46 lines
735 B
Markdown
46 lines
735 B
Markdown
# local-ai-test
|
|
|
|
## Setup Python virtual environment
|
|
|
|
Setup via `uv`:
|
|
|
|
```shell
|
|
uv venv --seed --python 3.12
|
|
```
|
|
|
|
```shell
|
|
source .venv/bin/activate
|
|
```
|
|
|
|
```shell
|
|
pip install -U openai
|
|
```
|
|
|
|
## How to run?
|
|
|
|
Export environments then run:
|
|
|
|
```shell
|
|
uv run main.py
|
|
```
|
|
|
|
Run with `tiny-encrypt` environments:
|
|
|
|
```shell
|
|
tiny-encrypt -X -k $(cat ~/.config/envs/LCOAL_DEFAULT_TINY_ENCRYPT_KEY_ID) \
|
|
~/secrets/alibaba-cloud-ai-cn.env.tinyenc -- go run main.go
|
|
```
|
|
|
|
Use local llm-proxy:
|
|
|
|
```shell
|
|
OPENAI_API_BASE=http://127.0.0.1:8080/ python main.py
|
|
|
|
OPENAI_API_BASE=http://127.0.0.1:8080/ python main-stream.py
|
|
```
|
|
|
|
## Reference
|
|
|
|
- https://help.aliyun.com/zh/model-studio/stream - 大模型服务平台百炼 / 文本生成 / 流式输出
|
|
|