mrleo1nid/gpu-rent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Seed Assistent personas as overlay folders and tighten Ollama/Assistent glue.

Browse Source 
gpu-rent now writes personas/<id>/ on the VM (not legacy personas.json), adds seed-personas/doctor checks, and shortens mid/high keep-alive now that Assistent parks the LLM before Generate.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Leonid Pershin
2026-08-22 01:00:14 +03:00
co-authored by Cursor
parent 789fa26918
commit 03ba4cb6ed
22 changed files with 430 additions and 53 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/llm.md
+14 -7
View File
@@ -81,18 +81,20 @@ $env:OLLAMA_HOST = "http://127.0.0.1:17811"
После успешного тега — **warmup**: 1-token `POST /api/chat`, чтобы веса (~6 GB VL) легли в VRAM до первого сообщения в Assistent. То же при `gpu-rent tunnel`, если `/api/ps` пуст. Промах warmup — warning, GPU не гасим.
Qwen3-VL (`huihui_ai/qwen3-vl-abliterated:…`) требует **Ollama ≥ 0.12.7**. Тег `:8b-instruct` — чат Instruct; `:latest` у этой библиотеки — Thinking, не ставить дефолтом.
### Пресеты (меню)
| # | ключ | tag / смысл |
| --- | --- | --- |
| 1 | **recommended** | `huihui_ai/qwen2.5-vl-abliterated:7b` — RU + vision, ~6 GB |
| 1 | **recommended** | `huihui_ai/qwen3-vl-abliterated:8b-instruct` — RU + vision, ~6.1 GB; запасной `huihui_ai/qwen2.5-vl-abliterated:7b` |
| 2 | light | `…:3b` — то же, мало VRAM (~3 GB) |
| 3 | text | `huihui_ai/qwen2.5-abliterate:7b` — RU, без vision (~5 GB) |
| 4 | big | `huihui_ai/qwen2.5-vl-abliterated:32b` — RU + vision (~21 GB) |
| 5 | empty | только runtime |
| — | keep | не трогать yaml |
Все пресеты (кроме `empty`) — **abliterated** Qwen2.5 с нормальным русским. Официальные censored-теги (`qwen2.5vl:…`) в меню нет.
Рекомендуемый пресет — **abliterated Qwen3-VL Instruct**; light / text / big пока Qwen2.5. Официальные censored-теги (`qwen2.5vl:…`) в меню нет.
`default: true` в yaml — preferred в логе; pull идёт по всему списку.
@@ -105,18 +107,22 @@ Unit `gpu-rent-ollama` читает `/mnt/swarm_data/.gpu-rent-gpu.json`:
| Tier (VRAM) | Flash Attn | KEEP_ALIVE | KV cache | GPU_OVERHEAD | CONTEXT |
| --- | --- | --- | --- | --- | --- |
| low (&lt;16 GiB) | off | 2m | q4_0 | 6 GiB | 8k |
| mid (16–23) | on* | 15m | q8_0 | 10 GiB | 16k |
| high (24–47) | on* | 15m | q8_0 | 14 GiB | 16k |
| mid (16–23) | on* | 5m | q8_0 | 10 GiB | 16k |
| high (24–47) | on* | 5m | q8_0 | 14 GiB | 16k |
| ultra (≥48) | on* | 30m | q8_0 | 20 GiB | 32k |
\*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=2`, `MAX_LOADED_MODELS=2` (chat VL + memory embed). Memory models use `use: memory` and a CPU Modelfile (`num_gpu 0`) so embed does not steal VRAM from the chat model. Ollama default `num_ctx` is 4096; we set `OLLAMA_CONTEXT_LENGTH` so Assistent + vision fits. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
\*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=2`, `MAX_LOADED_MODELS=2` (chat VL + memory embed). Memory models use `use: memory` and a CPU Modelfile (`num_gpu 0`) so embed does not steal VRAM from the chat model. Ollama default `num_ctx` is 4096; we set `OLLAMA_CONTEXT_LENGTH` so Assistent + vision fits. Mid/high keep-alive is **5m** because Assistent parks the chat model before Generate (`AssistentParkLlm`) and warms after. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
Personas: laptop `assistent-personas.yaml` → on `up` / `seed-personas` overlay folders under `/mnt/swarm_data/Assistent/personas/<id>/` (+ `_base/assistant.json` with `default_persona` / `num_ctx`). See [extensions.md](extensions.md).
`ollama-models.yaml` entries:
```yaml
- name: huihui_ai/qwen2.5-vl-abliterated:7b
- name: huihui_ai/qwen3-vl-abliterated:8b-instruct
use: chat
default: true
- name: huihui_ai/qwen2.5-vl-abliterated:7b
use: chat
- name: nomic-embed-text
use: memory
```
@@ -129,7 +135,8 @@ Roles are written to `/mnt/swarm_data/Assistent/ollama-roles.json` for the Assis
Busy (не гасить GPU):
- `ollama pull` (маркер младше ~45 мин; старше сбрасывается);
- очередь / loading SwarmUI (Generate) — даже если Ollama `/api/ps` пуст после ParkLlm;
- `ollama pull` (маркер младше ~3 ч; refresh во время pull);
- загруженная модель в Ollama.
Ошибка установки LLM на `up` — **fail** (не тихий лог).