mrleo1nid/gpu-rent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Support Ollama use: chat|memory and parallel embed beside VL.

Browse Source 
Pull nomic-embed-text for Assistent memory, write ollama-roles.json, CPU Modelfile, and raise MAX_LOADED_MODELS/NUM_PARALLEL to 2.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Leonid Pershin
2026-08-21 22:47:03 +03:00
co-authored by Cursor
parent 44f46d8190
commit 4081890b4c
8 changed files with 188 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/llm.md
+13 -1
View File
@@ -109,7 +109,19 @@ Unit `gpu-rent-ollama` читает `/mnt/swarm_data/.gpu-rent-gpu.json`:
| high (24–47) | on* | 15m | q8_0 | 14 GiB | 16k |
| ultra (≥48) | on* | 30m | q8_0 | 20 GiB | 32k |
\*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=1`, `MAX_LOADED_MODELS=1`. Ollama default `num_ctx` is 4096; we set `OLLAMA_CONTEXT_LENGTH` so Assistent + vision fits. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
\*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=2`, `MAX_LOADED_MODELS=2` (chat VL + memory embed). Memory models use `use: memory` and a CPU Modelfile (`num_gpu 0`) so embed does not steal VRAM from the chat model. Ollama default `num_ctx` is 4096; we set `OLLAMA_CONTEXT_LENGTH` so Assistent + vision fits. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
`ollama-models.yaml` entries:
```yaml
- name: huihui_ai/qwen2.5-vl-abliterated:7b
use: chat
default: true
- name: nomic-embed-text
use: memory
```
Roles are written to `/mnt/swarm_data/Assistent/ollama-roles.json` for the Assistent extension (chat select vs memory select).
---