Enhance GPU probing and performance tuning in provisioning

- Introduced GPU probing functionality to gather and store GPU specifications in `/mnt/swarm_data/.gpu-rent-gpu.json`, aiding in performance tuning.
- Updated `install_ollama.sh` and `install_llamacpp.sh` to utilize GPU information for configuring optimal runtime parameters.
- Enhanced `provision.py` to include GPU probing and performance tuning logic, ensuring better resource allocation for LLM operations.
- Improved documentation in `decisions.md`, `llm.md`, and `swarmui.md` to reflect changes in GPU handling and performance tuning processes.
- Added new tests to validate the GPU probing and model resolution logic, ensuring robustness in handling various GPU configurations.
This commit is contained in:
Leonid Pershin
2026-08-21 06:10:24 +03:00
parent 603165a4ba
commit 2ccb03f7d2
16 changed files with 1270 additions and 138 deletions
+19
View File
@@ -84,6 +84,23 @@ Community abliterate-модели без гарантий безопасност
---
## Автотюнинг Ollama под GPU
На установке `gpu-rent-ollama` читает `/mnt/swarm_data/.gpu-rent-gpu.json` и пишет env в systemd unit (одна карта вместе со SwarmUI):
| Tier (VRAM) | Flash Attn | KEEP_ALIVE | KV cache | GPU_OVERHEAD (запас под Swarm/Krea) |
| --- | --- | --- | --- | --- |
| low (<16GiB) | off | 2m | q4_0 | 6GiB |
| mid (1623) | on* | 5m | q8_0 | 10GiB |
| high (2447) | on* | 15m | q8_0 | 14GiB |
| ultra (≥48) | on* | 30m | q8_0 | 20GiB |
\*Flash на Ampere+ (compute ≥ 8.0). Всегда `NUM_PARALLEL=1`, `MAX_LOADED_MODELS=1`.
Файл: `/mnt/swarm_data/.gpu-rent-ollama.env` (пересоздаётся на каждом install Ollama).
---
## llama.cpp
1. `LLM_RUNTIME=llamacpp` или `up --llamacpp`
@@ -93,6 +110,8 @@ Community abliterate-модели без гарантий безопасност
Без GGUF unit может стартовать, но API бесполезен — смотри `gpu-rent logs` / `journalctl -u gpu-rent-llamacpp`.
Параметры `-ngl` / `-c` ставятся по тому же GPU probe (full offload на mid+, меньше слоёв и ctx на low).
---
## Idle-killer и LLM