Enhance GPU probing and performance tuning in provisioning
- Introduced GPU probing functionality to gather and store GPU specifications in `/mnt/swarm_data/.gpu-rent-gpu.json`, aiding in performance tuning. - Updated `install_ollama.sh` and `install_llamacpp.sh` to utilize GPU information for configuring optimal runtime parameters. - Enhanced `provision.py` to include GPU probing and performance tuning logic, ensuring better resource allocation for LLM operations. - Improved documentation in `decisions.md`, `llm.md`, and `swarmui.md` to reflect changes in GPU handling and performance tuning processes. - Added new tests to validate the GPU probing and model resolution logic, ensuring robustness in handling various GPU configurations.
This commit is contained in:
+19
@@ -84,6 +84,23 @@ Community abliterate-модели без гарантий безопасност
|
||||
|
||||
---
|
||||
|
||||
## Автотюнинг Ollama под GPU
|
||||
|
||||
На установке `gpu-rent-ollama` читает `/mnt/swarm_data/.gpu-rent-gpu.json` и пишет env в systemd unit (одна карта вместе со SwarmUI):
|
||||
|
||||
| Tier (VRAM) | Flash Attn | KEEP_ALIVE | KV cache | GPU_OVERHEAD (запас под Swarm/Krea) |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| low (<16 GiB) | off | 2m | q4_0 | 6 GiB |
|
||||
| mid (16–23) | on* | 5m | q8_0 | 10 GiB |
|
||||
| high (24–47) | on* | 15m | q8_0 | 14 GiB |
|
||||
| ultra (≥48) | on* | 30m | q8_0 | 20 GiB |
|
||||
|
||||
\*Flash на Ampere+ (compute ≥ 8.0). Всегда `NUM_PARALLEL=1`, `MAX_LOADED_MODELS=1`.
|
||||
|
||||
Файл: `/mnt/swarm_data/.gpu-rent-ollama.env` (пересоздаётся на каждом install Ollama).
|
||||
|
||||
---
|
||||
|
||||
## llama.cpp
|
||||
|
||||
1. `LLM_RUNTIME=llamacpp` или `up --llamacpp`
|
||||
@@ -93,6 +110,8 @@ Community abliterate-модели без гарантий безопасност
|
||||
|
||||
Без GGUF unit может стартовать, но API бесполезен — смотри `gpu-rent logs` / `journalctl -u gpu-rent-llamacpp`.
|
||||
|
||||
Параметры `-ngl` / `-c` ставятся по тому же GPU probe (full offload на mid+, меньше слоёв и ctx на low).
|
||||
|
||||
---
|
||||
|
||||
## Idle-killer и LLM
|
||||
|
||||
Reference in New Issue
Block a user