mrleo1nid/gpu-rent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update GPU performance tuning and documentation for context length

Browse Source 
- Enhanced the `OllamaTune` class to include a new `context_length` attribute, improving the configuration for different GPU tiers.
- Updated performance tuning logic to set appropriate context lengths for low, mid, high, and ultra tiers, ensuring optimal resource allocation.
- Modified installation scripts to reflect the new context length settings, enhancing the installation process for Ollama.
- Revised documentation to include context length details in the GPU performance table, providing clearer guidance for users.
- Added tests to validate the correct context length settings in various scenarios, ensuring robustness in performance tuning.
This commit is contained in:
Leonid Pershin
2026-08-21 20:05:33 +03:00
parent 5832c5cf75
commit a882964ce0
4 changed files with 39 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/llm.md
+7 -7
View File
@@ -100,14 +100,14 @@ $env:OLLAMA_HOST = "http://127.0.0.1:17811"
Unit `gpu-rent-ollama` читает `/mnt/swarm_data/.gpu-rent-gpu.json`:
| Tier (VRAM) | Flash Attn | KEEP_ALIVE | KV cache | GPU_OVERHEAD |
| --- | --- | --- | --- | --- |
| low (<16 GiB) | off | 2m | q4_0 | 6 GiB |
| mid (16–23) | on* | 5m | q8_0 | 10 GiB |
| high (24–47) | on* | 15m | q8_0 | 14 GiB |
| ultra (≥48) | on* | 30m | q8_0 | 20 GiB |
| Tier (VRAM) | Flash Attn | KEEP_ALIVE | KV cache | GPU_OVERHEAD | CONTEXT |
| --- | --- | --- | --- | --- | --- |
| low (<16 GiB) | off | 2m | q4_0 | 6 GiB | 8k |
| mid (16–23) | on* | 5m | q8_0 | 10 GiB | 16k |
| high (24–47) | on* | 15m | q8_0 | 14 GiB | 16k |
| ultra (≥48) | on* | 30m | q8_0 | 20 GiB | 32k |
\*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=1`, `MAX_LOADED_MODELS=1`. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
\*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=1`, `MAX_LOADED_MODELS=1`. Ollama default `num_ctx` is 4096; we set `OLLAMA_CONTEXT_LENGTH` so Assistent + vision fits. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
---