Seed Assistent personas as overlay folders and tighten Ollama/Assistent glue.
gpu-rent now writes personas/<id>/ on the VM (not legacy personas.json), adds seed-personas/doctor checks, and shortens mid/high keep-alive now that Assistent parks the LLM before Generate. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -1,6 +1,8 @@
|
||||
# Copy to assistent-personas.yaml (never commit the working copy).
|
||||
# Synced to the VM on each up → /mnt/swarm_data/Assistent/personas.yaml + personas.json
|
||||
# Assistent reads personas.json (overlay). Yaml is the laptop source for gpu-rent seed.
|
||||
# Synced to the VM on each up / seed-personas →
|
||||
# /mnt/swarm_data/Assistent/personas/<id>/{persona.json,extra.md}
|
||||
# /mnt/swarm_data/Assistent/_base/assistant.json (default_persona, num_ctx)
|
||||
# Yaml is the laptop source for gpu-rent seed; Assistent reads overlay folders.
|
||||
|
||||
default: neutral
|
||||
|
||||
|
||||
@@ -18,6 +18,8 @@
|
||||
|
||||
Файл маленький (danbooru.csv ≈ 3–4 MB). Killer из‑за него не откладываем, но на первом bootstrap качаем **до** старта UI, вместе с extensions.
|
||||
|
||||
Swarm Assistent индексирует тот же csv в FTS-таблицу `tags` внутри `Assistent/memory/assistent.sqlite` (без эмбедов). Первый чат после смены sha занимает несколько секунд; дальше `lookup_tags` ищет канон/алиасы. В `memory_hits` csv не попадает.
|
||||
|
||||
## Откуда файл
|
||||
|
||||
Дефолт — то, на что ссылается сама документация SwarmUI:
|
||||
|
||||
@@ -75,6 +75,7 @@ gpu-rent up --yes --ollama
|
||||
| `gpu-rent ssh` | Оболочка на VM |
|
||||
| `gpu-rent seed-models` | Докачать новые строки Civitai-манифеста на живой диск |
|
||||
| `gpu-rent seed-extensions` | Доклонировать/обновить git-репы; restart swarmui |
|
||||
| `gpu-rent seed-personas` | Пуш `assistent-personas.yaml` → overlay на VM (без полного up) |
|
||||
| `gpu-rent push` / `push-models` | Локальные деревья → VM |
|
||||
| `gpu-rent pull-output` | VM `Output/` → `./Output` |
|
||||
| `gpu-rent capture` / `capture all` | Инвентарь VM → merge **ссылок** в `models.yaml` + `extensions.yaml` (веса не качать) |
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@
|
||||
| Civitai хост | Дефолт API `civitai.red` (полный каталог). `.com` — SFW-витрина, NSFW с неё часто 404. Ссылки `.com`/`.red`/`.green` в манифесте принимаем. 404 → один retry на второй хост. Один токен на оба домена |
|
||||
| Пул GPU | Перед `up`/`flavors` сканируем `SCAN_POOLS` (дефолт `ru-6,ru-7`) по compute URL. Сегменты `ru-6a/b/c` берём из Nova `availability_zones`, не отдельными пулами. Пресет размера `FLAVOR_SIZE_PRESET` (дефолт `cheap` = 1×GPU, min vCPU/RAM). Диск уже есть — AZ диска, не дешёвый чужой сегмент. Первый запуск — дешёвый SKU в сегменте, где есть карта из `FLAVOR_PREFERENCE`. `.env` автоматом не пишем |
|
||||
| Манифест моделей | `<repo>/models.yaml`, типы: checkpoint / lora / vae / embedding / controlnet / upscaler. В git только `models.example.yaml` |
|
||||
| Расширения | `<repo>/extensions.yaml`: git-репы `swarmui` → `src/Extensions`, `comfy` → DLNodes. Поле `requires: none\|ollama\|any-llm` фильтрует по `LLM_RUNTIME`. Пример: swarm-assistent с `requires: ollama` |
|
||||
| Расширения | `<repo>/extensions.yaml`: git-репы `swarmui` → `src/Extensions`, `comfy` → DLNodes. Поле `requires: none\|ollama\|any-llm` фильтрует по `LLM_RUNTIME`. Пример: swarm-assistent с `requires: ollama`. Personas: `assistent-personas.yaml` → overlay `Assistent/personas/<id>/` на `up` / `seed-personas` |
|
||||
| Autocomplete | До первого старта: скачать word-list в `Data/Autocompletions`, прописать `DefaultUser.AutoComplete.Source`. На каждом `up` сверить GitHub blob sha и обновить файл, если изменился. Дефолт: `tags/danbooru.csv` из a1111-sd-webui-tagcomplete (как в доке SwarmUI) |
|
||||
| Доступ | Браузер на туннеле; MCP переключается на облако, пока оно живо; HTTP API SwarmUI через тот же туннель |
|
||||
| Preemptible | По умолчанию всегда. Обычный сервер — только `--no-spot` |
|
||||
|
||||
+1
-1
@@ -53,7 +53,7 @@ comfy:
|
||||
| `ollama` | только при `LLM_RUNTIME=ollama` |
|
||||
| `any-llm` | при `LLM_RUNTIME=ollama` |
|
||||
|
||||
Строки с несовпавшим `requires` пропускаются (лог), остальные ставятся как обычно. В `extensions.example.yaml` по умолчанию — **swarm-assistent** с `requires: ollama` (чат + доска Generate/Ref + persona/Cards, ветка `main`). На `up` сидится `assistent-personas.yaml` → `/mnt/swarm_data/Assistent/personas.yaml` **и** `personas.json` (Assistent читает только **json**).
|
||||
Строки с несовпавшим `requires` пропускаются (лог), остальные ставятся как обычно. В `extensions.example.yaml` по умолчанию — **swarm-assistent** с `requires: ollama` (чат + доска Generate/Ref + persona/Cards, ветка `main`). На `up` (и `gpu-rent seed-personas`) сидится `assistent-personas.yaml` → overlay `/mnt/swarm_data/Assistent/personas/<id>/persona.json` + `extra.md`, плюс `_base/assistant.json` (`default_persona`, опционально `num_ctx` с GPU tier). Legacy `personas.json` больше не пишется.
|
||||
|
||||
В git репозитория gpu-rent не коммитить рабочий список с лишними приватными URL сверх примера.
|
||||
|
||||
|
||||
+14
-7
@@ -81,18 +81,20 @@ $env:OLLAMA_HOST = "http://127.0.0.1:17811"
|
||||
|
||||
После успешного тега — **warmup**: 1-token `POST /api/chat`, чтобы веса (~6 GB VL) легли в VRAM до первого сообщения в Assistent. То же при `gpu-rent tunnel`, если `/api/ps` пуст. Промах warmup — warning, GPU не гасим.
|
||||
|
||||
Qwen3-VL (`huihui_ai/qwen3-vl-abliterated:…`) требует **Ollama ≥ 0.12.7**. Тег `:8b-instruct` — чат Instruct; `:latest` у этой библиотеки — Thinking, не ставить дефолтом.
|
||||
|
||||
### Пресеты (меню)
|
||||
|
||||
| # | ключ | tag / смысл |
|
||||
| --- | --- | --- |
|
||||
| 1 | **recommended** | `huihui_ai/qwen2.5-vl-abliterated:7b` — RU + vision, ~6 GB |
|
||||
| 1 | **recommended** | `huihui_ai/qwen3-vl-abliterated:8b-instruct` — RU + vision, ~6.1 GB; запасной `huihui_ai/qwen2.5-vl-abliterated:7b` |
|
||||
| 2 | light | `…:3b` — то же, мало VRAM (~3 GB) |
|
||||
| 3 | text | `huihui_ai/qwen2.5-abliterate:7b` — RU, без vision (~5 GB) |
|
||||
| 4 | big | `huihui_ai/qwen2.5-vl-abliterated:32b` — RU + vision (~21 GB) |
|
||||
| 5 | empty | только runtime |
|
||||
| — | keep | не трогать yaml |
|
||||
|
||||
Все пресеты (кроме `empty`) — **abliterated** Qwen2.5 с нормальным русским. Официальные censored-теги (`qwen2.5vl:…`) в меню нет.
|
||||
Рекомендуемый пресет — **abliterated Qwen3-VL Instruct**; light / text / big пока Qwen2.5. Официальные censored-теги (`qwen2.5vl:…`) в меню нет.
|
||||
|
||||
`default: true` в yaml — preferred в логе; pull идёт по всему списку.
|
||||
|
||||
@@ -105,18 +107,22 @@ Unit `gpu-rent-ollama` читает `/mnt/swarm_data/.gpu-rent-gpu.json`:
|
||||
| Tier (VRAM) | Flash Attn | KEEP_ALIVE | KV cache | GPU_OVERHEAD | CONTEXT |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| low (<16 GiB) | off | 2m | q4_0 | 6 GiB | 8k |
|
||||
| mid (16–23) | on* | 15m | q8_0 | 10 GiB | 16k |
|
||||
| high (24–47) | on* | 15m | q8_0 | 14 GiB | 16k |
|
||||
| mid (16–23) | on* | 5m | q8_0 | 10 GiB | 16k |
|
||||
| high (24–47) | on* | 5m | q8_0 | 14 GiB | 16k |
|
||||
| ultra (≥48) | on* | 30m | q8_0 | 20 GiB | 32k |
|
||||
|
||||
\*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=2`, `MAX_LOADED_MODELS=2` (chat VL + memory embed). Memory models use `use: memory` and a CPU Modelfile (`num_gpu 0`) so embed does not steal VRAM from the chat model. Ollama default `num_ctx` is 4096; we set `OLLAMA_CONTEXT_LENGTH` so Assistent + vision fits. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
|
||||
\*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=2`, `MAX_LOADED_MODELS=2` (chat VL + memory embed). Memory models use `use: memory` and a CPU Modelfile (`num_gpu 0`) so embed does not steal VRAM from the chat model. Ollama default `num_ctx` is 4096; we set `OLLAMA_CONTEXT_LENGTH` so Assistent + vision fits. Mid/high keep-alive is **5m** because Assistent parks the chat model before Generate (`AssistentParkLlm`) and warms after. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
|
||||
|
||||
Personas: laptop `assistent-personas.yaml` → on `up` / `seed-personas` overlay folders under `/mnt/swarm_data/Assistent/personas/<id>/` (+ `_base/assistant.json` with `default_persona` / `num_ctx`). See [extensions.md](extensions.md).
|
||||
|
||||
`ollama-models.yaml` entries:
|
||||
|
||||
```yaml
|
||||
- name: huihui_ai/qwen2.5-vl-abliterated:7b
|
||||
- name: huihui_ai/qwen3-vl-abliterated:8b-instruct
|
||||
use: chat
|
||||
default: true
|
||||
- name: huihui_ai/qwen2.5-vl-abliterated:7b
|
||||
use: chat
|
||||
- name: nomic-embed-text
|
||||
use: memory
|
||||
```
|
||||
@@ -129,7 +135,8 @@ Roles are written to `/mnt/swarm_data/Assistent/ollama-roles.json` for the Assis
|
||||
|
||||
Busy (не гасить GPU):
|
||||
|
||||
- `ollama pull` (маркер младше ~45 мин; старше сбрасывается);
|
||||
- очередь / loading SwarmUI (Generate) — даже если Ollama `/api/ps` пуст после ParkLlm;
|
||||
- `ollama pull` (маркер младше ~3 ч; refresh во время pull);
|
||||
- загруженная модель в Ollama.
|
||||
|
||||
Ошибка установки LLM на `up` — **fail** (не тихий лог).
|
||||
|
||||
@@ -2,13 +2,18 @@
|
||||
# name = exact tag for `ollama pull`.
|
||||
# use: chat — Assistent header select; use: memory — settings memory model.
|
||||
# Requirement: uncensored (abliterated) + Russian for chat. Empty models: [] → no pull.
|
||||
# Qwen3-VL tags need Ollama ≥ 0.12.7. Pin :8b-instruct — :latest is Thinking.
|
||||
|
||||
models:
|
||||
# Recommended (~6GB): vision + RU/EN, abliterated
|
||||
- name: huihui_ai/qwen2.5-vl-abliterated:7b
|
||||
# Recommended (~6.1GB): Qwen3-VL Instruct, vision + RU/EN, abliterated
|
||||
- name: huihui_ai/qwen3-vl-abliterated:8b-instruct
|
||||
use: chat
|
||||
default: true
|
||||
|
||||
# Previous default — still in Assistent chat select
|
||||
- name: huihui_ai/qwen2.5-vl-abliterated:7b
|
||||
use: chat
|
||||
|
||||
# Vector memory (CPU via num_gpu 0 Modelfile after pull) — not shown in chat select
|
||||
- name: nomic-embed-text
|
||||
use: memory
|
||||
|
||||
@@ -40,6 +40,11 @@ def collect_access_links(cfg: Config, *, tunneled: bool) -> list[AccessLink]:
|
||||
links.extend(
|
||||
[
|
||||
AccessLink("SwarmUI UI", base, "браузер"),
|
||||
AccessLink(
|
||||
"Assistent",
|
||||
base,
|
||||
"вкладка Assistent в SwarmUI",
|
||||
),
|
||||
AccessLink("SwarmUI API", f"{base}/API/", "HTTP JSON"),
|
||||
AccessLink("SwarmUI MCP", f"{base}/mcp", "Cursor mcp.json"),
|
||||
]
|
||||
|
||||
@@ -324,6 +324,16 @@ def status() -> None:
|
||||
table.add_row("idle-killer", f"{killer_line} · arm ok (сессия)")
|
||||
else:
|
||||
table.add_row("idle-killer", killer_line)
|
||||
try:
|
||||
from gpu_rent.provision import count_wanted_models_on_vm
|
||||
|
||||
wanted_n = count_wanted_models_on_vm(cfg, state.floating_ip)
|
||||
table.add_row(
|
||||
"Assistent wanted",
|
||||
f"{wanted_n} в очереди" if wanted_n else "пусто",
|
||||
)
|
||||
except Exception:
|
||||
table.add_row("Assistent wanted", "—")
|
||||
except GpuRentError as exc:
|
||||
table.add_row("диск used/free", f"SSH: {exc}")
|
||||
table.add_row("idle-killer", "нет SSH")
|
||||
@@ -932,6 +942,18 @@ def seed_extensions_cmd() -> None:
|
||||
_die(exc)
|
||||
|
||||
|
||||
@app.command("seed-personas")
|
||||
def seed_personas_cmd() -> None:
|
||||
"""Push assistent-personas.yaml → VM Assistent overlay (без полного up)."""
|
||||
try:
|
||||
from gpu_rent.provision import seed_assistent_personas
|
||||
|
||||
cfg, host = _live()
|
||||
seed_assistent_personas(cfg, host, log)
|
||||
except GpuRentError as exc:
|
||||
_die(exc)
|
||||
|
||||
|
||||
capture_app = typer.Typer(
|
||||
help=(
|
||||
"Снять с VM инвентарь → локальные манифесты (только ссылки, без весов). "
|
||||
|
||||
+63
-1
@@ -15,7 +15,7 @@ from gpu_rent.inventory import (
|
||||
pick_volume_type,
|
||||
rank_flavors,
|
||||
)
|
||||
from gpu_rent.manifests import parse_extensions, parse_models
|
||||
from gpu_rent.manifests import parse_extensions, parse_models, repo_dirname, repo_matches_runtime
|
||||
from gpu_rent.os_client import (
|
||||
compute_quotas,
|
||||
connect,
|
||||
@@ -243,10 +243,72 @@ def run_doctor() -> list[Check]:
|
||||
_civitai(cfg, checks)
|
||||
_huggingface(cfg, checks)
|
||||
_local_manifests(cfg, checks)
|
||||
_llm_assistent(cfg, checks)
|
||||
_local_folders(cfg, checks)
|
||||
return checks
|
||||
|
||||
|
||||
def _llm_assistent(cfg: Config, checks: list[Check]) -> None:
|
||||
"""When Ollama is on, warn if Assistent stack pieces are missing locally."""
|
||||
from gpu_rent.llm_runtime import normalize_runtime, parse_ollama_models
|
||||
from gpu_rent.paths import ollama_models_manifest_path
|
||||
|
||||
runtime = normalize_runtime(getattr(cfg, "llm_runtime", "none"))
|
||||
if runtime != "ollama":
|
||||
checks.append(Check("LLM / Assistent", True, False, f"LLM_RUNTIME={runtime}"))
|
||||
return
|
||||
|
||||
ollama_path = Path(
|
||||
getattr(cfg, "ollama_models_manifest", None) or ollama_models_manifest_path()
|
||||
)
|
||||
if not ollama_path.is_file():
|
||||
checks.append(
|
||||
Check(
|
||||
"ollama-models.yaml",
|
||||
False,
|
||||
True,
|
||||
f"нет {ollama_path} — Assistent будет пустой. gpu-rent setup / скопируй example",
|
||||
)
|
||||
)
|
||||
else:
|
||||
try:
|
||||
entries = parse_ollama_models(ollama_path)
|
||||
chat = [e for e in entries if getattr(e, "use", "chat") != "memory"]
|
||||
checks.append(
|
||||
Check(
|
||||
"ollama-models.yaml",
|
||||
True,
|
||||
True,
|
||||
f"{len(entries)} тег(ов), chat={len(chat)}",
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
checks.append(Check("ollama-models.yaml", False, True, str(exc)))
|
||||
|
||||
try:
|
||||
repos = parse_extensions(cfg.extensions_manifest)
|
||||
except ConfigError as exc:
|
||||
checks.append(Check("Assistent ext", False, True, str(exc)))
|
||||
return
|
||||
has_assistent = any(
|
||||
"assistent" in repo_dirname(r).lower()
|
||||
or "assistent" in (r.url or "").lower()
|
||||
for r in repos
|
||||
if repo_matches_runtime(r, runtime)
|
||||
)
|
||||
if has_assistent:
|
||||
checks.append(Check("Assistent ext", True, False, "swarm-assistent в extensions.yaml"))
|
||||
else:
|
||||
checks.append(
|
||||
Check(
|
||||
"Assistent ext",
|
||||
True,
|
||||
False,
|
||||
"нет swarm-assistent (requires:ollama) в extensions.yaml — чат не поставится",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _civitai(cfg: Config, checks: list[Check]) -> None:
|
||||
if not cfg.civitai_api_token:
|
||||
checks.append(
|
||||
|
||||
@@ -21,10 +21,13 @@ VALID_RUNTIMES = frozenset({"none", "ollama"})
|
||||
MEMORY_EMBED_MODEL = "nomic-embed-text"
|
||||
|
||||
OLLAMA_PRESETS: dict[str, list[str]] = {
|
||||
# Requirement: uncensored (abliterated) + solid Russian. Qwen2.5 family.
|
||||
# Vision tags preferred for SwarmUI prompt help with images.
|
||||
# Memory embed (nomic) is appended separately with use: memory.
|
||||
"recommended": ["huihui_ai/qwen2.5-vl-abliterated:7b"], # ~6 GB
|
||||
# Requirement: uncensored (abliterated) + solid Russian. Vision for prompt help.
|
||||
# Pin :8b-instruct — :latest on this library is Thinking, not chat.
|
||||
# Qwen3-VL needs Ollama ≥ 0.12.7. Memory embed (nomic) appended with use: memory.
|
||||
"recommended": [
|
||||
"huihui_ai/qwen3-vl-abliterated:8b-instruct", # ~6.1 GB, default
|
||||
"huihui_ai/qwen2.5-vl-abliterated:7b", # ~6 GB, second chat tag
|
||||
],
|
||||
"light": ["huihui_ai/qwen2.5-vl-abliterated:3b"], # ~3 GB
|
||||
"text": ["huihui_ai/qwen2.5-abliterate:7b"], # ~5 GB, no vision
|
||||
"big": ["huihui_ai/qwen2.5-vl-abliterated:32b"], # ~21 GB
|
||||
@@ -32,7 +35,7 @@ OLLAMA_PRESETS: dict[str, list[str]] = {
|
||||
}
|
||||
|
||||
OLLAMA_PRESET_LABELS: dict[str, str] = {
|
||||
"recommended": "VL 7B abliterate — RU + картинки (~6GB)",
|
||||
"recommended": "VL 8B Instruct abliterate — RU + картинки (~6GB)",
|
||||
"light": "VL 3B abliterate — RU + картинки, мало VRAM (~3GB)",
|
||||
"text": "7B abliterate text — RU, без vision (~5GB)",
|
||||
"big": "VL 32B abliterate — RU + картинки (~21GB)",
|
||||
@@ -40,10 +43,8 @@ OLLAMA_PRESET_LABELS: dict[str, str] = {
|
||||
"keep": "не менять ollama-models.yaml",
|
||||
}
|
||||
|
||||
# Deprecated text blob — prefer menu helpers below.
|
||||
PRESET_HELP = "\n".join(
|
||||
f"{k} — {v}" for k, v in OLLAMA_PRESET_LABELS.items() if k != "keep"
|
||||
)
|
||||
# Deprecated: use ollama_preset_menu / OLLAMA_PRESET_LABELS
|
||||
PRESET_HELP = "" # kept empty; menus use OLLAMA_PRESET_LABELS
|
||||
|
||||
LLM_RUNTIME_LABELS: dict[str, str] = {
|
||||
"none": "только SwarmUI",
|
||||
|
||||
@@ -77,16 +77,16 @@ def ollama_tune_for(info: GpuInfo) -> OllamaTune:
|
||||
note = "ultra: flash+q8 KV, 20GiB reserved for Swarm, ctx 32k, keep 30m"
|
||||
elif info.tier == TIER_HIGH:
|
||||
overhead = 14 * 1024**3
|
||||
keep = "15m"
|
||||
keep = "5m" # Assistent parks LLM before Generate; short keep is enough
|
||||
kv = "q8_0"
|
||||
ctx = 16384
|
||||
note = "high: flash+q8 KV, 14GiB reserved for Swarm, ctx 16k, keep 15m"
|
||||
note = "high: flash+q8 KV, 14GiB reserved for Swarm, ctx 16k, keep 5m"
|
||||
elif info.tier == TIER_MID:
|
||||
overhead = 10 * 1024**3
|
||||
keep = "15m"
|
||||
keep = "5m"
|
||||
kv = "q8_0"
|
||||
ctx = 16384
|
||||
note = "mid: flash+q8 KV, 10GiB reserved for Swarm, ctx 16k, keep 15m"
|
||||
note = "mid: flash+q8 KV, 10GiB reserved for Swarm, ctx 16k, keep 5m"
|
||||
else:
|
||||
overhead = 6 * 1024**3
|
||||
keep = "2m"
|
||||
|
||||
+136
-15
@@ -710,8 +710,55 @@ def seed_swarmui_api_keys(cfg: Config, host: str, log: Log) -> None:
|
||||
run_ssh(cfg, host, "rm -f /tmp/gpu-rent-swarm-api-keys.json", check=False)
|
||||
|
||||
|
||||
def _safe_persona_id(raw: object) -> str | None:
|
||||
s = str(raw or "").strip().replace("\\", "/")
|
||||
if not s or ".." in s or "/" in s:
|
||||
return None
|
||||
import re
|
||||
|
||||
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_\-]{0,63}", s):
|
||||
return None
|
||||
return s
|
||||
|
||||
|
||||
def _overlay_num_ctx(cfg: Config, host: str) -> int | None:
|
||||
"""Prefer GPU-tier context from .gpu-rent-gpu.json when present."""
|
||||
raw = run_ssh(
|
||||
cfg,
|
||||
host,
|
||||
f"test -f {DATA}/.gpu-rent-gpu.json && cat {DATA}/.gpu-rent-gpu.json || true",
|
||||
check=False,
|
||||
timeout=15,
|
||||
).strip()
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
probe = json.loads(raw)
|
||||
vram = int(probe.get("vram_mib") or probe.get("memory_total_mib") or 0)
|
||||
except (json.JSONDecodeError, TypeError, ValueError):
|
||||
return None
|
||||
if vram <= 0:
|
||||
return None
|
||||
from gpu_rent.perf_tiers import GpuInfo, ollama_tune_for, tier_for_vram_mib
|
||||
|
||||
info = GpuInfo(
|
||||
name=str(probe.get("name") or "gpu"),
|
||||
vram_mib=vram,
|
||||
compute_cap=str(probe.get("compute_cap") or "0.0"),
|
||||
uuid=str(probe.get("uuid") or ""),
|
||||
tier=tier_for_vram_mib(vram),
|
||||
)
|
||||
return ollama_tune_for(info).context_length
|
||||
|
||||
|
||||
def seed_assistent_personas(cfg: Config, host: str, log: Log) -> None:
|
||||
"""Push local assistent-personas.yaml → VM Assistent/personas.yaml + personas.json."""
|
||||
"""Push local assistent-personas.yaml → VM Assistent/personas/<id>/ overlay.
|
||||
|
||||
Laptop yaml stays the editor; on VM we write persona.json + extra.md and
|
||||
overlay assistant.json (default_persona, optional num_ctx). Does not clobber
|
||||
voice/likes/dislikes/rules already on disk. No longer writes legacy
|
||||
personas.json / personas.yaml dumps.
|
||||
"""
|
||||
from gpu_rent.paths import assistent_personas_example_path, assistent_personas_manifest_path
|
||||
|
||||
local = Path(
|
||||
@@ -728,24 +775,96 @@ def seed_assistent_personas(cfg: Config, host: str, log: Log) -> None:
|
||||
if not text.strip():
|
||||
log("assistent-personas: пустой файл — skip")
|
||||
return
|
||||
remote_dir = f"{DATA}/Assistent"
|
||||
remote_yaml = f"{remote_dir}/personas.yaml"
|
||||
remote_json = f"{remote_dir}/personas.json"
|
||||
run_ssh(cfg, host, f"mkdir -p {shlex.quote(remote_dir)}", check=False)
|
||||
put_text(cfg, host, remote_yaml, text if text.endswith("\n") else text + "\n")
|
||||
try:
|
||||
import yaml
|
||||
|
||||
data = yaml.safe_load(text) or {}
|
||||
put_text(
|
||||
cfg,
|
||||
host,
|
||||
remote_json,
|
||||
json.dumps(data, ensure_ascii=False, indent=2) + "\n",
|
||||
)
|
||||
except Exception as exc:
|
||||
log(f"assistent-personas: json convert — {exc}")
|
||||
log(f"assistent-personas → {remote_yaml}")
|
||||
log(f"assistent-personas: yaml parse — {exc}")
|
||||
return
|
||||
if not isinstance(data, dict):
|
||||
log("assistent-personas: корень должен быть mapping — skip")
|
||||
return
|
||||
|
||||
remote_dir = f"{DATA}/Assistent"
|
||||
run_ssh(cfg, host, f"mkdir -p {shlex.quote(remote_dir)}/personas {shlex.quote(remote_dir)}/_base", check=False)
|
||||
|
||||
default_id = _safe_persona_id(data.get("default")) or "neutral"
|
||||
personas = data.get("personas") or []
|
||||
written = 0
|
||||
if isinstance(personas, list):
|
||||
for row in personas:
|
||||
if not isinstance(row, dict):
|
||||
continue
|
||||
pid = _safe_persona_id(row.get("id"))
|
||||
if not pid:
|
||||
continue
|
||||
title = str(row.get("title") or pid).strip() or pid
|
||||
prompt = str(row.get("prompt") or "").strip()
|
||||
pdir = f"{remote_dir}/personas/{pid}"
|
||||
run_ssh(cfg, host, f"mkdir -p {shlex.quote(pdir)}", check=False)
|
||||
meta = {"title": title, "tagline": title, "accent": "#8b949e"}
|
||||
put_text(
|
||||
cfg,
|
||||
host,
|
||||
f"{pdir}/persona.json",
|
||||
json.dumps(meta, ensure_ascii=False, indent=2) + "\n",
|
||||
)
|
||||
if prompt:
|
||||
put_text(
|
||||
cfg,
|
||||
host,
|
||||
f"{pdir}/extra.md",
|
||||
prompt if prompt.endswith("\n") else prompt + "\n",
|
||||
)
|
||||
written += 1
|
||||
|
||||
assistant_overlay: dict = {"default_persona": default_id}
|
||||
num_ctx = _overlay_num_ctx(cfg, host)
|
||||
if num_ctx:
|
||||
assistant_overlay["num_ctx"] = num_ctx
|
||||
put_text(
|
||||
cfg,
|
||||
host,
|
||||
f"{remote_dir}/_base/assistant.json",
|
||||
json.dumps(assistant_overlay, ensure_ascii=False, indent=2) + "\n",
|
||||
)
|
||||
# Drop legacy dumps so Assistent does not double-inject prompts.
|
||||
run_ssh(
|
||||
cfg,
|
||||
host,
|
||||
f"rm -f {shlex.quote(remote_dir + '/personas.json')} "
|
||||
f"{shlex.quote(remote_dir + '/personas.yaml')}",
|
||||
check=False,
|
||||
)
|
||||
ctx_note = f", num_ctx={num_ctx}" if num_ctx else ""
|
||||
log(f"assistent-personas → overlay personas/{written} (default={default_id}{ctx_note})")
|
||||
|
||||
|
||||
def count_wanted_models_on_vm(cfg: Config, host: str) -> int:
|
||||
"""Count entries in Assistent wanted queue on the VM (best-effort)."""
|
||||
remote = f"{DATA}/.gpu-rent-wanted-models.yaml"
|
||||
raw = run_ssh(
|
||||
cfg,
|
||||
host,
|
||||
f"test -f {shlex.quote(remote)} && cat {shlex.quote(remote)} || true",
|
||||
check=False,
|
||||
timeout=20,
|
||||
).strip()
|
||||
if not raw:
|
||||
return 0
|
||||
try:
|
||||
import yaml
|
||||
|
||||
data = yaml.safe_load(raw) or {}
|
||||
except Exception:
|
||||
return sum(1 for line in raw.splitlines() if line.strip().startswith("- url:"))
|
||||
n = 0
|
||||
if isinstance(data, dict):
|
||||
for rows in data.values():
|
||||
if isinstance(rows, list):
|
||||
n += sum(1 for r in rows if isinstance(r, dict) and r.get("url"))
|
||||
return n
|
||||
|
||||
|
||||
def merge_wanted_models_from_vm(cfg: Config, host: str, log: Log) -> int:
|
||||
@@ -1072,6 +1191,8 @@ def provision_llm(cfg: Config, host: str, log: Log) -> None:
|
||||
)
|
||||
|
||||
# Drop LLM units that should not hold VRAM for this runtime.
|
||||
# Tombstone: still stop gpu-rent-llamacpp if an old disk left that unit behind
|
||||
# (llamacpp runtime was removed; do not reinstall it).
|
||||
if runtime == "none":
|
||||
log("LLM: none — останавливаю gpu-rent-ollama / gpu-rent-llamacpp если были")
|
||||
_stop_units("gpu-rent-ollama", "gpu-rent-llamacpp")
|
||||
@@ -1083,7 +1204,7 @@ def provision_llm(cfg: Config, host: str, log: Log) -> None:
|
||||
return
|
||||
still: list[str] = []
|
||||
if runtime == "ollama":
|
||||
_stop_units("gpu-rent-llamacpp")
|
||||
_stop_units("gpu-rent-llamacpp") # tombstone: disable leftover llamacpp unit
|
||||
log("LLM: ставим/запускаем Ollama")
|
||||
run_script_sudo(
|
||||
cfg,
|
||||
|
||||
@@ -129,7 +129,11 @@ def swarm_busy(swarm_url: str, timeout: float = 8.0) -> tuple[bool, str]:
|
||||
|
||||
|
||||
def llm_busy(timeout: float = 3.0) -> tuple[bool, str]:
|
||||
"""Ollama pull / loaded models count as busy."""
|
||||
"""Ollama pull / loaded models count as busy.
|
||||
|
||||
Empty /api/ps after AssistentParkLlm is NOT busy by itself — Swarm queue
|
||||
is checked separately in main() and keeps the GPU alive during Generate.
|
||||
"""
|
||||
pull_marker = DATA / ".gpu-rent-ollama-pulling"
|
||||
if pull_marker.is_file():
|
||||
try:
|
||||
|
||||
@@ -47,9 +47,9 @@ except ValueError:
|
||||
if gib >= 48:
|
||||
tier, overhead, keep, kv, flash, ctx = "ultra", 20 * 1024**3, "30m", "q8_0", True, 32768
|
||||
elif gib >= 24:
|
||||
tier, overhead, keep, kv, flash, ctx = "high", 14 * 1024**3, "15m", "q8_0", True, 16384
|
||||
tier, overhead, keep, kv, flash, ctx = "high", 14 * 1024**3, "5m", "q8_0", True, 16384
|
||||
elif gib >= 16:
|
||||
tier, overhead, keep, kv, flash, ctx = "mid", 10 * 1024**3, "15m", "q8_0", True, 16384
|
||||
tier, overhead, keep, kv, flash, ctx = "mid", 10 * 1024**3, "5m", "q8_0", True, 16384
|
||||
else:
|
||||
tier, overhead, keep, kv, flash, ctx = "low", 6 * 1024**3, "2m", "q4_0", False, 8192
|
||||
flash = bool(flash and (ampere or "A100" in name.upper() or "H100" in name.upper() or gib >= 16))
|
||||
|
||||
@@ -16,9 +16,10 @@ def test_collect_links_swarm_and_ollama(monkeypatch):
|
||||
links = collect_access_links(_Cfg(), tunneled=True)
|
||||
labels = [x.label for x in links]
|
||||
assert "SwarmUI UI" in labels
|
||||
assert "Assistent" in labels
|
||||
assert "SwarmUI MCP" in labels
|
||||
assert "Ollama API" in labels
|
||||
assert any("17811" in x.url for x in links)
|
||||
assert any(x.label == "Assistent" and "вкладка" in x.note for x in links)
|
||||
|
||||
|
||||
def test_collect_links_no_tunnel():
|
||||
@@ -57,7 +58,7 @@ def test_access_panel_hides_stale_llm_error_when_ollama_ok(monkeypatch):
|
||||
{
|
||||
"name": "ollama",
|
||||
"ok": True,
|
||||
"detail": "1 models (huihui_ai/qwen2.5-vl-abliterated:7b)",
|
||||
"detail": "1 models (huihui_ai/qwen3-vl-abliterated:8b-instruct)",
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Unit tests for assistent personas overlay seed helpers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from gpu_rent.provision import _safe_persona_id, count_wanted_models_on_vm, seed_assistent_personas
|
||||
|
||||
|
||||
def test_safe_persona_id():
|
||||
assert _safe_persona_id("neutral") == "neutral"
|
||||
assert _safe_persona_id("lewd_v2") == "lewd_v2"
|
||||
assert _safe_persona_id("../x") is None
|
||||
assert _safe_persona_id("a/b") is None
|
||||
assert _safe_persona_id("") is None
|
||||
|
||||
|
||||
def test_seed_assistent_personas_writes_overlay(tmp_path: Path, monkeypatch):
|
||||
yaml_path = tmp_path / "assistent-personas.yaml"
|
||||
yaml_path.write_text(
|
||||
"default: cinema\n"
|
||||
"personas:\n"
|
||||
" - id: cinema\n"
|
||||
" title: Кино\n"
|
||||
" prompt: |\n"
|
||||
" You are a DP.\n"
|
||||
" - id: neutral\n"
|
||||
" title: Нейтральный\n"
|
||||
" prompt: Calm.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
puts: dict[str, str] = {}
|
||||
ssh_cmds: list[str] = []
|
||||
|
||||
def fake_put(cfg, host, remote, text, mode=0o644):
|
||||
puts[remote] = text
|
||||
|
||||
def fake_ssh(cfg, host, cmd, check=False, timeout=60):
|
||||
ssh_cmds.append(cmd)
|
||||
if ".gpu-rent-gpu.json" in cmd:
|
||||
return json.dumps({"vram_mib": 24576, "name": "4090", "compute_cap": "8.9"})
|
||||
return ""
|
||||
|
||||
monkeypatch.setattr("gpu_rent.provision.put_text", fake_put)
|
||||
monkeypatch.setattr("gpu_rent.provision.run_ssh", fake_ssh)
|
||||
monkeypatch.setattr(
|
||||
"gpu_rent.paths.assistent_personas_manifest_path", lambda: yaml_path
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"gpu_rent.paths.assistent_personas_example_path", lambda: tmp_path / "missing"
|
||||
)
|
||||
|
||||
cfg = MagicMock()
|
||||
cfg.assistent_personas_manifest = str(yaml_path)
|
||||
logs: list[str] = []
|
||||
seed_assistent_personas(cfg, "1.2.3.4", logs.append)
|
||||
|
||||
assert any("personas/cinema/persona.json" in p for p in puts)
|
||||
assert any("personas/cinema/extra.md" in p for p in puts)
|
||||
assert "/mnt/swarm_data/Assistent/_base/assistant.json" in puts
|
||||
asst = json.loads(puts["/mnt/swarm_data/Assistent/_base/assistant.json"])
|
||||
assert asst["default_persona"] == "cinema"
|
||||
assert asst["num_ctx"] == 16384
|
||||
assert any("rm -f" in c and "personas.json" in c for c in ssh_cmds)
|
||||
assert not any(p.endswith("personas.yaml") for p in puts)
|
||||
assert not any(p.endswith("personas.json") for p in puts)
|
||||
assert any("overlay personas/2" in m for m in logs)
|
||||
|
||||
|
||||
def test_count_wanted_models(monkeypatch):
|
||||
raw = "lora:\n - url: https://civitai.red/x?modelVersionId=1\n title: A\n"
|
||||
|
||||
monkeypatch.setattr(
|
||||
"gpu_rent.provision.run_ssh",
|
||||
lambda *a, **k: raw,
|
||||
)
|
||||
assert count_wanted_models_on_vm(MagicMock(), "h") == 1
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Doctor LLM / Assistent local checks."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from gpu_rent.doctor import Check, _llm_assistent
|
||||
|
||||
|
||||
class _Cfg:
|
||||
llm_runtime = "ollama"
|
||||
ollama_models_manifest = None
|
||||
extensions_manifest = None
|
||||
|
||||
|
||||
def test_llm_assistent_warns_without_ollama_yaml(tmp_path: Path, monkeypatch):
|
||||
cfg = _Cfg()
|
||||
cfg.ollama_models_manifest = tmp_path / "missing-ollama.yaml"
|
||||
cfg.extensions_manifest = tmp_path / "extensions.yaml"
|
||||
cfg.extensions_manifest.write_text(
|
||||
"swarmui:\n"
|
||||
" - url: https://gitea.example/swarm-assistent.git\n"
|
||||
" dir: swarm-assistent\n"
|
||||
" requires: ollama\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"gpu_rent.paths.ollama_models_manifest_path",
|
||||
lambda: cfg.ollama_models_manifest,
|
||||
)
|
||||
checks: list[Check] = []
|
||||
_llm_assistent(cfg, checks)
|
||||
names = {c.name: c for c in checks}
|
||||
assert names["ollama-models.yaml"].ok is False
|
||||
assert names["ollama-models.yaml"].blocking is True
|
||||
assert names["Assistent ext"].ok is True
|
||||
|
||||
|
||||
def test_llm_assistent_ok_with_manifests(tmp_path: Path, monkeypatch):
|
||||
ollama = tmp_path / "ollama-models.yaml"
|
||||
ollama.write_text(
|
||||
"models:\n - name: foo:7b\n use: chat\n default: true\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
ext = tmp_path / "extensions.yaml"
|
||||
ext.write_text(
|
||||
"swarmui:\n"
|
||||
" - url: https://gitea.example/swarm-assistent.git\n"
|
||||
" dir: swarm-assistent\n"
|
||||
" requires: ollama\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
cfg = _Cfg()
|
||||
cfg.ollama_models_manifest = ollama
|
||||
cfg.extensions_manifest = ext
|
||||
cfg.llm_runtime = "ollama"
|
||||
checks: list[Check] = []
|
||||
_llm_assistent(cfg, checks)
|
||||
by = {c.name: c for c in checks}
|
||||
assert by["ollama-models.yaml"].ok
|
||||
assert "swarm-assistent" in by["Assistent ext"].detail
|
||||
@@ -56,7 +56,9 @@ def test_write_preset(tmp_path: Path):
|
||||
path = tmp_path / "out.yaml"
|
||||
write_ollama_models_preset(path, "recommended")
|
||||
entries = parse_ollama_models(path)
|
||||
assert entries[0].name == "huihui_ai/qwen2.5-vl-abliterated:7b"
|
||||
assert entries[0].name == "huihui_ai/qwen3-vl-abliterated:8b-instruct"
|
||||
assert entries[0].default is True
|
||||
assert entries[1].name == "huihui_ai/qwen2.5-vl-abliterated:7b"
|
||||
write_ollama_models_preset(path, "big")
|
||||
assert parse_ollama_models(path)[0].name == "huihui_ai/qwen2.5-vl-abliterated:32b"
|
||||
write_ollama_models_preset(path, "text")
|
||||
|
||||
@@ -43,10 +43,10 @@ def test_warmup_skips_when_ps_has_model(monkeypatch):
|
||||
def fake_urlopen(req, timeout=None):
|
||||
url = getattr(req, "full_url", str(req))
|
||||
assert "/api/ps" in url
|
||||
return FakeResp('{"models":[{"name":"huihui_ai/qwen2.5-vl-abliterated:7b"}]}')
|
||||
return FakeResp('{"models":[{"name":"huihui_ai/qwen3-vl-abliterated:8b-instruct"}]}')
|
||||
|
||||
monkeypatch.setattr(_mod.urllib.request, "urlopen", fake_urlopen)
|
||||
msg = _mod.warmup("huihui_ai/qwen2.5-vl-abliterated:7b", keep_alive="15m", num_ctx=16384, timeout=5)
|
||||
msg = _mod.warmup("huihui_ai/qwen3-vl-abliterated:8b-instruct", keep_alive="15m", num_ctx=16384, timeout=5)
|
||||
assert "skip" in msg
|
||||
assert "VRAM" in msg
|
||||
|
||||
|
||||
@@ -82,6 +82,6 @@ def test_ollama_mid_4090_context_16k():
|
||||
)
|
||||
tune = ollama_tune_for(info)
|
||||
assert tune.context_length == 16384
|
||||
assert tune.keep_alive == "15m"
|
||||
assert tune.keep_alive == "5m"
|
||||
assert "OLLAMA_CONTEXT_LENGTH=16384" in "\n".join(ollama_env_lines(tune))
|
||||
assert "OLLAMA_KEEP_ALIVE=15m" in "\n".join(ollama_env_lines(tune))
|
||||
assert "OLLAMA_KEEP_ALIVE=5m" in "\n".join(ollama_env_lines(tune))
|
||||
|
||||
@@ -86,7 +86,7 @@ def test_install_ollama_skips_restart_when_unit_unchanged():
|
||||
text = files("gpu_rent.remote").joinpath("install_ollama.sh").read_text(encoding="utf-8")
|
||||
assert "cmp -s" in text
|
||||
assert "skip restart" in text
|
||||
assert '"mid", 10 * 1024**3, "15m"' in text
|
||||
assert '"mid", 10 * 1024**3, "5m"' in text
|
||||
|
||||
|
||||
def test_provision_llm_skips_on_api_tags_not_cli_list():
|
||||
|
||||
Reference in New Issue
Block a user