swarm-assistent/README.md

# Swarm Assistent

SwarmUI extension for **collaborative Krea 2** prompting via **Ollama**: chat + board (Generate | Refs tabs), LoRA chips, **persona presets** (`Config/personas/`), **About the user** prefs + craft vector memory, model cards with Civitai fetch, img2img/inpaint, slash commands, auto Generate.

**Turn model:** one user message is one *turn*. A turn may fan out into nested LLM *hops* — Krea prompt prep, empty-patch retry, vision, auto-critique. Hops share one `HOP_BUDGET`, never re-read the user's text (their prompt is client-authored), and pass the busy gate that blocks new user sends. What a reply does to generation state is decided once, in `resolveTurnIntent`: the model's `actions:["generate"]` / `look_at` win, RU intent heuristics only back it up when the model forgets, and an explicit «запомни, не генерируй» vetoes both.

**Version 0.14.0** — **Чат = сессия генерации**: у каждого чата свои params/LoRA/checkpoint/кадр/refs; модель шлёт sparse-дельту + `generate`/`look_at`/`ask`; без вкладки Карточки и Civitai/wanted hops. **Сжатие контекста**: rolling-саммари той же Ollama-моделью, чип бюджета `N / num_ctx`, авто перед отправкой, `/compress`.

**Version 0.13.1** — Сборка 0.13.0: `using` для `WebSocket`/`HttpClient`, instance-методы с `Config`/`FilePath`, Sqlite dll рядом с extension (иначе вкладка не грузится / API пустые).

**Version 0.13.0** — **Реальный QLoRA-пайплайн**: `train_qlora.py` (TRL SFTTrainer + PEFT), HF-датасеты с маппингом (preset fiction title/tags→text), `max_samples`, полный post-train: safetensors → GGUF (`convert_lora_to_gguf.py`) → `ollama create` с `FROM ollama_base` + `ADAPTER`. Раннер: `builtin` + `custom`. Зависимости: `scripts/requirements-train.txt`.

**Version 0.12.1** — **Услышанное → агент**: одобренные примеры датасета сразу попадают в vector memory (`kind=heard`) и в контекст чата как `heard_examples` (без QLoRA). На вкладке «Датасет»: авто-подключение при одобрении, синхронизация всех, per-sample 🔗. Агент может запросить `heard_search`. Настройки: `training-agent.json`.

**Version 0.12.0** — App-level tabs (Чат / Карточки / **Обучение** / Настройки), боковая панель истории чатов, вкладка обучения LLM: курирование диалогов, импорт JSONL/CSV, Hugging Face datasets (фильтр совместимости), быстрый Ollama Modelfile, опциональный QLoRA-раннер с локаутом VRAM. HF token из SwarmUI User Settings (`huggingface_api`).

**Version 0.11.9** — Distilled client: esbuild bundle (`Assets/assistent.bundle.js`), unified patch keys (`Config/_base/patch-keys.json`), taste stack removed (UserPrefs only), chat storage merge + all-chats disk save, `write_prompt` → alias of `ordinary`. Builds on prior 0.11.9 turn-intent work.

**Version 0.11.9** — One turn, one decision. Nested hops (Krea prep, empty-patch retry, vision, critique) share a `turnHops` budget and pass the busy gate — Krea prep and the empty-patch retry were silently no-ops since 0.10.22/0.11.2. Generate / `look_at` are decided in a single `resolveTurnIntent`; `ensureGenerateAction`, `shouldHonorLookAt` and the `wantsGen`/`willGen`/`suppressGen` tangle are gone. Builds on 0.11.8.

**Version 0.11.8** — `session_exact` remembers applied params that differ from Exact (not only when the user typed the knob). `/debug ask` uses a hidden Q&A pack: 5–10 line explain, no JSON/generate, dump stays a system note. Builds on 0.11.7.

**Version 0.11.7** — Generate only for a real frame request: chat/opinions no longer auto-run Swarm. Context still counts («нарисуй», «ещё одну», «другая поза»), not only «генерируй». Builds on 0.11.6.

**Version 0.11.6** — Generate from context, not only «генерируй»: scene briefs / edits / «ещё» emit `actions:["generate"]`; client injects it if the model forgets. Builds on 0.11.5.

**Version 0.11.5** — Generate always carries a negative: model creates / supplements / echoes live; client pass-through if omitted. Builds on 0.11.4.

**Version 0.11.4** — Vision is opt-in: look via «Посмотри результат» / `/look`, or when the model truly needs pixels. No auto look_at / auto-critique after every Generate (checkboxes stay, default off). Builds on 0.11.3.

**Version 0.11.2** — Before Krea Generate, chat model maximally preps the prompt (EN + structure); skip prep hop only if already Krea-ready English. Builds on 0.11.1.

**Version 0.11.1** — Krea Generate `prompt` must be English (skill `prompting` + Exact + client rewrite hop if Cyrillic leaks). Builds on 0.11.0 variants.

**Version 0.11.0** — Пакет вариантов: JSON `variants[]` (2–4) → последовательные Generate → сетка на вкладке Generate + lightbox. Builds on 0.10.22 empty-patch fix.

**Version 0.10.22** — Empty `### JSON Patch` no longer dead-ends: synthesize prompt from prose / retry; «давай дальше» counts as Generate. Builds on 0.10.21 spinner fix.

**Version 0.10.21** — Board Generate spinner clears when Swarm finishes (`num_live_gens`), not only when the image URL string changes. Builds on 0.10.20 remember-without-gen.

**Version 0.10.20** — «Запомни / базовый промпт» no longer triggers Auto-Generate or auto look_at (even if the model sneaks `actions:["generate"]`). Builds on 0.10.19 slider echo fix.

**Version 0.10.19** — Вкус/Хорни no longer reset: Generate patches never apply `controls`; default-echo filtered always; bare «вкус» no longer disables the filter. Builds on 0.10.18 settings tab.

**Version 0.10.18** — Settings is a full subtab (Чат | Карточки | Настройки), not a header overlay. Builds on 0.10.17 post-Generate warm.

## Layout

- **Left — Board tabs:** **Generate** (live view, or a **variant grid** when the model emits `variants[]`) | **Refs** (reference grid + badge `N · vision M`); click a variant to select / open lightbox; **Посмотри результат** attaches the selected finished frame and asks for a verdict
- **Splitter:** drag to resize panes
- **Right:** Chat | Cards | Settings; persona / pack / Ollama chat model; **Ollama health** badge
- **Chips / slash:** loaded from `Config/_base/ui.json` (persona can override)

## Config (bundled + overlay)

```
Config/
  _base/           # defaults (assistant, ui, models/krea2, exact.json, core, packs, skills, memory-seed, identity)
  personas/<id>/   # sparse shelves: persona/bio/voice/humor/… + optional controls.json / exact.json / memory-seed
```

Disk overlay (wins over bundled): `/mnt/swarm_data/Assistent/` — same folder layout as `Config/`. Drop `_base/…` and `personas/<id>/…` to override. Plus runtime state:

```
Assistent/
  _base/ personas/<id>/    # overlay presets — same names as Config/, sparse
  settings.json            # embed_model, base_url, per-persona skills
  ollama-roles.json        # chat vs memory model tags (gpu-rent writes this)
  memory/assistent.sqlite  # craft RAG + user_prefs + tags FTS + chats + ui_state
```

## Client build

Sources live in `src/` (ES modules). The VM ships the committed bundle only (no Node required at runtime):

```bash
npm install
npm run build    # → Assets/assistent.bundle.js
npm run watch    # rebuild on save
npm test         # intent.js + patch.js
```

SwarmUI loads a single script: `Assets/assistent.bundle.js`.

Copy `personas/leonid/` → new id, edit only differing JSON. See `Config/personas/README.md`.

**Controls:** optional `controls.json` schema + `exact.controls` values. UI auto-draws every slider (`order`, `display: percent`). LLM may patch `"controls": {…}`. Values persist in overlay Exact (DeepMerge partial saves). Leonid: **Вкус** + **Хорни**; `/остынь`, `/horny-game`.

**Authoring:** pack `author_persona` + `/persona new` clones to overlay only. Delete overlay personas with the ✕ button or ⚙ → Личности (never from the model). Export/import `.assistent-persona.json` for sharing.

## Exact memory (KV)

- `Config/_base/exact.json` — canonical generation defaults, profiles (turbo/raw), aspect table, short facts
- Persona / disk overlays merge via DeepMerge (matching keys overwrite)
- Always injected into the system prompt; UI fills **empty** SwarmUI fields from Exact (no LLM call)
- Chat-session overrides (`session_exact`) last until persona change or clear chat — not written to disk
- Priority: core → user message → **About the user** → session_exact → exact (+ persona) → live fields → craft `memory_hits`

## About the user (UserPrefs)

Separate sqlite table `user_prefs` (not craft RAG):

- **Global** — every persona (e.g. “avoid blonde hair”)
- **Persona** — only the current agent
- Injected as `## About the user`; strength via `user_prefs_weight` / `user_prefs_max` in `assistant.json` (⚙ → О пользователе)
- Agent write: `actions: ["user_pref_upsert"]` + `user_prefs: [{key,text,scope}]`

## Craft vector memory

Two layers in `memory/assistent.sqlite` (`persona` column; empty = shared):

- **Shared** — `Config/_base/memory-seed/`, model cards, `scope: "shared"` upserts. Visible to every persona.
- **Personal** — `Config/personas/<id>/memory-seed/` and chat upserts (default). Never copied into shared. Other personas do not retrieve it.
- Retrieve = shared ∪ this persona (and `extends` parents). Hybrid **FTS5 + cosine**, kind quotas, `min_score`. Same `kind`+`key`: personal overwrites parent overwrites shared.
- Tools: `memory_get`, `memory_search`, `lookup_tags` (Danbooru csv in `Data/Autocompletions`, FTS, **no embeddings**).
- Soft craft notes only — Exact, About the user, and the user beat RAG for params
- ⚙ → **Крафт** lists rows with filters + clear (bundled seed is read-only)

## Chats and runtime KV

- Every chat (messages + Generate params snapshot) is a row in `assistent.sqlite`, newest **200** kept. History search uses FTS over title + body.
- All chats with messages are debounced to disk (not only the active one). Load merges disk + localStorage by `updatedAt`.
- UI state whitelists `park_llm` among other keys in sqlite `kv.ui_state`.
- `settings.json` and persona overlays stay files (layered merge + git). `.assistent.json` cards stay next to weights.
## VRAM handover

- Before Generate the chat model is unloaded only if **Park LLM** is enabled (`keep_alive: 0`) — default off
- After Generate the chat model is **always** force-warmed (`AssistentWarmLlm`) — Krea still often evicts VL from VRAM even without park
- Embed / memory models are never parked — reloading them would stall every retrieve

## UX

- **Send to Assistent** under Generate/History → Ref + Assistent tab
- Enter sends; Shift+Enter newline; Interrupt cancels chat epoch
- Manual **Apply + Generate** / `/gen` always generate; Auto-generate checkbox only for LLM auto-path
- **Посмотри результат** / auto-critique wait for a real Generate frame — model previews and unfinished batches are skipped
- Civitai Confirm required (unless auto-download); queued-but-missing models show a `⏳ wanted` badge in Cards

### Slash commands (client-side, no LLM)

| Command | Effect |
| --- | --- |
| `/help` | List commands |
| `/new` | New chat (current one is saved) |
| `/history` | Open saved chats |
| `/debug` | Short UI/Exact dump (no LLM) |
| `/debug ask` / `/why` | Dump + short model explanation |
| `/gen` | Generate now |
| `/look generate\|refN` | Attach that board window + ask the LLM to look |
| `/init` `/mask` `/clear` | Same as board buttons |
| `/interrupt` | Stop generation / cancel chat |
| `/aspect 16:9` | Set size from the official 1K table |
| `/seed lock\|random` | Lock or randomize seed |
| `/vary` | New seed, same prompt (+ generate if auto) |
| `/pack write\|ordinary\|critique\|compose\|params\|inpaint\|describe\|persona` | Switch pack |
| `/persona new\|clone\|save` | Overlay persona authoring |
| `/inventory` | Rescan моделей + обновить список LoRA |

## Requirements

- SwarmUI with a **Krea 2** checkpoint selected
- Ollama on `http://127.0.0.1:11434` **on the GPU VM** (gpu-rent `LLM_RUNTIME=ollama`)
- Chat model + memory embed (`use: memory` in `ollama-models.yaml`; gpu-rent creates CPU variant)
- Optional: Civitai API key in SwarmUI User Settings

## Install

```yaml
swarmui:
  - url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
    ref: main
    dir: swarm-assistent
    requires: ollama
```

Restart / rebuild SwarmUI after clone. gpu-rent: `seed-extensions` + restart.

## Packs & skills

**Packs** (one active): `ordinary` (default комбайн; covers write/critique/params flows), `write_prompt` (alias → same as ordinary), `critique_image`, `compose_scene`, `fix_params`, `inpaint_edit`, `describe_ref`, `author_persona`.

Patch fence keys: single source `Config/_base/patch-keys.json` → C# + client via `AssistentGetConfig.patch_keys`.

**Skills** (checkboxes): `prompting`, `creativity_sliders`, `memory` — procedures; encyclopedia numbers live in Exact, soft notes in memory-seed / RAG, human taste in UserPrefs.

**Personas:** `neutral`, `lewd`, `aggressive`, `cinema`, `leonid` under `Config/personas/`. Saved `terse` falls back to `aggressive`. Overlay clones via `/persona new` or ⚙ → Личности.

## API routes

| Route | Role |
| --- | --- |
| `AssistentListModels` | Ollama tags → `models` (chat) + `memory_models` |
| `AssistentGetConfig` | Merged preset for persona (ui, packs, skills, identity, controls, **patch_keys**) |
| `AssistentSaveSettings` | Overlay settings DeepMerge (skills, embed_model) |
| `AssistentSaveControls` | Persist Exact `controls` values for a persona (overlay) |
| `AssistentGetPersonaShelves` | Merged identity shelves + controls |
| `AssistentClonePersona` | Snapshot clone → overlay id |
| `AssistentSavePersona` | Sparse shelf write (overlay only) |
| `AssistentDeletePersona` | UI-only delete of overlay persona |
| `AssistentExportPersona` / `AssistentImportPersona` | Shareable `.assistent-persona.json` pack |
| `AssistentSaveKnobs` | Overlay `_base/assistant.json` + Exact turbo/raw profiles |
| `AssistentListInventory` | LoRA / checkpoint / wildcard inventory |
| `AssistentListPersonas` | Persona catalog |
| `AssistentListUserPrefs` / `AssistentUpsertUserPref` / `AssistentForgetUserPref` / `AssistentClearUserPrefs` | About the user |
| `AssistentChat` / `AssistentChatWS` | Chat (+ user prefs + hybrid craft memory + ask hops) |
| `AssistentListMemory` / `AssistentUpsertMemory` / `AssistentForgetMemory` / `AssistentClearMemory` | Craft vector store |
| `AssistentSearchMemory` / `AssistentGetMemory` | Hybrid search / exact kind+key |
| `AssistentLookupTags` | Danbooru csv FTS (no embeddings) |
| `AssistentListChats` / `AssistentGetChat` / `AssistentSaveChat` / `AssistentDeleteChat` | sqlite `chats` (optional `q` FTS) |
| `AssistentGetUiState` / `AssistentSaveUiState` | sqlite `kv.ui_state` |
| `AssistentParkLlm` / `AssistentWarmLlm` | Unload / reload the chat model in VRAM |
| `AssistentListTrainSamples` / `AssistentUpsertTrainSample` / `AssistentDeleteTrainSample` | Training samples in sqlite |
| `AssistentBuildDatasetFromChats` / `AssistentImportDataset` / `AssistentExportDataset` | Dataset from chats / file import / JSONL export |
| `AssistentCreateOllamaModel` | Build Ollama model from Modelfile (SYSTEM + few-shot) |
| `AssistentSearchHfDatasets` / `AssistentCheckHfDataset` / `AssistentPreviewHfDataset` / `AssistentImportHfDataset` | Hugging Face datasets (gated by schema) |
| `AssistentStartTrainJob` / `AssistentCancelTrainJob` / `AssistentGetTrainJob` / `AssistentTrainWS` | QLoRA runner + progress |
| `AssistentGetRunnerSettings` / `AssistentSaveRunnerSettings` | Python runner config overlay |
| `AssistentGetDatasetAgentSettings` / `AssistentSaveDatasetAgentSettings` | «Услышанное» → agent RAG (`training-agent.json`) |
| `AssistentLinkTrainSampleToAgent` / `AssistentUnlinkTrainSampleFromAgent` / `AssistentSyncDatasetToAgent` | Embed approved samples as `heard` memory |

## QLoRA setup (0.13.0)

1. Python env with CUDA: `pip install -r scripts/requirements-train.txt`
2. SwarmUI **User Settings** → `huggingface_api` (for HF base model download)
3. **Настройки → Модели**: runner kind = **builtin**, paths to `convert_lora_to_gguf.py` and **GGUF base** (same arch as HF base)
4. **Обучение → QLoRA**: HF base id, **Ollama base** (existing tag), output name, optional HF dataset (`krplt/ru-fictext-nsfw` auto-maps fiction preset)
5. Pipeline: train → `adapter_model.safetensors` → GGUF → `ollama create` with `ADAPTER`

Manual test checklist: small JSONL (5 pairs); HF dataset with `max_samples=50`; cancel job; missing deps (exit 2); missing gguf script (completed with note).

## License

MIT