Swarm Assistent

SwarmUI extension for collaborative Krea 2 prompting via Ollama: chat + board (Generate | Refs tabs), LoRA chips, persona presets (Config/personas/), About the user prefs + craft vector memory, model cards with Civitai fetch, img2img/inpaint, slash commands, auto Generate.

Turn model: one user message is one turn. A turn may fan out into nested LLM hops — Krea prompt prep, empty-patch retry, vision, auto-critique. Hops share one HOP_BUDGET, never re-read the user's text (their prompt is client-authored), and pass the busy gate that blocks new user sends. What a reply does to generation state is decided once, in resolveTurnIntent: the model's actions:["generate"] / look_at win, RU intent heuristics only back it up when the model forgets, and an explicit «запомни, не генерируй» vetoes both.

Version 0.14.0 — Чат = сессия генерации: у каждого чата свои params/LoRA/checkpoint/кадр/refs; модель шлёт sparse-дельту + generate/look_at/ask; без вкладки Карточки и Civitai/wanted hops. Сжатие контекста: rolling-саммари той же Ollama-моделью, чип бюджета N / num_ctx, авто перед отправкой, /compress.

Version 0.13.1 — Сборка 0.13.0: using для WebSocket/HttpClient, instance-методы с Config/FilePath, Sqlite dll рядом с extension (иначе вкладка не грузится / API пустые).

Version 0.13.0 — Реальный QLoRA-пайплайн: train_qlora.py (TRL SFTTrainer + PEFT), HF-датасеты с маппингом (preset fiction title/tags→text), max_samples, полный post-train: safetensors → GGUF (convert_lora_to_gguf.py) → ollama create с FROM ollama_base + ADAPTER. Раннер: builtin + custom. Зависимости: scripts/requirements-train.txt.

Version 0.12.1 — Услышанное → агент: одобренные примеры датасета сразу попадают в vector memory (kind=heard) и в контекст чата как heard_examples (без QLoRA). На вкладке «Датасет»: авто-подключение при одобрении, синхронизация всех, per-sample 🔗. Агент может запросить heard_search. Настройки: training-agent.json.

Version 0.12.0 — App-level tabs (Чат / Карточки / Обучение / Настройки), боковая панель истории чатов, вкладка обучения LLM: курирование диалогов, импорт JSONL/CSV, Hugging Face datasets (фильтр совместимости), быстрый Ollama Modelfile, опциональный QLoRA-раннер с локаутом VRAM. HF token из SwarmUI User Settings (huggingface_api).

Version 0.11.9 — Distilled client: esbuild bundle (Assets/assistent.bundle.js), unified patch keys (Config/_base/patch-keys.json), taste stack removed (UserPrefs only), chat storage merge + all-chats disk save, write_prompt → alias of ordinary. Builds on prior 0.11.9 turn-intent work.

Version 0.11.9 — One turn, one decision. Nested hops (Krea prep, empty-patch retry, vision, critique) share a turnHops budget and pass the busy gate — Krea prep and the empty-patch retry were silently no-ops since 0.10.22/0.11.2. Generate / look_at are decided in a single resolveTurnIntent; ensureGenerateAction, shouldHonorLookAt and the wantsGen/willGen/suppressGen tangle are gone. Builds on 0.11.8.

Version 0.11.8 — session_exact remembers applied params that differ from Exact (not only when the user typed the knob). /debug ask uses a hidden Q&A pack: 5–10 line explain, no JSON/generate, dump stays a system note. Builds on 0.11.7.

Version 0.11.7 — Generate only for a real frame request: chat/opinions no longer auto-run Swarm. Context still counts («нарисуй», «ещё одну», «другая поза»), not only «генерируй». Builds on 0.11.6.

Version 0.11.6 — Generate from context, not only «генерируй»: scene briefs / edits / «ещё» emit actions:["generate"]; client injects it if the model forgets. Builds on 0.11.5.

Version 0.11.5 — Generate always carries a negative: model creates / supplements / echoes live; client pass-through if omitted. Builds on 0.11.4.

Version 0.11.4 — Vision is opt-in: look via «Посмотри результат» / /look, or when the model truly needs pixels. No auto look_at / auto-critique after every Generate (checkboxes stay, default off). Builds on 0.11.3.

Version 0.11.2 — Before Krea Generate, chat model maximally preps the prompt (EN + structure); skip prep hop only if already Krea-ready English. Builds on 0.11.1.

Version 0.11.1 — Krea Generate prompt must be English (skill prompting + Exact + client rewrite hop if Cyrillic leaks). Builds on 0.11.0 variants.

Version 0.11.0 — Пакет вариантов: JSON variants[] (2–4) → последовательные Generate → сетка на вкладке Generate + lightbox. Builds on 0.10.22 empty-patch fix.

Version 0.10.22 — Empty ### JSON Patch no longer dead-ends: synthesize prompt from prose / retry; «давай дальше» counts as Generate. Builds on 0.10.21 spinner fix.

Version 0.10.21 — Board Generate spinner clears when Swarm finishes (num_live_gens), not only when the image URL string changes. Builds on 0.10.20 remember-without-gen.

Version 0.10.20 — «Запомни / базовый промпт» no longer triggers Auto-Generate or auto look_at (even if the model sneaks actions:["generate"]). Builds on 0.10.19 slider echo fix.

Version 0.10.19 — Вкус/Хорни no longer reset: Generate patches never apply controls; default-echo filtered always; bare «вкус» no longer disables the filter. Builds on 0.10.18 settings tab.

Version 0.10.18 — Settings is a full subtab (Чат | Карточки | Настройки), not a header overlay. Builds on 0.10.17 post-Generate warm.

Layout

• Left — Board tabs: Generate (live view, or a variant grid when the model emits variants[]) | Refs (reference grid + badge N · vision M); click a variant to select / open lightbox; Посмотри результат attaches the selected finished frame and asks for a verdict
• Splitter: drag to resize panes
• Right: Chat | Cards | Settings; persona / pack / Ollama chat model; Ollama health badge
• Chips / slash: loaded from Config/_base/ui.json (persona can override)

Config (bundled + overlay)

Config/
  _base/           # defaults (assistant, ui, models/krea2, exact.json, core, packs, skills, memory-seed, identity)
  personas/<id>/   # sparse shelves: persona/bio/voice/humor/… + optional controls.json / exact.json / memory-seed

Disk overlay (wins over bundled): /mnt/swarm_data/Assistent/ — same folder layout as Config/. Drop _base/… and personas/<id>/… to override. Plus runtime state:

Assistent/
  _base/ personas/<id>/    # overlay presets — same names as Config/, sparse
  settings.json            # embed_model, base_url, per-persona skills
  ollama-roles.json        # chat vs memory model tags (gpu-rent writes this)
  memory/assistent.sqlite  # craft RAG + user_prefs + tags FTS + chats + ui_state

Client build

Sources live in src/ (ES modules). The VM ships the committed bundle only (no Node required at runtime):

npm install
npm run build    # → Assets/assistent.bundle.js
npm run watch    # rebuild on save
npm test         # intent.js + patch.js

SwarmUI loads a single script: Assets/assistent.bundle.js.

Copy personas/leonid/ → new id, edit only differing JSON. See Config/personas/README.md.

Controls: optional controls.json schema + exact.controls values. UI auto-draws every slider (order, display: percent). LLM may patch "controls": {…}. Values persist in overlay Exact (DeepMerge partial saves). Leonid: Вкус + Хорни; /остынь, /horny-game.

Authoring: pack author_persona + /persona new clones to overlay only. Delete overlay personas with the ✕ button or ⚙ → Личности (never from the model). Export/import .assistent-persona.json for sharing.

Exact memory (KV)

• Config/_base/exact.json — canonical generation defaults, profiles (turbo/raw), aspect table, short facts
• Persona / disk overlays merge via DeepMerge (matching keys overwrite)
• Always injected into the system prompt; UI fills empty SwarmUI fields from Exact (no LLM call)
• Chat-session overrides (session_exact) last until persona change or clear chat — not written to disk
• Priority: core → user message → About the user → session_exact → exact (+ persona) → live fields → craft memory_hits

About the user (UserPrefs)

Separate sqlite table user_prefs (not craft RAG):

• Global — every persona (e.g. “avoid blonde hair”)
• Persona — only the current agent
• Injected as ## About the user; strength via user_prefs_weight / user_prefs_max in assistant.json (⚙ → О пользователе)
• Agent write: actions: ["user_pref_upsert"] + user_prefs: [{key,text,scope}]

Craft vector memory

Two layers in memory/assistent.sqlite (persona column; empty = shared):

• Shared — Config/_base/memory-seed/, model cards, scope: "shared" upserts. Visible to every persona.
• Personal — Config/personas/<id>/memory-seed/ and chat upserts (default). Never copied into shared. Other personas do not retrieve it.
• Retrieve = shared ∪ this persona (and extends parents). Hybrid FTS5 + cosine, kind quotas, min_score. Same kind+key: personal overwrites parent overwrites shared.
• Tools: memory_get, memory_search, lookup_tags (Danbooru csv in Data/Autocompletions, FTS, no embeddings).
• Soft craft notes only — Exact, About the user, and the user beat RAG for params
• ⚙ → Крафт lists rows with filters + clear (bundled seed is read-only)

Chats and runtime KV

• Every chat (messages + Generate params snapshot) is a row in assistent.sqlite, newest 200 kept. History search uses FTS over title + body.
• All chats with messages are debounced to disk (not only the active one). Load merges disk + localStorage by updatedAt.
• UI state whitelists park_llm among other keys in sqlite kv.ui_state.
• settings.json and persona overlays stay files (layered merge + git). .assistent.json cards stay next to weights.

VRAM handover

• Before Generate the chat model is unloaded only if Park LLM is enabled (keep_alive: 0) — default off
• After Generate the chat model is always force-warmed (AssistentWarmLlm) — Krea still often evicts VL from VRAM even without park
• Embed / memory models are never parked — reloading them would stall every retrieve

UX

• Send to Assistent under Generate/History → Ref + Assistent tab
• Enter sends; Shift+Enter newline; Interrupt cancels chat epoch
• Manual Apply + Generate / /gen always generate; Auto-generate checkbox only for LLM auto-path
• Посмотри результат / auto-critique wait for a real Generate frame — model previews and unfinished batches are skipped
• Civitai Confirm required (unless auto-download); queued-but-missing models show a ⏳ wanted badge in Cards

Slash commands (client-side, no LLM)

Command	Effect
/help	List commands
/new	New chat (current one is saved)
/history	Open saved chats
/debug	Short UI/Exact dump (no LLM)
/debug ask / /why	Dump + short model explanation
/gen	Generate now
/look generate|refN	Attach that board window + ask the LLM to look
/init /mask /clear	Same as board buttons
/interrupt	Stop generation / cancel chat
/aspect 16:9	Set size from the official 1K table
/seed lock|random	Lock or randomize seed
/vary	New seed, same prompt (+ generate if auto)
/pack write|ordinary|critique|compose|params|inpaint|describe|persona	Switch pack
/persona new|clone|save	Overlay persona authoring
/inventory	Rescan моделей + обновить список LoRA

Requirements

• SwarmUI with a Krea 2 checkpoint selected
• Ollama on http://127.0.0.1:11434 on the GPU VM (gpu-rent LLM_RUNTIME=ollama)
• Chat model + memory embed (use: memory in ollama-models.yaml; gpu-rent creates CPU variant)
• Optional: Civitai API key in SwarmUI User Settings

Install

swarmui:
  - url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
    ref: main
    dir: swarm-assistent
    requires: ollama

Restart / rebuild SwarmUI after clone. gpu-rent: seed-extensions + restart.

Packs & skills

Packs (one active): ordinary (default комбайн; covers write/critique/params flows), write_prompt (alias → same as ordinary), critique_image, compose_scene, fix_params, inpaint_edit, describe_ref, author_persona.

Patch fence keys: single source Config/_base/patch-keys.json → C# + client via AssistentGetConfig.patch_keys.

Skills (checkboxes): prompting, creativity_sliders, memory — procedures; encyclopedia numbers live in Exact, soft notes in memory-seed / RAG, human taste in UserPrefs.

Personas: neutral, lewd, aggressive, cinema, leonid under Config/personas/. Saved terse falls back to aggressive. Overlay clones via /persona new or ⚙ → Личности.

API routes

Route	Role
AssistentListModels	Ollama tags → models (chat) + memory_models
AssistentGetConfig	Merged preset for persona (ui, packs, skills, identity, controls, patch_keys)
AssistentSaveSettings	Overlay settings DeepMerge (skills, embed_model)
AssistentSaveControls	Persist Exact controls values for a persona (overlay)
AssistentGetPersonaShelves	Merged identity shelves + controls
AssistentClonePersona	Snapshot clone → overlay id
AssistentSavePersona	Sparse shelf write (overlay only)
AssistentDeletePersona	UI-only delete of overlay persona
AssistentExportPersona / AssistentImportPersona	Shareable .assistent-persona.json pack
AssistentSaveKnobs	Overlay _base/assistant.json + Exact turbo/raw profiles
AssistentListInventory	LoRA / checkpoint / wildcard inventory
AssistentListPersonas	Persona catalog
AssistentListUserPrefs / AssistentUpsertUserPref / AssistentForgetUserPref / AssistentClearUserPrefs	About the user
AssistentChat / AssistentChatWS	Chat (+ user prefs + hybrid craft memory + ask hops)
AssistentListMemory / AssistentUpsertMemory / AssistentForgetMemory / AssistentClearMemory	Craft vector store
AssistentSearchMemory / AssistentGetMemory	Hybrid search / exact kind+key
AssistentLookupTags	Danbooru csv FTS (no embeddings)
AssistentListChats / AssistentGetChat / AssistentSaveChat / AssistentDeleteChat	sqlite chats (optional q FTS)
AssistentGetUiState / AssistentSaveUiState	sqlite kv.ui_state
AssistentParkLlm / AssistentWarmLlm	Unload / reload the chat model in VRAM
AssistentListTrainSamples / AssistentUpsertTrainSample / AssistentDeleteTrainSample	Training samples in sqlite
AssistentBuildDatasetFromChats / AssistentImportDataset / AssistentExportDataset	Dataset from chats / file import / JSONL export
AssistentCreateOllamaModel	Build Ollama model from Modelfile (SYSTEM + few-shot)
AssistentSearchHfDatasets / AssistentCheckHfDataset / AssistentPreviewHfDataset / AssistentImportHfDataset	Hugging Face datasets (gated by schema)
AssistentStartTrainJob / AssistentCancelTrainJob / AssistentGetTrainJob / AssistentTrainWS	QLoRA runner + progress
AssistentGetRunnerSettings / AssistentSaveRunnerSettings	Python runner config overlay
AssistentGetDatasetAgentSettings / AssistentSaveDatasetAgentSettings	«Услышанное» → agent RAG (training-agent.json)
AssistentLinkTrainSampleToAgent / AssistentUnlinkTrainSampleFromAgent / AssistentSyncDatasetToAgent	Embed approved samples as heard memory

QLoRA setup (0.13.0)

1. Python env with CUDA: pip install -r scripts/requirements-train.txt
2. SwarmUI User Settings → huggingface_api (for HF base model download)
3. Настройки → Модели: runner kind = builtin, paths to convert_lora_to_gguf.py and GGUF base (same arch as HF base)
4. Обучение → QLoRA: HF base id, Ollama base (existing tag), output name, optional HF dataset (krplt/ru-fictext-nsfw auto-maps fiction preset)
5. Pipeline: train → adapter_model.safetensors → GGUF → ollama create with ADAPTER

Manual test checklist: small JSONL (5 pairs); HF dataset with max_samples=50; cancel job; missing deps (exit 2); missing gguf script (completed with note).

License

MIT