Swarm Assistent

SwarmUI extension for collaborative Krea 2 prompting via Ollama: chat + board (Generate | Refs tabs), LoRA chips, persona presets (Config/personas/), About the user prefs + craft vector memory, model cards with Civitai fetch, img2img/inpaint, slash commands, auto Generate.

Version 0.11.3 — Stream no longer aborts on weak JSON fences (so prompts/skill_load can finish); num_predict 3072. Builds on 0.11.2 Krea prep.

Version 0.11.2 — Before Krea Generate, chat model maximally preps the prompt (EN + structure); skip prep hop only if already Krea-ready English. Builds on 0.11.1.

Version 0.11.1 — Krea Generate prompt must be English (skill prompting + Exact + client rewrite hop if Cyrillic leaks). Builds on 0.11.0 variants.

Version 0.11.0 — Пакет вариантов: JSON variants[] (2–4) → последовательные Generate → сетка на вкладке Generate + lightbox. Builds on 0.10.22 empty-patch fix.

Version 0.10.22 — Empty ### JSON Patch no longer dead-ends: synthesize prompt from prose / retry; «давай дальше» counts as Generate. Builds on 0.10.21 spinner fix.

Version 0.10.21 — Board Generate spinner clears when Swarm finishes (num_live_gens), not only when the image URL string changes. Builds on 0.10.20 remember-without-gen.

Version 0.10.20 — «Запомни / базовый промпт» no longer triggers Auto-Generate or auto look_at (even if the model sneaks actions:["generate"]). Builds on 0.10.19 slider echo fix.

Version 0.10.19 — Вкус/Хорни no longer reset: Generate patches never apply controls; default-echo filtered always; bare «вкус» no longer disables the filter. Builds on 0.10.18 settings tab.

Version 0.10.18 — Settings is a full subtab (Чат | Карточки | Настройки), not a header overlay. Builds on 0.10.17 post-Generate warm.

Layout

• Left — Board tabs: Generate (live view, or a variant grid when the model emits variants[]) | Refs (reference grid + badge N · vision M); click a variant to select / open lightbox; Посмотри результат attaches the selected finished frame and asks for a verdict
• Splitter: drag to resize panes
• Right: Chat | Cards | Settings; persona / pack / Ollama chat model; Ollama health badge
• Chips / slash: loaded from Config/_base/ui.json (persona can override)

Config (bundled + overlay)

Config/
  _base/           # defaults (assistant, ui, models/krea2, exact.json, core, packs, skills, memory-seed, identity)
  personas/<id>/   # sparse shelves: persona/bio/voice/humor/… + optional controls.json / exact.json / memory-seed

Disk overlay (wins over bundled): /mnt/swarm_data/Assistent/ — same folder layout as Config/. Drop _base/… and personas/<id>/… to override. Plus runtime state:

Assistent/
  _base/ personas/<id>/    # overlay presets — same names as Config/, sparse
  settings.json            # embed_model, base_url, per-persona skills
  ollama-roles.json        # chat vs memory model tags (gpu-rent writes this)
  memory/assistent.sqlite  # craft RAG + user_prefs + tags FTS + chats + ui_state + taste (legacy)
  _migrated_json/          # one-shot archive of old chats/*.json, ui-state.json, taste.json

Copy personas/leonid/ → new id, edit only differing JSON. See Config/personas/README.md.

Controls: optional controls.json schema + exact.controls values. UI auto-draws every slider (order, display: percent). LLM may patch "controls": {…}. Values persist in overlay Exact (DeepMerge partial saves). Leonid: Вкус + Хорни; /остынь, /horny-game.

Authoring: pack author_persona + /persona new clones to overlay only. Delete overlay personas with the ✕ button or ⚙ → Личности (never from the model). Export/import .assistent-persona.json for sharing.

Exact memory (KV)

• Config/_base/exact.json — canonical generation defaults, profiles (turbo/raw), aspect table, short facts
• Persona / disk overlays merge via DeepMerge (matching keys overwrite)
• Always injected into the system prompt; UI fills empty SwarmUI fields from Exact (no LLM call)
• Chat-session overrides (session_exact) last until persona change or clear chat — not written to disk
• Priority: core → user message → About the user → session_exact → exact (+ persona) → live fields → craft memory_hits

About the user (UserPrefs)

Separate sqlite table user_prefs (not craft RAG):

• Global — every persona (e.g. “avoid blonde hair”)
• Persona — only the current agent
• Injected as ## About the user; strength via user_prefs_weight / user_prefs_max in assistant.json (⚙ → О пользователе)
• Agent write: actions: ["user_pref_upsert"] + user_prefs: [{key,text,scope}]
• Legacy kv.taste migrates once into global prefs

Craft vector memory

Two layers in memory/assistent.sqlite (persona column; empty = shared):

• Shared — Config/_base/memory-seed/, model cards, scope: "shared" upserts. Visible to every persona.
• Personal — Config/personas/<id>/memory-seed/ and chat upserts (default). Never copied into shared. Other personas do not retrieve it.
• Retrieve = shared ∪ this persona (and extends parents). Hybrid FTS5 + cosine, kind quotas, min_score. Same kind+key: personal overwrites parent overwrites shared.
• Tools: memory_get, memory_search, lookup_tags (Danbooru csv in Data/Autocompletions, FTS, no embeddings).
• Soft craft notes only — Exact, About the user, and the user beat RAG for params
• ⚙ → Крафт lists rows with filters + clear (bundled seed is read-only)

Chats and runtime KV

• Every chat (messages + Generate params snapshot) is a row in assistent.sqlite, newest 200 kept. History search uses FTS over title + body.
• First launch after 0.8.3 copies chats/*.json, ui-state.json, and taste.json into sqlite, then archives them under _migrated_json/.
• localStorage stays as a fast cache; on first run with an empty store the old swarm_assistent_chats_v1 browser history is migrated up once.
• UI state seeds a fresh browser only — anything already in localStorage wins, and auto_download is never restored as on
• settings.json and persona overlays stay files (layered merge + git). .assistent.json cards stay next to weights.

VRAM handover

• Before Generate the chat model is unloaded only if Park LLM is enabled (keep_alive: 0) — default off
• After Generate the chat model is always force-warmed (AssistentWarmLlm) — Krea still often evicts VL from VRAM even without park
• Embed / memory models are never parked — reloading them would stall every retrieve

UX

• Send to Assistent under Generate/History → Ref + Assistent tab
• Enter sends; Shift+Enter newline; Interrupt cancels chat epoch
• Manual Apply + Generate / /gen always generate; Auto-generate checkbox only for LLM auto-path
• Посмотри результат / auto-critique wait for a real Generate frame — model previews and unfinished batches are skipped
• Civitai Confirm required (unless auto-download); queued-but-missing models show a ⏳ wanted badge in Cards

Slash commands (client-side, no LLM)

Command	Effect
/help	List commands
/debug	Short UI/Exact dump (no LLM)
/debug ask / /why	Dump + short model explanation
/gen	Generate now
/look generate|refN	Attach that board window + ask the LLM to look
/init /mask /clear	Same as board buttons
/interrupt	Stop generation / cancel chat
/aspect 16:9	Set size from the official 1K table
/seed lock|random	Lock or randomize seed
/vary	New seed, same prompt (+ generate if auto)
/pack write|critique|…	Switch pack
/civitai <query>	Ask LLM to search Civitai
/inventory	Rescan models + refresh LoRA list

Requirements

• SwarmUI with a Krea 2 checkpoint selected
• Ollama on http://127.0.0.1:11434 on the GPU VM (gpu-rent LLM_RUNTIME=ollama)
• Chat model + memory embed (use: memory in ollama-models.yaml; gpu-rent creates CPU variant)
• Optional: Civitai API key in SwarmUI User Settings

Install

swarmui:
  - url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
    ref: main
    dir: swarm-assistent
    requires: ollama

Restart / rebuild SwarmUI after clone. gpu-rent: seed-extensions + restart.

Packs & skills

Packs (one active): ordinary (default комбайн), write_prompt, critique_image, compose_scene, fix_params, inpaint_edit, describe_ref, catalog_card, author_persona.

Skills (checkboxes): prompting, creativity_sliders, memory — procedures; encyclopedia numbers live in Exact, soft notes in memory-seed / RAG, human taste in UserPrefs.

Personas: neutral, lewd, aggressive, cinema, terse, leonid under Config/personas/. Overlay clones via /persona new or ⚙ → Личности.

API routes

Route	Role
AssistentListModels	Ollama tags → models (chat) + memory_models
AssistentGetConfig	Merged preset for persona (ui, packs, skills, identity, controls)
AssistentSaveControls	Persist Exact controls values for a persona (overlay)
AssistentGetPersonaShelves	Merged identity shelves + controls
AssistentClonePersona	Snapshot clone → overlay id
AssistentSavePersona	Sparse shelf write (overlay only)
AssistentDeletePersona	UI-only delete of overlay persona
AssistentExportPersona / AssistentImportPersona	Shareable .assistent-persona.json pack
AssistentSaveKnobs	Overlay _base/assistant.json + Exact turbo/raw profiles
AssistentGetSettings / AssistentSaveSettings	Overlay settings (skills, embed_model)
AssistentListInventory	LoRA / checkpoint / wildcard inventory
AssistentListPersonas	Persona catalog
AssistentGetPacks	Prompt pack texts
AssistentGetCard / AssistentSaveCard	.assistent.json cards (+ memory ingest)
AssistentGetCardMeta	Local sidecar + optional Civitai by-hash
AssistentEnqueueWanted / AssistentListWanted	Wanted YAML queue (write / read + count)
AssistentGetTaste / AssistentSaveTaste	sqlite kv.taste (legacy; prefer UserPrefs)
AssistentListUserPrefs / AssistentUpsertUserPref / AssistentForgetUserPref / AssistentClearUserPrefs	About the user
AssistentSearchCivitai	Civitai LoRA search
AssistentChat / AssistentChatWS	Chat (+ user prefs + hybrid craft memory + hops)
AssistentListMemory / AssistentUpsertMemory / AssistentForgetMemory / AssistentClearMemory	Craft vector store
AssistentSearchMemory / AssistentGetMemory	Hybrid search / exact kind+key
AssistentLookupTags	Danbooru csv FTS (no embeddings)
AssistentListChats / AssistentGetChat / AssistentSaveChat / AssistentDeleteChat	sqlite chats (optional q FTS)
AssistentGetUiState / AssistentSaveUiState	sqlite kv.ui_state
AssistentParkLlm / AssistentWarmLlm	Unload / reload the chat model in VRAM

License

MIT