Swarm Assistent

SwarmUI extension for collaborative Krea 2 prompting via Ollama: chat + multi-window board (live Generate + refs), LoRA/trigger awareness, applyable patches, img2img / inpaint, slash commands, aspect/seed chips, auto Generate, and Civitai search with Confirm.

Version 0.5.0 — denser Krea 2 knowledge in prompt packs, new patch fields, /help slash commands, composer chips.

Layout

• Left — Board: live Generate window + Ref windows (drop / paste / Send to Assistent / Snapshot gen). Per-window vision checkbox attaches that image to the next chat.
• Splitter: drag to resize panes
• Right (wider): chat — first visit shows a short how-to. Only the latest JSON proposal keeps action buttons; Apply + Generate spins/disables while a generation is running.
• Chips above the composer: aspect ratios, Seed lock / random, Vary.
• Top-right: prompt pack + settings (Ollama URL, model, auto-apply / auto-generate)

UX

• Send to Assistent under the current Generate image (and History) — adds a Ref window and opens the Assistent tab
• Drag / paste onto a Ref (drop on Generate snapshots into a new Ref)
• Snapshot gen copies the live Generate window into a Ref
• As Init / As Mask / Clear Init — selected window → Swarm Init Image / Mask Image
• Enter sends; Shift+Enter newline
• Auto-apply + Auto-generate (default on): patch from the LLM is applied and Generate runs when the patch changes prompt/params or includes actions: ["generate"]
• Pure Q&A without a patch does not start Generate
• Interrupt stops Swarm generation / clears busy state
• Civitai search cards require Confirm download (uses Swarm DoModelDownloadWS + stored civitai_api key). Auto-download is off by default.
• Pack auto-selects from the user message (critique / inpaint / params / compose / describe / write) unless you changed the dropdown yourself.

Slash commands (client-side, no LLM)

Command	Effect
/help	List commands
/gen	Generate now
/look generate|refN	Attach that board window + ask the LLM to look
/init /mask /clear	Same as board buttons
/interrupt	Stop generation
/aspect 16:9	Set size from the official 1K table
/seed lock|random	Lock or randomize seed
/vary	New seed, same prompt (+ generate if auto)
/pack write|critique|compose|params|inpaint|describe	Switch pack
/civitai <query>	Ask LLM to search Civitai (Confirm still required)

Requirements

• SwarmUI with a Krea 2 checkpoint selected
• Ollama on http://127.0.0.1:11434 on the GPU VM (gpu-rent LLM_RUNTIME=ollama). The browser talks to SwarmUI; SwarmUI proxies /api/tags and /api/chat. URL in settings must stay 127.0.0.1:11434, not the laptop tunnel port 17811.
• At least one pulled model (ollama pull / ollama-models.yaml). Empty /api/tags → empty Model dropdown.
• Optional: Civitai API key in SwarmUI User Settings for search/download.

Install

Clone into SwarmUI src/Extensions/swarm-assistent (or let gpu-rent seed it from extensions.yaml with requires: ollama):

swarmui:
  - url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
    ref: main
    dir: swarm-assistent
    requires: ollama

Restart / rebuild SwarmUI after clone.

Prompt packs

Pack	Role
base_krea2	Always injected: Krea 2 rules + JSON patch / actions contract
write_prompt	Craft / improve prompts (prose structure)
critique_image	Vision critique → fixes
compose_scene	Scene / moodboard via board refs → text
fix_params	Aspect / steps / CFG / seed / batch
inpaint_edit	Init Image img2img + Mask inpaint
describe_ref	Vision → Krea prompt (no generate by default)
catalog_card	Recommendation card JSON for a checkpoint/LoRA

Persona (tone) is a separate dropdown from pack — base_krea2 → persona → pack. Overlay: /mnt/swarm_data/Assistent/personas.json (seeded from gpu-rent assistent-personas.yaml).

Cards subtab: edit/save {stem}.assistent.json next to weights; live chat gets cards for the current checkpoint + enabled LoRAs only. Uninstalled models can enqueue .gpu-rent-wanted-models.yaml for the next up.

Live context (checkpoint, server inventory LoRAs + triggers, model_cards, wildcards, current params, persona) is injected every request.

Cloud-only Krea.ai features (moodboards UI, Generative Sliders, Creativity Raw/Low/Medium/High) are not in Swarm. The assistant emulates them with prompt language + board refs. Optional patch fields creativity / intensity / complexity / movement guide the LLM only.

Patch actions

{
  "prompt": "...",
  "loras": [{"name": "exact", "weight": 0.8, "triggers": ["..."]}],
  "aspect": "16:9",
  "width": 1376, "height": 768,
  "steps": 8, "cfg": 1,
  "seed": -1, "images": 1,
  "sigma_shift": 1.15,
  "creativity": "medium",
  "intensity": 0, "complexity": 0, "movement": 0,
  "vary": false, "lock_seed": false,
  "use_init_image": true,
  "init_creativity": 0.45,
  "use_mask_image": false,
  "clear_prompt_images": false,
  "look_at": ["generate"],
  "slot_to_init": "generate",
  "snapshot_generate": false,
  "pack": "critique_image",
  "actions": ["generate"],
  "search_query": null
}

• aspect — maps to official 1K sizes (1:1, 4:5, 2:3, 16:9, 9:16, 4:3, 3:2, 2.35:1)
• vary / lock_seed / images|batch — seed and batch helpers
• clear_prompt_images — strip <image…> embeds from the prompt box
• pack — switch the active prompt pack for a follow-up
• generate — auto-generate after apply (when enabled)
• look_at — hop vision from named board windows (generate, ref1, …)
• slot_to_init / slot_to_mask — copy that window into Swarm Init / Mask
• snapshot_generate — copy live Generate into a Ref
• search_civitai + search_query — server searches Civitai, second LLM hop, Confirm cards in UI
• interrupt — stop current generation

Mask convention: white = edit, black = keep. Creativity ≈ denoise (0–1).

Turbo defaults: steps 8, CFG 1 (never 0), sigma shift 1.15.

API routes

Route	Role
AssistentListModels	Ollama /api/tags
AssistentListInventory	LoRA / checkpoint / wildcard inventory from Swarm
AssistentSearchCivitai	Civitai LoRA search
AssistentGetPacks	Prompt pack texts
AssistentChat	HTTP chat (+ Civitai hop)
AssistentChatWS	Streaming chat WebSocket

License

MIT