Swarm Assistent

SwarmUI extension for collaborative Krea 2 prompting via Ollama: chat + multi-window board (live Generate + refs), LoRA/trigger awareness, applyable patches, img2img / inpaint, auto Generate, and Civitai search with Confirm.

Layout

• Left — Board: live Generate window + Ref windows (drop / paste / Send to Assistent / Snapshot gen). Per-window vision checkbox attaches that image to the next chat.
• Splitter: drag to resize panes
• Right (wider): chat — first visit shows a short how-to. Only the latest JSON proposal keeps action buttons; Apply + Generate spins/disables while a generation is running.
• Top-right: prompt pack + settings (Ollama URL, model, auto-apply / auto-generate)

UX

• Send to Assistent under the current Generate image (and History) — adds a Ref window and opens the Assistent tab
• Drag / paste onto a Ref (drop on Generate snapshots into a new Ref)
• Snapshot gen copies the live Generate window into a Ref
• As Init / As Mask / Clear Init — selected window → Swarm Init Image / Mask Image
• Enter sends; Shift+Enter newline
• Auto-apply + Auto-generate (default on): patch from the LLM is applied and Generate runs when the patch changes prompt/params or includes actions: ["generate"]
• Pure Q&A without a patch does not start Generate
• Interrupt stops Swarm generation / clears busy state
• Civitai search cards require Confirm download (uses Swarm DoModelDownloadWS + stored civitai_api key). Auto-download is off by default.

Requirements

• SwarmUI with a Krea 2 checkpoint selected
• Ollama on http://127.0.0.1:11434 on the GPU VM (gpu-rent LLM_RUNTIME=ollama). The browser talks to SwarmUI; SwarmUI proxies /api/tags and /api/chat. URL in settings must stay 127.0.0.1:11434, not the laptop tunnel port 17811.
• At least one pulled model (ollama pull / ollama-models.yaml). Empty /api/tags → empty Model dropdown.
• Optional: Civitai API key in SwarmUI User Settings for search/download.

Install

Clone into SwarmUI src/Extensions/swarm-assistent (or let gpu-rent seed it from extensions.yaml with requires: ollama):

swarmui:
  - url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
    ref: main
    dir: swarm-assistent
    requires: ollama

Restart / rebuild SwarmUI after clone.

Prompt packs

Pack	Role
base_krea2	Always injected: Krea 2 rules + JSON patch / actions contract
write_prompt	Craft / improve prompts
critique_image	Vision critique → fixes
compose_scene	Scene / moodboard
fix_params	Width/height/steps/CFG/seed/σ-shift
inpaint_edit	Init Image img2img + Mask inpaint

Live context (checkpoint, server inventory LoRAs + triggers, wildcards, current params) is injected every request.

Patch actions

{
  "prompt": "...",
  "loras": [{"name": "exact", "weight": 0.8, "triggers": ["..."]}],
  "width": 1024, "height": 1280, "steps": 8, "cfg": 1,
  "seed": -1, "sigma_shift": 1.15,
  "use_init_image": true,
  "init_creativity": 0.45,
  "use_mask_image": false,
  "look_at": ["generate"],
  "slot_to_init": "generate",
  "snapshot_generate": false,
  "actions": ["generate"],
  "search_query": null
}

• generate — auto-generate after apply (when enabled)
• look_at — hop vision from named board windows (generate, ref1, …)
• slot_to_init / slot_to_mask — copy that window into Swarm Init / Mask
• snapshot_generate — copy live Generate into a Ref
• search_civitai + search_query — server searches Civitai, second LLM hop, Confirm cards in UI
• interrupt — stop current generation

Mask convention: white = edit, black = keep. Creativity ≈ denoise (0–1).

API routes

Route	Role
AssistentListModels	Ollama /api/tags
AssistentListInventory	LoRA / checkpoint / wildcard inventory from Swarm
AssistentSearchCivitai	Civitai LoRA search
AssistentGetPacks	Prompt pack texts
AssistentChat	HTTP chat (+ Civitai hop)
AssistentChatWS	Streaming chat WebSocket

License

MIT