Swarm Assistent

SwarmUI extension for collaborative Krea 2 prompting via Ollama: chat + vision, LoRA/trigger awareness, applyable patches, img2img / inpaint, auto Generate, and Civitai search with Confirm.

Layout

• Left: vision reference (drop / paste / Send to Assistent from Generate)
• Splitter: drag to resize panes
• Right (wider): chat
• Top-right: prompt pack + settings (Ollama URL, model, auto-apply / auto-generate)

UX

• Send to Assistent under the current Generate image (and History) — loads vision + opens the Assistent tab
• Drag images from Generate/History or the OS onto the left pane
• Paste (Ctrl+V) while the Assistent tab is open
• Use current / Clear for the vision reference
• As Init / As Mask / Clear Init — wire the vision image into Swarm Init Image / Mask Image (img2img & inpaint)
• Enter sends; Shift+Enter newline
• Auto-apply + Auto-generate (default on): patch from the LLM is applied and Generate runs when the patch changes prompt/params or includes actions: ["generate"]
• Pure Q&A without a patch does not start Generate
• Interrupt stops Swarm generation / clears busy state
• Civitai search cards require Confirm download (uses Swarm DoModelDownloadWS + stored civitai_api key). Auto-download is off by default.

Requirements

• SwarmUI with a Krea 2 checkpoint selected
• Ollama on http://127.0.0.1:11434 on the GPU VM (gpu-rent LLM_RUNTIME=ollama). The browser talks to SwarmUI; SwarmUI proxies /api/tags and /api/chat. URL in settings must stay 127.0.0.1:11434, not the laptop tunnel port 17811.
• At least one pulled model (ollama pull / ollama-models.yaml). Empty /api/tags → empty Model dropdown.
• Optional: Civitai API key in SwarmUI User Settings for search/download.

Install

Clone into SwarmUI src/Extensions/swarm-assistent (or let gpu-rent seed it from extensions.yaml with requires: ollama):

swarmui:
  - url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
    ref: main
    dir: swarm-assistent
    requires: ollama

Restart / rebuild SwarmUI after clone.

Prompt packs

Pack	Role
base_krea2	Always injected: Krea 2 rules + JSON patch / actions contract
write_prompt	Craft / improve prompts
critique_image	Vision critique → fixes
compose_scene	Scene / moodboard
fix_params	Width/height/steps/CFG/seed/σ-shift
inpaint_edit	Init Image img2img + Mask inpaint

Live context (checkpoint, server inventory LoRAs + triggers, wildcards, current params) is injected every request.

Patch actions

{
  "prompt": "...",
  "loras": [{"name": "exact", "weight": 0.8, "triggers": ["..."]}],
  "width": 1024, "height": 1280, "steps": 8, "cfg": 1,
  "seed": -1, "sigma_shift": 1.15,
  "use_init_image": true,
  "init_creativity": 0.45,
  "use_mask_image": false,
  "actions": ["generate"],
  "search_query": null
}

• generate — auto-generate after apply (when enabled)
• use_init / use_mask — set Assistent vision (or current Generate) as Init / Mask
• search_civitai + search_query — server searches Civitai, second LLM hop, Confirm cards in UI
• interrupt — stop current generation

Mask convention: white = edit, black = keep. Creativity ≈ denoise (0–1).

API routes

Route	Role
AssistentListModels	Ollama /api/tags
AssistentListInventory	LoRA / checkpoint / wildcard inventory from Swarm
AssistentSearchCivitai	Civitai LoRA search
AssistentGetPacks	Prompt pack texts
AssistentChat	HTTP chat (+ Civitai hop)
AssistentChatWS	Streaming chat WebSocket

License

MIT