# Swarm Assistent

SwarmUI extension for **collaborative Krea 2** prompting via **Ollama**: chat + multi-window board (live Generate + refs), LoRA/trigger awareness, applyable patches, **img2img / inpaint**, slash commands, aspect/seed chips, auto Generate, and Civitai search with Confirm.

**Version 0.5.0** — denser Krea 2 knowledge in prompt packs, new patch fields, `/help` slash commands, composer chips.

## Layout

- **Left — Board:** live **Generate** window + **Ref** windows (drop / paste / *Send to Assistent* / Snapshot gen). Per-window **vision** checkbox attaches that image to the next chat.
- **Splitter:** drag to resize panes
- **Right (wider):** chat — first visit shows a short how-to. Only the **latest** JSON proposal keeps action buttons; Apply + Generate spins/disables while a generation is running.
- **Chips** above the composer: aspect ratios, Seed lock / random, Vary.
- **Top-right:** prompt pack + settings (Ollama URL, model, auto-apply / auto-generate)

## UX

- **Send to Assistent** under the current Generate image (and History) — adds a Ref window and opens the Assistent tab
- Drag / paste onto a Ref (drop on Generate snapshots into a new Ref)
- **Snapshot gen** copies the live Generate window into a Ref
- **As Init** / **As Mask** / **Clear Init** — selected window → Swarm `Init Image` / `Mask Image`
- Enter sends; Shift+Enter newline
- **Auto-apply** + **Auto-generate** (default on): patch from the LLM is applied and Generate runs when the patch changes prompt/params or includes `actions: ["generate"]`
- Pure Q&A without a patch does **not** start Generate
- **Interrupt** stops Swarm generation / clears busy state
- **Civitai** search cards require **Confirm download** (uses Swarm `DoModelDownloadWS` + stored `civitai_api` key). Auto-download is off by default.
- Pack auto-selects from the user message (critique / inpaint / params / compose / describe / write) unless you changed the dropdown yourself.

### Slash commands (client-side, no LLM)

| Command | Effect |
| --- | --- |
| `/help` | List commands |
| `/gen` | Generate now |
| `/look generate\|refN` | Attach that board window + ask the LLM to look |
| `/init` `/mask` `/clear` | Same as board buttons |
| `/interrupt` | Stop generation |
| `/aspect 16:9` | Set size from the official 1K table |
| `/seed lock\|random` | Lock or randomize seed |
| `/vary` | New seed, same prompt (+ generate if auto) |
| `/pack write\|critique\|compose\|params\|inpaint\|describe` | Switch pack |
| `/civitai <query>` | Ask LLM to search Civitai (Confirm still required) |

## Requirements

- SwarmUI with a **Krea 2** checkpoint selected
- Ollama on `http://127.0.0.1:11434` **on the GPU VM** (gpu-rent `LLM_RUNTIME=ollama`). The browser talks to SwarmUI; SwarmUI proxies `/api/tags` and `/api/chat`. URL in settings must stay `127.0.0.1:11434`, not the laptop tunnel port 17811.
- At least one pulled model (`ollama pull` / `ollama-models.yaml`). Empty `/api/tags` → empty Model dropdown.
- Optional: Civitai API key in SwarmUI User Settings for search/download.

## Install

Clone into SwarmUI `src/Extensions/swarm-assistent` (or let **gpu-rent** seed it from `extensions.yaml` with `requires: ollama`):

```yaml
swarmui:
  - url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
    ref: main
    dir: swarm-assistent
    requires: ollama
```

Restart / rebuild SwarmUI after clone.

## Prompt packs

| Pack | Role |
| --- | --- |
| `base_krea2` | Always injected: Krea 2 rules + JSON patch / actions contract |
| `write_prompt` | Craft / improve prompts (prose structure) |
| `critique_image` | Vision critique → fixes |
| `compose_scene` | Scene / moodboard via board refs → text |
| `fix_params` | Aspect / steps / CFG / seed / batch |
| `inpaint_edit` | Init Image img2img + Mask inpaint |
| `describe_ref` | Vision → Krea prompt (no generate by default) |
| `catalog_card` | Recommendation card JSON for a checkpoint/LoRA |

**Persona** (tone) is a separate dropdown from pack — `base_krea2` → persona → pack. Overlay: `/mnt/swarm_data/Assistent/personas.json` (seeded from gpu-rent `assistent-personas.yaml`).

**Cards** subtab: edit/save `{stem}.assistent.json` next to weights; live chat gets cards for the current checkpoint + enabled LoRAs only. Uninstalled models can enqueue `.gpu-rent-wanted-models.yaml` for the next `up`.

Live context (checkpoint, server inventory LoRAs + blurbs/triggers, `model_cards`, `taste_profile`, wildcards, current params, persona) is injected every request. Inventory refreshes each chat turn (disk rescan when stale / after download / `/inventory`). After Confirm download Assistent auto-writes a `.assistent.json` card.

**Wanted queue:** Cards → Enqueue wanted → `/mnt/swarm_data/.gpu-rent-wanted-models.yaml`. Laptop: `gpu-rent capture wanted` (also on `capture models` / seed `up`).

**Cloud-only Krea.ai features** (moodboards UI, Generative Sliders, Creativity Raw/Low/Medium/High) are **not** in Swarm. The assistant emulates them with prompt language + board refs. Optional patch fields `creativity` / `intensity` / `complexity` / `movement` guide the LLM only.

### Patch actions

```json
{
  "prompt": "...",
  "loras": [{"name": "exact", "weight": 0.8, "triggers": ["..."]}],
  "aspect": "16:9",
  "width": 1376, "height": 768,
  "steps": 8, "cfg": 1,
  "seed": -1, "images": 1,
  "sigma_shift": 1.15,
  "creativity": "medium",
  "intensity": 0, "complexity": 0, "movement": 0,
  "vary": false, "lock_seed": false,
  "use_init_image": true,
  "init_creativity": 0.45,
  "use_mask_image": false,
  "clear_prompt_images": false,
  "look_at": ["generate"],
  "slot_to_init": "generate",
  "snapshot_generate": false,
  "pack": "critique_image",
  "actions": ["generate"],
  "search_query": null
}
```

- `aspect` — maps to official 1K sizes (`1:1`, `4:5`, `2:3`, `16:9`, `9:16`, `4:3`, `3:2`, `2.35:1`)
- `vary` / `lock_seed` / `images`|`batch` — seed and batch helpers
- `clear_prompt_images` — strip `<image…>` embeds from the prompt box
- `pack` — switch the active prompt pack for a follow-up
- `generate` — auto-generate after apply (when enabled)
- `look_at` — hop vision from named board windows (`generate`, `ref1`, …)
- `slot_to_init` / `slot_to_mask` — copy that window into Swarm Init / Mask
- `snapshot_generate` — copy live Generate into a Ref
- `search_civitai` + `search_query` — server searches Civitai, second LLM hop, Confirm cards in UI
- `interrupt` — stop current generation

Mask convention: **white = edit**, black = keep. Creativity ≈ denoise (0–1).

Turbo defaults: steps **8**, CFG **1** (never 0), sigma shift **1.15**.

## API routes

| Route | Role |
| --- | --- |
| `AssistentListModels` | Ollama `/api/tags` |
| `AssistentListInventory` | LoRA / checkpoint / wildcard inventory from Swarm |
| `AssistentSearchCivitai` | Civitai LoRA search |
| `AssistentGetPacks` | Prompt pack texts |
| `AssistentChat` | HTTP chat (+ Civitai hop) |
| `AssistentChatWS` | Streaming chat WebSocket |

## License

MIT