143 lines
7.0 KiB
Markdown
143 lines
7.0 KiB
Markdown
# Swarm Assistent
|
||
|
||
SwarmUI extension for **collaborative Krea 2** prompting via **Ollama**: chat + multi-window board (live Generate + refs), LoRA/trigger awareness, applyable patches, **img2img / inpaint**, slash commands, aspect/seed chips, auto Generate, and Civitai search with Confirm.
|
||
|
||
**Version 0.5.0** — denser Krea 2 knowledge in prompt packs, new patch fields, `/help` slash commands, composer chips.
|
||
|
||
## Layout
|
||
|
||
- **Left — Board:** live **Generate** window + **Ref** windows (drop / paste / *Send to Assistent* / Snapshot gen). Per-window **vision** checkbox attaches that image to the next chat.
|
||
- **Splitter:** drag to resize panes
|
||
- **Right (wider):** chat — first visit shows a short how-to. Only the **latest** JSON proposal keeps action buttons; Apply + Generate spins/disables while a generation is running.
|
||
- **Chips** above the composer: aspect ratios, Seed lock / random, Vary.
|
||
- **Top-right:** prompt pack + settings (Ollama URL, model, auto-apply / auto-generate)
|
||
|
||
## UX
|
||
|
||
- **Send to Assistent** under the current Generate image (and History) — adds a Ref window and opens the Assistent tab
|
||
- Drag / paste onto a Ref (drop on Generate snapshots into a new Ref)
|
||
- **Snapshot gen** copies the live Generate window into a Ref
|
||
- **As Init** / **As Mask** / **Clear Init** — selected window → Swarm `Init Image` / `Mask Image`
|
||
- Enter sends; Shift+Enter newline
|
||
- **Auto-apply** + **Auto-generate** (default on): patch from the LLM is applied and Generate runs when the patch changes prompt/params or includes `actions: ["generate"]`
|
||
- Pure Q&A without a patch does **not** start Generate
|
||
- **Interrupt** stops Swarm generation / clears busy state
|
||
- **Civitai** search cards require **Confirm download** (uses Swarm `DoModelDownloadWS` + stored `civitai_api` key). Auto-download is off by default.
|
||
- Pack auto-selects from the user message (critique / inpaint / params / compose / describe / write) unless you changed the dropdown yourself.
|
||
|
||
### Slash commands (client-side, no LLM)
|
||
|
||
| Command | Effect |
|
||
| --- | --- |
|
||
| `/help` | List commands |
|
||
| `/gen` | Generate now |
|
||
| `/look generate\|refN` | Attach that board window + ask the LLM to look |
|
||
| `/init` `/mask` `/clear` | Same as board buttons |
|
||
| `/interrupt` | Stop generation |
|
||
| `/aspect 16:9` | Set size from the official 1K table |
|
||
| `/seed lock\|random` | Lock or randomize seed |
|
||
| `/vary` | New seed, same prompt (+ generate if auto) |
|
||
| `/pack write\|critique\|compose\|params\|inpaint\|describe` | Switch pack |
|
||
| `/civitai <query>` | Ask LLM to search Civitai (Confirm still required) |
|
||
|
||
## Requirements
|
||
|
||
- SwarmUI with a **Krea 2** checkpoint selected
|
||
- Ollama on `http://127.0.0.1:11434` **on the GPU VM** (gpu-rent `LLM_RUNTIME=ollama`). The browser talks to SwarmUI; SwarmUI proxies `/api/tags` and `/api/chat`. URL in settings must stay `127.0.0.1:11434`, not the laptop tunnel port 17811.
|
||
- At least one pulled model (`ollama pull` / `ollama-models.yaml`). Empty `/api/tags` → empty Model dropdown.
|
||
- Optional: Civitai API key in SwarmUI User Settings for search/download.
|
||
|
||
## Install
|
||
|
||
Clone into SwarmUI `src/Extensions/swarm-assistent` (or let **gpu-rent** seed it from `extensions.yaml` with `requires: ollama`):
|
||
|
||
```yaml
|
||
swarmui:
|
||
- url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
|
||
ref: main
|
||
dir: swarm-assistent
|
||
requires: ollama
|
||
```
|
||
|
||
Restart / rebuild SwarmUI after clone.
|
||
|
||
## Prompt packs
|
||
|
||
| Pack | Role |
|
||
| --- | --- |
|
||
| `base_krea2` | Always injected: Krea 2 rules + JSON patch / actions contract |
|
||
| `write_prompt` | Craft / improve prompts (prose structure) |
|
||
| `critique_image` | Vision critique → fixes |
|
||
| `compose_scene` | Scene / moodboard via board refs → text |
|
||
| `fix_params` | Aspect / steps / CFG / seed / batch |
|
||
| `inpaint_edit` | Init Image img2img + Mask inpaint |
|
||
| `describe_ref` | Vision → Krea prompt (no generate by default) |
|
||
| `catalog_card` | Recommendation card JSON for a checkpoint/LoRA |
|
||
|
||
**Persona** (tone) is a separate dropdown from pack — `base_krea2` → persona → pack. Overlay: `/mnt/swarm_data/Assistent/personas.json` (seeded from gpu-rent `assistent-personas.yaml`).
|
||
|
||
**Cards** subtab: edit/save `{stem}.assistent.json` next to weights; live chat gets cards for the current checkpoint + enabled LoRAs only. Uninstalled models can enqueue `.gpu-rent-wanted-models.yaml` for the next `up`.
|
||
|
||
Live context (checkpoint, server inventory LoRAs + blurbs/triggers, `model_cards`, `taste_profile`, wildcards, current params, persona) is injected every request. Inventory refreshes each chat turn (disk rescan when stale / after download / `/inventory`). After Confirm download Assistent auto-writes a `.assistent.json` card.
|
||
|
||
**Wanted queue:** Cards → Enqueue wanted → `/mnt/swarm_data/.gpu-rent-wanted-models.yaml`. Laptop: `gpu-rent capture wanted` (also on `capture models` / seed `up`).
|
||
|
||
**Cloud-only Krea.ai features** (moodboards UI, Generative Sliders, Creativity Raw/Low/Medium/High) are **not** in Swarm. The assistant emulates them with prompt language + board refs. Optional patch fields `creativity` / `intensity` / `complexity` / `movement` guide the LLM only.
|
||
|
||
### Patch actions
|
||
|
||
```json
|
||
{
|
||
"prompt": "...",
|
||
"loras": [{"name": "exact", "weight": 0.8, "triggers": ["..."]}],
|
||
"aspect": "16:9",
|
||
"width": 1376, "height": 768,
|
||
"steps": 8, "cfg": 1,
|
||
"seed": -1, "images": 1,
|
||
"sigma_shift": 1.15,
|
||
"creativity": "medium",
|
||
"intensity": 0, "complexity": 0, "movement": 0,
|
||
"vary": false, "lock_seed": false,
|
||
"use_init_image": true,
|
||
"init_creativity": 0.45,
|
||
"use_mask_image": false,
|
||
"clear_prompt_images": false,
|
||
"look_at": ["generate"],
|
||
"slot_to_init": "generate",
|
||
"snapshot_generate": false,
|
||
"pack": "critique_image",
|
||
"actions": ["generate"],
|
||
"search_query": null
|
||
}
|
||
```
|
||
|
||
- `aspect` — maps to official 1K sizes (`1:1`, `4:5`, `2:3`, `16:9`, `9:16`, `4:3`, `3:2`, `2.35:1`)
|
||
- `vary` / `lock_seed` / `images`|`batch` — seed and batch helpers
|
||
- `clear_prompt_images` — strip `<image…>` embeds from the prompt box
|
||
- `pack` — switch the active prompt pack for a follow-up
|
||
- `generate` — auto-generate after apply (when enabled)
|
||
- `look_at` — hop vision from named board windows (`generate`, `ref1`, …)
|
||
- `slot_to_init` / `slot_to_mask` — copy that window into Swarm Init / Mask
|
||
- `snapshot_generate` — copy live Generate into a Ref
|
||
- `search_civitai` + `search_query` — server searches Civitai, second LLM hop, Confirm cards in UI
|
||
- `interrupt` — stop current generation
|
||
|
||
Mask convention: **white = edit**, black = keep. Creativity ≈ denoise (0–1).
|
||
|
||
Turbo defaults: steps **8**, CFG **1** (never 0), sigma shift **1.15**.
|
||
|
||
## API routes
|
||
|
||
| Route | Role |
|
||
| --- | --- |
|
||
| `AssistentListModels` | Ollama `/api/tags` |
|
||
| `AssistentListInventory` | LoRA / checkpoint / wildcard inventory from Swarm |
|
||
| `AssistentSearchCivitai` | Civitai LoRA search |
|
||
| `AssistentGetPacks` | Prompt pack texts |
|
||
| `AssistentChat` | HTTP chat (+ Civitai hop) |
|
||
| `AssistentChatWS` | Streaming chat WebSocket |
|
||
|
||
## License
|
||
|
||
MIT
|