Leonid PershinandCursor
bb8cb990aa
Keep action buttons only on the latest patch, spin Apply+Generate while busy, and greet on first open. Co-authored-by: Cursor <cursoragent@cursor.com>
100 lines
4.2 KiB
Markdown
100 lines
4.2 KiB
Markdown
# Swarm Assistent
|
||
|
||
SwarmUI extension for **collaborative Krea 2** prompting via **Ollama**: chat + multi-window board (live Generate + refs), LoRA/trigger awareness, applyable patches, **img2img / inpaint**, auto Generate, and Civitai search with Confirm.
|
||
|
||
## Layout
|
||
|
||
- **Left — Board:** live **Generate** window + **Ref** windows (drop / paste / *Send to Assistent* / Snapshot gen). Per-window **vision** checkbox attaches that image to the next chat.
|
||
- **Splitter:** drag to resize panes
|
||
- **Right (wider):** chat — first visit shows a short how-to. Only the **latest** JSON proposal keeps action buttons; Apply + Generate spins/disables while a generation is running.
|
||
- **Top-right:** prompt pack + settings (Ollama URL, model, auto-apply / auto-generate)
|
||
|
||
## UX
|
||
|
||
- **Send to Assistent** under the current Generate image (and History) — adds a Ref window and opens the Assistent tab
|
||
- Drag / paste onto a Ref (drop on Generate snapshots into a new Ref)
|
||
- **Snapshot gen** copies the live Generate window into a Ref
|
||
- **As Init** / **As Mask** / **Clear Init** — selected window → Swarm `Init Image` / `Mask Image`
|
||
- Enter sends; Shift+Enter newline
|
||
- **Auto-apply** + **Auto-generate** (default on): patch from the LLM is applied and Generate runs when the patch changes prompt/params or includes `actions: ["generate"]`
|
||
- Pure Q&A without a patch does **not** start Generate
|
||
- **Interrupt** stops Swarm generation / clears busy state
|
||
- **Civitai** search cards require **Confirm download** (uses Swarm `DoModelDownloadWS` + stored `civitai_api` key). Auto-download is off by default.
|
||
|
||
## Requirements
|
||
|
||
- SwarmUI with a **Krea 2** checkpoint selected
|
||
- Ollama on `http://127.0.0.1:11434` **on the GPU VM** (gpu-rent `LLM_RUNTIME=ollama`). The browser talks to SwarmUI; SwarmUI proxies `/api/tags` and `/api/chat`. URL in settings must stay `127.0.0.1:11434`, not the laptop tunnel port 17811.
|
||
- At least one pulled model (`ollama pull` / `ollama-models.yaml`). Empty `/api/tags` → empty Model dropdown.
|
||
- Optional: Civitai API key in SwarmUI User Settings for search/download.
|
||
|
||
## Install
|
||
|
||
Clone into SwarmUI `src/Extensions/swarm-assistent` (or let **gpu-rent** seed it from `extensions.yaml` with `requires: ollama`):
|
||
|
||
```yaml
|
||
swarmui:
|
||
- url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
|
||
ref: main
|
||
dir: swarm-assistent
|
||
requires: ollama
|
||
```
|
||
|
||
Restart / rebuild SwarmUI after clone.
|
||
|
||
## Prompt packs
|
||
|
||
| Pack | Role |
|
||
| --- | --- |
|
||
| `base_krea2` | Always injected: Krea 2 rules + JSON patch / actions contract |
|
||
| `write_prompt` | Craft / improve prompts |
|
||
| `critique_image` | Vision critique → fixes |
|
||
| `compose_scene` | Scene / moodboard |
|
||
| `fix_params` | Width/height/steps/CFG/seed/σ-shift |
|
||
| `inpaint_edit` | Init Image img2img + Mask inpaint |
|
||
|
||
Live context (checkpoint, server inventory LoRAs + triggers, wildcards, current params) is injected every request.
|
||
|
||
### Patch actions
|
||
|
||
```json
|
||
{
|
||
"prompt": "...",
|
||
"loras": [{"name": "exact", "weight": 0.8, "triggers": ["..."]}],
|
||
"width": 1024, "height": 1280, "steps": 8, "cfg": 1,
|
||
"seed": -1, "sigma_shift": 1.15,
|
||
"use_init_image": true,
|
||
"init_creativity": 0.45,
|
||
"use_mask_image": false,
|
||
"look_at": ["generate"],
|
||
"slot_to_init": "generate",
|
||
"snapshot_generate": false,
|
||
"actions": ["generate"],
|
||
"search_query": null
|
||
}
|
||
```
|
||
|
||
- `generate` — auto-generate after apply (when enabled)
|
||
- `look_at` — hop vision from named board windows (`generate`, `ref1`, …)
|
||
- `slot_to_init` / `slot_to_mask` — copy that window into Swarm Init / Mask
|
||
- `snapshot_generate` — copy live Generate into a Ref
|
||
- `search_civitai` + `search_query` — server searches Civitai, second LLM hop, Confirm cards in UI
|
||
- `interrupt` — stop current generation
|
||
|
||
Mask convention: **white = edit**, black = keep. Creativity ≈ denoise (0–1).
|
||
|
||
## API routes
|
||
|
||
| Route | Role |
|
||
| --- | --- |
|
||
| `AssistentListModels` | Ollama `/api/tags` |
|
||
| `AssistentListInventory` | LoRA / checkpoint / wildcard inventory from Swarm |
|
||
| `AssistentSearchCivitai` | Civitai LoRA search |
|
||
| `AssistentGetPacks` | Prompt pack texts |
|
||
| `AssistentChat` | HTTP chat (+ Civitai hop) |
|
||
| `AssistentChatWS` | Streaming chat WebSocket |
|
||
|
||
## License
|
||
|
||
MIT
|