Leonid PershinandCursor
3380206c6a
Server inventory and streaming chat, auto-apply/generate with Interrupt, Civitai search cards (Confirm-only download), plus Init/Mask wiring and inpaint_edit pack. Co-authored-by: Cursor <cursoragent@cursor.com>
96 lines
3.8 KiB
Markdown
96 lines
3.8 KiB
Markdown
# Swarm Assistent
|
||
|
||
SwarmUI extension for **collaborative Krea 2** prompting via **Ollama**: chat + vision, LoRA/trigger awareness, applyable patches, **img2img / inpaint**, auto Generate, and Civitai search with Confirm.
|
||
|
||
## Layout
|
||
|
||
- **Left:** vision reference (drop / paste / *Send to Assistent* from Generate)
|
||
- **Splitter:** drag to resize panes
|
||
- **Right (wider):** chat
|
||
- **Top-right:** prompt pack + settings (Ollama URL, model, auto-apply / auto-generate)
|
||
|
||
## UX
|
||
|
||
- **Send to Assistent** under the current Generate image (and History) — loads vision + opens the Assistent tab
|
||
- Drag images from Generate/History or the OS onto the left pane
|
||
- Paste (`Ctrl+V`) while the Assistent tab is open
|
||
- **Use current** / **Clear** for the vision reference
|
||
- **As Init** / **As Mask** / **Clear Init** — wire the vision image into Swarm `Init Image` / `Mask Image` (img2img & inpaint)
|
||
- Enter sends; Shift+Enter newline
|
||
- **Auto-apply** + **Auto-generate** (default on): patch from the LLM is applied and Generate runs when the patch changes prompt/params or includes `actions: ["generate"]`
|
||
- Pure Q&A without a patch does **not** start Generate
|
||
- **Interrupt** stops Swarm generation / clears busy state
|
||
- **Civitai** search cards require **Confirm download** (uses Swarm `DoModelDownloadWS` + stored `civitai_api` key). Auto-download is off by default.
|
||
|
||
## Requirements
|
||
|
||
- SwarmUI with a **Krea 2** checkpoint selected
|
||
- Ollama on `http://127.0.0.1:11434` **on the GPU VM** (gpu-rent `LLM_RUNTIME=ollama`). The browser talks to SwarmUI; SwarmUI proxies `/api/tags` and `/api/chat`. URL in settings must stay `127.0.0.1:11434`, not the laptop tunnel port 17811.
|
||
- At least one pulled model (`ollama pull` / `ollama-models.yaml`). Empty `/api/tags` → empty Model dropdown.
|
||
- Optional: Civitai API key in SwarmUI User Settings for search/download.
|
||
|
||
## Install
|
||
|
||
Clone into SwarmUI `src/Extensions/swarm-assistent` (or let **gpu-rent** seed it from `extensions.yaml` with `requires: ollama`):
|
||
|
||
```yaml
|
||
swarmui:
|
||
- url: https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git
|
||
ref: main
|
||
dir: swarm-assistent
|
||
requires: ollama
|
||
```
|
||
|
||
Restart / rebuild SwarmUI after clone.
|
||
|
||
## Prompt packs
|
||
|
||
| Pack | Role |
|
||
| --- | --- |
|
||
| `base_krea2` | Always injected: Krea 2 rules + JSON patch / actions contract |
|
||
| `write_prompt` | Craft / improve prompts |
|
||
| `critique_image` | Vision critique → fixes |
|
||
| `compose_scene` | Scene / moodboard |
|
||
| `fix_params` | Width/height/steps/CFG/seed/σ-shift |
|
||
| `inpaint_edit` | Init Image img2img + Mask inpaint |
|
||
|
||
Live context (checkpoint, server inventory LoRAs + triggers, wildcards, current params) is injected every request.
|
||
|
||
### Patch actions
|
||
|
||
```json
|
||
{
|
||
"prompt": "...",
|
||
"loras": [{"name": "exact", "weight": 0.8, "triggers": ["..."]}],
|
||
"width": 1024, "height": 1280, "steps": 8, "cfg": 1,
|
||
"seed": -1, "sigma_shift": 1.15,
|
||
"use_init_image": true,
|
||
"init_creativity": 0.45,
|
||
"use_mask_image": false,
|
||
"actions": ["generate"],
|
||
"search_query": null
|
||
}
|
||
```
|
||
|
||
- `generate` — auto-generate after apply (when enabled)
|
||
- `use_init` / `use_mask` — set Assistent vision (or current Generate) as Init / Mask
|
||
- `search_civitai` + `search_query` — server searches Civitai, second LLM hop, Confirm cards in UI
|
||
- `interrupt` — stop current generation
|
||
|
||
Mask convention: **white = edit**, black = keep. Creativity ≈ denoise (0–1).
|
||
|
||
## API routes
|
||
|
||
| Route | Role |
|
||
| --- | --- |
|
||
| `AssistentListModels` | Ollama `/api/tags` |
|
||
| `AssistentListInventory` | LoRA / checkpoint / wildcard inventory from Swarm |
|
||
| `AssistentSearchCivitai` | Civitai LoRA search |
|
||
| `AssistentGetPacks` | Prompt pack texts |
|
||
| `AssistentChat` | HTTP chat (+ Civitai hop) |
|
||
| `AssistentChatWS` | Streaming chat WebSocket |
|
||
|
||
## License
|
||
|
||
MIT
|