Add Assistent reference books and remove training dataset output.

Introduce books/ with civitai-krea2 and HF fictext builders, sha-diff seed to VM, and drop train.jsonl from the civitai scrape pipeline.

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/architecture.md

@@ -44,8 +44,9 @@
 | `inventory` / `ux` / `placement` / `flavor_presets` | Flavors, квоты, AZ; preview / ServerPlan |
 | `session` | `cmd_up` / `cmd_stop` / adopt |
 | `bootstrap` + `remote/bootstrap.sh` | Идемпотентный first-boot; light без apt |
-| `provision` | extensions, autocomplete, civitai seed, LLM, push, idle-killer arm, start SwarmUI; seed `search.jsonl` → `Assistent/civitai-examples.jsonl` |
-| `civitai` / `civitai_dataset` | Civitai HTTP API; локальный scrape Krea2 → train/search jsonl (отдельный CLI) |
+| `provision` | extensions, autocomplete, books seed, LLM, push, idle-killer arm, start SwarmUI; `seed_books()` → `Assistent/books/` (sha diff) |
+| `civitai` / `civitai_dataset` | Civitai HTTP API; локальный scrape Krea2 → search jsonl + book `books/civitai-krea2` |
+| `books` | HF fictext + civitai book build; `meta.content_sha`; отдельный CLI + `seed-books` |
 | `huggingface` | HF probe / метадата / URL для seed и capture |
 | `capture` | Инвентарь VM → merge ссылок в локальные манифесты |
 | `doctor` | Preflight без mutating compute |

docs/cli.md

@@ -79,7 +79,8 @@ gpu-rent up --yes --ollama
 | `gpu-rent ssh` | Оболочка на VM |
 | `gpu-rent seed-models` | Докачать новые строки Civitai-манифеста на живой диск |
 | `gpu-rent seed-extensions` | Доклонировать/обновить git-репы; restart swarmui |
-| `gpu-rent seed-personas` | Пуш `assistent-extensions/` → `Assistent/extensions/` (+ `_base`); также `search.jsonl` → FTS, если есть |
+| `gpu-rent seed-personas` | Пуш `assistent-extensions/` → `Assistent/extensions/` (+ `_base`); также `seed-books` если есть `books/` |
+| `gpu-rent seed-books` | Push изменённых `books/*/` → `Assistent/books/` (`--force` — все) |
 | `gpu-rent push` / `push-models` | Локальные деревья → VM |
 | `gpu-rent pull-output` | VM `Output/` → `./Output` |
 | `gpu-rent capture` / `capture all` | Инвентарь VM → merge **ссылок** в `models.yaml` + `extensions.yaml` (веса не качать) |
@@ -102,10 +103,10 @@ gpu-rent up --yes --ollama
 | --- | --- |
 | `discover` | Топ Krea2 checkpoints/LoRAs → `catalog/models.json` |
 | `scrape` | Галереи (метаданные, **без** картинок); `--target`, `--per-version`, `--min-score` |
-| `split` | `by_kind/` / `by_rating/` / `train.jsonl` / `search.jsonl` |
+| `split` | `by_kind/` / `by_rating/` / `search.jsonl` + `books/civitai-krea2/` |
 | `all` | discover + scrape + split |
 
-Нужен `CIVITAI_API_TOKEN` (лучше `CIVITAI_API_HOST=civitai.red`). На `up` / `seed-personas` файл `datasets/civitai/search.jsonl` пушится на VM как `Assistent/civitai-examples.jsonl` (если есть).
+Нужен `CIVITAI_API_TOKEN` (лучше `CIVITAI_API_HOST=civitai.red`). Книги: `books-dataset.bat` / `python -m gpu_rent.books download fictext --variant both` (pyarrow: `pip install 'gpu-rent[books]'`). На `up` / `seed-books` изменённые `books/*/` → `Assistent/books/`.
 
 ---
 

docs/decisions.md

@@ -62,4 +62,4 @@
 | LLM runtime | Только `none` \| `ollama`. **llama.cpp снят** из продукта; leftover unit `gpu-rent-llamacpp` на `up` стопается |
 | Debug API | Принятая поверхность: localhost `:17821` на `up`/`tunnel` / `gpu-rent debug`; Assistent diagnose, session, chat-eval |
 | Assistent | First-class поверх SwarmUI + Ollama (чат, Cards, personas), не только «prompt-help» |
-| Civitai dataset | Локальный scrape метаданных Krea2 (без картинок) → `train.jsonl` / `search.jsonl`; на `up`/`seed-personas` `search.jsonl` → `Assistent/civitai-examples.jsonl` (FTS) |
+| Civitai dataset | Локальный scrape Krea2 (без картинок) → `search.jsonl` + book `books/civitai-krea2/`; на `up`/`seed-books` push при изменении `meta.content_sha` → `Assistent/books/` (FTS). Legacy `civitai-examples.jsonl` если books нет |

docs/extensions.md

@@ -70,7 +70,7 @@ Bundled в swarm-assistent: `neutral` (Нормальный), `aggressive`, `dre
 
 Формат пака: `assistent-pack.yaml` (`kind: persona`, `id: …`) + полки JSON в корне (или `personas/<id>/` для мульти-пака). На `up` также пишется `_base/assistant.json` (`default_persona: neutral`, опционально `num_ctx` с GPU tier). Seed **не** удаляет overlay-личности, созданные в UI. Legacy `assistent-personas/` / `personas.json` больше не сидятся.
 
-Если есть `datasets/civitai/search.jsonl`, на том же шаге пушится FTS-индекс → `/mnt/swarm_data/Assistent/civitai-examples.jsonl` ([datasets/README](../datasets/README.md)). Диагностика Assistent: Debug API `/assistent/*` ([cli.md](cli.md)).
+Если есть `books/` (или legacy `datasets/civitai/search.jsonl`), на том же шаге `seed-books` пушит FTS → `/mnt/swarm_data/Assistent/books/` ([books/README](../books/README.md), [knowledge-books.md](knowledge-books.md)). Диагностика Assistent: Debug API `/assistent/*` ([cli.md](cli.md)).
 
 В git репозитория gpu-rent не коммитить рабочий список с лишними приватными URL сверх примера.
 

docs/knowledge-books.md

@@ -0,0 +1,47 @@
+# Assistent Knowledge Hub (swarm-assistent)
+
+gpu-rent builds and seeds **books** locally. Runtime lives in [swarm-assistent](https://gitea.hsrv.site/mrleo1nid/swarm-assistent.git).
+
+## VM layout
+
+```text
+/mnt/swarm_data/Assistent/books/<book-id>/
+  book.yaml          # title, description, content_kind
+  search.jsonl       # FTS rows (unified contract)
+  meta.json          # row_count, content_sha, built_at
+  .gpu-rent-meta.json  # same sha for gpu-rent skip/push
+```
+
+## Unified search row
+
+```json
+{
+  "id": "civitai-krea2:42",
+  "book": "civitai-krea2",
+  "title": "",
+  "tags": ["woman"],
+  "text": "excerpt ≤500 chars",
+  "body": "longer chunk",
+  "rating": "pg13",
+  "meta": {}
+}
+```
+
+## swarm-assistent tasks (not in gpu-rent)
+
+1. **`Knowledge/IKnowledgeSource.cs`** + adapters: Book, MemorySeed, MemoryPersonal, Tag, Session
+2. **`AssistentKnowledgeHub`**: ListCatalog, Search, Upsert, Forget, RunPipeline, BuildSystemLayer
+3. **Remove training UI**: Обучение tab, AssistentTrainingJobs, AssistentListTrainSamples, taste APIs
+4. **HTTP** `AssistentChat` → `knowledge: {catalog, hops, results}`; legacy `civitai_results` shim
+5. **Persona UI** «Знания»: attach toggles; bundled personas default attach civitai-krea2 + ru-fictext-rplus
+6. **Skill `knowledge`**: list / search / upsert / forget
+
+## Default attach (bundled only)
+
+- `neutral`, `aggressive`, `dreamer`: civitai-krea2 + ru-fictext-rplus
+- UI-created personas: no auto-attach
+- Pack `leonid`: from `assistent-pack.yaml` `knowledge.attach`
+
+## Index on VM
+
+On book push or sha change: reindex `search.jsonl` → sqlite FTS `books_fts`. Progress in UI on first chat.

docs/llm.md

@@ -113,7 +113,7 @@ Unit `gpu-rent-ollama` читает `/mnt/swarm_data/.gpu-rent-gpu.json`:
 
 \*Flash на Ampere+ (compute ≥ 8.0). `NUM_PARALLEL=2`, `MAX_LOADED_MODELS=2` (chat VL + memory embed). Memory models use `use: memory` and a CPU Modelfile (`num_gpu 0`) so embed does not steal VRAM from the chat model. Ollama default `num_ctx` is 4096; we set `OLLAMA_CONTEXT_LENGTH` so Assistent + vision fits. Mid/high keep-alive is **5m** because Assistent parks the chat model before Generate (`AssistentParkLlm`) and warms after. Env: `/mnt/swarm_data/.gpu-rent-ollama.env`.
 
-Personas: bundled `neutral` / `aggressive` / `dreamer` in swarm-assistent; extra packs via `extensions.yaml` `assistent:` or laptop `assistent-extensions/` → `Assistent/extensions/` on `up` / `seed-personas` / `push` (+ `_base/assistant.json` with `default_persona` / `num_ctx`). See [extensions.md](extensions.md). Optional FTS examples: [datasets/README](../datasets/README.md) (`search.jsonl` → `Assistent/civitai-examples.jsonl`). Journal: `gpu-rent logs -u ollama`.
+Personas: bundled `neutral` / `aggressive` / `dreamer` in swarm-assistent; extra packs via `extensions.yaml` `assistent:` or laptop `assistent-extensions/` → `Assistent/extensions/` on `up` / `seed-personas` / `push` (+ `_base/assistant.json` with `default_persona` / `num_ctx`). See [extensions.md](extensions.md). Reference books: [books/README](../books/README.md) → `Assistent/books/` on `seed-books` ([knowledge-books.md](knowledge-books.md)). Journal: `gpu-rent logs -u ollama`.
 
 `ollama-models.yaml` entries:
 

docs/local-folders.md

@@ -26,7 +26,8 @@
 | Локально | На VM | Когда |
 | --- | --- | --- |
 | `assistent-extensions/<pack>/` | `/mnt/swarm_data/Assistent/extensions/<pack>/` | `up` / `seed-personas` / `push` (без `.git`); см. [extensions.md](extensions.md) |
-| `datasets/civitai/search.jsonl` | `Assistent/civitai-examples.jsonl` | `up` / `seed-personas`, если файл есть; см. [datasets/README](../datasets/README.md) |
+| `datasets/civitai/search.jsonl` | (local build) → `books/civitai-krea2/` | civitai-dataset split |
+| `books/<id>/` | `Assistent/books/<id>/` | `up` / `seed-personas` / `seed-books` if `meta.content_sha` changed; see [books/README](../books/README.md) |
 
 ## Push (Models / Wildcards / CustomWorkflows / packs)
 

docs/roadmap.md

@@ -90,7 +90,7 @@
 ## Уже в коде после v1
 
 - Debug API на `127.0.0.1:17821` (`gpu-rent debug` / sidecar на `up`/`tunnel`); Assistent session / chat-eval
-- `civitai-dataset` → `datasets/civitai/search.jsonl` → FTS push на `up` / `seed-personas`
+- `civitai-dataset` → `books/civitai-krea2/`; `gpu-rent books download fictext`; `seed-books` на `up` / `seed-personas` (sha diff)
 - `seed-personas`, `capture wanted`, `logs --unit swarm|ollama|killer|cloud-init`
 - Balance notify (`SELECTEL_API_TOKEN` + watchdog)