Leonid PershinandCursor
07bb521b70
Introduce books/ with civitai-krea2 and HF fictext builders, sha-diff seed to VM, and drop train.jsonl from the civitai scrape pipeline. Co-authored-by: Cursor <cursoragent@cursor.com>
28 lines
1.0 KiB
Markdown
28 lines
1.0 KiB
Markdown
# Assistent reference books (local)
|
|
|
|
FTS `search.jsonl` indexes for Swarm Assistent Knowledge Hub. **Not** in Typer main CLI — use:
|
|
|
|
```text
|
|
books-dataset.bat # python -m gpu_rent.books
|
|
python -m gpu_rent.books list
|
|
python -m gpu_rent.books build civitai
|
|
python -m gpu_rent.books download fictext --variant both
|
|
gpu-rent seed-books # push changed books → VM
|
|
```
|
|
|
|
## Layout
|
|
|
|
| Path | Role |
|
|
| --- | --- |
|
|
| `civitai-krea2/` | From `civitai-dataset split` → `book.yaml` + `search.jsonl` + `meta.json` |
|
|
| `ru-fictext-rplus/` | HF `ru-fictext-nsfw-data-r+.parquet` → chunked search rows |
|
|
| `ru-fictext-full/` | HF full parquet |
|
|
|
|
On `up` / `seed-personas` / `seed-books`: changed books (by `meta.content_sha`) → `/mnt/swarm_data/Assistent/books/<id>/`.
|
|
|
|
Civitai scrape: manual on laptop; `up` pushes when local book changed.
|
|
|
|
Fictext: `pip install 'gpu-rent[books]'` (pyarrow) for build.
|
|
|
|
See [docs/local-folders.md](../docs/local-folders.md), [datasets/README.md](../datasets/README.md).
|