Add Assistent reference books and remove training dataset output.
Introduce books/ with civitai-krea2 and HF fictext builders, sha-diff seed to VM, and drop train.jsonl from the civitai scrape pipeline. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
# Assistent reference books (local)
|
||||
|
||||
FTS `search.jsonl` indexes for Swarm Assistent Knowledge Hub. **Not** in Typer main CLI — use:
|
||||
|
||||
```text
|
||||
books-dataset.bat # python -m gpu_rent.books
|
||||
python -m gpu_rent.books list
|
||||
python -m gpu_rent.books build civitai
|
||||
python -m gpu_rent.books download fictext --variant both
|
||||
gpu-rent seed-books # push changed books → VM
|
||||
```
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | Role |
|
||||
| --- | --- |
|
||||
| `civitai-krea2/` | From `civitai-dataset split` → `book.yaml` + `search.jsonl` + `meta.json` |
|
||||
| `ru-fictext-rplus/` | HF `ru-fictext-nsfw-data-r+.parquet` → chunked search rows |
|
||||
| `ru-fictext-full/` | HF full parquet |
|
||||
|
||||
On `up` / `seed-personas` / `seed-books`: changed books (by `meta.content_sha`) → `/mnt/swarm_data/Assistent/books/<id>/`.
|
||||
|
||||
Civitai scrape: manual on laptop; `up` pushes when local book changed.
|
||||
|
||||
Fictext: `pip install 'gpu-rent[books]'` (pyarrow) for build.
|
||||
|
||||
See [docs/local-folders.md](../docs/local-folders.md), [datasets/README.md](../datasets/README.md).
|
||||
Reference in New Issue
Block a user