Leonid PershinandCursor
07bb521b70
Introduce books/ with civitai-krea2 and HF fictext builders, sha-diff seed to VM, and drop train.jsonl from the civitai scrape pipeline. Co-authored-by: Cursor <cursoragent@cursor.com>
Civitai Krea2 datasets (local)
Scraped generation metadata for Assistent books (FTS search). No image files. UI training (train.jsonl) removed.
Not part of gpu-rent Typer CLI and not models.yaml / gpu-rent push. Separate entrypoints:
civitai-dataset.bat # Windows (or .\civitai-dataset.ps1 / ./civitai-dataset.sh)
# default = all → discover + scrape + split (+ books/civitai-krea2)
python -m gpu_rent.civitai_dataset all
books-dataset.bat # python -m gpu_rent.books download fictext …
Needs CIVITAI_API_TOKEN and preferably CIVITAI_API_HOST=civitai.red in .env.
Commands
| Command | Flags | Role |
|---|---|---|
discover |
--top-checkpoints (20), --top-loras (30) |
Popular Krea2 ckpt/LoRAs (+ versions from models.yaml) → catalog/models.json |
scrape |
--target (2000), --per-version, --min-score |
Gallery metadata only; resume by id; NSFW→rating; skip minor tags |
split |
— | by_kind/ / by_rating/ / search.jsonl + books/civitai-krea2/ |
all |
same as discover+scrape | discover → scrape → split (launcher default) |
Global: --out = app root (writes under datasets/civitai).
Layout
| Path | Role |
|---|---|
civitai/catalog/models.json |
Discovered Krea2 checkpoints / LoRAs |
civitai/catalog/images.jsonl |
Canonical rows (resume by id) |
civitai/by_kind/ |
checkpoint vs lora |
civitai/by_rating/ |
pg / pg13 / r / x |
civitai/search.jsonl |
Compact Civitai FTS source → books/civitai-krea2 on split |
../books/civitai-krea2/ |
Assistent book (book.yaml, search.jsonl, meta.json) |
On up / seed-personas / seed-books: changed books push → /mnt/swarm_data/Assistent/books/ (sha diff). Legacy civitai-examples.jsonl only if books/ empty.