Leonid PershinandCursor
07bb521b70
Introduce books/ with civitai-krea2 and HF fictext builders, sha-diff seed to VM, and drop train.jsonl from the civitai scrape pipeline. Co-authored-by: Cursor <cursoragent@cursor.com>
42 lines
2.0 KiB
Markdown
42 lines
2.0 KiB
Markdown
# Civitai Krea2 datasets (local)
|
|
|
|
Scraped generation metadata for Assistent **books** (FTS search). **No image files.** UI training (`train.jsonl`) removed.
|
|
|
|
Not part of `gpu-rent` Typer CLI and **not** `models.yaml` / `gpu-rent push`. Separate entrypoints:
|
|
|
|
```text
|
|
civitai-dataset.bat # Windows (or .\civitai-dataset.ps1 / ./civitai-dataset.sh)
|
|
# default = all → discover + scrape + split (+ books/civitai-krea2)
|
|
python -m gpu_rent.civitai_dataset all
|
|
|
|
books-dataset.bat # python -m gpu_rent.books download fictext …
|
|
```
|
|
|
|
Needs `CIVITAI_API_TOKEN` and preferably `CIVITAI_API_HOST=civitai.red` in `.env`.
|
|
|
|
## Commands
|
|
|
|
| Command | Flags | Role |
|
|
| --- | --- | --- |
|
|
| `discover` | `--top-checkpoints` (20), `--top-loras` (30) | Popular Krea2 ckpt/LoRAs (+ versions from `models.yaml`) → `catalog/models.json` |
|
|
| `scrape` | `--target` (2000), `--per-version`, `--min-score` | Gallery metadata only; resume by id; NSFW→rating; skip minor tags |
|
|
| `split` | — | `by_kind/` / `by_rating/` / `search.jsonl` + `books/civitai-krea2/` |
|
|
| `all` | same as discover+scrape | discover → scrape → split (launcher default) |
|
|
|
|
Global: `--out` = app root (writes under `datasets/civitai`).
|
|
|
|
## Layout
|
|
|
|
| Path | Role |
|
|
| --- | --- |
|
|
| `civitai/catalog/models.json` | Discovered Krea2 checkpoints / LoRAs |
|
|
| `civitai/catalog/images.jsonl` | Canonical rows (resume by id) |
|
|
| `civitai/by_kind/` | checkpoint vs lora |
|
|
| `civitai/by_rating/` | pg / pg13 / r / x |
|
|
| `civitai/search.jsonl` | Compact Civitai FTS source → `books/civitai-krea2` on split |
|
|
| `../books/civitai-krea2/` | Assistent book (`book.yaml`, `search.jsonl`, `meta.json`) |
|
|
|
|
On `up` / `seed-personas` / `seed-books`: changed books push → `/mnt/swarm_data/Assistent/books/` (sha diff). Legacy `civitai-examples.jsonl` only if `books/` empty.
|
|
|
|
See [books/README.md](../books/README.md), [docs/cli.md](../docs/cli.md), [docs/local-folders.md](../docs/local-folders.md).
|