Leonid PershinandCursor
6dfbf4e702
Local civitai-dataset launchers collect ~2000 prompt/params rows without images; search.jsonl is pushed on up for cheap example lookup. Co-authored-by: Cursor <cursoragent@cursor.com>
1.9 KiB
1.9 KiB
Civitai Krea2 datasets (local)
Scraped generation metadata for Assistent training + cheap FTS search. No image files.
Not part of gpu-rent Typer CLI and not models.yaml / gpu-rent push. Separate entrypoint:
civitai-dataset.bat # Windows (or .\civitai-dataset.ps1 / ./civitai-dataset.sh)
# default = all → discover + scrape + split
python -m gpu_rent.civitai_dataset all
Needs CIVITAI_API_TOKEN and preferably CIVITAI_API_HOST=civitai.red in .env.
Commands
| Command | Flags | Role |
|---|---|---|
discover |
--top-checkpoints (20), --top-loras (30) |
Popular Krea2 ckpt/LoRAs (+ versions from models.yaml) → catalog/models.json |
scrape |
--target (2000), --per-version, --min-score |
Gallery metadata only; resume by id; NSFW→rating; skip minor tags |
split |
— | by_kind/ / by_rating/ / train.jsonl / search.jsonl |
all |
same as discover+scrape | discover → scrape → split (launcher default) |
Global: --out = app root (writes under datasets/civitai).
Layout
| Path | Role |
|---|---|
civitai/catalog/models.json |
Discovered Krea2 checkpoints / LoRAs |
civitai/catalog/images.jsonl |
Canonical rows (resume by id) |
civitai/by_kind/ |
checkpoint vs lora |
civitai/by_rating/ |
pg / pg13 / r / x |
civitai/train.jsonl |
Alpaca instruction/output for Assistent Обучение |
civitai/search.jsonl |
Compact FTS index → pushed to VM as Assistent/civitai-examples.jsonl on up / seed-personas |
Push of search.jsonl is skipped if the file is missing. Weights stay in models.md (Civitai seed / Models/ / capture).
Git ignores the data; this README stays.
See also: docs/cli.md (civitai-dataset block), docs/extensions.md, docs/local-folders.md.