Files
HVideoTool/CLAUDE.md
T

361 lines
26 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
Guidance for Claude Code (and other agents) working in this repository.
## Memory: EchoVault (read this first)
This project uses the **EchoVault** MCP server for persistent, cross-session memory.
Prior sessions store architectural decisions, fixed bugs, and gotchas there. Follow
this protocol every session:
1. **At session start — load context.** Call `memory_context` (project is
auto-detected from cwd) before doing any work. Use `memory_search` for specific
topics (e.g. "detector model", "classic-cv", "false positives").
2. **During work — search before re-deciding.** When the task touches an area that
may have prior context, `memory_search` it first instead of re-deriving decisions.
3. **Before ending a session — save what matters.** Call `memory_save` when you made
a design decision, fixed a bug (include root cause + fix), found a non-obvious
gotcha, or the user corrected/clarified a requirement. Pick the right `category`
(`decision` / `bug` / `pattern` / `learning` / `context`). Do **not** save trivia,
things obvious from the code, or duplicates.
EchoVault is the source of truth for *why* things are the way they are; this file is
the stable, high-level map. When they disagree, trust on-disk code first, then
EchoVault, then this file — and update whichever is stale.
## What this project is
**HVideoTool** is a Windows-first desktop GUI utility that **detects already-applied
censorship** (mosaic, pixelation, blur, black bars) in **images**, and draws outlines
over the detected censored regions. You open a **project** (see below); it runs each
image through a detector, draws the regions, and shows a detailed per-image list of
what it found.
> **Projects (this session).** Work is organized into **projects** — a project is a
> folder holding `project.json` (metadata + per-project settings) · `frames/` (the
> images) · `detections.json` (the detection cache) · `collections/Избранное` (the
> single default "favorites" collection). See `core/project.py`. The per-project settings (detector, model,
> threshold, restore engine) live in `project.json`; the global `settings.json` only
> seeds the **defaults for new projects**. The old "open a bare folder" flow is now
> "Импортировать папку как проект…" (copies images into a new project's `frames/`).
> **Scope is still narrow.** It used to extract frames from video, detect, and play
> back with overlays (a thread-based "project model"). That video *playback* pipeline
> was **removed**; the new "projects" are just an on-disk layout, NOT worker threads or
> playback. The tool remains a synchronous, single-threaded **image inspector**.
Keep this scope sharp:
- Primary job is **detection + overlay/inspection**. A **restoration** ("расцензурить")
step was added later (user-requested): on-demand (single frame **or** whole-project
batch into `restored/`), behind a `Restorer` interface. **Detection is YOLO-only and
restoration is DeepMosaics-only** — the noisy classic-CV detector (+ the `combined`
composite) and the cv2 `inpaint` baseline (filled but didn't reconstruct) were
**removed** as "works poorly". DeepMosaics has two engines: **image** (per-frame) and
**video** (BVDNet, temporal — uses neighbour frames). Its GPL-3.0 network code is
**vendored** under `core/restore/_deepmosaics/` and run in-process (user supplies only
the weights). Because of that vendoring the **whole project is GPL-3.0**. LADA
(BasicVSR++) is a possible future engine, not wired.
- Still **no diffusion / ControlNet / SDXL**. (`xinsir/controlnet-union-sdxl-1.0` was
rejected early — a generative *conditioning* model, not a censorship restorer. Don't
reintroduce it.) Restoration, if upgraded, uses a mosaic-removal model (DeepMosaics/
LADA), not a general text-to-image diffusion pipeline.
- It detects **already-censored** regions, not "content that should be censored"
(i.e. not an NSFW classifier).
- Video is only a one-shot frame-extraction convenience (see below): "Создать из
ролика…" makes a new project and decodes the clip into its `frames/`. Detection and
restoration operate on the project's images.
## Target environment
- **OS:** Windows 11 x64 (primary). Use PowerShell syntax in commands.
- **Python:** 3.11+.
- **GPU:** NVIDIA + CUDA via PyTorch, for the YOLO detector and the DeepMosaics
restorer. CPU fallback works but is slow (esp. DeepMosaics / the temporal BVDNet).
## Tech stack (decided)
| Concern | Choice |
|----------------|---------------------------------------------|
| GUI | PySide6 (Qt 6) — LGPL |
| Image IO | OpenCV (`opencv-python`) + NumPy, unicode-safe via `core/imageio.py` |
| Detector | Ultralytics YOLO (LADA weights) behind a pluggable interface (YOLO-only) |
Torch/CUDA + Ultralytics enter with the YOLO detector. Keep that dependency optional
(the `yolo` extra in `pyproject.toml` pulls only Ultralytics; torch is installed
separately per the README). Both detection (YOLO) and restoration (DeepMosaics) now
require torch — there's no longer a torch-free detector.
## Architecture (as implemented)
> This reflects the actual code on disk. The GUI is mostly synchronous, but the
> **heavy compute (detection + restoration) runs on a background thread** so the UI
> stays responsive — see `ui/workers.py` and the "Background jobs" bullet (this reverses
> the earlier "no worker threads" rule; `processEvents` can't unfreeze a single multi-
> second `detector.detect()`/DeepMosaics call). Work is organized into **projects**
> (`core/project.py`): a project folder holds `project.json` (metadata + per-project
> settings), `frames/` (the images), `detections.json` (the detection cache, at the
> project root — no longer a sidecar next to the images), and `collections/Избранное`
> (the favorites collection). The only video touch is a one-shot "Создать из ролика…"
> that creates a new project and decodes a clip into its `frames/` via the **ffmpeg
> CLI** (cv2.VideoCapture fallback; NOT PyAV). When code and this file disagree, trust
> the code.
```
hvideotool/
├── __main__.py # entry point + CLI (optional project path, --detector, --model)
├── app.py # QApplication bootstrap; run(config, target=None) — opens/auto-reopens a project
├── config.py # AppConfig + DetectionConfig/OverlayConfig (thresholds live here)
├── settings_store.py # new-project DEFAULTS + last/recent projects to ~/HVideoTool/settings.json
├── ui/
│ ├── main_window.py # the whole UI: toolbar + [file list | image view | detail table]
│ ├── workers.py # Job (QRunnable): runs detect/restore off-thread, results via Qt signals
│ └── image_view.py # renders an image + draws polygon/bbox overlays (QPainter); can highlight one
└── core/
├── imageio.py # unicode-safe imread/imwrite (np.fromfile + imdecode)
├── torch_info.py # probe torch/CUDA (gather/reason/install_hint) for the device badge; no Qt
├── project.py # Project: layout (project.json/frames/detections.json/collections) + per-project settings
├── video/
│ ├── extract.py # extract_frames(): ffmpeg CLI (cv2 fallback) -> JPGs; keyframe/step modes + downscale
│ └── frame.py # Frame dataclass (image BGR, index, pts) — the detector input type
├── restore/ # "un-censor" (DeepMosaics only; per-frame OR temporal)
│ ├── base.py # Restorer ABC: restore(image, dets, should_cancel) + restore_sequence (batch/temporal) + .temporal flag; Cancelled exc
│ ├── factory.py # build_restorer(name, config) -> deepmosaics | deepmosaics_video (lada = TODO)
│ ├── deepmosaics.py # DeepMosaicsRestorer (image, per-frame) + DeepMosaicsVideoRestorer (BVDNet, temporal); in-process, load once; uses _deepmosaics/
│ └── _deepmosaics/ # VENDORED DeepMosaics models/+util/ (GPL-3.0) — added to sys.path at import
└── detection/ # YOLO only
├── base.py # Detector ABC: detect(frame) -> list[Detection]
├── factory.py # build_detector(config) -> yolo (the only kind)
├── types.py # Detection (+ to_dict/from_dict), CensorType enum
├── cache.py # save/load the project detection cache (detections.json): cache_file + base_dir args
└── yolo.py # YoloDetector — Ultralytics YOLO-seg; lazy-imports torch/ultralytics
```
### How it works
- `MainWindow` holds the config, builds the detector lazily via `build_detector`
(cached by detector+model+conf in `_make_detector`), and keeps `_results: dict[path
-> list[Detection]]` as the detection cache.
- **Background jobs (`ui/workers.py`).** Detection and restoration are CPU-heavy and
would freeze the GUI, so they run on a `QThreadPool` thread via `Job` (a `QRunnable`
wrapping `fn(job)`); results return to the GUI through queued Qt signals
(`tick`/`progress`/`done`/`failed`). `MainWindow._start_job(fn, total, on_tick, on_done)`
starts one (only one at a time — `_busy` guards entry points), `_finish_job`/
`_on_job_failed` end it. `_make_detector`/`_make_restorer`, image reads, and
`engine.detect/restore` all run **inside the worker** (`_compute` is the pure
read+detect helper); the `fn` must touch NO Qt widgets — it emits plain data that the
GUI-thread slots (`_apply_detection`, restore `tick`) apply. `_begin_busy` disables
`model_action` for the duration (it'd race the running detector). This is the
deliberate exception to the old single-threaded rule (CPU YOLO/DeepMosaics per-call
latency can't be hidden with `processEvents`).
- **Device badge + CUDA diagnostics.** A clickable status-bar chip (`device_badge`) shows
"⚡ CUDA" (green) or "🖥 CPU" (orange). `_probe_device` runs `core/torch_info.gather()` in
a background `Job` at startup (it imports torch AND shells out to `nvidia-smi`, so it's off
the GUI thread) → `_set_device_badge`. `gather()` collects torch facts (version, built_cuda,
cuda_available, device_name) **and** NVIDIA facts (gpus, driver_version, max cuda_driver).
Clicking (`_show_device_info`) opens a diagnostic **QDialog** (not QMessageBox — its text
wasn't copyable): a read-only monospace `QPlainTextEdit` with `torch_info.analyze(info)`
`{summary, details, steps, command}` — a verdict on *why* it's on CPU (CPU-only `+cpu`
build / no GPU / driver-too-old-for-built-CUDA) and the exact pip fix. Buttons: "Скопировать
команду установки" (`_copy_install_command` → the recommended cu121/cu118 command, picked by
`recommend_channel` from the driver's CUDA) and "Проверить заново" (re-runs `_probe_device`).
`core/torch_info.py` is pure (no Qt); subprocess uses `CREATE_NO_WINDOW` on Windows.
- **Projects (`core/project.py`).** `MainWindow._project` is the open `Project`;
`_folder` is kept as a synonym for `project.frames_dir` so the rest of the code
(navigation, cache, tags) didn't need rewiring. Entry points: "Создать проект…"
(`_create_project`), "Открыть проект…" (`_open_project_dialog`), "Импортировать папку
как проект…" (`_import_folder_as_project` — copies images into a new project's
`frames/`, carries over an old `.hvideotool_detections.json` sidecar if present), and
a "Недавние проекты" submenu. `_open_project(project)` is the core open: it
`apply_to_config`s the project's settings, syncs the toolbar widgets without signal
loops (`_sync_settings_ui`), titles the window, records last/recent, and lists
`frames/`. On startup `app.run` opens the CLI `target` or auto-reopens
`settings_store.last_project()` (`_auto_open_last`).
- **Per-project settings.** Detector/model/threshold/restore engine live in
`project.json` (`Project.settings`, `_SETTING_KEYS`). `_persist_settings()` writes both
the global defaults (for new projects) **and** the open project. Global `settings.json`
is now only defaults + last/recent projects.
- Listing image files (`_IMAGE_EXTS`) from `frames/` uses a progress bar (bulk insert
with `setUpdatesEnabled(False)` + periodic `processEvents`), so a big project doesn't
freeze silently.
- **Viewing and detecting are decoupled on purpose** (so browsing stays instant even
with a slow CPU detector): selecting a file only *shows* it with its cached result
(header reads "не рассчитано" if none). Detection runs on **double-click**, the
"Рассчитать кадр" action (Space → `_recompute_current`, force-recomputes current), or
"Детектировать все" (whole folder, progress bar). Do NOT re-add auto-detect-on-select.
Results cache in `_results`; the file-list row gets a count suffix when computed.
Switching the model clears the cache (`_choose_model``_invalidate_results`).
- **Detection cache (persisted).** `_results` is mirrored to the project's
`detections.json` (`core/detection/cache.py`, at `project.cache_path`; keyed by
**basename** so it survives moving the project). `cache.save_results`/`load_results`
take the cache file and the image `base_dir` (= `project.frames_dir`) separately, since
the cache lives at the project root, not next to the images. It's tagged with
the detector identity (`_results_key` = detector + model + conf/imgsz); on open,
`_load_cached_results` reloads it **only on a key match** (else ignored, not shown as
current). Saved (`_save_results`, skipped when `_results` is empty so it never clobbers
a good cache with nothing) after detect-all (incl. cancel → partial), single
detect/recompute, move-to-collection, and on `closeEvent`.
**"Детектировать все" is incremental** (skips already-cached frames → resumes/top-ups);
**"Все заново"** (`_detect_all(force=True)`) clears the cache first (full regen);
**"Рассчитать кадр"/Space** always recomputes the one current frame. Empty list in the
cache = "checked, clean" (tinted green, no mark); `in _results` distinguishes it from
"not computed".
- **Favorites (curation).** Curation was simplified (user request) to a **single
default collection** — no create/select/browse UI. The "★ В избранное" button (left
pane, under the file list) / "В избранное" menu item / Ctrl+M → `_move_to_favorites`
**moves** (`shutil.move`, not copy) the `ExtendedSelection`-selected frames into
`project.favorites_dir` (`collections/Избранное`, `FAVORITES_DIR` in `core/project.py`,
created lazily on first move), removing them from list/`_files`/cache. `_unique_dest`
avoids clobbering (`foo.jpg``foo (1).jpg`). Use case: flag good frames into a
training/example set while inspecting detections.
- **Restoration ("Расцензурить кадр" / "Расцензурить все").** DeepMosaics-only, and it
**locates the mosaic itself** — so restoration is **fully decoupled from detection**: no
detector runs in either path (detections are passed as `[]`). Single-frame: a toolbar
action reads the current frame + runs `self._restorer` (built via `build_restorer`) **on
a background job** (`_restore_current`; `done` stores `_restored[path]` + shows it). "Показать
оригинал/результат" toggles (`_showing_restored`); "Сохранить результат" writes
`<stem>_restored.jpg` beside the frame. **Batch ("Расцензурить все" / "Все заново",
`_restore_all(force)`)** mirrors `_detect_all`: a single background job restores every
frame and writes results to the project's **`restored/`** dir (`Project.restored_dir`,
basename-mirrored, kept OUT of `frames/` so outputs aren't re-listed/re-restored); the
per-frame engine **skips frames already in `restored/`** unless `force` (resume). The
engines are **DeepMosaics** (`restore/deepmosaics.py`), run **in-process** from the
vendored `_deepmosaics/` code, loading the BiSeNet locator + generator **once** (lazy,
cached on the instance):
- `deepmosaics` (image, per-frame): reproduces `cleanmosaic_img_server` (locate mosaic
→ run generator on the crop → feather back), ~0.3 s/frame cached on CPU. Image model
`clean_youknow_resnet_9blocks.pth`.
- `deepmosaics_video` (**temporal, BVDNet**): `DeepMosaicsVideoRestorer`, `.temporal=True`.
Reproduces `cleanmosaic_video_fusion` — per target frame it feeds the net a window of
`T=5` neighbour frames sampled at step `S=3` around it (`N=2` each side, clamped at the
sequence edges) **plus its own previous output (recurrent)**, for temporal coherence.
Because of that recurrence it must run a **contiguous, ordered range** — it implements
`restore_sequence(count, get_frame, get_dets, emit, should_cancel)` (the batch run uses
it; single-frame `restore` degrades to a window of the same frame). Needs the **video**
weights `clean_youknow_video.pth` (+ `mosaic_position.pth` beside). `INPUT_SIZE=256`.
`restore_sequence` is on the `Restorer` ABC (default = independent per-frame loop);
`_restore_all` dispatches on `restorer.temporal` (temporal → `restore_sequence` over the
whole range; per-frame → resumable loop with skip-existing). `should_cancel`
(= `lambda: job.cancelled`) is polled so "■ Стоп" stops it; engines raise `Cancelled`,
which `Job.run` reports as a clean cancel. Engine + weights are set in `RestoreDialog`
(Файл → Движок восстановления…) — the model dropdown shows image vs video weights per
selected engine — persisted, and built lazily/cached in `_make_restorer`. NOTE:
DeepMosaics locates mosaics itself (its `mosaic_position.pth`) — our detections aren't
passed to it. To add another engine (e.g. LADA), implement `core/restore/base.Restorer`
(set `.temporal` + override `restore_sequence` if it needs neighbours) and register it in
`restore/factory.build_restorer`.
- **Navigation bar** under the image (`_build_nav_bar`): prev/next frame (◀ ▶, keys
`,`/`.`), a scrubber `frame_slider` across the whole sequence, a `pos_label`
("row / n"), and jump-to-detection (◀ детекция / детекция ▶, keys `[`/`]`,
`_step_hit` scans `_results` for the next non-empty frame). The slider and file list
are kept in sync via `_update_nav` guarded by `_nav_sync` (avoids signal loops); all
navigation ultimately drives `file_list.setCurrentRow`. The scrubber is a custom
`MarkerSlider` (`ui/marker_slider.py`) that paints cyan ticks at frames with
detections (`_refresh_marks` projects `_results` onto row indices; per-pixel deduped
so big folders stay cheap). File-list rows are tinted too (`_tag_file`): red =
censorship found, green = checked & clean. Both reset on `_invalidate_results`.
- **Cancellation (cooperative).** A single "■ Стоп" toolbar action (Esc) cancels the
running op. `_begin_busy(total)` / `_end_busy()` toggle `self._busy` + the Stop button +
the progress bar (`total=None` → indeterminate). For **background jobs** (detection,
restore) `_request_cancel` calls `self._job.cancel()`; the worker loop checks
`job.cancelled` between frames and restore polls it via `should_cancel`. The still-
synchronous loops (`_load_folder` listing, import copy, video extraction — its
`progress` cb returns `not self._cancel`) check `self._cancel` between `processEvents`
ticks. Entry points guard with `if self._busy: return` (notably `_move_to_favorites`,
which mutates `_files` that a detect-all job reads — so a snapshot/pending list is used).
`closeEvent` cancels a running job and `waitForDone(3000)` before tearing down.
- `image_view.ImageView` draws the image scaled-to-fit plus overlays. Overlay
visibility/threshold are applied at paint time. Selecting a row in the detail table
calls `set_highlight(i)` — that detection is drawn boldly (even below threshold) and
the rest dim. The detail table lists ALL detections (sorted by score), so sub-threshold
hits are still visible for debugging; the threshold only affects what's drawn.
### Separation of concerns
- `ui/` must not import `torch` / `ultralytics` directly. It builds detectors only via
`core/detection/factory.build_detector` and talks to `core/` through the `Detector`
interface and the `Detection`/`CensorType` types.
- Detection is YOLO-only and restoration is DeepMosaics-only. If you re-add an engine
kind, implement `core/detection/base.Detector` / `core/restore/base.Restorer`, register
the string in the respective `factory`, and add it to `DETECTORS`/`RESTORERS` in
`config.py` (and `normalize_config`). There's no detector dropdown anymore — the toolbar
just shows "Детектор: YOLO"; restoration engines are chosen in `RestoreDialog`.
## Commands
```powershell
python -m venv .venv; .\.venv\Scripts\Activate.ps1
pip install -e ".[yolo]" # YOLO needs ultralytics; install torch separately (README)
python -m hvideotool # reopen the last project (or create/open one in-app)
python -m hvideotool "C:\path\to\MyProject" --model models\lada_mosaic_detection_model_v4_accurate.pt
```
No formal test suite. Headless sanity check: set `QT_QPA_PLATFORM=offscreen`, build a
`MainWindow`, `Project.create(tmp)` + copy a few images into `frames/`,
`_open_project(project)`, drive `file_list.setCurrentRow(...)`, and read
`detail_table` / `detail_header`. Or run `build_detector(config).detect(...)` on a
frame directly.
## Conventions
- Match the style of surrounding code; keep `core/` free of Qt where reasonable.
- Type hints on public functions and the `Detector` interface.
- Model weights (`.pt`) and large media are **not** committed — keep them in `models/`
and `.gitignore`d.
- User-facing strings / README are in Russian; code identifiers and this file in English.
## Gotchas
- **Wrong YOLO model = "noise".** The YOLO detector needs a **censorship** model
(LADA `models\lada_mosaic_detection_model_v4_accurate.pt`). If `model_path` points at
a generic COCO model (e.g. the `yolo11n-seg.pt` in the repo root, which Ultralytics
auto-downloads / is the training base), it detects people/objects and maps them to
`CensorType.UNKNOWN` → purple boxes that look like noise. This was a real user trap.
**A model is auto-picked on project open** via `MainWindow._ensure_model()`
`_auto_find_model()`: it scans `./models/**.pt` and matches only filenames containing
`lada`/`mosaic` (so it skips the COCO `yolo11n-seg.pt` trap), no prompt; otherwise the
user picks via "Модель…" (`_choose_model`).
- **classic-CV / inpaint were removed (worked poorly).** The classic-CV detector was
noisy/approximate on real footage (false positives on skin/hair/fabric/JPEG; missed
real mosaic after downscale) and the `combined` mode + cv2 `inpaint` baseline went with
it. Detection is YOLO-only, restoration is DeepMosaics-only. `normalize_config` coerces
any leftover `classic`/`combined`/`inpaint` in old settings/projects to `yolo`/`deepmosaics`.
- **Domain matters.** LADA is trained on REAL video (JAV). It detects some anime mosaic
but not all. The real anime fix is *retraining* a YOLO11-seg (see `scripts/training/`).
- **YOLO detector = LADA weights** ([HF `ladaapp/lada`](https://huggingface.co/ladaapp/lada)).
YOLO **segmentation** model, classes `{0: mosaic_nsfw, 1: mosaic_sfw_head}` → both map
to `CensorType.MOSAIC` (`_name_to_type` matches "mosaic" in the class name). Detects
mosaic only. Weights + Ultralytics are AGPL-3.0 (accepted). `yolo.py` lazy-imports
`torch`/`ultralytics`.
- **No model weights in the repo.** Code must fail with a clear, actionable message
when the model path is missing — not a raw stack trace (`factory._require_model`,
`YoloDetector.__init__`).
- **CUDA/torch install is environment-specific.** Don't add torch to core deps; it
stays out (the `yolo` extra pulls only Ultralytics) and is installed separately.
- **CPU-only torch must not request CUDA.** A `+cpu` torch build raises "Torch not
compiled with CUDA enabled" the moment something calls `.cuda()`. Both engines guard
for this: `YoloDetector` picks `cuda` only when `torch.cuda.is_available()` (even an
explicit `yolo_device="cuda"` is downgraded to cpu); both `DeepMosaicsRestorer._ensure_loaded`
and `DeepMosaicsVideoRestorer._ensure_loaded` force `gpu_id="-1"` when CUDA is absent (the
vendored `model_util.todevice` / `data.im2tensor`/`to_tensor` call `.cuda()` for any
`gpu_id != "-1"`, e.g. the `dm_gpu="0"` default). So a wrong/CPU-only torch falls back to
CPU instead of crashing (the temporal BVDNet engine is heavy on CPU, though).
- **QImage from a numpy buffer must be `.copy()`d** (see `ImageView.set_image`),
otherwise it aliases a buffer that gets freed → garbage/crash.
- **Always use `core/imageio.py`** (`imread_unicode`/`imwrite_unicode`) for images —
`cv2.imread`/`imwrite` silently fail on non-ASCII Windows paths.
- Don't reintroduce any generative / ControlNet dependency, nor the removed video
*playback pipeline* (PyAV, producer/consumer worker threads, player, project session).
(The new `core/project.py` is an on-disk layout, not that thread-based "project
model".) NOTE: a **single** background `Job` thread for detect/restore (`ui/workers.py`)
IS in scope now (keeps the GUI responsive) — that's different from the rejected
multi-thread video pipeline. The one allowed video touch is
`core/video/extract.py` (one-shot decode → a new project's `frames/`, behind "Создать
из ролика…"): ffmpeg CLI — `_find_ffmpeg()` prefers PATH, else the binary bundled by
the `imageio-ffmpeg` dep, else cv2 fallback. Keyframe-only `-skip_frame nokey` is
~10× faster than every-frame; `-hwaccel` does NOT help (GPU transfer overhead). Use
ffmpeg/cv2, not PyAV, and keep it synchronous. Decoding every frame is the inherent
cost — the speed lever is decoding *fewer* frames (keyframes).