Refactor HVideoTool to support project-based workflow: introduced project management features, updated UI for project handling, and enhanced documentation in README and CLAUDE.md. The tool now organizes images and settings into projects, improving usability and detection caching.

2026-06-07 04:53:27 +03:00
parent 7f0121b7df
commit e27dfdf518
27 changed files with 2998 additions and 387 deletions
@@ -27,30 +27,41 @@ EchoVault, then this file — and update whichever is stale.

 **HVideoTool** is a Windows-first desktop GUI utility that **detects already-applied
 censorship** (mosaic, pixelation, blur, black bars) in **images**, and draws outlines
-over the detected censored regions. You open a folder of images; it runs each through
-a detector, draws the regions, and shows a detailed per-image list of what it found.
+over the detected censored regions. You open a **project** (see below); it runs each
+image through a detector, draws the regions, and shows a detailed per-image list of
+what it found.

-> **Scope was deliberately narrowed (this session).** It used to extract frames from
-> video, detect, and play back with overlays (a "project model" with worker threads).
-> That whole video pipeline was **removed** — the tool is now a simple **image-folder
-> inspector** for viewing/debugging detector output on test images. Pre-extract video
-> to frames externally if you need that.
+> **Projects (this session).** Work is organized into **projects** — a project is a
+> folder holding `project.json` (metadata + per-project settings) · `frames/` (the
+> images) · `detections.json` (the detection cache) · `collections/Избранное` (the
+> single default "favorites" collection). See `core/project.py`. The per-project settings (detector, model,
+> threshold, restore engine) live in `project.json`; the global `settings.json` only
+> seeds the **defaults for new projects**. The old "open a bare folder" flow is now
+> "Импортировать папку как проект…" (copies images into a new project's `frames/`).
+
+> **Scope is still narrow.** It used to extract frames from video, detect, and play
+> back with overlays (a thread-based "project model"). That video *playback* pipeline
+> was **removed**; the new "projects" are just an on-disk layout, NOT worker threads or
+> playback. The tool remains a synchronous, single-threaded **image inspector**.

 Keep this scope sharp:

 - Primary job is **detection + overlay/inspection**. A **restoration** ("расцензурить")
  step was added later (user-requested): per-frame, on-demand, behind a `Restorer`
-  interface. The shipped engine is a cv2 **inpaint baseline** (fills, does NOT truly
-  reconstruct); a generative engine (DeepMosaics / LADA BasicVSR++) is the intended
-  real engine but needs weights + a CUDA GPU and is not wired yet.
+  interface. Two engines: a cv2 **inpaint baseline** (fills, does NOT reconstruct) and
+  **DeepMosaics** — real generative mosaic removal, its GPL-3.0 network code **vendored**
+  under `core/restore/_deepmosaics/` and run in-process (user supplies only the weights).
+  Because of that vendoring the **whole project is GPL-3.0**. LADA (BasicVSR++) is a
+  possible future engine, not wired.
 - Still **no diffusion / ControlNet / SDXL**. (`xinsir/controlnet-union-sdxl-1.0` was
  rejected early — a generative *conditioning* model, not a censorship restorer. Don't
  reintroduce it.) Restoration, if upgraded, uses a mosaic-removal model (DeepMosaics/
  LADA), not a general text-to-image diffusion pipeline.
 - It detects **already-censored** regions, not "content that should be censored"
  (i.e. not an NSFW classifier).
- Video is only a one-shot frame-extraction convenience (see below); detection and
-  restoration operate on image folders.
+- Video is only a one-shot frame-extraction convenience (see below): "Создать из
+  ролика…" makes a new project and decodes the clip into its `frames/`. Detection and
+  restoration operate on the project's images.

 ## Target environment

@@ -75,37 +86,42 @@ torch present.
 ## Architecture (as implemented)

 > This reflects the actual code on disk. It is a synchronous, single-threaded GUI app
-> — no worker threads, no project/cache files. The only video touch is a one-shot
-> "Создать из ролика…" that decodes a clip to a folder of JPGs via the **ffmpeg CLI**
-> (cv2.VideoCapture fallback; NOT PyAV); detection still works on image folders only.
-> Detection runs on the GUI
-> thread (lazily per image, or via "Детектировать все"). When code and this file
-> disagree, trust the code.
+> — no worker threads. Work is organized into **projects** (`core/project.py`): a
+> project folder holds `project.json` (metadata + per-project settings), `frames/` (the
+> images), `detections.json` (the detection cache, at the project root — no longer a
+> sidecar next to the images), and `collections/Избранное` (the favorites collection). The only
+> video touch is a one-shot "Создать из ролика…" that creates a new project and decodes
+> a clip into its `frames/` via the **ffmpeg CLI** (cv2.VideoCapture fallback; NOT
+> PyAV). Detection runs on the GUI thread (lazily per image, or via "Детектировать
+> все"). When code and this file disagree, trust the code.

 ```
 hvideotool/
-├── __main__.py          # entry point + CLI (optional folder arg, --detector, --model)
-├── app.py               # QApplication bootstrap; run(config, folder=None)
+├── __main__.py          # entry point + CLI (optional project path, --detector, --model)
+├── app.py               # QApplication bootstrap; run(config, target=None) — opens/auto-reopens a project
 ├── config.py            # AppConfig + DetectionConfig/OverlayConfig (thresholds live here)
-├── settings_store.py    # persist detector/model/threshold/last_dir to ~/HVideoTool/settings.json
+├── settings_store.py    # new-project DEFAULTS + last/recent projects to ~/HVideoTool/settings.json
 ├── ui/
 │   ├── main_window.py   # the whole UI: toolbar + [file list | image view | detail table]
 │   └── image_view.py    # renders an image + draws polygon/bbox overlays (QPainter); can highlight one
 └── core/
    ├── imageio.py       # unicode-safe imread/imwrite (np.fromfile + imdecode)
+    ├── project.py       # Project: layout (project.json/frames/detections.json/collections) + per-project settings
    ├── video/
    │   ├── extract.py   # extract_frames(): ffmpeg CLI (cv2 fallback) -> JPGs; keyframe/step modes + downscale
    │   └── frame.py     # Frame dataclass (image BGR, index, pts) — the detector input type
    ├── restore/         # "un-censor" detected regions (per-frame)
-    │   ├── base.py      #   Restorer ABC: restore(image, detections) -> image
+    │   ├── base.py      #   Restorer ABC: restore(image, detections, should_cancel=None) -> image; Cancelled exc
    │   ├── factory.py   #   build_restorer(name, config) -> inpaint | deepmosaics (lada = TODO)
    │   ├── inpaint.py   #   InpaintRestorer (cv2) — baseline, fills not reconstructs
-    │   ├── deepmosaics.py #  DeepMosaicsRestorer — shells out to user-installed DeepMosaics
+    │   ├── deepmosaics.py #  DeepMosaicsRestorer — in-process, loads models once; uses _deepmosaics/
+    │   ├── _deepmosaics/ #  VENDORED DeepMosaics models/+util/ (GPL-3.0) — added to sys.path at import
    │   └── mask.py      #   detections_to_mask(shape, dets, dilate)
    └── detection/
        ├── base.py      # Detector ABC: detect(frame) -> list[Detection]
        ├── factory.py   # build_detector(config) -> classic | yolo | combined
        ├── types.py     # Detection (+ to_dict/from_dict), CensorType enum
+        ├── cache.py     # save/load the project detection cache (detections.json): cache_file + base_dir args
        ├── classic_cv.py  # ClassicCVDetector — heuristic; accepts a `types` filter
        ├── yolo.py      # YoloDetector — Ultralytics YOLO-seg; lazy-imports torch/ultralytics
        └── composite.py # CompositeDetector — merge detectors + IoU dedup
@@ -116,8 +132,23 @@ hvideotool/
 - `MainWindow` holds the config, builds the detector lazily via `build_detector`
  (cached by detector+model+conf in `_make_detector`), and keeps `_results: dict[path
  -> list[Detection]]` as the detection cache.
- "Открыть папку" lists image files (`_IMAGE_EXTS`) with a progress bar (bulk insert
-  with `setUpdatesEnabled(False)` + periodic `processEvents`), so a big folder doesn't
+- **Projects (`core/project.py`).** `MainWindow._project` is the open `Project`;
+  `_folder` is kept as a synonym for `project.frames_dir` so the rest of the code
+  (navigation, cache, tags) didn't need rewiring. Entry points: "Создать проект…"
+  (`_create_project`), "Открыть проект…" (`_open_project_dialog`), "Импортировать папку
+  как проект…" (`_import_folder_as_project` — copies images into a new project's
+  `frames/`, carries over an old `.hvideotool_detections.json` sidecar if present), and
+  a "Недавние проекты" submenu. `_open_project(project)` is the core open: it
+  `apply_to_config`s the project's settings, syncs the toolbar widgets without signal
+  loops (`_sync_settings_ui`), titles the window, records last/recent, and lists
+  `frames/`. On startup `app.run` opens the CLI `target` or auto-reopens
+  `settings_store.last_project()` (`_auto_open_last`).
+- **Per-project settings.** Detector/model/threshold/restore engine live in
+  `project.json` (`Project.settings`, `_SETTING_KEYS`). `_persist_settings()` writes both
+  the global defaults (for new projects) **and** the open project. Global `settings.json`
+  is now only defaults + last/recent projects.
+- Listing image files (`_IMAGE_EXTS`) from `frames/` uses a progress bar (bulk insert
+  with `setUpdatesEnabled(False)` + periodic `processEvents`), so a big project doesn't
  freeze silently.
 - **Viewing and detecting are decoupled on purpose** (so browsing stays instant even
  with a slow CPU detector): selecting a file only *shows* it with its cached result
@@ -126,38 +157,68 @@ hvideotool/
  "Детектировать все" (whole folder, progress bar). Do NOT re-add auto-detect-on-select.
  Results cache in `_results`; the file-list row gets a count suffix when computed.
  Switching detector/model clears the cache (`_invalidate_results`).
- **Collections (curation).** A combo in the left pane (next to the file list, since it
-  acts on the list selection — not on the toolbar, to keep that uncluttered)
-  (`collection_combo`) picks the active destination; `_refresh_collections()` repopulates it from sibling folders of the
-  opened folder (`_collections_base()` = the opened folder's parent, else
-  `~/HVideoTool/collections`) on load/create, so previously-made collections are
-  reselectable. Items carry the path in itemData; "— не выбрана —" = None,
-  "Выбрать папку…" = `"__browse__"` sentinel → `_browse_collection()` for an arbitrary
-  folder (kept in the combo even if outside base). "Создать…" makes a new one and
-  selects it. The file list is `ExtendedSelection`; "В коллекцию" / Ctrl+M **moves**
-  (`shutil.move`, not copy) the selected frames there, removing them from
-  list/`_files`/cache. `_unique_dest` avoids clobbering (`foo.jpg` → `foo (1).jpg`).
-  Use case: sort frames into a training/example set while inspecting detections.
+- **Detection cache (persisted).** `_results` is mirrored to the project's
+  `detections.json` (`core/detection/cache.py`, at `project.cache_path`; keyed by
+  **basename** so it survives moving the project). `cache.save_results`/`load_results`
+  take the cache file and the image `base_dir` (= `project.frames_dir`) separately, since
+  the cache lives at the project root, not next to the images. It's tagged with
+  the detector identity (`_results_key` = detector + model + conf/imgsz); on open,
+  `_load_cached_results` reloads it **only on a key match** (else ignored, not shown as
+  current). Saved (`_save_results`, skipped when `_results` is empty so it never clobbers
+  a good cache with nothing) after detect-all (incl. cancel → partial), single
+  detect/recompute, move-to-collection, and on `closeEvent`.
+  **"Детектировать все" is incremental** (skips already-cached frames → resumes/top-ups);
+  **"Все заново"** (`_detect_all(force=True)`) clears the cache first (full regen);
+  **"Рассчитать кадр"/Space** always recomputes the one current frame. Empty list in the
+  cache = "checked, clean" (tinted green, no mark); `in _results` distinguishes it from
+  "not computed".
+- **Favorites (curation).** Curation was simplified (user request) to a **single
+  default collection** — no create/select/browse UI. The "★ В избранное" button (left
+  pane, under the file list) / "В избранное" menu item / Ctrl+M → `_move_to_favorites`
+  **moves** (`shutil.move`, not copy) the `ExtendedSelection`-selected frames into
+  `project.favorites_dir` (`collections/Избранное`, `FAVORITES_DIR` in `core/project.py`,
+  created lazily on first move), removing them from list/`_files`/cache. `_unique_dest`
+  avoids clobbering (`foo.jpg` → `foo (1).jpg`). Use case: flag good frames into a
+  training/example set while inspecting detections.
 - **Restoration ("Расцензурить кадр").** Toolbar action runs `self._restorer` (built via
  `build_restorer`) on the current frame's detections (computing them first if needed),
  caches the result in `_restored[path]`, and shows it overlay-free. "Показать оригинал/
  результат" toggles (`_showing_restored`); "Сохранить результат" writes
-  `<stem>_restored.jpg` into the active collection (or beside the frame). The baseline
-  is cv2 inpaint; the real engine is **DeepMosaics** (`restore/deepmosaics.py`), which
-  shells out to a user-installed DeepMosaics repo (`deepmosaic.py --mode clean`, stdin
-  closed so its error `input()` can't hang; reads the newest image from a temp
-  result_dir). The engine + paths are set in `RestoreDialog` (Файл → Движок
-  восстановления…), persisted, and built lazily/cached in `_make_restorer` (like
-  `_make_detector`). NOTE: DeepMosaics locates mosaics itself (its `mosaic_position.pth`)
-  — our detections aren't passed to it. `_show` resets `_showing_restored` +
-  `_update_restore_actions`. To add another engine (e.g. LADA), implement
+  `<stem>_restored.jpg` beside the frame. The baseline
+  is cv2 inpaint; the real engine is **DeepMosaics** (`restore/deepmosaics.py`), run
+  **in-process** from the vendored `_deepmosaics/` code: it loads the BiSeNet locator +
+  clean generator **once** (lazy, cached on the instance) and per frame reproduces their
+  `cleanmosaic_img_server` (locate mosaic → run generator on the crop → feather back),
+  ~0.3 s/frame cached on CPU vs ~7 s when it spawned a subprocess. Use the **image**
+  model `clean_youknow_resnet_9blocks.pth` — the video model (BVDNet) is rejected per
+  frame (needs a neighbour). `should_cancel` is polled at entry (raises `Cancelled`).
+  The engine + weights are set in `RestoreDialog` (Файл → Движок восстановления…),
+  persisted, and built lazily/cached in `_make_restorer` (like `_make_detector`). NOTE:
+  DeepMosaics locates mosaics itself (its `mosaic_position.pth`, expected beside the
+  clean weights) — our detections aren't passed to it. `_show` resets `_showing_restored`
+  + `_update_restore_actions`. To add another engine (e.g. LADA), implement
  `core/restore/base.Restorer` and register it in `restore/factory.build_restorer`.
 - **Navigation bar** under the image (`_build_nav_bar`): prev/next frame (◀ ▶, keys
  `,`/`.`), a scrubber `frame_slider` across the whole sequence, a `pos_label`
  ("row / n"), and jump-to-detection (◀ детекция / детекция ▶, keys `[`/`]`,
  `_step_hit` scans `_results` for the next non-empty frame). The slider and file list
  are kept in sync via `_update_nav` guarded by `_nav_sync` (avoids signal loops); all
-  navigation ultimately drives `file_list.setCurrentRow`.
+  navigation ultimately drives `file_list.setCurrentRow`. The scrubber is a custom
+  `MarkerSlider` (`ui/marker_slider.py`) that paints cyan ticks at frames with
+  detections (`_refresh_marks` projects `_results` onto row indices; per-pixel deduped
+  so big folders stay cheap). File-list rows are tinted too (`_tag_file`): red =
+  censorship found, green = checked & clean. Both reset on `_invalidate_results`.
+- **Cancellation (cooperative, no threads).** A single "■ Стоп" toolbar action (Esc)
+  cancels the running long op. `_begin_busy(total)` / `_end_busy()` toggle `self._busy`
+  + the Stop button + the progress bar (`total=None` → indeterminate); `_request_cancel`
+  sets `self._cancel`; the loop-based ops (`_detect_all`, `_load_folder`) and video
+  extraction (its `progress` cb returns `not self._cancel`) check the flag between
+  `processEvents` ticks. Single-image restore passes `should_cancel=self._poll_cancel`
+  (which pumps `processEvents` then returns the flag) into `Restorer.restore`; only
+  DeepMosaics actually polls it (kills its subprocess + raises `Cancelled`) — cv2 ops are
+  instant. Entry points guard with `if self._busy: return` (notably `_move_to_collection`,
+  which mutates `_files` that `_detect_all` iterates). This keeps the synchronous,
+  single-threaded model — do NOT reintroduce worker threads for cancellation.
 - `image_view.ImageView` draws the image scaled-to-fit plus overlays. Overlay
  visibility/threshold are applied at paint time. Selecting a row in the detail table
  calls `set_highlight(i)` — that detection is drawn boldly (even below threshold) and
@@ -179,14 +240,15 @@ hvideotool/
 python -m venv .venv; .\.venv\Scripts\Activate.ps1
 pip install -e .                       # classic detector needs no torch/CUDA

-python -m hvideotool                   # open a folder in-app
-python -m hvideotool "C:\path\to\images" --detector yolo --model models\lada_mosaic_detection_model_v4_accurate.pt
+python -m hvideotool                   # reopen the last project (or create/open one in-app)
+python -m hvideotool "C:\path\to\MyProject" --detector yolo --model models\lada_mosaic_detection_model_v4_accurate.pt

 pip install -e ".[yolo]"               # + install torch separately, see README
 ```

 No formal test suite. Headless sanity check: set `QT_QPA_PLATFORM=offscreen`, build a
-`MainWindow`, `open_path(folder)`, drive `file_list.setCurrentRow(...)`, and read
+`MainWindow`, `Project.create(tmp)` + copy a few images into `frames/`,
+`_open_project(project)`, drive `file_list.setCurrentRow(...)`, and read
 `detail_table` / `detail_header`. Or run `build_detector(config).detect(...)` on a
 frame directly.

@@ -229,9 +291,10 @@ frame directly.
 - **Always use `core/imageio.py`** (`imread_unicode`/`imwrite_unicode`) for images —
  `cv2.imread`/`imwrite` silently fail on non-ASCII Windows paths.
 - Don't reintroduce any generative / ControlNet dependency, nor the removed video
-  *pipeline* (PyAV, project/cache, worker threads, playback). The one allowed video
-  touch is `core/video/extract.py` (one-shot decode → JPG folder, behind "Создать из
-  ролика…"): ffmpeg CLI — `_find_ffmpeg()` prefers PATH, else the binary bundled by
+  *playback pipeline* (PyAV, worker threads, player). (The new `core/project.py` is an
+  on-disk layout, not that thread-based "project model".) The one allowed video touch is
+  `core/video/extract.py` (one-shot decode → a new project's `frames/`, behind "Создать
+  из ролика…"): ffmpeg CLI — `_find_ffmpeg()` prefers PATH, else the binary bundled by
  the `imageio-ffmpeg` dep, else cv2 fallback. Keyframe-only `-skip_frame nokey` is
  ~10× faster than every-frame; `-hwaccel` does NOT help (GPU transfer overhead). Use
  ffmpeg/cv2, not PyAV, and keep it synchronous. Decoding every frame is the inherent