Implement multi-model detection in HVideoTool: updated the detection system to support multiple YOLO models simultaneously, enhancing detection capabilities. Reflected changes in the UI with a new model selection menu and updated documentation in README and CLAUDE.md to guide users on model management and configuration.
This commit is contained in:
@@ -51,7 +51,10 @@ Keep this scope sharp:
|
||||
batch into `restored/`), behind a `Restorer` interface. **Detection is YOLO-only and
|
||||
restoration is DeepMosaics-only** — the noisy classic-CV detector (+ the `combined`
|
||||
composite) and the cv2 `inpaint` baseline (filled but didn't reconstruct) were
|
||||
**removed** as "works poorly". DeepMosaics has two engines: **image** (per-frame) and
|
||||
**removed** as "works poorly". Detection is **multi-model** (ADetailer-style): drop YOLO
|
||||
weights under `models/yolo/<category>/`, tick which ones are active in the toolbar
|
||||
"Модели" menu, and a detect runs **all** ticked models and merges results (each tagged
|
||||
with its category → its own overlay colour). DeepMosaics has two engines: **image** (per-frame) and
|
||||
**video** (BVDNet, temporal — uses neighbour frames). Its GPL-3.0 network code is
|
||||
**vendored** under `core/restore/_deepmosaics/` and run in-process (user supplies only
|
||||
the weights). Because of that vendoring the **whole project is GPL-3.0**. LADA
|
||||
@@ -79,7 +82,7 @@ Keep this scope sharp:
|
||||
|----------------|---------------------------------------------|
|
||||
| GUI | PySide6 (Qt 6) — LGPL |
|
||||
| Image IO | OpenCV (`opencv-python`) + NumPy, unicode-safe via `core/imageio.py` |
|
||||
| Detector | Ultralytics YOLO (LADA weights) behind a pluggable interface (YOLO-only) |
|
||||
| Detector | Ultralytics YOLO, multi-model ensemble (models/yolo/<category>/*.pt) |
|
||||
|
||||
Torch/CUDA + Ultralytics enter with the YOLO detector. Keep that dependency optional
|
||||
(the `yolo` extra in `pyproject.toml` pulls only Ultralytics; torch is installed
|
||||
@@ -123,19 +126,31 @@ hvideotool/
|
||||
│ ├── factory.py # build_restorer(name, config) -> deepmosaics | deepmosaics_video (lada = TODO)
|
||||
│ ├── deepmosaics.py # DeepMosaicsRestorer (image, per-frame) + DeepMosaicsVideoRestorer (BVDNet, temporal); in-process, load once; uses _deepmosaics/
|
||||
│ └── _deepmosaics/ # VENDORED DeepMosaics models/+util/ (GPL-3.0) — added to sys.path at import
|
||||
└── detection/ # YOLO only
|
||||
└── detection/ # YOLO only, multi-model
|
||||
├── base.py # Detector ABC: detect(frame) -> list[Detection]
|
||||
├── factory.py # build_detector(config) -> yolo (the only kind)
|
||||
├── types.py # Detection (+ to_dict/from_dict), CensorType enum
|
||||
├── cache.py # save/load the project detection cache (detections.json): cache_file + base_dir args
|
||||
└── yolo.py # YoloDetector — Ultralytics YOLO-seg; lazy-imports torch/ultralytics
|
||||
├── factory.py # build_detector(config) -> MultiYoloDetector over config.detector_models
|
||||
├── registry.py # discover_models()/category_of() — scans models/yolo/<category>/*.pt
|
||||
├── multi.py # MultiYoloDetector — runs several YoloDetectors, concatenates results
|
||||
├── types.py # Detection (type + score + bbox + polygon + label/category; .display); CensorType enum
|
||||
├── cache.py # save/load the detection cache (detections.json); key = set of model basenames + conf/imgsz
|
||||
└── yolo.py # YoloDetector — Ultralytics YOLO-seg; lazy torch/ultralytics; tags dets with a category label
|
||||
```
|
||||
|
||||
### How it works
|
||||
|
||||
- `MainWindow` holds the config, builds the detector lazily via `build_detector`
|
||||
(cached by detector+model+conf in `_make_detector`), and keeps `_results: dict[path
|
||||
-> list[Detection]]` as the detection cache.
|
||||
(cached by the **selected-model set** + conf/imgsz in `_make_detector`), and keeps
|
||||
`_results: dict[path -> list[Detection]]` as the detection cache.
|
||||
- **Multi-model selection.** `config.detector_models` is the list of ticked YOLO weights
|
||||
(paths under `models/yolo/<category>/`). The toolbar "Модели" `QToolButton`/`QMenu`
|
||||
(`_rebuild_models_menu`, items grouped by category via `addSection`) toggles them
|
||||
(`_on_model_toggled` → persist + `_invalidate_results`); "Добавить модель…" copies a
|
||||
`.pt` into `models/yolo/<category>/`. On project open `_ensure_models` prunes vanished
|
||||
paths and, if nothing is selected, default-ticks every discovered model. `build_detector`
|
||||
builds one `YoloDetector` per selected model (tagged `label=category`) wrapped in a
|
||||
`MultiYoloDetector` that concatenates their detections (no cross-model dedup). Each
|
||||
`Detection` carries `label` (category); overlay colour + table group by `Detection.display`
|
||||
(label, else the CensorType) via `OverlayConfig.colors` + a stable `palette` fallback.
|
||||
- **Background jobs (`ui/workers.py`).** Detection and restoration are CPU-heavy and
|
||||
would freeze the GUI, so they run on a `QThreadPool` thread via `Job` (a `QRunnable`
|
||||
wrapping `fn(job)`); results return to the GUI through queued Qt signals
|
||||
@@ -276,11 +291,13 @@ hvideotool/
|
||||
- `ui/` must not import `torch` / `ultralytics` directly. It builds detectors only via
|
||||
`core/detection/factory.build_detector` and talks to `core/` through the `Detector`
|
||||
interface and the `Detection`/`CensorType` types.
|
||||
- Detection is YOLO-only and restoration is DeepMosaics-only. If you re-add an engine
|
||||
kind, implement `core/detection/base.Detector` / `core/restore/base.Restorer`, register
|
||||
the string in the respective `factory`, and add it to `DETECTORS`/`RESTORERS` in
|
||||
`config.py` (and `normalize_config`). There's no detector dropdown anymore — the toolbar
|
||||
just shows "Детектор: YOLO"; restoration engines are chosen in `RestoreDialog`.
|
||||
- Detection is YOLO-only (multi-model) and restoration is DeepMosaics-only. New detection
|
||||
kinds plug in by adding more `.pt` under `models/yolo/<category>/` — no code change. The
|
||||
toolbar shows a "Модели" menu of checkable models (no detector dropdown); restoration
|
||||
engines are chosen in `RestoreDialog`. If you re-add a different engine *kind*, implement
|
||||
`core/detection/base.Detector` / `core/restore/base.Restorer` and register it in the
|
||||
respective `factory`. (No legacy-settings migration is kept while in active development —
|
||||
old `settings.json`/`project.json` keys are simply ignored, not coerced.)
|
||||
|
||||
## Commands
|
||||
|
||||
@@ -308,20 +325,19 @@ frame directly.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Wrong YOLO model = "noise".** The YOLO detector needs a **censorship** model
|
||||
(LADA `models\lada_mosaic_detection_model_v4_accurate.pt`). If `model_path` points at
|
||||
a generic COCO model (e.g. the `yolo11n-seg.pt` in the repo root, which Ultralytics
|
||||
auto-downloads / is the training base), it detects people/objects and maps them to
|
||||
`CensorType.UNKNOWN` → purple boxes that look like noise. This was a real user trap.
|
||||
**A model is auto-picked on project open** via `MainWindow._ensure_model()` →
|
||||
`_auto_find_model()`: it scans `./models/**.pt` and matches only filenames containing
|
||||
`lada`/`mosaic` (so it skips the COCO `yolo11n-seg.pt` trap), no prompt; otherwise the
|
||||
user picks via "Модель…" (`_choose_model`).
|
||||
- **Models live under `models/yolo/<category>/`.** Discovery (`detection/registry.py`)
|
||||
scans that tree; the **category folder is the detection label + overlay colour** (e.g.
|
||||
`models/yolo/mosaic/lada.pt` → "mosaic", `models/yolo/face/…` → "face"). On open
|
||||
`_ensure_models` default-ticks all discovered models if the project has no selection.
|
||||
Put a **censorship** model in `mosaic/` (LADA `lada_mosaic_detection_model_v4_accurate.pt`);
|
||||
a generic COCO model (e.g. `yolo11n-seg.pt`) would detect people/objects → noise.
|
||||
Selecting many models multiplies per-frame time (each runs in turn).
|
||||
- **classic-CV / inpaint were removed (worked poorly).** The classic-CV detector was
|
||||
noisy/approximate on real footage (false positives on skin/hair/fabric/JPEG; missed
|
||||
real mosaic after downscale) and the `combined` mode + cv2 `inpaint` baseline went with
|
||||
it. Detection is YOLO-only, restoration is DeepMosaics-only. `normalize_config` coerces
|
||||
any leftover `classic`/`combined`/`inpaint` in old settings/projects to `yolo`/`deepmosaics`.
|
||||
it. Detection is YOLO-only, restoration is DeepMosaics-only. No backward-compat shims
|
||||
while in active development — stale keys in old `settings.json`/`project.json` are just
|
||||
ignored (a project with no valid model selection default-ticks all discovered models).
|
||||
- **Domain matters.** LADA is trained on REAL video (JAV). It detects some anime mosaic
|
||||
but not all. The real anime fix is *retraining* a YOLO11-seg (see `scripts/training/`).
|
||||
- **YOLO detector = LADA weights** ([HF `ladaapp/lada`](https://huggingface.co/ladaapp/lada)).
|
||||
|
||||
Reference in New Issue
Block a user