diff --git a/CLAUDE.md b/CLAUDE.md index 60b4907..b401d2a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,73 +1,56 @@ # CLAUDE.md -Конвенции этого репозитория и грабли, на которые здесь уже наступили. Читать до правок. +Конвенции репозитория. Читать до правок. + +**Перед правкой подсистемы прочитать её файл — там инварианты, которые ломаются молча:** + +| Правишь | Читай | +|---|---| +| `Core/Proxies/**`, `Infrastructure/Proxies/**` | [docs/proxies.md](docs/proxies.md) | +| `Collecting/**`, `Media/**`, `CollectViewModel`, `GalleryViewModel` | [docs/collecting.md](docs/collecting.md) | +| `.axaml`, стили, тестовые проекты, файлы сборки | [docs/avalonia.md](docs/avalonia.md) | ## Команды -Полный локальный гейт (формат → сборка → тесты) — то, что нужно прогнать перед коммитом: - -```bash -./build.ps1 -``` - -Отдельные шаги, если нужен только один из них: - -```bash -dotnet build AvParser.slnx -c Release -``` -```bash -dotnet test AvParser.slnx -c Release -``` -```bash -dotnet csharpier check . -``` -```bash -./run.ps1 -``` - -На Linux/macOS — `./build.sh` и `./run.sh` с теми же шагами. +`./build.ps1` — полный гейт (формат → сборка → тесты), прогнать перед коммитом. Отдельные шаги: +`dotnet build AvParser.slnx -c Release`, `dotnet test AvParser.slnx -c Release`, +`dotnet csharpier check .`, `./run.ps1`. На Linux/macOS — `./build.sh` и `./run.sh`. ## Слои `Core ← Infrastructure ← UI ← Desktop`, строго в одну сторону. -- **`AvParser.Core` не ссылается на Avalonia.** Это единственное ограничение, которое здесь - по-настоящему несущее: домен должен запускаться из CLI, worker-сервиса или бенчмарка. Как - только Avalonia станет доступна из домена, кто-нибудь потянется к `Dispatcher.UIThread` или - `IStorageProvider` внутри источника. -- **`AvParser.UI` — библиотека, а не exe.** Headless-тесты собирают настоящие View, не - подтягивая `Program.cs`, Serilog и контейнер. -- **`AvParser.Desktop` — тонкий composition root.** Логика туда не переезжает. +- **`Core` не ссылается на Avalonia** — домен должен запускаться из CLI, worker-а или бенчмарка. Дай + ему Avalonia, и кто-нибудь потянется к `Dispatcher.UIThread` внутри источника. +- **`UI` — библиотека, а не exe**: headless-тесты собирают настоящие View без `Program.cs`, Serilog и + контейнера. +- **`Desktop` — тонкий composition root.** Логика туда не переезжает. ## Добавить источник 1. Реализовать `IMediaSource` — в `Core/Collecting/Sources/`, если сети не нужно, иначе в `Infrastructure/Collecting/`. -2. Одна строка регистрации: `AddAvParserCore()` для доменного, `AddAvParserCollecting()` для - сетевого. -3. Два ключа в оба resx: `Source.{id}.Name` и `Source.{id}.Description` (без них источник - отрисуется английским текстом из самого класса, а не сломается). +2. Одна строка регистрации: `AddAvParserCore()` для доменного, `AddAvParserCollecting()` для сетевого. +3. Ключи `Source.{id}.Name` и `Source.{id}.Description` в оба resx (без них отрисуется английский + текст самого класса, а не сломается). -Всё. Каталог, страница «Сбор» и выпадающий список подхватят его сами. +Каталог, страница «Сбор» и список источников подхватят его сами. -**Источник ищет, а не качает.** Он отдаёт `MediaCandidate` — адрес плюс метаданные. Скачиванием, -редиректами, тайм-аутами, сниффингом и троттлингом занимается `MediaFetcher`, один на всех. Источник, -который сам лезет за байтами, дублирует всё это и почти наверняка неправильно. - -**Сетевой источник обязан переопределить `RequiresNetwork => true`** — иначе он не попадёт под гейт -и поедет напрямую в обход настройки. +- **Источник ищет, а не качает.** Он отдаёт `MediaCandidate`; скачивание, редиректы, тайм-ауты, + сниффинг и троттлинг — на `MediaFetcher`, одном на всех. +- **Сетевой обязан переопределить `RequiresNetwork => true`** — иначе поедет напрямую в обход гейта. ## Добавить страницу -1. Наследник `PageViewModel` в `UI/ViewModels/XxxViewModel.cs` (`Title`, `IconKey`). -2. `UI/Views/XxxView.axaml` — имя обязано соответствовать конвенции `ViewLocator`: - `...ViewModels.XxxViewModel` → `...Views.XxxView`. -3. Регистрация в `AddAvParserUI()`: конкретным типом **и** как `PageViewModel` — порядок этих - регистраций и есть порядок пунктов в рельсе навигации. +1. Наследник `PageViewModel` в `UI/ViewModels/XxxViewModel.cs` (`TitleKey`, `IconKey`). +2. `UI/Views/XxxView.axaml` — имя по конвенции `ViewLocator`: `...ViewModels.XxxViewModel` → + `...Views.XxxView`. +3. Регистрация в `AddAvParserUI()` конкретным типом **и** как `PageViewModel`; порядок этих + регистраций = порядок пунктов в рельсе навигации. ## ReactiveUI 24 (дистрибутив Primitives) -Это не классический ReactiveUI. `System.Reactive` не используется: +`System.Reactive` не используется: | Классика | Здесь | |---|---| @@ -77,207 +60,49 @@ dotnet csharpier check . | `RxApp.MainThreadScheduler` | `RxSchedulers.MainThreadScheduler` | | `TestScheduler` | `VirtualClock`, `ImmediateSequencer.Instance` | -Привычные имена операторов (`Select`, `Where`, `Throttle`, `DistinctUntilChanged`, -`CombineLatest`) **работают** — Primitives отдаёт оба набора. `using ReactiveUI.Primitives;` -нужен ради `Subscribe(Action)`. +Привычные операторы (`Select`, `Where`, `Throttle`, `DistinctUntilChanged`, `CombineLatest`) +работают; `using ReactiveUI.Primitives;` нужен ради `Subscribe(Action)`. -**ReactiveUI 24 не инициализируется сама.** Первый `WhenAnyValue` бросит -`InvalidOperationException`, пока не отработал builder. В приложении это делает -`AppBuilder.UseReactiveUI(...)`; в проекте VM-тестов — module initializer -`ReactiveUiBootstrap`. Новый тестовый проект без Avalonia обязан сделать то же самое. +**ReactiveUI 24 не инициализируется сама** — первый `WhenAnyValue` бросит +`InvalidOperationException`, пока не отработал builder: в приложении `AppBuilder.UseReactiveUI(...)`, +в VM-тестах module initializer `ReactiveUiBootstrap`. Новый тестовый проект без Avalonia обязан +сделать то же самое. ## Конвенции ViewModel -- **Каждая VM принимает `ISequencer? mainThread = null`** и использует его в `outputScheduler:` - и `ToProperty(..., scheduler)`. Именно это делает тесты синхронными: они передают - `ImmediateSequencer.Instance`. Без этого пришлось бы гонять диспетчер. -- У VM с необязательным `ISequencer` регистрация в DI — явная фабрика, а не по типу: иначе - выбор конструктора контейнером зависит от порядка регистраций. +- **Каждая VM принимает `ISequencer? mainThread = null`** и использует его в `outputScheduler:` и + `ToProperty(..., scheduler)`. Это и делает тесты синхронными — они передают `ImmediateSequencer`. +- **Регистрация такой VM в DI — явная фабрика**, не по типу: иначе выбор конструктора зависит от + порядка регистраций. - `[Reactive]` из `ReactiveUI.SourceGenerators` на partial-свойствах; класс — `partial`. +- **VM с производным от языка текстом переопределяет `OnLanguageChanged`** и зовёт `base`, иначе + заголовок страницы застрянет на прежнем языке. +- **VM, подписанная на синглтон** (пул, каталог, настройки), — `IDisposable` и отписывается. -## Добавить источник прокси +## Строки UI -1. Реализовать `IProxySource` (или `IMutableProxySource`, если список редактируемый). -2. Зарегистрировать как `IProxySource` в `AddAvParserProxies()`. Порядок регистрации = порядок - слияния; свой список идёт последним, чтобы пользовательский адрес перебивал фидовый. - -`ProxyPool` при обновлении **переиспользует существующие `ProxyEntry`** по `Endpoint.Key` — иначе -перезагрузка списка стирала бы всю накопленную статистику, а публичные фиды переиздаются каждые -несколько минут. - -## Инварианты прокси-пула - -- **Доступность определяется карантином, а не `Health`.** `Health` — это «что видели в последний - раз». Если исключать всё, что когда-либо падало, окно карантина становится бессмысленным, а - прокси теряется навсегда после первой же осечки. Это уже был баг, его ловит - `A_failing_proxy_is_quarantined_and_comes_back_later`. -- **Проба не трогает `SuccessCount`/`FailureCount`.** Эти счётчики про реальные запросы; свип по - паре тысяч прокси перезаписал бы всё, на чём держится взвешенный выбор. Провалившаяся проба - выставляет карантин через `RecordProbe(..., quarantineOnFailure:)`. -- **Лиза без вердикта нейтральна.** Отменённая операция — не вина прокси; считать это отказом - значит карантинить здоровые прокси на каждый Cancel. -- **`Select` и `Next` — зарезервированные слова для CA1716.** Метод стратегии называется `Pick`. -- **`LiveCount` считает только `Alive` и не в карантине.** На нём висит гейт сбора, поэтому - «доступна» (карантин истёк) и «живая» здесь намеренно расходятся: гейт не должен открываться - от одного лишь истечения окна. -- **Прогрев обрывается по достижении цели, а не проходит список до конца.** `WarmUpAsync` линкует - CTS и гасит остаток, как только набралось `MinimumLiveProxies`. Порядок кандидатов — - `WarmUpOrder()`, он публичный ровно затем, чтобы порядок проверялся без прогона проб. -- **`RestoreState` не выставляет `Health = Alive`.** «Работала вчера» живёт в отдельном - `WasAliveOnLastRun` и влияет только на порядок прогрева. Если восстанавливать как `Alive`, пул - отрапортует живыми тех, с кем не разговаривал: прогрев сочтёт цель достигнутой и не проверит - никого, а гейт сбора откроется по данным недельной давности. Ловит - `A_remembered_proxy_is_not_reported_live_until_it_answers_again`. -- **`RestoreState` не восстанавливает карантин.** Окно — стенные часы, между запусками могли пройти - сутки; перенос окна сажал бы прокси за то, что давно истекло. -- **Сохраняются только `HasEverAnswered`** = `SuccessCount > 0 || IsBelievedAlive`, где - «believed» = вердикт этой сессии, а без него — прошлой. Мёртвые в фиде исчисляются тысячами и - переиздаются каждые пять минут. Важен именно перенос: прогрев обрывается рано, поэтому - большинство запомненных заканчивают сессию непроверенными — строгое `Health == Alive` стирало бы - накопленный список за пару запусков. - -## Гейт сбора - -`IParser.RequiresNetwork` — дефолтная реализация возвращает `false`, поэтому добавление источника -остаётся однострочным. - -Гейт живёт в `CollectViewModel.RefreshProxyGate()` и складывается из трёх условий: источник сетевой, -`AllowDirectConnection` выключен, `LiveCount == 0`. Он пересчитывается по событию пула (с -throttle 250 мс — пул дёргается на каждый исход лизы), при смене источника и при смене настроек. - -Второй экземпляр того же правила — в `FetchOptions.RequireProxy`: гейт закрывает кнопку, а фетчер -бросает `ProxyUnavailableException` вместо тихого прямого запроса. Одного UI мало — запрос ушёл бы -с адреса пользователя ровно тогда, когда он просил этого не делать. - -`CollectView.axaml` держит баннер под `x:Name="ProxyGateBanner"`; `CollectViewTests` рендерит его -по-настоящему, потому что мёртвый биндинг `IsVisible` не ломает ни одного VM-теста. - -## Инварианты хранилища медиа - -- **В `blobs/` попадает только дочитанное.** Загрузка идёт во временный файл в соседнем каталоге на - том же томе и продвигается переименованием. Обрыв оставляет `.part`, который подметает следующий - старт, а не обрезанную картинку, неотличимую от настоящей навсегда. -- **Тип — по сигнатуре, никогда по URL, расширению или `Content-Type`.** Два из трёх выбирает тот, - кто отдаёт файл, и расширение на диске у пользователя не должно зависеть от чужого сервера. -- **`GIF89a` не доказывает анимацию**, и APNG не определяется по фиксированному префиксу: нужен обход - блоков (второй Image Descriptor) и чанков (`acTL` раньше первого `IDAT`). Тихая ошибка, поэтому - обходчики изолированы за `internal static` швами и проверяются на массивах байтов. -- **`ref_count` денормализован и пересчитывается, а не инкрементится.** Апсерт `item` может заменить - строку, указывавшую на другой blob, и слепой `+1` уехал бы навсегда. Есть - `VerifyReferenceCountsAsync`, и он часть замысла, а не отладка. -- **Журнал `seen_url` переживает чистку.** Иначе следующий прогон скачает заново ровно то, что - пользователь только что удалил. Терминальные исходы отделены от повторяемых: отказ описывает - момент, а не ресурс, и считать его окончательным значит терять контент на каждой сетевой икоте. -- **Вердикт лизы — про транспорт, а не про ресурс.** 404 и 429 — это успех прокси. Иначе пул - карантинил бы рабочие адреса ровно с той частотой, с какой встречаются мёртвые ссылки, а на - лимит отвечал бы сменой прокси, то есть обходом лимита. -- **Троттл поднимается только сигналами хоста** (429/503 с `Retry-After`) и никогда не приводит к - ротации прокси. Флажка «повторить через другую прокси при 429» в настройках быть не должно. -- **Жёсткая ссылка — привилегия ФС, а не гарантия.** Откат на копию удваивает расход диска, поэтому - достигнутый режим пишется в `showcase_mode` и виден в UI. -- **Имя из `SuggestedName` враждебно.** Остаётся только последний сегмент, разделители не - переживают, устройства Windows отодвигаются, расширение берётся из типа. - -## Добавить строку в UI - -1. Ключ и оба перевода — в `Strings.resx` и `Strings.ru.resx` (**оба**, иначе упадёт - `LocalizationTests.Russian_translates_every_english_key`). -2. В XAML — `{l:Loc Ключ}`, во ViewModel — `Localizer.Instance[...]` / `.Format(...)`. +1. Ключ и **оба** перевода — в `Strings.resx` и `Strings.ru.resx`, иначе упадёт + `LocalizationTests.Russian_translates_every_english_key`. +2. В XAML `{l:Loc Ключ}`, во ViewModel `Localizer.Instance[...]` / `.Format(...)`. Никакого хардкода в `Views/` кроме имени продукта и примеров адресов. -- **Счётчики — только через `Localizer.Plural`** с ключами `.One` / `.Few` / `.Many`. У русского - три формы; «{0} records» с приклеенным окончанием непереводимо. -- **Перечисления в списках — `LocalizedOption`**, не сырые значения. Конвертер разрешил бы - подпись один раз и не заметил смены языка. Идентичность обёртки — значение перечисления, чтобы - выбор не слетал. -- **Текст из домена переводится по коду.** `Core` о языках не знает: `ParseError` несёт `Code` и - `Arguments`, UI ищет `Collect.Error.{Code}` с откатом на `Message`. Имена источников — так же: - `Source.{id}.Name` с откатом на `DisplayName`, поэтому обещание «добавить источник = одна строка» - остаётся в силе. -- **VM, у которой есть производный от языка текст, переопределяет `OnLanguageChanged`** и зовёт - `base`. Без этого заголовок страницы останется на прежнем языке. - -## Грабли, уже оплаченные - -- **Селектор типа в Avalonia матчит точный тип.** `UserControl.shell` не матчит `ShellView` - (наследник `ReactiveUserControl`) и молча не делает ничего. Использовать - `:is(UserControl).shell`. Голый `.shell` тоже матчит, но тогда XAML-компилятор не может - вывести тип для `Setter` и падает с AVLN2200. -- **Style-сеттер навсегда проигрывает локальному значению.** Не стилизовать `IsPaneOpen` и - `DisplayMode` — они биндятся во ViewModel. -- **`IPseudoClasses.Set` требует ведущего `:`**. -- **`InvariantGlobalization` обязан быть `false`**: Semi.Avalonia строит `CultureInfo` в - статическом конструкторе и падает целиком. -- **Превьюер рефлексирует безпараметровый статический `BuildAvaloniaApp()`.** Необязательный - параметр ломает его вызов, вторая перегрузка — `AmbiguousMatchException`. -- **Compiled bindings включены по умолчанию** (Avalonia 12): `x:DataType` нужен на каждом - `UserControl` и каждом `DataTemplate`. -- **`Avalonia.Headless.XUnit` 12.x — это xUnit v3**, а не v2. Тестовые проекты — `Exe`. -- **Headless: ручной `Measure`/`Arrange` внутри окна бесполезен** — следующий проход layout - окна вернёт свой размер. Задавать ширину самому `Window`. Но и без окна нельзя: у - открепленного контрола не строится визуальное дерево. -- **csharpier — единственный владелец форматирования** (включая `.axaml` и `.csproj`). - `IDE0055` понижен до suggestion: два форматтера с `TreatWarningsAsErrors` дерутся насмерть. -- **`Condition="'$(Configuration)' == 'Debug'"` на `PackageReference` — ловушка.** - `dotnet restore` вычисляется с конфигурацией по умолчанию, поэтому пакет обязан - резолвиться, даже если эту конфигурацию никто не собирает. Так тут проехал мёртвый - `Avalonia.Diagnostics` (его нет под Avalonia 12): `dotnet build -c Release` работал, - а голый `dotnet restore` падал. -- **`Execute()` завершился ≠ `IsExecuting` уже false.** Второе публикуется на выходном - планировщике. Тест, который сразу после `await` дёргает команду, закрытую по чужому - `IsExecuting`, будет мигать под нагрузкой — ждите `CanExecute`, а не предполагайте. -- **Проект VM-тестов не параллелится.** `ReactiveUiBootstrap` ставит глобальные планировщики - ReactiveUI, то есть тесты делят изменяемое состояние независимо от их желания. -- **Инспектора в Avalonia 12 нет из коробки.** `Avalonia.Diagnostics` закончился на 11.3.x; - DevTools живут отдельно (`AvaloniaUI.DiagnosticsSupport` + `.WithDeveloperTools()`), со своей - установкой. Зависимость намеренно не добавлена. - -## Качество - -- Центральные версии пакетов — `Directory.Packages.props`. **Никаких `Version=` в csproj** - (иначе NU1008 на restore). -- `TreatWarningsAsErrors` включён; NuGet-advisory (`NU19xx`) выведены из ошибок, чтобы - свежая CVE не роняла сборку кода, который никто не трогал. -- Тестовые послабления анализаторов — в `tests/Directory.Build.props`, не в самих тестах. - -## Пути - -- **Медиа лежит рядом с exe, настройки — в профиле.** Это осознанное расхождение: конфигурация - пользовательская, а коллекция принадлежит установке и переезжает вместе с папкой. -- **`DefaultMediaDirectory()` проверяет запись пробным файлом**, а не только созданием каталога: - каталог создаётся и там, куда потом нельзя записать. При отказе — откат в профиль. -- **Смена каталога требует перезапуска.** `AppPaths` строится до контейнера (медиа-корень читается - из settings.json напрямую), так что применить его на лету нельзя без переподключения индекса, - blob-хранилища и кэша миниатюр. Настройки честно это говорят и показывают действующий путь. -- **Смена каталога не переносит файлы.** Перенос архива — отдельная операция с другой ценой ошибки. - -## Отображение медиа - -- **Миниатюры декодируются в нужную ширину**, а не декодируются целиком и потом масштабируются. - На архиве это разница между «работает» и «кончилась память». -- **Кэш владеет своими `Bitmap` и удаляет их при вытеснении**, поэтому вызывающий не должен их - освобождать — и поэтому ёмкость кэша обязана заметно превышать страницу галереи: вытесненная - картинка, которая ещё на экране, освободилась бы под рендерером. -- **Видео не декодируется**, и попытка была бы исключением на каждой плитке. `MediaKinds.IsImage` - отсекает это до всякого обращения к диску. -- **`{DynamicResource}` с несуществующим ключом молча не срабатывает** — свойство остаётся со - значением по умолчанию, фон не красится, кисть прозрачная. Это уже дважды доезжало до - скриншота; ловит `ResourceKeyTests` в обеих темах. -- **`{l:Loc}` с ключом, которого нет в обоих resx, тест паритета не поймает** — файлы согласованы - между собой. Ловит `LocalizationCoverageTests`. +- **Счётчики — только через `Localizer.Plural`** с ключами `.One` / `.Few` / `.Many`: у русского три + формы, «{0} records» с приклеенным окончанием непереводимо. +- **Перечисления в списках — `LocalizedOption`**, не сырые значения: конвертер разрешил бы подпись + один раз и не заметил смены языка. Идентичность обёртки — значение перечисления, чтобы выбор не + слетал. +- **Текст из домена переводится по коду.** `ParseError` несёт `Code` и `Arguments`, UI ищет + `Collect.Error.{Code}` с откатом на `Message`; имена источников — `Source.{id}.Name` с откатом на + `DisplayName`, поэтому «добавить источник = одна строка» остаётся в силе. ## Границы, выбранные намеренно -- **robots.txt не читается.** Оба источника v1 либо принадлежат пользователю, либо введены им - вручную, поэтому спрашивать разрешения не у кого. Это решение, а не забывчивость: **с появлением - третьего источника, который ходит по чужому сайту, robots.txt становится обязательным.** -- **Перебора идентификаторов нет и не планируется.** Источник перечисляет то, что сайт сам - публикует; подбор адресов — это не «сбор опубликованного», и прокси-пул существует ради лимитов - и доступности, а не ради их обхода. -- **Возобновления по `Range` нет.** Оборванная загрузка выбрасывается целиком; частичный файл в - хранилище дороже, чем повторное скачивание. +- **robots.txt не читается.** Источники либо принадлежат пользователю, либо введены им вручную. Это + решение, а не забывчивость: **с появлением источника, который ходит по чужому сайту, robots.txt + становится обязательным.** +- **Возобновления по `Range` нет**: частичный файл в хранилище дороже повторного скачивания. - **SVG не поддерживается сознательно** — это текст, он умеет исполнять скрипты и несёт XXE. BMP/ICO/HEIC/JPEG-XL просто отложены. -- **Перцептивных хешей нет.** Дедуп точный, по SHA-256; «похожие» картинки — отдельная задача. -- **Кадры из видео не извлекаются.** Это FFmpeg целиком ради одной картинки в плитке. +- **Перцептивных хешей нет**: дедуп точный, по SHA-256. +- **Кадры из видео не извлекаются** — это FFmpeg целиком ради одной картинки в плитке. diff --git a/docs/avalonia.md b/docs/avalonia.md new file mode 100644 index 0000000..8a5d835 --- /dev/null +++ b/docs/avalonia.md @@ -0,0 +1,63 @@ +# Avalonia, тесты и сборка: уже оплаченные грабли + +Читать перед правкой `.axaml`, стилей, тестовых проектов и файлов сборки. + +## XAML и стили + +- **Селектор типа матчит точный тип.** `UserControl.shell` не матчит `ShellView` (наследник + `ReactiveUserControl`) и молча не делает ничего — нужен `:is(UserControl).shell`. Голый `.shell` + тоже матчит, но тогда XAML-компилятор не выводит тип для `Setter` и падает с AVLN2200. +- **Style-сеттер навсегда проигрывает локальному значению.** Не стилизовать `IsPaneOpen` и + `DisplayMode` — они биндятся во ViewModel. +- **`IPseudoClasses.Set` требует ведущего `:`.** +- **Compiled bindings включены по умолчанию** (Avalonia 12): `x:DataType` нужен на каждом + `UserControl` и каждом `DataTemplate`. +- **`{DynamicResource}` с несуществующим ключом молча не срабатывает** — свойство остаётся со + значением по умолчанию, фон не красится, кисть прозрачная. Дважды доезжало до скриншота; ловит + `ResourceKeyTests` в обеих темах. +- **`{l:Loc}` с ключом, которого нет в обоих resx, тест паритета не поймает** — файлы согласованы + между собой. Ловит `LocalizationCoverageTests`. + +## Приложение + +- **`InvariantGlobalization` обязан быть `false`**: Semi.Avalonia строит `CultureInfo` в статическом + конструкторе и падает целиком. +- **Превьюер рефлексирует безпараметровый статический `BuildAvaloniaApp()`**: необязательный параметр + ломает его вызов, вторая перегрузка — `AmbiguousMatchException`. +- **Инспектора в Avalonia 12 нет из коробки.** `Avalonia.Diagnostics` закончился на 11.3.x; DevTools + живут отдельно (`AvaloniaUI.DiagnosticsSupport` + `.WithDeveloperTools()`), со своей установкой. + Зависимость намеренно не добавлена. + +## Тесты + +- **`Avalonia.Headless.XUnit` 12.x — это xUnit v3**, а не v2. Тестовые проекты — `Exe`. +- **Headless: ручной `Measure`/`Arrange` внутри окна бесполезен** — следующий проход layout окна + вернёт свой размер. Задавать ширину самому `Window`. Но и без окна нельзя: у открепленного контрола + не строится визуальное дерево. +- **`Execute()` завершился ≠ `IsExecuting` уже false.** Второе публикуется на выходном планировщике. + Тест, который сразу после `await` дёргает команду, закрытую по чужому `IsExecuting`, будет мигать + под нагрузкой — ждите `CanExecute`, а не предполагайте. +- **Проект VM-тестов не параллелится**: `ReactiveUiBootstrap` ставит глобальные планировщики + ReactiveUI, то есть тесты делят изменяемое состояние независимо от их желания. + +## Композиция + +- **`Dispose()` страницы обязан переживать второй вызов.** Каждая страница зарегистрирована дважды — + своим типом и как `PageViewModel`, — и контейнер учитывает её как disposable по обеим регистрациям, + то есть зовёт `Dispose()` два раза. `GalleryViewModel` отменял на втором вызове уже освобождённый + `CancellationTokenSource`, и приложение падало с `ObjectDisposedException` при каждом выходе. Ловят + тесты `Disposing_twice_is_safe`. + +## Сборка + +- **csharpier — единственный владелец форматирования** (включая `.axaml` и `.csproj`). `IDE0055` + понижен до suggestion: два форматтера с `TreatWarningsAsErrors` дерутся насмерть. +- **`Condition="'$(Configuration)' == 'Debug'"` на `PackageReference` — ловушка.** `dotnet restore` + вычисляется с конфигурацией по умолчанию, поэтому пакет обязан резолвиться, даже если эту + конфигурацию никто не собирает. Так тут проехал мёртвый `Avalonia.Diagnostics`: + `dotnet build -c Release` работал, а голый `dotnet restore` падал. +- Версии пакетов централизованы в `Directory.Packages.props` — **никаких `Version=` в csproj** + (иначе NU1008 на restore). +- `TreatWarningsAsErrors` включён; NuGet-advisory (`NU19xx`) выведены из ошибок, чтобы свежая CVE не + роняла сборку кода, который никто не трогал. +- Тестовые послабления анализаторов — в `tests/Directory.Build.props`, не в самих тестах. diff --git a/docs/collecting.md b/docs/collecting.md new file mode 100644 index 0000000..573dc8e --- /dev/null +++ b/docs/collecting.md @@ -0,0 +1,111 @@ +# Сбор и хранилище + +Читать перед правкой `Collecting/**`, `Media/**`, `CollectViewModel` и `GalleryViewModel`. + +## Гейт сбора + +`CollectViewModel.RefreshProxyGate()`: **хотя бы один отмеченный** источник требует прокси +(`MediaSourceViewModel.NeedsProxy` = сетевой и без своего разрешения на прямое подключение) и +`LiveCount == 0`. Пересчитывается по событию пула (throttle 250 мс — пул дёргается на каждый исход +лизы) и при смене отметок. Подсветка в списке для этого не годится: она про то, что редактируют, а не +про то, что запускают. + +- **Разрешение работать без прокси — настройка источника, а не приложения** + (`PatternSourceConfig.AllowDirectConnection`). Один хост может быть своим, где прокси бессмысленна, + а соседний — чужим, где прямое подключение это ровно то, чего пользователь избегал; общий тумблер + навязывал обоим разрешающий ответ. По умолчанию `false` — туда же приезжает старый + `sources.user.json`. +- **Один закрытый источник в прогоне закрывает весь прогон.** Разрешающий не может поручиться за + строгого: запрос, которого пользователь не хотел, всё равно ушёл бы с его адреса. +- **Второй экземпляр того же правила — `FetchOptions.RequireProxy`**, и он строится на источник в + `ProduceAsync`. Гейт гасит кнопку, а фетчер бросает `ProxyUnavailableException`; одного UI мало. + `CollectOptions.RequireProxy` по умолчанию `true`: молчание вызывающего — не разрешение. +- Баннер живёт под `x:Name="ProxyGateBanner"` и рендерится по-настоящему в `CollectViewTests`, потому + что мёртвый биндинг `IsVisible` не ломает ни одного VM-теста. + +## Прогон по нескольким источникам + +- **Отметка ≠ подсветка.** `MediaSourceViewModel.IsSelected` решает, что войдёт в прогон; выделение в + списке — что правят, удаляют и чистят. Отмеченный набор живёт в `AppSettings.CollectSourceIds` + строкой, а не списком: список сломал бы сравнение записи, и каждое сохранение выглядело бы + изменением. +- **Источники идут одновременно, а не по очереди.** Это не про пропускную способность: при нулевом + бюджете первый источник не заканчивается никогда, так что последовательный прогон был бы прогоном + по одному источнику со списком в руках. Слияние — `CollectViewModel.MergeAsync`; канал ограничен, а + `finally` обязан погасить продюсеров — оставить их писать в канал, который никто не читает, значит + подвесить их посреди записи в хранилище. +- **Глобальный лимит закачек делится на число источников, а не умножается**: каждый `CollectRunner` + поднимает своих воркеров, и пять источников по четыре — это двадцать соединений вместо четырёх. +- **`Limit = 0` — «пока не остановят».** В безлимитном режиме `PatternMediaSource` сбрасывает + множество виденных id по `SeenCapacity`: у прогона нет конца, значит и у множества не должно быть + роста. Настоящий дедуп держит журнал `seen_url`, а не оно. + +## Источники + +- **Пустое расширение — это `.jpg`, а не «без суффикса».** В редакторе это плейсхолдер, и + `PatternSourceConfig.NormaliseExtension` подставляет его в домене, потому что через `TryCreate` + проходят и форма, и загрузка `sources.user.json`. Голый `/{id}` почти всегда опечатка, которая + стоит целого прогона из 404. + +## Живой журнал + +- **Одна хронологическая лента вместо «результаты + ошибки»**: при переборе id почти всё промахи, и + смотрят на порядок, а не на две таблицы. +- **Пакетов мало, нужен ещё и тик.** `FlushInterval` (200 мс) существует потому, что на медленном + источнике буфер не добирает до `BatchSize` и страница выглядит зависшей. Буферы — + `ConcurrentQueue`, слив под `_flushGate`, иначе две гонки-выгрузки перемешают строки местами. +- **Строка хранит ключ и аргументы, а не готовое предложение** — смена языка посреди прогона иначе + оставит половину журнала по-английски. Литеральная половина (адрес, размер) не переводится никогда. +- **Адрес неудачи живёт в `ParseError.Subject`.** В `Message` его нет и быть не может: у шаблона + перевода фиксированные подстановки. Без него журнал говорит «сайт ответил 404» и не говорит, на + каком из десяти тысяч id. +- **Журнал ограничен `MaxLogEntries`** — у безлимитного прогона нет конца, а несрезанный список это + утечка памяти со скроллбаром. +- **Строки копируются**: множественное выделение, Ctrl+C и контекстное меню; текст строит + `CollectLogEntryViewModel.ToString()`, чтобы в буфер попало ровно то, что на экране, а не + повторный рендер, который тихо разъедется с шаблоном. Ctrl+C без выделения копирует весь журнал. + В Avalonia 12 `SetTextAsync` — расширение из `Avalonia.Input.Platform`, а не член `IClipboard`. + +## Хранилище медиа + +- **В `blobs/` попадает только дочитанное.** Загрузка идёт во временный файл в соседнем каталоге на + том же томе и продвигается переименованием. Обрыв оставляет `.part`, который подметает следующий + старт, а не обрезанную картинку, навсегда неотличимую от настоящей. +- **Тип — по сигнатуре, никогда по URL, расширению или `Content-Type`**: два из трёх выбирает тот, + кто отдаёт файл, и расширение на диске у пользователя не должно зависеть от чужого сервера. +- **`GIF89a` не доказывает анимацию**, и APNG не определяется по фиксированному префиксу: нужен обход + блоков (второй Image Descriptor) и чанков (`acTL` раньше первого `IDAT`). Ошибка тихая, поэтому + обходчики изолированы за `internal static` швами и проверяются на массивах байтов. +- **`ref_count` денормализован и пересчитывается, а не инкрементится**: апсерт `item` может заменить + строку, указывавшую на другой blob, и слепой `+1` уехал бы навсегда. `VerifyReferenceCountsAsync` — + часть замысла, а не отладка. +- **Журнал `seen_url` переживает чистку**, иначе следующий прогон скачает заново ровно то, что + пользователь только что удалил. Терминальные исходы отделены от повторяемых: отказ описывает + момент, а не ресурс, и считать его окончательным значит терять контент на каждой сетевой икоте. +- **Троттл поднимается только сигналами хоста** (429/503 с `Retry-After`) и никогда не приводит к + ротации прокси. Флажка «повторить через другую прокси при 429» в настройках быть не должно. +- **Жёсткая ссылка — привилегия ФС, а не гарантия.** Откат на копию удваивает расход диска, поэтому + достигнутый режим пишется в `showcase_mode` и виден в UI. +- **Имя из `SuggestedName` враждебно**: остаётся только последний сегмент, разделители не переживают, + устройства Windows отодвигаются, расширение берётся из типа. + +## Пути + +- **Медиа лежит рядом с exe, настройки — в профиле.** Расхождение осознанное: конфигурация + пользовательская, а коллекция принадлежит установке и переезжает вместе с папкой. +- **`DefaultMediaDirectory()` проверяет запись пробным файлом**, а не только созданием каталога: + каталог создаётся и там, куда потом нельзя писать. При отказе — откат в профиль. +- **Смена каталога требует перезапуска и не переносит файлы.** `AppPaths` строится до контейнера + (медиа-корень читается из settings.json напрямую), так что на лету это не применить без + переподключения индекса, blob-хранилища и кэша миниатюр. Настройки честно это говорят и показывают + действующий путь. + +## Отображение медиа + +- **Миниатюры декодируются сразу в нужную ширину**, а не декодируются целиком и потом масштабируются. + На архиве это разница между «работает» и «кончилась память». +- **Кэш владеет своими `Bitmap` и удаляет их при вытеснении**, поэтому вызывающий не должен их + освобождать — и поэтому ёмкость кэша обязана заметно превышать страницу галереи: вытесненная + картинка, которая ещё на экране, освободилась бы под рендерером. +- **Видео не декодируется**, попытка была бы исключением на каждой плитке: `MediaKinds.IsImage` + отсекает это до всякого обращения к диску. diff --git a/docs/proxies.md b/docs/proxies.md new file mode 100644 index 0000000..4001832 --- /dev/null +++ b/docs/proxies.md @@ -0,0 +1,63 @@ +# Прокси-пул + +Читать перед правкой `Core/Proxies/**` и `Infrastructure/Proxies/**`. + +## Добавить источник + +1. Реализовать `IProxySource` (или `IMutableProxySource`, если список редактируемый). +2. Зарегистрировать в `AddAvParserProxies()`. Порядок регистрации = порядок слияния; свой список идёт + последним, чтобы пользовательский адрес перебивал фидовый. + +## Инварианты + +- **`ProxyPool` переиспользует существующие `ProxyEntry` по `Endpoint.Key`** — иначе перезагрузка + списка стирала бы статистику, а публичные фиды переиздаются каждые несколько минут. +- **Доступность определяется карантином, а не `Health`.** `Health` — «что видели в последний раз»; + исключать всё когда-либо упавшее значит потерять прокси после первой осечки. Уже было багом, ловит + `A_failing_proxy_is_quarantined_and_comes_back_later`. +- **`LiveCount` считает только `Alive` и не в карантине.** На нём висит гейт сбора, поэтому + «доступна» (окно истекло) и «живая» намеренно расходятся: гейт не должен открываться от одного лишь + истечения окна. +- **Проба не трогает `SuccessCount`/`FailureCount`** — они про реальные запросы, а свип по паре тысяч + адресов перезаписал бы всё, на чём держится взвешенный выбор. Провал пробы даёт карантин через + `RecordProbe(..., quarantineOnFailure:)`. +- **Лиза без вердикта нейтральна**: отменённая операция — не вина прокси, иначе каждый Cancel + карантинил бы здоровые адреса. +- **Вердикт лизы — про транспорт, а не про ресурс.** 404 и 429 — успех прокси; иначе пул карантинил + бы рабочие адреса с частотой мёртвых ссылок, а на лимит отвечал бы сменой прокси, то есть обходом + лимита. +- **Прогрев обрывается по достижении цели**, а не проходит список до конца: `WarmUpAsync` линкует CTS + и гасит остаток на `MinimumLiveProxies`. Порядок кандидатов — `WarmUpOrder()`, публичный ровно ради + проверки без прогона проб. +- **Остаток дочищает `TopUpAsync` в фоне.** Прогрев по построению оставляет тысячи адресов в + `Unknown` — это читается как «проверка не запускалась», и, что хуже, сбор получает пул из десяти + прокси вместо сотен рабочих. Фоновый проход запускает `ProxyPoolLoader` после прогрева, с + четвертью `ProbeConcurrency`: пользовательский свип — это всплеск, за которым смотрят, а этот идёт + минутами и, возможно, параллельно с закачкой. +- **`TopUpAsync` трогает только `Unknown`.** Перепроверять мёртвых — это и есть свип; перепроверять + живых значит посадить в карантин прокси, которой прямо сейчас пользуется прогон. +- **`TopUpAsync` не делает `_strategy.Reset()`**, в отличие от свипа: сброс уронил бы липкий выбор + посреди сбора, причём ради записей, которых в этом выборе и нет. `Changed` он поднимает раз в + `ChangeInterval` проб — иначе страница молчит несколько минут. +- **Фоновый проход не идёт при `ProxyHealthCheck.Lazy`**: «проверять в момент выдачи» — это ровно + просьба не свипать. +- **`SaveStateAsync` сериализован семафором.** Писателей теперь двое (фоновый проход по завершении и + выход из приложения), а запись идёт через один и тот же временный файл с переименованием. +- **`RestoreState` не выставляет `Health = Alive`** — «работала вчера» живёт в `WasAliveOnLastRun` и + влияет только на порядок прогрева. Иначе пул отрапортует живыми тех, с кем не разговаривал: прогрев + сочтёт цель достигнутой, а гейт сбора откроется по данным недельной давности. Ловит + `A_remembered_proxy_is_not_reported_live_until_it_answers_again`. +- **`RestoreState` не восстанавливает карантин**: окно — стенные часы, между запусками могли пройти + сутки, и перенос сажал бы прокси за давно истёкшее. +- **Сохраняются только `HasEverAnswered`** = `SuccessCount > 0 || IsBelievedAlive` («believed» — + вердикт этой сессии, а без него прошлой). Прогрев обрывается рано, поэтому большинство запомненных + заканчивают сессию непроверенными — строгое `Health == Alive` стёрло бы накопленный список за пару + запусков. +- **`Select` и `Next` зарезервированы для CA1716** — метод стратегии называется `Pick`. + +## Статистика на «Обзоре» + +`DashboardViewModel` держит сводку пула и обновляет её по событию `IProxyPool.Changed` с throttle +250 мс — событие приходит на каждый исход лизы и на каждую пробу. «Живых» на карточке — это +`LiveCount`, то же число, что читает гейт сбора; `RefreshProxyStats()` публичен, чтобы тесты не ждали +throttle. diff --git a/src/AvParser.Core/Collecting/ICollectRunner.cs b/src/AvParser.Core/Collecting/ICollectRunner.cs index d3b7533..3ba6920 100644 --- a/src/AvParser.Core/Collecting/ICollectRunner.cs +++ b/src/AvParser.Core/Collecting/ICollectRunner.cs @@ -125,7 +125,12 @@ public sealed record CollectOptions public string UserAgent { get; init; } = "AvParser/0.1"; /// Whether a missing proxy is a hard failure rather than a direct connection. - public bool RequireProxy { get; init; } + /// + /// Defaults to the strict end: a caller that has not said otherwise has not asked to be seen + /// from their own address. The collect page sets it per source, since one host may be the user's + /// own service and the next a stranger's. + /// + public bool RequireProxy { get; init; } = true; /// How browsable copies point at their blobs. public ShowcaseMode ShowcaseMode { get; init; } = ShowcaseMode.HardLink; diff --git a/src/AvParser.Core/Collecting/IMediaSource.cs b/src/AvParser.Core/Collecting/IMediaSource.cs index 2b38554..a81fe80 100644 --- a/src/AvParser.Core/Collecting/IMediaSource.cs +++ b/src/AvParser.Core/Collecting/IMediaSource.cs @@ -1,3 +1,4 @@ +using AvParser.Core.Collecting.Sources; using AvParser.Core.Parsing; namespace AvParser.Core.Collecting; @@ -12,64 +13,130 @@ namespace AvParser.Core.Collecting; /// public interface IMediaSource : IParser; -/// Read-only view over every registered media source. +/// Live view over the user's media sources, with editing. +/// +/// No longer immutable: sources are user-added data now, so the list changes at runtime. Editing is +/// delegated to an ; the catalog re-materialises a +/// per stored config and raises so the UI can +/// rebuild. An empty catalog is a valid state — the collector shows an "add a source" prompt rather +/// than failing to start. +/// public interface IMediaSourceCatalog { - /// All registered sources, ordered by . + /// Every source, ordered by . IReadOnlyList Sources { get; } - /// The source used when nothing has been chosen yet. - IMediaSource DefaultSource { get; } + /// The stored configs behind the sources, for editing. + IReadOnlyList Configs { get; } /// Finds a source by its stable id; when unknown. IMediaSource? Find(string? id); - /// Finds a source by id, falling back to . - IMediaSource FindOrDefault(string? id) => Find(id) ?? DefaultSource; + /// Adds a source, returning the stored config with its assigned id. + Task AddAsync(PatternSourceConfig config, CancellationToken cancellationToken = default); + + /// Replaces the source with the same id. Returns whether one was present. + Task UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default); + + /// Removes a source by id. Returns whether it was present. + Task RemoveAsync(string id, CancellationToken cancellationToken = default); + + /// Raised after the source list changes. + event EventHandler? Changed; } /// -public sealed class MediaSourceCatalog : IMediaSourceCatalog +public sealed class MediaSourceCatalog : IMediaSourceCatalog, IDisposable { - private readonly Dictionary _byId; + private readonly IUserSourceStore _store; + private readonly object _gate = new(); - /// Builds a catalog from every source the container resolved. - /// Registered sources. - /// - /// Which source to land on. Named explicitly rather than taken as "first alphabetically", - /// because that would make the landing page depend on a display name — and would put a network - /// source there, so the app would open behind the proxy gate for no reason. - /// - /// No sources were registered, or two share an id. - public MediaSourceCatalog(IEnumerable sources, string? defaultId = null) + private Dictionary _byId = new(StringComparer.OrdinalIgnoreCase); + private IReadOnlyList _sources = []; + + /// Builds a catalog that tracks the store. + /// Where the user's source configs live. + public MediaSourceCatalog(IUserSourceStore store) { - ArgumentNullException.ThrowIfNull(sources); + _store = store ?? throw new ArgumentNullException(nameof(store)); - Sources = [.. sources.OrderBy(source => source.DisplayName, StringComparer.OrdinalIgnoreCase)]; - - if (Sources.Count == 0) - { - throw new ArgumentException("At least one media source must be registered.", nameof(sources)); - } - - _byId = new Dictionary(StringComparer.OrdinalIgnoreCase); - foreach (var source in Sources) - { - if (!_byId.TryAdd(source.Id, source)) - { - throw new ArgumentException($"Duplicate media source id '{source.Id}'.", nameof(sources)); - } - } - - DefaultSource = Find(defaultId) ?? Sources[0]; + Rebuild(); + _store.Changed += OnStoreChanged; } /// - public IReadOnlyList Sources { get; } + public event EventHandler? Changed; /// - public IMediaSource DefaultSource { get; } + public IReadOnlyList Sources + { + get + { + lock (_gate) + { + return _sources; + } + } + } /// - public IMediaSource? Find(string? id) => id is not null && _byId.TryGetValue(id, out var source) ? source : null; + public IReadOnlyList Configs => _store.List(); + + /// + public IMediaSource? Find(string? id) + { + if (id is null) + { + return null; + } + + lock (_gate) + { + return _byId.TryGetValue(id, out var source) ? source : null; + } + } + + /// + public Task AddAsync( + PatternSourceConfig config, + CancellationToken cancellationToken = default + ) => _store.AddAsync(config, cancellationToken); + + /// + public Task UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default) => + _store.UpdateAsync(config, cancellationToken); + + /// + public Task RemoveAsync(string id, CancellationToken cancellationToken = default) => + _store.RemoveAsync(id, cancellationToken); + + /// + public void Dispose() => _store.Changed -= OnStoreChanged; + + private void OnStoreChanged(object? sender, EventArgs e) + { + Rebuild(); + Changed?.Invoke(this, EventArgs.Empty); + } + + private void Rebuild() + { + var sources = _store + .List() + .Select(config => (IMediaSource)new PatternMediaSource(config)) + .OrderBy(source => source.DisplayName, StringComparer.OrdinalIgnoreCase) + .ToArray(); + + var byId = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var source in sources) + { + byId[source.Id] = source; + } + + lock (_gate) + { + _sources = sources; + _byId = byId; + } + } } diff --git a/src/AvParser.Core/Collecting/IUserSourceStore.cs b/src/AvParser.Core/Collecting/IUserSourceStore.cs new file mode 100644 index 0000000..ba3e32d --- /dev/null +++ b/src/AvParser.Core/Collecting/IUserSourceStore.cs @@ -0,0 +1,30 @@ +using AvParser.Core.Collecting.Sources; + +namespace AvParser.Core.Collecting; + +/// +/// The user's own list of pattern sources, persisted between runs. +/// +/// +/// The collector equivalent of IMutableProxySource: sources are now data the user edits, not +/// types the build ships, so they live in a file and this is the seam that reads and writes it. The +/// interface is in the domain and the persistence in infrastructure, so the catalog can depend on it +/// without the domain learning about the filesystem. +/// +public interface IUserSourceStore +{ + /// Every stored config, in insertion order. Implementations may cache. + IReadOnlyList List(); + + /// Adds a config. Returns the stored config, whose id is assigned if it had none. + Task AddAsync(PatternSourceConfig config, CancellationToken cancellationToken = default); + + /// Replaces the config with the same id. Returns whether one was present to replace. + Task UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default); + + /// Removes the config with this id. Returns whether it was present. + Task RemoveAsync(string id, CancellationToken cancellationToken = default); + + /// Raised after the list changes, so a catalog can rebuild its materialised sources. + event EventHandler? Changed; +} diff --git a/src/AvParser.Core/Collecting/Sources/PatternMediaSource.cs b/src/AvParser.Core/Collecting/Sources/PatternMediaSource.cs new file mode 100644 index 0000000..761b506 --- /dev/null +++ b/src/AvParser.Core/Collecting/Sources/PatternMediaSource.cs @@ -0,0 +1,202 @@ +using System.Runtime.CompilerServices; +using AvParser.Core.Parsing; + +namespace AvParser.Core.Collecting.Sources; + +/// +/// Generates candidate ids from a and hangs them off its base URL. +/// +/// +/// It only produces addresses — the fetcher does every byte of the downloading, redirect-following +/// and sniffing. Unlike a pasted list, this one hits a remote host on purpose +/// and in bulk, so is : the proxy gate exists for +/// exactly this. Which of the generated ids actually resolve is the fetcher's verdict, not the +/// source's; the source's only job is to hand out well-formed addresses within the pattern. +/// +public sealed class PatternMediaSource : IMediaSource +{ + /// How many candidates to emit each time the enumerator yields control. + private const int YieldInterval = 256; + + /// + /// Above this the id space is treated as effectively unbounded, so no exhaustion cap is applied. + /// + /// + /// Below it the space is small enough that a naive "generate until the budget is met" loop would + /// spin forever on duplicate ids once most of the space is drawn, so the budget is clamped to the + /// space size instead. + /// + private const long SmallSpaceThreshold = 5_000_000; + + /// How many ids an unbounded run remembers before it starts over. + /// + /// An uncapped run has no end, so the de-duplication set would grow until the process died. Past + /// a million draws from a space this large the chance of redrawing an id is negligible, and a + /// redraw is not a correctness problem anyway: the store's seen_url journal is what keeps + /// an address from being fetched twice. Only ever applied to unbounded runs — a bounded one over + /// a small space needs the full set to honour "never exceed the space". + /// + private const int SeenCapacity = 1 << 20; + + private readonly PatternSourceConfig _config; + private readonly string _alphabet; + + /// Wraps a config as a runnable source. + public PatternMediaSource(PatternSourceConfig config) + { + _config = config ?? throw new ArgumentNullException(nameof(config)); + _alphabet = config.AlphabetCharacters; + } + + /// The config this source runs. + public PatternSourceConfig Config => _config; + + /// + public string Id => _config.Id; + + /// + public string DisplayName => _config.Name; + + /// + public string Description => $"{_config.BaseUrl} — id {_config.MinLength}–{_config.MaxLength} from '{_alphabet}'"; + + /// + public bool RequiresNetwork => true; + + /// + public bool CanParse(MediaQuery input) => input is not null && _alphabet.Length > 0; + + /// + public async IAsyncEnumerable> ParseAsync( + MediaQuery input, + IProgress? progress, + [EnumeratorCancellation] CancellationToken cancellationToken + ) + { + ArgumentNullException.ThrowIfNull(input); + + if (_alphabet.Length == 0) + { + yield break; + } + + var budget = Budget(input.Limit); + var unbounded = budget == 0; + var seen = new HashSet(Math.Min(unbounded ? SeenCapacity : budget, 1 << 16)); + var emitted = 0; + + // A run of collisions this long means the space is all but drawn; stop rather than spin. The + // multiplier keeps a false trip vanishingly unlikely while the space still has room. + var stale = 0; + var staleLimit = Math.Max(1_000, _alphabet.Length * 64); + + while (unbounded || emitted < budget) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (unbounded && seen.Count >= SeenCapacity) + { + seen.Clear(); + } + + var id = NextId(); + if (!seen.Add(id)) + { + if (++stale >= staleLimit) + { + break; + } + + continue; + } + + stale = 0; + + yield return ParseOutcome.Success( + new MediaCandidate(new Uri(_config.BaseUrl, id + _config.Extension)) + { + SourceId = Id, + ExternalId = id, + Ordinal = emitted + 1, + } + ); + + emitted++; + + if (emitted % YieldInterval == 0) + { + progress?.Report(new ParseProgress(emitted, budget)); + await Task.Yield(); + } + } + + progress?.Report(new ParseProgress(emitted, budget)); + } + + /// + /// The number of candidates to generate: the caller's budget, capped at the id space when small. + /// + /// + /// Returns 0 for "keep going until the caller stops asking", which is what a limit of zero + /// means. That can only happen over a large space — a small one is always capped at its own size, + /// because past that point every further draw is a collision and the run would spin rather than + /// collect. + /// + private int Budget(int limit) + { + var space = SpaceSize(); + + if (space >= SmallSpaceThreshold) + { + return Math.Max(0, limit); + } + + return limit > 0 ? (int)Math.Min(limit, space) : (int)space; + } + + /// + /// Size of the id space, saturating at so large alphabets do + /// not overflow — the exact value past the threshold is never needed. + /// + private long SpaceSize() + { + long total = 0; + long power = 1; + + for (var length = 1; length <= _config.MaxLength; length++) + { + power = Math.Min(power * _alphabet.Length, SmallSpaceThreshold); + + if (length >= _config.MinLength) + { + total += power; + if (total >= SmallSpaceThreshold) + { + return SmallSpaceThreshold; + } + } + } + + return total; + } + + private string NextId() + { + var length = + _config.MinLength == _config.MaxLength + ? _config.MinLength + : Random.Shared.Next(_config.MinLength, _config.MaxLength + 1); + + return string.Create( + length, + _alphabet, + static (span, alphabet) => + { + for (var i = 0; i < span.Length; i++) + { + span[i] = alphabet[Random.Shared.Next(alphabet.Length)]; + } + } + ); + } +} diff --git a/src/AvParser.Core/Collecting/Sources/PatternSourceConfig.cs b/src/AvParser.Core/Collecting/Sources/PatternSourceConfig.cs new file mode 100644 index 0000000..889e6ce --- /dev/null +++ b/src/AvParser.Core/Collecting/Sources/PatternSourceConfig.cs @@ -0,0 +1,258 @@ +namespace AvParser.Core.Collecting.Sources; + +/// Which characters an id is drawn from. +public enum IdAlphabet +{ + /// Lower-case latin letters, az. + LettersLower = 0, + + /// Upper-case latin letters, AZ. + LettersUpper = 1, + + /// Both cases of latin letters. + Letters = 2, + + /// Decimal digits, 09. + Digits = 3, + + /// Letters of both cases and digits. + Alphanumeric = 4, + + /// Lower-case hexadecimal, 09 and af. + HexLower = 5, + + /// Whatever characters the user typed in . + Custom = 6, +} + +/// Turns an into the concrete set of characters to draw from. +public static class PatternAlphabet +{ + private const string Lower = "abcdefghijklmnopqrstuvwxyz"; + private const string Upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"; + private const string Digit = "0123456789"; + private const string Hex = "0123456789abcdef"; + + /// Resolves the character set, de-duplicated, for a config. + /// + /// Custom alphabets are de-duplicated so a pasted "aabbcc" does not skew the random draw toward + /// the repeated characters. An empty result means the config is unusable and is rejected by + /// before it ever reaches a source. + /// + public static string Resolve(IdAlphabet alphabet, string? customAlphabet = null) => + alphabet switch + { + IdAlphabet.LettersLower => Lower, + IdAlphabet.LettersUpper => Upper, + IdAlphabet.Letters => Lower + Upper, + IdAlphabet.Digits => Digit, + IdAlphabet.Alphanumeric => Lower + Upper + Digit, + IdAlphabet.HexLower => Hex, + IdAlphabet.Custom => Distinct(customAlphabet), + _ => string.Empty, + }; + + private static string Distinct(string? text) + { + if (string.IsNullOrEmpty(text)) + { + return string.Empty; + } + + var seen = new HashSet(text.Length); + var builder = new System.Text.StringBuilder(text.Length); + + foreach (var c in text) + { + // A separator inside an id would break the "one path segment" assumption the base URL + // relies on, so control and obvious URL-structural characters never make the alphabet. + if (char.IsControl(c) || c is '/' or '\\' or '?' or '#' or ' ') + { + continue; + } + + if (seen.Add(c)) + { + builder.Append(c); + } + } + + return builder.ToString(); + } +} + +/// Why a could not be built. +public enum PatternConfigError +{ + /// The config is valid. + None = 0, + + /// The display name was blank. + NameRequired, + + /// The base URL was blank, relative, or not http/https. + BaseUrlInvalid, + + /// The length range was outside 1..64 or had min greater than max. + LengthRangeInvalid, + + /// The alphabet resolved to no usable characters. + AlphabetEmpty, +} + +/// +/// A user-added source that generates candidate ids from a pattern and hangs them off a base URL. +/// +/// +/// This is the whole "a source is data, not a class" idea: one +/// implementation is parameterised by this record, so adding a source is adding a row to a file +/// rather than shipping a build. The record is what sources.user.json persists. +/// +/// Stable identifier; scopes storage and purge, and is remembered as the last choice. +/// User-given display name. +/// Absolute http/https address ending in /; ids resolve against it. +/// Shortest id to generate, inclusive. +/// Longest id to generate, inclusive. +/// Which characters ids are drawn from. +/// Characters used when is . +/// +/// Suffix appended after the id. A blank one from the editor becomes ; +/// a config built by hand may still carry for none. +/// +/// +/// Whether this source may run from the user's own address when no proxy is live. Per source rather +/// than per app: one host may be the user's own service, where a proxy is pointless, while the next +/// is a stranger's, where going direct is exactly what the user was avoiding — a global switch forced +/// the permissive answer on both. Defaults to , which is both the safe end and +/// where an older sources.user.json lands. +/// +public sealed record PatternSourceConfig( + string Id, + string Name, + Uri BaseUrl, + int MinLength, + int MaxLength, + IdAlphabet Alphabet, + string? CustomAlphabet, + string? Extension, + bool AllowDirectConnection = false +) +{ + /// Hard ceiling on id length, guarding both the UI spinner and a hand-edited file. + public const int MaxIdLength = 64; + + /// Suffix used when the editor's extension field is left empty. + /// + /// A blank field means "the usual one", not "no suffix": every host worth pointing this at + /// serves /{id}.jpg, and a bare /{id} is almost always a typo that costs a whole + /// run of 404s to discover. The editor shows this as the placeholder and the saved config keeps + /// it verbatim, so the field stops being blank once saved. + /// + public const string DefaultExtension = ".jpg"; + + /// The characters this config draws ids from. + public string AlphabetCharacters => PatternAlphabet.Resolve(Alphabet, CustomAlphabet); + + /// + /// Validates and normalises the inputs, generating an id when one is not supplied. + /// + /// + /// The single gate every config passes through, used by both the editor (for inline feedback) + /// and the store (so a hand-edited file cannot smuggle in an unusable source). Normalises the + /// base URL to a trailing slash and the extension to a leading dot, because both are joined by + /// naive concatenation downstream. + /// + public static PatternConfigError TryCreate( + string? name, + string? baseUrl, + int minLength, + int maxLength, + IdAlphabet alphabet, + string? customAlphabet, + string? extension, + bool allowDirectConnection, + out PatternSourceConfig? config, + string? id = null + ) + { + config = null; + + var trimmedName = name?.Trim() ?? string.Empty; + if (trimmedName.Length == 0) + { + return PatternConfigError.NameRequired; + } + + if ( + string.IsNullOrWhiteSpace(baseUrl) + || !Uri.TryCreate(baseUrl.Trim(), UriKind.Absolute, out var parsed) + || parsed.Scheme is not ("http" or "https") + ) + { + return PatternConfigError.BaseUrlInvalid; + } + + if (minLength < 1 || maxLength > MaxIdLength || minLength > maxLength) + { + return PatternConfigError.LengthRangeInvalid; + } + + if (PatternAlphabet.Resolve(alphabet, customAlphabet).Length == 0) + { + return PatternConfigError.AlphabetEmpty; + } + + config = new PatternSourceConfig( + string.IsNullOrWhiteSpace(id) ? NewId(trimmedName) : id!.Trim(), + trimmedName, + EnsureTrailingSlash(parsed), + minLength, + maxLength, + alphabet, + alphabet == IdAlphabet.Custom ? customAlphabet : null, + NormaliseExtension(extension), + allowDirectConnection + ); + + return PatternConfigError.None; + } + + private static Uri EnsureTrailingSlash(Uri url) + { + if (url.AbsoluteUri.EndsWith('/')) + { + return url; + } + + // A base URL without a trailing slash drops its last segment when a relative id is resolved + // against it, so "…/test1" would collect from "…/2df.jpg", not "…/test1/2df.jpg". + return new Uri(url.AbsoluteUri + "/", UriKind.Absolute); + } + + private static string NormaliseExtension(string? extension) + { + var trimmed = extension?.Trim(); + if (string.IsNullOrEmpty(trimmed)) + { + return DefaultExtension; + } + + return trimmed.StartsWith('.') ? trimmed : "." + trimmed; + } + + private static string NewId(string name) + { + var slug = new System.Text.StringBuilder(name.Length); + foreach (var c in name.ToLowerInvariant()) + { + slug.Append(char.IsLetterOrDigit(c) ? c : '-'); + } + + var trimmed = slug.ToString().Trim('-'); + var prefix = trimmed.Length == 0 ? "source" : trimmed; + + // A short random suffix keeps ids unique when two sources share a name, without leaking a + // full GUID into settings and logs. + return $"{prefix}-{Guid.NewGuid().ToString("N").AsSpan(0, 6)}"; + } +} diff --git a/src/AvParser.Core/Collecting/Sources/UrlListSource.cs b/src/AvParser.Core/Collecting/Sources/UrlListSource.cs deleted file mode 100644 index a729828..0000000 --- a/src/AvParser.Core/Collecting/Sources/UrlListSource.cs +++ /dev/null @@ -1,122 +0,0 @@ -using System.Runtime.CompilerServices; -using AvParser.Core.Parsing; - -namespace AvParser.Core.Collecting.Sources; - -/// -/// Reads addresses the user pasted in, one per line. -/// -/// -/// The only source that touches no network at all, which is why it lives in the domain and why -/// RequiresNetwork stays false: it discovers nothing, it just reads what it was handed. -/// Downloading those addresses is the fetcher's business and is gated separately. -/// -public sealed class UrlListSource : IMediaSource -{ - /// - public string Id => "url-list"; - - /// - public string DisplayName => "URL list"; - - /// - public string Description => "One address per line. Blank lines and lines starting with '#' are ignored."; - - /// - public bool CanParse(MediaQuery input) => - input is not null && TextLines.Split(input.Text).Any(line => TryParse(line, out _)); - - /// - public async IAsyncEnumerable> ParseAsync( - MediaQuery input, - IProgress? progress, - [EnumeratorCancellation] CancellationToken cancellationToken - ) - { - ArgumentNullException.ThrowIfNull(input); - - var lines = TextLines.Split(input.Text); - var ordinal = 0; - var yielded = 0; - - for (var index = 0; index < lines.Length; index++) - { - cancellationToken.ThrowIfCancellationRequested(); - - var line = lines[index]; - - if (TextLines.IsSkippable(line)) - { - continue; - } - - ordinal++; - - if (TryParse(line, out var url)) - { - yield return ParseOutcome.Success( - new MediaCandidate(url!) { SourceId = Id, Ordinal = ordinal } - ); - - yielded++; - - if (input.HasLimit && yielded >= input.Limit) - { - break; - } - } - else - { - // Named rather than silently dropped: a mistyped address in a pasted list of two - // hundred is otherwise impossible to find. The line goes in twice on purpose — as - // the English fallback message, and as the argument the translation substitutes. - var offending = line.Trim(); - - yield return ParseOutcome.Failure( - ParseError.Create( - ordinal, - "NotAnAddress", - $"'{offending}' is not an http or https address.", - offending - ) - ); - } - - if (ordinal % TextLines.ProgressInterval == 0) - { - progress?.Report(new ParseProgress(index + 1, lines.Length)); - } - - if (ordinal % TextLines.YieldInterval == 0) - { - await Task.Yield(); - } - } - - progress?.Report(new ParseProgress(lines.Length, lines.Length)); - } - - /// Accepts only absolute HTTP addresses. - /// - /// file: and data: are rejected here as well as at redirect time: a pasted list is - /// as likely to have come from somewhere else as to have been typed. - /// - private static bool TryParse(string line, out Uri? url) - { - url = null; - var text = line.Trim(); - - if (text.Length == 0 || !Uri.TryCreate(text, UriKind.Absolute, out var parsed)) - { - return false; - } - - if (parsed.Scheme is not ("http" or "https")) - { - return false; - } - - url = parsed; - return true; - } -} diff --git a/src/AvParser.Core/DependencyInjection/CoreServiceCollectionExtensions.cs b/src/AvParser.Core/DependencyInjection/CoreServiceCollectionExtensions.cs index 6636fcf..91e483b 100644 --- a/src/AvParser.Core/DependencyInjection/CoreServiceCollectionExtensions.cs +++ b/src/AvParser.Core/DependencyInjection/CoreServiceCollectionExtensions.cs @@ -1,5 +1,3 @@ -using AvParser.Core.Collecting; -using AvParser.Core.Collecting.Sources; using Microsoft.Extensions.DependencyInjection; namespace AvParser.Core.DependencyInjection; @@ -8,26 +6,13 @@ namespace AvParser.Core.DependencyInjection; public static class CoreServiceCollectionExtensions { /// - /// Registers every media source the domain itself provides. + /// Registers the domain services. Media sources are user-added data now, not registered types, + /// so they are built from the persisted store in the infrastructure layer rather than here. /// - /// - /// Adding a source is a one-line change here — that is the whole point of the - /// / split. Sources that need a - /// network live in the infrastructure layer and register themselves there. - /// public static IServiceCollection AddAvParserCore(this IServiceCollection services) { ArgumentNullException.ThrowIfNull(services); - services.AddSingleton(); - return services; } - - /// Id of the source the collector opens on. - /// - /// Named rather than left to alphabetical order, which would land on the network source and - /// open the page behind the proxy gate before the user has asked for anything. - /// - public const string DefaultMediaSourceId = "url-list"; } diff --git a/src/AvParser.Core/Parsing/ParseError.cs b/src/AvParser.Core/Parsing/ParseError.cs index b379d02..5d9457c 100644 --- a/src/AvParser.Core/Parsing/ParseError.cs +++ b/src/AvParser.Core/Parsing/ParseError.cs @@ -20,6 +20,15 @@ public sealed record ParseError(int Index, string Message) /// Values to substitute into the translated message. public IReadOnlyList Arguments { get; init; } = []; + /// What the failure was about — the address or the line it came from. + /// + /// Deliberately outside and : a translated template + /// has fixed placeholders, so an address cannot be appended to it without editing every + /// translation. The live collection log prints this beside the translated text, which is the + /// difference between "the site answered 404" and knowing which of ten thousand ids that was. + /// + public string? Subject { get; init; } + /// Creates an error carrying a translation code. public static ParseError Create(int index, string code, string message, params object?[] arguments) => new(index, message) { Code = code, Arguments = arguments }; diff --git a/src/AvParser.Core/Proxies/IProxyPool.cs b/src/AvParser.Core/Proxies/IProxyPool.cs index b14c789..4d91d6a 100644 --- a/src/AvParser.Core/Proxies/IProxyPool.cs +++ b/src/AvParser.Core/Proxies/IProxyPool.cs @@ -51,6 +51,24 @@ public interface IProxyPool IProgress? progress = null, CancellationToken cancellationToken = default ); + + /// + /// Probes whatever the warm-up never got to, slowly, while the app is doing something else. + /// + /// Probes in flight; deliberately far below a manual sweep's. + /// Optional progress, in checked-entry counts. + /// Stops the pass; entries already probed keep their verdict. + /// How many of the previously unchecked entries answered. + /// + /// The warm-up stops the moment it has enough live proxies, which on a free list of a few + /// thousand means thirty were checked and the rest are unknown — not dead. This finishes the + /// job in the background so the pool ends up genuinely checked without anyone waiting for it. + /// + Task TopUpAsync( + int concurrency, + IProgress? progress = null, + CancellationToken cancellationToken = default + ); } /// diff --git a/src/AvParser.Core/Proxies/ProxyOptions.cs b/src/AvParser.Core/Proxies/ProxyOptions.cs index afb9024..915baa7 100644 --- a/src/AvParser.Core/Proxies/ProxyOptions.cs +++ b/src/AvParser.Core/Proxies/ProxyOptions.cs @@ -113,15 +113,6 @@ public sealed record ProxyOptions /// public int MinimumLiveProxies { get; init; } = 10; - /// - /// Whether work may run without a proxy when none is available. - /// - /// - /// Only parsers that declare RequiresNetwork are affected; a parser working on pasted - /// text is never blocked. - /// - public bool AllowDirectConnection { get; init; } - /// Validates the options, throwing on values that would misbehave silently. /// A numeric option is out of range. public ProxyOptions Validated() diff --git a/src/AvParser.Core/Proxies/ProxyPool.cs b/src/AvParser.Core/Proxies/ProxyPool.cs index 3226c8a..446f7f7 100644 --- a/src/AvParser.Core/Proxies/ProxyPool.cs +++ b/src/AvParser.Core/Proxies/ProxyPool.cs @@ -5,6 +5,9 @@ namespace AvParser.Core.Proxies; /// public sealed class ProxyPool : IProxyPool { + /// How many probes a background top-up makes before it tells anyone. + private const int ChangeInterval = 32; + private readonly IReadOnlyList _sources; private readonly IProxyProbe _probe; private readonly TimeProvider _time; @@ -367,6 +370,77 @@ public sealed class ProxyPool : IProxyPool return LiveCount; } + /// + public async Task TopUpAsync( + int concurrency, + IProgress? progress = null, + CancellationToken cancellationToken = default + ) + { + ArgumentOutOfRangeException.ThrowIfLessThan(concurrency, 1); + + var options = Options; + + // Only what nobody has spoken to this session. Re-probing a known-dead entry would just be + // a sweep, and re-probing a live one could sideline the proxy a collect run is using. + var targets = WarmUpOrder().Where(entry => entry.Health == ProxyHealthState.Unknown).ToArray(); + var total = targets.Length; + + if (total == 0) + { + progress?.Report(new ProxySweepProgress(0, 0, LiveCount)); + return 0; + } + + using var limiter = new SemaphoreSlim(concurrency, concurrency); + var checkedCount = 0; + var aliveCount = 0; + + var work = targets.Select(async entry => + { + await limiter.WaitAsync(cancellationToken).ConfigureAwait(false); + + try + { + var result = await _probe.ProbeAsync(entry.Endpoint, options, cancellationToken).ConfigureAwait(false); + + entry.RecordProbe( + _time.GetUtcNow(), + result.Alive, + result.Latency, + result.Error, + quarantineOnFailure: options.BaseQuarantine + ); + + if (result.Alive) + { + Interlocked.Increment(ref aliveCount); + } + } + finally + { + limiter.Release(); + + var done = Interlocked.Increment(ref checkedCount); + progress?.Report(new ProxySweepProgress(done, total, Volatile.Read(ref aliveCount))); + + // Periodically rather than per probe: this pass runs for minutes over thousands of + // entries, and a page that only hears about it at the end looks stuck. + if (done % ChangeInterval == 0) + { + RaiseChanged(); + } + } + }); + + await Task.WhenAll(work).ConfigureAwait(false); + + // No _strategy.Reset() here, unlike a sweep: this runs while proxies are being handed out, + // and resetting would drop a sticky pick mid-collect for entries the pick is not even among. + RaiseChanged(); + return aliveCount; + } + /// /// Orders candidates for a warm-up: what worked last time, then what looks most promising. /// diff --git a/src/AvParser.Core/Settings/AppSettings.cs b/src/AvParser.Core/Settings/AppSettings.cs index 65de508..84af4bf 100644 --- a/src/AvParser.Core/Settings/AppSettings.cs +++ b/src/AvParser.Core/Settings/AppSettings.cs @@ -46,7 +46,6 @@ public enum AppTheme /// Per-proxy probe timeout, in seconds. /// How many probes run at once during a pool sweep. /// How many working proxies a startup warm-up aims for. -/// Whether network parsers may run without a proxy. /// Id of the media source selected last time; resolved leniently on load. /// How many downloads may be in flight at once. /// How many requests one origin may be serving at once. @@ -61,6 +60,16 @@ public enum AppTheme /// How browsable copies point at their blobs. /// Identifies the collector to origins that care. /// Where collected media goes; null keeps it beside the settings. +/// How many items to stop after collecting; 0 means no target. +/// +/// How many candidate ids one source may try in a run; 0 means until the user stops it. +/// +/// +/// Ids of the sources ticked to run, separated by ;. A joined string rather than a list +/// because keeps value equality — the settings service short-circuits a +/// no-op write by comparing records, and a list-valued member would make every save look like a +/// change. Same reasoning as . +/// public sealed record AppSettings( AppTheme Theme = AppTheme.System, AppLanguage Language = AppLanguage.System, @@ -76,7 +85,6 @@ public sealed record AppSettings( int ProxyProbeTimeoutSeconds = 8, int ProxyProbeConcurrency = 64, int ProxyMinimumLive = 10, - bool AllowDirectConnection = false, string? LastSourceId = null, int MaxConcurrentDownloads = 4, int MaxConcurrentPerHost = 2, @@ -90,9 +98,34 @@ public sealed record AppSettings( MediaKindFilter AllowedMediaKinds = MediaKindFilter.All, ShowcaseMode ShowcaseMode = ShowcaseMode.HardLink, string CollectUserAgent = "AvParser/0.1", - string? MediaRootOverride = null + string? MediaRootOverride = null, + int CollectTargetCount = 100, + int CollectAttemptBudget = 5000, + string? CollectSourceIds = null ) { + /// Separates ids inside . + public const char SourceIdSeparator = ';'; + + /// Splits a stored back into ids. + /// + /// A method rather than a property: every public property here is serialised, and a derived one + /// would be written back into the settings file as a second copy of the same list. + /// + public static IReadOnlyList SplitSourceIds(string? ids) => + string.IsNullOrWhiteSpace(ids) + ? [] + : ids.Split(SourceIdSeparator, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + + /// Joins ids for storage in . + public static string? JoinSourceIds(IEnumerable ids) + { + ArgumentNullException.ThrowIfNull(ids); + + var joined = string.Join(SourceIdSeparator, ids); + return joined.Length == 0 ? null : joined; + } + /// Projects the proxy-related settings onto . /// /// Settings are persisted as primitives so an old file still deserialises; the pool wants a @@ -115,7 +148,6 @@ public sealed record AppSettings( ProbeTimeout = TimeSpan.FromSeconds(Math.Clamp(ProxyProbeTimeoutSeconds, 1, 120)), ProbeConcurrency = Math.Clamp(ProxyProbeConcurrency, 1, 512), MinimumLiveProxies = Math.Clamp(ProxyMinimumLive, 1, 500), - AllowDirectConnection = AllowDirectConnection, }.Validated(); } @@ -142,7 +174,9 @@ public sealed record AppSettings( // to be an accident than an intention to collect nothing. AllowedKinds = AllowedMediaKinds == MediaKindFilter.None ? MediaKindFilter.All : AllowedMediaKinds, UserAgent = string.IsNullOrWhiteSpace(CollectUserAgent) ? "AvParser/0.1" : CollectUserAgent, - RequireProxy = !AllowDirectConnection, + // Whether a missing proxy is fatal is the source's own setting; the collect page + // overrides this per source. The default here is the strict one on purpose. + RequireProxy = true, ShowcaseMode = ShowcaseMode, }; } diff --git a/src/AvParser.Infrastructure/Collecting/CollectRunner.cs b/src/AvParser.Infrastructure/Collecting/CollectRunner.cs index 11bf5f1..b328bcb 100644 --- a/src/AvParser.Infrastructure/Collecting/CollectRunner.cs +++ b/src/AvParser.Infrastructure/Collecting/CollectRunner.cs @@ -311,7 +311,10 @@ public sealed class CollectRunner( result.ErrorCode ?? "Failed", result.ErrorDetail ?? candidate.Url.AbsoluteUri, result.ErrorDetail ?? candidate.Url.AbsoluteUri - ) + ) with + { + Subject = candidate.Url.AbsoluteUri, + } ); } diff --git a/src/AvParser.Infrastructure/Collecting/JsonUserSourceStore.cs b/src/AvParser.Infrastructure/Collecting/JsonUserSourceStore.cs new file mode 100644 index 0000000..ac18ac1 --- /dev/null +++ b/src/AvParser.Infrastructure/Collecting/JsonUserSourceStore.cs @@ -0,0 +1,281 @@ +using System.Text.Json; +using System.Text.Json.Serialization; +using AvParser.Core.Collecting; +using AvParser.Core.Collecting.Sources; +using AvParser.Infrastructure.Storage; +using Microsoft.Extensions.Logging; + +namespace AvParser.Infrastructure.Collecting; + +/// One user-added pattern source, as persisted. +/// +/// String-typed rather than storing a directly: the config carries +/// a and an enum, and a hand-edited or older file must degrade to "skip this row" +/// rather than throw. Every record is re-validated through +/// on load, so persistence stays forgiving while the in-memory list stays well-formed. +/// +public sealed record PatternSourceRecord( + string Id, + string Name, + string BaseUrl, + int MinLength, + int MaxLength, + string Alphabet, + string? CustomAlphabet, + string? Extension, + bool AllowDirectConnection = false +); + +/// Source-generated serialiser metadata for the user's source list. +[JsonSourceGenerationOptions(WriteIndented = true, PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase)] +[JsonSerializable(typeof(List))] +internal sealed partial class UserSourceJsonContext : JsonSerializerContext; + +/// +/// The user's own media-source list, persisted next to the settings. +/// +/// +/// The collector twin of CustomProxySource: a separate file the user can back up and share, +/// read once and cached, written atomically. Mutations raise so the catalog +/// re-materialises its sources. +/// +public sealed class JsonUserSourceStore : IUserSourceStore, IDisposable +{ + private readonly IAppPaths _paths; + private readonly ILogger _logger; + private readonly SemaphoreSlim _gate = new(1, 1); + private readonly object _cacheLock = new(); + + private List? _records; + private IReadOnlyList _configs = []; + + /// Creates the store. + public JsonUserSourceStore(IAppPaths paths, ILogger logger) + { + _paths = paths ?? throw new ArgumentNullException(nameof(paths)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + /// + public event EventHandler? Changed; + + /// + public IReadOnlyList List() + { + lock (_cacheLock) + { + EnsureLoaded(); + return _configs; + } + } + + /// + public async Task AddAsync( + PatternSourceConfig config, + CancellationToken cancellationToken = default + ) + { + ArgumentNullException.ThrowIfNull(config); + + await _gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + var records = LoadRecords(); + + // A colliding id means a hand-edited file or a repeated slug; take the incoming one as + // authoritative and replace rather than grow a second row with the same id. + records.RemoveAll(r => string.Equals(r.Id, config.Id, StringComparison.OrdinalIgnoreCase)); + records.Add(ToRecord(config)); + + await SaveAsync(records, cancellationToken).ConfigureAwait(false); + } + finally + { + _gate.Release(); + } + + Changed?.Invoke(this, EventArgs.Empty); + return config; + } + + /// + public async Task UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(config); + + bool replaced; + + await _gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + var records = LoadRecords(); + var index = records.FindIndex(r => string.Equals(r.Id, config.Id, StringComparison.OrdinalIgnoreCase)); + + if (index < 0) + { + return false; + } + + records[index] = ToRecord(config); + replaced = true; + + await SaveAsync(records, cancellationToken).ConfigureAwait(false); + } + finally + { + _gate.Release(); + } + + Changed?.Invoke(this, EventArgs.Empty); + return replaced; + } + + /// + public async Task RemoveAsync(string id, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(id); + + bool removed; + + await _gate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + var records = LoadRecords(); + removed = records.RemoveAll(r => string.Equals(r.Id, id, StringComparison.OrdinalIgnoreCase)) > 0; + + if (!removed) + { + return false; + } + + await SaveAsync(records, cancellationToken).ConfigureAwait(false); + } + finally + { + _gate.Release(); + } + + Changed?.Invoke(this, EventArgs.Empty); + return removed; + } + + /// + public void Dispose() => _gate.Dispose(); + + private static PatternSourceRecord ToRecord(PatternSourceConfig config) => + new( + config.Id, + config.Name, + config.BaseUrl.AbsoluteUri, + config.MinLength, + config.MaxLength, + config.Alphabet.ToString(), + config.CustomAlphabet, + config.Extension, + config.AllowDirectConnection + ); + + private static PatternSourceConfig? ToConfig(PatternSourceRecord record) + { + if (!Enum.TryParse(record.Alphabet, ignoreCase: true, out var alphabet)) + { + return null; + } + + return + PatternSourceConfig.TryCreate( + record.Name, + record.BaseUrl, + record.MinLength, + record.MaxLength, + alphabet, + record.CustomAlphabet, + record.Extension, + record.AllowDirectConnection, + out var config, + record.Id + ) == PatternConfigError.None + ? config + : null; + } + + private void EnsureLoaded() + { + if (_records is not null) + { + return; + } + + LoadRecords(); + } + + private List LoadRecords() + { + if (_records is not null) + { + return _records; + } + + List records; + + try + { + records = File.Exists(_paths.UserSourcesFile) + ? JsonSerializer.Deserialize( + File.ReadAllText(_paths.UserSourcesFile), + UserSourceJsonContext.Default.ListPatternSourceRecord + ) ?? [] + : []; + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or JsonException) + { + _logger.LogWarning(ex, "Could not read {Path}; starting with an empty source list", _paths.UserSourcesFile); + records = []; + } + + Cache(records); + return records; + } + + private void Cache(List records) + { + var configs = new List(records.Count); + foreach (var record in records) + { + var config = ToConfig(record); + if (config is null) + { + _logger.LogWarning("Skipping unusable stored source {Id}", record.Id); + continue; + } + + configs.Add(config); + } + + lock (_cacheLock) + { + _records = records; + _configs = configs; + } + } + + private async Task SaveAsync(List records, CancellationToken cancellationToken) + { + Cache(records); + + try + { + Directory.CreateDirectory(Path.GetDirectoryName(_paths.UserSourcesFile)!); + + var temp = _paths.UserSourcesFile + ".tmp"; + var json = JsonSerializer.Serialize(records, UserSourceJsonContext.Default.ListPatternSourceRecord); + + await File.WriteAllTextAsync(temp, json, cancellationToken).ConfigureAwait(false); + File.Move(temp, _paths.UserSourcesFile, overwrite: true); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + _logger.LogWarning(ex, "Could not write {Path}", _paths.UserSourcesFile); + } + } +} diff --git a/src/AvParser.Infrastructure/Collecting/OwnServiceSource.cs b/src/AvParser.Infrastructure/Collecting/OwnServiceSource.cs deleted file mode 100644 index 43a4230..0000000 --- a/src/AvParser.Infrastructure/Collecting/OwnServiceSource.cs +++ /dev/null @@ -1,293 +0,0 @@ -using System.Globalization; -using System.Runtime.CompilerServices; -using System.Text.Json; -using AvParser.Core.Collecting; -using AvParser.Core.Parsing; -using AvParser.Core.Settings; -using AvParser.Infrastructure.Proxies; -using Microsoft.Extensions.Logging; - -namespace AvParser.Infrastructure.Collecting; - -/// -/// Lists what a service you run holds, by asking it. -/// -/// -/// -/// A service that knows its own contents can simply say so, which is why this is a listing client -/// and not a crawler: one request per page returns exactly what is there, in order, with metadata, -/// and stops when the pages run out. -/// -/// -/// The expected response is deliberately loose, because the service on the other end is the user's -/// own and should not have to be rewritten to match us. Either of these works: -/// -/// -/// { "items": [ { "url": "...", "id": "...", "name": "...", "published": "...", "size": 1234, -/// "tags": ["a"] } ], "next": "cursor" } -/// [ "https://host/one.png", "https://host/two.gif" ] -/// -/// -/// Paging follows next until it is absent. A page that repeats a cursor stops the walk -/// rather than looping for ever. -/// -/// -public sealed class OwnServiceSource( - IProxiedHttpClientFactory clients, - ISettingsService settings, - ILogger logger -) : IMediaSource -{ - /// Stops a service that keeps handing back pages from running the collector for ever. - private const int MaxPages = 10_000; - - private readonly IProxiedHttpClientFactory _clients = clients ?? throw new ArgumentNullException(nameof(clients)); - private readonly ISettingsService _settings = settings ?? throw new ArgumentNullException(nameof(settings)); - private readonly ILogger _logger = logger ?? throw new ArgumentNullException(nameof(logger)); - - /// - public string Id => "own-service"; - - /// - public string DisplayName => "Own service"; - - /// - public string Description => "Reads the listing endpoint of a service you run."; - - /// - public bool RequiresNetwork => true; - - /// - public bool CanParse(MediaQuery input) => - input?.Endpoint is { IsAbsoluteUri: true } endpoint && endpoint.Scheme is "http" or "https"; - - /// - public async IAsyncEnumerable> ParseAsync( - MediaQuery input, - IProgress? progress, - [EnumeratorCancellation] CancellationToken cancellationToken - ) - { - ArgumentNullException.ThrowIfNull(input); - - if (!CanParse(input)) - { - yield return ParseOutcome.Failure( - ParseError.Create(0, "NoEndpoint", "No listing endpoint was configured.") - ); - yield break; - } - - var requireProxy = !_settings.Current.AllowDirectConnection; - - using var leased = await _clients - .LeaseAsync(HttpClientTimeouts.Default, requireProxy, cancellationToken) - .ConfigureAwait(false); - - var cursor = input.Cursor; - var seenCursors = new HashSet(StringComparer.Ordinal); - var ordinal = 0; - - for (var page = 0; page < MaxPages; page++) - { - cancellationToken.ThrowIfCancellationRequested(); - - var url = WithCursor(input.Endpoint!, cursor); - - // Read and parse inside the try, report outside it: a yield cannot live in a catch. - Page parsed = default; - ParseError? error = null; - - try - { - var body = await leased.Client.GetStringAsync(url, cancellationToken).ConfigureAwait(false); - leased.Lease?.ReportSuccess(); - - parsed = ReadPage(body, input.Endpoint!); - } - catch (Exception ex) when (ex is HttpRequestException or IOException) - { - leased.Lease?.ReportFailure(ex.Message); - _logger.LogWarning(ex, "Could not read the listing at {Url}", url); - error = ParseError.Create(ordinal, "ListingFailed", ex.Message, ex.Message); - } - catch (JsonException ex) - { - // The transport was fine; the service answered with something we cannot read. - error = ParseError.Create(ordinal, "ListingMalformed", ex.Message, ex.Message); - } - - if (error is not null) - { - yield return ParseOutcome.Failure(error); - yield break; - } - - foreach (var candidate in parsed.Items) - { - ordinal++; - - yield return ParseOutcome.Success(candidate with { SourceId = Id, Ordinal = ordinal }); - - if (input.HasLimit && ordinal >= input.Limit) - { - yield break; - } - } - - // Total is unknown until the last page, so progress stays indeterminate and only the - // running count moves. - progress?.Report(new ParseProgress(ordinal, 0)); - - if (string.IsNullOrEmpty(parsed.Next) || !seenCursors.Add(parsed.Next)) - { - yield break; - } - - cursor = parsed.Next; - } - - _logger.LogWarning("Stopped paging {Endpoint} after {Pages} pages", input.Endpoint, MaxPages); - } - - /// One page of a listing. - /// Candidates on this page. - /// Cursor for the following page, or null when this was the last. - internal readonly record struct Page(IReadOnlyList Items, string? Next); - - private static Uri WithCursor(Uri endpoint, string? cursor) - { - if (string.IsNullOrEmpty(cursor)) - { - return endpoint; - } - - var separator = string.IsNullOrEmpty(endpoint.Query) ? '?' : '&'; - - return new Uri($"{endpoint.AbsoluteUri}{separator}cursor={Uri.EscapeDataString(cursor)}"); - } - - /// - /// Reads either shape of listing. - /// - /// - /// Hand-parsed with rather than deserialised into a type: the two - /// accepted shapes and the string-or-object item would need a converter each, and being lenient - /// about a contract the user controls is the whole point. - /// - internal static Page ReadPage(string json, Uri baseAddress) - { - using var document = JsonDocument.Parse(json); - var root = document.RootElement; - - var items = new List(); - string? next = null; - - var array = root.ValueKind switch - { - JsonValueKind.Array => root, - JsonValueKind.Object when root.TryGetProperty("items", out var found) => found, - _ => default, - }; - - if (root.ValueKind == JsonValueKind.Object && root.TryGetProperty("next", out var cursor)) - { - next = cursor.ValueKind == JsonValueKind.String ? cursor.GetString() : null; - } - - if (array.ValueKind != JsonValueKind.Array) - { - return new Page(items, next); - } - - foreach (var element in array.EnumerateArray()) - { - if (ReadItem(element, baseAddress) is { } candidate) - { - items.Add(candidate); - } - } - - return new Page(items, next); - } - - private static MediaCandidate? ReadItem(JsonElement element, Uri baseAddress) - { - if (element.ValueKind == JsonValueKind.String) - { - return Resolve(element.GetString(), baseAddress) is { } bare ? new MediaCandidate(bare) : null; - } - - if (element.ValueKind != JsonValueKind.Object) - { - return null; - } - - var url = Resolve(ReadString(element, "url") ?? ReadString(element, "href"), baseAddress); - - if (url is null) - { - return null; - } - - var candidate = new MediaCandidate(url) - { - ExternalId = ReadString(element, "id"), - SuggestedName = ReadString(element, "name") ?? ReadString(element, "filename"), - }; - - if ( - ReadString(element, "published") is { } published - && DateTimeOffset.TryParse( - published, - CultureInfo.InvariantCulture, - DateTimeStyles.RoundtripKind, - out var when - ) - ) - { - candidate = candidate with { PublishedUtc = when }; - } - - if (element.TryGetProperty("size", out var size) && size.TryGetInt64(out var length)) - { - candidate = candidate with { ExpectedLength = length }; - } - - if (element.TryGetProperty("tags", out var tags) && tags.ValueKind == JsonValueKind.Array) - { - candidate = candidate with - { - Tags = - [ - .. tags.EnumerateArray() - .Where(tag => tag.ValueKind == JsonValueKind.String) - .Select(tag => tag.GetString()!), - ], - }; - } - - return candidate; - } - - private static string? ReadString(JsonElement element, string name) => - element.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.String - ? value.GetString() - : null; - - /// Resolves a listed address, allowing relative paths against the endpoint. - private static Uri? Resolve(string? value, Uri baseAddress) - { - if (string.IsNullOrWhiteSpace(value)) - { - return null; - } - - if (!Uri.TryCreate(baseAddress, value.Trim(), out var resolved)) - { - return null; - } - - return resolved.Scheme is "http" or "https" ? resolved : null; - } -} diff --git a/src/AvParser.Infrastructure/DependencyInjection/InfrastructureServiceCollectionExtensions.cs b/src/AvParser.Infrastructure/DependencyInjection/InfrastructureServiceCollectionExtensions.cs index eae9626..8bd15cb 100644 --- a/src/AvParser.Infrastructure/DependencyInjection/InfrastructureServiceCollectionExtensions.cs +++ b/src/AvParser.Infrastructure/DependencyInjection/InfrastructureServiceCollectionExtensions.cs @@ -1,5 +1,4 @@ using AvParser.Core.Collecting; -using AvParser.Core.DependencyInjection; using AvParser.Core.Proxies; using AvParser.Core.Settings; using AvParser.Infrastructure.Collecting; @@ -94,13 +93,12 @@ public static class InfrastructureServiceCollectionExtensions services.AddSingleton(); services.AddSingleton(); - services.AddSingleton(); - - // Resolves sources from both assemblies: the container gathers every IMediaSource - // registration regardless of which project declared it. + // Sources are user-added data now: the store owns the persisted list, and the catalog + // materialises one source per stored config and tracks the store for edits. + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); services.AddSingleton(sp => new MediaSourceCatalog( - sp.GetServices(), - CoreServiceCollectionExtensions.DefaultMediaSourceId + sp.GetRequiredService() )); return services; diff --git a/src/AvParser.Infrastructure/Proxies/ProxyPoolLoader.cs b/src/AvParser.Infrastructure/Proxies/ProxyPoolLoader.cs index 11d0b6a..672cfcb 100644 --- a/src/AvParser.Infrastructure/Proxies/ProxyPoolLoader.cs +++ b/src/AvParser.Infrastructure/Proxies/ProxyPoolLoader.cs @@ -22,24 +22,49 @@ public interface IProxyPoolLoader /// Whether the initial load has finished. bool IsLoaded { get; } + /// Whether the background pass over the unchecked remainder is still running. + bool IsToppingUp { get; } + /// Writes the current pool state so the next launch can start from it. Task SaveStateAsync(CancellationToken cancellationToken = default); } /// public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore, ILogger logger) - : IProxyPoolLoader + : IProxyPoolLoader, + IDisposable { + /// + /// Share of the sweep concurrency the background pass is allowed. + /// + /// + /// A manual sweep is a burst the user asked for and is watching; this one runs for minutes, + /// unwatched, possibly while a collection is downloading. A quarter is enough to finish a few + /// thousand addresses in a few minutes without competing with the thing the user actually + /// started. + /// + private const int BackgroundProbeShare = 4; + private readonly IProxyPool _pool = pool ?? throw new ArgumentNullException(nameof(pool)); private readonly IProxyStateStore _stateStore = stateStore ?? throw new ArgumentNullException(nameof(stateStore)); private readonly ILogger _logger = logger ?? throw new ArgumentNullException(nameof(logger)); private readonly Lock _gate = new(); + private readonly CancellationTokenSource _shutdown = new(); + private readonly SemaphoreSlim _saving = new(1, 1); private Task? _load; + private Task? _topUp; + private int _disposed; /// public bool IsLoaded => _load is { IsCompleted: true }; + /// + public bool IsToppingUp => _topUp is { IsCompleted: false }; + + /// The background pass, so tests can await it instead of polling a flag. + internal Task? TopUp => _topUp; + /// public Task EnsureLoadedAsync() { @@ -59,8 +84,23 @@ public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore } /// - public Task SaveStateAsync(CancellationToken cancellationToken = default) => - _stateStore.SaveAsync(_pool.Entries, cancellationToken); + /// + /// Serialised: the background pass saves when it finishes and the app saves on shutdown, and + /// both writes go through the same temporary file before being renamed into place. + /// + public async Task SaveStateAsync(CancellationToken cancellationToken = default) + { + await _saving.WaitAsync(cancellationToken).ConfigureAwait(false); + + try + { + await _stateStore.SaveAsync(_pool.Entries, cancellationToken).ConfigureAwait(false); + } + finally + { + _saving.Release(); + } + } private async Task LoadAsync() { @@ -97,6 +137,8 @@ public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore await SaveStateAsync(CancellationToken.None).ConfigureAwait(false); + StartTopUp(options); + return new ProxyPoolLoadResult(total, live, restored); } catch (Exception ex) @@ -107,4 +149,69 @@ public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore return new ProxyPoolLoadResult(_pool.Entries.Count, _pool.LiveCount, 0); } } + + /// + /// Finishes checking whatever the warm-up skipped, in the background. + /// + /// + /// The warm-up stops at , so on a feed of a few + /// thousand it leaves almost everything unknown — which reads as "nothing was checked" and, worse, + /// means the next collection has a pool of ten instead of the hundreds that actually answer. Not + /// awaited: startup is over by now, and nobody should wait for this. Skipped under + /// , where the user has asked for no sweeping at all. + /// + private void StartTopUp(ProxyOptions options) + { + if (options.HealthCheck != ProxyHealthCheck.Pool || _shutdown.IsCancellationRequested) + { + return; + } + + var concurrency = Math.Max(1, options.ProbeConcurrency / BackgroundProbeShare); + + _topUp = Task.Run( + async () => + { + try + { + var found = await _pool.TopUpAsync(concurrency, null, _shutdown.Token).ConfigureAwait(false); + + _logger.LogInformation( + "Background proxy check finished: {Found} more live, {Live} live of {Total}", + found, + _pool.LiveCount, + _pool.Entries.Count + ); + + // Everything just learned is worth carrying to the next launch; without this the + // pass would be repeated from scratch every time. Uncancellable on purpose: a + // write stopped halfway leaves a stray temp file and saves nothing. + await SaveStateAsync(CancellationToken.None).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + // The app is closing; what was probed keeps its verdict. + } + catch (Exception ex) + { + _logger.LogWarning(ex, "The background proxy check stopped early"); + } + }, + CancellationToken.None + ); + } + + /// + /// Stops the background pass; the container disposes this on shutdown. + public void Dispose() + { + if (Interlocked.Exchange(ref _disposed, 1) == 1) + { + return; + } + + _shutdown.Cancel(); + _shutdown.Dispose(); + _saving.Dispose(); + } } diff --git a/src/AvParser.Infrastructure/Storage/AppPaths.cs b/src/AvParser.Infrastructure/Storage/AppPaths.cs index 6bdf0bf..4020921 100644 --- a/src/AvParser.Infrastructure/Storage/AppPaths.cs +++ b/src/AvParser.Infrastructure/Storage/AppPaths.cs @@ -21,6 +21,13 @@ public interface IAppPaths /// Separate from the custom list: this one is derived data the app rewrites itself. string ProxyStateFile { get; } + /// Full path of the user's own media-source list. + /// + /// A default member computed from rather than a constructor-set one, + /// so adding it does not break the test doubles that implement this interface by hand. + /// + string UserSourcesFile => Path.Combine(DataDirectory, "sources.user.json"); + /// Directory holding rolling log files. string LogDirectory { get; } diff --git a/src/AvParser.UI/DependencyInjection/UiServiceCollectionExtensions.cs b/src/AvParser.UI/DependencyInjection/UiServiceCollectionExtensions.cs index 53bfc3d..c310a60 100644 --- a/src/AvParser.UI/DependencyInjection/UiServiceCollectionExtensions.cs +++ b/src/AvParser.UI/DependencyInjection/UiServiceCollectionExtensions.cs @@ -36,7 +36,12 @@ public static class UiServiceCollectionExtensions services.AddSingleton(); services.AddSingleton(); - services.AddSingleton(); + services.AddSingleton(static sp => new DashboardViewModel( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService(), + sp + )); services.AddSingleton(static sp => new SettingsViewModel( sp.GetRequiredService(), sp.GetRequiredService(), diff --git a/src/AvParser.UI/Localization/Strings.resx b/src/AvParser.UI/Localization/Strings.resx index 0961f6b..eaf0ea8 100644 --- a/src/AvParser.UI/Localization/Strings.resx +++ b/src/AvParser.UI/Localization/Strings.resx @@ -88,6 +88,30 @@ DATA DIRECTORY + + Proxy pool + + + KNOWN + + + LIVE + + + SIDELINED + + + UNCHECKED + + + Average response: {0} ms + + + Requests through a proxy: {0} of {1} worked ({2}) + + + The pool is empty. Add your own addresses, or switch the feed on in settings. + POOL @@ -385,11 +409,8 @@ {0} in the pool, {1} live. - - Allow network sources to run without a proxy - - - When off, a source that fetches anything refuses to start until at least one proxy answers. Sources that only read pasted text are never blocked. + + Whether a source may run without a proxy is now set per source, in the source editor on the Collect page. LIVE PROXIES TO FIND AT STARTUP @@ -400,8 +421,11 @@ Collect - - SOURCE + + SOURCES + + + Tick every source the run should cover; they run side by side. ACTIONS @@ -415,17 +439,107 @@ Clear the results - - ADDRESSES + + TARGET - - One address per line. Lines starting with '#' are ignored. + + ATTEMPTS - - LISTING ENDPOINT + + Per source. 0 — until you stop it. - - https://my-service.local/api/list + + 0 — no limit. + + + Add source + + + Edit source + + + Delete source + + + No sources yet. Add one to start collecting. + + + SOURCE + + + NAME + + + My image service + + + BASE URL + + + https://imgtest.example/test1/ + + + MIN LENGTH + + + MAX LENGTH + + + May run without a proxy + + + Off by default: with no live proxy this source refuses to start rather than fetching from your own address. Turn it on for a host you own, or one you do not mind being seen from here. + + + EXTENSION + + + ID CHARACTERS + + + CUSTOM CHARACTERS + + + The exact characters ids may use + + + Save + + + Cancel + + + Give the source a name. + + + The base URL must be an absolute http or https address. + + + The id length range must be between 1 and 64, with min no greater than max. + + + The chosen characters resolve to an empty set. + + + Lower-case letters (a–z) + + + Upper-case letters (A–Z) + + + Letters (a–z, A–Z) + + + Digits (0–9) + + + Letters and digits + + + Hexadecimal (0–9, a–f) + + + Custom… Fetch everything again @@ -436,8 +550,41 @@ COLLECTED - - ERRORS + + LOG + + + Run started: {0}, attempts {1}, target {2} + + + started + + + out of candidates + + + Run finished. {0} + + + Run stopped. {0} + + + no limit + + + Copy + + + Copy the whole log + + + {0} source + + + {0} sources + + + {0} sources This source needs a working proxy and none is live. Check the proxy list, or allow direct connections in settings. @@ -493,30 +640,9 @@ {0} errors - - URL list - - - Addresses you paste in, one per line. - - - Own service - - - Reads the listing endpoint of a service you run. - '{0}' is not an http or https address. - - No listing endpoint was given. - - - Could not read the listing: {0} - - - The listing could not be understood: {0} - The source stopped: {0} diff --git a/src/AvParser.UI/Localization/Strings.ru.resx b/src/AvParser.UI/Localization/Strings.ru.resx index 6ade22e..59739c2 100644 --- a/src/AvParser.UI/Localization/Strings.ru.resx +++ b/src/AvParser.UI/Localization/Strings.ru.resx @@ -88,6 +88,30 @@ КАТАЛОГ ДАННЫХ + + Пул прокси + + + ВСЕГО + + + ЖИВЫХ + + + В КАРАНТИНЕ + + + НЕ ПРОВЕРЕНО + + + Средний отклик: {0} мс + + + Запросов через прокси: {0} из {1} удались ({2}) + + + Пул пуст. Добавьте свои адреса или включите фид в настройках. + ПУЛ @@ -385,11 +409,8 @@ В пуле {0}, живых {1}. - - Разрешить сетевым источникам работать без прокси - - - Когда выключено, источник, который куда-то ходит, не запустится, пока не ответит хотя бы одна прокси. Источники, читающие вставленный текст, не блокируются никогда. + + Разрешение работать без прокси теперь задаётся у каждого источника отдельно — в редакторе источника на странице «Сбор». СКОЛЬКО ЖИВЫХ ИСКАТЬ ПРИ СТАРТЕ @@ -400,8 +421,11 @@ Сбор - - ИСТОЧНИК + + ИСТОЧНИКИ + + + Отметьте источники для запуска — они работают одновременно. ДЕЙСТВИЯ @@ -415,17 +439,107 @@ Очистить результаты - - АДРЕСА + + ЦЕЛЬ - - По одному адресу в строке. Строки, начинающиеся с «#», игнорируются. + + ПОПЫТКИ - - АДРЕС ЛИСТИНГА + + На каждый источник. 0 — пока не остановите. - - https://my-service.local/api/list + + 0 — без ограничения. + + + Добавить источник + + + Изменить источник + + + Удалить источник + + + Источников пока нет. Добавьте, чтобы начать сбор. + + + ИСТОЧНИК + + + НАЗВАНИЕ + + + Мой сервис картинок + + + БАЗОВЫЙ URL + + + https://imgtest.example/test1/ + + + МИН. ДЛИНА + + + МАКС. ДЛИНА + + + Может работать без прокси + + + По умолчанию выключено: без живой прокси источник не запустится, а не пойдёт с вашего адреса. Включайте для своего хоста или там, где вас не смущает быть видимым отсюда. + + + РАСШИРЕНИЕ + + + СИМВОЛЫ ID + + + СВОИ СИМВОЛЫ + + + Точный набор символов для id + + + Сохранить + + + Отмена + + + Задайте название источника. + + + Базовый URL должен быть абсолютным адресом http или https. + + + Длина id должна быть от 1 до 64, причём минимум не больше максимума. + + + Выбранные символы дают пустой набор. + + + Строчные буквы (a–z) + + + Прописные буквы (A–Z) + + + Буквы (a–z, A–Z) + + + Цифры (0–9) + + + Буквы и цифры + + + Шестнадцатеричные (0–9, a–f) + + + Свои… Скачать всё заново @@ -436,8 +550,41 @@ СОБРАНО - - ОШИБКИ + + ЖУРНАЛ + + + Запуск: {0}, попыток {1}, цель {2} + + + запущен + + + кандидаты закончились + + + Готово. {0} + + + Остановлено. {0} + + + без ограничения + + + Копировать + + + Копировать весь журнал + + + {0} источник + + + {0} источника + + + {0} источников Этому источнику нужна рабочая прокси, а живых нет. Проверьте список прокси или разрешите прямое подключение в настройках. @@ -493,30 +640,9 @@ {0} ошибок - - Список ссылок - - - Адреса, которые вы вставляете сами, по одному в строке. - - - Свой сервис - - - Читает листинг сервиса, который вы держите сами. - «{0}» — не http- и не https-адрес. - - Не указан адрес листинга. - - - Не удалось прочитать листинг: {0} - - - Листинг не удалось разобрать: {0} - Источник остановился: {0} diff --git a/src/AvParser.UI/Styles/Icons.axaml b/src/AvParser.UI/Styles/Icons.axaml index c28410a..99be998 100644 --- a/src/AvParser.UI/Styles/Icons.axaml +++ b/src/AvParser.UI/Styles/Icons.axaml @@ -59,6 +59,10 @@ M9 3h6l1 1h4v2H4V4h4l1-1zM6 7h12l-1 13H7L6 7z + + M3 17.25V21h3.75L17.81 9.94l-3.75-3.75L3 17.25zM20.71 7.04a1 1 0 0 0 0-1.41l-2.34-2.34a1 1 0 0 0-1.41 0l-1.83 1.83 3.75 3.75 1.83-1.83z + + M12 2.5 13.9 9l6.6 1.9-6.6 1.9L12 19.4l-1.9-6.6L3.5 11 10.1 9 12 2.5z diff --git a/src/AvParser.UI/ViewModels/CollectErrorViewModel.cs b/src/AvParser.UI/ViewModels/CollectErrorViewModel.cs deleted file mode 100644 index f2ebf84..0000000 --- a/src/AvParser.UI/ViewModels/CollectErrorViewModel.cs +++ /dev/null @@ -1,40 +0,0 @@ -using AvParser.Core.Parsing; -using AvParser.UI.Localization; -using ReactiveUI; - -namespace AvParser.UI.ViewModels; - -/// One row of the collect error list, with its message translated. -/// -/// Sources and the fetcher are part of the domain and produce English text plus a code. This resolves -/// Collect.Error.{Code} and falls back to the domain's own wording, so a code that has not -/// been translated yet still says something useful instead of showing a missing-key marker. -/// -public sealed class CollectErrorViewModel(ParseError error) : ReactiveObject -{ - /// The underlying error. - public ParseError Error { get; } = error ?? throw new ArgumentNullException(nameof(error)); - - /// 1-based position of the offending item: a pasted line, or a place in a listing. - public int Index => Error.Index; - - /// Translated message. - public string Text - { - get - { - if (Error.Code is not { Length: > 0 } code) - { - return Error.Message; - } - - var template = Localizer.Instance.GetOrDefault($"Collect.Error.{code}", Error.Message); - return Error.Arguments.Count == 0 - ? template - : string.Format(Localizer.Instance.Culture, template, [.. Error.Arguments]); - } - } - - /// Re-reads the translated message. - public void Refresh() => this.RaisePropertyChanged(nameof(Text)); -} diff --git a/src/AvParser.UI/ViewModels/CollectLogEntryViewModel.cs b/src/AvParser.UI/ViewModels/CollectLogEntryViewModel.cs new file mode 100644 index 0000000..aaba383 --- /dev/null +++ b/src/AvParser.UI/ViewModels/CollectLogEntryViewModel.cs @@ -0,0 +1,171 @@ +using System.Globalization; +using AvParser.Core.Collecting; +using AvParser.Core.Parsing; +using AvParser.UI.Localization; +using ReactiveUI; + +namespace AvParser.UI.ViewModels; + +/// How loud one log line is, which is also how it is coloured. +public enum CollectLogLevel +{ + /// Lifecycle: a run started, a source finished. + Info = 0, + + /// Something was collected. + Success = 1, + + /// Nothing was collected, but nothing went wrong either: a duplicate or a skip. + Warning = 2, + + /// A failure. The run carries on regardless. + Error = 3, +} + +/// +/// One line of the live collection log. +/// +/// +/// Holds the resource key and its arguments rather than the finished sentence: switching language +/// mid-run must not leave a page of half-English text behind. The detail half — an address, a size — +/// is literal and stays as it is, because it never was English. +/// +public sealed class CollectLogEntryViewModel : ReactiveObject +{ + private readonly string _key; + private readonly string? _fallback; + private readonly object?[] _arguments; + private readonly string? _detail; + + private CollectLogEntryViewModel( + CollectLogLevel level, + string key, + string? fallback, + object?[] arguments, + string? detail, + string? source + ) + { + Level = level; + Source = source; + Time = DateTime.Now; + + _key = key; + _fallback = fallback; + _arguments = arguments; + _detail = detail; + } + + /// Severity of the line. + public CollectLogLevel Level { get; } + + /// Name of the source it came from; for whole-run lines. + public string? Source { get; } + + /// When it happened, on the wall clock. + public DateTime Time { get; } + + /// Timestamp, seconds precision — a log with milliseconds is unreadable at this rate. + public string TimeText => Time.ToString("HH:mm:ss", CultureInfo.InvariantCulture); + + /// Single-character severity marker. Symbols rather than words, so nothing to translate. + public string LevelText => + Level switch + { + CollectLogLevel.Success => "+", + CollectLogLevel.Warning => "~", + CollectLogLevel.Error => "×", + _ => "·", + }; + + /// Whether this line reports a failure; drives the red chip. + public bool IsError => Level == CollectLogLevel.Error; + + /// Whether this line reports something collected; drives the green chip. + public bool IsSuccess => Level == CollectLogLevel.Success; + + /// Whether there is a source to name on the line. + public bool HasSource => !string.IsNullOrEmpty(Source); + + /// The translated message, with its literal detail appended. + public string Text + { + get + { + var template = _fallback is null + ? Localizer.Instance[_key] + : Localizer.Instance.GetOrDefault(_key, _fallback); + + var message = + _arguments.Length == 0 ? template : string.Format(Localizer.Instance.Culture, template, _arguments); + + return _detail is null ? message : $"{message} {_detail}"; + } + } + + /// Re-reads the translated half of the line. + public void Refresh() => this.RaisePropertyChanged(nameof(Text)); + + /// + /// The line as one string: what a copy puts on the clipboard. + /// + /// + /// Built here rather than in the view so the copied text is the line the user is looking at — + /// same timestamp, same source, same wording — instead of a re-render that quietly drifts from + /// the template. The severity marker is left out: it is a colour cue, not information. + /// + public override string ToString() => HasSource ? $"{TimeText} {Source} {Text}" : $"{TimeText} {Text}"; + + /// Joins lines for the clipboard, in the order they were logged. + public static string ToText(IEnumerable entries) + { + ArgumentNullException.ThrowIfNull(entries); + + return string.Join(Environment.NewLine, entries.Select(entry => entry.ToString())); + } + + /// A lifecycle line: run started, source finished, and so on. + public static CollectLogEntryViewModel Message( + CollectLogLevel level, + string key, + string? source = null, + params object?[] arguments + ) => new(level, key, null, arguments, null, source); + + /// A line for one collected, duplicated or skipped item. + public static CollectLogEntryViewModel ForItem(CollectedItem item, string? source) + { + ArgumentNullException.ThrowIfNull(item); + + var level = item.Status switch + { + CollectStatus.Stored => CollectLogLevel.Success, + _ => CollectLogLevel.Warning, + }; + + var size = CollectedItemViewModel.FormatSize(item.Blob.Length); + var detail = size.Length == 0 ? item.Candidate.Url.AbsoluteUri : $"{item.Candidate.Url.AbsoluteUri} {size}"; + + return new CollectLogEntryViewModel(level, $"Collect.Item.{item.Status}", null, [], detail, source); + } + + /// A line for one failure, translated by code with the domain's wording as fallback. + /// + /// The domain produces English text plus a code; this resolves Collect.Error.{Code} and + /// falls back to , so a code nobody has translated yet still says + /// something useful instead of showing a missing-key marker. + /// + public static CollectLogEntryViewModel ForError(ParseError error, string? source) + { + ArgumentNullException.ThrowIfNull(error); + + return new CollectLogEntryViewModel( + CollectLogLevel.Error, + $"Collect.Error.{error.Code}", + error.Message, + [.. error.Arguments], + error.Subject, + source + ); + } +} diff --git a/src/AvParser.UI/ViewModels/CollectViewModel.cs b/src/AvParser.UI/ViewModels/CollectViewModel.cs index d7b41c4..401bfe3 100644 --- a/src/AvParser.UI/ViewModels/CollectViewModel.cs +++ b/src/AvParser.UI/ViewModels/CollectViewModel.cs @@ -1,7 +1,12 @@ +using System.Collections.Concurrent; using System.Collections.ObjectModel; +using System.ComponentModel; using System.Diagnostics; using System.Globalization; +using System.Runtime.CompilerServices; +using System.Threading.Channels; using AvParser.Core.Collecting; +using AvParser.Core.Collecting.Sources; using AvParser.Core.Parsing; using AvParser.Core.Proxies; using AvParser.Core.Settings; @@ -34,6 +39,25 @@ public partial class CollectViewModel : PageViewModel, IDisposable /// private const int MaxDisplayedItems = 20_000; + /// + /// Log lines kept on screen; older ones fall off the top. + /// + /// + /// A run with no attempt budget has no end, so an uncapped log is a memory leak with a scroll + /// bar. This is a tail, not an archive: Serilog still has the whole run. + /// + private const int MaxLogEntries = 2000; + + /// + /// How often buffered rows and log lines are pushed to the UI while a run is going. + /// + /// + /// Batching alone cannot make a log live: with a slow source the buffer sits below + /// and the page looks stopped until the next item happens to arrive. + /// The tick is what makes "in real time" true rather than "in real time, eventually". + /// + private static readonly TimeSpan FlushInterval = TimeSpan.FromMilliseconds(200); + private readonly IMediaSourceCatalog _catalog; private readonly ISettingsService _settings; private readonly IProxyPool _proxyPool; @@ -44,22 +68,48 @@ public partial class CollectViewModel : PageViewModel, IDisposable private readonly ILogger _logger; private readonly ISequencer _mainThread; private readonly ObservableAsPropertyHelper _isBusy; + private readonly ObservableAsPropertyHelper _showCustomAlphabet; private readonly Signal _proxyChanged = new(); + // Filled from the run loop and drained by the flush; a queue rather than a list because the + // periodic flush runs on a timer thread while the loop is still writing. + private readonly ConcurrentQueue _itemBuffer = new(); + private readonly ConcurrentQueue _logBuffer = new(); + + /// Serialises flushes so two of them cannot interleave the log out of order. + private readonly Lock _flushGate = new(); + private CancellationTokenSource? _cancellation; - /// Addresses pasted by the user, one per line. - [Reactive] - public partial string InputText { get; set; } + private int _disposed; - /// Listing endpoint, for sources that ask a service what it holds. - [Reactive] - public partial string EndpointText { get; set; } + /// Which source to select after the next catalog rebuild; null keeps the current one. + private string? _pendingSelectId; - /// Source applied by . + /// Which sources to tick after the first build; null keeps whatever is ticked now. + private HashSet? _pendingCheckedIds; + + /// Source applied by ; when none exist. [Reactive] - public partial MediaSourceViewModel SelectedSource { get; set; } + public partial MediaSourceViewModel? SelectedSource { get; set; } + + /// How many items to stop after collecting; 0 collects until the budget runs out. + [Reactive] + public partial int TargetCount { get; set; } + + /// How many candidate ids each source may try; 0 runs until the user stops it. + /// + /// Per source, not per run: two ticked sources with a budget of a thousand try a thousand ids + /// each. Zero on both this and is a run with no end, which is the + /// point — a scan of an id space is finished when the user says so. + /// + [Reactive] + public partial int AttemptBudget { get; set; } + + /// How many sources are ticked to run. + [Reactive] + public partial int RunnableCount { get; private set; } /// Completion of the running collection, 0.0 to 1.0. [Reactive] @@ -84,26 +134,56 @@ public partial class CollectViewModel : PageViewModel, IDisposable /// /// Whether the selected source needs the network but has no working proxy to use. /// - /// - /// Only network sources are gated. A source that reads addresses the user pasted has nothing to - /// route, and blocking it would make the app unusable whenever the public lists are down. - /// [Reactive] public partial bool IsBlockedWithoutProxy { get; set; } + // ----- Source editor ----- + + /// Whether the add/edit form is showing. + [Reactive] + public partial bool IsEditorOpen { get; set; } + + /// Id being edited; when adding a new source. + [Reactive] + public partial string? EditingId { get; set; } + + /// Editor field: display name. + [Reactive] + public partial string EditorName { get; set; } + + /// Editor field: base URL the ids hang off. + [Reactive] + public partial string EditorBaseUrl { get; set; } + + /// Editor field: shortest id length. + [Reactive] + public partial int EditorMinLength { get; set; } + + /// Editor field: longest id length. + [Reactive] + public partial int EditorMaxLength { get; set; } + + /// Editor field: which characters an id is drawn from. + [Reactive] + public partial LocalizedOption EditorAlphabet { get; set; } + + /// Editor field: characters for a custom alphabet. + [Reactive] + public partial string EditorCustomAlphabet { get; set; } + + /// Editor field: extension appended after the id, such as .jpg. + [Reactive] + public partial string EditorExtension { get; set; } + + /// Editor field: whether this source may run without a proxy. + [Reactive] + public partial bool EditorAllowDirect { get; set; } + + /// Validation message shown in the editor; when valid. + [Reactive] + public partial string? EditorError { get; set; } + /// Creates the page. - /// Available sources. - /// Used to remember the selected source. - /// Consulted for the live count that gates network sources. - /// Runs the collection. - /// Consulted for totals, and asked to purge. - /// Decodes row previews. - /// Resolves the navigation service lazily, to keep pages acyclic. - /// Diagnostics. - /// - /// Scheduler used to marshal collection and progress updates back to the UI thread. Tests pass - /// to make everything synchronous. - /// public CollectViewModel( IMediaSourceCatalog catalog, ISettingsService settings, @@ -126,10 +206,27 @@ public partial class CollectViewModel : PageViewModel, IDisposable _logger = logger ?? throw new ArgumentNullException(nameof(logger)); _mainThread = mainThread ?? RxSchedulers.MainThreadScheduler; - InputText = string.Empty; - EndpointText = string.Empty; - Sources = [.. catalog.Sources.Select(source => new MediaSourceViewModel(source))]; - SelectedSource = Sources.First(source => source.Id == catalog.FindOrDefault(settings.Current.LastSourceId).Id); + AlphabetOptions = LocalizedOption.ForAll(); + EditorName = string.Empty; + EditorBaseUrl = string.Empty; + EditorCustomAlphabet = string.Empty; + EditorExtension = string.Empty; + EditorMinLength = 6; + EditorMaxLength = 8; + EditorAlphabet = OptionFor(IdAlphabet.Alphanumeric); + + TargetCount = settings.Current.CollectTargetCount; + AttemptBudget = settings.Current.CollectAttemptBudget; + + Sources = []; + _pendingSelectId = settings.Current.LastSourceId; + _pendingCheckedIds = new HashSet( + AppSettings.SplitSourceIds(settings.Current.CollectSourceIds), + StringComparer.OrdinalIgnoreCase + ); + RebuildSources(); + + _catalog.Changed += OnCatalogChanged; // The pool changes on every lease outcome and on every probe, so coalesce before // re-evaluating whether the source is allowed to run. @@ -141,25 +238,41 @@ public partial class CollectViewModel : PageViewModel, IDisposable RefreshProxyGate(); + _showCustomAlphabet = this.WhenAnyValue(x => x.EditorAlphabet) + .Select(static option => option?.Value == IdAlphabet.Custom) + .ToProperty(this, nameof(ShowCustomAlphabet), false, _mainThread); + + // No condition on the budget: zero now means "until stopped", not "nothing to do". var canCollect = this.WhenAnyValue( - x => x.InputText, - x => x.EndpointText, - x => x.SelectedSource, + x => x.RunnableCount, x => x.IsBlockedWithoutProxy, - static (text, endpoint, source, blocked) => (text, endpoint, source, blocked) + x => x.IsEditorOpen, + static (runnable, blocked, editing) => runnable > 0 && !blocked && !editing ) - .Select(static state => !state.blocked && HasWork(state.source, state.text, state.endpoint)) .DistinctUntilChanged(); CollectCommand = ReactiveCommand.CreateFromTask(RunCollectAsync, canCollect, _mainThread); + _isBusy = CollectCommand.IsExecuting.ToProperty(this, nameof(IsBusy), false, _mainThread); + + var idle = CollectCommand.IsExecuting.Select(static running => !running); + var hasSelection = this.WhenAnyValue( + x => x.SelectedSource, + x => x.IsEditorOpen, + static (source, editing) => source is not null && !editing + ); + var canManageSelected = hasSelection.CombineLatest(idle, static (selected, notBusy) => selected && notBusy); + var canAdd = idle.CombineLatest( + this.WhenAnyValue(x => x.IsEditorOpen), + static (notBusy, editing) => notBusy && !editing + ); + var editorOpen = this.WhenAnyValue(x => x.IsEditorOpen); + GoToProxiesCommand = ReactiveCommand.Create( () => _services.GetRequiredService().NavigateTo(), outputScheduler: _mainThread ); - _isBusy = CollectCommand.IsExecuting.ToProperty(this, nameof(IsBusy), false, _mainThread); - CancelCommand = ReactiveCommand.Create(() => _cancellation?.Cancel(), CollectCommand.IsExecuting, _mainThread); ClearCommand = ReactiveCommand.Create( @@ -169,29 +282,43 @@ public partial class CollectViewModel : PageViewModel, IDisposable StatusMessage = null; Progress = 0d; }, - CollectCommand.IsExecuting.Select(static running => !running), + idle, _mainThread ); + AddSourceCommand = ReactiveCommand.Create(OpenEditorForAdd, canAdd, _mainThread); + EditSourceCommand = ReactiveCommand.Create(OpenEditorForEdit, canManageSelected, _mainThread); + SaveSourceCommand = ReactiveCommand.CreateFromTask(SaveSourceAsync, editorOpen, _mainThread); + CancelEditCommand = ReactiveCommand.Create(CloseEditor, editorOpen, _mainThread); + RemoveSourceCommand = ReactiveCommand.CreateFromTask(RemoveSourceAsync, canManageSelected, _mainThread); + + PurgeCommand = ReactiveCommand.CreateFromTask(PurgeAsync, canManageSelected, _mainThread); + // Remember the choice; the debounced settings service coalesces the writes. Switching // source can also change whether the gate applies, since only network sources are gated. this.WhenAnyValue(x => x.SelectedSource) .Where(static source => source is not null) .Subscribe(source => { - _settings.Update(current => current with { LastSourceId = source.Id }); + _settings.Update(current => current with { LastSourceId = source!.Id }); RefreshProxyGate(); }); - PurgeCommand = ReactiveCommand.CreateFromTask( - PurgeAsync, - CollectCommand.IsExecuting.Select(static running => !running), - _mainThread - ); + this.WhenAnyValue(x => x.TargetCount) + .DistinctUntilChanged() + .Subscribe(value => _settings.Update(current => current with { CollectTargetCount = Math.Max(0, value) })); + + this.WhenAnyValue(x => x.AttemptBudget) + .DistinctUntilChanged() + .Subscribe(value => + _settings.Update(current => current with { CollectAttemptBudget = Math.Max(0, value) }) + ); _settings.Changes.Subscribe(_ => RefreshProxyGate()); CollectCommand.ThrownExceptions.Subscribe(OnCommandFailed); + SaveSourceCommand.ThrownExceptions.Subscribe(OnCommandFailed); + RemoveSourceCommand.ThrownExceptions.Subscribe(OnCommandFailed); PurgeCommand.ThrownExceptions.Subscribe(OnCommandFailed); // Not awaited — a constructor cannot be — and it never throws. @@ -204,14 +331,34 @@ public partial class CollectViewModel : PageViewModel, IDisposable /// public override string IconKey => "IconDownload"; - /// Every registered source, for the picker. - public IReadOnlyList Sources { get; } + /// Every user-added source, for the picker. + public ObservableCollection Sources { get; } + + /// Whether any source exists; drives the "add a source" empty state. + public bool HasSources => Sources.Count > 0; + + /// Alphabet choices for the editor. + public IReadOnlyList> AlphabetOptions { get; } + + /// Whether the custom-alphabet field applies to the current editor choice. + public bool ShowCustomAlphabet => _showCustomAlphabet.Value; /// Collected items, capped at . public ObservableCollection Items { get; } = []; - /// Per-item failures. A failure never aborts the run. - public ObservableCollection Errors { get; } = []; + /// + /// What the run is doing, newest last, capped at . + /// + /// + /// One chronological list rather than a results list beside a failures list: during a scan of an + /// id space almost everything is a failure, and what the user is watching for is the order — + /// which addresses answered, and how quickly the misses are going by. + /// + public ObservableCollection Log { get; } = []; + + /// How many failures the log currently holds; failures never abort a run. + [Reactive] + public partial int ErrorCount { get; private set; } /// Whether a collection is currently running. public bool IsBusy => _isBusy.Value; @@ -231,53 +378,235 @@ public partial class CollectViewModel : PageViewModel, IDisposable /// Removes everything the selected source has collected. public ReactiveCommand PurgeCommand { get; } + /// Opens the editor to add a source. + public ReactiveCommand AddSourceCommand { get; } + + /// Opens the editor to change the selected source. + public ReactiveCommand EditSourceCommand { get; } + + /// Saves the editor, adding or updating the source. + public ReactiveCommand SaveSourceCommand { get; } + + /// Closes the editor without saving. + public ReactiveCommand CancelEditCommand { get; } + + /// Deletes the selected source. + public ReactiveCommand RemoveSourceCommand { get; } + /// Explains why collecting is blocked. public string ProxyRequiredMessage => Localizer.Instance["Collect.ProxyRequired"]; /// Re-evaluates the proxy gate. Exposed so tests can drive it without waiting. - public void RefreshProxyGate() - { - var settings = _settings.Current; + /// + /// Asks the ticked sources, not the highlighted one: a run that includes one gated source is + /// blocked even if the highlight happens to be resting on another. Whether a source is gated is + /// its own setting now — one host may be the user's own service, the next a stranger's. + /// + public void RefreshProxyGate() => + IsBlockedWithoutProxy = RunnableSources().Any(source => source.NeedsProxy) && _proxyPool.LiveCount == 0; - IsBlockedWithoutProxy = - SelectedSource.RequiresNetwork && !settings.AllowDirectConnection && _proxyPool.LiveCount == 0; + /// The sources a run would cover. + public IReadOnlyList RunnableSources() => [.. Sources.Where(source => source.IsSelected)]; + + private LocalizedOption OptionFor(IdAlphabet alphabet) => + AlphabetOptions.First(option => option.Value == alphabet); + + private void OnCatalogChanged(object? sender, EventArgs e) => OnUi(RebuildSources); + + private void RebuildSources() + { + var preferredId = _pendingSelectId ?? SelectedSource?.Id; + _pendingSelectId = null; + + // After the first build the ticks on screen are the truth; the stored ids only seed them. + var checkedIds = + _pendingCheckedIds + ?? new HashSet( + Sources.Where(source => source.IsSelected).Select(source => source.Id), + StringComparer.OrdinalIgnoreCase + ); + _pendingCheckedIds = null; + + foreach (var existing in Sources) + { + existing.PropertyChanged -= OnSourcePropertyChanged; + existing.Dispose(); + } + + Sources.Clear(); + foreach (var source in _catalog.Sources) + { + var row = new MediaSourceViewModel(source) { IsSelected = checkedIds.Contains(source.Id) }; + row.PropertyChanged += OnSourcePropertyChanged; + Sources.Add(row); + } + + SelectedSource = Sources.FirstOrDefault(source => source.Id == preferredId) ?? Sources.FirstOrDefault(); + + // Nothing ticked — a first run, or the last ticked source was just deleted. Tick the + // highlighted one so the page arrives ready to run rather than with the button greyed out. + if (SelectedSource is not null && !Sources.Any(source => source.IsSelected)) + { + SelectedSource.IsSelected = true; + } + + this.RaisePropertyChanged(nameof(HasSources)); + OnRunnableSourcesChanged(); } - /// Whether the source has been given enough to work with. - private static bool HasWork(MediaSourceViewModel? source, string text, string endpoint) => - source is not null - && ( - source.UsesEndpoint - ? Uri.TryCreate(endpoint, UriKind.Absolute, out var parsed) && parsed.Scheme is "http" or "https" - : !string.IsNullOrWhiteSpace(text) + private void OnSourcePropertyChanged(object? sender, PropertyChangedEventArgs e) + { + if (e.PropertyName == nameof(MediaSourceViewModel.IsSelected)) + { + OnRunnableSourcesChanged(); + } + } + + /// Remembers the ticked set, re-counts it and re-checks the gate against it. + private void OnRunnableSourcesChanged() + { + var runnable = RunnableSources(); + + RunnableCount = runnable.Count; + _settings.Update(current => + current with + { + CollectSourceIds = AppSettings.JoinSourceIds(runnable.Select(source => source.Id)), + } ); + RefreshProxyGate(); + } + + private void OpenEditorForAdd() + { + LoadEditor(null); + IsEditorOpen = true; + } + + private void OpenEditorForEdit() + { + if (SelectedSource?.Config is { } config) + { + LoadEditor(config); + IsEditorOpen = true; + } + } + + private void LoadEditor(PatternSourceConfig? config) + { + EditingId = config?.Id; + EditorName = config?.Name ?? string.Empty; + EditorBaseUrl = config?.BaseUrl.AbsoluteUri ?? string.Empty; + EditorMinLength = config?.MinLength ?? 6; + EditorMaxLength = config?.MaxLength ?? 8; + EditorAlphabet = OptionFor(config?.Alphabet ?? IdAlphabet.Alphanumeric); + EditorCustomAlphabet = config?.CustomAlphabet ?? string.Empty; + EditorExtension = config?.Extension ?? string.Empty; + EditorAllowDirect = config?.AllowDirectConnection ?? false; + EditorError = null; + } + + private void CloseEditor() + { + IsEditorOpen = false; + EditorError = null; + } + + private async Task SaveSourceAsync() + { + var result = PatternSourceConfig.TryCreate( + EditorName, + EditorBaseUrl, + EditorMinLength, + EditorMaxLength, + EditorAlphabet.Value, + EditorCustomAlphabet, + EditorExtension, + EditorAllowDirect, + out var config, + EditingId + ); + + if (result != PatternConfigError.None || config is null) + { + EditorError = Localizer.Instance[$"Collect.SourceError.{result}"]; + return; + } + + // The rebuild lands on the id just saved, so adding a source selects it. + _pendingSelectId = config.Id; + + if (EditingId is null) + { + await _catalog.AddAsync(config).ConfigureAwait(false); + } + else + { + await _catalog.UpdateAsync(config).ConfigureAwait(false); + } + + OnUi(CloseEditor); + } + + private async Task RemoveSourceAsync() + { + var id = SelectedSource?.Id; + if (id is null) + { + return; + } + + await _catalog.RemoveAsync(id).ConfigureAwait(false); + } + private void OnProxyPoolChanged(object? sender, EventArgs e) => _proxyChanged.OnNext(RxVoid.Default); private async Task RunCollectAsync(CancellationToken commandToken) { + var sources = RunnableSources(); + if (sources.Count == 0) + { + return; + } + using var cancellation = CancellationTokenSource.CreateLinkedTokenSource(commandToken); _cancellation = cancellation; - var source = SelectedSource.Source; - var query = BuildQuery(); + var query = new MediaQuery(Limit: Math.Max(0, AttemptBudget)); var options = _settings.Current.ToCollectOptions() with { ForceRefetch = ForceRefetch }; + // Each source runs its own workers, so the in-flight budget is shared out rather than + // multiplied: five sources at four downloads each would be twenty connections, not four. + var perSource = options with + { + MaxConcurrentDownloads = Math.Max(1, options.MaxConcurrentDownloads / sources.Count), + }; + + var target = TargetCount; + ClearResults(); Progress = 0d; - IsProgressIndeterminate = true; + IsProgressIndeterminate = target <= 0; StatusMessage = null; - var itemBuffer = new List(BatchSize); - var errorBuffer = new List(16); - var progress = new Progress(value => - OnUi(() => - { - IsProgressIndeterminate = value.IsIndeterminate; - Progress = value.Fraction; - }) + var loc = Localizer.Instance; + WriteLog( + CollectLogEntryViewModel.Message( + CollectLogLevel.Info, + "Collect.Log.Started", + null, + loc.Plural("Collect.Count.Sources", sources.Count), + query.HasLimit ? query.Limit : loc["Collect.Log.Unlimited"], + target > 0 ? target : loc["Collect.Log.Unlimited"] + ) ); + foreach (var source in sources) + { + WriteLog(CollectLogEntryViewModel.Message(CollectLogLevel.Info, "Collect.Log.SourceStarted", source.Name)); + } + var stopwatch = Stopwatch.StartNew(); var stored = 0; var duplicates = 0; @@ -286,14 +615,30 @@ public partial class CollectViewModel : PageViewModel, IDisposable var truncated = false; var cancelled = false; + // Nothing else pushes rows while a run is going, so the tick is what keeps the log live + // through a stretch where items arrive slower than one per batch. + var pump = new Timer(_ => SafeFlush(), null, FlushInterval, FlushInterval); + try { await foreach ( - var outcome in _runner - .RunAsync(source, query, options, progress, cancellation.Token) - .ConfigureAwait(false) + var (source, result) in MergeAsync(sources, query, perSource, cancellation.Token).ConfigureAwait(false) ) { + // No outcome means that source has run out of candidates; carried through the + // channel rather than logged where it happens, so the line lands in order. + if (result is not { } outcome) + { + WriteLog( + CollectLogEntryViewModel.Message( + CollectLogLevel.Info, + "Collect.Log.SourceFinished", + source.Name + ) + ); + continue; + } + if (outcome.IsSuccess) { var item = outcome.Value!; @@ -313,22 +658,37 @@ public partial class CollectViewModel : PageViewModel, IDisposable if (stored + duplicates + skipped <= MaxDisplayedItems) { - itemBuffer.Add(new CollectedItemViewModel(item, _thumbnails)); + _itemBuffer.Enqueue(new CollectedItemViewModel(item, _thumbnails)); } else { truncated = true; } + + WriteLog(CollectLogEntryViewModel.ForItem(item, source.Name)); + + if (target > 0 && item.Status == CollectStatus.Stored) + { + var fraction = Math.Clamp((double)stored / target, 0d, 1d); + OnUi(() => Progress = fraction); + } } else { failed++; - errorBuffer.Add(outcome.Error); + WriteLog(CollectLogEntryViewModel.ForError(outcome.Error, source.Name)); } - if (itemBuffer.Count >= BatchSize) + if (_itemBuffer.Count >= BatchSize || _logBuffer.Count >= BatchSize) { - FlushBuffers(itemBuffer, errorBuffer); + FlushBuffers(); + } + + // A satisfied target is a normal finish, not a cancellation: break out cleanly rather + // than cancelling, so the run reports "done", not "stopped". + if (target > 0 && stored >= target) + { + break; } } } @@ -338,13 +698,24 @@ public partial class CollectViewModel : PageViewModel, IDisposable } finally { + await pump.DisposeAsync().ConfigureAwait(false); _cancellation = null; - FlushBuffers(itemBuffer, errorBuffer); stopwatch.Stop(); } var summary = BuildSummary(stored, duplicates, skipped, failed, stopwatch.Elapsed, truncated, cancelled); + WriteLog( + CollectLogEntryViewModel.Message( + cancelled ? CollectLogLevel.Warning : CollectLogLevel.Info, + cancelled ? "Collect.Log.Stopped" : "Collect.Log.Finished", + null, + summary + ) + ); + + FlushBuffers(); + OnUi(() => { StatusMessage = summary; @@ -353,8 +724,8 @@ public partial class CollectViewModel : PageViewModel, IDisposable }); _logger.LogInformation( - "Collected with {Source}: {Stored} new, {Duplicates} duplicate, {Skipped} skipped, {Failed} failed in {Elapsed}", - source.Id, + "Collected with {Sources}: {Stored} new, {Duplicates} duplicate, {Skipped} skipped, {Failed} failed in {Elapsed}", + string.Join(", ", sources.Select(source => source.Id)), stored, duplicates, skipped, @@ -364,14 +735,148 @@ public partial class CollectViewModel : PageViewModel, IDisposable } /// - /// Removes what the selected source collected. + /// Runs every source at once and interleaves what they find. /// /// - /// Scoped to one source rather than emptying the store: content another source also holds - /// survives, which is exactly what the reference count in the index is for. + /// + /// Concurrent rather than one after another, and that is not a throughput preference: with no + /// attempt budget the first source never finishes, so a sequential run would be a single-source + /// run wearing a list. + /// + /// + /// The channel is bounded so a fast source cannot queue its whole listing in memory while a slow + /// one is still connecting, and the finally stops the producers before returning — + /// leaving them writing into a channel nobody reads would deadlock them mid-store. + /// /// + private async IAsyncEnumerable<(MediaSourceViewModel Source, ParseOutcome? Outcome)> MergeAsync( + IReadOnlyList sources, + MediaQuery query, + CollectOptions options, + [EnumeratorCancellation] CancellationToken cancellationToken + ) + { + using var run = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + + var channel = Channel.CreateBounded<(MediaSourceViewModel, ParseOutcome?)>( + new BoundedChannelOptions(Math.Max(BatchSize, sources.Count * 8)) { FullMode = BoundedChannelFullMode.Wait } + ); + + var producers = sources + .Select(source => Task.Run(() => ProduceAsync(source, query, options, channel.Writer, run.Token))) + .ToArray(); + + var completion = Task.Run( + async () => + { + try + { + await Task.WhenAll(producers).ConfigureAwait(false); + } + finally + { + channel.Writer.TryComplete(); + } + }, + CancellationToken.None + ); + + try + { + await foreach (var pair in channel.Reader.ReadAllAsync(cancellationToken).ConfigureAwait(false)) + { + yield return pair; + } + } + finally + { + await run.CancelAsync().ConfigureAwait(false); + + try + { + await completion.ConfigureAwait(false); + } + catch (Exception ex) when (ex is OperationCanceledException or ChannelClosedException) + { + // Expected once the run is stopped or the target has been met. + } + } + } + + /// Drives one source and writes what it produces into the shared channel. + private async Task ProduceAsync( + MediaSourceViewModel source, + MediaQuery query, + CollectOptions options, + ChannelWriter<(MediaSourceViewModel, ParseOutcome?)> writer, + CancellationToken token + ) + { + try + { + // Whether a missing proxy is fatal is the source's own setting, so the policy is built + // here rather than once for the whole run. The UI gate reads the same flag; this is the + // half that matters, because without it the request would leave the user's own address. + var forSource = options with + { + RequireProxy = !source.AllowsDirectConnection, + }; + + await foreach ( + var outcome in _runner + .RunAsync(source.Source, query, forSource, null, token) + .WithCancellation(token) + .ConfigureAwait(false) + ) + { + await writer.WriteAsync((source, outcome), token).ConfigureAwait(false); + } + } + catch (OperationCanceledException) + { + // Stopped, or another source met the target first. Nothing more to say about it. + return; + } + catch (Exception ex) + { + // One source falling over must not take the others down with it. + _logger.LogError(ex, "The source {Source} failed", source.Id); + + var error = ParseError.Create(0, "SourceFailed", ex.Message, ex.Message); + await TryWriteAsync(writer, (source, ParseOutcome.Failure(error)), token) + .ConfigureAwait(false); + } + + await TryWriteAsync(writer, (source, null), token).ConfigureAwait(false); + } + + /// Writes unless the run is already going down, in which case there is nobody to tell. + private static async Task TryWriteAsync( + ChannelWriter<(MediaSourceViewModel, ParseOutcome?)> writer, + (MediaSourceViewModel, ParseOutcome?) message, + CancellationToken token + ) + { + try + { + await writer.WriteAsync(message, token).ConfigureAwait(false); + } + catch (Exception ex) when (ex is OperationCanceledException or ChannelClosedException) + { + // Expected on the cancellation path. + } + } + + /// + /// Removes what the selected source collected. + /// private async Task PurgeAsync(CancellationToken cancellationToken) { + if (SelectedSource is null) + { + return; + } + var sourceId = SelectedSource.Id; var result = await _store.PurgeAsync(new PurgeOptions(sourceId), cancellationToken).ConfigureAwait(false); @@ -412,23 +917,9 @@ public partial class CollectViewModel : PageViewModel, IDisposable } } - private MediaQuery BuildQuery() - { - var endpoint = - SelectedSource.UsesEndpoint && Uri.TryCreate(EndpointText, UriKind.Absolute, out var parsed) - ? parsed - : null; - - return new MediaQuery(InputText, endpoint); - } - /// /// Builds the outcome line out of translated, plural-aware fragments. /// - /// - /// Assembled from pieces rather than one format string per case: Russian needs three forms for - /// a counted noun, so "{0} images" with an English plural glued on cannot be translated. - /// internal static string BuildSummary( int stored, int duplicates, @@ -468,39 +959,100 @@ public partial class CollectViewModel : PageViewModel, IDisposable return text; } - private void FlushBuffers(List items, List errors) + /// Queues one log line for the next flush. + private void WriteLog(CollectLogEntryViewModel entry) => _logBuffer.Enqueue(entry); + + /// Flushes from the timer thread, where an escaping exception would kill the process. + private void SafeFlush() { - if (items.Count == 0 && errors.Count == 0) + try + { + FlushBuffers(); + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Could not push collected rows to the page"); + } + } + + /// Moves everything buffered onto the UI collections, oldest first. + /// + /// Draining under the lock is what keeps the log in order: two flushes racing — the tick and the + /// batch — could otherwise queue their scheduled callbacks the wrong way round. + /// + private void FlushBuffers() + { + CollectedItemViewModel[] itemBatch; + CollectLogEntryViewModel[] logBatch; + + lock (_flushGate) + { + itemBatch = Drain(_itemBuffer); + logBatch = Drain(_logBuffer); + + if (itemBatch.Length == 0 && logBatch.Length == 0) + { + return; + } + + OnUi(() => + { + foreach (var item in itemBatch) + { + Items.Add(item); + } + + foreach (var entry in logBatch) + { + Log.Add(entry); + } + + while (Log.Count > MaxLogEntries) + { + Log.RemoveAt(0); + } + + ErrorCount += logBatch.Count(entry => entry.IsError); + }); + } + } + + private static T[] Drain(ConcurrentQueue queue) + { + if (queue.IsEmpty) + { + return []; + } + + var drained = new List(queue.Count); + while (queue.TryDequeue(out var value)) + { + drained.Add(value); + } + + return [.. drained]; + } + + /// + /// The pool and catalog are singletons and would otherwise keep this page alive. + public void Dispose() + { + // The container disposes every page twice — once per registration; see GalleryViewModel. + if (Interlocked.Exchange(ref _disposed, 1) == 1) { return; } - // Copy before clearing: the scheduled callback may run after the loop has refilled these. - var itemBatch = items.ToArray(); - var errorBatch = errors.ToArray(); - items.Clear(); - errors.Clear(); - - OnUi(() => - { - foreach (var item in itemBatch) - { - Items.Add(item); - } - - foreach (var error in errorBatch) - { - Errors.Add(new CollectErrorViewModel(error)); - } - }); - } - - /// - /// The pool is a singleton and would otherwise keep this page alive for the process. - public void Dispose() - { _proxyPool.Changed -= OnProxyPoolChanged; + _catalog.Changed -= OnCatalogChanged; _proxyChanged.Dispose(); + + foreach (var source in Sources) + { + source.PropertyChanged -= OnSourcePropertyChanged; + source.Dispose(); + } + GC.SuppressFinalize(this); } @@ -519,16 +1071,20 @@ public partial class CollectViewModel : PageViewModel, IDisposable item.Refresh(); } - foreach (var error in Errors) + foreach (var entry in Log) { - error.Refresh(); + entry.Refresh(); } } private void ClearResults() { + while (_itemBuffer.TryDequeue(out _)) { } + while (_logBuffer.TryDequeue(out _)) { } + Items.Clear(); - Errors.Clear(); + Log.Clear(); + ErrorCount = 0; } private void OnCommandFailed(Exception exception) diff --git a/src/AvParser.UI/ViewModels/DashboardViewModel.cs b/src/AvParser.UI/ViewModels/DashboardViewModel.cs index 85693d7..be9222a 100644 --- a/src/AvParser.UI/ViewModels/DashboardViewModel.cs +++ b/src/AvParser.UI/ViewModels/DashboardViewModel.cs @@ -1,31 +1,76 @@ +using System.Globalization; using AvParser.Core.Collecting; +using AvParser.Core.Proxies; using AvParser.Infrastructure.Storage; +using AvParser.UI.Localization; using AvParser.UI.Navigation; using Microsoft.Extensions.DependencyInjection; using ReactiveUI; using ReactiveUI.Primitives; +using ReactiveUI.Primitives.Concurrency; +using ReactiveUI.Primitives.Signals; +using ReactiveUI.SourceGenerators; namespace AvParser.UI.ViewModels; -/// Landing page: what is registered, where data lives, and shortcuts into the app. -public sealed class DashboardViewModel : PageViewModel +/// Landing page: what is registered, how the proxy pool is doing, and where data lives. +public sealed partial class DashboardViewModel : PageViewModel, IDisposable { private readonly IServiceProvider _services; + private readonly IProxyPool _pool; + private readonly ISequencer _mainThread; + private readonly Signal _poolChanged = new(); + + private int _disposed; + + /// Entries in the pool, feed and custom alike. + [Reactive] + public partial int ProxyTotal { get; private set; } + + /// How many answered this session and are not sidelined — the number the gate reads. + [Reactive] + public partial int ProxyLive { get; private set; } + + /// How many are serving a quarantine right now. + [Reactive] + public partial int ProxyQuarantined { get; private set; } + + /// How many nobody has spoken to yet this session. + [Reactive] + public partial int ProxyUnchecked { get; private set; } + + /// Average round trip of the proxies that answered; when none did. + [Reactive] + public partial string? ProxyLatency { get; private set; } + + /// Requests that went through a proxy and how many worked; null before any were made. + [Reactive] + public partial string? ProxyRequests { get; private set; } /// Creates the dashboard. /// Registered media sources, shown as cards. /// Where the app writes settings and logs. + /// The proxy pool, summarised on the page. /// /// Used to resolve at click time rather than at construction /// time. Injecting it directly would be a cycle: the navigation service is built from every /// page, so a page cannot also depend on it up front. /// - public DashboardViewModel(IMediaSourceCatalog catalog, IAppPaths paths, IServiceProvider services) + /// Scheduler for UI-affine updates; tests pass an immediate one. + public DashboardViewModel( + IMediaSourceCatalog catalog, + IAppPaths paths, + IProxyPool pool, + IServiceProvider services, + ISequencer? mainThread = null + ) { ArgumentNullException.ThrowIfNull(catalog); ArgumentNullException.ThrowIfNull(paths); _services = services ?? throw new ArgumentNullException(nameof(services)); + _pool = pool ?? throw new ArgumentNullException(nameof(pool)); + _mainThread = mainThread ?? RxSchedulers.MainThreadScheduler; Sources = [.. catalog.Sources.Select(source => new MediaSourceViewModel(source))]; DataDirectory = paths.DataDirectory; @@ -33,6 +78,17 @@ public sealed class DashboardViewModel : PageViewModel GoToCollectCommand = ReactiveCommand.Create(() => Navigate()); GoToSettingsCommand = ReactiveCommand.Create(() => Navigate()); + GoToProxiesCommand = ReactiveCommand.Create(() => Navigate()); + + // The pool fires on every lease outcome and every probe — a sweep of a few thousand entries + // would otherwise repaint the page a few thousand times. + _pool.Changed += OnPoolChanged; + _poolChanged + .Throttle(TimeSpan.FromMilliseconds(250), _mainThread) + .ObserveOn(_mainThread) + .Subscribe(_ => RefreshProxyStats()); + + RefreshProxyStats(); } /// @@ -50,12 +106,89 @@ public sealed class DashboardViewModel : PageViewModel /// Where collected media is written. public string MediaDirectory { get; } + /// Whether the pool holds anything at all; drives the empty state. + public bool HasProxies => ProxyTotal > 0; + /// Jumps to the Collect page. public ReactiveCommand GoToCollectCommand { get; } /// Jumps to the Settings page. public ReactiveCommand GoToSettingsCommand { get; } + /// Jumps to the Proxies page. + public ReactiveCommand GoToProxiesCommand { get; } + + /// Re-reads the pool. Exposed so tests can drive it without waiting for the throttle. + public void RefreshProxyStats() + { + var entries = _pool.Entries; + var now = DateTimeOffset.UtcNow; + var culture = CultureInfo.CurrentCulture; + + ProxyTotal = entries.Count; + ProxyLive = _pool.LiveCount; + ProxyQuarantined = entries.Count(entry => entry.IsQuarantined(now)); + ProxyUnchecked = entries.Count(entry => entry.Health == ProxyHealthState.Unknown); + + var latencies = entries + .Where(entry => entry.Latency is not null) + .Select(entry => entry.Latency!.Value) + .ToArray(); + + ProxyLatency = + latencies.Length == 0 + ? null + : Localizer.Instance.Format( + "Dashboard.Proxy.Latency", + latencies.Average(latency => latency.TotalMilliseconds).ToString("N0", culture) + ); + + var successes = entries.Sum(entry => entry.SuccessCount); + var attempts = successes + entries.Sum(entry => entry.FailureCount); + + ProxyRequests = + attempts == 0 + ? null + : Localizer.Instance.Format( + "Dashboard.Proxy.Requests", + successes.ToString("N0", culture), + attempts.ToString("N0", culture), + ((double)successes / attempts).ToString("P0", culture) + ); + + this.RaisePropertyChanged(nameof(HasProxies)); + } + + /// + /// The pool is a singleton and would otherwise keep this page alive. + public void Dispose() + { + // The container disposes every page twice — once per registration; see GalleryViewModel. + if (Interlocked.Exchange(ref _disposed, 1) == 1) + { + return; + } + + _pool.Changed -= OnPoolChanged; + _poolChanged.Dispose(); + + foreach (var source in Sources) + { + source.Dispose(); + } + } + + /// + protected override void OnLanguageChanged() + { + base.OnLanguageChanged(); + + // The latency and request lines were formatted in the previous language. + RefreshProxyStats(); + } + + private void OnPoolChanged(object? sender, EventArgs e) => _poolChanged.OnNext(RxVoid.Default); + private void Navigate() where TPage : PageViewModel => _services.GetRequiredService().NavigateTo(); } diff --git a/src/AvParser.UI/ViewModels/GalleryViewModel.cs b/src/AvParser.UI/ViewModels/GalleryViewModel.cs index c735fb9..4716965 100644 --- a/src/AvParser.UI/ViewModels/GalleryViewModel.cs +++ b/src/AvParser.UI/ViewModels/GalleryViewModel.cs @@ -317,10 +317,24 @@ public partial class GalleryViewModel : PageViewModel, IDisposable } /// + /// + /// Has to survive being called twice. Every page is registered under its own type and + /// under PageViewModel, and the container tracks the instance for both registrations, so + /// it disposes each page twice on shutdown. Cancelling an already-disposed source threw + /// out of ServiceProvider.Dispose and took the + /// process down on every exit. + /// public void Dispose() { - _loading?.Cancel(); - _loading?.Dispose(); + // Detached first: a load starting after this must not resurrect the field. + var loading = Interlocked.Exchange(ref _loading, null); + + if (loading is not null) + { + loading.Cancel(); + loading.Dispose(); + } + GC.SuppressFinalize(this); } diff --git a/src/AvParser.UI/ViewModels/MediaSourceViewModel.cs b/src/AvParser.UI/ViewModels/MediaSourceViewModel.cs index 1e716cc..d4c6a7f 100644 --- a/src/AvParser.UI/ViewModels/MediaSourceViewModel.cs +++ b/src/AvParser.UI/ViewModels/MediaSourceViewModel.cs @@ -1,4 +1,5 @@ using AvParser.Core.Collecting; +using AvParser.Core.Collecting.Sources; using AvParser.UI.Localization; using ReactiveUI; @@ -7,12 +8,15 @@ namespace AvParser.UI.ViewModels; /// A media source paired with its translated name and description. /// /// The domain knows nothing about languages, so carries English text. -/// This looks the id up as Source.{id}.Name and falls back to what the source itself says, -/// which keeps the "add a source = one registration line" promise intact: a new source works -/// untranslated rather than rendering a missing-key marker. +/// This looks the id up as Source.{id}.Name and falls back to what the source itself says — +/// which is what user-added sources always do, since their name is data the user typed, not a resx +/// key. Disposable because the catalog rebuilds these on every edit, and each one holds a +/// language-changed subscription that would otherwise leak. /// -public sealed class MediaSourceViewModel : ReactiveObject +public sealed class MediaSourceViewModel : ReactiveObject, IDisposable { + private bool _isSelected; + /// Wraps a source. public MediaSourceViewModel(IMediaSource source) { @@ -20,9 +24,24 @@ public sealed class MediaSourceViewModel : ReactiveObject Localizer.Instance.LanguageChanged += OnLanguageChanged; } + /// Whether the next run includes this source. + /// + /// Ticking is what a run acts on; the list's highlight only says which source the editor, + /// the delete button and the purge button are aimed at. Keeping the two apart is what lets a + /// run cover five sources while the user edits a sixth. + /// + public bool IsSelected + { + get => _isSelected; + set => this.RaiseAndSetIfChanged(ref _isSelected, value); + } + /// The source itself. public IMediaSource Source { get; } + /// The config behind the source, when it is a pattern source (they all are today). + public PatternSourceConfig? Config => (Source as PatternMediaSource)?.Config; + /// Stable identifier. public string Id => Source.Id; @@ -32,15 +51,27 @@ public sealed class MediaSourceViewModel : ReactiveObject /// Translated description, or the source's own when untranslated. public string Description => Localizer.Instance.GetOrDefault($"Source.{Id}.Description", Source.Description); - /// Whether this source needs a working proxy before it may run. + /// Whether this source goes to the network at all. public bool RequiresNetwork => Source.RequiresNetwork; - /// Whether this source reads a listing endpoint rather than pasted text. - public bool UsesEndpoint => Source.RequiresNetwork; + /// + /// Whether this source may run from the user's own address when no proxy is live. + /// + /// + /// Anything that is not a configured pattern source answers : the safe + /// end, since a source with nowhere to say otherwise has not said otherwise. + /// + public bool AllowsDirectConnection => Config?.AllowDirectConnection ?? false; + + /// Whether a run including this source is gated on a live proxy. + public bool NeedsProxy => RequiresNetwork && !AllowsDirectConnection; /// public override string ToString() => Name; + /// + public void Dispose() => Localizer.Instance.LanguageChanged -= OnLanguageChanged; + private void OnLanguageChanged(object? sender, EventArgs e) { this.RaisePropertyChanged(nameof(Name)); diff --git a/src/AvParser.UI/ViewModels/ProxiesViewModel.cs b/src/AvParser.UI/ViewModels/ProxiesViewModel.cs index f2dd306..8d89f4c 100644 --- a/src/AvParser.UI/ViewModels/ProxiesViewModel.cs +++ b/src/AvParser.UI/ViewModels/ProxiesViewModel.cs @@ -44,6 +44,8 @@ public partial class ProxiesViewModel : PageViewModel, IDisposable private readonly Dictionary _rows = new(StringComparer.Ordinal); private readonly Signal _poolChanged = new(); + private int _disposed; + /// Free-text filter over address and country. [Reactive] public partial string SearchText { get; set; } @@ -328,6 +330,12 @@ public partial class ProxiesViewModel : PageViewModel, IDisposable /// public void Dispose() { + // The container disposes every page twice — once per registration; see GalleryViewModel. + if (Interlocked.Exchange(ref _disposed, 1) == 1) + { + return; + } + _pool.Changed -= OnPoolChanged; _poolChanged.Dispose(); GC.SuppressFinalize(this); diff --git a/src/AvParser.UI/ViewModels/SettingsViewModel.cs b/src/AvParser.UI/ViewModels/SettingsViewModel.cs index 32317ea..ddc4870 100644 --- a/src/AvParser.UI/ViewModels/SettingsViewModel.cs +++ b/src/AvParser.UI/ViewModels/SettingsViewModel.cs @@ -63,10 +63,6 @@ public partial class SettingsViewModel : PageViewModel [Reactive] public partial int ProxyMinimumLive { get; set; } - /// Whether network parsers may run with no proxy available. - [Reactive] - public partial bool AllowDirectConnection { get; set; } - /// How many downloads may be in flight at once. [Reactive] public partial int MaxConcurrentDownloads { get; set; } @@ -136,7 +132,6 @@ public partial class SettingsViewModel : PageViewModel ProxyProbeTimeoutSeconds = current.ProxyProbeTimeoutSeconds; ProxyProbeConcurrency = current.ProxyProbeConcurrency; ProxyMinimumLive = current.ProxyMinimumLive; - AllowDirectConnection = current.AllowDirectConnection; MaxConcurrentDownloads = current.MaxConcurrentDownloads; MaxConcurrentPerHost = current.MaxConcurrentPerHost; HostDelayMs = current.HostDelayMs; @@ -168,8 +163,7 @@ public partial class SettingsViewModel : PageViewModel x => x.ProxyProbeTimeoutSeconds, x => x.ProxyProbeConcurrency, x => x.ProxyMinimumLive, - x => x.AllowDirectConnection, - (_, _, _, _, _, _, _, _) => RxVoid.Default + (_, _, _, _, _, _, _) => RxVoid.Default ) .Throttle(TimeSpan.FromMilliseconds(200), scheduler) .ObserveOn(scheduler) @@ -345,7 +339,6 @@ public partial class SettingsViewModel : PageViewModel ProxyProbeTimeoutSeconds = ProxyProbeTimeoutSeconds, ProxyProbeConcurrency = ProxyProbeConcurrency, ProxyMinimumLive = ProxyMinimumLive, - AllowDirectConnection = AllowDirectConnection, MaxConcurrentDownloads = MaxConcurrentDownloads, MaxConcurrentPerHost = MaxConcurrentPerHost, HostDelayMs = HostDelayMs, diff --git a/src/AvParser.UI/Views/CollectView.axaml b/src/AvParser.UI/Views/CollectView.axaml index 476d601..8115b01 100644 --- a/src/AvParser.UI/Views/CollectView.axaml +++ b/src/AvParser.UI/Views/CollectView.axaml @@ -7,25 +7,83 @@ x:Class="AvParser.UI.Views.CollectView" x:DataType="vm:CollectViewModel" > - + - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -52,7 +110,12 @@ - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/AvParser.UI/Views/SettingsView.axaml b/src/AvParser.UI/Views/SettingsView.axaml index a97ca95..f5ff178 100644 --- a/src/AvParser.UI/Views/SettingsView.axaml +++ b/src/AvParser.UI/Views/SettingsView.axaml @@ -141,10 +141,7 @@ - - - - + @@ -221,7 +218,7 @@ Command="{Binding BrowseMediaDirectoryCommand}" Content="{l:Loc Settings.MediaDirectoryBrowse}" /> - + diff --git a/test-service/.dockerignore b/test-service/.dockerignore new file mode 100644 index 0000000..11d926c --- /dev/null +++ b/test-service/.dockerignore @@ -0,0 +1,3 @@ +bin/ +obj/ +Properties/launchSettings.json diff --git a/test-service/AvParser.ImgTestService.csproj b/test-service/AvParser.ImgTestService.csproj new file mode 100644 index 0000000..7d2af3e --- /dev/null +++ b/test-service/AvParser.ImgTestService.csproj @@ -0,0 +1,10 @@ + + + net10.0 + enable + enable + AvParser.ImgTestService + true + + + diff --git a/test-service/Config.cs b/test-service/Config.cs new file mode 100644 index 0000000..d4a8ed5 --- /dev/null +++ b/test-service/Config.cs @@ -0,0 +1,164 @@ +using System.Security.Cryptography; + +namespace AvParser.ImgTestService; + +/// Top-level service options, bound from the ImgTest configuration section. +public sealed class ImgTestOptions +{ + /// Fraction of otherwise-valid ids that answer 404, to emulate a sparse site. 0 disables. + public double MissRate { get; set; } + + /// The endpoints to expose, each with its own id shape. + public List Endpoints { get; set; } = []; + + /// Resolves every endpoint and indexes it by path segment. + public Dictionary BuildLookup() + { + var lookup = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var endpoint in Endpoints) + { + var resolved = endpoint.Resolve(); + if (resolved is not null) + { + lookup[resolved.Path] = resolved; + } + } + + return lookup; + } + + /// + /// Whether an id should 404 despite being well-formed, deterministically by id. + /// + /// + /// Deterministic on purpose: a given id is either always present or always absent, so a + /// collector's "seen this url" journal and its duplicate detection stay meaningful across runs. + /// + public static bool IsMiss(string core, double missRate) + { + if (missRate <= 0) + { + return false; + } + + if (missRate >= 1) + { + return true; + } + + var hash = SHA256.HashData(System.Text.Encoding.UTF8.GetBytes("miss:" + core)); + var value = ((uint)hash[0] << 24) | ((uint)hash[1] << 16) | ((uint)hash[2] << 8) | hash[3]; + return value / (double)uint.MaxValue < missRate; + } +} + +/// One endpoint's configuration, as written in appsettings. +public sealed class EndpointOptions +{ + /// First path segment, e.g. test1 for /test1/{id}. + public string Path { get; set; } = string.Empty; + + /// Shortest id core (excluding any extension). + public int MinLength { get; set; } + + /// Longest id core (excluding any extension). + public int MaxLength { get; set; } + + /// Named character set: Letters, LettersLower, LettersUpper, Digits, Alphanumeric, HexLower, or Custom. + public string Alphabet { get; set; } = "Alphanumeric"; + + /// Exact characters, used when is Custom. + public string? Chars { get; set; } + + /// Accepted trailing extensions, e.g. .jpg; empty means none is expected. + public List Extensions { get; set; } = []; + + /// Turns this into a resolved endpoint, or null when unusable. + public ResolvedEndpoint? Resolve() + { + var allowed = ResolveAlphabet(Alphabet, Chars); + if (string.IsNullOrWhiteSpace(Path) || MinLength < 1 || MaxLength < MinLength || allowed.Count == 0) + { + return null; + } + + return new ResolvedEndpoint( + Path.Trim('/'), + MinLength, + MaxLength, + allowed, + [.. Extensions.Where(e => !string.IsNullOrWhiteSpace(e)).Select(Normalise)] + ); + } + + private static string Normalise(string extension) + { + var trimmed = extension.Trim(); + return trimmed.StartsWith('.') ? trimmed : "." + trimmed; + } + + private static HashSet ResolveAlphabet(string name, string? chars) + { + const string lower = "abcdefghijklmnopqrstuvwxyz"; + const string upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"; + const string digit = "0123456789"; + const string hex = "0123456789abcdef"; + + var set = name.Trim().ToLowerInvariant() switch + { + "letterslower" or "lower" => lower, + "lettersupper" or "upper" => upper, + "letters" => lower + upper, + "digits" or "digit" => digit, + "alphanumeric" or "alnum" => lower + upper + digit, + "hexlower" or "hex" => hex, + "custom" => chars ?? string.Empty, + _ => string.Empty, + }; + + return [.. set]; + } +} + +/// A validated endpoint ready to match requests. +public sealed record ResolvedEndpoint( + string Path, + int MinLength, + int MaxLength, + HashSet Allowed, + string[] Extensions +) +{ + /// Removes a recognised trailing extension from the raw path segment. + public string StripExtension(string id) + { + foreach (var extension in Extensions) + { + if (id.EndsWith(extension, StringComparison.OrdinalIgnoreCase)) + { + return id[..^extension.Length]; + } + } + + return id; + } + + /// Whether an id core is the right length and made only of allowed characters. + public bool Matches(string core) + { + if (core.Length < MinLength || core.Length > MaxLength) + { + return false; + } + + foreach (var c in core) + { + if (!Allowed.Contains(c)) + { + return false; + } + } + + return true; + } +} diff --git a/test-service/Directory.Build.props b/test-service/Directory.Build.props new file mode 100644 index 0000000..2239637 --- /dev/null +++ b/test-service/Directory.Build.props @@ -0,0 +1,10 @@ + + + + false + + diff --git a/test-service/Dockerfile b/test-service/Dockerfile new file mode 100644 index 0000000..45887dc --- /dev/null +++ b/test-service/Dockerfile @@ -0,0 +1,20 @@ +# Build the service, then run it on the smaller ASP.NET runtime image. +FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build +WORKDIR /src + +COPY Directory.Build.props ./ +COPY AvParser.ImgTestService.csproj ./ +RUN dotnet restore AvParser.ImgTestService.csproj + +COPY . ./ +RUN dotnet publish AvParser.ImgTestService.csproj -c Release -o /app --no-restore + +FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime +WORKDIR /app +COPY --from=build /app ./ + +# Listen on 8080 inside the container; a reverse proxy terminates TLS for the public domain. +ENV ASPNETCORE_URLS=http://+:8080 +EXPOSE 8080 + +ENTRYPOINT ["dotnet", "AvParser.ImgTestService.dll"] diff --git a/test-service/Identicon.cs b/test-service/Identicon.cs new file mode 100644 index 0000000..68a60f8 --- /dev/null +++ b/test-service/Identicon.cs @@ -0,0 +1,198 @@ +using System.IO.Compression; +using System.Security.Cryptography; + +namespace AvParser.ImgTestService; + +/// +/// Renders a deterministic identicon PNG from an id, with a hand-written encoder. +/// +/// +/// No image library and no native dependency on purpose: the container stays tiny and runs the same +/// on any architecture. The same id always yields the same bytes, so a collector that fetches an id +/// twice gets one blob, and a duplicate-detection test has something stable to assert against. +/// +public static class Identicon +{ + private const int Cells = 5; + private const int CellSize = 48; + private const int Margin = 20; + private const int Size = (Cells * CellSize) + (2 * Margin); + + /// Builds the PNG bytes for an id. + public static byte[] Render(string id) + { + var hash = SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(id)); + + // A saturated foreground from the first bytes, on a near-white background. + var (fr, fg, fb) = Foreground(hash); + byte br = 0xF2, + bg = 0xF2, + bb = 0xF4; + + var pixels = new byte[Size * Size * 3]; + FillBackground(pixels, br, bg, bb); + + // A 5x5 grid, mirrored left-to-right, so the icon reads as a single symmetric shape. Bit + // source is the tail of the hash, one bit per cell in the left three columns. + for (var col = 0; col < (Cells + 1) / 2; col++) + { + for (var row = 0; row < Cells; row++) + { + var bit = hash[(col * Cells) + row] & 1; + if (bit == 0) + { + continue; + } + + PaintCell(pixels, col, row, fr, fg, fb); + PaintCell(pixels, Cells - 1 - col, row, fr, fg, fb); + } + } + + return Encode(pixels, Size, Size); + } + + private static (byte R, byte G, byte B) Foreground(byte[] hash) + { + // Pick a hue-ish colour that is never too pale: keep at least one channel low and one high. + var r = (byte)(60 + (hash[0] % 160)); + var g = (byte)(60 + (hash[1] % 160)); + var b = (byte)(60 + (hash[2] % 160)); + return (r, g, b); + } + + private static void FillBackground(byte[] pixels, byte r, byte g, byte b) + { + for (var i = 0; i < pixels.Length; i += 3) + { + pixels[i] = r; + pixels[i + 1] = g; + pixels[i + 2] = b; + } + } + + private static void PaintCell(byte[] pixels, int col, int row, byte r, byte g, byte b) + { + var x0 = Margin + (col * CellSize); + var y0 = Margin + (row * CellSize); + + for (var y = y0; y < y0 + CellSize; y++) + { + var rowStart = y * Size * 3; + for (var x = x0; x < x0 + CellSize; x++) + { + var i = rowStart + (x * 3); + pixels[i] = r; + pixels[i + 1] = g; + pixels[i + 2] = b; + } + } + } + + /// Encodes RGB pixels as a PNG (colour type 2, 8-bit). + private static byte[] Encode(byte[] rgb, int width, int height) + { + using var output = new MemoryStream(); + + // Signature. + output.Write([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]); + + Span ihdr = stackalloc byte[13]; + WriteBigEndian(ihdr[..4], (uint)width); + WriteBigEndian(ihdr.Slice(4, 4), (uint)height); + ihdr[8] = 8; // bit depth + ihdr[9] = 2; // colour type: truecolour RGB + ihdr[10] = 0; // compression + ihdr[11] = 0; // filter + ihdr[12] = 0; // interlace + WriteChunk(output, "IHDR"u8, ihdr); + + // Raw scanlines: a leading filter byte (0 = none) then the row's RGB bytes. + var stride = width * 3; + var raw = new byte[height * (stride + 1)]; + for (var y = 0; y < height; y++) + { + var src = y * stride; + var dst = y * (stride + 1); + raw[dst] = 0; + Array.Copy(rgb, src, raw, dst + 1, stride); + } + + using (var compressed = new MemoryStream()) + { + using (var zlib = new ZLibStream(compressed, CompressionLevel.Optimal, leaveOpen: true)) + { + zlib.Write(raw, 0, raw.Length); + } + + WriteChunk(output, "IDAT"u8, compressed.ToArray()); + } + + WriteChunk(output, "IEND"u8, []); + + return output.ToArray(); + } + + private static void WriteChunk(Stream stream, ReadOnlySpan type, ReadOnlySpan data) + { + Span length = stackalloc byte[4]; + WriteBigEndian(length, (uint)data.Length); + stream.Write(length); + stream.Write(type); + stream.Write(data); + + var crc = Crc32.Compute(type, data); + Span crcBytes = stackalloc byte[4]; + WriteBigEndian(crcBytes, crc); + stream.Write(crcBytes); + } + + private static void WriteBigEndian(Span destination, uint value) + { + destination[0] = (byte)(value >> 24); + destination[1] = (byte)(value >> 16); + destination[2] = (byte)(value >> 8); + destination[3] = (byte)value; + } +} + +/// Minimal CRC-32 (PNG polynomial), enough to seal chunks. +internal static class Crc32 +{ + private static readonly uint[] Table = BuildTable(); + + public static uint Compute(ReadOnlySpan type, ReadOnlySpan data) + { + var crc = 0xFFFFFFFFu; + crc = Update(crc, type); + crc = Update(crc, data); + return crc ^ 0xFFFFFFFFu; + } + + private static uint Update(uint crc, ReadOnlySpan bytes) + { + foreach (var b in bytes) + { + crc = Table[(crc ^ b) & 0xFF] ^ (crc >> 8); + } + + return crc; + } + + private static uint[] BuildTable() + { + var table = new uint[256]; + for (var n = 0u; n < 256; n++) + { + var c = n; + for (var k = 0; k < 8; k++) + { + c = (c & 1) != 0 ? 0xEDB88320u ^ (c >> 1) : c >> 1; + } + + table[n] = c; + } + + return table; + } +} diff --git a/test-service/Program.cs b/test-service/Program.cs new file mode 100644 index 0000000..1210f22 --- /dev/null +++ b/test-service/Program.cs @@ -0,0 +1,57 @@ +using System.Text; +using AvParser.ImgTestService; + +var builder = WebApplication.CreateBuilder(args); + +var options = builder.Configuration.GetSection("ImgTest").Get() ?? new ImgTestOptions(); +var endpoints = options.BuildLookup(); + +var app = builder.Build(); + +// A plain index so opening the host in a browser explains what is here. +app.MapGet( + "/", + () => + { + var text = new StringBuilder(); + text.AppendLine("AvParser image test service"); + text.AppendLine(); + text.AppendLine($"miss rate: {options.MissRate:0.###}"); + text.AppendLine("endpoints:"); + foreach (var endpoint in endpoints.Values) + { + var ext = endpoint.Extensions.Length == 0 ? "(none)" : string.Join(" ", endpoint.Extensions); + text.AppendLine( + $" /{endpoint.Path}/{{id}} length {endpoint.MinLength}-{endpoint.MaxLength}, " + + $"{endpoint.Allowed.Count} chars, ext {ext}" + ); + } + + return Results.Text(text.ToString(), "text/plain; charset=utf-8"); + } +); + +app.MapGet("/healthz", () => Results.Text("ok", "text/plain")); + +// One handler for every configured endpoint: the first segment selects the config, the second is +// the id (optionally with an extension). A miss and an unknown endpoint are both a plain 404. +app.MapGet( + "/{endpoint}/{id}", + (string endpoint, string id) => + { + if (!endpoints.TryGetValue(endpoint, out var config)) + { + return Results.NotFound(); + } + + var core = config.StripExtension(id); + if (!config.Matches(core) || ImgTestOptions.IsMiss(core, options.MissRate)) + { + return Results.NotFound(); + } + + return Results.Bytes(Identicon.Render(core), "image/png"); + } +); + +app.Run(); diff --git a/test-service/Properties/launchSettings.json b/test-service/Properties/launchSettings.json new file mode 100644 index 0000000..533494d --- /dev/null +++ b/test-service/Properties/launchSettings.json @@ -0,0 +1,12 @@ +{ + "profiles": { + "http": { + "commandName": "Project", + "dotnetRunMessages": true, + "applicationUrl": "http://localhost:8080", + "environmentVariables": { + "ASPNETCORE_ENVIRONMENT": "Development" + } + } + } +} diff --git a/test-service/README.md b/test-service/README.md new file mode 100644 index 0000000..e7bbd6b --- /dev/null +++ b/test-service/README.md @@ -0,0 +1,66 @@ +# AvParser image test service + +A tiny, dependency-free web service that serves a deterministic image for any id matching a +configured pattern. It exists as a **target** for the app's pattern sources: point a source at one of +its endpoints and let the collector generate ids. + +Each valid id returns a unique identicon PNG (derived from `SHA-256(id)`), so fetching the same id +twice yields the same bytes — which is what makes the collector's duplicate detection observable. + +## Endpoints + +Endpoints are declared in [`appsettings.json`](appsettings.json) under `ImgTest:Endpoints`. The +defaults: + +| Path | Id length | Characters | Extension | +| --------------- | --------- | ----------------- | --------- | +| `/test1/{id}` | 6–8 | letters + digits | `.jpg` | +| `/test2/{id}` | 8–12 | digits only | none | +| `/test3/{id}` | 8 | hex (`0-9a-f`) | `.png` | + +Anything not matching a pattern (`/test1/xx`, `/test2/abcd1234`) returns `404`. `GET /` lists the +endpoints; `GET /healthz` returns `ok`. + +Example: `GET /test1/2dfhyuj.jpg` → `200 image/png`. + +### Adding an endpoint + +Add an object to `ImgTest:Endpoints`: + +```json +{ "Path": "test4", "MinLength": 4, "MaxLength": 6, "Alphabet": "Custom", "Chars": "abcdef012", "Extensions": [".png"] } +``` + +`Alphabet` is one of `LettersLower`, `LettersUpper`, `Letters`, `Digits`, `Alphanumeric`, `HexLower`, +or `Custom` (which uses `Chars`). + +## Miss rate + +Set `ImgTest:MissRate` (0..1) to make a deterministic fraction of otherwise-valid ids answer `404`, +emulating a site where most guessed ids do not exist. It is stable per id, so a run is repeatable. +Override at deploy time with the `ImgTest__MissRate` environment variable. + +## Run + +```bash +docker compose up --build +``` + +The container listens on `http://localhost:8080`. It speaks plain HTTP — **TLS for a public domain +such as `https://imgtest.example` is terminated by your own reverse proxy**, which forwards to this +container. A Caddy example: + +``` +imgtest.example { + reverse_proxy localhost:8080 +} +``` + +Or run it directly without Docker: + +```bash +dotnet run +``` + +> This service is intentionally **not** part of `AvParser.slnx` and is not built by `./build.ps1`. +> It is a standalone support tool with its own Docker context. diff --git a/test-service/appsettings.json b/test-service/appsettings.json new file mode 100644 index 0000000..e4804c6 --- /dev/null +++ b/test-service/appsettings.json @@ -0,0 +1,34 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + "ImgTest": { + "MissRate": 0.0, + "Endpoints": [ + { + "Path": "test1", + "MinLength": 6, + "MaxLength": 8, + "Alphabet": "Alphanumeric", + "Extensions": [".jpg"] + }, + { + "Path": "test2", + "MinLength": 8, + "MaxLength": 12, + "Alphabet": "Digits", + "Extensions": [] + }, + { + "Path": "test3", + "MinLength": 8, + "MaxLength": 8, + "Alphabet": "HexLower", + "Extensions": [".png"] + } + ] + } +} diff --git a/test-service/docker-compose.yml b/test-service/docker-compose.yml new file mode 100644 index 0000000..f381797 --- /dev/null +++ b/test-service/docker-compose.yml @@ -0,0 +1,12 @@ +services: + imgtest: + build: . + image: avparser-imgtest + container_name: avparser-imgtest + restart: unless-stopped + ports: + # host:container — change the host port to suit; the reverse proxy points at it. + - "8080:8080" + environment: + # Raise this (0..1) to make a fraction of valid ids answer 404, emulating a sparse site. + - ImgTest__MissRate=0.0 diff --git a/tests/AvParser.Core.Tests/Collecting/MediaSourceCatalogTests.cs b/tests/AvParser.Core.Tests/Collecting/MediaSourceCatalogTests.cs new file mode 100644 index 0000000..ebd4ed1 --- /dev/null +++ b/tests/AvParser.Core.Tests/Collecting/MediaSourceCatalogTests.cs @@ -0,0 +1,110 @@ +using AvParser.Core.Collecting; +using AvParser.Core.Collecting.Sources; + +namespace AvParser.Core.Tests.Collecting; + +public class MediaSourceCatalogTests +{ + private sealed class FakeStore(IEnumerable? seed = null) : IUserSourceStore + { + private readonly List _configs = seed?.ToList() ?? []; + + public event EventHandler? Changed; + + public IReadOnlyList List() => [.. _configs]; + + public Task AddAsync( + PatternSourceConfig config, + CancellationToken cancellationToken = default + ) + { + _configs.Add(config); + Changed?.Invoke(this, EventArgs.Empty); + return Task.FromResult(config); + } + + public Task UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default) + { + var index = _configs.FindIndex(c => c.Id == config.Id); + if (index < 0) + { + return Task.FromResult(false); + } + + _configs[index] = config; + Changed?.Invoke(this, EventArgs.Empty); + return Task.FromResult(true); + } + + public Task RemoveAsync(string id, CancellationToken cancellationToken = default) + { + var removed = _configs.RemoveAll(c => c.Id == id) > 0; + if (removed) + { + Changed?.Invoke(this, EventArgs.Empty); + } + + return Task.FromResult(removed); + } + } + + private static PatternSourceConfig Config(string id, string name) + { + PatternSourceConfig.TryCreate( + name, + "https://h/x/", + 6, + 8, + IdAlphabet.Digits, + null, + null, + allowDirectConnection: false, + out var config, + id + ); + return config!; + } + + [Fact] + public void An_empty_store_is_a_valid_empty_catalog() + { + using var catalog = new MediaSourceCatalog(new FakeStore()); + + catalog.Sources.ShouldBeEmpty(); + catalog.Find("anything").ShouldBeNull(); + } + + [Fact] + public void Sources_are_materialised_from_the_stored_configs() + { + using var catalog = new MediaSourceCatalog(new FakeStore([Config("s1", "Beta"), Config("s2", "Alpha")])); + + // Ordered by display name. + catalog.Sources.Select(s => s.Id).ShouldBe(["s2", "s1"]); + catalog.Find("s1").ShouldNotBeNull().DisplayName.ShouldBe("Beta"); + } + + [Fact] + public async Task Adding_through_the_catalog_rebuilds_and_signals() + { + var store = new FakeStore(); + using var catalog = new MediaSourceCatalog(store); + var changed = 0; + catalog.Changed += (_, _) => changed++; + + await catalog.AddAsync(Config("s1", "One"), TestContext.Current.CancellationToken); + + changed.ShouldBe(1); + catalog.Sources.ShouldHaveSingleItem().Id.ShouldBe("s1"); + } + + [Fact] + public async Task Removing_through_the_catalog_drops_the_source() + { + using var catalog = new MediaSourceCatalog(new FakeStore([Config("s1", "One")])); + + await catalog.RemoveAsync("s1", TestContext.Current.CancellationToken); + + catalog.Sources.ShouldBeEmpty(); + } +} diff --git a/tests/AvParser.Core.Tests/Collecting/PatternMediaSourceTests.cs b/tests/AvParser.Core.Tests/Collecting/PatternMediaSourceTests.cs new file mode 100644 index 0000000..097e214 --- /dev/null +++ b/tests/AvParser.Core.Tests/Collecting/PatternMediaSourceTests.cs @@ -0,0 +1,283 @@ +using AvParser.Core.Collecting; +using AvParser.Core.Collecting.Sources; + +namespace AvParser.Core.Tests.Collecting; + +public class PatternMediaSourceTests +{ + private static PatternSourceConfig Config( + int min = 6, + int max = 8, + IdAlphabet alphabet = IdAlphabet.Alphanumeric, + string? custom = null, + string? extension = ".jpg", + string url = "https://imgtest.example/test1/", + bool allowDirect = false + ) + { + PatternSourceConfig.TryCreate("Test", url, min, max, alphabet, custom, extension, allowDirect, out var config); + return config!; + } + + private static async Task> Collect(IMediaSource source, int limit) + { + var list = new List(); + await foreach (var outcome in source.ParseAsync(new MediaQuery(Limit: limit), null, CancellationToken.None)) + { + if (outcome.IsSuccess) + { + list.Add(outcome.Value!); + } + } + + return list; + } + + [Fact] + public void A_pattern_source_needs_the_network() + { + new PatternMediaSource(Config()).RequiresNetwork.ShouldBeTrue(); + } + + [Fact] + public async Task Generated_ids_stay_within_the_length_range_and_alphabet() + { + var source = new PatternMediaSource(Config(min: 4, max: 6, alphabet: IdAlphabet.Digits, extension: null)); + + var candidates = await Collect(source, 300); + + candidates.Count.ShouldBe(300); + foreach (var candidate in candidates) + { + var id = candidate.ExternalId.ShouldNotBeNull(); + id.Length.ShouldBeInRange(4, 6); + id.ShouldAllBe(c => char.IsAsciiDigit(c)); + } + } + + [Fact] + public async Task Candidates_hang_the_id_off_the_base_url_with_the_extension() + { + var source = new PatternMediaSource(Config(min: 6, max: 6, alphabet: IdAlphabet.HexLower, extension: ".png")); + + var candidate = (await Collect(source, 1)).ShouldHaveSingleItem(); + + candidate.SourceId.ShouldBe(source.Id); + candidate.Url.AbsoluteUri.ShouldBe($"https://imgtest.example/test1/{candidate.ExternalId}.png"); + } + + [Fact] + public async Task An_unset_extension_still_lands_on_a_jpg_address() + { + var source = new PatternMediaSource(Config(min: 6, max: 6, alphabet: IdAlphabet.HexLower, extension: null)); + + var candidate = (await Collect(source, 1)).ShouldHaveSingleItem(); + + candidate.Url.AbsoluteUri.ShouldBe($"https://imgtest.example/test1/{candidate.ExternalId}.jpg"); + } + + [Fact] + public async Task The_attempt_budget_bounds_how_many_are_generated() + { + var source = new PatternMediaSource(Config(min: 6, max: 8, alphabet: IdAlphabet.Alphanumeric)); + + (await Collect(source, 40)).Count.ShouldBe(40); + } + + [Fact] + public async Task A_small_space_is_never_exceeded_however_large_the_budget() + { + // Digits of length 2 is a hundred ids; asking for a thousand must not loop forever. + var source = new PatternMediaSource(Config(min: 2, max: 2, alphabet: IdAlphabet.Digits, extension: null)); + + var candidates = await Collect(source, 1000); + + candidates.Count.ShouldBeLessThanOrEqualTo(100); + candidates.Select(c => c.ExternalId).Distinct().Count().ShouldBe(candidates.Count); + } + + [Fact] + public async Task No_budget_means_the_source_keeps_going_until_it_is_stopped() + { + // A limit of zero used to mean "one candidate", which made an unlimited run collect nothing. + var source = new PatternMediaSource(Config(min: 8, max: 8, alphabet: IdAlphabet.Alphanumeric)); + using var stop = new CancellationTokenSource(); + + var seen = 0; + + try + { + await foreach (var outcome in source.ParseAsync(new MediaQuery(), null, stop.Token)) + { + outcome.IsSuccess.ShouldBeTrue(); + + if (++seen >= 5_000) + { + await stop.CancelAsync(); + } + } + } + catch (OperationCanceledException) + { + // The only way an unbounded run ends. + } + + seen.ShouldBe(5_000); + } + + [Fact] + public async Task A_small_space_still_bounds_an_unlimited_run() + { + // Two digits is a hundred ids: with no budget the run must end at the space, not spin on + // collisions for ever. + var source = new PatternMediaSource(Config(min: 2, max: 2, alphabet: IdAlphabet.Digits, extension: null)); + + var candidates = await Collect(source, 0); + + // Not exactly a hundred: the draw is random, so the last few ids may be given up on rather + // than waited for. What matters is that the run ends and never repeats itself. + candidates.Count.ShouldBeInRange(90, 100); + candidates.Select(c => c.ExternalId).Distinct().Count().ShouldBe(candidates.Count); + } + + [Fact] + public async Task Generated_ids_are_unique_within_a_run() + { + var source = new PatternMediaSource(Config(min: 8, max: 8, alphabet: IdAlphabet.Alphanumeric, extension: null)); + + var candidates = await Collect(source, 500); + + candidates.Select(c => c.ExternalId).Distinct().Count().ShouldBe(500); + } +} + +public class PatternAlphabetTests +{ + [Theory] + [InlineData(IdAlphabet.LettersLower, "abcdefghijklmnopqrstuvwxyz")] + [InlineData(IdAlphabet.Digits, "0123456789")] + [InlineData(IdAlphabet.HexLower, "0123456789abcdef")] + public void Named_sets_resolve_to_their_characters(IdAlphabet alphabet, string expected) + { + PatternAlphabet.Resolve(alphabet).ShouldBe(expected); + } + + [Fact] + public void Letters_and_alphanumeric_combine_the_sets() + { + PatternAlphabet.Resolve(IdAlphabet.Letters).Length.ShouldBe(52); + PatternAlphabet.Resolve(IdAlphabet.Alphanumeric).Length.ShouldBe(62); + } + + [Fact] + public void A_custom_set_is_de_duplicated_and_stripped_of_separators() + { + // Repeats would skew the draw; a slash would break the "one path segment" assumption. + PatternAlphabet.Resolve(IdAlphabet.Custom, "aab/c c").ShouldBe("abc"); + } + + [Fact] + public void An_empty_custom_set_resolves_to_nothing() + { + PatternAlphabet.Resolve(IdAlphabet.Custom, " ").ShouldBeEmpty(); + } +} + +public class PatternSourceConfigTests +{ + [Fact] + public void A_valid_config_normalises_the_base_url_and_extension() + { + var error = PatternSourceConfig.TryCreate( + " My source ", + "https://imgtest.example/test1", + 6, + 8, + IdAlphabet.Alphanumeric, + null, + "jpg", + allowDirectConnection: false, + out var config + ); + + error.ShouldBe(PatternConfigError.None); + config.ShouldNotBeNull(); + config.Name.ShouldBe("My source"); + config.BaseUrl.AbsoluteUri.ShouldBe("https://imgtest.example/test1/"); + config.Extension.ShouldBe(".jpg"); + config.Id.ShouldNotBeNullOrWhiteSpace(); + } + + [Fact] + public void A_blank_extension_becomes_the_default_one() + { + // The editor shows ".jpg" as the placeholder, so an untouched field means "the usual one", + // not "no suffix" — the latter costs a whole run of 404s to discover. + var error = PatternSourceConfig.TryCreate( + "N", + "https://h/x/", + 6, + 8, + IdAlphabet.Digits, + null, + " ", + allowDirectConnection: false, + out var config + ); + + error.ShouldBe(PatternConfigError.None); + config.ShouldNotBeNull().Extension.ShouldBe(PatternSourceConfig.DefaultExtension); + } + + [Fact] + public void A_supplied_id_is_kept() + { + PatternSourceConfig.TryCreate( + "N", + "https://h/x/", + 6, + 8, + IdAlphabet.Digits, + null, + null, + allowDirectConnection: false, + out var config, + "fixed-id" + ); + + config.ShouldNotBeNull().Id.ShouldBe("fixed-id"); + } + + [Theory] + [InlineData("", "https://h/x/", 6, 8, IdAlphabet.Digits, PatternConfigError.NameRequired)] + [InlineData("N", "not-a-url", 6, 8, IdAlphabet.Digits, PatternConfigError.BaseUrlInvalid)] + [InlineData("N", "ftp://h/x/", 6, 8, IdAlphabet.Digits, PatternConfigError.BaseUrlInvalid)] + [InlineData("N", "https://h/x/", 0, 8, IdAlphabet.Digits, PatternConfigError.LengthRangeInvalid)] + [InlineData("N", "https://h/x/", 8, 6, IdAlphabet.Digits, PatternConfigError.LengthRangeInvalid)] + [InlineData("N", "https://h/x/", 6, 65, IdAlphabet.Digits, PatternConfigError.LengthRangeInvalid)] + [InlineData("N", "https://h/x/", 6, 8, IdAlphabet.Custom, PatternConfigError.AlphabetEmpty)] + public void Invalid_inputs_are_rejected_with_a_reason( + string name, + string url, + int min, + int max, + IdAlphabet alphabet, + PatternConfigError expected + ) + { + var error = PatternSourceConfig.TryCreate( + name, + url, + min, + max, + alphabet, + null, + null, + allowDirectConnection: false, + out var config + ); + + error.ShouldBe(expected); + config.ShouldBeNull(); + } +} diff --git a/tests/AvParser.Core.Tests/Collecting/UrlListSourceTests.cs b/tests/AvParser.Core.Tests/Collecting/UrlListSourceTests.cs deleted file mode 100644 index 82d38b3..0000000 --- a/tests/AvParser.Core.Tests/Collecting/UrlListSourceTests.cs +++ /dev/null @@ -1,198 +0,0 @@ -using AvParser.Core.Collecting; -using AvParser.Core.Collecting.Sources; -using AvParser.Core.Parsing; - -namespace AvParser.Core.Tests.Collecting; - -public class UrlListSourceTests -{ - private static readonly IMediaSource Source = new UrlListSource(); - - private static async Task>> RunAsync(string text, int limit = 0) - { - var results = new List>(); - - await foreach ( - var outcome in Source.ParseAsync( - new MediaQuery(text, Limit: limit), - null, - TestContext.Current.CancellationToken - ) - ) - { - results.Add(outcome); - } - - return results; - } - - [Fact] - public async Task Each_line_becomes_a_candidate() - { - var results = await RunAsync("https://example.test/a.png\nhttps://example.test/b.gif"); - - results.Count.ShouldBe(2); - results.ShouldAllBe(r => r.IsSuccess); - results[0].Value!.Url.AbsoluteUri.ShouldBe("https://example.test/a.png"); - results[1].Value!.Ordinal.ShouldBe(2); - } - - [Fact] - public async Task Blank_lines_and_comments_are_ignored() - { - var results = await RunAsync( - """ - # my list - https://example.test/a.png - - # indented comment - https://example.test/b.png - """ - ); - - results.Count.ShouldBe(2); - } - - [Fact] - public async Task Whitespace_around_an_address_is_forgiven() - { - var results = await RunAsync(" https://example.test/a.png "); - - results.ShouldHaveSingleItem().IsSuccess.ShouldBeTrue(); - } - - [Fact] - public async Task A_bad_line_is_named_rather_than_dropped() - { - // In a paste of two hundred addresses a silently skipped typo is unfindable. - var results = await RunAsync("https://example.test/a.png\nnot an address\nhttps://example.test/b.png"); - - results.Count.ShouldBe(3); - results[1].IsSuccess.ShouldBeFalse(); - results[1].Error!.Code.ShouldBe("NotAnAddress"); - results[1].Error!.Arguments.ShouldContain("not an address"); - } - - [Theory] - [InlineData("file:///etc/passwd")] - [InlineData("data:image/png;base64,AAAA")] - [InlineData("ftp://example.test/a.png")] - [InlineData("javascript:alert(1)")] - public async Task Only_http_addresses_are_accepted(string line) - { - // A pasted list is as likely to have come from somewhere else as to have been typed. - var results = await RunAsync(line); - - results.ShouldHaveSingleItem().IsSuccess.ShouldBeFalse(); - } - - [Fact] - public async Task A_limit_stops_the_listing_early() - { - var text = string.Join('\n', Enumerable.Range(0, 50).Select(i => $"https://example.test/{i}.png")); - - var results = await RunAsync(text, limit: 5); - - results.Count(r => r.IsSuccess).ShouldBe(5); - } - - [Fact] - public void The_source_needs_no_network_of_its_own() - { - // It discovers nothing; downloading what it read is the fetcher's job and is gated there. - Source.RequiresNetwork.ShouldBeFalse(); - } - - [Theory] - [InlineData("https://example.test/a.png", true)] - [InlineData("# only a comment", false)] - [InlineData("", false)] - [InlineData("nonsense", false)] - public void CanParse_answers_without_doing_any_work(string text, bool expected) => - Source.CanParse(new MediaQuery(text)).ShouldBe(expected); - - [Fact] - public async Task Cancellation_is_honoured() - { - using var cancellation = new CancellationTokenSource(); - await cancellation.CancelAsync(); - - await Should.ThrowAsync(async () => - { - await foreach (var _ in Source.ParseAsync(new MediaQuery("https://a.test/x.png"), null, cancellation.Token)) - { - // Draining is the point; the first move must already throw. - } - }); - } -} - -public class MediaSourceCatalogTests -{ - [Fact] - public void Sources_are_ordered_by_display_name() - { - var catalog = new MediaSourceCatalog([new FakeSource("z", "Zebra"), new FakeSource("a", "Aardvark")]); - - catalog.Sources.Select(s => s.Id).ShouldBe(["a", "z"]); - } - - [Fact] - public void The_named_default_wins_over_alphabetical_order() - { - // Otherwise the landing page depends on a display name, and would open on the network - // source — behind the proxy gate — before the user has asked for anything. - var catalog = new MediaSourceCatalog([new FakeSource("a", "Aardvark"), new FakeSource("z", "Zebra")], "z"); - - catalog.DefaultSource.Id.ShouldBe("z"); - } - - [Fact] - public void An_unknown_default_falls_back_rather_than_throwing() - { - var catalog = new MediaSourceCatalog([new FakeSource("a", "Aardvark")], "removed-in-a-past-version"); - - catalog.DefaultSource.Id.ShouldBe("a"); - } - - [Fact] - public void Lookup_ignores_case_and_reports_a_miss() - { - IMediaSourceCatalog catalog = new MediaSourceCatalog([new FakeSource("url-list", "URL list")]); - - catalog.Find("URL-LIST").ShouldNotBeNull(); - catalog.Find("nope").ShouldBeNull(); - catalog.FindOrDefault("nope").Id.ShouldBe("url-list"); - } - - [Fact] - public void An_empty_registration_is_rejected() => - Should.Throw(() => new MediaSourceCatalog([])); - - [Fact] - public void Two_sources_sharing_an_id_are_rejected() => - Should.Throw(() => - new MediaSourceCatalog([new FakeSource("same", "One"), new FakeSource("same", "Two")]) - ); - - private sealed class FakeSource(string id, string name) : IMediaSource - { - public string Id => id; - - public string DisplayName => name; - - public string Description => string.Empty; - - public bool CanParse(MediaQuery input) => true; - - public async IAsyncEnumerable> ParseAsync( - MediaQuery input, - IProgress? progress, - [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken - ) - { - await Task.Yield(); - yield break; - } - } -} diff --git a/tests/AvParser.Core.Tests/Proxies/ProxyPoolWarmUpTests.cs b/tests/AvParser.Core.Tests/Proxies/ProxyPoolWarmUpTests.cs index eb80e8b..839a0bf 100644 --- a/tests/AvParser.Core.Tests/Proxies/ProxyPoolWarmUpTests.cs +++ b/tests/AvParser.Core.Tests/Proxies/ProxyPoolWarmUpTests.cs @@ -102,4 +102,57 @@ public class ProxyPoolWarmUpTests (await pool.WarmUpAsync(1, cancellationToken: TestContext.Current.CancellationToken)).ShouldBe(0); pool.LiveCount.ShouldBe(0); } + + [Fact] + public async Task The_top_up_checks_what_the_warm_up_skipped() + { + // The warm-up leaves almost everything unknown by design; without this pass the pool looks + // — and behaves — as if it held two proxies rather than the fifty that answer. + var pool = Build(out var probe, out _, [.. Enumerable.Range(0, 50).Select(index => $"h{index}")]); + await pool.RefreshAsync(TestContext.Current.CancellationToken); + probe.DefaultAlive = true; + + await pool.WarmUpAsync(2, cancellationToken: TestContext.Current.CancellationToken); + var afterWarmUp = probe.ProbeCount; + + afterWarmUp.ShouldBeLessThan(50); + + var found = await pool.TopUpAsync(4, cancellationToken: TestContext.Current.CancellationToken); + + found.ShouldBe(50 - afterWarmUp); + pool.LiveCount.ShouldBe(50); + pool.Entries.ShouldAllBe(entry => entry.Health == ProxyHealthState.Alive); + } + + [Fact] + public async Task The_top_up_does_not_re_probe_what_is_already_known() + { + var pool = Build(out var probe, out _, "a", "b"); + await pool.RefreshAsync(TestContext.Current.CancellationToken); + probe.DefaultAlive = true; + + await pool.TopUpAsync(2, cancellationToken: TestContext.Current.CancellationToken); + var first = probe.ProbeCount; + + // Everything has a verdict now, so a second pass has nothing to do — re-probing would just + // be a sweep, and a sweep is something the user asks for. + (await pool.TopUpAsync(2, cancellationToken: TestContext.Current.CancellationToken)).ShouldBe(0); + probe.ProbeCount.ShouldBe(first); + } + + [Fact] + public async Task A_stopped_top_up_keeps_what_it_learned() + { + var pool = Build(out var probe, out _, [.. Enumerable.Range(0, 20).Select(index => $"h{index}")]); + await pool.RefreshAsync(TestContext.Current.CancellationToken); + probe.DefaultAlive = true; + + using var stop = new CancellationTokenSource(); + await stop.CancelAsync(); + + await Should.ThrowAsync(() => pool.TopUpAsync(2, null, stop.Token)); + + // Cancelled before anything was probed, so nothing is claimed to be live either. + pool.LiveCount.ShouldBe(0); + } } diff --git a/tests/AvParser.Infrastructure.Tests/Collecting/CollectRunnerTests.cs b/tests/AvParser.Infrastructure.Tests/Collecting/CollectRunnerTests.cs index 9476e99..49d1e01 100644 --- a/tests/AvParser.Infrastructure.Tests/Collecting/CollectRunnerTests.cs +++ b/tests/AvParser.Infrastructure.Tests/Collecting/CollectRunnerTests.cs @@ -138,8 +138,7 @@ public sealed class CollectRunnerTests : IAsyncLifetime NullLogger.Instance ); - // Direct is allowed here: these tests are about the runner, not the proxy gate. - _settings = new FixedSettings(new AppSettings { AllowDirectConnection = true }); + _settings = new FixedSettings(new AppSettings()); _fetcher = new ScriptedFetcher(_blobs); _throttle = new HostThrottle(4, TimeSpan.Zero, NullLogger.Instance); _runner = new CollectRunner(_fetcher, _store, _throttle, NullLogger.Instance); @@ -354,66 +353,3 @@ public sealed class CollectRunnerTests : IAsyncLifetime } } } - -public class OwnServiceListingTests -{ - private static readonly Uri Endpoint = new("https://own.test/api/list"); - - [Fact] - public void An_object_with_items_is_read() - { - var page = OwnServiceSource.ReadPage( - """ - { "items": [ { "url": "https://own.test/a.png", "id": "42", "name": "kitten", - "published": "2026-08-13T10:00:00Z", "size": 4096, "tags": ["cats"] } ], - "next": "page2" } - """, - Endpoint - ); - - var item = page.Items.ShouldHaveSingleItem(); - item.Url.AbsoluteUri.ShouldBe("https://own.test/a.png"); - item.ExternalId.ShouldBe("42"); - item.SuggestedName.ShouldBe("kitten"); - item.ExpectedLength.ShouldBe(4096); - item.Tags.ShouldBe(["cats"]); - page.Next.ShouldBe("page2"); - } - - [Fact] - public void A_bare_array_of_addresses_is_read() - { - // The service on the other end is the user's own; it should not have to be rewritten to - // match a schema we invented. - var page = OwnServiceSource.ReadPage("""["https://own.test/a.png", "https://own.test/b.gif"]""", Endpoint); - - page.Items.Count.ShouldBe(2); - page.Next.ShouldBeNull(); - } - - [Fact] - public void Relative_addresses_resolve_against_the_endpoint() - { - var page = OwnServiceSource.ReadPage("""{"items":[{"url":"/files/a.png"}]}""", Endpoint); - - page.Items.ShouldHaveSingleItem().Url.AbsoluteUri.ShouldBe("https://own.test/files/a.png"); - } - - [Fact] - public void Entries_without_a_usable_address_are_dropped() - { - var page = OwnServiceSource.ReadPage( - """{"items":[{"name":"no url"}, {"url":"data:image/png;base64,AA"}, {"url":"https://own.test/ok.png"}]}""", - Endpoint - ); - - page.Items.ShouldHaveSingleItem().Url.AbsoluteUri.ShouldBe("https://own.test/ok.png"); - } - - [Fact] - public void An_empty_listing_is_not_an_error() - { - OwnServiceSource.ReadPage("""{"items":[]}""", Endpoint).Items.ShouldBeEmpty(); - OwnServiceSource.ReadPage("[]", Endpoint).Items.ShouldBeEmpty(); - } -} diff --git a/tests/AvParser.Infrastructure.Tests/Collecting/JsonUserSourceStoreTests.cs b/tests/AvParser.Infrastructure.Tests/Collecting/JsonUserSourceStoreTests.cs new file mode 100644 index 0000000..eae639b --- /dev/null +++ b/tests/AvParser.Infrastructure.Tests/Collecting/JsonUserSourceStoreTests.cs @@ -0,0 +1,124 @@ +using AvParser.Core.Collecting.Sources; +using AvParser.Infrastructure.Collecting; +using AvParser.Infrastructure.Storage; +using Microsoft.Extensions.Logging.Abstractions; + +namespace AvParser.Infrastructure.Tests.Collecting; + +public sealed class JsonUserSourceStoreTests : IDisposable +{ + private readonly string _directory = Path.Combine( + Path.GetTempPath(), + "AvParserTests", + Guid.NewGuid().ToString("N") + ); + + private JsonUserSourceStore Create() => new(new AppPaths(_directory), NullLogger.Instance); + + private static PatternSourceConfig Config(string id, string name = "Test", bool allowDirect = false) + { + PatternSourceConfig.TryCreate( + name, + "https://imgtest.example/test1/", + 6, + 8, + IdAlphabet.Alphanumeric, + null, + ".jpg", + allowDirect, + out var config, + id + ); + return config!; + } + + public void Dispose() + { + if (Directory.Exists(_directory)) + { + Directory.Delete(_directory, recursive: true); + } + } + + [Fact] + public void An_absent_file_reads_as_an_empty_list() + { + using var store = Create(); + + store.List().ShouldBeEmpty(); + } + + [Fact] + public async Task Added_sources_survive_a_reload() + { + using (var store = Create()) + { + await store.AddAsync(Config("s1", "Kept"), TestContext.Current.CancellationToken); + } + + // A brand-new instance reads from disk rather than from the in-memory cache. + using var reopened = Create(); + + var config = reopened.List().ShouldHaveSingleItem(); + config.Id.ShouldBe("s1"); + config.Name.ShouldBe("Kept"); + config.BaseUrl.AbsoluteUri.ShouldBe("https://imgtest.example/test1/"); + config.Extension.ShouldBe(".jpg"); + } + + [Fact] + public async Task Updating_replaces_the_matching_config() + { + using var store = Create(); + await store.AddAsync(Config("s1", "Before"), TestContext.Current.CancellationToken); + + var updated = Config("s1", "After"); + (await store.UpdateAsync(updated, TestContext.Current.CancellationToken)).ShouldBeTrue(); + + store.List().ShouldHaveSingleItem().Name.ShouldBe("After"); + } + + [Fact] + public async Task Updating_an_unknown_id_changes_nothing() + { + using var store = Create(); + + (await store.UpdateAsync(Config("ghost"), TestContext.Current.CancellationToken)).ShouldBeFalse(); + store.List().ShouldBeEmpty(); + } + + [Fact] + public async Task Removing_takes_the_config_out() + { + using var store = Create(); + await store.AddAsync(Config("s1"), TestContext.Current.CancellationToken); + + (await store.RemoveAsync("s1", TestContext.Current.CancellationToken)).ShouldBeTrue(); + store.List().ShouldBeEmpty(); + } + + [Fact] + public async Task Mutations_raise_the_changed_event() + { + using var store = Create(); + var changed = 0; + store.Changed += (_, _) => changed++; + + await store.AddAsync(Config("s1"), TestContext.Current.CancellationToken); + await store.RemoveAsync("s1", TestContext.Current.CancellationToken); + + changed.ShouldBe(2); + } + + [Fact] + public async Task A_corrupt_file_reads_as_an_empty_list() + { + Directory.CreateDirectory(_directory); + IAppPaths paths = new AppPaths(_directory); + await File.WriteAllTextAsync(paths.UserSourcesFile, "{ not json ]", TestContext.Current.CancellationToken); + + using var store = Create(); + + store.List().ShouldBeEmpty(); + } +} diff --git a/tests/AvParser.Infrastructure.Tests/JsonSettingsServiceTests.cs b/tests/AvParser.Infrastructure.Tests/JsonSettingsServiceTests.cs index 255a1a1..683fbbf 100644 --- a/tests/AvParser.Infrastructure.Tests/JsonSettingsServiceTests.cs +++ b/tests/AvParser.Infrastructure.Tests/JsonSettingsServiceTests.cs @@ -156,10 +156,13 @@ public sealed class JsonSettingsServiceTests : IDisposable } [Fact] - public void The_proxy_gate_setting_reaches_the_collector() + public void The_collector_defaults_to_proxy_only() { - new AppSettings(AllowDirectConnection: false).ToCollectOptions().RequireProxy.ShouldBeTrue(); - new AppSettings(AllowDirectConnection: true).ToCollectOptions().RequireProxy.ShouldBeFalse(); + // Whether a source may go direct is that source's own setting now; what the app-wide + // options must never do is default to the permissive answer. + new AppSettings() + .ToCollectOptions() + .RequireProxy.ShouldBeTrue(); } [Fact] diff --git a/tests/AvParser.Infrastructure.Tests/ProxyPoolLoaderTests.cs b/tests/AvParser.Infrastructure.Tests/ProxyPoolLoaderTests.cs index 7045d07..2be82d1 100644 --- a/tests/AvParser.Infrastructure.Tests/ProxyPoolLoaderTests.cs +++ b/tests/AvParser.Infrastructure.Tests/ProxyPoolLoaderTests.cs @@ -92,6 +92,58 @@ public class ProxyPoolLoaderTests (await loader.EnsureLoadedAsync()).Total.ShouldBe(0); } + [Fact] + public async Task What_the_warm_up_skipped_is_checked_in_the_background() + { + // The warm-up stops at the target, which on a real feed leaves thousands unknown. Without + // this pass the pool reports ten live out of a few thousand and the rest were never asked. + var source = new CountingSource(); + var pool = new ProxyPool( + [source], + new AliveProbe(), + new ProxyOptions { MinimumLiveProxies = 1, ProbeConcurrency = 1 } + ); + using var loader = new ProxyPoolLoader(pool, new MemoryStateStore(), NullLogger.Instance); + + var result = await loader.EnsureLoadedAsync(); + + // One live was enough to finish starting up; the other is still unknown at this point. + result.Live.ShouldBe(1); + + await loader.TopUp.ShouldNotBeNull(); + + pool.LiveCount.ShouldBe(2); + loader.IsToppingUp.ShouldBeFalse(); + } + + [Fact] + public async Task The_background_check_is_skipped_when_the_user_asked_for_lazy_probing() + { + // Lazy means "check a proxy when you hand it out"; sweeping the list behind the user's back + // is exactly what they switched off. + var pool = new ProxyPool( + [new CountingSource()], + new AliveProbe(), + new ProxyOptions { HealthCheck = ProxyHealthCheck.Lazy } + ); + using var loader = new ProxyPoolLoader(pool, new MemoryStateStore(), NullLogger.Instance); + + await loader.EnsureLoadedAsync(); + + loader.TopUp.ShouldBeNull(); + } + + [Fact] + public async Task Disposing_twice_is_safe() + { + var loader = Build(out _, out _); + await loader.EnsureLoadedAsync(); + + loader.Dispose(); + + Should.NotThrow(loader.Dispose); + } + private sealed class MemoryStateStore : IProxyStateStore { public Dictionary State { get; } = new(StringComparer.Ordinal); @@ -150,4 +202,13 @@ public class ProxyPoolLoaderTests CancellationToken cancellationToken = default ) => Task.FromResult(ProxyProbeResult.Failure("not used")); } + + private sealed class AliveProbe : IProxyProbe + { + public Task ProbeAsync( + ProxyEndpoint endpoint, + ProxyOptions options, + CancellationToken cancellationToken = default + ) => Task.FromResult(ProxyProbeResult.Success(TimeSpan.FromMilliseconds(15))); + } } diff --git a/tests/AvParser.UI.HeadlessTests/CollectViewTests.cs b/tests/AvParser.UI.HeadlessTests/CollectViewTests.cs index 1337959..acc192b 100644 --- a/tests/AvParser.UI.HeadlessTests/CollectViewTests.cs +++ b/tests/AvParser.UI.HeadlessTests/CollectViewTests.cs @@ -1,41 +1,63 @@ -using System.Runtime.CompilerServices; using Avalonia.Controls; using Avalonia.Headless.XUnit; using Avalonia.Threading; using Avalonia.VisualTree; using AvParser.Core.Collecting; +using AvParser.Core.Collecting.Sources; using AvParser.Core.Parsing; using AvParser.Core.Proxies; using AvParser.Core.Settings; using AvParser.UI.ViewModels; using AvParser.UI.Views; using Microsoft.Extensions.Logging.Abstractions; +using ReactiveUI.Primitives; using ReactiveUI.Primitives.Concurrency; namespace AvParser.UI.HeadlessTests; public class CollectViewTests { - private sealed class StubSource(string id, string name, bool network) : IMediaSource + private sealed class FakeUserSourceStore(IEnumerable? seed = null) : IUserSourceStore { - public string Id => id; + private readonly List _configs = seed?.ToList() ?? []; - public string DisplayName => name; + public event EventHandler? Changed; - public string Description => "A source"; + public IReadOnlyList List() => [.. _configs]; - public bool RequiresNetwork => network; - - public bool CanParse(MediaQuery input) => true; - - public async IAsyncEnumerable> ParseAsync( - MediaQuery input, - IProgress? progress, - [EnumeratorCancellation] CancellationToken cancellationToken + public Task AddAsync( + PatternSourceConfig config, + CancellationToken cancellationToken = default ) { - await Task.Yield(); - yield break; + _configs.RemoveAll(c => c.Id == config.Id); + _configs.Add(config); + Changed?.Invoke(this, EventArgs.Empty); + return Task.FromResult(config); + } + + public Task UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default) + { + var index = _configs.FindIndex(c => c.Id == config.Id); + if (index < 0) + { + return Task.FromResult(false); + } + + _configs[index] = config; + Changed?.Invoke(this, EventArgs.Empty); + return Task.FromResult(true); + } + + public Task RemoveAsync(string id, CancellationToken cancellationToken = default) + { + var removed = _configs.RemoveAll(c => c.Id == id) > 0; + if (removed) + { + Changed?.Invoke(this, EventArgs.Empty); + } + + return Task.FromResult(removed); } } @@ -46,7 +68,7 @@ public class CollectViewTests MediaQuery query, CollectOptions options, IProgress? progress, - [EnumeratorCancellation] CancellationToken cancellationToken + [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken ) { await Task.Yield(); @@ -59,15 +81,34 @@ public class CollectViewTests public object? GetService(Type serviceType) => null; } - private static (CollectView View, CollectViewModel ViewModel, Window Window) ShowPage(bool networkSource) + private static PatternSourceConfig Config(string id = "s1", bool allowDirect = false) { - IMediaSource[] sources = networkSource - ? [new StubSource("url-list", "URL list", false), new StubSource("own-service", "Own service", true)] - : [new StubSource("url-list", "URL list", false)]; + PatternSourceConfig.TryCreate( + "Test", + "https://imgtest.example/test1/", + 6, + 8, + IdAlphabet.Alphanumeric, + null, + ".jpg", + allowDirect, + out var config, + id + ); + return config!; + } + + private static (CollectView View, CollectViewModel ViewModel, Window Window) ShowPage( + bool allowDirect, + bool withSource = true + ) + { + // Whether a run may go without a proxy is the source's setting now, not the app's. + var store = new FakeUserSourceStore(withSource ? [Config(allowDirect: allowDirect)] : []); var viewModel = new CollectViewModel( - new MediaSourceCatalog(sources, networkSource ? "own-service" : "url-list"), - new FakeSettingsService(new AppSettings { LastSourceId = networkSource ? "own-service" : "url-list" }), + new MediaSourceCatalog(store), + new FakeSettingsService(new AppSettings { LastSourceId = "s1" }), new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()), new IdleRunner(), new FakeMediaStore(), @@ -96,26 +137,26 @@ public class CollectViewTests [AvaloniaFact] public void The_page_renders() { - var (view, _, _) = ShowPage(networkSource: false); + var (view, _, _) = ShowPage(allowDirect: true); view.GetVisualDescendants().OfType().ShouldNotBeEmpty(); } [AvaloniaFact] - public void No_banner_is_shown_for_a_source_that_needs_no_network() + public void No_banner_is_shown_when_direct_connections_are_allowed() { - var (view, viewModel, _) = ShowPage(networkSource: false); + var (view, viewModel, _) = ShowPage(allowDirect: true); viewModel.IsBlockedWithoutProxy.ShouldBeFalse(); Banner(view).IsVisible.ShouldBeFalse(); } [AvaloniaFact] - public void A_blocked_network_source_puts_the_banner_on_screen() + public void A_blocked_source_puts_the_banner_on_screen() { // Rendered rather than asserted on the view model: an IsVisible binding that never fires // leaves the page silently unhelpful, which is exactly the failure this guards. - var (view, viewModel, _) = ShowPage(networkSource: true); + var (view, viewModel, _) = ShowPage(allowDirect: false); Dispatcher.UIThread.RunJobs(); viewModel.IsBlockedWithoutProxy.ShouldBeTrue(); @@ -125,7 +166,7 @@ public class CollectViewTests [AvaloniaFact] public void The_banner_offers_a_way_to_the_proxies_page() { - var (view, viewModel, _) = ShowPage(networkSource: true); + var (view, viewModel, _) = ShowPage(allowDirect: false); Dispatcher.UIThread.RunJobs(); var button = Banner(view).GetVisualDescendants().OfType