Refactor media source handling and update collection options
- Updated `IMediaSourceCatalog` to support user-added media sources, allowing dynamic editing and management of sources. - Removed the `UrlListSource` class as its functionality is now integrated into the new catalog structure. - Enhanced `CollectOptions` to default `RequireProxy` to true, ensuring stricter handling of proxy requirements. - Improved error handling in `ParseError` to include a `Subject` field for better context on failures. - Adjusted dependency injection to reflect changes in media source management, removing old source registrations. - Introduced background proxy checks to ensure a more robust proxy pool management during collection processes. These changes streamline the media collection process and improve the overall user experience by providing clearer error reporting and more flexible source management.
This commit is contained in:
@@ -1,73 +1,56 @@
|
|||||||
# CLAUDE.md
|
# CLAUDE.md
|
||||||
|
|
||||||
Конвенции этого репозитория и грабли, на которые здесь уже наступили. Читать до правок.
|
Конвенции репозитория. Читать до правок.
|
||||||
|
|
||||||
|
**Перед правкой подсистемы прочитать её файл — там инварианты, которые ломаются молча:**
|
||||||
|
|
||||||
|
| Правишь | Читай |
|
||||||
|
|---|---|
|
||||||
|
| `Core/Proxies/**`, `Infrastructure/Proxies/**` | [docs/proxies.md](docs/proxies.md) |
|
||||||
|
| `Collecting/**`, `Media/**`, `CollectViewModel`, `GalleryViewModel` | [docs/collecting.md](docs/collecting.md) |
|
||||||
|
| `.axaml`, стили, тестовые проекты, файлы сборки | [docs/avalonia.md](docs/avalonia.md) |
|
||||||
|
|
||||||
## Команды
|
## Команды
|
||||||
|
|
||||||
Полный локальный гейт (формат → сборка → тесты) — то, что нужно прогнать перед коммитом:
|
`./build.ps1` — полный гейт (формат → сборка → тесты), прогнать перед коммитом. Отдельные шаги:
|
||||||
|
`dotnet build AvParser.slnx -c Release`, `dotnet test AvParser.slnx -c Release`,
|
||||||
```bash
|
`dotnet csharpier check .`, `./run.ps1`. На Linux/macOS — `./build.sh` и `./run.sh`.
|
||||||
./build.ps1
|
|
||||||
```
|
|
||||||
|
|
||||||
Отдельные шаги, если нужен только один из них:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
dotnet build AvParser.slnx -c Release
|
|
||||||
```
|
|
||||||
```bash
|
|
||||||
dotnet test AvParser.slnx -c Release
|
|
||||||
```
|
|
||||||
```bash
|
|
||||||
dotnet csharpier check .
|
|
||||||
```
|
|
||||||
```bash
|
|
||||||
./run.ps1
|
|
||||||
```
|
|
||||||
|
|
||||||
На Linux/macOS — `./build.sh` и `./run.sh` с теми же шагами.
|
|
||||||
|
|
||||||
## Слои
|
## Слои
|
||||||
|
|
||||||
`Core ← Infrastructure ← UI ← Desktop`, строго в одну сторону.
|
`Core ← Infrastructure ← UI ← Desktop`, строго в одну сторону.
|
||||||
|
|
||||||
- **`AvParser.Core` не ссылается на Avalonia.** Это единственное ограничение, которое здесь
|
- **`Core` не ссылается на Avalonia** — домен должен запускаться из CLI, worker-а или бенчмарка. Дай
|
||||||
по-настоящему несущее: домен должен запускаться из CLI, worker-сервиса или бенчмарка. Как
|
ему Avalonia, и кто-нибудь потянется к `Dispatcher.UIThread` внутри источника.
|
||||||
только Avalonia станет доступна из домена, кто-нибудь потянется к `Dispatcher.UIThread` или
|
- **`UI` — библиотека, а не exe**: headless-тесты собирают настоящие View без `Program.cs`, Serilog и
|
||||||
`IStorageProvider` внутри источника.
|
контейнера.
|
||||||
- **`AvParser.UI` — библиотека, а не exe.** Headless-тесты собирают настоящие View, не
|
- **`Desktop` — тонкий composition root.** Логика туда не переезжает.
|
||||||
подтягивая `Program.cs`, Serilog и контейнер.
|
|
||||||
- **`AvParser.Desktop` — тонкий composition root.** Логика туда не переезжает.
|
|
||||||
|
|
||||||
## Добавить источник
|
## Добавить источник
|
||||||
|
|
||||||
1. Реализовать `IMediaSource` — в `Core/Collecting/Sources/`, если сети не нужно, иначе в
|
1. Реализовать `IMediaSource` — в `Core/Collecting/Sources/`, если сети не нужно, иначе в
|
||||||
`Infrastructure/Collecting/`.
|
`Infrastructure/Collecting/`.
|
||||||
2. Одна строка регистрации: `AddAvParserCore()` для доменного, `AddAvParserCollecting()` для
|
2. Одна строка регистрации: `AddAvParserCore()` для доменного, `AddAvParserCollecting()` для сетевого.
|
||||||
сетевого.
|
3. Ключи `Source.{id}.Name` и `Source.{id}.Description` в оба resx (без них отрисуется английский
|
||||||
3. Два ключа в оба resx: `Source.{id}.Name` и `Source.{id}.Description` (без них источник
|
текст самого класса, а не сломается).
|
||||||
отрисуется английским текстом из самого класса, а не сломается).
|
|
||||||
|
|
||||||
Всё. Каталог, страница «Сбор» и выпадающий список подхватят его сами.
|
Каталог, страница «Сбор» и список источников подхватят его сами.
|
||||||
|
|
||||||
**Источник ищет, а не качает.** Он отдаёт `MediaCandidate` — адрес плюс метаданные. Скачиванием,
|
- **Источник ищет, а не качает.** Он отдаёт `MediaCandidate`; скачивание, редиректы, тайм-ауты,
|
||||||
редиректами, тайм-аутами, сниффингом и троттлингом занимается `MediaFetcher`, один на всех. Источник,
|
сниффинг и троттлинг — на `MediaFetcher`, одном на всех.
|
||||||
который сам лезет за байтами, дублирует всё это и почти наверняка неправильно.
|
- **Сетевой обязан переопределить `RequiresNetwork => true`** — иначе поедет напрямую в обход гейта.
|
||||||
|
|
||||||
**Сетевой источник обязан переопределить `RequiresNetwork => true`** — иначе он не попадёт под гейт
|
|
||||||
и поедет напрямую в обход настройки.
|
|
||||||
|
|
||||||
## Добавить страницу
|
## Добавить страницу
|
||||||
|
|
||||||
1. Наследник `PageViewModel` в `UI/ViewModels/XxxViewModel.cs` (`Title`, `IconKey`).
|
1. Наследник `PageViewModel` в `UI/ViewModels/XxxViewModel.cs` (`TitleKey`, `IconKey`).
|
||||||
2. `UI/Views/XxxView.axaml` — имя обязано соответствовать конвенции `ViewLocator`:
|
2. `UI/Views/XxxView.axaml` — имя по конвенции `ViewLocator`: `...ViewModels.XxxViewModel` →
|
||||||
`...ViewModels.XxxViewModel` → `...Views.XxxView`.
|
`...Views.XxxView`.
|
||||||
3. Регистрация в `AddAvParserUI()`: конкретным типом **и** как `PageViewModel` — порядок этих
|
3. Регистрация в `AddAvParserUI()` конкретным типом **и** как `PageViewModel`; порядок этих
|
||||||
регистраций и есть порядок пунктов в рельсе навигации.
|
регистраций = порядок пунктов в рельсе навигации.
|
||||||
|
|
||||||
## ReactiveUI 24 (дистрибутив Primitives)
|
## ReactiveUI 24 (дистрибутив Primitives)
|
||||||
|
|
||||||
Это не классический ReactiveUI. `System.Reactive` не используется:
|
`System.Reactive` не используется:
|
||||||
|
|
||||||
| Классика | Здесь |
|
| Классика | Здесь |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -77,207 +60,49 @@ dotnet csharpier check .
|
|||||||
| `RxApp.MainThreadScheduler` | `RxSchedulers.MainThreadScheduler` |
|
| `RxApp.MainThreadScheduler` | `RxSchedulers.MainThreadScheduler` |
|
||||||
| `TestScheduler` | `VirtualClock`, `ImmediateSequencer.Instance` |
|
| `TestScheduler` | `VirtualClock`, `ImmediateSequencer.Instance` |
|
||||||
|
|
||||||
Привычные имена операторов (`Select`, `Where`, `Throttle`, `DistinctUntilChanged`,
|
Привычные операторы (`Select`, `Where`, `Throttle`, `DistinctUntilChanged`, `CombineLatest`)
|
||||||
`CombineLatest`) **работают** — Primitives отдаёт оба набора. `using ReactiveUI.Primitives;`
|
работают; `using ReactiveUI.Primitives;` нужен ради `Subscribe(Action<T>)`.
|
||||||
нужен ради `Subscribe(Action<T>)`.
|
|
||||||
|
|
||||||
**ReactiveUI 24 не инициализируется сама.** Первый `WhenAnyValue` бросит
|
**ReactiveUI 24 не инициализируется сама** — первый `WhenAnyValue` бросит
|
||||||
`InvalidOperationException`, пока не отработал builder. В приложении это делает
|
`InvalidOperationException`, пока не отработал builder: в приложении `AppBuilder.UseReactiveUI(...)`,
|
||||||
`AppBuilder.UseReactiveUI(...)`; в проекте VM-тестов — module initializer
|
в VM-тестах module initializer `ReactiveUiBootstrap`. Новый тестовый проект без Avalonia обязан
|
||||||
`ReactiveUiBootstrap`. Новый тестовый проект без Avalonia обязан сделать то же самое.
|
сделать то же самое.
|
||||||
|
|
||||||
## Конвенции ViewModel
|
## Конвенции ViewModel
|
||||||
|
|
||||||
- **Каждая VM принимает `ISequencer? mainThread = null`** и использует его в `outputScheduler:`
|
- **Каждая VM принимает `ISequencer? mainThread = null`** и использует его в `outputScheduler:` и
|
||||||
и `ToProperty(..., scheduler)`. Именно это делает тесты синхронными: они передают
|
`ToProperty(..., scheduler)`. Это и делает тесты синхронными — они передают `ImmediateSequencer`.
|
||||||
`ImmediateSequencer.Instance`. Без этого пришлось бы гонять диспетчер.
|
- **Регистрация такой VM в DI — явная фабрика**, не по типу: иначе выбор конструктора зависит от
|
||||||
- У VM с необязательным `ISequencer` регистрация в DI — явная фабрика, а не по типу: иначе
|
порядка регистраций.
|
||||||
выбор конструктора контейнером зависит от порядка регистраций.
|
|
||||||
- `[Reactive]` из `ReactiveUI.SourceGenerators` на partial-свойствах; класс — `partial`.
|
- `[Reactive]` из `ReactiveUI.SourceGenerators` на partial-свойствах; класс — `partial`.
|
||||||
|
- **VM с производным от языка текстом переопределяет `OnLanguageChanged`** и зовёт `base`, иначе
|
||||||
|
заголовок страницы застрянет на прежнем языке.
|
||||||
|
- **VM, подписанная на синглтон** (пул, каталог, настройки), — `IDisposable` и отписывается.
|
||||||
|
|
||||||
## Добавить источник прокси
|
## Строки UI
|
||||||
|
|
||||||
1. Реализовать `IProxySource` (или `IMutableProxySource`, если список редактируемый).
|
1. Ключ и **оба** перевода — в `Strings.resx` и `Strings.ru.resx`, иначе упадёт
|
||||||
2. Зарегистрировать как `IProxySource` в `AddAvParserProxies()`. Порядок регистрации = порядок
|
`LocalizationTests.Russian_translates_every_english_key`.
|
||||||
слияния; свой список идёт последним, чтобы пользовательский адрес перебивал фидовый.
|
2. В XAML `{l:Loc Ключ}`, во ViewModel `Localizer.Instance[...]` / `.Format(...)`.
|
||||||
|
|
||||||
`ProxyPool` при обновлении **переиспользует существующие `ProxyEntry`** по `Endpoint.Key` — иначе
|
|
||||||
перезагрузка списка стирала бы всю накопленную статистику, а публичные фиды переиздаются каждые
|
|
||||||
несколько минут.
|
|
||||||
|
|
||||||
## Инварианты прокси-пула
|
|
||||||
|
|
||||||
- **Доступность определяется карантином, а не `Health`.** `Health` — это «что видели в последний
|
|
||||||
раз». Если исключать всё, что когда-либо падало, окно карантина становится бессмысленным, а
|
|
||||||
прокси теряется навсегда после первой же осечки. Это уже был баг, его ловит
|
|
||||||
`A_failing_proxy_is_quarantined_and_comes_back_later`.
|
|
||||||
- **Проба не трогает `SuccessCount`/`FailureCount`.** Эти счётчики про реальные запросы; свип по
|
|
||||||
паре тысяч прокси перезаписал бы всё, на чём держится взвешенный выбор. Провалившаяся проба
|
|
||||||
выставляет карантин через `RecordProbe(..., quarantineOnFailure:)`.
|
|
||||||
- **Лиза без вердикта нейтральна.** Отменённая операция — не вина прокси; считать это отказом
|
|
||||||
значит карантинить здоровые прокси на каждый Cancel.
|
|
||||||
- **`Select` и `Next` — зарезервированные слова для CA1716.** Метод стратегии называется `Pick`.
|
|
||||||
- **`LiveCount` считает только `Alive` и не в карантине.** На нём висит гейт сбора, поэтому
|
|
||||||
«доступна» (карантин истёк) и «живая» здесь намеренно расходятся: гейт не должен открываться
|
|
||||||
от одного лишь истечения окна.
|
|
||||||
- **Прогрев обрывается по достижении цели, а не проходит список до конца.** `WarmUpAsync` линкует
|
|
||||||
CTS и гасит остаток, как только набралось `MinimumLiveProxies`. Порядок кандидатов —
|
|
||||||
`WarmUpOrder()`, он публичный ровно затем, чтобы порядок проверялся без прогона проб.
|
|
||||||
- **`RestoreState` не выставляет `Health = Alive`.** «Работала вчера» живёт в отдельном
|
|
||||||
`WasAliveOnLastRun` и влияет только на порядок прогрева. Если восстанавливать как `Alive`, пул
|
|
||||||
отрапортует живыми тех, с кем не разговаривал: прогрев сочтёт цель достигнутой и не проверит
|
|
||||||
никого, а гейт сбора откроется по данным недельной давности. Ловит
|
|
||||||
`A_remembered_proxy_is_not_reported_live_until_it_answers_again`.
|
|
||||||
- **`RestoreState` не восстанавливает карантин.** Окно — стенные часы, между запусками могли пройти
|
|
||||||
сутки; перенос окна сажал бы прокси за то, что давно истекло.
|
|
||||||
- **Сохраняются только `HasEverAnswered`** = `SuccessCount > 0 || IsBelievedAlive`, где
|
|
||||||
«believed» = вердикт этой сессии, а без него — прошлой. Мёртвые в фиде исчисляются тысячами и
|
|
||||||
переиздаются каждые пять минут. Важен именно перенос: прогрев обрывается рано, поэтому
|
|
||||||
большинство запомненных заканчивают сессию непроверенными — строгое `Health == Alive` стирало бы
|
|
||||||
накопленный список за пару запусков.
|
|
||||||
|
|
||||||
## Гейт сбора
|
|
||||||
|
|
||||||
`IParser.RequiresNetwork` — дефолтная реализация возвращает `false`, поэтому добавление источника
|
|
||||||
остаётся однострочным.
|
|
||||||
|
|
||||||
Гейт живёт в `CollectViewModel.RefreshProxyGate()` и складывается из трёх условий: источник сетевой,
|
|
||||||
`AllowDirectConnection` выключен, `LiveCount == 0`. Он пересчитывается по событию пула (с
|
|
||||||
throttle 250 мс — пул дёргается на каждый исход лизы), при смене источника и при смене настроек.
|
|
||||||
|
|
||||||
Второй экземпляр того же правила — в `FetchOptions.RequireProxy`: гейт закрывает кнопку, а фетчер
|
|
||||||
бросает `ProxyUnavailableException` вместо тихого прямого запроса. Одного UI мало — запрос ушёл бы
|
|
||||||
с адреса пользователя ровно тогда, когда он просил этого не делать.
|
|
||||||
|
|
||||||
`CollectView.axaml` держит баннер под `x:Name="ProxyGateBanner"`; `CollectViewTests` рендерит его
|
|
||||||
по-настоящему, потому что мёртвый биндинг `IsVisible` не ломает ни одного VM-теста.
|
|
||||||
|
|
||||||
## Инварианты хранилища медиа
|
|
||||||
|
|
||||||
- **В `blobs/` попадает только дочитанное.** Загрузка идёт во временный файл в соседнем каталоге на
|
|
||||||
том же томе и продвигается переименованием. Обрыв оставляет `.part`, который подметает следующий
|
|
||||||
старт, а не обрезанную картинку, неотличимую от настоящей навсегда.
|
|
||||||
- **Тип — по сигнатуре, никогда по URL, расширению или `Content-Type`.** Два из трёх выбирает тот,
|
|
||||||
кто отдаёт файл, и расширение на диске у пользователя не должно зависеть от чужого сервера.
|
|
||||||
- **`GIF89a` не доказывает анимацию**, и APNG не определяется по фиксированному префиксу: нужен обход
|
|
||||||
блоков (второй Image Descriptor) и чанков (`acTL` раньше первого `IDAT`). Тихая ошибка, поэтому
|
|
||||||
обходчики изолированы за `internal static` швами и проверяются на массивах байтов.
|
|
||||||
- **`ref_count` денормализован и пересчитывается, а не инкрементится.** Апсерт `item` может заменить
|
|
||||||
строку, указывавшую на другой blob, и слепой `+1` уехал бы навсегда. Есть
|
|
||||||
`VerifyReferenceCountsAsync`, и он часть замысла, а не отладка.
|
|
||||||
- **Журнал `seen_url` переживает чистку.** Иначе следующий прогон скачает заново ровно то, что
|
|
||||||
пользователь только что удалил. Терминальные исходы отделены от повторяемых: отказ описывает
|
|
||||||
момент, а не ресурс, и считать его окончательным значит терять контент на каждой сетевой икоте.
|
|
||||||
- **Вердикт лизы — про транспорт, а не про ресурс.** 404 и 429 — это успех прокси. Иначе пул
|
|
||||||
карантинил бы рабочие адреса ровно с той частотой, с какой встречаются мёртвые ссылки, а на
|
|
||||||
лимит отвечал бы сменой прокси, то есть обходом лимита.
|
|
||||||
- **Троттл поднимается только сигналами хоста** (429/503 с `Retry-After`) и никогда не приводит к
|
|
||||||
ротации прокси. Флажка «повторить через другую прокси при 429» в настройках быть не должно.
|
|
||||||
- **Жёсткая ссылка — привилегия ФС, а не гарантия.** Откат на копию удваивает расход диска, поэтому
|
|
||||||
достигнутый режим пишется в `showcase_mode` и виден в UI.
|
|
||||||
- **Имя из `SuggestedName` враждебно.** Остаётся только последний сегмент, разделители не
|
|
||||||
переживают, устройства Windows отодвигаются, расширение берётся из типа.
|
|
||||||
|
|
||||||
## Добавить строку в UI
|
|
||||||
|
|
||||||
1. Ключ и оба перевода — в `Strings.resx` и `Strings.ru.resx` (**оба**, иначе упадёт
|
|
||||||
`LocalizationTests.Russian_translates_every_english_key`).
|
|
||||||
2. В XAML — `{l:Loc Ключ}`, во ViewModel — `Localizer.Instance[...]` / `.Format(...)`.
|
|
||||||
|
|
||||||
Никакого хардкода в `Views/` кроме имени продукта и примеров адресов.
|
Никакого хардкода в `Views/` кроме имени продукта и примеров адресов.
|
||||||
|
|
||||||
- **Счётчики — только через `Localizer.Plural`** с ключами `.One` / `.Few` / `.Many`. У русского
|
- **Счётчики — только через `Localizer.Plural`** с ключами `.One` / `.Few` / `.Many`: у русского три
|
||||||
три формы; «{0} records» с приклеенным окончанием непереводимо.
|
формы, «{0} records» с приклеенным окончанием непереводимо.
|
||||||
- **Перечисления в списках — `LocalizedOption<T>`**, не сырые значения. Конвертер разрешил бы
|
- **Перечисления в списках — `LocalizedOption<T>`**, не сырые значения: конвертер разрешил бы подпись
|
||||||
подпись один раз и не заметил смены языка. Идентичность обёртки — значение перечисления, чтобы
|
один раз и не заметил смены языка. Идентичность обёртки — значение перечисления, чтобы выбор не
|
||||||
выбор не слетал.
|
слетал.
|
||||||
- **Текст из домена переводится по коду.** `Core` о языках не знает: `ParseError` несёт `Code` и
|
- **Текст из домена переводится по коду.** `ParseError` несёт `Code` и `Arguments`, UI ищет
|
||||||
`Arguments`, UI ищет `Collect.Error.{Code}` с откатом на `Message`. Имена источников — так же:
|
`Collect.Error.{Code}` с откатом на `Message`; имена источников — `Source.{id}.Name` с откатом на
|
||||||
`Source.{id}.Name` с откатом на `DisplayName`, поэтому обещание «добавить источник = одна строка»
|
`DisplayName`, поэтому «добавить источник = одна строка» остаётся в силе.
|
||||||
остаётся в силе.
|
|
||||||
- **VM, у которой есть производный от языка текст, переопределяет `OnLanguageChanged`** и зовёт
|
|
||||||
`base`. Без этого заголовок страницы останется на прежнем языке.
|
|
||||||
|
|
||||||
## Грабли, уже оплаченные
|
|
||||||
|
|
||||||
- **Селектор типа в Avalonia матчит точный тип.** `UserControl.shell` не матчит `ShellView`
|
|
||||||
(наследник `ReactiveUserControl<T>`) и молча не делает ничего. Использовать
|
|
||||||
`:is(UserControl).shell`. Голый `.shell` тоже матчит, но тогда XAML-компилятор не может
|
|
||||||
вывести тип для `Setter` и падает с AVLN2200.
|
|
||||||
- **Style-сеттер навсегда проигрывает локальному значению.** Не стилизовать `IsPaneOpen` и
|
|
||||||
`DisplayMode` — они биндятся во ViewModel.
|
|
||||||
- **`IPseudoClasses.Set` требует ведущего `:`**.
|
|
||||||
- **`InvariantGlobalization` обязан быть `false`**: Semi.Avalonia строит `CultureInfo` в
|
|
||||||
статическом конструкторе и падает целиком.
|
|
||||||
- **Превьюер рефлексирует безпараметровый статический `BuildAvaloniaApp()`.** Необязательный
|
|
||||||
параметр ломает его вызов, вторая перегрузка — `AmbiguousMatchException`.
|
|
||||||
- **Compiled bindings включены по умолчанию** (Avalonia 12): `x:DataType` нужен на каждом
|
|
||||||
`UserControl` и каждом `DataTemplate`.
|
|
||||||
- **`Avalonia.Headless.XUnit` 12.x — это xUnit v3**, а не v2. Тестовые проекты — `Exe`.
|
|
||||||
- **Headless: ручной `Measure`/`Arrange` внутри окна бесполезен** — следующий проход layout
|
|
||||||
окна вернёт свой размер. Задавать ширину самому `Window`. Но и без окна нельзя: у
|
|
||||||
открепленного контрола не строится визуальное дерево.
|
|
||||||
- **csharpier — единственный владелец форматирования** (включая `.axaml` и `.csproj`).
|
|
||||||
`IDE0055` понижен до suggestion: два форматтера с `TreatWarningsAsErrors` дерутся насмерть.
|
|
||||||
- **`Condition="'$(Configuration)' == 'Debug'"` на `PackageReference` — ловушка.**
|
|
||||||
`dotnet restore` вычисляется с конфигурацией по умолчанию, поэтому пакет обязан
|
|
||||||
резолвиться, даже если эту конфигурацию никто не собирает. Так тут проехал мёртвый
|
|
||||||
`Avalonia.Diagnostics` (его нет под Avalonia 12): `dotnet build -c Release` работал,
|
|
||||||
а голый `dotnet restore` падал.
|
|
||||||
- **`Execute()` завершился ≠ `IsExecuting` уже false.** Второе публикуется на выходном
|
|
||||||
планировщике. Тест, который сразу после `await` дёргает команду, закрытую по чужому
|
|
||||||
`IsExecuting`, будет мигать под нагрузкой — ждите `CanExecute`, а не предполагайте.
|
|
||||||
- **Проект VM-тестов не параллелится.** `ReactiveUiBootstrap` ставит глобальные планировщики
|
|
||||||
ReactiveUI, то есть тесты делят изменяемое состояние независимо от их желания.
|
|
||||||
- **Инспектора в Avalonia 12 нет из коробки.** `Avalonia.Diagnostics` закончился на 11.3.x;
|
|
||||||
DevTools живут отдельно (`AvaloniaUI.DiagnosticsSupport` + `.WithDeveloperTools()`), со своей
|
|
||||||
установкой. Зависимость намеренно не добавлена.
|
|
||||||
|
|
||||||
## Качество
|
|
||||||
|
|
||||||
- Центральные версии пакетов — `Directory.Packages.props`. **Никаких `Version=` в csproj**
|
|
||||||
(иначе NU1008 на restore).
|
|
||||||
- `TreatWarningsAsErrors` включён; NuGet-advisory (`NU19xx`) выведены из ошибок, чтобы
|
|
||||||
свежая CVE не роняла сборку кода, который никто не трогал.
|
|
||||||
- Тестовые послабления анализаторов — в `tests/Directory.Build.props`, не в самих тестах.
|
|
||||||
|
|
||||||
## Пути
|
|
||||||
|
|
||||||
- **Медиа лежит рядом с exe, настройки — в профиле.** Это осознанное расхождение: конфигурация
|
|
||||||
пользовательская, а коллекция принадлежит установке и переезжает вместе с папкой.
|
|
||||||
- **`DefaultMediaDirectory()` проверяет запись пробным файлом**, а не только созданием каталога:
|
|
||||||
каталог создаётся и там, куда потом нельзя записать. При отказе — откат в профиль.
|
|
||||||
- **Смена каталога требует перезапуска.** `AppPaths` строится до контейнера (медиа-корень читается
|
|
||||||
из settings.json напрямую), так что применить его на лету нельзя без переподключения индекса,
|
|
||||||
blob-хранилища и кэша миниатюр. Настройки честно это говорят и показывают действующий путь.
|
|
||||||
- **Смена каталога не переносит файлы.** Перенос архива — отдельная операция с другой ценой ошибки.
|
|
||||||
|
|
||||||
## Отображение медиа
|
|
||||||
|
|
||||||
- **Миниатюры декодируются в нужную ширину**, а не декодируются целиком и потом масштабируются.
|
|
||||||
На архиве это разница между «работает» и «кончилась память».
|
|
||||||
- **Кэш владеет своими `Bitmap` и удаляет их при вытеснении**, поэтому вызывающий не должен их
|
|
||||||
освобождать — и поэтому ёмкость кэша обязана заметно превышать страницу галереи: вытесненная
|
|
||||||
картинка, которая ещё на экране, освободилась бы под рендерером.
|
|
||||||
- **Видео не декодируется**, и попытка была бы исключением на каждой плитке. `MediaKinds.IsImage`
|
|
||||||
отсекает это до всякого обращения к диску.
|
|
||||||
- **`{DynamicResource}` с несуществующим ключом молча не срабатывает** — свойство остаётся со
|
|
||||||
значением по умолчанию, фон не красится, кисть прозрачная. Это уже дважды доезжало до
|
|
||||||
скриншота; ловит `ResourceKeyTests` в обеих темах.
|
|
||||||
- **`{l:Loc}` с ключом, которого нет в обоих resx, тест паритета не поймает** — файлы согласованы
|
|
||||||
между собой. Ловит `LocalizationCoverageTests`.
|
|
||||||
|
|
||||||
## Границы, выбранные намеренно
|
## Границы, выбранные намеренно
|
||||||
|
|
||||||
- **robots.txt не читается.** Оба источника v1 либо принадлежат пользователю, либо введены им
|
- **robots.txt не читается.** Источники либо принадлежат пользователю, либо введены им вручную. Это
|
||||||
вручную, поэтому спрашивать разрешения не у кого. Это решение, а не забывчивость: **с появлением
|
решение, а не забывчивость: **с появлением источника, который ходит по чужому сайту, robots.txt
|
||||||
третьего источника, который ходит по чужому сайту, robots.txt становится обязательным.**
|
становится обязательным.**
|
||||||
- **Перебора идентификаторов нет и не планируется.** Источник перечисляет то, что сайт сам
|
- **Возобновления по `Range` нет**: частичный файл в хранилище дороже повторного скачивания.
|
||||||
публикует; подбор адресов — это не «сбор опубликованного», и прокси-пул существует ради лимитов
|
|
||||||
и доступности, а не ради их обхода.
|
|
||||||
- **Возобновления по `Range` нет.** Оборванная загрузка выбрасывается целиком; частичный файл в
|
|
||||||
хранилище дороже, чем повторное скачивание.
|
|
||||||
- **SVG не поддерживается сознательно** — это текст, он умеет исполнять скрипты и несёт XXE.
|
- **SVG не поддерживается сознательно** — это текст, он умеет исполнять скрипты и несёт XXE.
|
||||||
BMP/ICO/HEIC/JPEG-XL просто отложены.
|
BMP/ICO/HEIC/JPEG-XL просто отложены.
|
||||||
- **Перцептивных хешей нет.** Дедуп точный, по SHA-256; «похожие» картинки — отдельная задача.
|
- **Перцептивных хешей нет**: дедуп точный, по SHA-256.
|
||||||
- **Кадры из видео не извлекаются.** Это FFmpeg целиком ради одной картинки в плитке.
|
- **Кадры из видео не извлекаются** — это FFmpeg целиком ради одной картинки в плитке.
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# Avalonia, тесты и сборка: уже оплаченные грабли
|
||||||
|
|
||||||
|
Читать перед правкой `.axaml`, стилей, тестовых проектов и файлов сборки.
|
||||||
|
|
||||||
|
## XAML и стили
|
||||||
|
|
||||||
|
- **Селектор типа матчит точный тип.** `UserControl.shell` не матчит `ShellView` (наследник
|
||||||
|
`ReactiveUserControl<T>`) и молча не делает ничего — нужен `:is(UserControl).shell`. Голый `.shell`
|
||||||
|
тоже матчит, но тогда XAML-компилятор не выводит тип для `Setter` и падает с AVLN2200.
|
||||||
|
- **Style-сеттер навсегда проигрывает локальному значению.** Не стилизовать `IsPaneOpen` и
|
||||||
|
`DisplayMode` — они биндятся во ViewModel.
|
||||||
|
- **`IPseudoClasses.Set` требует ведущего `:`.**
|
||||||
|
- **Compiled bindings включены по умолчанию** (Avalonia 12): `x:DataType` нужен на каждом
|
||||||
|
`UserControl` и каждом `DataTemplate`.
|
||||||
|
- **`{DynamicResource}` с несуществующим ключом молча не срабатывает** — свойство остаётся со
|
||||||
|
значением по умолчанию, фон не красится, кисть прозрачная. Дважды доезжало до скриншота; ловит
|
||||||
|
`ResourceKeyTests` в обеих темах.
|
||||||
|
- **`{l:Loc}` с ключом, которого нет в обоих resx, тест паритета не поймает** — файлы согласованы
|
||||||
|
между собой. Ловит `LocalizationCoverageTests`.
|
||||||
|
|
||||||
|
## Приложение
|
||||||
|
|
||||||
|
- **`InvariantGlobalization` обязан быть `false`**: Semi.Avalonia строит `CultureInfo` в статическом
|
||||||
|
конструкторе и падает целиком.
|
||||||
|
- **Превьюер рефлексирует безпараметровый статический `BuildAvaloniaApp()`**: необязательный параметр
|
||||||
|
ломает его вызов, вторая перегрузка — `AmbiguousMatchException`.
|
||||||
|
- **Инспектора в Avalonia 12 нет из коробки.** `Avalonia.Diagnostics` закончился на 11.3.x; DevTools
|
||||||
|
живут отдельно (`AvaloniaUI.DiagnosticsSupport` + `.WithDeveloperTools()`), со своей установкой.
|
||||||
|
Зависимость намеренно не добавлена.
|
||||||
|
|
||||||
|
## Тесты
|
||||||
|
|
||||||
|
- **`Avalonia.Headless.XUnit` 12.x — это xUnit v3**, а не v2. Тестовые проекты — `Exe`.
|
||||||
|
- **Headless: ручной `Measure`/`Arrange` внутри окна бесполезен** — следующий проход layout окна
|
||||||
|
вернёт свой размер. Задавать ширину самому `Window`. Но и без окна нельзя: у открепленного контрола
|
||||||
|
не строится визуальное дерево.
|
||||||
|
- **`Execute()` завершился ≠ `IsExecuting` уже false.** Второе публикуется на выходном планировщике.
|
||||||
|
Тест, который сразу после `await` дёргает команду, закрытую по чужому `IsExecuting`, будет мигать
|
||||||
|
под нагрузкой — ждите `CanExecute`, а не предполагайте.
|
||||||
|
- **Проект VM-тестов не параллелится**: `ReactiveUiBootstrap` ставит глобальные планировщики
|
||||||
|
ReactiveUI, то есть тесты делят изменяемое состояние независимо от их желания.
|
||||||
|
|
||||||
|
## Композиция
|
||||||
|
|
||||||
|
- **`Dispose()` страницы обязан переживать второй вызов.** Каждая страница зарегистрирована дважды —
|
||||||
|
своим типом и как `PageViewModel`, — и контейнер учитывает её как disposable по обеим регистрациям,
|
||||||
|
то есть зовёт `Dispose()` два раза. `GalleryViewModel` отменял на втором вызове уже освобождённый
|
||||||
|
`CancellationTokenSource`, и приложение падало с `ObjectDisposedException` при каждом выходе. Ловят
|
||||||
|
тесты `Disposing_twice_is_safe`.
|
||||||
|
|
||||||
|
## Сборка
|
||||||
|
|
||||||
|
- **csharpier — единственный владелец форматирования** (включая `.axaml` и `.csproj`). `IDE0055`
|
||||||
|
понижен до suggestion: два форматтера с `TreatWarningsAsErrors` дерутся насмерть.
|
||||||
|
- **`Condition="'$(Configuration)' == 'Debug'"` на `PackageReference` — ловушка.** `dotnet restore`
|
||||||
|
вычисляется с конфигурацией по умолчанию, поэтому пакет обязан резолвиться, даже если эту
|
||||||
|
конфигурацию никто не собирает. Так тут проехал мёртвый `Avalonia.Diagnostics`:
|
||||||
|
`dotnet build -c Release` работал, а голый `dotnet restore` падал.
|
||||||
|
- Версии пакетов централизованы в `Directory.Packages.props` — **никаких `Version=` в csproj**
|
||||||
|
(иначе NU1008 на restore).
|
||||||
|
- `TreatWarningsAsErrors` включён; NuGet-advisory (`NU19xx`) выведены из ошибок, чтобы свежая CVE не
|
||||||
|
роняла сборку кода, который никто не трогал.
|
||||||
|
- Тестовые послабления анализаторов — в `tests/Directory.Build.props`, не в самих тестах.
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# Сбор и хранилище
|
||||||
|
|
||||||
|
Читать перед правкой `Collecting/**`, `Media/**`, `CollectViewModel` и `GalleryViewModel`.
|
||||||
|
|
||||||
|
## Гейт сбора
|
||||||
|
|
||||||
|
`CollectViewModel.RefreshProxyGate()`: **хотя бы один отмеченный** источник требует прокси
|
||||||
|
(`MediaSourceViewModel.NeedsProxy` = сетевой и без своего разрешения на прямое подключение) и
|
||||||
|
`LiveCount == 0`. Пересчитывается по событию пула (throttle 250 мс — пул дёргается на каждый исход
|
||||||
|
лизы) и при смене отметок. Подсветка в списке для этого не годится: она про то, что редактируют, а не
|
||||||
|
про то, что запускают.
|
||||||
|
|
||||||
|
- **Разрешение работать без прокси — настройка источника, а не приложения**
|
||||||
|
(`PatternSourceConfig.AllowDirectConnection`). Один хост может быть своим, где прокси бессмысленна,
|
||||||
|
а соседний — чужим, где прямое подключение это ровно то, чего пользователь избегал; общий тумблер
|
||||||
|
навязывал обоим разрешающий ответ. По умолчанию `false` — туда же приезжает старый
|
||||||
|
`sources.user.json`.
|
||||||
|
- **Один закрытый источник в прогоне закрывает весь прогон.** Разрешающий не может поручиться за
|
||||||
|
строгого: запрос, которого пользователь не хотел, всё равно ушёл бы с его адреса.
|
||||||
|
- **Второй экземпляр того же правила — `FetchOptions.RequireProxy`**, и он строится на источник в
|
||||||
|
`ProduceAsync`. Гейт гасит кнопку, а фетчер бросает `ProxyUnavailableException`; одного UI мало.
|
||||||
|
`CollectOptions.RequireProxy` по умолчанию `true`: молчание вызывающего — не разрешение.
|
||||||
|
- Баннер живёт под `x:Name="ProxyGateBanner"` и рендерится по-настоящему в `CollectViewTests`, потому
|
||||||
|
что мёртвый биндинг `IsVisible` не ломает ни одного VM-теста.
|
||||||
|
|
||||||
|
## Прогон по нескольким источникам
|
||||||
|
|
||||||
|
- **Отметка ≠ подсветка.** `MediaSourceViewModel.IsSelected` решает, что войдёт в прогон; выделение в
|
||||||
|
списке — что правят, удаляют и чистят. Отмеченный набор живёт в `AppSettings.CollectSourceIds`
|
||||||
|
строкой, а не списком: список сломал бы сравнение записи, и каждое сохранение выглядело бы
|
||||||
|
изменением.
|
||||||
|
- **Источники идут одновременно, а не по очереди.** Это не про пропускную способность: при нулевом
|
||||||
|
бюджете первый источник не заканчивается никогда, так что последовательный прогон был бы прогоном
|
||||||
|
по одному источнику со списком в руках. Слияние — `CollectViewModel.MergeAsync`; канал ограничен, а
|
||||||
|
`finally` обязан погасить продюсеров — оставить их писать в канал, который никто не читает, значит
|
||||||
|
подвесить их посреди записи в хранилище.
|
||||||
|
- **Глобальный лимит закачек делится на число источников, а не умножается**: каждый `CollectRunner`
|
||||||
|
поднимает своих воркеров, и пять источников по четыре — это двадцать соединений вместо четырёх.
|
||||||
|
- **`Limit = 0` — «пока не остановят».** В безлимитном режиме `PatternMediaSource` сбрасывает
|
||||||
|
множество виденных id по `SeenCapacity`: у прогона нет конца, значит и у множества не должно быть
|
||||||
|
роста. Настоящий дедуп держит журнал `seen_url`, а не оно.
|
||||||
|
|
||||||
|
## Источники
|
||||||
|
|
||||||
|
- **Пустое расширение — это `.jpg`, а не «без суффикса».** В редакторе это плейсхолдер, и
|
||||||
|
`PatternSourceConfig.NormaliseExtension` подставляет его в домене, потому что через `TryCreate`
|
||||||
|
проходят и форма, и загрузка `sources.user.json`. Голый `/{id}` почти всегда опечатка, которая
|
||||||
|
стоит целого прогона из 404.
|
||||||
|
|
||||||
|
## Живой журнал
|
||||||
|
|
||||||
|
- **Одна хронологическая лента вместо «результаты + ошибки»**: при переборе id почти всё промахи, и
|
||||||
|
смотрят на порядок, а не на две таблицы.
|
||||||
|
- **Пакетов мало, нужен ещё и тик.** `FlushInterval` (200 мс) существует потому, что на медленном
|
||||||
|
источнике буфер не добирает до `BatchSize` и страница выглядит зависшей. Буферы —
|
||||||
|
`ConcurrentQueue`, слив под `_flushGate`, иначе две гонки-выгрузки перемешают строки местами.
|
||||||
|
- **Строка хранит ключ и аргументы, а не готовое предложение** — смена языка посреди прогона иначе
|
||||||
|
оставит половину журнала по-английски. Литеральная половина (адрес, размер) не переводится никогда.
|
||||||
|
- **Адрес неудачи живёт в `ParseError.Subject`.** В `Message` его нет и быть не может: у шаблона
|
||||||
|
перевода фиксированные подстановки. Без него журнал говорит «сайт ответил 404» и не говорит, на
|
||||||
|
каком из десяти тысяч id.
|
||||||
|
- **Журнал ограничен `MaxLogEntries`** — у безлимитного прогона нет конца, а несрезанный список это
|
||||||
|
утечка памяти со скроллбаром.
|
||||||
|
- **Строки копируются**: множественное выделение, Ctrl+C и контекстное меню; текст строит
|
||||||
|
`CollectLogEntryViewModel.ToString()`, чтобы в буфер попало ровно то, что на экране, а не
|
||||||
|
повторный рендер, который тихо разъедется с шаблоном. Ctrl+C без выделения копирует весь журнал.
|
||||||
|
В Avalonia 12 `SetTextAsync` — расширение из `Avalonia.Input.Platform`, а не член `IClipboard`.
|
||||||
|
|
||||||
|
## Хранилище медиа
|
||||||
|
|
||||||
|
- **В `blobs/` попадает только дочитанное.** Загрузка идёт во временный файл в соседнем каталоге на
|
||||||
|
том же томе и продвигается переименованием. Обрыв оставляет `.part`, который подметает следующий
|
||||||
|
старт, а не обрезанную картинку, навсегда неотличимую от настоящей.
|
||||||
|
- **Тип — по сигнатуре, никогда по URL, расширению или `Content-Type`**: два из трёх выбирает тот,
|
||||||
|
кто отдаёт файл, и расширение на диске у пользователя не должно зависеть от чужого сервера.
|
||||||
|
- **`GIF89a` не доказывает анимацию**, и APNG не определяется по фиксированному префиксу: нужен обход
|
||||||
|
блоков (второй Image Descriptor) и чанков (`acTL` раньше первого `IDAT`). Ошибка тихая, поэтому
|
||||||
|
обходчики изолированы за `internal static` швами и проверяются на массивах байтов.
|
||||||
|
- **`ref_count` денормализован и пересчитывается, а не инкрементится**: апсерт `item` может заменить
|
||||||
|
строку, указывавшую на другой blob, и слепой `+1` уехал бы навсегда. `VerifyReferenceCountsAsync` —
|
||||||
|
часть замысла, а не отладка.
|
||||||
|
- **Журнал `seen_url` переживает чистку**, иначе следующий прогон скачает заново ровно то, что
|
||||||
|
пользователь только что удалил. Терминальные исходы отделены от повторяемых: отказ описывает
|
||||||
|
момент, а не ресурс, и считать его окончательным значит терять контент на каждой сетевой икоте.
|
||||||
|
- **Троттл поднимается только сигналами хоста** (429/503 с `Retry-After`) и никогда не приводит к
|
||||||
|
ротации прокси. Флажка «повторить через другую прокси при 429» в настройках быть не должно.
|
||||||
|
- **Жёсткая ссылка — привилегия ФС, а не гарантия.** Откат на копию удваивает расход диска, поэтому
|
||||||
|
достигнутый режим пишется в `showcase_mode` и виден в UI.
|
||||||
|
- **Имя из `SuggestedName` враждебно**: остаётся только последний сегмент, разделители не переживают,
|
||||||
|
устройства Windows отодвигаются, расширение берётся из типа.
|
||||||
|
|
||||||
|
## Пути
|
||||||
|
|
||||||
|
- **Медиа лежит рядом с exe, настройки — в профиле.** Расхождение осознанное: конфигурация
|
||||||
|
пользовательская, а коллекция принадлежит установке и переезжает вместе с папкой.
|
||||||
|
- **`DefaultMediaDirectory()` проверяет запись пробным файлом**, а не только созданием каталога:
|
||||||
|
каталог создаётся и там, куда потом нельзя писать. При отказе — откат в профиль.
|
||||||
|
- **Смена каталога требует перезапуска и не переносит файлы.** `AppPaths` строится до контейнера
|
||||||
|
(медиа-корень читается из settings.json напрямую), так что на лету это не применить без
|
||||||
|
переподключения индекса, blob-хранилища и кэша миниатюр. Настройки честно это говорят и показывают
|
||||||
|
действующий путь.
|
||||||
|
|
||||||
|
## Отображение медиа
|
||||||
|
|
||||||
|
- **Миниатюры декодируются сразу в нужную ширину**, а не декодируются целиком и потом масштабируются.
|
||||||
|
На архиве это разница между «работает» и «кончилась память».
|
||||||
|
- **Кэш владеет своими `Bitmap` и удаляет их при вытеснении**, поэтому вызывающий не должен их
|
||||||
|
освобождать — и поэтому ёмкость кэша обязана заметно превышать страницу галереи: вытесненная
|
||||||
|
картинка, которая ещё на экране, освободилась бы под рендерером.
|
||||||
|
- **Видео не декодируется**, попытка была бы исключением на каждой плитке: `MediaKinds.IsImage`
|
||||||
|
отсекает это до всякого обращения к диску.
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# Прокси-пул
|
||||||
|
|
||||||
|
Читать перед правкой `Core/Proxies/**` и `Infrastructure/Proxies/**`.
|
||||||
|
|
||||||
|
## Добавить источник
|
||||||
|
|
||||||
|
1. Реализовать `IProxySource` (или `IMutableProxySource`, если список редактируемый).
|
||||||
|
2. Зарегистрировать в `AddAvParserProxies()`. Порядок регистрации = порядок слияния; свой список идёт
|
||||||
|
последним, чтобы пользовательский адрес перебивал фидовый.
|
||||||
|
|
||||||
|
## Инварианты
|
||||||
|
|
||||||
|
- **`ProxyPool` переиспользует существующие `ProxyEntry` по `Endpoint.Key`** — иначе перезагрузка
|
||||||
|
списка стирала бы статистику, а публичные фиды переиздаются каждые несколько минут.
|
||||||
|
- **Доступность определяется карантином, а не `Health`.** `Health` — «что видели в последний раз»;
|
||||||
|
исключать всё когда-либо упавшее значит потерять прокси после первой осечки. Уже было багом, ловит
|
||||||
|
`A_failing_proxy_is_quarantined_and_comes_back_later`.
|
||||||
|
- **`LiveCount` считает только `Alive` и не в карантине.** На нём висит гейт сбора, поэтому
|
||||||
|
«доступна» (окно истекло) и «живая» намеренно расходятся: гейт не должен открываться от одного лишь
|
||||||
|
истечения окна.
|
||||||
|
- **Проба не трогает `SuccessCount`/`FailureCount`** — они про реальные запросы, а свип по паре тысяч
|
||||||
|
адресов перезаписал бы всё, на чём держится взвешенный выбор. Провал пробы даёт карантин через
|
||||||
|
`RecordProbe(..., quarantineOnFailure:)`.
|
||||||
|
- **Лиза без вердикта нейтральна**: отменённая операция — не вина прокси, иначе каждый Cancel
|
||||||
|
карантинил бы здоровые адреса.
|
||||||
|
- **Вердикт лизы — про транспорт, а не про ресурс.** 404 и 429 — успех прокси; иначе пул карантинил
|
||||||
|
бы рабочие адреса с частотой мёртвых ссылок, а на лимит отвечал бы сменой прокси, то есть обходом
|
||||||
|
лимита.
|
||||||
|
- **Прогрев обрывается по достижении цели**, а не проходит список до конца: `WarmUpAsync` линкует CTS
|
||||||
|
и гасит остаток на `MinimumLiveProxies`. Порядок кандидатов — `WarmUpOrder()`, публичный ровно ради
|
||||||
|
проверки без прогона проб.
|
||||||
|
- **Остаток дочищает `TopUpAsync` в фоне.** Прогрев по построению оставляет тысячи адресов в
|
||||||
|
`Unknown` — это читается как «проверка не запускалась», и, что хуже, сбор получает пул из десяти
|
||||||
|
прокси вместо сотен рабочих. Фоновый проход запускает `ProxyPoolLoader` после прогрева, с
|
||||||
|
четвертью `ProbeConcurrency`: пользовательский свип — это всплеск, за которым смотрят, а этот идёт
|
||||||
|
минутами и, возможно, параллельно с закачкой.
|
||||||
|
- **`TopUpAsync` трогает только `Unknown`.** Перепроверять мёртвых — это и есть свип; перепроверять
|
||||||
|
живых значит посадить в карантин прокси, которой прямо сейчас пользуется прогон.
|
||||||
|
- **`TopUpAsync` не делает `_strategy.Reset()`**, в отличие от свипа: сброс уронил бы липкий выбор
|
||||||
|
посреди сбора, причём ради записей, которых в этом выборе и нет. `Changed` он поднимает раз в
|
||||||
|
`ChangeInterval` проб — иначе страница молчит несколько минут.
|
||||||
|
- **Фоновый проход не идёт при `ProxyHealthCheck.Lazy`**: «проверять в момент выдачи» — это ровно
|
||||||
|
просьба не свипать.
|
||||||
|
- **`SaveStateAsync` сериализован семафором.** Писателей теперь двое (фоновый проход по завершении и
|
||||||
|
выход из приложения), а запись идёт через один и тот же временный файл с переименованием.
|
||||||
|
- **`RestoreState` не выставляет `Health = Alive`** — «работала вчера» живёт в `WasAliveOnLastRun` и
|
||||||
|
влияет только на порядок прогрева. Иначе пул отрапортует живыми тех, с кем не разговаривал: прогрев
|
||||||
|
сочтёт цель достигнутой, а гейт сбора откроется по данным недельной давности. Ловит
|
||||||
|
`A_remembered_proxy_is_not_reported_live_until_it_answers_again`.
|
||||||
|
- **`RestoreState` не восстанавливает карантин**: окно — стенные часы, между запусками могли пройти
|
||||||
|
сутки, и перенос сажал бы прокси за давно истёкшее.
|
||||||
|
- **Сохраняются только `HasEverAnswered`** = `SuccessCount > 0 || IsBelievedAlive` («believed» —
|
||||||
|
вердикт этой сессии, а без него прошлой). Прогрев обрывается рано, поэтому большинство запомненных
|
||||||
|
заканчивают сессию непроверенными — строгое `Health == Alive` стёрло бы накопленный список за пару
|
||||||
|
запусков.
|
||||||
|
- **`Select` и `Next` зарезервированы для CA1716** — метод стратегии называется `Pick`.
|
||||||
|
|
||||||
|
## Статистика на «Обзоре»
|
||||||
|
|
||||||
|
`DashboardViewModel` держит сводку пула и обновляет её по событию `IProxyPool.Changed` с throttle
|
||||||
|
250 мс — событие приходит на каждый исход лизы и на каждую пробу. «Живых» на карточке — это
|
||||||
|
`LiveCount`, то же число, что читает гейт сбора; `RefreshProxyStats()` публичен, чтобы тесты не ждали
|
||||||
|
throttle.
|
||||||
@@ -125,7 +125,12 @@ public sealed record CollectOptions
|
|||||||
public string UserAgent { get; init; } = "AvParser/0.1";
|
public string UserAgent { get; init; } = "AvParser/0.1";
|
||||||
|
|
||||||
/// <summary>Whether a missing proxy is a hard failure rather than a direct connection.</summary>
|
/// <summary>Whether a missing proxy is a hard failure rather than a direct connection.</summary>
|
||||||
public bool RequireProxy { get; init; }
|
/// <remarks>
|
||||||
|
/// Defaults to the strict end: a caller that has not said otherwise has not asked to be seen
|
||||||
|
/// from their own address. The collect page sets it per source, since one host may be the user's
|
||||||
|
/// own service and the next a stranger's.
|
||||||
|
/// </remarks>
|
||||||
|
public bool RequireProxy { get; init; } = true;
|
||||||
|
|
||||||
/// <summary>How browsable copies point at their blobs.</summary>
|
/// <summary>How browsable copies point at their blobs.</summary>
|
||||||
public ShowcaseMode ShowcaseMode { get; init; } = ShowcaseMode.HardLink;
|
public ShowcaseMode ShowcaseMode { get; init; } = ShowcaseMode.HardLink;
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
using AvParser.Core.Parsing;
|
using AvParser.Core.Parsing;
|
||||||
|
|
||||||
namespace AvParser.Core.Collecting;
|
namespace AvParser.Core.Collecting;
|
||||||
@@ -12,64 +13,130 @@ namespace AvParser.Core.Collecting;
|
|||||||
/// </remarks>
|
/// </remarks>
|
||||||
public interface IMediaSource : IParser<MediaQuery, MediaCandidate>;
|
public interface IMediaSource : IParser<MediaQuery, MediaCandidate>;
|
||||||
|
|
||||||
/// <summary>Read-only view over every registered media source.</summary>
|
/// <summary>Live view over the user's media sources, with editing.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// No longer immutable: sources are user-added data now, so the list changes at runtime. Editing is
|
||||||
|
/// delegated to an <see cref="IUserSourceStore"/>; the catalog re-materialises a
|
||||||
|
/// <see cref="PatternMediaSource"/> per stored config and raises <see cref="Changed"/> so the UI can
|
||||||
|
/// rebuild. An empty catalog is a valid state — the collector shows an "add a source" prompt rather
|
||||||
|
/// than failing to start.
|
||||||
|
/// </remarks>
|
||||||
public interface IMediaSourceCatalog
|
public interface IMediaSourceCatalog
|
||||||
{
|
{
|
||||||
/// <summary>All registered sources, ordered by <see cref="IParser{TInput,TOutput}.DisplayName"/>.</summary>
|
/// <summary>Every source, ordered by <see cref="IParser{TInput,TOutput}.DisplayName"/>.</summary>
|
||||||
IReadOnlyList<IMediaSource> Sources { get; }
|
IReadOnlyList<IMediaSource> Sources { get; }
|
||||||
|
|
||||||
/// <summary>The source used when nothing has been chosen yet.</summary>
|
/// <summary>The stored configs behind the sources, for editing.</summary>
|
||||||
IMediaSource DefaultSource { get; }
|
IReadOnlyList<PatternSourceConfig> Configs { get; }
|
||||||
|
|
||||||
/// <summary>Finds a source by its stable id; <see langword="null"/> when unknown.</summary>
|
/// <summary>Finds a source by its stable id; <see langword="null"/> when unknown.</summary>
|
||||||
IMediaSource? Find(string? id);
|
IMediaSource? Find(string? id);
|
||||||
|
|
||||||
/// <summary>Finds a source by id, falling back to <see cref="DefaultSource"/>.</summary>
|
/// <summary>Adds a source, returning the stored config with its assigned id.</summary>
|
||||||
IMediaSource FindOrDefault(string? id) => Find(id) ?? DefaultSource;
|
Task<PatternSourceConfig> AddAsync(PatternSourceConfig config, CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
/// <summary>Replaces the source with the same id. Returns whether one was present.</summary>
|
||||||
|
Task<bool> UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
/// <summary>Removes a source by id. Returns whether it was present.</summary>
|
||||||
|
Task<bool> RemoveAsync(string id, CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
/// <summary>Raised after the source list changes.</summary>
|
||||||
|
event EventHandler? Changed;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc cref="IMediaSourceCatalog" />
|
/// <inheritdoc cref="IMediaSourceCatalog" />
|
||||||
public sealed class MediaSourceCatalog : IMediaSourceCatalog
|
public sealed class MediaSourceCatalog : IMediaSourceCatalog, IDisposable
|
||||||
{
|
{
|
||||||
private readonly Dictionary<string, IMediaSource> _byId;
|
private readonly IUserSourceStore _store;
|
||||||
|
private readonly object _gate = new();
|
||||||
|
|
||||||
/// <summary>Builds a catalog from every source the container resolved.</summary>
|
private Dictionary<string, IMediaSource> _byId = new(StringComparer.OrdinalIgnoreCase);
|
||||||
/// <param name="sources">Registered sources.</param>
|
private IReadOnlyList<IMediaSource> _sources = [];
|
||||||
/// <param name="defaultId">
|
|
||||||
/// Which source to land on. Named explicitly rather than taken as "first alphabetically",
|
/// <summary>Builds a catalog that tracks the store.</summary>
|
||||||
/// because that would make the landing page depend on a display name — and would put a network
|
/// <param name="store">Where the user's source configs live.</param>
|
||||||
/// source there, so the app would open behind the proxy gate for no reason.
|
public MediaSourceCatalog(IUserSourceStore store)
|
||||||
/// </param>
|
|
||||||
/// <exception cref="ArgumentException">No sources were registered, or two share an id.</exception>
|
|
||||||
public MediaSourceCatalog(IEnumerable<IMediaSource> sources, string? defaultId = null)
|
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(sources);
|
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||||
|
|
||||||
Sources = [.. sources.OrderBy(source => source.DisplayName, StringComparer.OrdinalIgnoreCase)];
|
Rebuild();
|
||||||
|
_store.Changed += OnStoreChanged;
|
||||||
if (Sources.Count == 0)
|
|
||||||
{
|
|
||||||
throw new ArgumentException("At least one media source must be registered.", nameof(sources));
|
|
||||||
}
|
|
||||||
|
|
||||||
_byId = new Dictionary<string, IMediaSource>(StringComparer.OrdinalIgnoreCase);
|
|
||||||
foreach (var source in Sources)
|
|
||||||
{
|
|
||||||
if (!_byId.TryAdd(source.Id, source))
|
|
||||||
{
|
|
||||||
throw new ArgumentException($"Duplicate media source id '{source.Id}'.", nameof(sources));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
DefaultSource = Find(defaultId) ?? Sources[0];
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public IReadOnlyList<IMediaSource> Sources { get; }
|
public event EventHandler? Changed;
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public IMediaSource DefaultSource { get; }
|
public IReadOnlyList<IMediaSource> Sources
|
||||||
|
{
|
||||||
|
get
|
||||||
|
{
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
return _sources;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public IMediaSource? Find(string? id) => id is not null && _byId.TryGetValue(id, out var source) ? source : null;
|
public IReadOnlyList<PatternSourceConfig> Configs => _store.List();
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public IMediaSource? Find(string? id)
|
||||||
|
{
|
||||||
|
if (id is null)
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
return _byId.TryGetValue(id, out var source) ? source : null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<PatternSourceConfig> AddAsync(
|
||||||
|
PatternSourceConfig config,
|
||||||
|
CancellationToken cancellationToken = default
|
||||||
|
) => _store.AddAsync(config, cancellationToken);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<bool> UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default) =>
|
||||||
|
_store.UpdateAsync(config, cancellationToken);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public Task<bool> RemoveAsync(string id, CancellationToken cancellationToken = default) =>
|
||||||
|
_store.RemoveAsync(id, cancellationToken);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public void Dispose() => _store.Changed -= OnStoreChanged;
|
||||||
|
|
||||||
|
private void OnStoreChanged(object? sender, EventArgs e)
|
||||||
|
{
|
||||||
|
Rebuild();
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void Rebuild()
|
||||||
|
{
|
||||||
|
var sources = _store
|
||||||
|
.List()
|
||||||
|
.Select(config => (IMediaSource)new PatternMediaSource(config))
|
||||||
|
.OrderBy(source => source.DisplayName, StringComparer.OrdinalIgnoreCase)
|
||||||
|
.ToArray();
|
||||||
|
|
||||||
|
var byId = new Dictionary<string, IMediaSource>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
foreach (var source in sources)
|
||||||
|
{
|
||||||
|
byId[source.Id] = source;
|
||||||
|
}
|
||||||
|
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
_sources = sources;
|
||||||
|
_byId = byId;
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
|
|
||||||
|
namespace AvParser.Core.Collecting;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The user's own list of pattern sources, persisted between runs.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The collector equivalent of <c>IMutableProxySource</c>: sources are now data the user edits, not
|
||||||
|
/// types the build ships, so they live in a file and this is the seam that reads and writes it. The
|
||||||
|
/// interface is in the domain and the persistence in infrastructure, so the catalog can depend on it
|
||||||
|
/// without the domain learning about the filesystem.
|
||||||
|
/// </remarks>
|
||||||
|
public interface IUserSourceStore
|
||||||
|
{
|
||||||
|
/// <summary>Every stored config, in insertion order. Implementations may cache.</summary>
|
||||||
|
IReadOnlyList<PatternSourceConfig> List();
|
||||||
|
|
||||||
|
/// <summary>Adds a config. Returns the stored config, whose id is assigned if it had none.</summary>
|
||||||
|
Task<PatternSourceConfig> AddAsync(PatternSourceConfig config, CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
/// <summary>Replaces the config with the same id. Returns whether one was present to replace.</summary>
|
||||||
|
Task<bool> UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
/// <summary>Removes the config with this id. Returns whether it was present.</summary>
|
||||||
|
Task<bool> RemoveAsync(string id, CancellationToken cancellationToken = default);
|
||||||
|
|
||||||
|
/// <summary>Raised after the list changes, so a catalog can rebuild its materialised sources.</summary>
|
||||||
|
event EventHandler? Changed;
|
||||||
|
}
|
||||||
@@ -0,0 +1,202 @@
|
|||||||
|
using System.Runtime.CompilerServices;
|
||||||
|
using AvParser.Core.Parsing;
|
||||||
|
|
||||||
|
namespace AvParser.Core.Collecting.Sources;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Generates candidate ids from a <see cref="PatternSourceConfig"/> and hangs them off its base URL.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// It only produces addresses — the fetcher does every byte of the downloading, redirect-following
|
||||||
|
/// and sniffing. Unlike a pasted list, this one hits a remote host on purpose
|
||||||
|
/// and in bulk, so <see cref="RequiresNetwork"/> is <see langword="true"/>: the proxy gate exists for
|
||||||
|
/// exactly this. Which of the generated ids actually resolve is the fetcher's verdict, not the
|
||||||
|
/// source's; the source's only job is to hand out well-formed addresses within the pattern.
|
||||||
|
/// </remarks>
|
||||||
|
public sealed class PatternMediaSource : IMediaSource
|
||||||
|
{
|
||||||
|
/// <summary>How many candidates to emit each time the enumerator yields control.</summary>
|
||||||
|
private const int YieldInterval = 256;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Above this the id space is treated as effectively unbounded, so no exhaustion cap is applied.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Below it the space is small enough that a naive "generate until the budget is met" loop would
|
||||||
|
/// spin forever on duplicate ids once most of the space is drawn, so the budget is clamped to the
|
||||||
|
/// space size instead.
|
||||||
|
/// </remarks>
|
||||||
|
private const long SmallSpaceThreshold = 5_000_000;
|
||||||
|
|
||||||
|
/// <summary>How many ids an unbounded run remembers before it starts over.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// An uncapped run has no end, so the de-duplication set would grow until the process died. Past
|
||||||
|
/// a million draws from a space this large the chance of redrawing an id is negligible, and a
|
||||||
|
/// redraw is not a correctness problem anyway: the store's <c>seen_url</c> journal is what keeps
|
||||||
|
/// an address from being fetched twice. Only ever applied to unbounded runs — a bounded one over
|
||||||
|
/// a small space needs the full set to honour "never exceed the space".
|
||||||
|
/// </remarks>
|
||||||
|
private const int SeenCapacity = 1 << 20;
|
||||||
|
|
||||||
|
private readonly PatternSourceConfig _config;
|
||||||
|
private readonly string _alphabet;
|
||||||
|
|
||||||
|
/// <summary>Wraps a config as a runnable source.</summary>
|
||||||
|
public PatternMediaSource(PatternSourceConfig config)
|
||||||
|
{
|
||||||
|
_config = config ?? throw new ArgumentNullException(nameof(config));
|
||||||
|
_alphabet = config.AlphabetCharacters;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>The config this source runs.</summary>
|
||||||
|
public PatternSourceConfig Config => _config;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public string Id => _config.Id;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public string DisplayName => _config.Name;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public string Description => $"{_config.BaseUrl} — id {_config.MinLength}–{_config.MaxLength} from '{_alphabet}'";
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public bool RequiresNetwork => true;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public bool CanParse(MediaQuery input) => input is not null && _alphabet.Length > 0;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async IAsyncEnumerable<ParseOutcome<MediaCandidate>> ParseAsync(
|
||||||
|
MediaQuery input,
|
||||||
|
IProgress<ParseProgress>? progress,
|
||||||
|
[EnumeratorCancellation] CancellationToken cancellationToken
|
||||||
|
)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(input);
|
||||||
|
|
||||||
|
if (_alphabet.Length == 0)
|
||||||
|
{
|
||||||
|
yield break;
|
||||||
|
}
|
||||||
|
|
||||||
|
var budget = Budget(input.Limit);
|
||||||
|
var unbounded = budget == 0;
|
||||||
|
var seen = new HashSet<string>(Math.Min(unbounded ? SeenCapacity : budget, 1 << 16));
|
||||||
|
var emitted = 0;
|
||||||
|
|
||||||
|
// A run of collisions this long means the space is all but drawn; stop rather than spin. The
|
||||||
|
// multiplier keeps a false trip vanishingly unlikely while the space still has room.
|
||||||
|
var stale = 0;
|
||||||
|
var staleLimit = Math.Max(1_000, _alphabet.Length * 64);
|
||||||
|
|
||||||
|
while (unbounded || emitted < budget)
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
if (unbounded && seen.Count >= SeenCapacity)
|
||||||
|
{
|
||||||
|
seen.Clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
var id = NextId();
|
||||||
|
if (!seen.Add(id))
|
||||||
|
{
|
||||||
|
if (++stale >= staleLimit)
|
||||||
|
{
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
stale = 0;
|
||||||
|
|
||||||
|
yield return ParseOutcome<MediaCandidate>.Success(
|
||||||
|
new MediaCandidate(new Uri(_config.BaseUrl, id + _config.Extension))
|
||||||
|
{
|
||||||
|
SourceId = Id,
|
||||||
|
ExternalId = id,
|
||||||
|
Ordinal = emitted + 1,
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
emitted++;
|
||||||
|
|
||||||
|
if (emitted % YieldInterval == 0)
|
||||||
|
{
|
||||||
|
progress?.Report(new ParseProgress(emitted, budget));
|
||||||
|
await Task.Yield();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
progress?.Report(new ParseProgress(emitted, budget));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The number of candidates to generate: the caller's budget, capped at the id space when small.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Returns <c>0</c> for "keep going until the caller stops asking", which is what a limit of zero
|
||||||
|
/// means. That can only happen over a large space — a small one is always capped at its own size,
|
||||||
|
/// because past that point every further draw is a collision and the run would spin rather than
|
||||||
|
/// collect.
|
||||||
|
/// </remarks>
|
||||||
|
private int Budget(int limit)
|
||||||
|
{
|
||||||
|
var space = SpaceSize();
|
||||||
|
|
||||||
|
if (space >= SmallSpaceThreshold)
|
||||||
|
{
|
||||||
|
return Math.Max(0, limit);
|
||||||
|
}
|
||||||
|
|
||||||
|
return limit > 0 ? (int)Math.Min(limit, space) : (int)space;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Size of the id space, saturating at <see cref="SmallSpaceThreshold"/> so large alphabets do
|
||||||
|
/// not overflow — the exact value past the threshold is never needed.
|
||||||
|
/// </summary>
|
||||||
|
private long SpaceSize()
|
||||||
|
{
|
||||||
|
long total = 0;
|
||||||
|
long power = 1;
|
||||||
|
|
||||||
|
for (var length = 1; length <= _config.MaxLength; length++)
|
||||||
|
{
|
||||||
|
power = Math.Min(power * _alphabet.Length, SmallSpaceThreshold);
|
||||||
|
|
||||||
|
if (length >= _config.MinLength)
|
||||||
|
{
|
||||||
|
total += power;
|
||||||
|
if (total >= SmallSpaceThreshold)
|
||||||
|
{
|
||||||
|
return SmallSpaceThreshold;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return total;
|
||||||
|
}
|
||||||
|
|
||||||
|
private string NextId()
|
||||||
|
{
|
||||||
|
var length =
|
||||||
|
_config.MinLength == _config.MaxLength
|
||||||
|
? _config.MinLength
|
||||||
|
: Random.Shared.Next(_config.MinLength, _config.MaxLength + 1);
|
||||||
|
|
||||||
|
return string.Create(
|
||||||
|
length,
|
||||||
|
_alphabet,
|
||||||
|
static (span, alphabet) =>
|
||||||
|
{
|
||||||
|
for (var i = 0; i < span.Length; i++)
|
||||||
|
{
|
||||||
|
span[i] = alphabet[Random.Shared.Next(alphabet.Length)];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,258 @@
|
|||||||
|
namespace AvParser.Core.Collecting.Sources;
|
||||||
|
|
||||||
|
/// <summary>Which characters an id is drawn from.</summary>
|
||||||
|
public enum IdAlphabet
|
||||||
|
{
|
||||||
|
/// <summary>Lower-case latin letters, <c>a</c>–<c>z</c>.</summary>
|
||||||
|
LettersLower = 0,
|
||||||
|
|
||||||
|
/// <summary>Upper-case latin letters, <c>A</c>–<c>Z</c>.</summary>
|
||||||
|
LettersUpper = 1,
|
||||||
|
|
||||||
|
/// <summary>Both cases of latin letters.</summary>
|
||||||
|
Letters = 2,
|
||||||
|
|
||||||
|
/// <summary>Decimal digits, <c>0</c>–<c>9</c>.</summary>
|
||||||
|
Digits = 3,
|
||||||
|
|
||||||
|
/// <summary>Letters of both cases and digits.</summary>
|
||||||
|
Alphanumeric = 4,
|
||||||
|
|
||||||
|
/// <summary>Lower-case hexadecimal, <c>0</c>–<c>9</c> and <c>a</c>–<c>f</c>.</summary>
|
||||||
|
HexLower = 5,
|
||||||
|
|
||||||
|
/// <summary>Whatever characters the user typed in <see cref="PatternSourceConfig.CustomAlphabet"/>.</summary>
|
||||||
|
Custom = 6,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Turns an <see cref="IdAlphabet"/> into the concrete set of characters to draw from.</summary>
|
||||||
|
public static class PatternAlphabet
|
||||||
|
{
|
||||||
|
private const string Lower = "abcdefghijklmnopqrstuvwxyz";
|
||||||
|
private const string Upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
|
||||||
|
private const string Digit = "0123456789";
|
||||||
|
private const string Hex = "0123456789abcdef";
|
||||||
|
|
||||||
|
/// <summary>Resolves the character set, de-duplicated, for a config.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Custom alphabets are de-duplicated so a pasted "aabbcc" does not skew the random draw toward
|
||||||
|
/// the repeated characters. An empty result means the config is unusable and is rejected by
|
||||||
|
/// <see cref="PatternSourceConfig.TryCreate"/> before it ever reaches a source.
|
||||||
|
/// </remarks>
|
||||||
|
public static string Resolve(IdAlphabet alphabet, string? customAlphabet = null) =>
|
||||||
|
alphabet switch
|
||||||
|
{
|
||||||
|
IdAlphabet.LettersLower => Lower,
|
||||||
|
IdAlphabet.LettersUpper => Upper,
|
||||||
|
IdAlphabet.Letters => Lower + Upper,
|
||||||
|
IdAlphabet.Digits => Digit,
|
||||||
|
IdAlphabet.Alphanumeric => Lower + Upper + Digit,
|
||||||
|
IdAlphabet.HexLower => Hex,
|
||||||
|
IdAlphabet.Custom => Distinct(customAlphabet),
|
||||||
|
_ => string.Empty,
|
||||||
|
};
|
||||||
|
|
||||||
|
private static string Distinct(string? text)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrEmpty(text))
|
||||||
|
{
|
||||||
|
return string.Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
var seen = new HashSet<char>(text.Length);
|
||||||
|
var builder = new System.Text.StringBuilder(text.Length);
|
||||||
|
|
||||||
|
foreach (var c in text)
|
||||||
|
{
|
||||||
|
// A separator inside an id would break the "one path segment" assumption the base URL
|
||||||
|
// relies on, so control and obvious URL-structural characters never make the alphabet.
|
||||||
|
if (char.IsControl(c) || c is '/' or '\\' or '?' or '#' or ' ')
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (seen.Add(c))
|
||||||
|
{
|
||||||
|
builder.Append(c);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return builder.ToString();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Why a <see cref="PatternSourceConfig"/> could not be built.</summary>
|
||||||
|
public enum PatternConfigError
|
||||||
|
{
|
||||||
|
/// <summary>The config is valid.</summary>
|
||||||
|
None = 0,
|
||||||
|
|
||||||
|
/// <summary>The display name was blank.</summary>
|
||||||
|
NameRequired,
|
||||||
|
|
||||||
|
/// <summary>The base URL was blank, relative, or not http/https.</summary>
|
||||||
|
BaseUrlInvalid,
|
||||||
|
|
||||||
|
/// <summary>The length range was outside 1..64 or had min greater than max.</summary>
|
||||||
|
LengthRangeInvalid,
|
||||||
|
|
||||||
|
/// <summary>The alphabet resolved to no usable characters.</summary>
|
||||||
|
AlphabetEmpty,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A user-added source that generates candidate ids from a pattern and hangs them off a base URL.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// This is the whole "a source is data, not a class" idea: one <see cref="PatternMediaSource"/>
|
||||||
|
/// implementation is parameterised by this record, so adding a source is adding a row to a file
|
||||||
|
/// rather than shipping a build. The record is what <c>sources.user.json</c> persists.
|
||||||
|
/// </remarks>
|
||||||
|
/// <param name="Id">Stable identifier; scopes storage and purge, and is remembered as the last choice.</param>
|
||||||
|
/// <param name="Name">User-given display name.</param>
|
||||||
|
/// <param name="BaseUrl">Absolute http/https address ending in <c>/</c>; ids resolve against it.</param>
|
||||||
|
/// <param name="MinLength">Shortest id to generate, inclusive.</param>
|
||||||
|
/// <param name="MaxLength">Longest id to generate, inclusive.</param>
|
||||||
|
/// <param name="Alphabet">Which characters ids are drawn from.</param>
|
||||||
|
/// <param name="CustomAlphabet">Characters used when <paramref name="Alphabet"/> is <see cref="IdAlphabet.Custom"/>.</param>
|
||||||
|
/// <param name="Extension">
|
||||||
|
/// Suffix appended after the id. A blank one from the editor becomes <see cref="DefaultExtension"/>;
|
||||||
|
/// a config built by hand may still carry <see langword="null"/> for none.
|
||||||
|
/// </param>
|
||||||
|
/// <param name="AllowDirectConnection">
|
||||||
|
/// Whether this source may run from the user's own address when no proxy is live. Per source rather
|
||||||
|
/// than per app: one host may be the user's own service, where a proxy is pointless, while the next
|
||||||
|
/// is a stranger's, where going direct is exactly what the user was avoiding — a global switch forced
|
||||||
|
/// the permissive answer on both. Defaults to <see langword="false"/>, which is both the safe end and
|
||||||
|
/// where an older <c>sources.user.json</c> lands.
|
||||||
|
/// </param>
|
||||||
|
public sealed record PatternSourceConfig(
|
||||||
|
string Id,
|
||||||
|
string Name,
|
||||||
|
Uri BaseUrl,
|
||||||
|
int MinLength,
|
||||||
|
int MaxLength,
|
||||||
|
IdAlphabet Alphabet,
|
||||||
|
string? CustomAlphabet,
|
||||||
|
string? Extension,
|
||||||
|
bool AllowDirectConnection = false
|
||||||
|
)
|
||||||
|
{
|
||||||
|
/// <summary>Hard ceiling on id length, guarding both the UI spinner and a hand-edited file.</summary>
|
||||||
|
public const int MaxIdLength = 64;
|
||||||
|
|
||||||
|
/// <summary>Suffix used when the editor's extension field is left empty.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// A blank field means "the usual one", not "no suffix": every host worth pointing this at
|
||||||
|
/// serves <c>/{id}.jpg</c>, and a bare <c>/{id}</c> is almost always a typo that costs a whole
|
||||||
|
/// run of 404s to discover. The editor shows this as the placeholder and the saved config keeps
|
||||||
|
/// it verbatim, so the field stops being blank once saved.
|
||||||
|
/// </remarks>
|
||||||
|
public const string DefaultExtension = ".jpg";
|
||||||
|
|
||||||
|
/// <summary>The characters this config draws ids from.</summary>
|
||||||
|
public string AlphabetCharacters => PatternAlphabet.Resolve(Alphabet, CustomAlphabet);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Validates and normalises the inputs, generating an id when one is not supplied.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The single gate every config passes through, used by both the editor (for inline feedback)
|
||||||
|
/// and the store (so a hand-edited file cannot smuggle in an unusable source). Normalises the
|
||||||
|
/// base URL to a trailing slash and the extension to a leading dot, because both are joined by
|
||||||
|
/// naive concatenation downstream.
|
||||||
|
/// </remarks>
|
||||||
|
public static PatternConfigError TryCreate(
|
||||||
|
string? name,
|
||||||
|
string? baseUrl,
|
||||||
|
int minLength,
|
||||||
|
int maxLength,
|
||||||
|
IdAlphabet alphabet,
|
||||||
|
string? customAlphabet,
|
||||||
|
string? extension,
|
||||||
|
bool allowDirectConnection,
|
||||||
|
out PatternSourceConfig? config,
|
||||||
|
string? id = null
|
||||||
|
)
|
||||||
|
{
|
||||||
|
config = null;
|
||||||
|
|
||||||
|
var trimmedName = name?.Trim() ?? string.Empty;
|
||||||
|
if (trimmedName.Length == 0)
|
||||||
|
{
|
||||||
|
return PatternConfigError.NameRequired;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
string.IsNullOrWhiteSpace(baseUrl)
|
||||||
|
|| !Uri.TryCreate(baseUrl.Trim(), UriKind.Absolute, out var parsed)
|
||||||
|
|| parsed.Scheme is not ("http" or "https")
|
||||||
|
)
|
||||||
|
{
|
||||||
|
return PatternConfigError.BaseUrlInvalid;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (minLength < 1 || maxLength > MaxIdLength || minLength > maxLength)
|
||||||
|
{
|
||||||
|
return PatternConfigError.LengthRangeInvalid;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (PatternAlphabet.Resolve(alphabet, customAlphabet).Length == 0)
|
||||||
|
{
|
||||||
|
return PatternConfigError.AlphabetEmpty;
|
||||||
|
}
|
||||||
|
|
||||||
|
config = new PatternSourceConfig(
|
||||||
|
string.IsNullOrWhiteSpace(id) ? NewId(trimmedName) : id!.Trim(),
|
||||||
|
trimmedName,
|
||||||
|
EnsureTrailingSlash(parsed),
|
||||||
|
minLength,
|
||||||
|
maxLength,
|
||||||
|
alphabet,
|
||||||
|
alphabet == IdAlphabet.Custom ? customAlphabet : null,
|
||||||
|
NormaliseExtension(extension),
|
||||||
|
allowDirectConnection
|
||||||
|
);
|
||||||
|
|
||||||
|
return PatternConfigError.None;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static Uri EnsureTrailingSlash(Uri url)
|
||||||
|
{
|
||||||
|
if (url.AbsoluteUri.EndsWith('/'))
|
||||||
|
{
|
||||||
|
return url;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A base URL without a trailing slash drops its last segment when a relative id is resolved
|
||||||
|
// against it, so "…/test1" would collect from "…/2df.jpg", not "…/test1/2df.jpg".
|
||||||
|
return new Uri(url.AbsoluteUri + "/", UriKind.Absolute);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string NormaliseExtension(string? extension)
|
||||||
|
{
|
||||||
|
var trimmed = extension?.Trim();
|
||||||
|
if (string.IsNullOrEmpty(trimmed))
|
||||||
|
{
|
||||||
|
return DefaultExtension;
|
||||||
|
}
|
||||||
|
|
||||||
|
return trimmed.StartsWith('.') ? trimmed : "." + trimmed;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string NewId(string name)
|
||||||
|
{
|
||||||
|
var slug = new System.Text.StringBuilder(name.Length);
|
||||||
|
foreach (var c in name.ToLowerInvariant())
|
||||||
|
{
|
||||||
|
slug.Append(char.IsLetterOrDigit(c) ? c : '-');
|
||||||
|
}
|
||||||
|
|
||||||
|
var trimmed = slug.ToString().Trim('-');
|
||||||
|
var prefix = trimmed.Length == 0 ? "source" : trimmed;
|
||||||
|
|
||||||
|
// A short random suffix keeps ids unique when two sources share a name, without leaking a
|
||||||
|
// full GUID into settings and logs.
|
||||||
|
return $"{prefix}-{Guid.NewGuid().ToString("N").AsSpan(0, 6)}";
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,122 +0,0 @@
|
|||||||
using System.Runtime.CompilerServices;
|
|
||||||
using AvParser.Core.Parsing;
|
|
||||||
|
|
||||||
namespace AvParser.Core.Collecting.Sources;
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// Reads addresses the user pasted in, one per line.
|
|
||||||
/// </summary>
|
|
||||||
/// <remarks>
|
|
||||||
/// The only source that touches no network at all, which is why it lives in the domain and why
|
|
||||||
/// <c>RequiresNetwork</c> stays false: it discovers nothing, it just reads what it was handed.
|
|
||||||
/// Downloading those addresses is the fetcher's business and is gated separately.
|
|
||||||
/// </remarks>
|
|
||||||
public sealed class UrlListSource : IMediaSource
|
|
||||||
{
|
|
||||||
/// <inheritdoc />
|
|
||||||
public string Id => "url-list";
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public string DisplayName => "URL list";
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public string Description => "One address per line. Blank lines and lines starting with '#' are ignored.";
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public bool CanParse(MediaQuery input) =>
|
|
||||||
input is not null && TextLines.Split(input.Text).Any(line => TryParse(line, out _));
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public async IAsyncEnumerable<ParseOutcome<MediaCandidate>> ParseAsync(
|
|
||||||
MediaQuery input,
|
|
||||||
IProgress<ParseProgress>? progress,
|
|
||||||
[EnumeratorCancellation] CancellationToken cancellationToken
|
|
||||||
)
|
|
||||||
{
|
|
||||||
ArgumentNullException.ThrowIfNull(input);
|
|
||||||
|
|
||||||
var lines = TextLines.Split(input.Text);
|
|
||||||
var ordinal = 0;
|
|
||||||
var yielded = 0;
|
|
||||||
|
|
||||||
for (var index = 0; index < lines.Length; index++)
|
|
||||||
{
|
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
|
||||||
|
|
||||||
var line = lines[index];
|
|
||||||
|
|
||||||
if (TextLines.IsSkippable(line))
|
|
||||||
{
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
ordinal++;
|
|
||||||
|
|
||||||
if (TryParse(line, out var url))
|
|
||||||
{
|
|
||||||
yield return ParseOutcome<MediaCandidate>.Success(
|
|
||||||
new MediaCandidate(url!) { SourceId = Id, Ordinal = ordinal }
|
|
||||||
);
|
|
||||||
|
|
||||||
yielded++;
|
|
||||||
|
|
||||||
if (input.HasLimit && yielded >= input.Limit)
|
|
||||||
{
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
else
|
|
||||||
{
|
|
||||||
// Named rather than silently dropped: a mistyped address in a pasted list of two
|
|
||||||
// hundred is otherwise impossible to find. The line goes in twice on purpose — as
|
|
||||||
// the English fallback message, and as the argument the translation substitutes.
|
|
||||||
var offending = line.Trim();
|
|
||||||
|
|
||||||
yield return ParseOutcome<MediaCandidate>.Failure(
|
|
||||||
ParseError.Create(
|
|
||||||
ordinal,
|
|
||||||
"NotAnAddress",
|
|
||||||
$"'{offending}' is not an http or https address.",
|
|
||||||
offending
|
|
||||||
)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (ordinal % TextLines.ProgressInterval == 0)
|
|
||||||
{
|
|
||||||
progress?.Report(new ParseProgress(index + 1, lines.Length));
|
|
||||||
}
|
|
||||||
|
|
||||||
if (ordinal % TextLines.YieldInterval == 0)
|
|
||||||
{
|
|
||||||
await Task.Yield();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
progress?.Report(new ParseProgress(lines.Length, lines.Length));
|
|
||||||
}
|
|
||||||
|
|
||||||
/// <summary>Accepts only absolute HTTP addresses.</summary>
|
|
||||||
/// <remarks>
|
|
||||||
/// <c>file:</c> and <c>data:</c> are rejected here as well as at redirect time: a pasted list is
|
|
||||||
/// as likely to have come from somewhere else as to have been typed.
|
|
||||||
/// </remarks>
|
|
||||||
private static bool TryParse(string line, out Uri? url)
|
|
||||||
{
|
|
||||||
url = null;
|
|
||||||
var text = line.Trim();
|
|
||||||
|
|
||||||
if (text.Length == 0 || !Uri.TryCreate(text, UriKind.Absolute, out var parsed))
|
|
||||||
{
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (parsed.Scheme is not ("http" or "https"))
|
|
||||||
{
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
url = parsed;
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,5 +1,3 @@
|
|||||||
using AvParser.Core.Collecting;
|
|
||||||
using AvParser.Core.Collecting.Sources;
|
|
||||||
using Microsoft.Extensions.DependencyInjection;
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
|
||||||
namespace AvParser.Core.DependencyInjection;
|
namespace AvParser.Core.DependencyInjection;
|
||||||
@@ -8,26 +6,13 @@ namespace AvParser.Core.DependencyInjection;
|
|||||||
public static class CoreServiceCollectionExtensions
|
public static class CoreServiceCollectionExtensions
|
||||||
{
|
{
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Registers every media source the domain itself provides.
|
/// Registers the domain services. Media sources are user-added data now, not registered types,
|
||||||
|
/// so they are built from the persisted store in the infrastructure layer rather than here.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <remarks>
|
|
||||||
/// Adding a source is a one-line change here — that is the whole point of the
|
|
||||||
/// <see cref="IMediaSource"/> / <see cref="IMediaSourceCatalog"/> split. Sources that need a
|
|
||||||
/// network live in the infrastructure layer and register themselves there.
|
|
||||||
/// </remarks>
|
|
||||||
public static IServiceCollection AddAvParserCore(this IServiceCollection services)
|
public static IServiceCollection AddAvParserCore(this IServiceCollection services)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(services);
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
services.AddSingleton<IMediaSource, UrlListSource>();
|
|
||||||
|
|
||||||
return services;
|
return services;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Id of the source the collector opens on.</summary>
|
|
||||||
/// <remarks>
|
|
||||||
/// Named rather than left to alphabetical order, which would land on the network source and
|
|
||||||
/// open the page behind the proxy gate before the user has asked for anything.
|
|
||||||
/// </remarks>
|
|
||||||
public const string DefaultMediaSourceId = "url-list";
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -20,6 +20,15 @@ public sealed record ParseError(int Index, string Message)
|
|||||||
/// <summary>Values to substitute into the translated message.</summary>
|
/// <summary>Values to substitute into the translated message.</summary>
|
||||||
public IReadOnlyList<object?> Arguments { get; init; } = [];
|
public IReadOnlyList<object?> Arguments { get; init; } = [];
|
||||||
|
|
||||||
|
/// <summary>What the failure was about — the address or the line it came from.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Deliberately outside <see cref="Message"/> and <see cref="Arguments"/>: a translated template
|
||||||
|
/// has fixed placeholders, so an address cannot be appended to it without editing every
|
||||||
|
/// translation. The live collection log prints this beside the translated text, which is the
|
||||||
|
/// difference between "the site answered 404" and knowing which of ten thousand ids that was.
|
||||||
|
/// </remarks>
|
||||||
|
public string? Subject { get; init; }
|
||||||
|
|
||||||
/// <summary>Creates an error carrying a translation code.</summary>
|
/// <summary>Creates an error carrying a translation code.</summary>
|
||||||
public static ParseError Create(int index, string code, string message, params object?[] arguments) =>
|
public static ParseError Create(int index, string code, string message, params object?[] arguments) =>
|
||||||
new(index, message) { Code = code, Arguments = arguments };
|
new(index, message) { Code = code, Arguments = arguments };
|
||||||
|
|||||||
@@ -51,6 +51,24 @@ public interface IProxyPool
|
|||||||
IProgress<ProxySweepProgress>? progress = null,
|
IProgress<ProxySweepProgress>? progress = null,
|
||||||
CancellationToken cancellationToken = default
|
CancellationToken cancellationToken = default
|
||||||
);
|
);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Probes whatever the warm-up never got to, slowly, while the app is doing something else.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="concurrency">Probes in flight; deliberately far below a manual sweep's.</param>
|
||||||
|
/// <param name="progress">Optional progress, in checked-entry counts.</param>
|
||||||
|
/// <param name="cancellationToken">Stops the pass; entries already probed keep their verdict.</param>
|
||||||
|
/// <returns>How many of the previously unchecked entries answered.</returns>
|
||||||
|
/// <remarks>
|
||||||
|
/// The warm-up stops the moment it has enough live proxies, which on a free list of a few
|
||||||
|
/// thousand means thirty were checked and the rest are unknown — not dead. This finishes the
|
||||||
|
/// job in the background so the pool ends up genuinely checked without anyone waiting for it.
|
||||||
|
/// </remarks>
|
||||||
|
Task<int> TopUpAsync(
|
||||||
|
int concurrency,
|
||||||
|
IProgress<ProxySweepProgress>? progress = null,
|
||||||
|
CancellationToken cancellationToken = default
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
|
|||||||
@@ -113,15 +113,6 @@ public sealed record ProxyOptions
|
|||||||
/// </remarks>
|
/// </remarks>
|
||||||
public int MinimumLiveProxies { get; init; } = 10;
|
public int MinimumLiveProxies { get; init; } = 10;
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// Whether work may run without a proxy when none is available.
|
|
||||||
/// </summary>
|
|
||||||
/// <remarks>
|
|
||||||
/// Only parsers that declare <c>RequiresNetwork</c> are affected; a parser working on pasted
|
|
||||||
/// text is never blocked.
|
|
||||||
/// </remarks>
|
|
||||||
public bool AllowDirectConnection { get; init; }
|
|
||||||
|
|
||||||
/// <summary>Validates the options, throwing on values that would misbehave silently.</summary>
|
/// <summary>Validates the options, throwing on values that would misbehave silently.</summary>
|
||||||
/// <exception cref="ArgumentOutOfRangeException">A numeric option is out of range.</exception>
|
/// <exception cref="ArgumentOutOfRangeException">A numeric option is out of range.</exception>
|
||||||
public ProxyOptions Validated()
|
public ProxyOptions Validated()
|
||||||
|
|||||||
@@ -5,6 +5,9 @@ namespace AvParser.Core.Proxies;
|
|||||||
/// <inheritdoc cref="IProxyPool" />
|
/// <inheritdoc cref="IProxyPool" />
|
||||||
public sealed class ProxyPool : IProxyPool
|
public sealed class ProxyPool : IProxyPool
|
||||||
{
|
{
|
||||||
|
/// <summary>How many probes a background top-up makes before it tells anyone.</summary>
|
||||||
|
private const int ChangeInterval = 32;
|
||||||
|
|
||||||
private readonly IReadOnlyList<IProxySource> _sources;
|
private readonly IReadOnlyList<IProxySource> _sources;
|
||||||
private readonly IProxyProbe _probe;
|
private readonly IProxyProbe _probe;
|
||||||
private readonly TimeProvider _time;
|
private readonly TimeProvider _time;
|
||||||
@@ -367,6 +370,77 @@ public sealed class ProxyPool : IProxyPool
|
|||||||
return LiveCount;
|
return LiveCount;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async Task<int> TopUpAsync(
|
||||||
|
int concurrency,
|
||||||
|
IProgress<ProxySweepProgress>? progress = null,
|
||||||
|
CancellationToken cancellationToken = default
|
||||||
|
)
|
||||||
|
{
|
||||||
|
ArgumentOutOfRangeException.ThrowIfLessThan(concurrency, 1);
|
||||||
|
|
||||||
|
var options = Options;
|
||||||
|
|
||||||
|
// Only what nobody has spoken to this session. Re-probing a known-dead entry would just be
|
||||||
|
// a sweep, and re-probing a live one could sideline the proxy a collect run is using.
|
||||||
|
var targets = WarmUpOrder().Where(entry => entry.Health == ProxyHealthState.Unknown).ToArray();
|
||||||
|
var total = targets.Length;
|
||||||
|
|
||||||
|
if (total == 0)
|
||||||
|
{
|
||||||
|
progress?.Report(new ProxySweepProgress(0, 0, LiveCount));
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
using var limiter = new SemaphoreSlim(concurrency, concurrency);
|
||||||
|
var checkedCount = 0;
|
||||||
|
var aliveCount = 0;
|
||||||
|
|
||||||
|
var work = targets.Select(async entry =>
|
||||||
|
{
|
||||||
|
await limiter.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var result = await _probe.ProbeAsync(entry.Endpoint, options, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
entry.RecordProbe(
|
||||||
|
_time.GetUtcNow(),
|
||||||
|
result.Alive,
|
||||||
|
result.Latency,
|
||||||
|
result.Error,
|
||||||
|
quarantineOnFailure: options.BaseQuarantine
|
||||||
|
);
|
||||||
|
|
||||||
|
if (result.Alive)
|
||||||
|
{
|
||||||
|
Interlocked.Increment(ref aliveCount);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
limiter.Release();
|
||||||
|
|
||||||
|
var done = Interlocked.Increment(ref checkedCount);
|
||||||
|
progress?.Report(new ProxySweepProgress(done, total, Volatile.Read(ref aliveCount)));
|
||||||
|
|
||||||
|
// Periodically rather than per probe: this pass runs for minutes over thousands of
|
||||||
|
// entries, and a page that only hears about it at the end looks stuck.
|
||||||
|
if (done % ChangeInterval == 0)
|
||||||
|
{
|
||||||
|
RaiseChanged();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
await Task.WhenAll(work).ConfigureAwait(false);
|
||||||
|
|
||||||
|
// No _strategy.Reset() here, unlike a sweep: this runs while proxies are being handed out,
|
||||||
|
// and resetting would drop a sticky pick mid-collect for entries the pick is not even among.
|
||||||
|
RaiseChanged();
|
||||||
|
return aliveCount;
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Orders candidates for a warm-up: what worked last time, then what looks most promising.
|
/// Orders candidates for a warm-up: what worked last time, then what looks most promising.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
|||||||
@@ -46,7 +46,6 @@ public enum AppTheme
|
|||||||
/// <param name="ProxyProbeTimeoutSeconds">Per-proxy probe timeout, in seconds.</param>
|
/// <param name="ProxyProbeTimeoutSeconds">Per-proxy probe timeout, in seconds.</param>
|
||||||
/// <param name="ProxyProbeConcurrency">How many probes run at once during a pool sweep.</param>
|
/// <param name="ProxyProbeConcurrency">How many probes run at once during a pool sweep.</param>
|
||||||
/// <param name="ProxyMinimumLive">How many working proxies a startup warm-up aims for.</param>
|
/// <param name="ProxyMinimumLive">How many working proxies a startup warm-up aims for.</param>
|
||||||
/// <param name="AllowDirectConnection">Whether network parsers may run without a proxy.</param>
|
|
||||||
/// <param name="LastSourceId">Id of the media source selected last time; resolved leniently on load.</param>
|
/// <param name="LastSourceId">Id of the media source selected last time; resolved leniently on load.</param>
|
||||||
/// <param name="MaxConcurrentDownloads">How many downloads may be in flight at once.</param>
|
/// <param name="MaxConcurrentDownloads">How many downloads may be in flight at once.</param>
|
||||||
/// <param name="MaxConcurrentPerHost">How many requests one origin may be serving at once.</param>
|
/// <param name="MaxConcurrentPerHost">How many requests one origin may be serving at once.</param>
|
||||||
@@ -61,6 +60,16 @@ public enum AppTheme
|
|||||||
/// <param name="ShowcaseMode">How browsable copies point at their blobs.</param>
|
/// <param name="ShowcaseMode">How browsable copies point at their blobs.</param>
|
||||||
/// <param name="CollectUserAgent">Identifies the collector to origins that care.</param>
|
/// <param name="CollectUserAgent">Identifies the collector to origins that care.</param>
|
||||||
/// <param name="MediaRootOverride">Where collected media goes; null keeps it beside the settings.</param>
|
/// <param name="MediaRootOverride">Where collected media goes; null keeps it beside the settings.</param>
|
||||||
|
/// <param name="CollectTargetCount">How many items to stop after collecting; 0 means no target.</param>
|
||||||
|
/// <param name="CollectAttemptBudget">
|
||||||
|
/// How many candidate ids one source may try in a run; 0 means until the user stops it.
|
||||||
|
/// </param>
|
||||||
|
/// <param name="CollectSourceIds">
|
||||||
|
/// Ids of the sources ticked to run, separated by <c>;</c>. A joined string rather than a list
|
||||||
|
/// because <see cref="AppSettings"/> keeps value equality — the settings service short-circuits a
|
||||||
|
/// no-op write by comparing records, and a list-valued member would make every save look like a
|
||||||
|
/// change. Same reasoning as <see cref="MediaKindFilter"/>.
|
||||||
|
/// </param>
|
||||||
public sealed record AppSettings(
|
public sealed record AppSettings(
|
||||||
AppTheme Theme = AppTheme.System,
|
AppTheme Theme = AppTheme.System,
|
||||||
AppLanguage Language = AppLanguage.System,
|
AppLanguage Language = AppLanguage.System,
|
||||||
@@ -76,7 +85,6 @@ public sealed record AppSettings(
|
|||||||
int ProxyProbeTimeoutSeconds = 8,
|
int ProxyProbeTimeoutSeconds = 8,
|
||||||
int ProxyProbeConcurrency = 64,
|
int ProxyProbeConcurrency = 64,
|
||||||
int ProxyMinimumLive = 10,
|
int ProxyMinimumLive = 10,
|
||||||
bool AllowDirectConnection = false,
|
|
||||||
string? LastSourceId = null,
|
string? LastSourceId = null,
|
||||||
int MaxConcurrentDownloads = 4,
|
int MaxConcurrentDownloads = 4,
|
||||||
int MaxConcurrentPerHost = 2,
|
int MaxConcurrentPerHost = 2,
|
||||||
@@ -90,9 +98,34 @@ public sealed record AppSettings(
|
|||||||
MediaKindFilter AllowedMediaKinds = MediaKindFilter.All,
|
MediaKindFilter AllowedMediaKinds = MediaKindFilter.All,
|
||||||
ShowcaseMode ShowcaseMode = ShowcaseMode.HardLink,
|
ShowcaseMode ShowcaseMode = ShowcaseMode.HardLink,
|
||||||
string CollectUserAgent = "AvParser/0.1",
|
string CollectUserAgent = "AvParser/0.1",
|
||||||
string? MediaRootOverride = null
|
string? MediaRootOverride = null,
|
||||||
|
int CollectTargetCount = 100,
|
||||||
|
int CollectAttemptBudget = 5000,
|
||||||
|
string? CollectSourceIds = null
|
||||||
)
|
)
|
||||||
{
|
{
|
||||||
|
/// <summary>Separates ids inside <see cref="CollectSourceIds"/>.</summary>
|
||||||
|
public const char SourceIdSeparator = ';';
|
||||||
|
|
||||||
|
/// <summary>Splits a stored <see cref="CollectSourceIds"/> back into ids.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// A method rather than a property: every public property here is serialised, and a derived one
|
||||||
|
/// would be written back into the settings file as a second copy of the same list.
|
||||||
|
/// </remarks>
|
||||||
|
public static IReadOnlyList<string> SplitSourceIds(string? ids) =>
|
||||||
|
string.IsNullOrWhiteSpace(ids)
|
||||||
|
? []
|
||||||
|
: ids.Split(SourceIdSeparator, StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);
|
||||||
|
|
||||||
|
/// <summary>Joins ids for storage in <see cref="CollectSourceIds"/>.</summary>
|
||||||
|
public static string? JoinSourceIds(IEnumerable<string> ids)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(ids);
|
||||||
|
|
||||||
|
var joined = string.Join(SourceIdSeparator, ids);
|
||||||
|
return joined.Length == 0 ? null : joined;
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Projects the proxy-related settings onto <see cref="ProxyOptions"/>.</summary>
|
/// <summary>Projects the proxy-related settings onto <see cref="ProxyOptions"/>.</summary>
|
||||||
/// <remarks>
|
/// <remarks>
|
||||||
/// Settings are persisted as primitives so an old file still deserialises; the pool wants a
|
/// Settings are persisted as primitives so an old file still deserialises; the pool wants a
|
||||||
@@ -115,7 +148,6 @@ public sealed record AppSettings(
|
|||||||
ProbeTimeout = TimeSpan.FromSeconds(Math.Clamp(ProxyProbeTimeoutSeconds, 1, 120)),
|
ProbeTimeout = TimeSpan.FromSeconds(Math.Clamp(ProxyProbeTimeoutSeconds, 1, 120)),
|
||||||
ProbeConcurrency = Math.Clamp(ProxyProbeConcurrency, 1, 512),
|
ProbeConcurrency = Math.Clamp(ProxyProbeConcurrency, 1, 512),
|
||||||
MinimumLiveProxies = Math.Clamp(ProxyMinimumLive, 1, 500),
|
MinimumLiveProxies = Math.Clamp(ProxyMinimumLive, 1, 500),
|
||||||
AllowDirectConnection = AllowDirectConnection,
|
|
||||||
}.Validated();
|
}.Validated();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -142,7 +174,9 @@ public sealed record AppSettings(
|
|||||||
// to be an accident than an intention to collect nothing.
|
// to be an accident than an intention to collect nothing.
|
||||||
AllowedKinds = AllowedMediaKinds == MediaKindFilter.None ? MediaKindFilter.All : AllowedMediaKinds,
|
AllowedKinds = AllowedMediaKinds == MediaKindFilter.None ? MediaKindFilter.All : AllowedMediaKinds,
|
||||||
UserAgent = string.IsNullOrWhiteSpace(CollectUserAgent) ? "AvParser/0.1" : CollectUserAgent,
|
UserAgent = string.IsNullOrWhiteSpace(CollectUserAgent) ? "AvParser/0.1" : CollectUserAgent,
|
||||||
RequireProxy = !AllowDirectConnection,
|
// Whether a missing proxy is fatal is the source's own setting; the collect page
|
||||||
|
// overrides this per source. The default here is the strict one on purpose.
|
||||||
|
RequireProxy = true,
|
||||||
ShowcaseMode = ShowcaseMode,
|
ShowcaseMode = ShowcaseMode,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -311,7 +311,10 @@ public sealed class CollectRunner(
|
|||||||
result.ErrorCode ?? "Failed",
|
result.ErrorCode ?? "Failed",
|
||||||
result.ErrorDetail ?? candidate.Url.AbsoluteUri,
|
result.ErrorDetail ?? candidate.Url.AbsoluteUri,
|
||||||
result.ErrorDetail ?? candidate.Url.AbsoluteUri
|
result.ErrorDetail ?? candidate.Url.AbsoluteUri
|
||||||
)
|
) with
|
||||||
|
{
|
||||||
|
Subject = candidate.Url.AbsoluteUri,
|
||||||
|
}
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,281 @@
|
|||||||
|
using System.Text.Json;
|
||||||
|
using System.Text.Json.Serialization;
|
||||||
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
|
using AvParser.Infrastructure.Storage;
|
||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
|
||||||
|
namespace AvParser.Infrastructure.Collecting;
|
||||||
|
|
||||||
|
/// <summary>One user-added pattern source, as persisted.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// String-typed rather than storing a <see cref="PatternSourceConfig"/> directly: the config carries
|
||||||
|
/// a <see cref="Uri"/> and an enum, and a hand-edited or older file must degrade to "skip this row"
|
||||||
|
/// rather than throw. Every record is re-validated through <see cref="PatternSourceConfig.TryCreate"/>
|
||||||
|
/// on load, so persistence stays forgiving while the in-memory list stays well-formed.
|
||||||
|
/// </remarks>
|
||||||
|
public sealed record PatternSourceRecord(
|
||||||
|
string Id,
|
||||||
|
string Name,
|
||||||
|
string BaseUrl,
|
||||||
|
int MinLength,
|
||||||
|
int MaxLength,
|
||||||
|
string Alphabet,
|
||||||
|
string? CustomAlphabet,
|
||||||
|
string? Extension,
|
||||||
|
bool AllowDirectConnection = false
|
||||||
|
);
|
||||||
|
|
||||||
|
/// <summary>Source-generated serialiser metadata for the user's source list.</summary>
|
||||||
|
[JsonSourceGenerationOptions(WriteIndented = true, PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase)]
|
||||||
|
[JsonSerializable(typeof(List<PatternSourceRecord>))]
|
||||||
|
internal sealed partial class UserSourceJsonContext : JsonSerializerContext;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The user's own media-source list, persisted next to the settings.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The collector twin of <c>CustomProxySource</c>: a separate file the user can back up and share,
|
||||||
|
/// read once and cached, written atomically. Mutations raise <see cref="Changed"/> so the catalog
|
||||||
|
/// re-materialises its sources.
|
||||||
|
/// </remarks>
|
||||||
|
public sealed class JsonUserSourceStore : IUserSourceStore, IDisposable
|
||||||
|
{
|
||||||
|
private readonly IAppPaths _paths;
|
||||||
|
private readonly ILogger<JsonUserSourceStore> _logger;
|
||||||
|
private readonly SemaphoreSlim _gate = new(1, 1);
|
||||||
|
private readonly object _cacheLock = new();
|
||||||
|
|
||||||
|
private List<PatternSourceRecord>? _records;
|
||||||
|
private IReadOnlyList<PatternSourceConfig> _configs = [];
|
||||||
|
|
||||||
|
/// <summary>Creates the store.</summary>
|
||||||
|
public JsonUserSourceStore(IAppPaths paths, ILogger<JsonUserSourceStore> logger)
|
||||||
|
{
|
||||||
|
_paths = paths ?? throw new ArgumentNullException(nameof(paths));
|
||||||
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public event EventHandler? Changed;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public IReadOnlyList<PatternSourceConfig> List()
|
||||||
|
{
|
||||||
|
lock (_cacheLock)
|
||||||
|
{
|
||||||
|
EnsureLoaded();
|
||||||
|
return _configs;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async Task<PatternSourceConfig> AddAsync(
|
||||||
|
PatternSourceConfig config,
|
||||||
|
CancellationToken cancellationToken = default
|
||||||
|
)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(config);
|
||||||
|
|
||||||
|
await _gate.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var records = LoadRecords();
|
||||||
|
|
||||||
|
// A colliding id means a hand-edited file or a repeated slug; take the incoming one as
|
||||||
|
// authoritative and replace rather than grow a second row with the same id.
|
||||||
|
records.RemoveAll(r => string.Equals(r.Id, config.Id, StringComparison.OrdinalIgnoreCase));
|
||||||
|
records.Add(ToRecord(config));
|
||||||
|
|
||||||
|
await SaveAsync(records, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_gate.Release();
|
||||||
|
}
|
||||||
|
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return config;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async Task<bool> UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(config);
|
||||||
|
|
||||||
|
bool replaced;
|
||||||
|
|
||||||
|
await _gate.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var records = LoadRecords();
|
||||||
|
var index = records.FindIndex(r => string.Equals(r.Id, config.Id, StringComparison.OrdinalIgnoreCase));
|
||||||
|
|
||||||
|
if (index < 0)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
records[index] = ToRecord(config);
|
||||||
|
replaced = true;
|
||||||
|
|
||||||
|
await SaveAsync(records, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_gate.Release();
|
||||||
|
}
|
||||||
|
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return replaced;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public async Task<bool> RemoveAsync(string id, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
ArgumentException.ThrowIfNullOrWhiteSpace(id);
|
||||||
|
|
||||||
|
bool removed;
|
||||||
|
|
||||||
|
await _gate.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var records = LoadRecords();
|
||||||
|
removed = records.RemoveAll(r => string.Equals(r.Id, id, StringComparison.OrdinalIgnoreCase)) > 0;
|
||||||
|
|
||||||
|
if (!removed)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
await SaveAsync(records, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_gate.Release();
|
||||||
|
}
|
||||||
|
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return removed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public void Dispose() => _gate.Dispose();
|
||||||
|
|
||||||
|
private static PatternSourceRecord ToRecord(PatternSourceConfig config) =>
|
||||||
|
new(
|
||||||
|
config.Id,
|
||||||
|
config.Name,
|
||||||
|
config.BaseUrl.AbsoluteUri,
|
||||||
|
config.MinLength,
|
||||||
|
config.MaxLength,
|
||||||
|
config.Alphabet.ToString(),
|
||||||
|
config.CustomAlphabet,
|
||||||
|
config.Extension,
|
||||||
|
config.AllowDirectConnection
|
||||||
|
);
|
||||||
|
|
||||||
|
private static PatternSourceConfig? ToConfig(PatternSourceRecord record)
|
||||||
|
{
|
||||||
|
if (!Enum.TryParse<IdAlphabet>(record.Alphabet, ignoreCase: true, out var alphabet))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
return
|
||||||
|
PatternSourceConfig.TryCreate(
|
||||||
|
record.Name,
|
||||||
|
record.BaseUrl,
|
||||||
|
record.MinLength,
|
||||||
|
record.MaxLength,
|
||||||
|
alphabet,
|
||||||
|
record.CustomAlphabet,
|
||||||
|
record.Extension,
|
||||||
|
record.AllowDirectConnection,
|
||||||
|
out var config,
|
||||||
|
record.Id
|
||||||
|
) == PatternConfigError.None
|
||||||
|
? config
|
||||||
|
: null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void EnsureLoaded()
|
||||||
|
{
|
||||||
|
if (_records is not null)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
LoadRecords();
|
||||||
|
}
|
||||||
|
|
||||||
|
private List<PatternSourceRecord> LoadRecords()
|
||||||
|
{
|
||||||
|
if (_records is not null)
|
||||||
|
{
|
||||||
|
return _records;
|
||||||
|
}
|
||||||
|
|
||||||
|
List<PatternSourceRecord> records;
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
records = File.Exists(_paths.UserSourcesFile)
|
||||||
|
? JsonSerializer.Deserialize(
|
||||||
|
File.ReadAllText(_paths.UserSourcesFile),
|
||||||
|
UserSourceJsonContext.Default.ListPatternSourceRecord
|
||||||
|
) ?? []
|
||||||
|
: [];
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or JsonException)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "Could not read {Path}; starting with an empty source list", _paths.UserSourcesFile);
|
||||||
|
records = [];
|
||||||
|
}
|
||||||
|
|
||||||
|
Cache(records);
|
||||||
|
return records;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void Cache(List<PatternSourceRecord> records)
|
||||||
|
{
|
||||||
|
var configs = new List<PatternSourceConfig>(records.Count);
|
||||||
|
foreach (var record in records)
|
||||||
|
{
|
||||||
|
var config = ToConfig(record);
|
||||||
|
if (config is null)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("Skipping unusable stored source {Id}", record.Id);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
configs.Add(config);
|
||||||
|
}
|
||||||
|
|
||||||
|
lock (_cacheLock)
|
||||||
|
{
|
||||||
|
_records = records;
|
||||||
|
_configs = configs;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task SaveAsync(List<PatternSourceRecord> records, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
Cache(records);
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
Directory.CreateDirectory(Path.GetDirectoryName(_paths.UserSourcesFile)!);
|
||||||
|
|
||||||
|
var temp = _paths.UserSourcesFile + ".tmp";
|
||||||
|
var json = JsonSerializer.Serialize(records, UserSourceJsonContext.Default.ListPatternSourceRecord);
|
||||||
|
|
||||||
|
await File.WriteAllTextAsync(temp, json, cancellationToken).ConfigureAwait(false);
|
||||||
|
File.Move(temp, _paths.UserSourcesFile, overwrite: true);
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "Could not write {Path}", _paths.UserSourcesFile);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,293 +0,0 @@
|
|||||||
using System.Globalization;
|
|
||||||
using System.Runtime.CompilerServices;
|
|
||||||
using System.Text.Json;
|
|
||||||
using AvParser.Core.Collecting;
|
|
||||||
using AvParser.Core.Parsing;
|
|
||||||
using AvParser.Core.Settings;
|
|
||||||
using AvParser.Infrastructure.Proxies;
|
|
||||||
using Microsoft.Extensions.Logging;
|
|
||||||
|
|
||||||
namespace AvParser.Infrastructure.Collecting;
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// Lists what a service you run holds, by asking it.
|
|
||||||
/// </summary>
|
|
||||||
/// <remarks>
|
|
||||||
/// <para>
|
|
||||||
/// A service that knows its own contents can simply say so, which is why this is a listing client
|
|
||||||
/// and not a crawler: one request per page returns exactly what is there, in order, with metadata,
|
|
||||||
/// and stops when the pages run out.
|
|
||||||
/// </para>
|
|
||||||
/// <para>
|
|
||||||
/// The expected response is deliberately loose, because the service on the other end is the user's
|
|
||||||
/// own and should not have to be rewritten to match us. Either of these works:
|
|
||||||
/// </para>
|
|
||||||
/// <code>
|
|
||||||
/// { "items": [ { "url": "...", "id": "...", "name": "...", "published": "...", "size": 1234,
|
|
||||||
/// "tags": ["a"] } ], "next": "cursor" }
|
|
||||||
/// [ "https://host/one.png", "https://host/two.gif" ]
|
|
||||||
/// </code>
|
|
||||||
/// <para>
|
|
||||||
/// Paging follows <c>next</c> until it is absent. A page that repeats a cursor stops the walk
|
|
||||||
/// rather than looping for ever.
|
|
||||||
/// </para>
|
|
||||||
/// </remarks>
|
|
||||||
public sealed class OwnServiceSource(
|
|
||||||
IProxiedHttpClientFactory clients,
|
|
||||||
ISettingsService settings,
|
|
||||||
ILogger<OwnServiceSource> logger
|
|
||||||
) : IMediaSource
|
|
||||||
{
|
|
||||||
/// <summary>Stops a service that keeps handing back pages from running the collector for ever.</summary>
|
|
||||||
private const int MaxPages = 10_000;
|
|
||||||
|
|
||||||
private readonly IProxiedHttpClientFactory _clients = clients ?? throw new ArgumentNullException(nameof(clients));
|
|
||||||
private readonly ISettingsService _settings = settings ?? throw new ArgumentNullException(nameof(settings));
|
|
||||||
private readonly ILogger<OwnServiceSource> _logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public string Id => "own-service";
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public string DisplayName => "Own service";
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public string Description => "Reads the listing endpoint of a service you run.";
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public bool RequiresNetwork => true;
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public bool CanParse(MediaQuery input) =>
|
|
||||||
input?.Endpoint is { IsAbsoluteUri: true } endpoint && endpoint.Scheme is "http" or "https";
|
|
||||||
|
|
||||||
/// <inheritdoc />
|
|
||||||
public async IAsyncEnumerable<ParseOutcome<MediaCandidate>> ParseAsync(
|
|
||||||
MediaQuery input,
|
|
||||||
IProgress<ParseProgress>? progress,
|
|
||||||
[EnumeratorCancellation] CancellationToken cancellationToken
|
|
||||||
)
|
|
||||||
{
|
|
||||||
ArgumentNullException.ThrowIfNull(input);
|
|
||||||
|
|
||||||
if (!CanParse(input))
|
|
||||||
{
|
|
||||||
yield return ParseOutcome<MediaCandidate>.Failure(
|
|
||||||
ParseError.Create(0, "NoEndpoint", "No listing endpoint was configured.")
|
|
||||||
);
|
|
||||||
yield break;
|
|
||||||
}
|
|
||||||
|
|
||||||
var requireProxy = !_settings.Current.AllowDirectConnection;
|
|
||||||
|
|
||||||
using var leased = await _clients
|
|
||||||
.LeaseAsync(HttpClientTimeouts.Default, requireProxy, cancellationToken)
|
|
||||||
.ConfigureAwait(false);
|
|
||||||
|
|
||||||
var cursor = input.Cursor;
|
|
||||||
var seenCursors = new HashSet<string>(StringComparer.Ordinal);
|
|
||||||
var ordinal = 0;
|
|
||||||
|
|
||||||
for (var page = 0; page < MaxPages; page++)
|
|
||||||
{
|
|
||||||
cancellationToken.ThrowIfCancellationRequested();
|
|
||||||
|
|
||||||
var url = WithCursor(input.Endpoint!, cursor);
|
|
||||||
|
|
||||||
// Read and parse inside the try, report outside it: a yield cannot live in a catch.
|
|
||||||
Page parsed = default;
|
|
||||||
ParseError? error = null;
|
|
||||||
|
|
||||||
try
|
|
||||||
{
|
|
||||||
var body = await leased.Client.GetStringAsync(url, cancellationToken).ConfigureAwait(false);
|
|
||||||
leased.Lease?.ReportSuccess();
|
|
||||||
|
|
||||||
parsed = ReadPage(body, input.Endpoint!);
|
|
||||||
}
|
|
||||||
catch (Exception ex) when (ex is HttpRequestException or IOException)
|
|
||||||
{
|
|
||||||
leased.Lease?.ReportFailure(ex.Message);
|
|
||||||
_logger.LogWarning(ex, "Could not read the listing at {Url}", url);
|
|
||||||
error = ParseError.Create(ordinal, "ListingFailed", ex.Message, ex.Message);
|
|
||||||
}
|
|
||||||
catch (JsonException ex)
|
|
||||||
{
|
|
||||||
// The transport was fine; the service answered with something we cannot read.
|
|
||||||
error = ParseError.Create(ordinal, "ListingMalformed", ex.Message, ex.Message);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (error is not null)
|
|
||||||
{
|
|
||||||
yield return ParseOutcome<MediaCandidate>.Failure(error);
|
|
||||||
yield break;
|
|
||||||
}
|
|
||||||
|
|
||||||
foreach (var candidate in parsed.Items)
|
|
||||||
{
|
|
||||||
ordinal++;
|
|
||||||
|
|
||||||
yield return ParseOutcome<MediaCandidate>.Success(candidate with { SourceId = Id, Ordinal = ordinal });
|
|
||||||
|
|
||||||
if (input.HasLimit && ordinal >= input.Limit)
|
|
||||||
{
|
|
||||||
yield break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Total is unknown until the last page, so progress stays indeterminate and only the
|
|
||||||
// running count moves.
|
|
||||||
progress?.Report(new ParseProgress(ordinal, 0));
|
|
||||||
|
|
||||||
if (string.IsNullOrEmpty(parsed.Next) || !seenCursors.Add(parsed.Next))
|
|
||||||
{
|
|
||||||
yield break;
|
|
||||||
}
|
|
||||||
|
|
||||||
cursor = parsed.Next;
|
|
||||||
}
|
|
||||||
|
|
||||||
_logger.LogWarning("Stopped paging {Endpoint} after {Pages} pages", input.Endpoint, MaxPages);
|
|
||||||
}
|
|
||||||
|
|
||||||
/// <summary>One page of a listing.</summary>
|
|
||||||
/// <param name="Items">Candidates on this page.</param>
|
|
||||||
/// <param name="Next">Cursor for the following page, or null when this was the last.</param>
|
|
||||||
internal readonly record struct Page(IReadOnlyList<MediaCandidate> Items, string? Next);
|
|
||||||
|
|
||||||
private static Uri WithCursor(Uri endpoint, string? cursor)
|
|
||||||
{
|
|
||||||
if (string.IsNullOrEmpty(cursor))
|
|
||||||
{
|
|
||||||
return endpoint;
|
|
||||||
}
|
|
||||||
|
|
||||||
var separator = string.IsNullOrEmpty(endpoint.Query) ? '?' : '&';
|
|
||||||
|
|
||||||
return new Uri($"{endpoint.AbsoluteUri}{separator}cursor={Uri.EscapeDataString(cursor)}");
|
|
||||||
}
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// Reads either shape of listing.
|
|
||||||
/// </summary>
|
|
||||||
/// <remarks>
|
|
||||||
/// Hand-parsed with <see cref="JsonDocument"/> rather than deserialised into a type: the two
|
|
||||||
/// accepted shapes and the string-or-object item would need a converter each, and being lenient
|
|
||||||
/// about a contract the user controls is the whole point.
|
|
||||||
/// </remarks>
|
|
||||||
internal static Page ReadPage(string json, Uri baseAddress)
|
|
||||||
{
|
|
||||||
using var document = JsonDocument.Parse(json);
|
|
||||||
var root = document.RootElement;
|
|
||||||
|
|
||||||
var items = new List<MediaCandidate>();
|
|
||||||
string? next = null;
|
|
||||||
|
|
||||||
var array = root.ValueKind switch
|
|
||||||
{
|
|
||||||
JsonValueKind.Array => root,
|
|
||||||
JsonValueKind.Object when root.TryGetProperty("items", out var found) => found,
|
|
||||||
_ => default,
|
|
||||||
};
|
|
||||||
|
|
||||||
if (root.ValueKind == JsonValueKind.Object && root.TryGetProperty("next", out var cursor))
|
|
||||||
{
|
|
||||||
next = cursor.ValueKind == JsonValueKind.String ? cursor.GetString() : null;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (array.ValueKind != JsonValueKind.Array)
|
|
||||||
{
|
|
||||||
return new Page(items, next);
|
|
||||||
}
|
|
||||||
|
|
||||||
foreach (var element in array.EnumerateArray())
|
|
||||||
{
|
|
||||||
if (ReadItem(element, baseAddress) is { } candidate)
|
|
||||||
{
|
|
||||||
items.Add(candidate);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return new Page(items, next);
|
|
||||||
}
|
|
||||||
|
|
||||||
private static MediaCandidate? ReadItem(JsonElement element, Uri baseAddress)
|
|
||||||
{
|
|
||||||
if (element.ValueKind == JsonValueKind.String)
|
|
||||||
{
|
|
||||||
return Resolve(element.GetString(), baseAddress) is { } bare ? new MediaCandidate(bare) : null;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (element.ValueKind != JsonValueKind.Object)
|
|
||||||
{
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
var url = Resolve(ReadString(element, "url") ?? ReadString(element, "href"), baseAddress);
|
|
||||||
|
|
||||||
if (url is null)
|
|
||||||
{
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
var candidate = new MediaCandidate(url)
|
|
||||||
{
|
|
||||||
ExternalId = ReadString(element, "id"),
|
|
||||||
SuggestedName = ReadString(element, "name") ?? ReadString(element, "filename"),
|
|
||||||
};
|
|
||||||
|
|
||||||
if (
|
|
||||||
ReadString(element, "published") is { } published
|
|
||||||
&& DateTimeOffset.TryParse(
|
|
||||||
published,
|
|
||||||
CultureInfo.InvariantCulture,
|
|
||||||
DateTimeStyles.RoundtripKind,
|
|
||||||
out var when
|
|
||||||
)
|
|
||||||
)
|
|
||||||
{
|
|
||||||
candidate = candidate with { PublishedUtc = when };
|
|
||||||
}
|
|
||||||
|
|
||||||
if (element.TryGetProperty("size", out var size) && size.TryGetInt64(out var length))
|
|
||||||
{
|
|
||||||
candidate = candidate with { ExpectedLength = length };
|
|
||||||
}
|
|
||||||
|
|
||||||
if (element.TryGetProperty("tags", out var tags) && tags.ValueKind == JsonValueKind.Array)
|
|
||||||
{
|
|
||||||
candidate = candidate with
|
|
||||||
{
|
|
||||||
Tags =
|
|
||||||
[
|
|
||||||
.. tags.EnumerateArray()
|
|
||||||
.Where(tag => tag.ValueKind == JsonValueKind.String)
|
|
||||||
.Select(tag => tag.GetString()!),
|
|
||||||
],
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
return candidate;
|
|
||||||
}
|
|
||||||
|
|
||||||
private static string? ReadString(JsonElement element, string name) =>
|
|
||||||
element.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.String
|
|
||||||
? value.GetString()
|
|
||||||
: null;
|
|
||||||
|
|
||||||
/// <summary>Resolves a listed address, allowing relative paths against the endpoint.</summary>
|
|
||||||
private static Uri? Resolve(string? value, Uri baseAddress)
|
|
||||||
{
|
|
||||||
if (string.IsNullOrWhiteSpace(value))
|
|
||||||
{
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!Uri.TryCreate(baseAddress, value.Trim(), out var resolved))
|
|
||||||
{
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
return resolved.Scheme is "http" or "https" ? resolved : null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+5
-7
@@ -1,5 +1,4 @@
|
|||||||
using AvParser.Core.Collecting;
|
using AvParser.Core.Collecting;
|
||||||
using AvParser.Core.DependencyInjection;
|
|
||||||
using AvParser.Core.Proxies;
|
using AvParser.Core.Proxies;
|
||||||
using AvParser.Core.Settings;
|
using AvParser.Core.Settings;
|
||||||
using AvParser.Infrastructure.Collecting;
|
using AvParser.Infrastructure.Collecting;
|
||||||
@@ -94,13 +93,12 @@ public static class InfrastructureServiceCollectionExtensions
|
|||||||
services.AddSingleton<IMediaFetcher, MediaFetcher>();
|
services.AddSingleton<IMediaFetcher, MediaFetcher>();
|
||||||
services.AddSingleton<ICollectRunner, CollectRunner>();
|
services.AddSingleton<ICollectRunner, CollectRunner>();
|
||||||
|
|
||||||
services.AddSingleton<IMediaSource, OwnServiceSource>();
|
// Sources are user-added data now: the store owns the persisted list, and the catalog
|
||||||
|
// materialises one source per stored config and tracks the store for edits.
|
||||||
// Resolves sources from both assemblies: the container gathers every IMediaSource
|
services.AddSingleton<JsonUserSourceStore>();
|
||||||
// registration regardless of which project declared it.
|
services.AddSingleton<IUserSourceStore>(sp => sp.GetRequiredService<JsonUserSourceStore>());
|
||||||
services.AddSingleton<IMediaSourceCatalog>(sp => new MediaSourceCatalog(
|
services.AddSingleton<IMediaSourceCatalog>(sp => new MediaSourceCatalog(
|
||||||
sp.GetServices<IMediaSource>(),
|
sp.GetRequiredService<IUserSourceStore>()
|
||||||
CoreServiceCollectionExtensions.DefaultMediaSourceId
|
|
||||||
));
|
));
|
||||||
|
|
||||||
return services;
|
return services;
|
||||||
|
|||||||
@@ -22,24 +22,49 @@ public interface IProxyPoolLoader
|
|||||||
/// <summary>Whether the initial load has finished.</summary>
|
/// <summary>Whether the initial load has finished.</summary>
|
||||||
bool IsLoaded { get; }
|
bool IsLoaded { get; }
|
||||||
|
|
||||||
|
/// <summary>Whether the background pass over the unchecked remainder is still running.</summary>
|
||||||
|
bool IsToppingUp { get; }
|
||||||
|
|
||||||
/// <summary>Writes the current pool state so the next launch can start from it.</summary>
|
/// <summary>Writes the current pool state so the next launch can start from it.</summary>
|
||||||
Task SaveStateAsync(CancellationToken cancellationToken = default);
|
Task SaveStateAsync(CancellationToken cancellationToken = default);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc cref="IProxyPoolLoader" />
|
/// <inheritdoc cref="IProxyPoolLoader" />
|
||||||
public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore, ILogger<ProxyPoolLoader> logger)
|
public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore, ILogger<ProxyPoolLoader> logger)
|
||||||
: IProxyPoolLoader
|
: IProxyPoolLoader,
|
||||||
|
IDisposable
|
||||||
{
|
{
|
||||||
|
/// <summary>
|
||||||
|
/// Share of the sweep concurrency the background pass is allowed.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// A manual sweep is a burst the user asked for and is watching; this one runs for minutes,
|
||||||
|
/// unwatched, possibly while a collection is downloading. A quarter is enough to finish a few
|
||||||
|
/// thousand addresses in a few minutes without competing with the thing the user actually
|
||||||
|
/// started.
|
||||||
|
/// </remarks>
|
||||||
|
private const int BackgroundProbeShare = 4;
|
||||||
|
|
||||||
private readonly IProxyPool _pool = pool ?? throw new ArgumentNullException(nameof(pool));
|
private readonly IProxyPool _pool = pool ?? throw new ArgumentNullException(nameof(pool));
|
||||||
private readonly IProxyStateStore _stateStore = stateStore ?? throw new ArgumentNullException(nameof(stateStore));
|
private readonly IProxyStateStore _stateStore = stateStore ?? throw new ArgumentNullException(nameof(stateStore));
|
||||||
private readonly ILogger<ProxyPoolLoader> _logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
private readonly ILogger<ProxyPoolLoader> _logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
private readonly Lock _gate = new();
|
private readonly Lock _gate = new();
|
||||||
|
private readonly CancellationTokenSource _shutdown = new();
|
||||||
|
private readonly SemaphoreSlim _saving = new(1, 1);
|
||||||
|
|
||||||
private Task<ProxyPoolLoadResult>? _load;
|
private Task<ProxyPoolLoadResult>? _load;
|
||||||
|
private Task? _topUp;
|
||||||
|
private int _disposed;
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public bool IsLoaded => _load is { IsCompleted: true };
|
public bool IsLoaded => _load is { IsCompleted: true };
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public bool IsToppingUp => _topUp is { IsCompleted: false };
|
||||||
|
|
||||||
|
/// <summary>The background pass, so tests can await it instead of polling a flag.</summary>
|
||||||
|
internal Task? TopUp => _topUp;
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public Task<ProxyPoolLoadResult> EnsureLoadedAsync()
|
public Task<ProxyPoolLoadResult> EnsureLoadedAsync()
|
||||||
{
|
{
|
||||||
@@ -59,8 +84,23 @@ public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public Task SaveStateAsync(CancellationToken cancellationToken = default) =>
|
/// <remarks>
|
||||||
_stateStore.SaveAsync(_pool.Entries, cancellationToken);
|
/// Serialised: the background pass saves when it finishes and the app saves on shutdown, and
|
||||||
|
/// both writes go through the same temporary file before being renamed into place.
|
||||||
|
/// </remarks>
|
||||||
|
public async Task SaveStateAsync(CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
await _saving.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await _stateStore.SaveAsync(_pool.Entries, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_saving.Release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private async Task<ProxyPoolLoadResult> LoadAsync()
|
private async Task<ProxyPoolLoadResult> LoadAsync()
|
||||||
{
|
{
|
||||||
@@ -97,6 +137,8 @@ public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore
|
|||||||
|
|
||||||
await SaveStateAsync(CancellationToken.None).ConfigureAwait(false);
|
await SaveStateAsync(CancellationToken.None).ConfigureAwait(false);
|
||||||
|
|
||||||
|
StartTopUp(options);
|
||||||
|
|
||||||
return new ProxyPoolLoadResult(total, live, restored);
|
return new ProxyPoolLoadResult(total, live, restored);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
@@ -107,4 +149,69 @@ public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore
|
|||||||
return new ProxyPoolLoadResult(_pool.Entries.Count, _pool.LiveCount, 0);
|
return new ProxyPoolLoadResult(_pool.Entries.Count, _pool.LiveCount, 0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Finishes checking whatever the warm-up skipped, in the background.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The warm-up stops at <see cref="ProxyOptions.MinimumLiveProxies"/>, so on a feed of a few
|
||||||
|
/// thousand it leaves almost everything unknown — which reads as "nothing was checked" and, worse,
|
||||||
|
/// means the next collection has a pool of ten instead of the hundreds that actually answer. Not
|
||||||
|
/// awaited: startup is over by now, and nobody should wait for this. Skipped under
|
||||||
|
/// <see cref="ProxyHealthCheck.Lazy"/>, where the user has asked for no sweeping at all.
|
||||||
|
/// </remarks>
|
||||||
|
private void StartTopUp(ProxyOptions options)
|
||||||
|
{
|
||||||
|
if (options.HealthCheck != ProxyHealthCheck.Pool || _shutdown.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var concurrency = Math.Max(1, options.ProbeConcurrency / BackgroundProbeShare);
|
||||||
|
|
||||||
|
_topUp = Task.Run(
|
||||||
|
async () =>
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var found = await _pool.TopUpAsync(concurrency, null, _shutdown.Token).ConfigureAwait(false);
|
||||||
|
|
||||||
|
_logger.LogInformation(
|
||||||
|
"Background proxy check finished: {Found} more live, {Live} live of {Total}",
|
||||||
|
found,
|
||||||
|
_pool.LiveCount,
|
||||||
|
_pool.Entries.Count
|
||||||
|
);
|
||||||
|
|
||||||
|
// Everything just learned is worth carrying to the next launch; without this the
|
||||||
|
// pass would be repeated from scratch every time. Uncancellable on purpose: a
|
||||||
|
// write stopped halfway leaves a stray temp file and saves nothing.
|
||||||
|
await SaveStateAsync(CancellationToken.None).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException)
|
||||||
|
{
|
||||||
|
// The app is closing; what was probed keeps its verdict.
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "The background proxy check stopped early");
|
||||||
|
}
|
||||||
|
},
|
||||||
|
CancellationToken.None
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
/// <remarks>Stops the background pass; the container disposes this on shutdown.</remarks>
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
if (Interlocked.Exchange(ref _disposed, 1) == 1)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
_shutdown.Cancel();
|
||||||
|
_shutdown.Dispose();
|
||||||
|
_saving.Dispose();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21,6 +21,13 @@ public interface IAppPaths
|
|||||||
/// <remarks>Separate from the custom list: this one is derived data the app rewrites itself.</remarks>
|
/// <remarks>Separate from the custom list: this one is derived data the app rewrites itself.</remarks>
|
||||||
string ProxyStateFile { get; }
|
string ProxyStateFile { get; }
|
||||||
|
|
||||||
|
/// <summary>Full path of the user's own media-source list.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// A default member computed from <see cref="DataDirectory"/> rather than a constructor-set one,
|
||||||
|
/// so adding it does not break the test doubles that implement this interface by hand.
|
||||||
|
/// </remarks>
|
||||||
|
string UserSourcesFile => Path.Combine(DataDirectory, "sources.user.json");
|
||||||
|
|
||||||
/// <summary>Directory holding rolling log files.</summary>
|
/// <summary>Directory holding rolling log files.</summary>
|
||||||
string LogDirectory { get; }
|
string LogDirectory { get; }
|
||||||
|
|
||||||
|
|||||||
@@ -36,7 +36,12 @@ public static class UiServiceCollectionExtensions
|
|||||||
services.AddSingleton<ILocalizationService, LocalizationService>();
|
services.AddSingleton<ILocalizationService, LocalizationService>();
|
||||||
services.AddSingleton<INavigationService, NavigationService>();
|
services.AddSingleton<INavigationService, NavigationService>();
|
||||||
|
|
||||||
services.AddSingleton<DashboardViewModel>();
|
services.AddSingleton<DashboardViewModel>(static sp => new DashboardViewModel(
|
||||||
|
sp.GetRequiredService<IMediaSourceCatalog>(),
|
||||||
|
sp.GetRequiredService<IAppPaths>(),
|
||||||
|
sp.GetRequiredService<IProxyPool>(),
|
||||||
|
sp
|
||||||
|
));
|
||||||
services.AddSingleton<SettingsViewModel>(static sp => new SettingsViewModel(
|
services.AddSingleton<SettingsViewModel>(static sp => new SettingsViewModel(
|
||||||
sp.GetRequiredService<ISettingsService>(),
|
sp.GetRequiredService<ISettingsService>(),
|
||||||
sp.GetRequiredService<IThemeService>(),
|
sp.GetRequiredService<IThemeService>(),
|
||||||
|
|||||||
@@ -88,6 +88,30 @@
|
|||||||
<data name="Dashboard.DataDirectory" xml:space="preserve">
|
<data name="Dashboard.DataDirectory" xml:space="preserve">
|
||||||
<value>DATA DIRECTORY</value>
|
<value>DATA DIRECTORY</value>
|
||||||
</data>
|
</data>
|
||||||
|
<data name="Dashboard.Proxies" xml:space="preserve">
|
||||||
|
<value>Proxy pool</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Total" xml:space="preserve">
|
||||||
|
<value>KNOWN</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Live" xml:space="preserve">
|
||||||
|
<value>LIVE</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Quarantined" xml:space="preserve">
|
||||||
|
<value>SIDELINED</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Unchecked" xml:space="preserve">
|
||||||
|
<value>UNCHECKED</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Latency" xml:space="preserve">
|
||||||
|
<value>Average response: {0} ms</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Requests" xml:space="preserve">
|
||||||
|
<value>Requests through a proxy: {0} of {1} worked ({2})</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Empty" xml:space="preserve">
|
||||||
|
<value>The pool is empty. Add your own addresses, or switch the feed on in settings.</value>
|
||||||
|
</data>
|
||||||
<data name="Proxies.Pool" xml:space="preserve">
|
<data name="Proxies.Pool" xml:space="preserve">
|
||||||
<value>POOL</value>
|
<value>POOL</value>
|
||||||
</data>
|
</data>
|
||||||
@@ -385,11 +409,8 @@
|
|||||||
<data name="Proxies.Status.Ready" xml:space="preserve">
|
<data name="Proxies.Status.Ready" xml:space="preserve">
|
||||||
<value>{0} in the pool, {1} live.</value>
|
<value>{0} in the pool, {1} live.</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Settings.AllowDirect" xml:space="preserve">
|
<data name="Settings.AllowDirectMoved" xml:space="preserve">
|
||||||
<value>Allow network sources to run without a proxy</value>
|
<value>Whether a source may run without a proxy is now set per source, in the source editor on the Collect page.</value>
|
||||||
</data>
|
|
||||||
<data name="Settings.AllowDirectHint" xml:space="preserve">
|
|
||||||
<value>When off, a source that fetches anything refuses to start until at least one proxy answers. Sources that only read pasted text are never blocked.</value>
|
|
||||||
</data>
|
</data>
|
||||||
<data name="Settings.MinimumLive" xml:space="preserve">
|
<data name="Settings.MinimumLive" xml:space="preserve">
|
||||||
<value>LIVE PROXIES TO FIND AT STARTUP</value>
|
<value>LIVE PROXIES TO FIND AT STARTUP</value>
|
||||||
@@ -400,8 +421,11 @@
|
|||||||
<data name="Page.Collect" xml:space="preserve">
|
<data name="Page.Collect" xml:space="preserve">
|
||||||
<value>Collect</value>
|
<value>Collect</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Source" xml:space="preserve">
|
<data name="Collect.Sources" xml:space="preserve">
|
||||||
<value>SOURCE</value>
|
<value>SOURCES</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourcesHint" xml:space="preserve">
|
||||||
|
<value>Tick every source the run should cover; they run side by side.</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Actions" xml:space="preserve">
|
<data name="Collect.Actions" xml:space="preserve">
|
||||||
<value>ACTIONS</value>
|
<value>ACTIONS</value>
|
||||||
@@ -415,17 +439,107 @@
|
|||||||
<data name="Collect.ClearTip" xml:space="preserve">
|
<data name="Collect.ClearTip" xml:space="preserve">
|
||||||
<value>Clear the results</value>
|
<value>Clear the results</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Input" xml:space="preserve">
|
<data name="Collect.Target" xml:space="preserve">
|
||||||
<value>ADDRESSES</value>
|
<value>TARGET</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.InputPlaceholder" xml:space="preserve">
|
<data name="Collect.AttemptBudget" xml:space="preserve">
|
||||||
<value>One address per line. Lines starting with '#' are ignored.</value>
|
<value>ATTEMPTS</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Endpoint" xml:space="preserve">
|
<data name="Collect.AttemptBudgetHint" xml:space="preserve">
|
||||||
<value>LISTING ENDPOINT</value>
|
<value>Per source. 0 — until you stop it.</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.EndpointPlaceholder" xml:space="preserve">
|
<data name="Collect.ZeroIsUnlimited" xml:space="preserve">
|
||||||
<value>https://my-service.local/api/list</value>
|
<value>0 — no limit.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.AddSource" xml:space="preserve">
|
||||||
|
<value>Add source</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.EditSource" xml:space="preserve">
|
||||||
|
<value>Edit source</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.RemoveSource" xml:space="preserve">
|
||||||
|
<value>Delete source</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.NoSources" xml:space="preserve">
|
||||||
|
<value>No sources yet. Add one to start collecting.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceEditor" xml:space="preserve">
|
||||||
|
<value>SOURCE</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceName" xml:space="preserve">
|
||||||
|
<value>NAME</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceNamePlaceholder" xml:space="preserve">
|
||||||
|
<value>My image service</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceBaseUrl" xml:space="preserve">
|
||||||
|
<value>BASE URL</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceBaseUrlPlaceholder" xml:space="preserve">
|
||||||
|
<value>https://imgtest.example/test1/</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceMinLength" xml:space="preserve">
|
||||||
|
<value>MIN LENGTH</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceMaxLength" xml:space="preserve">
|
||||||
|
<value>MAX LENGTH</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceAllowDirect" xml:space="preserve">
|
||||||
|
<value>May run without a proxy</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceAllowDirectHint" xml:space="preserve">
|
||||||
|
<value>Off by default: with no live proxy this source refuses to start rather than fetching from your own address. Turn it on for a host you own, or one you do not mind being seen from here.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceExtension" xml:space="preserve">
|
||||||
|
<value>EXTENSION</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceAlphabet" xml:space="preserve">
|
||||||
|
<value>ID CHARACTERS</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceCustomAlphabet" xml:space="preserve">
|
||||||
|
<value>CUSTOM CHARACTERS</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceCustomAlphabetPlaceholder" xml:space="preserve">
|
||||||
|
<value>The exact characters ids may use</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceSave" xml:space="preserve">
|
||||||
|
<value>Save</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceCancel" xml:space="preserve">
|
||||||
|
<value>Cancel</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceError.NameRequired" xml:space="preserve">
|
||||||
|
<value>Give the source a name.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceError.BaseUrlInvalid" xml:space="preserve">
|
||||||
|
<value>The base URL must be an absolute http or https address.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceError.LengthRangeInvalid" xml:space="preserve">
|
||||||
|
<value>The id length range must be between 1 and 64, with min no greater than max.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceError.AlphabetEmpty" xml:space="preserve">
|
||||||
|
<value>The chosen characters resolve to an empty set.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.LettersLower" xml:space="preserve">
|
||||||
|
<value>Lower-case letters (a–z)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.LettersUpper" xml:space="preserve">
|
||||||
|
<value>Upper-case letters (A–Z)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.Letters" xml:space="preserve">
|
||||||
|
<value>Letters (a–z, A–Z)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.Digits" xml:space="preserve">
|
||||||
|
<value>Digits (0–9)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.Alphanumeric" xml:space="preserve">
|
||||||
|
<value>Letters and digits</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.HexLower" xml:space="preserve">
|
||||||
|
<value>Hexadecimal (0–9, a–f)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.Custom" xml:space="preserve">
|
||||||
|
<value>Custom…</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.ForceRefetch" xml:space="preserve">
|
<data name="Collect.ForceRefetch" xml:space="preserve">
|
||||||
<value>Fetch everything again</value>
|
<value>Fetch everything again</value>
|
||||||
@@ -436,8 +550,41 @@
|
|||||||
<data name="Collect.Items" xml:space="preserve">
|
<data name="Collect.Items" xml:space="preserve">
|
||||||
<value>COLLECTED</value>
|
<value>COLLECTED</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Errors" xml:space="preserve">
|
<data name="Collect.Log" xml:space="preserve">
|
||||||
<value>ERRORS</value>
|
<value>LOG</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Started" xml:space="preserve">
|
||||||
|
<value>Run started: {0}, attempts {1}, target {2}</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.SourceStarted" xml:space="preserve">
|
||||||
|
<value>started</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.SourceFinished" xml:space="preserve">
|
||||||
|
<value>out of candidates</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Finished" xml:space="preserve">
|
||||||
|
<value>Run finished. {0}</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Stopped" xml:space="preserve">
|
||||||
|
<value>Run stopped. {0}</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Unlimited" xml:space="preserve">
|
||||||
|
<value>no limit</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Copy" xml:space="preserve">
|
||||||
|
<value>Copy</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.CopyAll" xml:space="preserve">
|
||||||
|
<value>Copy the whole log</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Count.Sources.One" xml:space="preserve">
|
||||||
|
<value>{0} source</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Count.Sources.Few" xml:space="preserve">
|
||||||
|
<value>{0} sources</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Count.Sources.Many" xml:space="preserve">
|
||||||
|
<value>{0} sources</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.ProxyRequired" xml:space="preserve">
|
<data name="Collect.ProxyRequired" xml:space="preserve">
|
||||||
<value>This source needs a working proxy and none is live. Check the proxy list, or allow direct connections in settings.</value>
|
<value>This source needs a working proxy and none is live. Check the proxy list, or allow direct connections in settings.</value>
|
||||||
@@ -493,30 +640,9 @@
|
|||||||
<data name="Collect.Count.Errors.Many" xml:space="preserve">
|
<data name="Collect.Count.Errors.Many" xml:space="preserve">
|
||||||
<value>{0} errors</value>
|
<value>{0} errors</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Source.url-list.Name" xml:space="preserve">
|
|
||||||
<value>URL list</value>
|
|
||||||
</data>
|
|
||||||
<data name="Source.url-list.Description" xml:space="preserve">
|
|
||||||
<value>Addresses you paste in, one per line.</value>
|
|
||||||
</data>
|
|
||||||
<data name="Source.own-service.Name" xml:space="preserve">
|
|
||||||
<value>Own service</value>
|
|
||||||
</data>
|
|
||||||
<data name="Source.own-service.Description" xml:space="preserve">
|
|
||||||
<value>Reads the listing endpoint of a service you run.</value>
|
|
||||||
</data>
|
|
||||||
<data name="Collect.Error.NotAnAddress" xml:space="preserve">
|
<data name="Collect.Error.NotAnAddress" xml:space="preserve">
|
||||||
<value>'{0}' is not an http or https address.</value>
|
<value>'{0}' is not an http or https address.</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Error.NoEndpoint" xml:space="preserve">
|
|
||||||
<value>No listing endpoint was given.</value>
|
|
||||||
</data>
|
|
||||||
<data name="Collect.Error.ListingFailed" xml:space="preserve">
|
|
||||||
<value>Could not read the listing: {0}</value>
|
|
||||||
</data>
|
|
||||||
<data name="Collect.Error.ListingMalformed" xml:space="preserve">
|
|
||||||
<value>The listing could not be understood: {0}</value>
|
|
||||||
</data>
|
|
||||||
<data name="Collect.Error.SourceFailed" xml:space="preserve">
|
<data name="Collect.Error.SourceFailed" xml:space="preserve">
|
||||||
<value>The source stopped: {0}</value>
|
<value>The source stopped: {0}</value>
|
||||||
</data>
|
</data>
|
||||||
|
|||||||
@@ -88,6 +88,30 @@
|
|||||||
<data name="Dashboard.DataDirectory" xml:space="preserve">
|
<data name="Dashboard.DataDirectory" xml:space="preserve">
|
||||||
<value>КАТАЛОГ ДАННЫХ</value>
|
<value>КАТАЛОГ ДАННЫХ</value>
|
||||||
</data>
|
</data>
|
||||||
|
<data name="Dashboard.Proxies" xml:space="preserve">
|
||||||
|
<value>Пул прокси</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Total" xml:space="preserve">
|
||||||
|
<value>ВСЕГО</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Live" xml:space="preserve">
|
||||||
|
<value>ЖИВЫХ</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Quarantined" xml:space="preserve">
|
||||||
|
<value>В КАРАНТИНЕ</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Unchecked" xml:space="preserve">
|
||||||
|
<value>НЕ ПРОВЕРЕНО</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Latency" xml:space="preserve">
|
||||||
|
<value>Средний отклик: {0} мс</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Requests" xml:space="preserve">
|
||||||
|
<value>Запросов через прокси: {0} из {1} удались ({2})</value>
|
||||||
|
</data>
|
||||||
|
<data name="Dashboard.Proxy.Empty" xml:space="preserve">
|
||||||
|
<value>Пул пуст. Добавьте свои адреса или включите фид в настройках.</value>
|
||||||
|
</data>
|
||||||
<data name="Proxies.Pool" xml:space="preserve">
|
<data name="Proxies.Pool" xml:space="preserve">
|
||||||
<value>ПУЛ</value>
|
<value>ПУЛ</value>
|
||||||
</data>
|
</data>
|
||||||
@@ -385,11 +409,8 @@
|
|||||||
<data name="Proxies.Status.Ready" xml:space="preserve">
|
<data name="Proxies.Status.Ready" xml:space="preserve">
|
||||||
<value>В пуле {0}, живых {1}.</value>
|
<value>В пуле {0}, живых {1}.</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Settings.AllowDirect" xml:space="preserve">
|
<data name="Settings.AllowDirectMoved" xml:space="preserve">
|
||||||
<value>Разрешить сетевым источникам работать без прокси</value>
|
<value>Разрешение работать без прокси теперь задаётся у каждого источника отдельно — в редакторе источника на странице «Сбор».</value>
|
||||||
</data>
|
|
||||||
<data name="Settings.AllowDirectHint" xml:space="preserve">
|
|
||||||
<value>Когда выключено, источник, который куда-то ходит, не запустится, пока не ответит хотя бы одна прокси. Источники, читающие вставленный текст, не блокируются никогда.</value>
|
|
||||||
</data>
|
</data>
|
||||||
<data name="Settings.MinimumLive" xml:space="preserve">
|
<data name="Settings.MinimumLive" xml:space="preserve">
|
||||||
<value>СКОЛЬКО ЖИВЫХ ИСКАТЬ ПРИ СТАРТЕ</value>
|
<value>СКОЛЬКО ЖИВЫХ ИСКАТЬ ПРИ СТАРТЕ</value>
|
||||||
@@ -400,8 +421,11 @@
|
|||||||
<data name="Page.Collect" xml:space="preserve">
|
<data name="Page.Collect" xml:space="preserve">
|
||||||
<value>Сбор</value>
|
<value>Сбор</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Source" xml:space="preserve">
|
<data name="Collect.Sources" xml:space="preserve">
|
||||||
<value>ИСТОЧНИК</value>
|
<value>ИСТОЧНИКИ</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourcesHint" xml:space="preserve">
|
||||||
|
<value>Отметьте источники для запуска — они работают одновременно.</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Actions" xml:space="preserve">
|
<data name="Collect.Actions" xml:space="preserve">
|
||||||
<value>ДЕЙСТВИЯ</value>
|
<value>ДЕЙСТВИЯ</value>
|
||||||
@@ -415,17 +439,107 @@
|
|||||||
<data name="Collect.ClearTip" xml:space="preserve">
|
<data name="Collect.ClearTip" xml:space="preserve">
|
||||||
<value>Очистить результаты</value>
|
<value>Очистить результаты</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Input" xml:space="preserve">
|
<data name="Collect.Target" xml:space="preserve">
|
||||||
<value>АДРЕСА</value>
|
<value>ЦЕЛЬ</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.InputPlaceholder" xml:space="preserve">
|
<data name="Collect.AttemptBudget" xml:space="preserve">
|
||||||
<value>По одному адресу в строке. Строки, начинающиеся с «#», игнорируются.</value>
|
<value>ПОПЫТКИ</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Endpoint" xml:space="preserve">
|
<data name="Collect.AttemptBudgetHint" xml:space="preserve">
|
||||||
<value>АДРЕС ЛИСТИНГА</value>
|
<value>На каждый источник. 0 — пока не остановите.</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.EndpointPlaceholder" xml:space="preserve">
|
<data name="Collect.ZeroIsUnlimited" xml:space="preserve">
|
||||||
<value>https://my-service.local/api/list</value>
|
<value>0 — без ограничения.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.AddSource" xml:space="preserve">
|
||||||
|
<value>Добавить источник</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.EditSource" xml:space="preserve">
|
||||||
|
<value>Изменить источник</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.RemoveSource" xml:space="preserve">
|
||||||
|
<value>Удалить источник</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.NoSources" xml:space="preserve">
|
||||||
|
<value>Источников пока нет. Добавьте, чтобы начать сбор.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceEditor" xml:space="preserve">
|
||||||
|
<value>ИСТОЧНИК</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceName" xml:space="preserve">
|
||||||
|
<value>НАЗВАНИЕ</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceNamePlaceholder" xml:space="preserve">
|
||||||
|
<value>Мой сервис картинок</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceBaseUrl" xml:space="preserve">
|
||||||
|
<value>БАЗОВЫЙ URL</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceBaseUrlPlaceholder" xml:space="preserve">
|
||||||
|
<value>https://imgtest.example/test1/</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceMinLength" xml:space="preserve">
|
||||||
|
<value>МИН. ДЛИНА</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceMaxLength" xml:space="preserve">
|
||||||
|
<value>МАКС. ДЛИНА</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceAllowDirect" xml:space="preserve">
|
||||||
|
<value>Может работать без прокси</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceAllowDirectHint" xml:space="preserve">
|
||||||
|
<value>По умолчанию выключено: без живой прокси источник не запустится, а не пойдёт с вашего адреса. Включайте для своего хоста или там, где вас не смущает быть видимым отсюда.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceExtension" xml:space="preserve">
|
||||||
|
<value>РАСШИРЕНИЕ</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceAlphabet" xml:space="preserve">
|
||||||
|
<value>СИМВОЛЫ ID</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceCustomAlphabet" xml:space="preserve">
|
||||||
|
<value>СВОИ СИМВОЛЫ</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceCustomAlphabetPlaceholder" xml:space="preserve">
|
||||||
|
<value>Точный набор символов для id</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceSave" xml:space="preserve">
|
||||||
|
<value>Сохранить</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceCancel" xml:space="preserve">
|
||||||
|
<value>Отмена</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceError.NameRequired" xml:space="preserve">
|
||||||
|
<value>Задайте название источника.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceError.BaseUrlInvalid" xml:space="preserve">
|
||||||
|
<value>Базовый URL должен быть абсолютным адресом http или https.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceError.LengthRangeInvalid" xml:space="preserve">
|
||||||
|
<value>Длина id должна быть от 1 до 64, причём минимум не больше максимума.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.SourceError.AlphabetEmpty" xml:space="preserve">
|
||||||
|
<value>Выбранные символы дают пустой набор.</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.LettersLower" xml:space="preserve">
|
||||||
|
<value>Строчные буквы (a–z)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.LettersUpper" xml:space="preserve">
|
||||||
|
<value>Прописные буквы (A–Z)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.Letters" xml:space="preserve">
|
||||||
|
<value>Буквы (a–z, A–Z)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.Digits" xml:space="preserve">
|
||||||
|
<value>Цифры (0–9)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.Alphanumeric" xml:space="preserve">
|
||||||
|
<value>Буквы и цифры</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.HexLower" xml:space="preserve">
|
||||||
|
<value>Шестнадцатеричные (0–9, a–f)</value>
|
||||||
|
</data>
|
||||||
|
<data name="Enum.IdAlphabet.Custom" xml:space="preserve">
|
||||||
|
<value>Свои…</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.ForceRefetch" xml:space="preserve">
|
<data name="Collect.ForceRefetch" xml:space="preserve">
|
||||||
<value>Скачать всё заново</value>
|
<value>Скачать всё заново</value>
|
||||||
@@ -436,8 +550,41 @@
|
|||||||
<data name="Collect.Items" xml:space="preserve">
|
<data name="Collect.Items" xml:space="preserve">
|
||||||
<value>СОБРАНО</value>
|
<value>СОБРАНО</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Errors" xml:space="preserve">
|
<data name="Collect.Log" xml:space="preserve">
|
||||||
<value>ОШИБКИ</value>
|
<value>ЖУРНАЛ</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Started" xml:space="preserve">
|
||||||
|
<value>Запуск: {0}, попыток {1}, цель {2}</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.SourceStarted" xml:space="preserve">
|
||||||
|
<value>запущен</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.SourceFinished" xml:space="preserve">
|
||||||
|
<value>кандидаты закончились</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Finished" xml:space="preserve">
|
||||||
|
<value>Готово. {0}</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Stopped" xml:space="preserve">
|
||||||
|
<value>Остановлено. {0}</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Unlimited" xml:space="preserve">
|
||||||
|
<value>без ограничения</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.Copy" xml:space="preserve">
|
||||||
|
<value>Копировать</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Log.CopyAll" xml:space="preserve">
|
||||||
|
<value>Копировать весь журнал</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Count.Sources.One" xml:space="preserve">
|
||||||
|
<value>{0} источник</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Count.Sources.Few" xml:space="preserve">
|
||||||
|
<value>{0} источника</value>
|
||||||
|
</data>
|
||||||
|
<data name="Collect.Count.Sources.Many" xml:space="preserve">
|
||||||
|
<value>{0} источников</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.ProxyRequired" xml:space="preserve">
|
<data name="Collect.ProxyRequired" xml:space="preserve">
|
||||||
<value>Этому источнику нужна рабочая прокси, а живых нет. Проверьте список прокси или разрешите прямое подключение в настройках.</value>
|
<value>Этому источнику нужна рабочая прокси, а живых нет. Проверьте список прокси или разрешите прямое подключение в настройках.</value>
|
||||||
@@ -493,30 +640,9 @@
|
|||||||
<data name="Collect.Count.Errors.Many" xml:space="preserve">
|
<data name="Collect.Count.Errors.Many" xml:space="preserve">
|
||||||
<value>{0} ошибок</value>
|
<value>{0} ошибок</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Source.url-list.Name" xml:space="preserve">
|
|
||||||
<value>Список ссылок</value>
|
|
||||||
</data>
|
|
||||||
<data name="Source.url-list.Description" xml:space="preserve">
|
|
||||||
<value>Адреса, которые вы вставляете сами, по одному в строке.</value>
|
|
||||||
</data>
|
|
||||||
<data name="Source.own-service.Name" xml:space="preserve">
|
|
||||||
<value>Свой сервис</value>
|
|
||||||
</data>
|
|
||||||
<data name="Source.own-service.Description" xml:space="preserve">
|
|
||||||
<value>Читает листинг сервиса, который вы держите сами.</value>
|
|
||||||
</data>
|
|
||||||
<data name="Collect.Error.NotAnAddress" xml:space="preserve">
|
<data name="Collect.Error.NotAnAddress" xml:space="preserve">
|
||||||
<value>«{0}» — не http- и не https-адрес.</value>
|
<value>«{0}» — не http- и не https-адрес.</value>
|
||||||
</data>
|
</data>
|
||||||
<data name="Collect.Error.NoEndpoint" xml:space="preserve">
|
|
||||||
<value>Не указан адрес листинга.</value>
|
|
||||||
</data>
|
|
||||||
<data name="Collect.Error.ListingFailed" xml:space="preserve">
|
|
||||||
<value>Не удалось прочитать листинг: {0}</value>
|
|
||||||
</data>
|
|
||||||
<data name="Collect.Error.ListingMalformed" xml:space="preserve">
|
|
||||||
<value>Листинг не удалось разобрать: {0}</value>
|
|
||||||
</data>
|
|
||||||
<data name="Collect.Error.SourceFailed" xml:space="preserve">
|
<data name="Collect.Error.SourceFailed" xml:space="preserve">
|
||||||
<value>Источник остановился: {0}</value>
|
<value>Источник остановился: {0}</value>
|
||||||
</data>
|
</data>
|
||||||
|
|||||||
@@ -59,6 +59,10 @@
|
|||||||
|
|
||||||
<StreamGeometry x:Key="IconTrash">M9 3h6l1 1h4v2H4V4h4l1-1zM6 7h12l-1 13H7L6 7z</StreamGeometry>
|
<StreamGeometry x:Key="IconTrash">M9 3h6l1 1h4v2H4V4h4l1-1zM6 7h12l-1 13H7L6 7z</StreamGeometry>
|
||||||
|
|
||||||
|
<StreamGeometry x:Key="IconEdit">
|
||||||
|
M3 17.25V21h3.75L17.81 9.94l-3.75-3.75L3 17.25zM20.71 7.04a1 1 0 0 0 0-1.41l-2.34-2.34a1 1 0 0 0-1.41 0l-1.83 1.83 3.75 3.75 1.83-1.83z
|
||||||
|
</StreamGeometry>
|
||||||
|
|
||||||
<StreamGeometry x:Key="IconSparkle">
|
<StreamGeometry x:Key="IconSparkle">
|
||||||
M12 2.5 13.9 9l6.6 1.9-6.6 1.9L12 19.4l-1.9-6.6L3.5 11 10.1 9 12 2.5z
|
M12 2.5 13.9 9l6.6 1.9-6.6 1.9L12 19.4l-1.9-6.6L3.5 11 10.1 9 12 2.5z
|
||||||
</StreamGeometry>
|
</StreamGeometry>
|
||||||
|
|||||||
@@ -1,40 +0,0 @@
|
|||||||
using AvParser.Core.Parsing;
|
|
||||||
using AvParser.UI.Localization;
|
|
||||||
using ReactiveUI;
|
|
||||||
|
|
||||||
namespace AvParser.UI.ViewModels;
|
|
||||||
|
|
||||||
/// <summary>One row of the collect error list, with its message translated.</summary>
|
|
||||||
/// <remarks>
|
|
||||||
/// Sources and the fetcher are part of the domain and produce English text plus a code. This resolves
|
|
||||||
/// <c>Collect.Error.{Code}</c> and falls back to the domain's own wording, so a code that has not
|
|
||||||
/// been translated yet still says something useful instead of showing a missing-key marker.
|
|
||||||
/// </remarks>
|
|
||||||
public sealed class CollectErrorViewModel(ParseError error) : ReactiveObject
|
|
||||||
{
|
|
||||||
/// <summary>The underlying error.</summary>
|
|
||||||
public ParseError Error { get; } = error ?? throw new ArgumentNullException(nameof(error));
|
|
||||||
|
|
||||||
/// <summary>1-based position of the offending item: a pasted line, or a place in a listing.</summary>
|
|
||||||
public int Index => Error.Index;
|
|
||||||
|
|
||||||
/// <summary>Translated message.</summary>
|
|
||||||
public string Text
|
|
||||||
{
|
|
||||||
get
|
|
||||||
{
|
|
||||||
if (Error.Code is not { Length: > 0 } code)
|
|
||||||
{
|
|
||||||
return Error.Message;
|
|
||||||
}
|
|
||||||
|
|
||||||
var template = Localizer.Instance.GetOrDefault($"Collect.Error.{code}", Error.Message);
|
|
||||||
return Error.Arguments.Count == 0
|
|
||||||
? template
|
|
||||||
: string.Format(Localizer.Instance.Culture, template, [.. Error.Arguments]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// <summary>Re-reads the translated message.</summary>
|
|
||||||
public void Refresh() => this.RaisePropertyChanged(nameof(Text));
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
using System.Globalization;
|
||||||
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Parsing;
|
||||||
|
using AvParser.UI.Localization;
|
||||||
|
using ReactiveUI;
|
||||||
|
|
||||||
|
namespace AvParser.UI.ViewModels;
|
||||||
|
|
||||||
|
/// <summary>How loud one log line is, which is also how it is coloured.</summary>
|
||||||
|
public enum CollectLogLevel
|
||||||
|
{
|
||||||
|
/// <summary>Lifecycle: a run started, a source finished.</summary>
|
||||||
|
Info = 0,
|
||||||
|
|
||||||
|
/// <summary>Something was collected.</summary>
|
||||||
|
Success = 1,
|
||||||
|
|
||||||
|
/// <summary>Nothing was collected, but nothing went wrong either: a duplicate or a skip.</summary>
|
||||||
|
Warning = 2,
|
||||||
|
|
||||||
|
/// <summary>A failure. The run carries on regardless.</summary>
|
||||||
|
Error = 3,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// One line of the live collection log.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Holds the resource key and its arguments rather than the finished sentence: switching language
|
||||||
|
/// mid-run must not leave a page of half-English text behind. The detail half — an address, a size —
|
||||||
|
/// is literal and stays as it is, because it never was English.
|
||||||
|
/// </remarks>
|
||||||
|
public sealed class CollectLogEntryViewModel : ReactiveObject
|
||||||
|
{
|
||||||
|
private readonly string _key;
|
||||||
|
private readonly string? _fallback;
|
||||||
|
private readonly object?[] _arguments;
|
||||||
|
private readonly string? _detail;
|
||||||
|
|
||||||
|
private CollectLogEntryViewModel(
|
||||||
|
CollectLogLevel level,
|
||||||
|
string key,
|
||||||
|
string? fallback,
|
||||||
|
object?[] arguments,
|
||||||
|
string? detail,
|
||||||
|
string? source
|
||||||
|
)
|
||||||
|
{
|
||||||
|
Level = level;
|
||||||
|
Source = source;
|
||||||
|
Time = DateTime.Now;
|
||||||
|
|
||||||
|
_key = key;
|
||||||
|
_fallback = fallback;
|
||||||
|
_arguments = arguments;
|
||||||
|
_detail = detail;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Severity of the line.</summary>
|
||||||
|
public CollectLogLevel Level { get; }
|
||||||
|
|
||||||
|
/// <summary>Name of the source it came from; <see langword="null"/> for whole-run lines.</summary>
|
||||||
|
public string? Source { get; }
|
||||||
|
|
||||||
|
/// <summary>When it happened, on the wall clock.</summary>
|
||||||
|
public DateTime Time { get; }
|
||||||
|
|
||||||
|
/// <summary>Timestamp, seconds precision — a log with milliseconds is unreadable at this rate.</summary>
|
||||||
|
public string TimeText => Time.ToString("HH:mm:ss", CultureInfo.InvariantCulture);
|
||||||
|
|
||||||
|
/// <summary>Single-character severity marker. Symbols rather than words, so nothing to translate.</summary>
|
||||||
|
public string LevelText =>
|
||||||
|
Level switch
|
||||||
|
{
|
||||||
|
CollectLogLevel.Success => "+",
|
||||||
|
CollectLogLevel.Warning => "~",
|
||||||
|
CollectLogLevel.Error => "×",
|
||||||
|
_ => "·",
|
||||||
|
};
|
||||||
|
|
||||||
|
/// <summary>Whether this line reports a failure; drives the red chip.</summary>
|
||||||
|
public bool IsError => Level == CollectLogLevel.Error;
|
||||||
|
|
||||||
|
/// <summary>Whether this line reports something collected; drives the green chip.</summary>
|
||||||
|
public bool IsSuccess => Level == CollectLogLevel.Success;
|
||||||
|
|
||||||
|
/// <summary>Whether there is a source to name on the line.</summary>
|
||||||
|
public bool HasSource => !string.IsNullOrEmpty(Source);
|
||||||
|
|
||||||
|
/// <summary>The translated message, with its literal detail appended.</summary>
|
||||||
|
public string Text
|
||||||
|
{
|
||||||
|
get
|
||||||
|
{
|
||||||
|
var template = _fallback is null
|
||||||
|
? Localizer.Instance[_key]
|
||||||
|
: Localizer.Instance.GetOrDefault(_key, _fallback);
|
||||||
|
|
||||||
|
var message =
|
||||||
|
_arguments.Length == 0 ? template : string.Format(Localizer.Instance.Culture, template, _arguments);
|
||||||
|
|
||||||
|
return _detail is null ? message : $"{message} {_detail}";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Re-reads the translated half of the line.</summary>
|
||||||
|
public void Refresh() => this.RaisePropertyChanged(nameof(Text));
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The line as one string: what a copy puts on the clipboard.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Built here rather than in the view so the copied text is the line the user is looking at —
|
||||||
|
/// same timestamp, same source, same wording — instead of a re-render that quietly drifts from
|
||||||
|
/// the template. The severity marker is left out: it is a colour cue, not information.
|
||||||
|
/// </remarks>
|
||||||
|
public override string ToString() => HasSource ? $"{TimeText} {Source} {Text}" : $"{TimeText} {Text}";
|
||||||
|
|
||||||
|
/// <summary>Joins lines for the clipboard, in the order they were logged.</summary>
|
||||||
|
public static string ToText(IEnumerable<CollectLogEntryViewModel> entries)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(entries);
|
||||||
|
|
||||||
|
return string.Join(Environment.NewLine, entries.Select(entry => entry.ToString()));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A lifecycle line: run started, source finished, and so on.</summary>
|
||||||
|
public static CollectLogEntryViewModel Message(
|
||||||
|
CollectLogLevel level,
|
||||||
|
string key,
|
||||||
|
string? source = null,
|
||||||
|
params object?[] arguments
|
||||||
|
) => new(level, key, null, arguments, null, source);
|
||||||
|
|
||||||
|
/// <summary>A line for one collected, duplicated or skipped item.</summary>
|
||||||
|
public static CollectLogEntryViewModel ForItem(CollectedItem item, string? source)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(item);
|
||||||
|
|
||||||
|
var level = item.Status switch
|
||||||
|
{
|
||||||
|
CollectStatus.Stored => CollectLogLevel.Success,
|
||||||
|
_ => CollectLogLevel.Warning,
|
||||||
|
};
|
||||||
|
|
||||||
|
var size = CollectedItemViewModel.FormatSize(item.Blob.Length);
|
||||||
|
var detail = size.Length == 0 ? item.Candidate.Url.AbsoluteUri : $"{item.Candidate.Url.AbsoluteUri} {size}";
|
||||||
|
|
||||||
|
return new CollectLogEntryViewModel(level, $"Collect.Item.{item.Status}", null, [], detail, source);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A line for one failure, translated by code with the domain's wording as fallback.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The domain produces English text plus a code; this resolves <c>Collect.Error.{Code}</c> and
|
||||||
|
/// falls back to <see cref="ParseError.Message"/>, so a code nobody has translated yet still says
|
||||||
|
/// something useful instead of showing a missing-key marker.
|
||||||
|
/// </remarks>
|
||||||
|
public static CollectLogEntryViewModel ForError(ParseError error, string? source)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(error);
|
||||||
|
|
||||||
|
return new CollectLogEntryViewModel(
|
||||||
|
CollectLogLevel.Error,
|
||||||
|
$"Collect.Error.{error.Code}",
|
||||||
|
error.Message,
|
||||||
|
[.. error.Arguments],
|
||||||
|
error.Subject,
|
||||||
|
source
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -1,31 +1,76 @@
|
|||||||
|
using System.Globalization;
|
||||||
using AvParser.Core.Collecting;
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Proxies;
|
||||||
using AvParser.Infrastructure.Storage;
|
using AvParser.Infrastructure.Storage;
|
||||||
|
using AvParser.UI.Localization;
|
||||||
using AvParser.UI.Navigation;
|
using AvParser.UI.Navigation;
|
||||||
using Microsoft.Extensions.DependencyInjection;
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
using ReactiveUI;
|
using ReactiveUI;
|
||||||
using ReactiveUI.Primitives;
|
using ReactiveUI.Primitives;
|
||||||
|
using ReactiveUI.Primitives.Concurrency;
|
||||||
|
using ReactiveUI.Primitives.Signals;
|
||||||
|
using ReactiveUI.SourceGenerators;
|
||||||
|
|
||||||
namespace AvParser.UI.ViewModels;
|
namespace AvParser.UI.ViewModels;
|
||||||
|
|
||||||
/// <summary>Landing page: what is registered, where data lives, and shortcuts into the app.</summary>
|
/// <summary>Landing page: what is registered, how the proxy pool is doing, and where data lives.</summary>
|
||||||
public sealed class DashboardViewModel : PageViewModel
|
public sealed partial class DashboardViewModel : PageViewModel, IDisposable
|
||||||
{
|
{
|
||||||
private readonly IServiceProvider _services;
|
private readonly IServiceProvider _services;
|
||||||
|
private readonly IProxyPool _pool;
|
||||||
|
private readonly ISequencer _mainThread;
|
||||||
|
private readonly Signal<RxVoid> _poolChanged = new();
|
||||||
|
|
||||||
|
private int _disposed;
|
||||||
|
|
||||||
|
/// <summary>Entries in the pool, feed and custom alike.</summary>
|
||||||
|
[Reactive]
|
||||||
|
public partial int ProxyTotal { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>How many answered this session and are not sidelined — the number the gate reads.</summary>
|
||||||
|
[Reactive]
|
||||||
|
public partial int ProxyLive { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>How many are serving a quarantine right now.</summary>
|
||||||
|
[Reactive]
|
||||||
|
public partial int ProxyQuarantined { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>How many nobody has spoken to yet this session.</summary>
|
||||||
|
[Reactive]
|
||||||
|
public partial int ProxyUnchecked { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>Average round trip of the proxies that answered; <see langword="null"/> when none did.</summary>
|
||||||
|
[Reactive]
|
||||||
|
public partial string? ProxyLatency { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>Requests that went through a proxy and how many worked; null before any were made.</summary>
|
||||||
|
[Reactive]
|
||||||
|
public partial string? ProxyRequests { get; private set; }
|
||||||
|
|
||||||
/// <summary>Creates the dashboard.</summary>
|
/// <summary>Creates the dashboard.</summary>
|
||||||
/// <param name="catalog">Registered media sources, shown as cards.</param>
|
/// <param name="catalog">Registered media sources, shown as cards.</param>
|
||||||
/// <param name="paths">Where the app writes settings and logs.</param>
|
/// <param name="paths">Where the app writes settings and logs.</param>
|
||||||
|
/// <param name="pool">The proxy pool, summarised on the page.</param>
|
||||||
/// <param name="services">
|
/// <param name="services">
|
||||||
/// Used to resolve <see cref="INavigationService"/> at click time rather than at construction
|
/// Used to resolve <see cref="INavigationService"/> at click time rather than at construction
|
||||||
/// time. Injecting it directly would be a cycle: the navigation service is built from every
|
/// time. Injecting it directly would be a cycle: the navigation service is built from every
|
||||||
/// page, so a page cannot also depend on it up front.
|
/// page, so a page cannot also depend on it up front.
|
||||||
/// </param>
|
/// </param>
|
||||||
public DashboardViewModel(IMediaSourceCatalog catalog, IAppPaths paths, IServiceProvider services)
|
/// <param name="mainThread">Scheduler for UI-affine updates; tests pass an immediate one.</param>
|
||||||
|
public DashboardViewModel(
|
||||||
|
IMediaSourceCatalog catalog,
|
||||||
|
IAppPaths paths,
|
||||||
|
IProxyPool pool,
|
||||||
|
IServiceProvider services,
|
||||||
|
ISequencer? mainThread = null
|
||||||
|
)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(catalog);
|
ArgumentNullException.ThrowIfNull(catalog);
|
||||||
ArgumentNullException.ThrowIfNull(paths);
|
ArgumentNullException.ThrowIfNull(paths);
|
||||||
|
|
||||||
_services = services ?? throw new ArgumentNullException(nameof(services));
|
_services = services ?? throw new ArgumentNullException(nameof(services));
|
||||||
|
_pool = pool ?? throw new ArgumentNullException(nameof(pool));
|
||||||
|
_mainThread = mainThread ?? RxSchedulers.MainThreadScheduler;
|
||||||
|
|
||||||
Sources = [.. catalog.Sources.Select(source => new MediaSourceViewModel(source))];
|
Sources = [.. catalog.Sources.Select(source => new MediaSourceViewModel(source))];
|
||||||
DataDirectory = paths.DataDirectory;
|
DataDirectory = paths.DataDirectory;
|
||||||
@@ -33,6 +78,17 @@ public sealed class DashboardViewModel : PageViewModel
|
|||||||
|
|
||||||
GoToCollectCommand = ReactiveCommand.Create(() => Navigate<CollectViewModel>());
|
GoToCollectCommand = ReactiveCommand.Create(() => Navigate<CollectViewModel>());
|
||||||
GoToSettingsCommand = ReactiveCommand.Create(() => Navigate<SettingsViewModel>());
|
GoToSettingsCommand = ReactiveCommand.Create(() => Navigate<SettingsViewModel>());
|
||||||
|
GoToProxiesCommand = ReactiveCommand.Create(() => Navigate<ProxiesViewModel>());
|
||||||
|
|
||||||
|
// The pool fires on every lease outcome and every probe — a sweep of a few thousand entries
|
||||||
|
// would otherwise repaint the page a few thousand times.
|
||||||
|
_pool.Changed += OnPoolChanged;
|
||||||
|
_poolChanged
|
||||||
|
.Throttle(TimeSpan.FromMilliseconds(250), _mainThread)
|
||||||
|
.ObserveOn(_mainThread)
|
||||||
|
.Subscribe(_ => RefreshProxyStats());
|
||||||
|
|
||||||
|
RefreshProxyStats();
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
@@ -50,12 +106,89 @@ public sealed class DashboardViewModel : PageViewModel
|
|||||||
/// <summary>Where collected media is written.</summary>
|
/// <summary>Where collected media is written.</summary>
|
||||||
public string MediaDirectory { get; }
|
public string MediaDirectory { get; }
|
||||||
|
|
||||||
|
/// <summary>Whether the pool holds anything at all; drives the empty state.</summary>
|
||||||
|
public bool HasProxies => ProxyTotal > 0;
|
||||||
|
|
||||||
/// <summary>Jumps to the Collect page.</summary>
|
/// <summary>Jumps to the Collect page.</summary>
|
||||||
public ReactiveCommand<RxVoid, RxVoid> GoToCollectCommand { get; }
|
public ReactiveCommand<RxVoid, RxVoid> GoToCollectCommand { get; }
|
||||||
|
|
||||||
/// <summary>Jumps to the Settings page.</summary>
|
/// <summary>Jumps to the Settings page.</summary>
|
||||||
public ReactiveCommand<RxVoid, RxVoid> GoToSettingsCommand { get; }
|
public ReactiveCommand<RxVoid, RxVoid> GoToSettingsCommand { get; }
|
||||||
|
|
||||||
|
/// <summary>Jumps to the Proxies page.</summary>
|
||||||
|
public ReactiveCommand<RxVoid, RxVoid> GoToProxiesCommand { get; }
|
||||||
|
|
||||||
|
/// <summary>Re-reads the pool. Exposed so tests can drive it without waiting for the throttle.</summary>
|
||||||
|
public void RefreshProxyStats()
|
||||||
|
{
|
||||||
|
var entries = _pool.Entries;
|
||||||
|
var now = DateTimeOffset.UtcNow;
|
||||||
|
var culture = CultureInfo.CurrentCulture;
|
||||||
|
|
||||||
|
ProxyTotal = entries.Count;
|
||||||
|
ProxyLive = _pool.LiveCount;
|
||||||
|
ProxyQuarantined = entries.Count(entry => entry.IsQuarantined(now));
|
||||||
|
ProxyUnchecked = entries.Count(entry => entry.Health == ProxyHealthState.Unknown);
|
||||||
|
|
||||||
|
var latencies = entries
|
||||||
|
.Where(entry => entry.Latency is not null)
|
||||||
|
.Select(entry => entry.Latency!.Value)
|
||||||
|
.ToArray();
|
||||||
|
|
||||||
|
ProxyLatency =
|
||||||
|
latencies.Length == 0
|
||||||
|
? null
|
||||||
|
: Localizer.Instance.Format(
|
||||||
|
"Dashboard.Proxy.Latency",
|
||||||
|
latencies.Average(latency => latency.TotalMilliseconds).ToString("N0", culture)
|
||||||
|
);
|
||||||
|
|
||||||
|
var successes = entries.Sum(entry => entry.SuccessCount);
|
||||||
|
var attempts = successes + entries.Sum(entry => entry.FailureCount);
|
||||||
|
|
||||||
|
ProxyRequests =
|
||||||
|
attempts == 0
|
||||||
|
? null
|
||||||
|
: Localizer.Instance.Format(
|
||||||
|
"Dashboard.Proxy.Requests",
|
||||||
|
successes.ToString("N0", culture),
|
||||||
|
attempts.ToString("N0", culture),
|
||||||
|
((double)successes / attempts).ToString("P0", culture)
|
||||||
|
);
|
||||||
|
|
||||||
|
this.RaisePropertyChanged(nameof(HasProxies));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
/// <remarks>The pool is a singleton and would otherwise keep this page alive.</remarks>
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
// The container disposes every page twice — once per registration; see GalleryViewModel.
|
||||||
|
if (Interlocked.Exchange(ref _disposed, 1) == 1)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
_pool.Changed -= OnPoolChanged;
|
||||||
|
_poolChanged.Dispose();
|
||||||
|
|
||||||
|
foreach (var source in Sources)
|
||||||
|
{
|
||||||
|
source.Dispose();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void OnLanguageChanged()
|
||||||
|
{
|
||||||
|
base.OnLanguageChanged();
|
||||||
|
|
||||||
|
// The latency and request lines were formatted in the previous language.
|
||||||
|
RefreshProxyStats();
|
||||||
|
}
|
||||||
|
|
||||||
|
private void OnPoolChanged(object? sender, EventArgs e) => _poolChanged.OnNext(RxVoid.Default);
|
||||||
|
|
||||||
private void Navigate<TPage>()
|
private void Navigate<TPage>()
|
||||||
where TPage : PageViewModel => _services.GetRequiredService<INavigationService>().NavigateTo<TPage>();
|
where TPage : PageViewModel => _services.GetRequiredService<INavigationService>().NavigateTo<TPage>();
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -317,10 +317,24 @@ public partial class GalleryViewModel : PageViewModel, IDisposable
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
|
/// <remarks>
|
||||||
|
/// Has to survive being called twice. Every page is registered under its own type <b>and</b>
|
||||||
|
/// under <c>PageViewModel</c>, and the container tracks the instance for both registrations, so
|
||||||
|
/// it disposes each page twice on shutdown. Cancelling an already-disposed source threw
|
||||||
|
/// <see cref="ObjectDisposedException"/> out of <c>ServiceProvider.Dispose</c> and took the
|
||||||
|
/// process down on every exit.
|
||||||
|
/// </remarks>
|
||||||
public void Dispose()
|
public void Dispose()
|
||||||
{
|
{
|
||||||
_loading?.Cancel();
|
// Detached first: a load starting after this must not resurrect the field.
|
||||||
_loading?.Dispose();
|
var loading = Interlocked.Exchange(ref _loading, null);
|
||||||
|
|
||||||
|
if (loading is not null)
|
||||||
|
{
|
||||||
|
loading.Cancel();
|
||||||
|
loading.Dispose();
|
||||||
|
}
|
||||||
|
|
||||||
GC.SuppressFinalize(this);
|
GC.SuppressFinalize(this);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
using AvParser.Core.Collecting;
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
using AvParser.UI.Localization;
|
using AvParser.UI.Localization;
|
||||||
using ReactiveUI;
|
using ReactiveUI;
|
||||||
|
|
||||||
@@ -7,12 +8,15 @@ namespace AvParser.UI.ViewModels;
|
|||||||
/// <summary>A media source paired with its translated name and description.</summary>
|
/// <summary>A media source paired with its translated name and description.</summary>
|
||||||
/// <remarks>
|
/// <remarks>
|
||||||
/// The domain knows nothing about languages, so <see cref="IMediaSource"/> carries English text.
|
/// The domain knows nothing about languages, so <see cref="IMediaSource"/> carries English text.
|
||||||
/// This looks the id up as <c>Source.{id}.Name</c> and falls back to what the source itself says,
|
/// This looks the id up as <c>Source.{id}.Name</c> and falls back to what the source itself says —
|
||||||
/// which keeps the "add a source = one registration line" promise intact: a new source works
|
/// which is what user-added sources always do, since their name is data the user typed, not a resx
|
||||||
/// untranslated rather than rendering a missing-key marker.
|
/// key. Disposable because the catalog rebuilds these on every edit, and each one holds a
|
||||||
|
/// language-changed subscription that would otherwise leak.
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
public sealed class MediaSourceViewModel : ReactiveObject
|
public sealed class MediaSourceViewModel : ReactiveObject, IDisposable
|
||||||
{
|
{
|
||||||
|
private bool _isSelected;
|
||||||
|
|
||||||
/// <summary>Wraps a source.</summary>
|
/// <summary>Wraps a source.</summary>
|
||||||
public MediaSourceViewModel(IMediaSource source)
|
public MediaSourceViewModel(IMediaSource source)
|
||||||
{
|
{
|
||||||
@@ -20,9 +24,24 @@ public sealed class MediaSourceViewModel : ReactiveObject
|
|||||||
Localizer.Instance.LanguageChanged += OnLanguageChanged;
|
Localizer.Instance.LanguageChanged += OnLanguageChanged;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>Whether the next run includes this source.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Ticking is what a run acts on; the list's highlight only says which source the editor,
|
||||||
|
/// the delete button and the purge button are aimed at. Keeping the two apart is what lets a
|
||||||
|
/// run cover five sources while the user edits a sixth.
|
||||||
|
/// </remarks>
|
||||||
|
public bool IsSelected
|
||||||
|
{
|
||||||
|
get => _isSelected;
|
||||||
|
set => this.RaiseAndSetIfChanged(ref _isSelected, value);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>The source itself.</summary>
|
/// <summary>The source itself.</summary>
|
||||||
public IMediaSource Source { get; }
|
public IMediaSource Source { get; }
|
||||||
|
|
||||||
|
/// <summary>The config behind the source, when it is a pattern source (they all are today).</summary>
|
||||||
|
public PatternSourceConfig? Config => (Source as PatternMediaSource)?.Config;
|
||||||
|
|
||||||
/// <summary>Stable identifier.</summary>
|
/// <summary>Stable identifier.</summary>
|
||||||
public string Id => Source.Id;
|
public string Id => Source.Id;
|
||||||
|
|
||||||
@@ -32,15 +51,27 @@ public sealed class MediaSourceViewModel : ReactiveObject
|
|||||||
/// <summary>Translated description, or the source's own when untranslated.</summary>
|
/// <summary>Translated description, or the source's own when untranslated.</summary>
|
||||||
public string Description => Localizer.Instance.GetOrDefault($"Source.{Id}.Description", Source.Description);
|
public string Description => Localizer.Instance.GetOrDefault($"Source.{Id}.Description", Source.Description);
|
||||||
|
|
||||||
/// <summary>Whether this source needs a working proxy before it may run.</summary>
|
/// <summary>Whether this source goes to the network at all.</summary>
|
||||||
public bool RequiresNetwork => Source.RequiresNetwork;
|
public bool RequiresNetwork => Source.RequiresNetwork;
|
||||||
|
|
||||||
/// <summary>Whether this source reads a listing endpoint rather than pasted text.</summary>
|
/// <summary>
|
||||||
public bool UsesEndpoint => Source.RequiresNetwork;
|
/// Whether this source may run from the user's own address when no proxy is live.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Anything that is not a configured pattern source answers <see langword="false"/>: the safe
|
||||||
|
/// end, since a source with nowhere to say otherwise has not said otherwise.
|
||||||
|
/// </remarks>
|
||||||
|
public bool AllowsDirectConnection => Config?.AllowDirectConnection ?? false;
|
||||||
|
|
||||||
|
/// <summary>Whether a run including this source is gated on a live proxy.</summary>
|
||||||
|
public bool NeedsProxy => RequiresNetwork && !AllowsDirectConnection;
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public override string ToString() => Name;
|
public override string ToString() => Name;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public void Dispose() => Localizer.Instance.LanguageChanged -= OnLanguageChanged;
|
||||||
|
|
||||||
private void OnLanguageChanged(object? sender, EventArgs e)
|
private void OnLanguageChanged(object? sender, EventArgs e)
|
||||||
{
|
{
|
||||||
this.RaisePropertyChanged(nameof(Name));
|
this.RaisePropertyChanged(nameof(Name));
|
||||||
|
|||||||
@@ -44,6 +44,8 @@ public partial class ProxiesViewModel : PageViewModel, IDisposable
|
|||||||
private readonly Dictionary<string, ProxyRowViewModel> _rows = new(StringComparer.Ordinal);
|
private readonly Dictionary<string, ProxyRowViewModel> _rows = new(StringComparer.Ordinal);
|
||||||
private readonly Signal<RxVoid> _poolChanged = new();
|
private readonly Signal<RxVoid> _poolChanged = new();
|
||||||
|
|
||||||
|
private int _disposed;
|
||||||
|
|
||||||
/// <summary>Free-text filter over address and country.</summary>
|
/// <summary>Free-text filter over address and country.</summary>
|
||||||
[Reactive]
|
[Reactive]
|
||||||
public partial string SearchText { get; set; }
|
public partial string SearchText { get; set; }
|
||||||
@@ -328,6 +330,12 @@ public partial class ProxiesViewModel : PageViewModel, IDisposable
|
|||||||
/// </remarks>
|
/// </remarks>
|
||||||
public void Dispose()
|
public void Dispose()
|
||||||
{
|
{
|
||||||
|
// The container disposes every page twice — once per registration; see GalleryViewModel.
|
||||||
|
if (Interlocked.Exchange(ref _disposed, 1) == 1)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
_pool.Changed -= OnPoolChanged;
|
_pool.Changed -= OnPoolChanged;
|
||||||
_poolChanged.Dispose();
|
_poolChanged.Dispose();
|
||||||
GC.SuppressFinalize(this);
|
GC.SuppressFinalize(this);
|
||||||
|
|||||||
@@ -63,10 +63,6 @@ public partial class SettingsViewModel : PageViewModel
|
|||||||
[Reactive]
|
[Reactive]
|
||||||
public partial int ProxyMinimumLive { get; set; }
|
public partial int ProxyMinimumLive { get; set; }
|
||||||
|
|
||||||
/// <summary>Whether network parsers may run with no proxy available.</summary>
|
|
||||||
[Reactive]
|
|
||||||
public partial bool AllowDirectConnection { get; set; }
|
|
||||||
|
|
||||||
/// <summary>How many downloads may be in flight at once.</summary>
|
/// <summary>How many downloads may be in flight at once.</summary>
|
||||||
[Reactive]
|
[Reactive]
|
||||||
public partial int MaxConcurrentDownloads { get; set; }
|
public partial int MaxConcurrentDownloads { get; set; }
|
||||||
@@ -136,7 +132,6 @@ public partial class SettingsViewModel : PageViewModel
|
|||||||
ProxyProbeTimeoutSeconds = current.ProxyProbeTimeoutSeconds;
|
ProxyProbeTimeoutSeconds = current.ProxyProbeTimeoutSeconds;
|
||||||
ProxyProbeConcurrency = current.ProxyProbeConcurrency;
|
ProxyProbeConcurrency = current.ProxyProbeConcurrency;
|
||||||
ProxyMinimumLive = current.ProxyMinimumLive;
|
ProxyMinimumLive = current.ProxyMinimumLive;
|
||||||
AllowDirectConnection = current.AllowDirectConnection;
|
|
||||||
MaxConcurrentDownloads = current.MaxConcurrentDownloads;
|
MaxConcurrentDownloads = current.MaxConcurrentDownloads;
|
||||||
MaxConcurrentPerHost = current.MaxConcurrentPerHost;
|
MaxConcurrentPerHost = current.MaxConcurrentPerHost;
|
||||||
HostDelayMs = current.HostDelayMs;
|
HostDelayMs = current.HostDelayMs;
|
||||||
@@ -168,8 +163,7 @@ public partial class SettingsViewModel : PageViewModel
|
|||||||
x => x.ProxyProbeTimeoutSeconds,
|
x => x.ProxyProbeTimeoutSeconds,
|
||||||
x => x.ProxyProbeConcurrency,
|
x => x.ProxyProbeConcurrency,
|
||||||
x => x.ProxyMinimumLive,
|
x => x.ProxyMinimumLive,
|
||||||
x => x.AllowDirectConnection,
|
(_, _, _, _, _, _, _) => RxVoid.Default
|
||||||
(_, _, _, _, _, _, _, _) => RxVoid.Default
|
|
||||||
)
|
)
|
||||||
.Throttle(TimeSpan.FromMilliseconds(200), scheduler)
|
.Throttle(TimeSpan.FromMilliseconds(200), scheduler)
|
||||||
.ObserveOn(scheduler)
|
.ObserveOn(scheduler)
|
||||||
@@ -345,7 +339,6 @@ public partial class SettingsViewModel : PageViewModel
|
|||||||
ProxyProbeTimeoutSeconds = ProxyProbeTimeoutSeconds,
|
ProxyProbeTimeoutSeconds = ProxyProbeTimeoutSeconds,
|
||||||
ProxyProbeConcurrency = ProxyProbeConcurrency,
|
ProxyProbeConcurrency = ProxyProbeConcurrency,
|
||||||
ProxyMinimumLive = ProxyMinimumLive,
|
ProxyMinimumLive = ProxyMinimumLive,
|
||||||
AllowDirectConnection = AllowDirectConnection,
|
|
||||||
MaxConcurrentDownloads = MaxConcurrentDownloads,
|
MaxConcurrentDownloads = MaxConcurrentDownloads,
|
||||||
MaxConcurrentPerHost = MaxConcurrentPerHost,
|
MaxConcurrentPerHost = MaxConcurrentPerHost,
|
||||||
HostDelayMs = HostDelayMs,
|
HostDelayMs = HostDelayMs,
|
||||||
|
|||||||
@@ -7,25 +7,83 @@
|
|||||||
x:Class="AvParser.UI.Views.CollectView"
|
x:Class="AvParser.UI.Views.CollectView"
|
||||||
x:DataType="vm:CollectViewModel"
|
x:DataType="vm:CollectViewModel"
|
||||||
>
|
>
|
||||||
<Grid RowDefinitions="Auto,Auto,Auto,*">
|
<Grid RowDefinitions="Auto,Auto,Auto,Auto,*">
|
||||||
<!-- ===== Toolbar ===== -->
|
<!-- ===== Toolbar ===== -->
|
||||||
<Border Grid.Row="0" Classes="card" Margin="0,0,0,12">
|
<Border Grid.Row="0" Classes="card" Margin="0,0,0,12">
|
||||||
<StackPanel Spacing="12">
|
<StackPanel Spacing="12">
|
||||||
<WrapPanel Orientation="Horizontal">
|
<WrapPanel Orientation="Horizontal">
|
||||||
<StackPanel Spacing="4" Margin="0,0,16,8" MinWidth="240">
|
<StackPanel Spacing="4" Margin="0,0,16,8" MinWidth="280">
|
||||||
<TextBlock Classes="caption" Text="{l:Loc Collect.Source}" />
|
<TextBlock Classes="caption" Text="{l:Loc Collect.Sources}" />
|
||||||
<ComboBox
|
<StackPanel Orientation="Horizontal" Spacing="8">
|
||||||
|
<Border
|
||||||
|
BorderThickness="1"
|
||||||
|
BorderBrush="{DynamicResource AppBorderBrush}"
|
||||||
|
CornerRadius="4"
|
||||||
|
MinWidth="240"
|
||||||
|
MaxHeight="120"
|
||||||
|
>
|
||||||
|
<ListBox
|
||||||
|
x:Name="SourceList"
|
||||||
ItemsSource="{Binding Sources}"
|
ItemsSource="{Binding Sources}"
|
||||||
SelectedItem="{Binding SelectedSource}"
|
SelectedItem="{Binding SelectedSource}"
|
||||||
IsEnabled="{Binding IsBusy, Converter={x:Static conv:AppConverters.Not}}"
|
IsEnabled="{Binding IsBusy, Converter={x:Static conv:AppConverters.Not}}"
|
||||||
HorizontalAlignment="Stretch"
|
Background="Transparent"
|
||||||
|
BorderThickness="0"
|
||||||
|
SelectionMode="Single"
|
||||||
>
|
>
|
||||||
<ComboBox.ItemTemplate>
|
<ListBox.ItemTemplate>
|
||||||
<DataTemplate x:DataType="vm:MediaSourceViewModel">
|
<DataTemplate x:DataType="vm:MediaSourceViewModel">
|
||||||
<TextBlock Text="{Binding Name}" />
|
<CheckBox
|
||||||
|
IsChecked="{Binding IsSelected}"
|
||||||
|
Content="{Binding Name}"
|
||||||
|
ToolTip.Tip="{Binding Description}"
|
||||||
|
/>
|
||||||
</DataTemplate>
|
</DataTemplate>
|
||||||
</ComboBox.ItemTemplate>
|
</ListBox.ItemTemplate>
|
||||||
</ComboBox>
|
</ListBox>
|
||||||
|
</Border>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4" VerticalAlignment="Top">
|
||||||
|
<Button Classes="icon" Command="{Binding AddSourceCommand}" ToolTip.Tip="{l:Loc Collect.AddSource}">
|
||||||
|
<PathIcon Classes="glyph" Data="{DynamicResource IconPlus}" />
|
||||||
|
</Button>
|
||||||
|
<Button Classes="icon" Command="{Binding EditSourceCommand}" ToolTip.Tip="{l:Loc Collect.EditSource}">
|
||||||
|
<PathIcon Classes="glyph" Data="{DynamicResource IconEdit}" />
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
Classes="icon"
|
||||||
|
Command="{Binding RemoveSourceCommand}"
|
||||||
|
ToolTip.Tip="{l:Loc Collect.RemoveSource}"
|
||||||
|
>
|
||||||
|
<PathIcon Classes="glyph" Data="{DynamicResource IconTrash}" />
|
||||||
|
</Button>
|
||||||
|
</StackPanel>
|
||||||
|
</StackPanel>
|
||||||
|
<TextBlock Classes="muted caption" Text="{l:Loc Collect.SourcesHint}" TextWrapping="Wrap" MaxWidth="280" />
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4" Margin="0,0,16,8" MinWidth="140">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.Target}" />
|
||||||
|
<NumericUpDown
|
||||||
|
Value="{Binding TargetCount}"
|
||||||
|
Minimum="0"
|
||||||
|
Maximum="1000000"
|
||||||
|
Increment="10"
|
||||||
|
FormatString="0"
|
||||||
|
/>
|
||||||
|
<TextBlock Classes="muted caption" Text="{l:Loc Collect.ZeroIsUnlimited}" TextWrapping="Wrap" />
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4" Margin="0,0,16,8" MinWidth="140">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.AttemptBudget}" />
|
||||||
|
<NumericUpDown
|
||||||
|
Value="{Binding AttemptBudget}"
|
||||||
|
Minimum="0"
|
||||||
|
Maximum="100000000"
|
||||||
|
Increment="100"
|
||||||
|
FormatString="0"
|
||||||
|
/>
|
||||||
|
<TextBlock Classes="muted caption" Text="{l:Loc Collect.AttemptBudgetHint}" TextWrapping="Wrap" />
|
||||||
</StackPanel>
|
</StackPanel>
|
||||||
|
|
||||||
<StackPanel Spacing="4" Margin="0,0,0,8">
|
<StackPanel Spacing="4" Margin="0,0,0,8">
|
||||||
@@ -52,7 +110,12 @@
|
|||||||
</StackPanel>
|
</StackPanel>
|
||||||
</WrapPanel>
|
</WrapPanel>
|
||||||
|
|
||||||
<TextBlock Classes="muted" Text="{Binding SelectedSource.Description}" TextWrapping="Wrap" />
|
<TextBlock
|
||||||
|
Classes="muted"
|
||||||
|
Text="{Binding SelectedSource.Description}"
|
||||||
|
TextWrapping="Wrap"
|
||||||
|
IsVisible="{Binding SelectedSource, Converter={x:Static ObjectConverters.IsNotNull}}"
|
||||||
|
/>
|
||||||
|
|
||||||
<CheckBox
|
<CheckBox
|
||||||
IsChecked="{Binding ForceRefetch}"
|
IsChecked="{Binding ForceRefetch}"
|
||||||
@@ -84,10 +147,85 @@
|
|||||||
</StackPanel>
|
</StackPanel>
|
||||||
</Border>
|
</Border>
|
||||||
|
|
||||||
|
<!-- ===== Source editor ===== -->
|
||||||
|
<Border x:Name="SourceEditor" Grid.Row="1" Classes="card" Margin="0,0,0,12" IsVisible="{Binding IsEditorOpen}">
|
||||||
|
<StackPanel Spacing="12">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.SourceEditor}" />
|
||||||
|
|
||||||
|
<StackPanel Spacing="4">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.SourceName}" />
|
||||||
|
<TextBox Text="{Binding EditorName}" PlaceholderText="{l:Loc Collect.SourceNamePlaceholder}" />
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.SourceBaseUrl}" />
|
||||||
|
<TextBox Text="{Binding EditorBaseUrl}" PlaceholderText="{l:Loc Collect.SourceBaseUrlPlaceholder}" />
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<Grid ColumnDefinitions="*,12,*,12,*">
|
||||||
|
<StackPanel Grid.Column="0" Spacing="4">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.SourceMinLength}" />
|
||||||
|
<NumericUpDown Value="{Binding EditorMinLength}" Minimum="1" Maximum="64" Increment="1" FormatString="0" />
|
||||||
|
</StackPanel>
|
||||||
|
<StackPanel Grid.Column="2" Spacing="4">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.SourceMaxLength}" />
|
||||||
|
<NumericUpDown Value="{Binding EditorMaxLength}" Minimum="1" Maximum="64" Increment="1" FormatString="0" />
|
||||||
|
</StackPanel>
|
||||||
|
<StackPanel Grid.Column="4" Spacing="4">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.SourceExtension}" />
|
||||||
|
<TextBox Text="{Binding EditorExtension}" PlaceholderText=".jpg" />
|
||||||
|
</StackPanel>
|
||||||
|
</Grid>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.SourceAlphabet}" />
|
||||||
|
<ComboBox
|
||||||
|
ItemsSource="{Binding AlphabetOptions}"
|
||||||
|
SelectedItem="{Binding EditorAlphabet}"
|
||||||
|
HorizontalAlignment="Stretch"
|
||||||
|
>
|
||||||
|
<ComboBox.ItemTemplate>
|
||||||
|
<DataTemplate x:DataType="l:LocalizedOption">
|
||||||
|
<TextBlock Text="{Binding Label}" />
|
||||||
|
</DataTemplate>
|
||||||
|
</ComboBox.ItemTemplate>
|
||||||
|
</ComboBox>
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4" IsVisible="{Binding ShowCustomAlphabet}">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Collect.SourceCustomAlphabet}" />
|
||||||
|
<TextBox
|
||||||
|
Text="{Binding EditorCustomAlphabet}"
|
||||||
|
PlaceholderText="{l:Loc Collect.SourceCustomAlphabetPlaceholder}"
|
||||||
|
/>
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<CheckBox
|
||||||
|
x:Name="EditorAllowDirect"
|
||||||
|
IsChecked="{Binding EditorAllowDirect}"
|
||||||
|
Content="{l:Loc Collect.SourceAllowDirect}"
|
||||||
|
ToolTip.Tip="{l:Loc Collect.SourceAllowDirectHint}"
|
||||||
|
/>
|
||||||
|
|
||||||
|
<TextBlock
|
||||||
|
Classes="muted"
|
||||||
|
Foreground="{DynamicResource AppDangerBrush}"
|
||||||
|
Text="{Binding EditorError}"
|
||||||
|
TextWrapping="Wrap"
|
||||||
|
IsVisible="{Binding EditorError, Converter={x:Static ObjectConverters.IsNotNull}}"
|
||||||
|
/>
|
||||||
|
|
||||||
|
<StackPanel Orientation="Horizontal" Spacing="8" HorizontalAlignment="Right">
|
||||||
|
<Button Command="{Binding CancelEditCommand}" Content="{l:Loc Collect.SourceCancel}" />
|
||||||
|
<Button Classes="primary" Command="{Binding SaveSourceCommand}" Content="{l:Loc Collect.SourceSave}" />
|
||||||
|
</StackPanel>
|
||||||
|
</StackPanel>
|
||||||
|
</Border>
|
||||||
|
|
||||||
<!-- ===== Blocked without a proxy ===== -->
|
<!-- ===== Blocked without a proxy ===== -->
|
||||||
<Border
|
<Border
|
||||||
x:Name="ProxyGateBanner"
|
x:Name="ProxyGateBanner"
|
||||||
Grid.Row="1"
|
Grid.Row="2"
|
||||||
Classes="card"
|
Classes="card"
|
||||||
Margin="0,0,0,12"
|
Margin="0,0,0,12"
|
||||||
Background="{DynamicResource AppDangerSoftBrush}"
|
Background="{DynamicResource AppDangerSoftBrush}"
|
||||||
@@ -121,7 +259,7 @@
|
|||||||
</Border>
|
</Border>
|
||||||
|
|
||||||
<!-- ===== Progress and status ===== -->
|
<!-- ===== Progress and status ===== -->
|
||||||
<StackPanel Grid.Row="2" Spacing="8" Margin="0,0,0,12">
|
<StackPanel Grid.Row="3" Spacing="8" Margin="0,0,0,12">
|
||||||
<ProgressBar
|
<ProgressBar
|
||||||
Minimum="0"
|
Minimum="0"
|
||||||
Maximum="1"
|
Maximum="1"
|
||||||
@@ -138,63 +276,28 @@
|
|||||||
/>
|
/>
|
||||||
</StackPanel>
|
</StackPanel>
|
||||||
|
|
||||||
<!-- ===== Input and results ===== -->
|
<!-- ===== Results ===== -->
|
||||||
<Grid Grid.Row="3" ColumnDefinitions="*,8,1.4*">
|
<Grid Grid.Row="4" RowDefinitions="*,Auto">
|
||||||
<Border Grid.Column="0" Classes="card" Padding="0">
|
<!-- Empty state: no sources yet -->
|
||||||
<DockPanel LastChildFill="True">
|
|
||||||
<!-- IsVisible belongs on the Border, not the caption inside it: hiding only the text
|
|
||||||
leaves the padding and the divider behind as an empty bar. -->
|
|
||||||
<Border
|
<Border
|
||||||
DockPanel.Dock="Top"
|
x:Name="EmptyState"
|
||||||
Padding="16,12"
|
Grid.Row="0"
|
||||||
BorderThickness="0,0,0,1"
|
Classes="card"
|
||||||
BorderBrush="{DynamicResource AppBorderBrush}"
|
IsVisible="{Binding HasSources, Converter={x:Static conv:AppConverters.Not}}"
|
||||||
IsVisible="{Binding SelectedSource.UsesEndpoint}"
|
|
||||||
>
|
>
|
||||||
<TextBlock Classes="caption" Text="{l:Loc Collect.Endpoint}" />
|
<StackPanel Spacing="12" HorizontalAlignment="Center" VerticalAlignment="Center">
|
||||||
</Border>
|
<PathIcon Classes="glyph" Data="{DynamicResource IconDownload}" Opacity="0.5" Width="32" Height="32" />
|
||||||
|
<TextBlock Classes="muted" Text="{l:Loc Collect.NoSources}" TextWrapping="Wrap" TextAlignment="Center" />
|
||||||
<!-- Endpoint sources take one address; the rest take a pasted list. -->
|
<Button
|
||||||
<TextBox
|
Classes="primary"
|
||||||
DockPanel.Dock="Top"
|
HorizontalAlignment="Center"
|
||||||
Margin="16,12"
|
Command="{Binding AddSourceCommand}"
|
||||||
Text="{Binding EndpointText}"
|
Content="{l:Loc Collect.AddSource}"
|
||||||
PlaceholderText="{l:Loc Collect.EndpointPlaceholder}"
|
|
||||||
IsVisible="{Binding SelectedSource.UsesEndpoint}"
|
|
||||||
/>
|
/>
|
||||||
|
</StackPanel>
|
||||||
<Border
|
|
||||||
DockPanel.Dock="Top"
|
|
||||||
Padding="16,12"
|
|
||||||
BorderThickness="0,0,0,1"
|
|
||||||
BorderBrush="{DynamicResource AppBorderBrush}"
|
|
||||||
IsVisible="{Binding SelectedSource.UsesEndpoint, Converter={x:Static conv:AppConverters.Not}}"
|
|
||||||
>
|
|
||||||
<TextBlock Classes="caption" Text="{l:Loc Collect.Input}" />
|
|
||||||
</Border>
|
</Border>
|
||||||
|
|
||||||
<TextBox
|
<Border Grid.Row="0" Classes="card" Padding="0" IsVisible="{Binding HasSources}">
|
||||||
Text="{Binding InputText}"
|
|
||||||
AcceptsReturn="True"
|
|
||||||
VerticalContentAlignment="Top"
|
|
||||||
AcceptsTab="False"
|
|
||||||
TextWrapping="NoWrap"
|
|
||||||
PlaceholderText="{l:Loc Collect.InputPlaceholder}"
|
|
||||||
BorderThickness="0"
|
|
||||||
Background="Transparent"
|
|
||||||
FontFamily="Cascadia Code,Consolas,Menlo,DejaVu Sans Mono,monospace"
|
|
||||||
FontSize="{DynamicResource FontSizeBody}"
|
|
||||||
IsVisible="{Binding SelectedSource.UsesEndpoint, Converter={x:Static conv:AppConverters.Not}}"
|
|
||||||
ScrollViewer.HorizontalScrollBarVisibility="Auto"
|
|
||||||
ScrollViewer.VerticalScrollBarVisibility="Auto"
|
|
||||||
/>
|
|
||||||
</DockPanel>
|
|
||||||
</Border>
|
|
||||||
|
|
||||||
<GridSplitter Grid.Column="1" ResizeDirection="Columns" Background="Transparent" />
|
|
||||||
|
|
||||||
<Grid Grid.Column="2" RowDefinitions="*,Auto">
|
|
||||||
<Border Grid.Row="0" Classes="card" Padding="0">
|
|
||||||
<DockPanel LastChildFill="True">
|
<DockPanel LastChildFill="True">
|
||||||
<Border
|
<Border
|
||||||
DockPanel.Dock="Top"
|
DockPanel.Dock="Top"
|
||||||
@@ -237,7 +340,7 @@
|
|||||||
Text="{Binding Address}"
|
Text="{Binding Address}"
|
||||||
VerticalAlignment="Center"
|
VerticalAlignment="Center"
|
||||||
TextTrimming="CharacterEllipsis"
|
TextTrimming="CharacterEllipsis"
|
||||||
MaxWidth="420"
|
MaxWidth="520"
|
||||||
/>
|
/>
|
||||||
|
|
||||||
<Border Classes="chip accent" VerticalAlignment="Center">
|
<Border Classes="chip accent" VerticalAlignment="Center">
|
||||||
@@ -253,13 +356,15 @@
|
|||||||
</DockPanel>
|
</DockPanel>
|
||||||
</Border>
|
</Border>
|
||||||
|
|
||||||
|
<!-- ===== Live log ===== -->
|
||||||
<Border
|
<Border
|
||||||
|
x:Name="LogPanel"
|
||||||
Grid.Row="1"
|
Grid.Row="1"
|
||||||
Classes="card"
|
Classes="card"
|
||||||
Margin="0,8,0,0"
|
Margin="0,8,0,0"
|
||||||
Padding="0"
|
Padding="0"
|
||||||
MaxHeight="180"
|
MaxHeight="220"
|
||||||
IsVisible="{Binding Errors.Count, Converter={x:Static conv:AppConverters.IsPositive}}"
|
IsVisible="{Binding Log.Count, Converter={x:Static conv:AppConverters.IsPositive}}"
|
||||||
>
|
>
|
||||||
<DockPanel LastChildFill="True">
|
<DockPanel LastChildFill="True">
|
||||||
<Border
|
<Border
|
||||||
@@ -269,27 +374,65 @@
|
|||||||
BorderBrush="{DynamicResource AppBorderBrush}"
|
BorderBrush="{DynamicResource AppBorderBrush}"
|
||||||
>
|
>
|
||||||
<StackPanel Orientation="Horizontal" Spacing="8">
|
<StackPanel Orientation="Horizontal" Spacing="8">
|
||||||
<PathIcon
|
<PathIcon Classes="glyph" Data="{DynamicResource IconDocument}" VerticalAlignment="Center" />
|
||||||
Classes="glyph"
|
<TextBlock Classes="caption" Text="{l:Loc Collect.Log}" VerticalAlignment="Center" />
|
||||||
Data="{DynamicResource IconAlert}"
|
<Border Classes="chip">
|
||||||
Foreground="{DynamicResource AppDangerBrush}"
|
<TextBlock Classes="mono caption" Text="{Binding Log.Count}" />
|
||||||
VerticalAlignment="Center"
|
</Border>
|
||||||
/>
|
<Border
|
||||||
<TextBlock Classes="caption" Text="{l:Loc Collect.Errors}" VerticalAlignment="Center" />
|
Classes="chip danger"
|
||||||
<Border Classes="chip danger">
|
IsVisible="{Binding ErrorCount, Converter={x:Static conv:AppConverters.IsPositive}}"
|
||||||
<TextBlock Classes="mono caption" Text="{Binding Errors.Count}" />
|
>
|
||||||
|
<TextBlock Classes="mono caption" Text="{Binding ErrorCount}" />
|
||||||
</Border>
|
</Border>
|
||||||
</StackPanel>
|
</StackPanel>
|
||||||
</Border>
|
</Border>
|
||||||
|
|
||||||
<ListBox ItemsSource="{Binding Errors}" Background="Transparent" BorderThickness="0">
|
<ListBox
|
||||||
|
x:Name="CollectLog"
|
||||||
|
ItemsSource="{Binding Log}"
|
||||||
|
Background="Transparent"
|
||||||
|
BorderThickness="0"
|
||||||
|
SelectionMode="Multiple"
|
||||||
|
KeyDown="OnLogKeyDown"
|
||||||
|
>
|
||||||
|
<ListBox.ContextMenu>
|
||||||
|
<ContextMenu>
|
||||||
|
<MenuItem
|
||||||
|
x:Name="CopyLogSelection"
|
||||||
|
Header="{l:Loc Collect.Log.Copy}"
|
||||||
|
InputGesture="Ctrl+C"
|
||||||
|
Click="OnCopyLogSelection"
|
||||||
|
/>
|
||||||
|
<MenuItem x:Name="CopyLogAll" Header="{l:Loc Collect.Log.CopyAll}" Click="OnCopyLogAll" />
|
||||||
|
</ContextMenu>
|
||||||
|
</ListBox.ContextMenu>
|
||||||
<ListBox.ItemTemplate>
|
<ListBox.ItemTemplate>
|
||||||
<DataTemplate x:DataType="vm:CollectErrorViewModel">
|
<DataTemplate x:DataType="vm:CollectLogEntryViewModel">
|
||||||
<StackPanel Orientation="Horizontal" Spacing="10">
|
<StackPanel Orientation="Horizontal" Spacing="10">
|
||||||
<Border Classes="chip danger" VerticalAlignment="Center" MinWidth="44">
|
<TextBlock Classes="mono caption muted" Text="{Binding TimeText}" VerticalAlignment="Center" />
|
||||||
<TextBlock Classes="mono caption" Text="{Binding Index}" HorizontalAlignment="Center" />
|
|
||||||
|
<Border
|
||||||
|
Classes="chip"
|
||||||
|
Classes.danger="{Binding IsError}"
|
||||||
|
Classes.ok="{Binding IsSuccess}"
|
||||||
|
VerticalAlignment="Center"
|
||||||
|
MinWidth="24"
|
||||||
|
>
|
||||||
|
<TextBlock Classes="mono caption" Text="{Binding LevelText}" HorizontalAlignment="Center" />
|
||||||
</Border>
|
</Border>
|
||||||
<TextBlock Classes="muted" Text="{Binding Text}" VerticalAlignment="Center" TextWrapping="Wrap" />
|
|
||||||
|
<Border Classes="chip accent" VerticalAlignment="Center" IsVisible="{Binding HasSource}">
|
||||||
|
<TextBlock Classes="caption" Text="{Binding Source}" />
|
||||||
|
</Border>
|
||||||
|
|
||||||
|
<TextBlock
|
||||||
|
Classes="mono caption"
|
||||||
|
Text="{Binding Text}"
|
||||||
|
ToolTip.Tip="{Binding Text}"
|
||||||
|
VerticalAlignment="Center"
|
||||||
|
TextTrimming="CharacterEllipsis"
|
||||||
|
/>
|
||||||
</StackPanel>
|
</StackPanel>
|
||||||
</DataTemplate>
|
</DataTemplate>
|
||||||
</ListBox.ItemTemplate>
|
</ListBox.ItemTemplate>
|
||||||
@@ -298,5 +441,4 @@
|
|||||||
</Border>
|
</Border>
|
||||||
</Grid>
|
</Grid>
|
||||||
</Grid>
|
</Grid>
|
||||||
</Grid>
|
|
||||||
</UserControl>
|
</UserControl>
|
||||||
|
|||||||
@@ -1,13 +1,132 @@
|
|||||||
|
using System.Collections.Specialized;
|
||||||
using Avalonia.Controls;
|
using Avalonia.Controls;
|
||||||
|
using Avalonia.Input;
|
||||||
|
using Avalonia.Input.Platform;
|
||||||
using Avalonia.Markup.Xaml;
|
using Avalonia.Markup.Xaml;
|
||||||
|
using AvParser.UI.ViewModels;
|
||||||
|
|
||||||
namespace AvParser.UI.Views;
|
namespace AvParser.UI.Views;
|
||||||
|
|
||||||
/// <summary>Source picker, toolbar and streamed collection results.</summary>
|
/// <summary>Source picker, toolbar and streamed collection results.</summary>
|
||||||
public partial class CollectView : UserControl
|
public partial class CollectView : UserControl
|
||||||
{
|
{
|
||||||
|
private INotifyCollectionChanged? _log;
|
||||||
|
|
||||||
/// <summary>Creates the view.</summary>
|
/// <summary>Creates the view.</summary>
|
||||||
public CollectView() => InitializeComponent();
|
public CollectView()
|
||||||
|
{
|
||||||
|
InitializeComponent();
|
||||||
|
DetachedFromVisualTree += (_, _) => Unsubscribe();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
/// <remarks>
|
||||||
|
/// A log that does not follow itself is a log the user has to drag a scroll bar through while it
|
||||||
|
/// is still being written, so the newest line is kept in view. This lives in the view rather than
|
||||||
|
/// the view model on purpose: it is scrolling, not state.
|
||||||
|
/// </remarks>
|
||||||
|
protected override void OnDataContextChanged(EventArgs e)
|
||||||
|
{
|
||||||
|
base.OnDataContextChanged(e);
|
||||||
|
|
||||||
|
Unsubscribe();
|
||||||
|
|
||||||
|
if (DataContext is CollectViewModel page)
|
||||||
|
{
|
||||||
|
_log = page.Log;
|
||||||
|
_log.CollectionChanged += OnLogChanged;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private void InitializeComponent() => AvaloniaXamlLoader.Load(this);
|
private void InitializeComponent() => AvaloniaXamlLoader.Load(this);
|
||||||
|
|
||||||
|
private void Unsubscribe()
|
||||||
|
{
|
||||||
|
if (_log is not null)
|
||||||
|
{
|
||||||
|
_log.CollectionChanged -= OnLogChanged;
|
||||||
|
_log = null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void OnLogChanged(object? sender, NotifyCollectionChangedEventArgs e)
|
||||||
|
{
|
||||||
|
if (e.Action != NotifyCollectionChangedAction.Add || sender is not System.Collections.IList list)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var last = list.Count - 1;
|
||||||
|
if (last >= 0)
|
||||||
|
{
|
||||||
|
this.FindControl<ListBox>("CollectLog")?.ScrollIntoView(last);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void OnLogKeyDown(object? sender, KeyEventArgs e)
|
||||||
|
{
|
||||||
|
if (e.Key != Key.C || !e.KeyModifiers.HasFlag(KeyModifiers.Control))
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handled either way: falling through would let the gesture reach whatever is behind and
|
||||||
|
// copy something the user did not point at.
|
||||||
|
e.Handled = true;
|
||||||
|
CopySelectedLines();
|
||||||
|
}
|
||||||
|
|
||||||
|
private void OnCopyLogSelection(object? sender, Avalonia.Interactivity.RoutedEventArgs e) => CopySelectedLines();
|
||||||
|
|
||||||
|
private void OnCopyLogAll(object? sender, Avalonia.Interactivity.RoutedEventArgs e)
|
||||||
|
{
|
||||||
|
if (DataContext is CollectViewModel page)
|
||||||
|
{
|
||||||
|
Copy(CollectLogEntryViewModel.ToText(page.Log));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Copies the selected lines, or the whole log when nothing is selected.</summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// The fallback is deliberate: Ctrl+C with no selection is far more likely to mean "give me the
|
||||||
|
/// log" than to mean nothing at all.
|
||||||
|
/// </remarks>
|
||||||
|
private void CopySelectedLines()
|
||||||
|
{
|
||||||
|
var list = this.FindControl<ListBox>("CollectLog");
|
||||||
|
if (list is null)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var selected = list.SelectedItems?.OfType<CollectLogEntryViewModel>().ToArray() ?? [];
|
||||||
|
|
||||||
|
if (selected.Length == 0)
|
||||||
|
{
|
||||||
|
if (DataContext is CollectViewModel page)
|
||||||
|
{
|
||||||
|
Copy(CollectLogEntryViewModel.ToText(page.Log));
|
||||||
|
}
|
||||||
|
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
Copy(CollectLogEntryViewModel.ToText(selected));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <remarks>
|
||||||
|
/// <c>SetTextAsync</c> is an extension in Avalonia 12 (<c>Avalonia.Input.Platform</c>), not a
|
||||||
|
/// member of <c>IClipboard</c>; without that using it fails to compile as a missing member.
|
||||||
|
/// </remarks>
|
||||||
|
private void Copy(string text)
|
||||||
|
{
|
||||||
|
if (text.Length == 0)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Not awaited: the clipboard write is fire-and-forget from a UI gesture, and there is
|
||||||
|
// nothing useful to do if the platform refuses it.
|
||||||
|
_ = TopLevel.GetTopLevel(this)?.Clipboard?.SetTextAsync(text);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@
|
|||||||
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
|
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
|
||||||
xmlns:l="clr-namespace:AvParser.UI.Localization"
|
xmlns:l="clr-namespace:AvParser.UI.Localization"
|
||||||
xmlns:vm="clr-namespace:AvParser.UI.ViewModels"
|
xmlns:vm="clr-namespace:AvParser.UI.ViewModels"
|
||||||
|
xmlns:conv="clr-namespace:AvParser.UI.Converters"
|
||||||
x:Class="AvParser.UI.Views.DashboardView"
|
x:Class="AvParser.UI.Views.DashboardView"
|
||||||
x:DataType="vm:DashboardViewModel"
|
x:DataType="vm:DashboardViewModel"
|
||||||
>
|
>
|
||||||
@@ -37,6 +38,69 @@
|
|||||||
</ItemsControl>
|
</ItemsControl>
|
||||||
</StackPanel>
|
</StackPanel>
|
||||||
|
|
||||||
|
<!-- ===== Proxy pool ===== -->
|
||||||
|
<StackPanel Spacing="12">
|
||||||
|
<TextBlock Classes="subtitle" Text="{l:Loc Dashboard.Proxies}" />
|
||||||
|
|
||||||
|
<Border x:Name="ProxyStats" Classes="card">
|
||||||
|
<StackPanel Spacing="16">
|
||||||
|
<WrapPanel Orientation="Horizontal">
|
||||||
|
<StackPanel Spacing="4" MinWidth="140" Margin="0,0,16,8">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Dashboard.Proxy.Total}" />
|
||||||
|
<TextBlock Classes="title mono" Text="{Binding ProxyTotal}" />
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4" MinWidth="140" Margin="0,0,16,8">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Dashboard.Proxy.Live}" />
|
||||||
|
<TextBlock
|
||||||
|
Classes="title mono"
|
||||||
|
Text="{Binding ProxyLive}"
|
||||||
|
Foreground="{DynamicResource AppSuccessBrush}"
|
||||||
|
/>
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4" MinWidth="140" Margin="0,0,16,8">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Dashboard.Proxy.Quarantined}" />
|
||||||
|
<TextBlock
|
||||||
|
Classes="title mono"
|
||||||
|
Text="{Binding ProxyQuarantined}"
|
||||||
|
Foreground="{DynamicResource AppDangerBrush}"
|
||||||
|
/>
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
|
<StackPanel Spacing="4" MinWidth="140" Margin="0,0,16,8">
|
||||||
|
<TextBlock Classes="caption" Text="{l:Loc Dashboard.Proxy.Unchecked}" />
|
||||||
|
<TextBlock Classes="title mono muted" Text="{Binding ProxyUnchecked}" />
|
||||||
|
</StackPanel>
|
||||||
|
</WrapPanel>
|
||||||
|
|
||||||
|
<TextBlock
|
||||||
|
Classes="muted"
|
||||||
|
Text="{Binding ProxyLatency}"
|
||||||
|
IsVisible="{Binding ProxyLatency, Converter={x:Static ObjectConverters.IsNotNull}}"
|
||||||
|
/>
|
||||||
|
<TextBlock
|
||||||
|
Classes="muted"
|
||||||
|
Text="{Binding ProxyRequests}"
|
||||||
|
IsVisible="{Binding ProxyRequests, Converter={x:Static ObjectConverters.IsNotNull}}"
|
||||||
|
/>
|
||||||
|
<TextBlock
|
||||||
|
Classes="muted"
|
||||||
|
Text="{l:Loc Dashboard.Proxy.Empty}"
|
||||||
|
TextWrapping="Wrap"
|
||||||
|
IsVisible="{Binding HasProxies, Converter={x:Static conv:AppConverters.Not}}"
|
||||||
|
/>
|
||||||
|
|
||||||
|
<Button HorizontalAlignment="Left" Command="{Binding GoToProxiesCommand}">
|
||||||
|
<StackPanel Orientation="Horizontal" Spacing="8">
|
||||||
|
<PathIcon Classes="glyph" Data="{DynamicResource IconShield}" />
|
||||||
|
<TextBlock Text="{l:Loc Page.Proxies}" />
|
||||||
|
</StackPanel>
|
||||||
|
</Button>
|
||||||
|
</StackPanel>
|
||||||
|
</Border>
|
||||||
|
</StackPanel>
|
||||||
|
|
||||||
<StackPanel Spacing="12">
|
<StackPanel Spacing="12">
|
||||||
<TextBlock Classes="subtitle" Text="{l:Loc Dashboard.GetStarted}" />
|
<TextBlock Classes="subtitle" Text="{l:Loc Dashboard.GetStarted}" />
|
||||||
<StackPanel Orientation="Horizontal" Spacing="12">
|
<StackPanel Orientation="Horizontal" Spacing="12">
|
||||||
|
|||||||
@@ -141,10 +141,7 @@
|
|||||||
<TextBlock Classes="muted" Text="{l:Loc Settings.MinimumLiveHint}" TextWrapping="Wrap" />
|
<TextBlock Classes="muted" Text="{l:Loc Settings.MinimumLiveHint}" TextWrapping="Wrap" />
|
||||||
</StackPanel>
|
</StackPanel>
|
||||||
|
|
||||||
<StackPanel Spacing="6">
|
<TextBlock Classes="muted" Text="{l:Loc Settings.AllowDirectMoved}" TextWrapping="Wrap" />
|
||||||
<CheckBox IsChecked="{Binding AllowDirectConnection}" Content="{l:Loc Settings.AllowDirect}" />
|
|
||||||
<TextBlock Classes="muted" Text="{l:Loc Settings.AllowDirectHint}" TextWrapping="Wrap" />
|
|
||||||
</StackPanel>
|
|
||||||
</StackPanel>
|
</StackPanel>
|
||||||
</Border>
|
</Border>
|
||||||
|
|
||||||
@@ -221,7 +218,7 @@
|
|||||||
Command="{Binding BrowseMediaDirectoryCommand}"
|
Command="{Binding BrowseMediaDirectoryCommand}"
|
||||||
Content="{l:Loc Settings.MediaDirectoryBrowse}"
|
Content="{l:Loc Settings.MediaDirectoryBrowse}"
|
||||||
/>
|
/>
|
||||||
<TextBox Text="{Binding MediaRootOverride}" Watermark="{Binding MediaDirectory}" />
|
<TextBox Text="{Binding MediaRootOverride}" PlaceholderText="{Binding MediaDirectory}" />
|
||||||
</DockPanel>
|
</DockPanel>
|
||||||
<TextBlock Classes="muted" Text="{l:Loc Settings.MediaDirectoryHint}" TextWrapping="Wrap" />
|
<TextBlock Classes="muted" Text="{l:Loc Settings.MediaDirectoryHint}" TextWrapping="Wrap" />
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,3 @@
|
|||||||
|
bin/
|
||||||
|
obj/
|
||||||
|
Properties/launchSettings.json
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
<Project Sdk="Microsoft.NET.Sdk.Web">
|
||||||
|
<PropertyGroup>
|
||||||
|
<TargetFramework>net10.0</TargetFramework>
|
||||||
|
<Nullable>enable</Nullable>
|
||||||
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
|
<RootNamespace>AvParser.ImgTestService</RootNamespace>
|
||||||
|
<InvariantGlobalization>true</InvariantGlobalization>
|
||||||
|
<!-- No PackageReference: the Web SDK pulls ASP.NET Core in through the shared framework. -->
|
||||||
|
</PropertyGroup>
|
||||||
|
</Project>
|
||||||
@@ -0,0 +1,164 @@
|
|||||||
|
using System.Security.Cryptography;
|
||||||
|
|
||||||
|
namespace AvParser.ImgTestService;
|
||||||
|
|
||||||
|
/// <summary>Top-level service options, bound from the <c>ImgTest</c> configuration section.</summary>
|
||||||
|
public sealed class ImgTestOptions
|
||||||
|
{
|
||||||
|
/// <summary>Fraction of otherwise-valid ids that answer 404, to emulate a sparse site. 0 disables.</summary>
|
||||||
|
public double MissRate { get; set; }
|
||||||
|
|
||||||
|
/// <summary>The endpoints to expose, each with its own id shape.</summary>
|
||||||
|
public List<EndpointOptions> Endpoints { get; set; } = [];
|
||||||
|
|
||||||
|
/// <summary>Resolves every endpoint and indexes it by path segment.</summary>
|
||||||
|
public Dictionary<string, ResolvedEndpoint> BuildLookup()
|
||||||
|
{
|
||||||
|
var lookup = new Dictionary<string, ResolvedEndpoint>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
foreach (var endpoint in Endpoints)
|
||||||
|
{
|
||||||
|
var resolved = endpoint.Resolve();
|
||||||
|
if (resolved is not null)
|
||||||
|
{
|
||||||
|
lookup[resolved.Path] = resolved;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return lookup;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Whether an id should 404 despite being well-formed, deterministically by id.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Deterministic on purpose: a given id is either always present or always absent, so a
|
||||||
|
/// collector's "seen this url" journal and its duplicate detection stay meaningful across runs.
|
||||||
|
/// </remarks>
|
||||||
|
public static bool IsMiss(string core, double missRate)
|
||||||
|
{
|
||||||
|
if (missRate <= 0)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (missRate >= 1)
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
var hash = SHA256.HashData(System.Text.Encoding.UTF8.GetBytes("miss:" + core));
|
||||||
|
var value = ((uint)hash[0] << 24) | ((uint)hash[1] << 16) | ((uint)hash[2] << 8) | hash[3];
|
||||||
|
return value / (double)uint.MaxValue < missRate;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>One endpoint's configuration, as written in appsettings.</summary>
|
||||||
|
public sealed class EndpointOptions
|
||||||
|
{
|
||||||
|
/// <summary>First path segment, e.g. <c>test1</c> for <c>/test1/{id}</c>.</summary>
|
||||||
|
public string Path { get; set; } = string.Empty;
|
||||||
|
|
||||||
|
/// <summary>Shortest id core (excluding any extension).</summary>
|
||||||
|
public int MinLength { get; set; }
|
||||||
|
|
||||||
|
/// <summary>Longest id core (excluding any extension).</summary>
|
||||||
|
public int MaxLength { get; set; }
|
||||||
|
|
||||||
|
/// <summary>Named character set: Letters, LettersLower, LettersUpper, Digits, Alphanumeric, HexLower, or Custom.</summary>
|
||||||
|
public string Alphabet { get; set; } = "Alphanumeric";
|
||||||
|
|
||||||
|
/// <summary>Exact characters, used when <see cref="Alphabet"/> is Custom.</summary>
|
||||||
|
public string? Chars { get; set; }
|
||||||
|
|
||||||
|
/// <summary>Accepted trailing extensions, e.g. <c>.jpg</c>; empty means none is expected.</summary>
|
||||||
|
public List<string> Extensions { get; set; } = [];
|
||||||
|
|
||||||
|
/// <summary>Turns this into a resolved endpoint, or null when unusable.</summary>
|
||||||
|
public ResolvedEndpoint? Resolve()
|
||||||
|
{
|
||||||
|
var allowed = ResolveAlphabet(Alphabet, Chars);
|
||||||
|
if (string.IsNullOrWhiteSpace(Path) || MinLength < 1 || MaxLength < MinLength || allowed.Count == 0)
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
return new ResolvedEndpoint(
|
||||||
|
Path.Trim('/'),
|
||||||
|
MinLength,
|
||||||
|
MaxLength,
|
||||||
|
allowed,
|
||||||
|
[.. Extensions.Where(e => !string.IsNullOrWhiteSpace(e)).Select(Normalise)]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string Normalise(string extension)
|
||||||
|
{
|
||||||
|
var trimmed = extension.Trim();
|
||||||
|
return trimmed.StartsWith('.') ? trimmed : "." + trimmed;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static HashSet<char> ResolveAlphabet(string name, string? chars)
|
||||||
|
{
|
||||||
|
const string lower = "abcdefghijklmnopqrstuvwxyz";
|
||||||
|
const string upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
|
||||||
|
const string digit = "0123456789";
|
||||||
|
const string hex = "0123456789abcdef";
|
||||||
|
|
||||||
|
var set = name.Trim().ToLowerInvariant() switch
|
||||||
|
{
|
||||||
|
"letterslower" or "lower" => lower,
|
||||||
|
"lettersupper" or "upper" => upper,
|
||||||
|
"letters" => lower + upper,
|
||||||
|
"digits" or "digit" => digit,
|
||||||
|
"alphanumeric" or "alnum" => lower + upper + digit,
|
||||||
|
"hexlower" or "hex" => hex,
|
||||||
|
"custom" => chars ?? string.Empty,
|
||||||
|
_ => string.Empty,
|
||||||
|
};
|
||||||
|
|
||||||
|
return [.. set];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A validated endpoint ready to match requests.</summary>
|
||||||
|
public sealed record ResolvedEndpoint(
|
||||||
|
string Path,
|
||||||
|
int MinLength,
|
||||||
|
int MaxLength,
|
||||||
|
HashSet<char> Allowed,
|
||||||
|
string[] Extensions
|
||||||
|
)
|
||||||
|
{
|
||||||
|
/// <summary>Removes a recognised trailing extension from the raw path segment.</summary>
|
||||||
|
public string StripExtension(string id)
|
||||||
|
{
|
||||||
|
foreach (var extension in Extensions)
|
||||||
|
{
|
||||||
|
if (id.EndsWith(extension, StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
return id[..^extension.Length];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Whether an id core is the right length and made only of allowed characters.</summary>
|
||||||
|
public bool Matches(string core)
|
||||||
|
{
|
||||||
|
if (core.Length < MinLength || core.Length > MaxLength)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach (var c in core)
|
||||||
|
{
|
||||||
|
if (!Allowed.Contains(c))
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
<Project>
|
||||||
|
<!--
|
||||||
|
Stops the upward walk so the test service does not inherit the main repo's build props
|
||||||
|
(central package versions, TreatWarningsAsErrors, analyzers). It is a standalone image built
|
||||||
|
only inside its own Docker context, never as part of AvParser.slnx.
|
||||||
|
-->
|
||||||
|
<PropertyGroup>
|
||||||
|
<ManagePackageVersionsCentrally>false</ManagePackageVersionsCentrally>
|
||||||
|
</PropertyGroup>
|
||||||
|
</Project>
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Build the service, then run it on the smaller ASP.NET runtime image.
|
||||||
|
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
COPY Directory.Build.props ./
|
||||||
|
COPY AvParser.ImgTestService.csproj ./
|
||||||
|
RUN dotnet restore AvParser.ImgTestService.csproj
|
||||||
|
|
||||||
|
COPY . ./
|
||||||
|
RUN dotnet publish AvParser.ImgTestService.csproj -c Release -o /app --no-restore
|
||||||
|
|
||||||
|
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=build /app ./
|
||||||
|
|
||||||
|
# Listen on 8080 inside the container; a reverse proxy terminates TLS for the public domain.
|
||||||
|
ENV ASPNETCORE_URLS=http://+:8080
|
||||||
|
EXPOSE 8080
|
||||||
|
|
||||||
|
ENTRYPOINT ["dotnet", "AvParser.ImgTestService.dll"]
|
||||||
@@ -0,0 +1,198 @@
|
|||||||
|
using System.IO.Compression;
|
||||||
|
using System.Security.Cryptography;
|
||||||
|
|
||||||
|
namespace AvParser.ImgTestService;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Renders a deterministic identicon PNG from an id, with a hand-written encoder.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// No image library and no native dependency on purpose: the container stays tiny and runs the same
|
||||||
|
/// on any architecture. The same id always yields the same bytes, so a collector that fetches an id
|
||||||
|
/// twice gets one blob, and a duplicate-detection test has something stable to assert against.
|
||||||
|
/// </remarks>
|
||||||
|
public static class Identicon
|
||||||
|
{
|
||||||
|
private const int Cells = 5;
|
||||||
|
private const int CellSize = 48;
|
||||||
|
private const int Margin = 20;
|
||||||
|
private const int Size = (Cells * CellSize) + (2 * Margin);
|
||||||
|
|
||||||
|
/// <summary>Builds the PNG bytes for an id.</summary>
|
||||||
|
public static byte[] Render(string id)
|
||||||
|
{
|
||||||
|
var hash = SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(id));
|
||||||
|
|
||||||
|
// A saturated foreground from the first bytes, on a near-white background.
|
||||||
|
var (fr, fg, fb) = Foreground(hash);
|
||||||
|
byte br = 0xF2,
|
||||||
|
bg = 0xF2,
|
||||||
|
bb = 0xF4;
|
||||||
|
|
||||||
|
var pixels = new byte[Size * Size * 3];
|
||||||
|
FillBackground(pixels, br, bg, bb);
|
||||||
|
|
||||||
|
// A 5x5 grid, mirrored left-to-right, so the icon reads as a single symmetric shape. Bit
|
||||||
|
// source is the tail of the hash, one bit per cell in the left three columns.
|
||||||
|
for (var col = 0; col < (Cells + 1) / 2; col++)
|
||||||
|
{
|
||||||
|
for (var row = 0; row < Cells; row++)
|
||||||
|
{
|
||||||
|
var bit = hash[(col * Cells) + row] & 1;
|
||||||
|
if (bit == 0)
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
PaintCell(pixels, col, row, fr, fg, fb);
|
||||||
|
PaintCell(pixels, Cells - 1 - col, row, fr, fg, fb);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return Encode(pixels, Size, Size);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static (byte R, byte G, byte B) Foreground(byte[] hash)
|
||||||
|
{
|
||||||
|
// Pick a hue-ish colour that is never too pale: keep at least one channel low and one high.
|
||||||
|
var r = (byte)(60 + (hash[0] % 160));
|
||||||
|
var g = (byte)(60 + (hash[1] % 160));
|
||||||
|
var b = (byte)(60 + (hash[2] % 160));
|
||||||
|
return (r, g, b);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void FillBackground(byte[] pixels, byte r, byte g, byte b)
|
||||||
|
{
|
||||||
|
for (var i = 0; i < pixels.Length; i += 3)
|
||||||
|
{
|
||||||
|
pixels[i] = r;
|
||||||
|
pixels[i + 1] = g;
|
||||||
|
pixels[i + 2] = b;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void PaintCell(byte[] pixels, int col, int row, byte r, byte g, byte b)
|
||||||
|
{
|
||||||
|
var x0 = Margin + (col * CellSize);
|
||||||
|
var y0 = Margin + (row * CellSize);
|
||||||
|
|
||||||
|
for (var y = y0; y < y0 + CellSize; y++)
|
||||||
|
{
|
||||||
|
var rowStart = y * Size * 3;
|
||||||
|
for (var x = x0; x < x0 + CellSize; x++)
|
||||||
|
{
|
||||||
|
var i = rowStart + (x * 3);
|
||||||
|
pixels[i] = r;
|
||||||
|
pixels[i + 1] = g;
|
||||||
|
pixels[i + 2] = b;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Encodes RGB pixels as a PNG (colour type 2, 8-bit).</summary>
|
||||||
|
private static byte[] Encode(byte[] rgb, int width, int height)
|
||||||
|
{
|
||||||
|
using var output = new MemoryStream();
|
||||||
|
|
||||||
|
// Signature.
|
||||||
|
output.Write([0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]);
|
||||||
|
|
||||||
|
Span<byte> ihdr = stackalloc byte[13];
|
||||||
|
WriteBigEndian(ihdr[..4], (uint)width);
|
||||||
|
WriteBigEndian(ihdr.Slice(4, 4), (uint)height);
|
||||||
|
ihdr[8] = 8; // bit depth
|
||||||
|
ihdr[9] = 2; // colour type: truecolour RGB
|
||||||
|
ihdr[10] = 0; // compression
|
||||||
|
ihdr[11] = 0; // filter
|
||||||
|
ihdr[12] = 0; // interlace
|
||||||
|
WriteChunk(output, "IHDR"u8, ihdr);
|
||||||
|
|
||||||
|
// Raw scanlines: a leading filter byte (0 = none) then the row's RGB bytes.
|
||||||
|
var stride = width * 3;
|
||||||
|
var raw = new byte[height * (stride + 1)];
|
||||||
|
for (var y = 0; y < height; y++)
|
||||||
|
{
|
||||||
|
var src = y * stride;
|
||||||
|
var dst = y * (stride + 1);
|
||||||
|
raw[dst] = 0;
|
||||||
|
Array.Copy(rgb, src, raw, dst + 1, stride);
|
||||||
|
}
|
||||||
|
|
||||||
|
using (var compressed = new MemoryStream())
|
||||||
|
{
|
||||||
|
using (var zlib = new ZLibStream(compressed, CompressionLevel.Optimal, leaveOpen: true))
|
||||||
|
{
|
||||||
|
zlib.Write(raw, 0, raw.Length);
|
||||||
|
}
|
||||||
|
|
||||||
|
WriteChunk(output, "IDAT"u8, compressed.ToArray());
|
||||||
|
}
|
||||||
|
|
||||||
|
WriteChunk(output, "IEND"u8, []);
|
||||||
|
|
||||||
|
return output.ToArray();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void WriteChunk(Stream stream, ReadOnlySpan<byte> type, ReadOnlySpan<byte> data)
|
||||||
|
{
|
||||||
|
Span<byte> length = stackalloc byte[4];
|
||||||
|
WriteBigEndian(length, (uint)data.Length);
|
||||||
|
stream.Write(length);
|
||||||
|
stream.Write(type);
|
||||||
|
stream.Write(data);
|
||||||
|
|
||||||
|
var crc = Crc32.Compute(type, data);
|
||||||
|
Span<byte> crcBytes = stackalloc byte[4];
|
||||||
|
WriteBigEndian(crcBytes, crc);
|
||||||
|
stream.Write(crcBytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void WriteBigEndian(Span<byte> destination, uint value)
|
||||||
|
{
|
||||||
|
destination[0] = (byte)(value >> 24);
|
||||||
|
destination[1] = (byte)(value >> 16);
|
||||||
|
destination[2] = (byte)(value >> 8);
|
||||||
|
destination[3] = (byte)value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Minimal CRC-32 (PNG polynomial), enough to seal chunks.</summary>
|
||||||
|
internal static class Crc32
|
||||||
|
{
|
||||||
|
private static readonly uint[] Table = BuildTable();
|
||||||
|
|
||||||
|
public static uint Compute(ReadOnlySpan<byte> type, ReadOnlySpan<byte> data)
|
||||||
|
{
|
||||||
|
var crc = 0xFFFFFFFFu;
|
||||||
|
crc = Update(crc, type);
|
||||||
|
crc = Update(crc, data);
|
||||||
|
return crc ^ 0xFFFFFFFFu;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static uint Update(uint crc, ReadOnlySpan<byte> bytes)
|
||||||
|
{
|
||||||
|
foreach (var b in bytes)
|
||||||
|
{
|
||||||
|
crc = Table[(crc ^ b) & 0xFF] ^ (crc >> 8);
|
||||||
|
}
|
||||||
|
|
||||||
|
return crc;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static uint[] BuildTable()
|
||||||
|
{
|
||||||
|
var table = new uint[256];
|
||||||
|
for (var n = 0u; n < 256; n++)
|
||||||
|
{
|
||||||
|
var c = n;
|
||||||
|
for (var k = 0; k < 8; k++)
|
||||||
|
{
|
||||||
|
c = (c & 1) != 0 ? 0xEDB88320u ^ (c >> 1) : c >> 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
table[n] = c;
|
||||||
|
}
|
||||||
|
|
||||||
|
return table;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
using System.Text;
|
||||||
|
using AvParser.ImgTestService;
|
||||||
|
|
||||||
|
var builder = WebApplication.CreateBuilder(args);
|
||||||
|
|
||||||
|
var options = builder.Configuration.GetSection("ImgTest").Get<ImgTestOptions>() ?? new ImgTestOptions();
|
||||||
|
var endpoints = options.BuildLookup();
|
||||||
|
|
||||||
|
var app = builder.Build();
|
||||||
|
|
||||||
|
// A plain index so opening the host in a browser explains what is here.
|
||||||
|
app.MapGet(
|
||||||
|
"/",
|
||||||
|
() =>
|
||||||
|
{
|
||||||
|
var text = new StringBuilder();
|
||||||
|
text.AppendLine("AvParser image test service");
|
||||||
|
text.AppendLine();
|
||||||
|
text.AppendLine($"miss rate: {options.MissRate:0.###}");
|
||||||
|
text.AppendLine("endpoints:");
|
||||||
|
foreach (var endpoint in endpoints.Values)
|
||||||
|
{
|
||||||
|
var ext = endpoint.Extensions.Length == 0 ? "(none)" : string.Join(" ", endpoint.Extensions);
|
||||||
|
text.AppendLine(
|
||||||
|
$" /{endpoint.Path}/{{id}} length {endpoint.MinLength}-{endpoint.MaxLength}, "
|
||||||
|
+ $"{endpoint.Allowed.Count} chars, ext {ext}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return Results.Text(text.ToString(), "text/plain; charset=utf-8");
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
app.MapGet("/healthz", () => Results.Text("ok", "text/plain"));
|
||||||
|
|
||||||
|
// One handler for every configured endpoint: the first segment selects the config, the second is
|
||||||
|
// the id (optionally with an extension). A miss and an unknown endpoint are both a plain 404.
|
||||||
|
app.MapGet(
|
||||||
|
"/{endpoint}/{id}",
|
||||||
|
(string endpoint, string id) =>
|
||||||
|
{
|
||||||
|
if (!endpoints.TryGetValue(endpoint, out var config))
|
||||||
|
{
|
||||||
|
return Results.NotFound();
|
||||||
|
}
|
||||||
|
|
||||||
|
var core = config.StripExtension(id);
|
||||||
|
if (!config.Matches(core) || ImgTestOptions.IsMiss(core, options.MissRate))
|
||||||
|
{
|
||||||
|
return Results.NotFound();
|
||||||
|
}
|
||||||
|
|
||||||
|
return Results.Bytes(Identicon.Render(core), "image/png");
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
app.Run();
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
{
|
||||||
|
"profiles": {
|
||||||
|
"http": {
|
||||||
|
"commandName": "Project",
|
||||||
|
"dotnetRunMessages": true,
|
||||||
|
"applicationUrl": "http://localhost:8080",
|
||||||
|
"environmentVariables": {
|
||||||
|
"ASPNETCORE_ENVIRONMENT": "Development"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# AvParser image test service
|
||||||
|
|
||||||
|
A tiny, dependency-free web service that serves a deterministic image for any id matching a
|
||||||
|
configured pattern. It exists as a **target** for the app's pattern sources: point a source at one of
|
||||||
|
its endpoints and let the collector generate ids.
|
||||||
|
|
||||||
|
Each valid id returns a unique identicon PNG (derived from `SHA-256(id)`), so fetching the same id
|
||||||
|
twice yields the same bytes — which is what makes the collector's duplicate detection observable.
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
Endpoints are declared in [`appsettings.json`](appsettings.json) under `ImgTest:Endpoints`. The
|
||||||
|
defaults:
|
||||||
|
|
||||||
|
| Path | Id length | Characters | Extension |
|
||||||
|
| --------------- | --------- | ----------------- | --------- |
|
||||||
|
| `/test1/{id}` | 6–8 | letters + digits | `.jpg` |
|
||||||
|
| `/test2/{id}` | 8–12 | digits only | none |
|
||||||
|
| `/test3/{id}` | 8 | hex (`0-9a-f`) | `.png` |
|
||||||
|
|
||||||
|
Anything not matching a pattern (`/test1/xx`, `/test2/abcd1234`) returns `404`. `GET /` lists the
|
||||||
|
endpoints; `GET /healthz` returns `ok`.
|
||||||
|
|
||||||
|
Example: `GET /test1/2dfhyuj.jpg` → `200 image/png`.
|
||||||
|
|
||||||
|
### Adding an endpoint
|
||||||
|
|
||||||
|
Add an object to `ImgTest:Endpoints`:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "Path": "test4", "MinLength": 4, "MaxLength": 6, "Alphabet": "Custom", "Chars": "abcdef012", "Extensions": [".png"] }
|
||||||
|
```
|
||||||
|
|
||||||
|
`Alphabet` is one of `LettersLower`, `LettersUpper`, `Letters`, `Digits`, `Alphanumeric`, `HexLower`,
|
||||||
|
or `Custom` (which uses `Chars`).
|
||||||
|
|
||||||
|
## Miss rate
|
||||||
|
|
||||||
|
Set `ImgTest:MissRate` (0..1) to make a deterministic fraction of otherwise-valid ids answer `404`,
|
||||||
|
emulating a site where most guessed ids do not exist. It is stable per id, so a run is repeatable.
|
||||||
|
Override at deploy time with the `ImgTest__MissRate` environment variable.
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose up --build
|
||||||
|
```
|
||||||
|
|
||||||
|
The container listens on `http://localhost:8080`. It speaks plain HTTP — **TLS for a public domain
|
||||||
|
such as `https://imgtest.example` is terminated by your own reverse proxy**, which forwards to this
|
||||||
|
container. A Caddy example:
|
||||||
|
|
||||||
|
```
|
||||||
|
imgtest.example {
|
||||||
|
reverse_proxy localhost:8080
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Or run it directly without Docker:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dotnet run
|
||||||
|
```
|
||||||
|
|
||||||
|
> This service is intentionally **not** part of `AvParser.slnx` and is not built by `./build.ps1`.
|
||||||
|
> It is a standalone support tool with its own Docker context.
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
{
|
||||||
|
"Logging": {
|
||||||
|
"LogLevel": {
|
||||||
|
"Default": "Information",
|
||||||
|
"Microsoft.AspNetCore": "Warning"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"ImgTest": {
|
||||||
|
"MissRate": 0.0,
|
||||||
|
"Endpoints": [
|
||||||
|
{
|
||||||
|
"Path": "test1",
|
||||||
|
"MinLength": 6,
|
||||||
|
"MaxLength": 8,
|
||||||
|
"Alphabet": "Alphanumeric",
|
||||||
|
"Extensions": [".jpg"]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Path": "test2",
|
||||||
|
"MinLength": 8,
|
||||||
|
"MaxLength": 12,
|
||||||
|
"Alphabet": "Digits",
|
||||||
|
"Extensions": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Path": "test3",
|
||||||
|
"MinLength": 8,
|
||||||
|
"MaxLength": 8,
|
||||||
|
"Alphabet": "HexLower",
|
||||||
|
"Extensions": [".png"]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
services:
|
||||||
|
imgtest:
|
||||||
|
build: .
|
||||||
|
image: avparser-imgtest
|
||||||
|
container_name: avparser-imgtest
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
# host:container — change the host port to suit; the reverse proxy points at it.
|
||||||
|
- "8080:8080"
|
||||||
|
environment:
|
||||||
|
# Raise this (0..1) to make a fraction of valid ids answer 404, emulating a sparse site.
|
||||||
|
- ImgTest__MissRate=0.0
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
|
|
||||||
|
namespace AvParser.Core.Tests.Collecting;
|
||||||
|
|
||||||
|
public class MediaSourceCatalogTests
|
||||||
|
{
|
||||||
|
private sealed class FakeStore(IEnumerable<PatternSourceConfig>? seed = null) : IUserSourceStore
|
||||||
|
{
|
||||||
|
private readonly List<PatternSourceConfig> _configs = seed?.ToList() ?? [];
|
||||||
|
|
||||||
|
public event EventHandler? Changed;
|
||||||
|
|
||||||
|
public IReadOnlyList<PatternSourceConfig> List() => [.. _configs];
|
||||||
|
|
||||||
|
public Task<PatternSourceConfig> AddAsync(
|
||||||
|
PatternSourceConfig config,
|
||||||
|
CancellationToken cancellationToken = default
|
||||||
|
)
|
||||||
|
{
|
||||||
|
_configs.Add(config);
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return Task.FromResult(config);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<bool> UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
var index = _configs.FindIndex(c => c.Id == config.Id);
|
||||||
|
if (index < 0)
|
||||||
|
{
|
||||||
|
return Task.FromResult(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
_configs[index] = config;
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return Task.FromResult(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<bool> RemoveAsync(string id, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
var removed = _configs.RemoveAll(c => c.Id == id) > 0;
|
||||||
|
if (removed)
|
||||||
|
{
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
}
|
||||||
|
|
||||||
|
return Task.FromResult(removed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static PatternSourceConfig Config(string id, string name)
|
||||||
|
{
|
||||||
|
PatternSourceConfig.TryCreate(
|
||||||
|
name,
|
||||||
|
"https://h/x/",
|
||||||
|
6,
|
||||||
|
8,
|
||||||
|
IdAlphabet.Digits,
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
allowDirectConnection: false,
|
||||||
|
out var config,
|
||||||
|
id
|
||||||
|
);
|
||||||
|
return config!;
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void An_empty_store_is_a_valid_empty_catalog()
|
||||||
|
{
|
||||||
|
using var catalog = new MediaSourceCatalog(new FakeStore());
|
||||||
|
|
||||||
|
catalog.Sources.ShouldBeEmpty();
|
||||||
|
catalog.Find("anything").ShouldBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Sources_are_materialised_from_the_stored_configs()
|
||||||
|
{
|
||||||
|
using var catalog = new MediaSourceCatalog(new FakeStore([Config("s1", "Beta"), Config("s2", "Alpha")]));
|
||||||
|
|
||||||
|
// Ordered by display name.
|
||||||
|
catalog.Sources.Select(s => s.Id).ShouldBe(["s2", "s1"]);
|
||||||
|
catalog.Find("s1").ShouldNotBeNull().DisplayName.ShouldBe("Beta");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Adding_through_the_catalog_rebuilds_and_signals()
|
||||||
|
{
|
||||||
|
var store = new FakeStore();
|
||||||
|
using var catalog = new MediaSourceCatalog(store);
|
||||||
|
var changed = 0;
|
||||||
|
catalog.Changed += (_, _) => changed++;
|
||||||
|
|
||||||
|
await catalog.AddAsync(Config("s1", "One"), TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
changed.ShouldBe(1);
|
||||||
|
catalog.Sources.ShouldHaveSingleItem().Id.ShouldBe("s1");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Removing_through_the_catalog_drops_the_source()
|
||||||
|
{
|
||||||
|
using var catalog = new MediaSourceCatalog(new FakeStore([Config("s1", "One")]));
|
||||||
|
|
||||||
|
await catalog.RemoveAsync("s1", TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
catalog.Sources.ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,283 @@
|
|||||||
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
|
|
||||||
|
namespace AvParser.Core.Tests.Collecting;
|
||||||
|
|
||||||
|
public class PatternMediaSourceTests
|
||||||
|
{
|
||||||
|
private static PatternSourceConfig Config(
|
||||||
|
int min = 6,
|
||||||
|
int max = 8,
|
||||||
|
IdAlphabet alphabet = IdAlphabet.Alphanumeric,
|
||||||
|
string? custom = null,
|
||||||
|
string? extension = ".jpg",
|
||||||
|
string url = "https://imgtest.example/test1/",
|
||||||
|
bool allowDirect = false
|
||||||
|
)
|
||||||
|
{
|
||||||
|
PatternSourceConfig.TryCreate("Test", url, min, max, alphabet, custom, extension, allowDirect, out var config);
|
||||||
|
return config!;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static async Task<List<MediaCandidate>> Collect(IMediaSource source, int limit)
|
||||||
|
{
|
||||||
|
var list = new List<MediaCandidate>();
|
||||||
|
await foreach (var outcome in source.ParseAsync(new MediaQuery(Limit: limit), null, CancellationToken.None))
|
||||||
|
{
|
||||||
|
if (outcome.IsSuccess)
|
||||||
|
{
|
||||||
|
list.Add(outcome.Value!);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return list;
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void A_pattern_source_needs_the_network()
|
||||||
|
{
|
||||||
|
new PatternMediaSource(Config()).RequiresNetwork.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Generated_ids_stay_within_the_length_range_and_alphabet()
|
||||||
|
{
|
||||||
|
var source = new PatternMediaSource(Config(min: 4, max: 6, alphabet: IdAlphabet.Digits, extension: null));
|
||||||
|
|
||||||
|
var candidates = await Collect(source, 300);
|
||||||
|
|
||||||
|
candidates.Count.ShouldBe(300);
|
||||||
|
foreach (var candidate in candidates)
|
||||||
|
{
|
||||||
|
var id = candidate.ExternalId.ShouldNotBeNull();
|
||||||
|
id.Length.ShouldBeInRange(4, 6);
|
||||||
|
id.ShouldAllBe(c => char.IsAsciiDigit(c));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Candidates_hang_the_id_off_the_base_url_with_the_extension()
|
||||||
|
{
|
||||||
|
var source = new PatternMediaSource(Config(min: 6, max: 6, alphabet: IdAlphabet.HexLower, extension: ".png"));
|
||||||
|
|
||||||
|
var candidate = (await Collect(source, 1)).ShouldHaveSingleItem();
|
||||||
|
|
||||||
|
candidate.SourceId.ShouldBe(source.Id);
|
||||||
|
candidate.Url.AbsoluteUri.ShouldBe($"https://imgtest.example/test1/{candidate.ExternalId}.png");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task An_unset_extension_still_lands_on_a_jpg_address()
|
||||||
|
{
|
||||||
|
var source = new PatternMediaSource(Config(min: 6, max: 6, alphabet: IdAlphabet.HexLower, extension: null));
|
||||||
|
|
||||||
|
var candidate = (await Collect(source, 1)).ShouldHaveSingleItem();
|
||||||
|
|
||||||
|
candidate.Url.AbsoluteUri.ShouldBe($"https://imgtest.example/test1/{candidate.ExternalId}.jpg");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task The_attempt_budget_bounds_how_many_are_generated()
|
||||||
|
{
|
||||||
|
var source = new PatternMediaSource(Config(min: 6, max: 8, alphabet: IdAlphabet.Alphanumeric));
|
||||||
|
|
||||||
|
(await Collect(source, 40)).Count.ShouldBe(40);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task A_small_space_is_never_exceeded_however_large_the_budget()
|
||||||
|
{
|
||||||
|
// Digits of length 2 is a hundred ids; asking for a thousand must not loop forever.
|
||||||
|
var source = new PatternMediaSource(Config(min: 2, max: 2, alphabet: IdAlphabet.Digits, extension: null));
|
||||||
|
|
||||||
|
var candidates = await Collect(source, 1000);
|
||||||
|
|
||||||
|
candidates.Count.ShouldBeLessThanOrEqualTo(100);
|
||||||
|
candidates.Select(c => c.ExternalId).Distinct().Count().ShouldBe(candidates.Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task No_budget_means_the_source_keeps_going_until_it_is_stopped()
|
||||||
|
{
|
||||||
|
// A limit of zero used to mean "one candidate", which made an unlimited run collect nothing.
|
||||||
|
var source = new PatternMediaSource(Config(min: 8, max: 8, alphabet: IdAlphabet.Alphanumeric));
|
||||||
|
using var stop = new CancellationTokenSource();
|
||||||
|
|
||||||
|
var seen = 0;
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await foreach (var outcome in source.ParseAsync(new MediaQuery(), null, stop.Token))
|
||||||
|
{
|
||||||
|
outcome.IsSuccess.ShouldBeTrue();
|
||||||
|
|
||||||
|
if (++seen >= 5_000)
|
||||||
|
{
|
||||||
|
await stop.CancelAsync();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException)
|
||||||
|
{
|
||||||
|
// The only way an unbounded run ends.
|
||||||
|
}
|
||||||
|
|
||||||
|
seen.ShouldBe(5_000);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task A_small_space_still_bounds_an_unlimited_run()
|
||||||
|
{
|
||||||
|
// Two digits is a hundred ids: with no budget the run must end at the space, not spin on
|
||||||
|
// collisions for ever.
|
||||||
|
var source = new PatternMediaSource(Config(min: 2, max: 2, alphabet: IdAlphabet.Digits, extension: null));
|
||||||
|
|
||||||
|
var candidates = await Collect(source, 0);
|
||||||
|
|
||||||
|
// Not exactly a hundred: the draw is random, so the last few ids may be given up on rather
|
||||||
|
// than waited for. What matters is that the run ends and never repeats itself.
|
||||||
|
candidates.Count.ShouldBeInRange(90, 100);
|
||||||
|
candidates.Select(c => c.ExternalId).Distinct().Count().ShouldBe(candidates.Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Generated_ids_are_unique_within_a_run()
|
||||||
|
{
|
||||||
|
var source = new PatternMediaSource(Config(min: 8, max: 8, alphabet: IdAlphabet.Alphanumeric, extension: null));
|
||||||
|
|
||||||
|
var candidates = await Collect(source, 500);
|
||||||
|
|
||||||
|
candidates.Select(c => c.ExternalId).Distinct().Count().ShouldBe(500);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public class PatternAlphabetTests
|
||||||
|
{
|
||||||
|
[Theory]
|
||||||
|
[InlineData(IdAlphabet.LettersLower, "abcdefghijklmnopqrstuvwxyz")]
|
||||||
|
[InlineData(IdAlphabet.Digits, "0123456789")]
|
||||||
|
[InlineData(IdAlphabet.HexLower, "0123456789abcdef")]
|
||||||
|
public void Named_sets_resolve_to_their_characters(IdAlphabet alphabet, string expected)
|
||||||
|
{
|
||||||
|
PatternAlphabet.Resolve(alphabet).ShouldBe(expected);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Letters_and_alphanumeric_combine_the_sets()
|
||||||
|
{
|
||||||
|
PatternAlphabet.Resolve(IdAlphabet.Letters).Length.ShouldBe(52);
|
||||||
|
PatternAlphabet.Resolve(IdAlphabet.Alphanumeric).Length.ShouldBe(62);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void A_custom_set_is_de_duplicated_and_stripped_of_separators()
|
||||||
|
{
|
||||||
|
// Repeats would skew the draw; a slash would break the "one path segment" assumption.
|
||||||
|
PatternAlphabet.Resolve(IdAlphabet.Custom, "aab/c c").ShouldBe("abc");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void An_empty_custom_set_resolves_to_nothing()
|
||||||
|
{
|
||||||
|
PatternAlphabet.Resolve(IdAlphabet.Custom, " ").ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public class PatternSourceConfigTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void A_valid_config_normalises_the_base_url_and_extension()
|
||||||
|
{
|
||||||
|
var error = PatternSourceConfig.TryCreate(
|
||||||
|
" My source ",
|
||||||
|
"https://imgtest.example/test1",
|
||||||
|
6,
|
||||||
|
8,
|
||||||
|
IdAlphabet.Alphanumeric,
|
||||||
|
null,
|
||||||
|
"jpg",
|
||||||
|
allowDirectConnection: false,
|
||||||
|
out var config
|
||||||
|
);
|
||||||
|
|
||||||
|
error.ShouldBe(PatternConfigError.None);
|
||||||
|
config.ShouldNotBeNull();
|
||||||
|
config.Name.ShouldBe("My source");
|
||||||
|
config.BaseUrl.AbsoluteUri.ShouldBe("https://imgtest.example/test1/");
|
||||||
|
config.Extension.ShouldBe(".jpg");
|
||||||
|
config.Id.ShouldNotBeNullOrWhiteSpace();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void A_blank_extension_becomes_the_default_one()
|
||||||
|
{
|
||||||
|
// The editor shows ".jpg" as the placeholder, so an untouched field means "the usual one",
|
||||||
|
// not "no suffix" — the latter costs a whole run of 404s to discover.
|
||||||
|
var error = PatternSourceConfig.TryCreate(
|
||||||
|
"N",
|
||||||
|
"https://h/x/",
|
||||||
|
6,
|
||||||
|
8,
|
||||||
|
IdAlphabet.Digits,
|
||||||
|
null,
|
||||||
|
" ",
|
||||||
|
allowDirectConnection: false,
|
||||||
|
out var config
|
||||||
|
);
|
||||||
|
|
||||||
|
error.ShouldBe(PatternConfigError.None);
|
||||||
|
config.ShouldNotBeNull().Extension.ShouldBe(PatternSourceConfig.DefaultExtension);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void A_supplied_id_is_kept()
|
||||||
|
{
|
||||||
|
PatternSourceConfig.TryCreate(
|
||||||
|
"N",
|
||||||
|
"https://h/x/",
|
||||||
|
6,
|
||||||
|
8,
|
||||||
|
IdAlphabet.Digits,
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
allowDirectConnection: false,
|
||||||
|
out var config,
|
||||||
|
"fixed-id"
|
||||||
|
);
|
||||||
|
|
||||||
|
config.ShouldNotBeNull().Id.ShouldBe("fixed-id");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Theory]
|
||||||
|
[InlineData("", "https://h/x/", 6, 8, IdAlphabet.Digits, PatternConfigError.NameRequired)]
|
||||||
|
[InlineData("N", "not-a-url", 6, 8, IdAlphabet.Digits, PatternConfigError.BaseUrlInvalid)]
|
||||||
|
[InlineData("N", "ftp://h/x/", 6, 8, IdAlphabet.Digits, PatternConfigError.BaseUrlInvalid)]
|
||||||
|
[InlineData("N", "https://h/x/", 0, 8, IdAlphabet.Digits, PatternConfigError.LengthRangeInvalid)]
|
||||||
|
[InlineData("N", "https://h/x/", 8, 6, IdAlphabet.Digits, PatternConfigError.LengthRangeInvalid)]
|
||||||
|
[InlineData("N", "https://h/x/", 6, 65, IdAlphabet.Digits, PatternConfigError.LengthRangeInvalid)]
|
||||||
|
[InlineData("N", "https://h/x/", 6, 8, IdAlphabet.Custom, PatternConfigError.AlphabetEmpty)]
|
||||||
|
public void Invalid_inputs_are_rejected_with_a_reason(
|
||||||
|
string name,
|
||||||
|
string url,
|
||||||
|
int min,
|
||||||
|
int max,
|
||||||
|
IdAlphabet alphabet,
|
||||||
|
PatternConfigError expected
|
||||||
|
)
|
||||||
|
{
|
||||||
|
var error = PatternSourceConfig.TryCreate(
|
||||||
|
name,
|
||||||
|
url,
|
||||||
|
min,
|
||||||
|
max,
|
||||||
|
alphabet,
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
allowDirectConnection: false,
|
||||||
|
out var config
|
||||||
|
);
|
||||||
|
|
||||||
|
error.ShouldBe(expected);
|
||||||
|
config.ShouldBeNull();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,198 +0,0 @@
|
|||||||
using AvParser.Core.Collecting;
|
|
||||||
using AvParser.Core.Collecting.Sources;
|
|
||||||
using AvParser.Core.Parsing;
|
|
||||||
|
|
||||||
namespace AvParser.Core.Tests.Collecting;
|
|
||||||
|
|
||||||
public class UrlListSourceTests
|
|
||||||
{
|
|
||||||
private static readonly IMediaSource Source = new UrlListSource();
|
|
||||||
|
|
||||||
private static async Task<List<ParseOutcome<MediaCandidate>>> RunAsync(string text, int limit = 0)
|
|
||||||
{
|
|
||||||
var results = new List<ParseOutcome<MediaCandidate>>();
|
|
||||||
|
|
||||||
await foreach (
|
|
||||||
var outcome in Source.ParseAsync(
|
|
||||||
new MediaQuery(text, Limit: limit),
|
|
||||||
null,
|
|
||||||
TestContext.Current.CancellationToken
|
|
||||||
)
|
|
||||||
)
|
|
||||||
{
|
|
||||||
results.Add(outcome);
|
|
||||||
}
|
|
||||||
|
|
||||||
return results;
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public async Task Each_line_becomes_a_candidate()
|
|
||||||
{
|
|
||||||
var results = await RunAsync("https://example.test/a.png\nhttps://example.test/b.gif");
|
|
||||||
|
|
||||||
results.Count.ShouldBe(2);
|
|
||||||
results.ShouldAllBe(r => r.IsSuccess);
|
|
||||||
results[0].Value!.Url.AbsoluteUri.ShouldBe("https://example.test/a.png");
|
|
||||||
results[1].Value!.Ordinal.ShouldBe(2);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public async Task Blank_lines_and_comments_are_ignored()
|
|
||||||
{
|
|
||||||
var results = await RunAsync(
|
|
||||||
"""
|
|
||||||
# my list
|
|
||||||
https://example.test/a.png
|
|
||||||
|
|
||||||
# indented comment
|
|
||||||
https://example.test/b.png
|
|
||||||
"""
|
|
||||||
);
|
|
||||||
|
|
||||||
results.Count.ShouldBe(2);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public async Task Whitespace_around_an_address_is_forgiven()
|
|
||||||
{
|
|
||||||
var results = await RunAsync(" https://example.test/a.png ");
|
|
||||||
|
|
||||||
results.ShouldHaveSingleItem().IsSuccess.ShouldBeTrue();
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public async Task A_bad_line_is_named_rather_than_dropped()
|
|
||||||
{
|
|
||||||
// In a paste of two hundred addresses a silently skipped typo is unfindable.
|
|
||||||
var results = await RunAsync("https://example.test/a.png\nnot an address\nhttps://example.test/b.png");
|
|
||||||
|
|
||||||
results.Count.ShouldBe(3);
|
|
||||||
results[1].IsSuccess.ShouldBeFalse();
|
|
||||||
results[1].Error!.Code.ShouldBe("NotAnAddress");
|
|
||||||
results[1].Error!.Arguments.ShouldContain("not an address");
|
|
||||||
}
|
|
||||||
|
|
||||||
[Theory]
|
|
||||||
[InlineData("file:///etc/passwd")]
|
|
||||||
[InlineData("data:image/png;base64,AAAA")]
|
|
||||||
[InlineData("ftp://example.test/a.png")]
|
|
||||||
[InlineData("javascript:alert(1)")]
|
|
||||||
public async Task Only_http_addresses_are_accepted(string line)
|
|
||||||
{
|
|
||||||
// A pasted list is as likely to have come from somewhere else as to have been typed.
|
|
||||||
var results = await RunAsync(line);
|
|
||||||
|
|
||||||
results.ShouldHaveSingleItem().IsSuccess.ShouldBeFalse();
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public async Task A_limit_stops_the_listing_early()
|
|
||||||
{
|
|
||||||
var text = string.Join('\n', Enumerable.Range(0, 50).Select(i => $"https://example.test/{i}.png"));
|
|
||||||
|
|
||||||
var results = await RunAsync(text, limit: 5);
|
|
||||||
|
|
||||||
results.Count(r => r.IsSuccess).ShouldBe(5);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void The_source_needs_no_network_of_its_own()
|
|
||||||
{
|
|
||||||
// It discovers nothing; downloading what it read is the fetcher's job and is gated there.
|
|
||||||
Source.RequiresNetwork.ShouldBeFalse();
|
|
||||||
}
|
|
||||||
|
|
||||||
[Theory]
|
|
||||||
[InlineData("https://example.test/a.png", true)]
|
|
||||||
[InlineData("# only a comment", false)]
|
|
||||||
[InlineData("", false)]
|
|
||||||
[InlineData("nonsense", false)]
|
|
||||||
public void CanParse_answers_without_doing_any_work(string text, bool expected) =>
|
|
||||||
Source.CanParse(new MediaQuery(text)).ShouldBe(expected);
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public async Task Cancellation_is_honoured()
|
|
||||||
{
|
|
||||||
using var cancellation = new CancellationTokenSource();
|
|
||||||
await cancellation.CancelAsync();
|
|
||||||
|
|
||||||
await Should.ThrowAsync<OperationCanceledException>(async () =>
|
|
||||||
{
|
|
||||||
await foreach (var _ in Source.ParseAsync(new MediaQuery("https://a.test/x.png"), null, cancellation.Token))
|
|
||||||
{
|
|
||||||
// Draining is the point; the first move must already throw.
|
|
||||||
}
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public class MediaSourceCatalogTests
|
|
||||||
{
|
|
||||||
[Fact]
|
|
||||||
public void Sources_are_ordered_by_display_name()
|
|
||||||
{
|
|
||||||
var catalog = new MediaSourceCatalog([new FakeSource("z", "Zebra"), new FakeSource("a", "Aardvark")]);
|
|
||||||
|
|
||||||
catalog.Sources.Select(s => s.Id).ShouldBe(["a", "z"]);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void The_named_default_wins_over_alphabetical_order()
|
|
||||||
{
|
|
||||||
// Otherwise the landing page depends on a display name, and would open on the network
|
|
||||||
// source — behind the proxy gate — before the user has asked for anything.
|
|
||||||
var catalog = new MediaSourceCatalog([new FakeSource("a", "Aardvark"), new FakeSource("z", "Zebra")], "z");
|
|
||||||
|
|
||||||
catalog.DefaultSource.Id.ShouldBe("z");
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void An_unknown_default_falls_back_rather_than_throwing()
|
|
||||||
{
|
|
||||||
var catalog = new MediaSourceCatalog([new FakeSource("a", "Aardvark")], "removed-in-a-past-version");
|
|
||||||
|
|
||||||
catalog.DefaultSource.Id.ShouldBe("a");
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void Lookup_ignores_case_and_reports_a_miss()
|
|
||||||
{
|
|
||||||
IMediaSourceCatalog catalog = new MediaSourceCatalog([new FakeSource("url-list", "URL list")]);
|
|
||||||
|
|
||||||
catalog.Find("URL-LIST").ShouldNotBeNull();
|
|
||||||
catalog.Find("nope").ShouldBeNull();
|
|
||||||
catalog.FindOrDefault("nope").Id.ShouldBe("url-list");
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void An_empty_registration_is_rejected() =>
|
|
||||||
Should.Throw<ArgumentException>(() => new MediaSourceCatalog([]));
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void Two_sources_sharing_an_id_are_rejected() =>
|
|
||||||
Should.Throw<ArgumentException>(() =>
|
|
||||||
new MediaSourceCatalog([new FakeSource("same", "One"), new FakeSource("same", "Two")])
|
|
||||||
);
|
|
||||||
|
|
||||||
private sealed class FakeSource(string id, string name) : IMediaSource
|
|
||||||
{
|
|
||||||
public string Id => id;
|
|
||||||
|
|
||||||
public string DisplayName => name;
|
|
||||||
|
|
||||||
public string Description => string.Empty;
|
|
||||||
|
|
||||||
public bool CanParse(MediaQuery input) => true;
|
|
||||||
|
|
||||||
public async IAsyncEnumerable<ParseOutcome<MediaCandidate>> ParseAsync(
|
|
||||||
MediaQuery input,
|
|
||||||
IProgress<ParseProgress>? progress,
|
|
||||||
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken
|
|
||||||
)
|
|
||||||
{
|
|
||||||
await Task.Yield();
|
|
||||||
yield break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -102,4 +102,57 @@ public class ProxyPoolWarmUpTests
|
|||||||
(await pool.WarmUpAsync(1, cancellationToken: TestContext.Current.CancellationToken)).ShouldBe(0);
|
(await pool.WarmUpAsync(1, cancellationToken: TestContext.Current.CancellationToken)).ShouldBe(0);
|
||||||
pool.LiveCount.ShouldBe(0);
|
pool.LiveCount.ShouldBe(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task The_top_up_checks_what_the_warm_up_skipped()
|
||||||
|
{
|
||||||
|
// The warm-up leaves almost everything unknown by design; without this pass the pool looks
|
||||||
|
// — and behaves — as if it held two proxies rather than the fifty that answer.
|
||||||
|
var pool = Build(out var probe, out _, [.. Enumerable.Range(0, 50).Select(index => $"h{index}")]);
|
||||||
|
await pool.RefreshAsync(TestContext.Current.CancellationToken);
|
||||||
|
probe.DefaultAlive = true;
|
||||||
|
|
||||||
|
await pool.WarmUpAsync(2, cancellationToken: TestContext.Current.CancellationToken);
|
||||||
|
var afterWarmUp = probe.ProbeCount;
|
||||||
|
|
||||||
|
afterWarmUp.ShouldBeLessThan(50);
|
||||||
|
|
||||||
|
var found = await pool.TopUpAsync(4, cancellationToken: TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
found.ShouldBe(50 - afterWarmUp);
|
||||||
|
pool.LiveCount.ShouldBe(50);
|
||||||
|
pool.Entries.ShouldAllBe(entry => entry.Health == ProxyHealthState.Alive);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task The_top_up_does_not_re_probe_what_is_already_known()
|
||||||
|
{
|
||||||
|
var pool = Build(out var probe, out _, "a", "b");
|
||||||
|
await pool.RefreshAsync(TestContext.Current.CancellationToken);
|
||||||
|
probe.DefaultAlive = true;
|
||||||
|
|
||||||
|
await pool.TopUpAsync(2, cancellationToken: TestContext.Current.CancellationToken);
|
||||||
|
var first = probe.ProbeCount;
|
||||||
|
|
||||||
|
// Everything has a verdict now, so a second pass has nothing to do — re-probing would just
|
||||||
|
// be a sweep, and a sweep is something the user asks for.
|
||||||
|
(await pool.TopUpAsync(2, cancellationToken: TestContext.Current.CancellationToken)).ShouldBe(0);
|
||||||
|
probe.ProbeCount.ShouldBe(first);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task A_stopped_top_up_keeps_what_it_learned()
|
||||||
|
{
|
||||||
|
var pool = Build(out var probe, out _, [.. Enumerable.Range(0, 20).Select(index => $"h{index}")]);
|
||||||
|
await pool.RefreshAsync(TestContext.Current.CancellationToken);
|
||||||
|
probe.DefaultAlive = true;
|
||||||
|
|
||||||
|
using var stop = new CancellationTokenSource();
|
||||||
|
await stop.CancelAsync();
|
||||||
|
|
||||||
|
await Should.ThrowAsync<OperationCanceledException>(() => pool.TopUpAsync(2, null, stop.Token));
|
||||||
|
|
||||||
|
// Cancelled before anything was probed, so nothing is claimed to be live either.
|
||||||
|
pool.LiveCount.ShouldBe(0);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -138,8 +138,7 @@ public sealed class CollectRunnerTests : IAsyncLifetime
|
|||||||
NullLogger<MediaStore>.Instance
|
NullLogger<MediaStore>.Instance
|
||||||
);
|
);
|
||||||
|
|
||||||
// Direct is allowed here: these tests are about the runner, not the proxy gate.
|
_settings = new FixedSettings(new AppSettings());
|
||||||
_settings = new FixedSettings(new AppSettings { AllowDirectConnection = true });
|
|
||||||
_fetcher = new ScriptedFetcher(_blobs);
|
_fetcher = new ScriptedFetcher(_blobs);
|
||||||
_throttle = new HostThrottle(4, TimeSpan.Zero, NullLogger<CollectRunnerTests>.Instance);
|
_throttle = new HostThrottle(4, TimeSpan.Zero, NullLogger<CollectRunnerTests>.Instance);
|
||||||
_runner = new CollectRunner(_fetcher, _store, _throttle, NullLogger<CollectRunner>.Instance);
|
_runner = new CollectRunner(_fetcher, _store, _throttle, NullLogger<CollectRunner>.Instance);
|
||||||
@@ -354,66 +353,3 @@ public sealed class CollectRunnerTests : IAsyncLifetime
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public class OwnServiceListingTests
|
|
||||||
{
|
|
||||||
private static readonly Uri Endpoint = new("https://own.test/api/list");
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void An_object_with_items_is_read()
|
|
||||||
{
|
|
||||||
var page = OwnServiceSource.ReadPage(
|
|
||||||
"""
|
|
||||||
{ "items": [ { "url": "https://own.test/a.png", "id": "42", "name": "kitten",
|
|
||||||
"published": "2026-08-13T10:00:00Z", "size": 4096, "tags": ["cats"] } ],
|
|
||||||
"next": "page2" }
|
|
||||||
""",
|
|
||||||
Endpoint
|
|
||||||
);
|
|
||||||
|
|
||||||
var item = page.Items.ShouldHaveSingleItem();
|
|
||||||
item.Url.AbsoluteUri.ShouldBe("https://own.test/a.png");
|
|
||||||
item.ExternalId.ShouldBe("42");
|
|
||||||
item.SuggestedName.ShouldBe("kitten");
|
|
||||||
item.ExpectedLength.ShouldBe(4096);
|
|
||||||
item.Tags.ShouldBe(["cats"]);
|
|
||||||
page.Next.ShouldBe("page2");
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void A_bare_array_of_addresses_is_read()
|
|
||||||
{
|
|
||||||
// The service on the other end is the user's own; it should not have to be rewritten to
|
|
||||||
// match a schema we invented.
|
|
||||||
var page = OwnServiceSource.ReadPage("""["https://own.test/a.png", "https://own.test/b.gif"]""", Endpoint);
|
|
||||||
|
|
||||||
page.Items.Count.ShouldBe(2);
|
|
||||||
page.Next.ShouldBeNull();
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void Relative_addresses_resolve_against_the_endpoint()
|
|
||||||
{
|
|
||||||
var page = OwnServiceSource.ReadPage("""{"items":[{"url":"/files/a.png"}]}""", Endpoint);
|
|
||||||
|
|
||||||
page.Items.ShouldHaveSingleItem().Url.AbsoluteUri.ShouldBe("https://own.test/files/a.png");
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void Entries_without_a_usable_address_are_dropped()
|
|
||||||
{
|
|
||||||
var page = OwnServiceSource.ReadPage(
|
|
||||||
"""{"items":[{"name":"no url"}, {"url":"data:image/png;base64,AA"}, {"url":"https://own.test/ok.png"}]}""",
|
|
||||||
Endpoint
|
|
||||||
);
|
|
||||||
|
|
||||||
page.Items.ShouldHaveSingleItem().Url.AbsoluteUri.ShouldBe("https://own.test/ok.png");
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void An_empty_listing_is_not_an_error()
|
|
||||||
{
|
|
||||||
OwnServiceSource.ReadPage("""{"items":[]}""", Endpoint).Items.ShouldBeEmpty();
|
|
||||||
OwnServiceSource.ReadPage("[]", Endpoint).Items.ShouldBeEmpty();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -0,0 +1,124 @@
|
|||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
|
using AvParser.Infrastructure.Collecting;
|
||||||
|
using AvParser.Infrastructure.Storage;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
|
||||||
|
namespace AvParser.Infrastructure.Tests.Collecting;
|
||||||
|
|
||||||
|
public sealed class JsonUserSourceStoreTests : IDisposable
|
||||||
|
{
|
||||||
|
private readonly string _directory = Path.Combine(
|
||||||
|
Path.GetTempPath(),
|
||||||
|
"AvParserTests",
|
||||||
|
Guid.NewGuid().ToString("N")
|
||||||
|
);
|
||||||
|
|
||||||
|
private JsonUserSourceStore Create() => new(new AppPaths(_directory), NullLogger<JsonUserSourceStore>.Instance);
|
||||||
|
|
||||||
|
private static PatternSourceConfig Config(string id, string name = "Test", bool allowDirect = false)
|
||||||
|
{
|
||||||
|
PatternSourceConfig.TryCreate(
|
||||||
|
name,
|
||||||
|
"https://imgtest.example/test1/",
|
||||||
|
6,
|
||||||
|
8,
|
||||||
|
IdAlphabet.Alphanumeric,
|
||||||
|
null,
|
||||||
|
".jpg",
|
||||||
|
allowDirect,
|
||||||
|
out var config,
|
||||||
|
id
|
||||||
|
);
|
||||||
|
return config!;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
if (Directory.Exists(_directory))
|
||||||
|
{
|
||||||
|
Directory.Delete(_directory, recursive: true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void An_absent_file_reads_as_an_empty_list()
|
||||||
|
{
|
||||||
|
using var store = Create();
|
||||||
|
|
||||||
|
store.List().ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Added_sources_survive_a_reload()
|
||||||
|
{
|
||||||
|
using (var store = Create())
|
||||||
|
{
|
||||||
|
await store.AddAsync(Config("s1", "Kept"), TestContext.Current.CancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
// A brand-new instance reads from disk rather than from the in-memory cache.
|
||||||
|
using var reopened = Create();
|
||||||
|
|
||||||
|
var config = reopened.List().ShouldHaveSingleItem();
|
||||||
|
config.Id.ShouldBe("s1");
|
||||||
|
config.Name.ShouldBe("Kept");
|
||||||
|
config.BaseUrl.AbsoluteUri.ShouldBe("https://imgtest.example/test1/");
|
||||||
|
config.Extension.ShouldBe(".jpg");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Updating_replaces_the_matching_config()
|
||||||
|
{
|
||||||
|
using var store = Create();
|
||||||
|
await store.AddAsync(Config("s1", "Before"), TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
var updated = Config("s1", "After");
|
||||||
|
(await store.UpdateAsync(updated, TestContext.Current.CancellationToken)).ShouldBeTrue();
|
||||||
|
|
||||||
|
store.List().ShouldHaveSingleItem().Name.ShouldBe("After");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Updating_an_unknown_id_changes_nothing()
|
||||||
|
{
|
||||||
|
using var store = Create();
|
||||||
|
|
||||||
|
(await store.UpdateAsync(Config("ghost"), TestContext.Current.CancellationToken)).ShouldBeFalse();
|
||||||
|
store.List().ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Removing_takes_the_config_out()
|
||||||
|
{
|
||||||
|
using var store = Create();
|
||||||
|
await store.AddAsync(Config("s1"), TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
(await store.RemoveAsync("s1", TestContext.Current.CancellationToken)).ShouldBeTrue();
|
||||||
|
store.List().ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Mutations_raise_the_changed_event()
|
||||||
|
{
|
||||||
|
using var store = Create();
|
||||||
|
var changed = 0;
|
||||||
|
store.Changed += (_, _) => changed++;
|
||||||
|
|
||||||
|
await store.AddAsync(Config("s1"), TestContext.Current.CancellationToken);
|
||||||
|
await store.RemoveAsync("s1", TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
changed.ShouldBe(2);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task A_corrupt_file_reads_as_an_empty_list()
|
||||||
|
{
|
||||||
|
Directory.CreateDirectory(_directory);
|
||||||
|
IAppPaths paths = new AppPaths(_directory);
|
||||||
|
await File.WriteAllTextAsync(paths.UserSourcesFile, "{ not json ]", TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
using var store = Create();
|
||||||
|
|
||||||
|
store.List().ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -156,10 +156,13 @@ public sealed class JsonSettingsServiceTests : IDisposable
|
|||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void The_proxy_gate_setting_reaches_the_collector()
|
public void The_collector_defaults_to_proxy_only()
|
||||||
{
|
{
|
||||||
new AppSettings(AllowDirectConnection: false).ToCollectOptions().RequireProxy.ShouldBeTrue();
|
// Whether a source may go direct is that source's own setting now; what the app-wide
|
||||||
new AppSettings(AllowDirectConnection: true).ToCollectOptions().RequireProxy.ShouldBeFalse();
|
// options must never do is default to the permissive answer.
|
||||||
|
new AppSettings()
|
||||||
|
.ToCollectOptions()
|
||||||
|
.RequireProxy.ShouldBeTrue();
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
|
|||||||
@@ -92,6 +92,58 @@ public class ProxyPoolLoaderTests
|
|||||||
(await loader.EnsureLoadedAsync()).Total.ShouldBe(0);
|
(await loader.EnsureLoadedAsync()).Total.ShouldBe(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task What_the_warm_up_skipped_is_checked_in_the_background()
|
||||||
|
{
|
||||||
|
// The warm-up stops at the target, which on a real feed leaves thousands unknown. Without
|
||||||
|
// this pass the pool reports ten live out of a few thousand and the rest were never asked.
|
||||||
|
var source = new CountingSource();
|
||||||
|
var pool = new ProxyPool(
|
||||||
|
[source],
|
||||||
|
new AliveProbe(),
|
||||||
|
new ProxyOptions { MinimumLiveProxies = 1, ProbeConcurrency = 1 }
|
||||||
|
);
|
||||||
|
using var loader = new ProxyPoolLoader(pool, new MemoryStateStore(), NullLogger<ProxyPoolLoader>.Instance);
|
||||||
|
|
||||||
|
var result = await loader.EnsureLoadedAsync();
|
||||||
|
|
||||||
|
// One live was enough to finish starting up; the other is still unknown at this point.
|
||||||
|
result.Live.ShouldBe(1);
|
||||||
|
|
||||||
|
await loader.TopUp.ShouldNotBeNull();
|
||||||
|
|
||||||
|
pool.LiveCount.ShouldBe(2);
|
||||||
|
loader.IsToppingUp.ShouldBeFalse();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task The_background_check_is_skipped_when_the_user_asked_for_lazy_probing()
|
||||||
|
{
|
||||||
|
// Lazy means "check a proxy when you hand it out"; sweeping the list behind the user's back
|
||||||
|
// is exactly what they switched off.
|
||||||
|
var pool = new ProxyPool(
|
||||||
|
[new CountingSource()],
|
||||||
|
new AliveProbe(),
|
||||||
|
new ProxyOptions { HealthCheck = ProxyHealthCheck.Lazy }
|
||||||
|
);
|
||||||
|
using var loader = new ProxyPoolLoader(pool, new MemoryStateStore(), NullLogger<ProxyPoolLoader>.Instance);
|
||||||
|
|
||||||
|
await loader.EnsureLoadedAsync();
|
||||||
|
|
||||||
|
loader.TopUp.ShouldBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Disposing_twice_is_safe()
|
||||||
|
{
|
||||||
|
var loader = Build(out _, out _);
|
||||||
|
await loader.EnsureLoadedAsync();
|
||||||
|
|
||||||
|
loader.Dispose();
|
||||||
|
|
||||||
|
Should.NotThrow(loader.Dispose);
|
||||||
|
}
|
||||||
|
|
||||||
private sealed class MemoryStateStore : IProxyStateStore
|
private sealed class MemoryStateStore : IProxyStateStore
|
||||||
{
|
{
|
||||||
public Dictionary<string, ProxyStateRecord> State { get; } = new(StringComparer.Ordinal);
|
public Dictionary<string, ProxyStateRecord> State { get; } = new(StringComparer.Ordinal);
|
||||||
@@ -150,4 +202,13 @@ public class ProxyPoolLoaderTests
|
|||||||
CancellationToken cancellationToken = default
|
CancellationToken cancellationToken = default
|
||||||
) => Task.FromResult(ProxyProbeResult.Failure("not used"));
|
) => Task.FromResult(ProxyProbeResult.Failure("not used"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private sealed class AliveProbe : IProxyProbe
|
||||||
|
{
|
||||||
|
public Task<ProxyProbeResult> ProbeAsync(
|
||||||
|
ProxyEndpoint endpoint,
|
||||||
|
ProxyOptions options,
|
||||||
|
CancellationToken cancellationToken = default
|
||||||
|
) => Task.FromResult(ProxyProbeResult.Success(TimeSpan.FromMilliseconds(15)));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,41 +1,63 @@
|
|||||||
using System.Runtime.CompilerServices;
|
|
||||||
using Avalonia.Controls;
|
using Avalonia.Controls;
|
||||||
using Avalonia.Headless.XUnit;
|
using Avalonia.Headless.XUnit;
|
||||||
using Avalonia.Threading;
|
using Avalonia.Threading;
|
||||||
using Avalonia.VisualTree;
|
using Avalonia.VisualTree;
|
||||||
using AvParser.Core.Collecting;
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
using AvParser.Core.Parsing;
|
using AvParser.Core.Parsing;
|
||||||
using AvParser.Core.Proxies;
|
using AvParser.Core.Proxies;
|
||||||
using AvParser.Core.Settings;
|
using AvParser.Core.Settings;
|
||||||
using AvParser.UI.ViewModels;
|
using AvParser.UI.ViewModels;
|
||||||
using AvParser.UI.Views;
|
using AvParser.UI.Views;
|
||||||
using Microsoft.Extensions.Logging.Abstractions;
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using ReactiveUI.Primitives;
|
||||||
using ReactiveUI.Primitives.Concurrency;
|
using ReactiveUI.Primitives.Concurrency;
|
||||||
|
|
||||||
namespace AvParser.UI.HeadlessTests;
|
namespace AvParser.UI.HeadlessTests;
|
||||||
|
|
||||||
public class CollectViewTests
|
public class CollectViewTests
|
||||||
{
|
{
|
||||||
private sealed class StubSource(string id, string name, bool network) : IMediaSource
|
private sealed class FakeUserSourceStore(IEnumerable<PatternSourceConfig>? seed = null) : IUserSourceStore
|
||||||
{
|
{
|
||||||
public string Id => id;
|
private readonly List<PatternSourceConfig> _configs = seed?.ToList() ?? [];
|
||||||
|
|
||||||
public string DisplayName => name;
|
public event EventHandler? Changed;
|
||||||
|
|
||||||
public string Description => "A source";
|
public IReadOnlyList<PatternSourceConfig> List() => [.. _configs];
|
||||||
|
|
||||||
public bool RequiresNetwork => network;
|
public Task<PatternSourceConfig> AddAsync(
|
||||||
|
PatternSourceConfig config,
|
||||||
public bool CanParse(MediaQuery input) => true;
|
CancellationToken cancellationToken = default
|
||||||
|
|
||||||
public async IAsyncEnumerable<ParseOutcome<MediaCandidate>> ParseAsync(
|
|
||||||
MediaQuery input,
|
|
||||||
IProgress<ParseProgress>? progress,
|
|
||||||
[EnumeratorCancellation] CancellationToken cancellationToken
|
|
||||||
)
|
)
|
||||||
{
|
{
|
||||||
await Task.Yield();
|
_configs.RemoveAll(c => c.Id == config.Id);
|
||||||
yield break;
|
_configs.Add(config);
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return Task.FromResult(config);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<bool> UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
var index = _configs.FindIndex(c => c.Id == config.Id);
|
||||||
|
if (index < 0)
|
||||||
|
{
|
||||||
|
return Task.FromResult(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
_configs[index] = config;
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return Task.FromResult(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<bool> RemoveAsync(string id, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
var removed = _configs.RemoveAll(c => c.Id == id) > 0;
|
||||||
|
if (removed)
|
||||||
|
{
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
}
|
||||||
|
|
||||||
|
return Task.FromResult(removed);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -46,7 +68,7 @@ public class CollectViewTests
|
|||||||
MediaQuery query,
|
MediaQuery query,
|
||||||
CollectOptions options,
|
CollectOptions options,
|
||||||
IProgress<ParseProgress>? progress,
|
IProgress<ParseProgress>? progress,
|
||||||
[EnumeratorCancellation] CancellationToken cancellationToken
|
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken
|
||||||
)
|
)
|
||||||
{
|
{
|
||||||
await Task.Yield();
|
await Task.Yield();
|
||||||
@@ -59,15 +81,34 @@ public class CollectViewTests
|
|||||||
public object? GetService(Type serviceType) => null;
|
public object? GetService(Type serviceType) => null;
|
||||||
}
|
}
|
||||||
|
|
||||||
private static (CollectView View, CollectViewModel ViewModel, Window Window) ShowPage(bool networkSource)
|
private static PatternSourceConfig Config(string id = "s1", bool allowDirect = false)
|
||||||
{
|
{
|
||||||
IMediaSource[] sources = networkSource
|
PatternSourceConfig.TryCreate(
|
||||||
? [new StubSource("url-list", "URL list", false), new StubSource("own-service", "Own service", true)]
|
"Test",
|
||||||
: [new StubSource("url-list", "URL list", false)];
|
"https://imgtest.example/test1/",
|
||||||
|
6,
|
||||||
|
8,
|
||||||
|
IdAlphabet.Alphanumeric,
|
||||||
|
null,
|
||||||
|
".jpg",
|
||||||
|
allowDirect,
|
||||||
|
out var config,
|
||||||
|
id
|
||||||
|
);
|
||||||
|
return config!;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static (CollectView View, CollectViewModel ViewModel, Window Window) ShowPage(
|
||||||
|
bool allowDirect,
|
||||||
|
bool withSource = true
|
||||||
|
)
|
||||||
|
{
|
||||||
|
// Whether a run may go without a proxy is the source's setting now, not the app's.
|
||||||
|
var store = new FakeUserSourceStore(withSource ? [Config(allowDirect: allowDirect)] : []);
|
||||||
|
|
||||||
var viewModel = new CollectViewModel(
|
var viewModel = new CollectViewModel(
|
||||||
new MediaSourceCatalog(sources, networkSource ? "own-service" : "url-list"),
|
new MediaSourceCatalog(store),
|
||||||
new FakeSettingsService(new AppSettings { LastSourceId = networkSource ? "own-service" : "url-list" }),
|
new FakeSettingsService(new AppSettings { LastSourceId = "s1" }),
|
||||||
new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
|
new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
|
||||||
new IdleRunner(),
|
new IdleRunner(),
|
||||||
new FakeMediaStore(),
|
new FakeMediaStore(),
|
||||||
@@ -96,26 +137,26 @@ public class CollectViewTests
|
|||||||
[AvaloniaFact]
|
[AvaloniaFact]
|
||||||
public void The_page_renders()
|
public void The_page_renders()
|
||||||
{
|
{
|
||||||
var (view, _, _) = ShowPage(networkSource: false);
|
var (view, _, _) = ShowPage(allowDirect: true);
|
||||||
|
|
||||||
view.GetVisualDescendants().OfType<ListBox>().ShouldNotBeEmpty();
|
view.GetVisualDescendants().OfType<ListBox>().ShouldNotBeEmpty();
|
||||||
}
|
}
|
||||||
|
|
||||||
[AvaloniaFact]
|
[AvaloniaFact]
|
||||||
public void No_banner_is_shown_for_a_source_that_needs_no_network()
|
public void No_banner_is_shown_when_direct_connections_are_allowed()
|
||||||
{
|
{
|
||||||
var (view, viewModel, _) = ShowPage(networkSource: false);
|
var (view, viewModel, _) = ShowPage(allowDirect: true);
|
||||||
|
|
||||||
viewModel.IsBlockedWithoutProxy.ShouldBeFalse();
|
viewModel.IsBlockedWithoutProxy.ShouldBeFalse();
|
||||||
Banner(view).IsVisible.ShouldBeFalse();
|
Banner(view).IsVisible.ShouldBeFalse();
|
||||||
}
|
}
|
||||||
|
|
||||||
[AvaloniaFact]
|
[AvaloniaFact]
|
||||||
public void A_blocked_network_source_puts_the_banner_on_screen()
|
public void A_blocked_source_puts_the_banner_on_screen()
|
||||||
{
|
{
|
||||||
// Rendered rather than asserted on the view model: an IsVisible binding that never fires
|
// Rendered rather than asserted on the view model: an IsVisible binding that never fires
|
||||||
// leaves the page silently unhelpful, which is exactly the failure this guards.
|
// leaves the page silently unhelpful, which is exactly the failure this guards.
|
||||||
var (view, viewModel, _) = ShowPage(networkSource: true);
|
var (view, viewModel, _) = ShowPage(allowDirect: false);
|
||||||
Dispatcher.UIThread.RunJobs();
|
Dispatcher.UIThread.RunJobs();
|
||||||
|
|
||||||
viewModel.IsBlockedWithoutProxy.ShouldBeTrue();
|
viewModel.IsBlockedWithoutProxy.ShouldBeTrue();
|
||||||
@@ -125,7 +166,7 @@ public class CollectViewTests
|
|||||||
[AvaloniaFact]
|
[AvaloniaFact]
|
||||||
public void The_banner_offers_a_way_to_the_proxies_page()
|
public void The_banner_offers_a_way_to_the_proxies_page()
|
||||||
{
|
{
|
||||||
var (view, viewModel, _) = ShowPage(networkSource: true);
|
var (view, viewModel, _) = ShowPage(allowDirect: false);
|
||||||
Dispatcher.UIThread.RunJobs();
|
Dispatcher.UIThread.RunJobs();
|
||||||
|
|
||||||
var button = Banner(view).GetVisualDescendants().OfType<Button>().ShouldHaveSingleItem();
|
var button = Banner(view).GetVisualDescendants().OfType<Button>().ShouldHaveSingleItem();
|
||||||
@@ -136,8 +177,7 @@ public class CollectViewTests
|
|||||||
[AvaloniaFact]
|
[AvaloniaFact]
|
||||||
public void A_blocked_page_will_not_run_the_collector()
|
public void A_blocked_page_will_not_run_the_collector()
|
||||||
{
|
{
|
||||||
var (view, viewModel, _) = ShowPage(networkSource: true);
|
var (view, viewModel, _) = ShowPage(allowDirect: false);
|
||||||
viewModel.EndpointText = "https://own.test/api/list";
|
|
||||||
Dispatcher.UIThread.RunJobs();
|
Dispatcher.UIThread.RunJobs();
|
||||||
|
|
||||||
var run = view.GetVisualDescendants()
|
var run = view.GetVisualDescendants()
|
||||||
@@ -148,26 +188,65 @@ public class CollectViewTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
[AvaloniaFact]
|
[AvaloniaFact]
|
||||||
public void An_endpoint_source_shows_an_address_box_rather_than_a_paste_box()
|
public void With_no_sources_the_empty_state_is_shown()
|
||||||
{
|
{
|
||||||
var (view, _, _) = ShowPage(networkSource: true);
|
var (view, _, _) = ShowPage(allowDirect: true, withSource: false);
|
||||||
Dispatcher.UIThread.RunJobs();
|
Dispatcher.UIThread.RunJobs();
|
||||||
|
|
||||||
var boxes = view.GetVisualDescendants().OfType<TextBox>().Where(box => box.IsEffectivelyVisible).ToList();
|
var empty = view.FindControl<Border>("EmptyState").ShouldNotBeNull();
|
||||||
|
empty.IsEffectivelyVisible.ShouldBeTrue();
|
||||||
boxes.ShouldHaveSingleItem();
|
|
||||||
boxes[0].PlaceholderText.ShouldNotBeNull().ShouldContain("api/list");
|
|
||||||
}
|
}
|
||||||
|
|
||||||
[AvaloniaFact]
|
[AvaloniaFact]
|
||||||
public void A_pasted_list_source_shows_the_paste_box()
|
public void The_log_panel_appears_as_soon_as_there_is_something_to_show()
|
||||||
{
|
{
|
||||||
var (view, _, _) = ShowPage(networkSource: false);
|
// Rendered rather than asserted on the view model: a log nobody can see is the whole failure
|
||||||
|
// this guards, and an IsVisible binding that never fires breaks no view-model test.
|
||||||
|
var (view, viewModel, _) = ShowPage(allowDirect: true);
|
||||||
|
|
||||||
|
var panel = view.FindControl<Border>("LogPanel").ShouldNotBeNull();
|
||||||
|
panel.IsVisible.ShouldBeFalse();
|
||||||
|
|
||||||
|
viewModel.Log.Add(CollectLogEntryViewModel.Message(CollectLogLevel.Info, "Collect.Log.SourceStarted", "Test"));
|
||||||
Dispatcher.UIThread.RunJobs();
|
Dispatcher.UIThread.RunJobs();
|
||||||
|
|
||||||
var boxes = view.GetVisualDescendants().OfType<TextBox>().Where(box => box.IsEffectivelyVisible).ToList();
|
panel.IsEffectivelyVisible.ShouldBeTrue();
|
||||||
|
view.FindControl<ListBox>("CollectLog").ShouldNotBeNull().ItemCount.ShouldBe(1);
|
||||||
|
|
||||||
boxes.ShouldHaveSingleItem();
|
// A log that grows without bound pushes the collected list off the page.
|
||||||
boxes[0].AcceptsReturn.ShouldBeTrue();
|
panel.Bounds.Height.ShouldBeLessThanOrEqualTo(220);
|
||||||
|
|
||||||
|
// Lines have to be selectable in bulk and copyable, which is the point of a log you can
|
||||||
|
// paste into a bug report.
|
||||||
|
var list = view.FindControl<ListBox>("CollectLog").ShouldNotBeNull();
|
||||||
|
list.SelectionMode.HasFlag(SelectionMode.Multiple).ShouldBeTrue();
|
||||||
|
list.ContextMenu.ShouldNotBeNull().Items.Count.ShouldBe(2);
|
||||||
|
}
|
||||||
|
|
||||||
|
[AvaloniaFact]
|
||||||
|
public void Every_source_is_offered_with_a_tick_box()
|
||||||
|
{
|
||||||
|
var (view, viewModel, _) = ShowPage(allowDirect: true);
|
||||||
|
|
||||||
|
var list = view.FindControl<ListBox>("SourceList").ShouldNotBeNull();
|
||||||
|
|
||||||
|
list.ItemCount.ShouldBe(1);
|
||||||
|
viewModel.RunnableSources().ShouldHaveSingleItem();
|
||||||
|
view.GetVisualDescendants().OfType<CheckBox>().ShouldContain(box => box.IsChecked == true);
|
||||||
|
|
||||||
|
// The picker sits in the toolbar: it scrolls, it does not grow with the catalog.
|
||||||
|
list.Bounds.Height.ShouldBeLessThanOrEqualTo(120);
|
||||||
|
}
|
||||||
|
|
||||||
|
[AvaloniaFact]
|
||||||
|
public void Adding_a_source_reveals_the_editor()
|
||||||
|
{
|
||||||
|
var (view, viewModel, _) = ShowPage(allowDirect: true, withSource: false);
|
||||||
|
|
||||||
|
viewModel.AddSourceCommand.Execute().Subscribe();
|
||||||
|
Dispatcher.UIThread.RunJobs();
|
||||||
|
|
||||||
|
var editor = view.FindControl<Border>("SourceEditor").ShouldNotBeNull();
|
||||||
|
editor.IsEffectivelyVisible.ShouldBeTrue();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -122,6 +122,8 @@ internal sealed class FakeProxyPoolLoader : IProxyPoolLoader
|
|||||||
{
|
{
|
||||||
public bool IsLoaded => true;
|
public bool IsLoaded => true;
|
||||||
|
|
||||||
|
public bool IsToppingUp => false;
|
||||||
|
|
||||||
public Task<ProxyPoolLoadResult> EnsureLoadedAsync() => Task.FromResult(new ProxyPoolLoadResult(0, 0, 0));
|
public Task<ProxyPoolLoadResult> EnsureLoadedAsync() => Task.FromResult(new ProxyPoolLoadResult(0, 0, 0));
|
||||||
|
|
||||||
public Task SaveStateAsync(CancellationToken cancellationToken = default) => Task.CompletedTask;
|
public Task SaveStateAsync(CancellationToken cancellationToken = default) => Task.CompletedTask;
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
using System.Runtime.CompilerServices;
|
using System.Runtime.CompilerServices;
|
||||||
using AvParser.Core.Collecting;
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Collecting.Sources;
|
||||||
using AvParser.Core.Parsing;
|
using AvParser.Core.Parsing;
|
||||||
using AvParser.Core.Proxies;
|
using AvParser.Core.Proxies;
|
||||||
using AvParser.Core.Settings;
|
using AvParser.Core.Settings;
|
||||||
@@ -16,9 +17,12 @@ public class CollectViewModelTests
|
|||||||
/// <summary>A runner that returns a canned stream instead of touching a network.</summary>
|
/// <summary>A runner that returns a canned stream instead of touching a network.</summary>
|
||||||
private sealed class FakeRunner : ICollectRunner
|
private sealed class FakeRunner : ICollectRunner
|
||||||
{
|
{
|
||||||
|
private readonly Lock _gate = new();
|
||||||
|
|
||||||
public List<ParseOutcome<CollectedItem>> Results { get; } = [];
|
public List<ParseOutcome<CollectedItem>> Results { get; } = [];
|
||||||
|
|
||||||
public int Runs { get; private set; }
|
/// <summary>Ids the runner was asked to run, one entry per call.</summary>
|
||||||
|
public List<string> RunSourceIds { get; } = [];
|
||||||
|
|
||||||
public CollectOptions? LastOptions { get; private set; }
|
public CollectOptions? LastOptions { get; private set; }
|
||||||
|
|
||||||
@@ -34,7 +38,11 @@ public class CollectViewModelTests
|
|||||||
[EnumeratorCancellation] CancellationToken cancellationToken
|
[EnumeratorCancellation] CancellationToken cancellationToken
|
||||||
)
|
)
|
||||||
{
|
{
|
||||||
Runs++;
|
lock (_gate)
|
||||||
|
{
|
||||||
|
RunSourceIds.Add(source.Id);
|
||||||
|
}
|
||||||
|
|
||||||
LastOptions = options;
|
LastOptions = options;
|
||||||
LastQuery = query;
|
LastQuery = query;
|
||||||
|
|
||||||
@@ -54,26 +62,48 @@ public class CollectViewModelTests
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private sealed class StubSource(string id, string name, bool network) : IMediaSource
|
/// <summary>In-memory source list standing in for the persisted store.</summary>
|
||||||
|
private sealed class FakeUserSourceStore(IEnumerable<PatternSourceConfig>? seed = null) : IUserSourceStore
|
||||||
{
|
{
|
||||||
public string Id => id;
|
private readonly List<PatternSourceConfig> _configs = seed?.ToList() ?? [];
|
||||||
|
|
||||||
public string DisplayName => name;
|
public event EventHandler? Changed;
|
||||||
|
|
||||||
public string Description => string.Empty;
|
public IReadOnlyList<PatternSourceConfig> List() => [.. _configs];
|
||||||
|
|
||||||
public bool RequiresNetwork => network;
|
public Task<PatternSourceConfig> AddAsync(
|
||||||
|
PatternSourceConfig config,
|
||||||
public bool CanParse(MediaQuery input) => true;
|
CancellationToken cancellationToken = default
|
||||||
|
|
||||||
public async IAsyncEnumerable<ParseOutcome<MediaCandidate>> ParseAsync(
|
|
||||||
MediaQuery input,
|
|
||||||
IProgress<ParseProgress>? progress,
|
|
||||||
[EnumeratorCancellation] CancellationToken cancellationToken
|
|
||||||
)
|
)
|
||||||
{
|
{
|
||||||
await Task.Yield();
|
_configs.RemoveAll(c => c.Id == config.Id);
|
||||||
yield break;
|
_configs.Add(config);
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return Task.FromResult(config);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<bool> UpdateAsync(PatternSourceConfig config, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
var index = _configs.FindIndex(c => c.Id == config.Id);
|
||||||
|
if (index < 0)
|
||||||
|
{
|
||||||
|
return Task.FromResult(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
_configs[index] = config;
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
return Task.FromResult(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task<bool> RemoveAsync(string id, CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
var removed = _configs.RemoveAll(c => c.Id == id) > 0;
|
||||||
|
if (removed)
|
||||||
|
{
|
||||||
|
Changed?.Invoke(this, EventArgs.Empty);
|
||||||
|
}
|
||||||
|
|
||||||
|
return Task.FromResult(removed);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -82,9 +112,36 @@ public class CollectViewModelTests
|
|||||||
public object? GetService(Type serviceType) => null;
|
public object? GetService(Type serviceType) => null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A source that may run without a proxy, which is what most of these tests need: the gate is a
|
||||||
|
/// per-source setting now, so a config that requires one would block every run.
|
||||||
|
/// </summary>
|
||||||
|
private static PatternSourceConfig Config(
|
||||||
|
string id,
|
||||||
|
string name = "Test",
|
||||||
|
string url = "https://imgtest.example/test1/",
|
||||||
|
bool allowDirect = true
|
||||||
|
)
|
||||||
|
{
|
||||||
|
PatternSourceConfig.TryCreate(
|
||||||
|
name,
|
||||||
|
url,
|
||||||
|
6,
|
||||||
|
8,
|
||||||
|
IdAlphabet.Alphanumeric,
|
||||||
|
null,
|
||||||
|
".jpg",
|
||||||
|
allowDirect,
|
||||||
|
out var config,
|
||||||
|
id
|
||||||
|
);
|
||||||
|
|
||||||
|
return config!;
|
||||||
|
}
|
||||||
|
|
||||||
private static CollectedItem Item(string url, CollectStatus status, long length = 4096) =>
|
private static CollectedItem Item(string url, CollectStatus status, long length = 4096) =>
|
||||||
new(
|
new(
|
||||||
new MediaCandidate(new Uri(url)) { SourceId = "url-list", Ordinal = 1 },
|
new MediaCandidate(new Uri(url)) { SourceId = "s1", Ordinal = 1 },
|
||||||
MediaBlob.Create(new string('a', 64), MediaKind.Png, length),
|
MediaBlob.Create(new string('a', 64), MediaKind.Png, length),
|
||||||
status
|
status
|
||||||
);
|
);
|
||||||
@@ -92,24 +149,20 @@ public class CollectViewModelTests
|
|||||||
private static (CollectViewModel Page, FakeRunner Runner, FakeSettingsService Settings) Build(
|
private static (CollectViewModel Page, FakeRunner Runner, FakeSettingsService Settings) Build(
|
||||||
AppSettings? settings = null,
|
AppSettings? settings = null,
|
||||||
IProxyPool? proxyPool = null,
|
IProxyPool? proxyPool = null,
|
||||||
bool includeNetworkSource = false
|
IEnumerable<PatternSourceConfig>? configs = null
|
||||||
)
|
)
|
||||||
{
|
{
|
||||||
IMediaSource[] sources = includeNetworkSource
|
var store = new FakeUserSourceStore(configs ?? [Config("s1")]);
|
||||||
? [new StubSource("url-list", "URL list", false), new StubSource("own-service", "Own service", true)]
|
var catalog = new MediaSourceCatalog(store);
|
||||||
: [new StubSource("url-list", "URL list", false)];
|
var settingsService = new FakeSettingsService(settings ?? new AppSettings());
|
||||||
|
|
||||||
var catalog = new MediaSourceCatalog(sources, "url-list");
|
|
||||||
var settingsService = new FakeSettingsService(settings);
|
|
||||||
var runner = new FakeRunner();
|
var runner = new FakeRunner();
|
||||||
var store = new FakeMediaStore();
|
|
||||||
|
|
||||||
var page = new CollectViewModel(
|
var page = new CollectViewModel(
|
||||||
catalog,
|
catalog,
|
||||||
settingsService,
|
settingsService,
|
||||||
proxyPool ?? new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
|
proxyPool ?? new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
|
||||||
runner,
|
runner,
|
||||||
store,
|
new FakeMediaStore(),
|
||||||
new FakeThumbnailCache(),
|
new FakeThumbnailCache(),
|
||||||
new EmptyServiceProvider(),
|
new EmptyServiceProvider(),
|
||||||
NullLogger<CollectViewModel>.Instance,
|
NullLogger<CollectViewModel>.Instance,
|
||||||
@@ -122,79 +175,142 @@ public class CollectViewModelTests
|
|||||||
private static Task RunAsync(CollectViewModel page) => page.CollectCommand.Execute().ToTask();
|
private static Task RunAsync(CollectViewModel page) => page.CollectCommand.Execute().ToTask();
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void The_page_opens_on_the_source_that_needs_no_proxy()
|
public void An_empty_catalog_selects_nothing_and_offers_to_add_one()
|
||||||
{
|
{
|
||||||
// Otherwise the app lands behind the gate before the user has asked for anything.
|
var (page, _, _) = Build(configs: []);
|
||||||
var (page, _, _) = Build(includeNetworkSource: true);
|
|
||||||
|
|
||||||
page.SelectedSource.Id.ShouldBe("url-list");
|
page.SelectedSource.ShouldBeNull();
|
||||||
page.IsBlockedWithoutProxy.ShouldBeFalse();
|
page.HasSources.ShouldBeFalse();
|
||||||
|
|
||||||
|
var canExecute = true;
|
||||||
|
using var subscription = page.CollectCommand.CanExecute.Subscribe(value => canExecute = value);
|
||||||
|
canExecute.ShouldBeFalse();
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void The_last_used_source_is_restored()
|
public void The_last_used_source_is_restored()
|
||||||
{
|
{
|
||||||
var (page, _, _) = Build(
|
var (page, _, _) = Build(
|
||||||
new AppSettings { LastSourceId = "own-service", AllowDirectConnection = true },
|
new AppSettings { LastSourceId = "s2" },
|
||||||
includeNetworkSource: true
|
configs: [Config("s1", "Alpha"), Config("s2", "Beta")]
|
||||||
);
|
);
|
||||||
|
|
||||||
page.SelectedSource.Id.ShouldBe("own-service");
|
page.SelectedSource.ShouldNotBeNull().Id.ShouldBe("s2");
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void An_unknown_remembered_source_falls_back_instead_of_throwing()
|
public void An_unknown_remembered_source_falls_back_instead_of_throwing()
|
||||||
{
|
{
|
||||||
var (page, _, _) = Build(new AppSettings { LastSourceId = "removed-in-a-past-version" });
|
var (page, _, _) = Build(new AppSettings { LastSourceId = "gone" });
|
||||||
|
|
||||||
page.SelectedSource.Id.ShouldBe("url-list");
|
page.SelectedSource.ShouldNotBeNull().Id.ShouldBe("s1");
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Choosing_a_source_remembers_it()
|
public void Choosing_a_source_remembers_it()
|
||||||
{
|
{
|
||||||
var (page, _, settings) = Build(new AppSettings { AllowDirectConnection = true }, includeNetworkSource: true);
|
var (page, _, settings) = Build(configs: [Config("s1", "Alpha"), Config("s2", "Beta")]);
|
||||||
|
|
||||||
page.SelectedSource = page.Sources.Single(source => source.Id == "own-service");
|
page.SelectedSource = page.Sources.Single(source => source.Id == "s2");
|
||||||
|
|
||||||
settings.Current.LastSourceId.ShouldBe("own-service");
|
settings.Current.LastSourceId.ShouldBe("s2");
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Collecting_needs_something_to_collect()
|
public void Collecting_needs_at_least_one_ticked_source()
|
||||||
{
|
{
|
||||||
var (page, _, _) = Build();
|
var (page, _, _) = Build();
|
||||||
var canExecute = true;
|
var canExecute = false;
|
||||||
using var subscription = page.CollectCommand.CanExecute.Subscribe(value => canExecute = value);
|
using var subscription = page.CollectCommand.CanExecute.Subscribe(value => canExecute = value);
|
||||||
|
|
||||||
canExecute.ShouldBeFalse();
|
|
||||||
|
|
||||||
page.InputText = "https://example.test/a.png";
|
|
||||||
canExecute.ShouldBeTrue();
|
canExecute.ShouldBeTrue();
|
||||||
|
|
||||||
page.InputText = " ";
|
page.Sources.Single().IsSelected = false;
|
||||||
canExecute.ShouldBeFalse();
|
canExecute.ShouldBeFalse();
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void An_endpoint_source_wants_an_address_not_pasted_text()
|
public void An_unlimited_budget_does_not_block_the_button()
|
||||||
{
|
{
|
||||||
var (page, _, _) = Build(new AppSettings { AllowDirectConnection = true }, includeNetworkSource: true);
|
// Zero used to mean "nothing to do"; it now means "until stopped", which is a running state,
|
||||||
page.SelectedSource = page.Sources.Single(source => source.Id == "own-service");
|
// not a disabled one.
|
||||||
|
var (page, _, _) = Build();
|
||||||
var canExecute = true;
|
var canExecute = false;
|
||||||
using var subscription = page.CollectCommand.CanExecute.Subscribe(value => canExecute = value);
|
using var subscription = page.CollectCommand.CanExecute.Subscribe(value => canExecute = value);
|
||||||
|
|
||||||
page.InputText = "https://example.test/a.png";
|
page.AttemptBudget = 0;
|
||||||
canExecute.ShouldBeFalse();
|
page.TargetCount = 0;
|
||||||
|
|
||||||
page.EndpointText = "not an address";
|
|
||||||
canExecute.ShouldBeFalse();
|
|
||||||
|
|
||||||
page.EndpointText = "https://own.test/api/list";
|
|
||||||
canExecute.ShouldBeTrue();
|
canExecute.ShouldBeTrue();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task An_unlimited_budget_reaches_the_source_as_no_limit()
|
||||||
|
{
|
||||||
|
var (page, runner, settings) = Build();
|
||||||
|
page.AttemptBudget = 0;
|
||||||
|
|
||||||
|
await RunAsync(page);
|
||||||
|
|
||||||
|
runner.LastQuery!.HasLimit.ShouldBeFalse();
|
||||||
|
settings.Current.CollectAttemptBudget.ShouldBe(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Every_ticked_source_runs_in_the_same_collection()
|
||||||
|
{
|
||||||
|
var (page, runner, _) = Build(configs: [Config("s1", "Alpha"), Config("s2", "Beta")]);
|
||||||
|
foreach (var source in page.Sources)
|
||||||
|
{
|
||||||
|
source.IsSelected = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
runner.Results.Add(ParseOutcome<CollectedItem>.Success(Item("https://a.test/1.png", CollectStatus.Stored)));
|
||||||
|
|
||||||
|
await RunAsync(page);
|
||||||
|
|
||||||
|
runner.RunSourceIds.Order().ShouldBe(["s1", "s2"]);
|
||||||
|
page.Items.Count.ShouldBe(2);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void The_ticked_set_is_remembered()
|
||||||
|
{
|
||||||
|
var (page, _, settings) = Build(configs: [Config("s1", "Alpha"), Config("s2", "Beta")]);
|
||||||
|
|
||||||
|
page.Sources.Single(source => source.Id == "s2").IsSelected = true;
|
||||||
|
|
||||||
|
AppSettings.SplitSourceIds(settings.Current.CollectSourceIds).Order().ShouldBe(["s1", "s2"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void A_remembered_ticked_set_is_restored()
|
||||||
|
{
|
||||||
|
var (page, _, _) = Build(
|
||||||
|
new AppSettings { CollectSourceIds = "s2" },
|
||||||
|
configs: [Config("s1", "Alpha"), Config("s2", "Beta")]
|
||||||
|
);
|
||||||
|
|
||||||
|
page.RunnableSources().ShouldHaveSingleItem().Id.ShouldBe("s2");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task The_log_records_what_happened_as_it_happens()
|
||||||
|
{
|
||||||
|
var (page, runner, _) = Build();
|
||||||
|
runner.Results.AddRange([
|
||||||
|
ParseOutcome<CollectedItem>.Success(Item("https://a.test/1.png", CollectStatus.Stored)),
|
||||||
|
ParseOutcome<CollectedItem>.Failure(ParseError.Create(2, "TooLarge", "too big")),
|
||||||
|
]);
|
||||||
|
|
||||||
|
await RunAsync(page);
|
||||||
|
|
||||||
|
page.ErrorCount.ShouldBe(1);
|
||||||
|
page.Log.Select(entry => entry.Text).ShouldContain(text => text.Contains("https://a.test/1.png"));
|
||||||
|
page.Log.Select(entry => entry.Text).ShouldContain("Larger than the size limit.");
|
||||||
|
page.Log[^1].Text.ShouldStartWith("Run finished.");
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task Results_land_in_the_list_and_the_summary_counts_them()
|
public async Task Results_land_in_the_list_and_the_summary_counts_them()
|
||||||
{
|
{
|
||||||
@@ -205,11 +321,10 @@ public class CollectViewModelTests
|
|||||||
ParseOutcome<CollectedItem>.Success(Item("https://a.test/3.png", CollectStatus.Skipped)),
|
ParseOutcome<CollectedItem>.Success(Item("https://a.test/3.png", CollectStatus.Skipped)),
|
||||||
]);
|
]);
|
||||||
|
|
||||||
page.InputText = "https://a.test/1.png";
|
|
||||||
await RunAsync(page);
|
await RunAsync(page);
|
||||||
|
|
||||||
page.Items.Count.ShouldBe(3);
|
page.Items.Count.ShouldBe(3);
|
||||||
page.Errors.ShouldBeEmpty();
|
page.ErrorCount.ShouldBe(0);
|
||||||
var summary = page.StatusMessage.ShouldNotBeNull();
|
var summary = page.StatusMessage.ShouldNotBeNull();
|
||||||
summary.ShouldContain("1 image");
|
summary.ShouldContain("1 image");
|
||||||
summary.ShouldContain("already held");
|
summary.ShouldContain("already held");
|
||||||
@@ -225,11 +340,10 @@ public class CollectViewModelTests
|
|||||||
ParseOutcome<CollectedItem>.Failure(ParseError.Create(2, "TooLarge", "too big")),
|
ParseOutcome<CollectedItem>.Failure(ParseError.Create(2, "TooLarge", "too big")),
|
||||||
]);
|
]);
|
||||||
|
|
||||||
page.InputText = "https://a.test/1.png";
|
|
||||||
await RunAsync(page);
|
await RunAsync(page);
|
||||||
|
|
||||||
page.Items.ShouldHaveSingleItem();
|
page.Items.ShouldHaveSingleItem();
|
||||||
page.Errors.ShouldHaveSingleItem().Text.ShouldBe("Larger than the size limit.");
|
page.Log.Single(entry => entry.IsError).Text.ShouldBe("Larger than the size limit.");
|
||||||
page.StatusMessage!.ShouldContain("1 error");
|
page.StatusMessage!.ShouldContain("1 error");
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -237,7 +351,6 @@ public class CollectViewModelTests
|
|||||||
public async Task The_force_refetch_switch_reaches_the_runner()
|
public async Task The_force_refetch_switch_reaches_the_runner()
|
||||||
{
|
{
|
||||||
var (page, runner, _) = Build();
|
var (page, runner, _) = Build();
|
||||||
page.InputText = "https://a.test/1.png";
|
|
||||||
page.ForceRefetch = true;
|
page.ForceRefetch = true;
|
||||||
|
|
||||||
await RunAsync(page);
|
await RunAsync(page);
|
||||||
@@ -246,15 +359,33 @@ public class CollectViewModelTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task The_pasted_text_reaches_the_query()
|
public async Task The_attempt_budget_becomes_the_query_limit()
|
||||||
{
|
{
|
||||||
var (page, runner, _) = Build();
|
var (page, runner, _) = Build();
|
||||||
page.InputText = "https://a.test/1.png\nhttps://a.test/2.png";
|
page.AttemptBudget = 250;
|
||||||
|
|
||||||
await RunAsync(page);
|
await RunAsync(page);
|
||||||
|
|
||||||
runner.LastQuery!.Text.ShouldContain("2.png");
|
runner.LastQuery!.Limit.ShouldBe(250);
|
||||||
runner.LastQuery.Endpoint.ShouldBeNull();
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Collecting_stops_once_the_target_is_reached()
|
||||||
|
{
|
||||||
|
var (page, runner, _) = Build();
|
||||||
|
for (var i = 0; i < 5; i++)
|
||||||
|
{
|
||||||
|
runner.Results.Add(
|
||||||
|
ParseOutcome<CollectedItem>.Success(Item($"https://a.test/{i}.png", CollectStatus.Stored))
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
page.TargetCount = 2;
|
||||||
|
await RunAsync(page);
|
||||||
|
|
||||||
|
page.Items.Count.ShouldBe(2);
|
||||||
|
// Reaching the target is a clean finish, not a stop: the summary must not read "Stopped".
|
||||||
|
page.StatusMessage.ShouldNotBeNull().ShouldNotContain("Stopped");
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
@@ -262,7 +393,6 @@ public class CollectViewModelTests
|
|||||||
{
|
{
|
||||||
var (page, runner, _) = Build();
|
var (page, runner, _) = Build();
|
||||||
runner.Results.Add(ParseOutcome<CollectedItem>.Success(Item("https://a.test/1.png", CollectStatus.Stored)));
|
runner.Results.Add(ParseOutcome<CollectedItem>.Success(Item("https://a.test/1.png", CollectStatus.Stored)));
|
||||||
page.InputText = "https://a.test/1.png";
|
|
||||||
|
|
||||||
await RunAsync(page);
|
await RunAsync(page);
|
||||||
await RunAsync(page);
|
await RunAsync(page);
|
||||||
@@ -270,39 +400,52 @@ public class CollectViewModelTests
|
|||||||
page.Items.ShouldHaveSingleItem();
|
page.Items.ShouldHaveSingleItem();
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void A_local_source_runs_with_no_proxy_at_all()
|
|
||||||
{
|
|
||||||
var (page, _, _) = Build();
|
|
||||||
|
|
||||||
page.IsBlockedWithoutProxy.ShouldBeFalse();
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void A_network_source_is_blocked_while_nothing_is_live()
|
public void A_network_source_is_blocked_while_nothing_is_live()
|
||||||
{
|
{
|
||||||
var (page, _, _) = Build(new AppSettings { LastSourceId = "own-service" }, includeNetworkSource: true);
|
var (page, _, _) = Build(configs: [Config("s1", allowDirect: false)]);
|
||||||
|
|
||||||
page.IsBlockedWithoutProxy.ShouldBeTrue();
|
page.IsBlockedWithoutProxy.ShouldBeTrue();
|
||||||
|
|
||||||
var canExecute = true;
|
var canExecute = true;
|
||||||
using var subscription = page.CollectCommand.CanExecute.Subscribe(value => canExecute = value);
|
using var subscription = page.CollectCommand.CanExecute.Subscribe(value => canExecute = value);
|
||||||
page.EndpointText = "https://own.test/api/list";
|
|
||||||
|
|
||||||
canExecute.ShouldBeFalse();
|
canExecute.ShouldBeFalse();
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Allowing_direct_connections_lifts_the_gate()
|
public void A_source_allowed_to_go_direct_lifts_the_gate_for_itself()
|
||||||
{
|
{
|
||||||
var (page, _, _) = Build(
|
var (page, _, _) = Build(configs: [Config("s1", allowDirect: true)]);
|
||||||
new AppSettings { LastSourceId = "own-service", AllowDirectConnection = true },
|
|
||||||
includeNetworkSource: true
|
|
||||||
);
|
|
||||||
|
|
||||||
page.IsBlockedWithoutProxy.ShouldBeFalse();
|
page.IsBlockedWithoutProxy.ShouldBeFalse();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void One_gated_source_in_the_run_blocks_the_whole_run()
|
||||||
|
{
|
||||||
|
// The permissive source cannot vouch for the strict one: the request the user did not want
|
||||||
|
// leaving their own address would leave it anyway.
|
||||||
|
var (page, _, _) = Build(
|
||||||
|
configs: [Config("s1", "Open", allowDirect: true), Config("s2", "Strict", allowDirect: false)]
|
||||||
|
);
|
||||||
|
|
||||||
|
page.Sources.Single(source => source.Id == "s2").IsSelected = true;
|
||||||
|
|
||||||
|
page.IsBlockedWithoutProxy.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Each_source_carries_its_own_proxy_policy_into_the_run()
|
||||||
|
{
|
||||||
|
var (page, runner, _) = Build(configs: [Config("s1", allowDirect: true)]);
|
||||||
|
|
||||||
|
await RunAsync(page);
|
||||||
|
|
||||||
|
// The gate is only half of it: the fetcher has to be told too, or a source that should be
|
||||||
|
// blocked would simply go direct with nobody the wiser.
|
||||||
|
runner.LastOptions.ShouldNotBeNull().RequireProxy.ShouldBeFalse();
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task A_network_source_runs_once_a_proxy_answers()
|
public async Task A_network_source_runs_once_a_proxy_answers()
|
||||||
{
|
{
|
||||||
@@ -313,7 +456,7 @@ public class CollectViewModelTests
|
|||||||
new ProxyOptions()
|
new ProxyOptions()
|
||||||
);
|
);
|
||||||
|
|
||||||
var (page, _, _) = Build(new AppSettings { LastSourceId = "own-service" }, pool, includeNetworkSource: true);
|
var (page, _, _) = Build(proxyPool: pool, configs: [Config("s1", allowDirect: false)]);
|
||||||
page.IsBlockedWithoutProxy.ShouldBeTrue();
|
page.IsBlockedWithoutProxy.ShouldBeTrue();
|
||||||
|
|
||||||
await pool.RefreshAsync(TestContext.Current.CancellationToken);
|
await pool.RefreshAsync(TestContext.Current.CancellationToken);
|
||||||
@@ -324,21 +467,68 @@ public class CollectViewModelTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Switching_away_from_a_network_source_lifts_the_gate()
|
public async Task Adding_a_source_stores_it_and_selects_it()
|
||||||
{
|
{
|
||||||
var (page, _, _) = Build(new AppSettings { LastSourceId = "own-service" }, includeNetworkSource: true);
|
var (page, _, _) = Build(configs: []);
|
||||||
page.IsBlockedWithoutProxy.ShouldBeTrue();
|
|
||||||
|
|
||||||
page.SelectedSource = page.Sources.Single(source => source.Id == "url-list");
|
page.AddSourceCommand.Execute().Subscribe();
|
||||||
|
page.EditorName = "New one";
|
||||||
|
page.EditorBaseUrl = "https://imgtest.example/test2/";
|
||||||
|
page.EditorMinLength = 8;
|
||||||
|
page.EditorMaxLength = 12;
|
||||||
|
page.EditorAlphabet = page.AlphabetOptions.Single(o => o.Value == IdAlphabet.Digits);
|
||||||
|
|
||||||
page.IsBlockedWithoutProxy.ShouldBeFalse();
|
await page.SaveSourceCommand.Execute().ToTask(TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
page.IsEditorOpen.ShouldBeFalse();
|
||||||
|
page.Sources.ShouldHaveSingleItem();
|
||||||
|
page.SelectedSource.ShouldNotBeNull().Name.ShouldBe("New one");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task An_invalid_source_reports_an_error_and_is_not_added()
|
||||||
|
{
|
||||||
|
var (page, _, _) = Build(configs: []);
|
||||||
|
|
||||||
|
page.AddSourceCommand.Execute().Subscribe();
|
||||||
|
page.EditorName = "Bad";
|
||||||
|
page.EditorBaseUrl = "not a url";
|
||||||
|
|
||||||
|
await page.SaveSourceCommand.Execute().ToTask(TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
page.EditorError.ShouldNotBeNull();
|
||||||
|
page.IsEditorOpen.ShouldBeTrue();
|
||||||
|
page.Sources.ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Editing_a_source_updates_it_in_place()
|
||||||
|
{
|
||||||
|
var (page, _, _) = Build(configs: [Config("s1", "Before")]);
|
||||||
|
|
||||||
|
page.EditSourceCommand.Execute().Subscribe();
|
||||||
|
page.EditorName = "After";
|
||||||
|
|
||||||
|
await page.SaveSourceCommand.Execute().ToTask(TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
page.Sources.ShouldHaveSingleItem().Name.ShouldBe("After");
|
||||||
|
page.SelectedSource.ShouldNotBeNull().Id.ShouldBe("s1");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Removing_a_source_drops_it()
|
||||||
|
{
|
||||||
|
var (page, _, _) = Build(configs: [Config("s1", "Alpha"), Config("s2", "Beta")]);
|
||||||
|
page.SelectedSource = page.Sources.Single(source => source.Id == "s2");
|
||||||
|
|
||||||
|
await page.RemoveSourceCommand.Execute().ToTask(TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
page.Sources.ShouldHaveSingleItem().Id.ShouldBe("s1");
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public async Task Purging_removes_only_the_selected_source()
|
public async Task Purging_removes_only_the_selected_source()
|
||||||
{
|
{
|
||||||
// Scoped rather than emptying the store: content another source also holds must survive,
|
|
||||||
// which is exactly what the index's reference count is for.
|
|
||||||
var (page, _, _) = Build();
|
var (page, _, _) = Build();
|
||||||
|
|
||||||
await page.PurgeCommand.Execute().ToTask(TestContext.Current.CancellationToken);
|
await page.PurgeCommand.Execute().ToTask(TestContext.Current.CancellationToken);
|
||||||
@@ -354,6 +544,35 @@ public class CollectViewModelTests
|
|||||||
page.StorageSummary.ShouldNotBeNull().ShouldContain("in the store");
|
page.StorageSummary.ShouldNotBeNull().ShouldContain("in the store");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Log_lines_copy_as_the_text_that_is_on_screen()
|
||||||
|
{
|
||||||
|
var (page, runner, _) = Build();
|
||||||
|
runner.Results.Add(ParseOutcome<CollectedItem>.Success(Item("https://a.test/1.png", CollectStatus.Stored)));
|
||||||
|
|
||||||
|
await RunAsync(page);
|
||||||
|
|
||||||
|
var line = page.Log.First(entry => entry.IsSuccess).ToString();
|
||||||
|
|
||||||
|
line.ShouldContain("https://a.test/1.png");
|
||||||
|
line.ShouldContain("Test"); // the source name, as shown in the chip
|
||||||
|
line.ShouldStartWith(page.Log[0].TimeText[..2]); // a timestamp, not a bare message
|
||||||
|
|
||||||
|
var text = CollectLogEntryViewModel.ToText(page.Log);
|
||||||
|
text.Split(Environment.NewLine).Length.ShouldBe(page.Log.Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Disposing_twice_is_safe()
|
||||||
|
{
|
||||||
|
// The container disposes each page once per registration — its own type and PageViewModel.
|
||||||
|
var (page, _, _) = Build();
|
||||||
|
|
||||||
|
page.Dispose();
|
||||||
|
|
||||||
|
Should.NotThrow(page.Dispose);
|
||||||
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Sizes_read_the_way_a_file_manager_shows_them()
|
public void Sizes_read_the_way_a_file_manager_shows_them()
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -0,0 +1,160 @@
|
|||||||
|
using AvParser.Core.Collecting;
|
||||||
|
using AvParser.Core.Proxies;
|
||||||
|
using AvParser.Infrastructure.Storage;
|
||||||
|
using AvParser.UI.Tests.Fakes;
|
||||||
|
using AvParser.UI.ViewModels;
|
||||||
|
using ReactiveUI.Primitives.Concurrency;
|
||||||
|
|
||||||
|
namespace AvParser.UI.Tests;
|
||||||
|
|
||||||
|
public class DashboardViewModelTests
|
||||||
|
{
|
||||||
|
private sealed class TestPaths : IAppPaths
|
||||||
|
{
|
||||||
|
public string DataDirectory => Path.Combine(Path.GetTempPath(), "AvParserTests");
|
||||||
|
|
||||||
|
public string SettingsFile => Path.Combine(DataDirectory, "settings.json");
|
||||||
|
|
||||||
|
public string CustomProxiesFile => Path.Combine(DataDirectory, "proxies.custom.json");
|
||||||
|
|
||||||
|
public string ProxyStateFile => Path.Combine(DataDirectory, "proxies.state.json");
|
||||||
|
|
||||||
|
public string LogDirectory => Path.Combine(DataDirectory, "logs");
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class EmptyServiceProvider : IServiceProvider
|
||||||
|
{
|
||||||
|
public object? GetService(Type serviceType) => null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class EmptyUserSourceStore : IUserSourceStore
|
||||||
|
{
|
||||||
|
public event EventHandler? Changed
|
||||||
|
{
|
||||||
|
add { }
|
||||||
|
remove { }
|
||||||
|
}
|
||||||
|
|
||||||
|
public IReadOnlyList<AvParser.Core.Collecting.Sources.PatternSourceConfig> List() => [];
|
||||||
|
|
||||||
|
public Task<AvParser.Core.Collecting.Sources.PatternSourceConfig> AddAsync(
|
||||||
|
AvParser.Core.Collecting.Sources.PatternSourceConfig config,
|
||||||
|
CancellationToken cancellationToken = default
|
||||||
|
) => Task.FromResult(config);
|
||||||
|
|
||||||
|
public Task<bool> UpdateAsync(
|
||||||
|
AvParser.Core.Collecting.Sources.PatternSourceConfig config,
|
||||||
|
CancellationToken cancellationToken = default
|
||||||
|
) => Task.FromResult(false);
|
||||||
|
|
||||||
|
public Task<bool> RemoveAsync(string id, CancellationToken cancellationToken = default) =>
|
||||||
|
Task.FromResult(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static DashboardViewModel Build(IProxyPool pool) =>
|
||||||
|
new(
|
||||||
|
new MediaSourceCatalog(new EmptyUserSourceStore()),
|
||||||
|
new TestPaths(),
|
||||||
|
pool,
|
||||||
|
new EmptyServiceProvider(),
|
||||||
|
ImmediateSequencer.Instance
|
||||||
|
);
|
||||||
|
|
||||||
|
private static ProxyEndpoint Endpoint(string host) => new(ProxyProtocol.Http, host, 8080);
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void An_empty_pool_reports_nothing_rather_than_zeroes_dressed_up_as_health()
|
||||||
|
{
|
||||||
|
using var page = Build(new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()));
|
||||||
|
|
||||||
|
page.ProxyTotal.ShouldBe(0);
|
||||||
|
page.HasProxies.ShouldBeFalse();
|
||||||
|
page.ProxyLatency.ShouldBeNull();
|
||||||
|
page.ProxyRequests.ShouldBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task The_pool_summary_counts_live_and_unchecked_entries()
|
||||||
|
{
|
||||||
|
var alive = Endpoint("1.2.3.4");
|
||||||
|
var dead = Endpoint("5.6.7.8");
|
||||||
|
var pool = new ProxyPool(
|
||||||
|
[new FakeProxySource([alive, dead])],
|
||||||
|
new FakeProxyProbe().Set(alive, alive: true),
|
||||||
|
new ProxyOptions()
|
||||||
|
);
|
||||||
|
|
||||||
|
using var page = Build(pool);
|
||||||
|
|
||||||
|
await pool.RefreshAsync(TestContext.Current.CancellationToken);
|
||||||
|
page.RefreshProxyStats();
|
||||||
|
|
||||||
|
page.ProxyTotal.ShouldBe(2);
|
||||||
|
page.HasProxies.ShouldBeTrue();
|
||||||
|
// Nothing has been probed yet, so nothing is live however promising the list looks.
|
||||||
|
page.ProxyLive.ShouldBe(0);
|
||||||
|
page.ProxyUnchecked.ShouldBe(2);
|
||||||
|
|
||||||
|
await pool.SweepAsync(cancellationToken: TestContext.Current.CancellationToken);
|
||||||
|
page.RefreshProxyStats();
|
||||||
|
|
||||||
|
page.ProxyLive.ShouldBe(1);
|
||||||
|
page.ProxyUnchecked.ShouldBe(0);
|
||||||
|
page.ProxyLatency.ShouldNotBeNull().ShouldContain("20");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Real_requests_show_up_as_a_success_rate()
|
||||||
|
{
|
||||||
|
var endpoint = Endpoint("1.2.3.4");
|
||||||
|
var pool = new ProxyPool(
|
||||||
|
[new FakeProxySource([endpoint])],
|
||||||
|
new FakeProxyProbe().Set(endpoint, alive: true),
|
||||||
|
new ProxyOptions()
|
||||||
|
);
|
||||||
|
|
||||||
|
using var page = Build(pool);
|
||||||
|
|
||||||
|
await pool.RefreshAsync(TestContext.Current.CancellationToken);
|
||||||
|
await pool.SweepAsync(cancellationToken: TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
using (var lease = await pool.AcquireAsync(TestContext.Current.CancellationToken))
|
||||||
|
{
|
||||||
|
lease.ShouldNotBeNull().ReportSuccess(TimeSpan.FromMilliseconds(30));
|
||||||
|
}
|
||||||
|
|
||||||
|
page.RefreshProxyStats();
|
||||||
|
|
||||||
|
page.ProxyRequests.ShouldNotBeNull().ShouldContain("1 of 1");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Disposing_twice_is_safe()
|
||||||
|
{
|
||||||
|
// The container disposes each page once per registration — its own type and PageViewModel.
|
||||||
|
var page = Build(new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()));
|
||||||
|
|
||||||
|
page.Dispose();
|
||||||
|
|
||||||
|
Should.NotThrow(page.Dispose);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task The_summary_follows_the_pool_without_being_asked()
|
||||||
|
{
|
||||||
|
// The page has to react to a sweep it did not start; nobody presses refresh on a dashboard.
|
||||||
|
var endpoint = Endpoint("1.2.3.4");
|
||||||
|
var pool = new ProxyPool(
|
||||||
|
[new FakeProxySource([endpoint])],
|
||||||
|
new FakeProxyProbe().Set(endpoint, alive: true),
|
||||||
|
new ProxyOptions()
|
||||||
|
);
|
||||||
|
|
||||||
|
using var page = Build(pool);
|
||||||
|
page.ProxyTotal.ShouldBe(0);
|
||||||
|
|
||||||
|
await pool.RefreshAsync(TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
page.ProxyTotal.ShouldBe(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -170,4 +170,18 @@ public class GalleryViewModelTests
|
|||||||
page.StatusMessage.ShouldNotBeNull().ShouldContain("Could not read the store");
|
page.StatusMessage.ShouldNotBeNull().ShouldContain("Could not read the store");
|
||||||
page.IsLoading.ShouldBeFalse();
|
page.IsLoading.ShouldBeFalse();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Disposing_twice_is_safe()
|
||||||
|
{
|
||||||
|
// Not hypothetical: every page is registered under its own type and under PageViewModel, so
|
||||||
|
// the container disposes it once per registration. The second call used to cancel an
|
||||||
|
// already-disposed token source and bring the process down on every exit.
|
||||||
|
var (page, _, _) = Build(Media("https://a.test/1.png"));
|
||||||
|
await page.LoadAsync(0, TestContext.Current.CancellationToken);
|
||||||
|
|
||||||
|
page.Dispose();
|
||||||
|
|
||||||
|
Should.NotThrow(page.Dispose);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -229,4 +229,15 @@ public class ProxiesViewModelTests
|
|||||||
// and would keep rebuilding its rows in the background.
|
// and would keep rebuilding its rows in the background.
|
||||||
Should.NotThrow(() => pool.Configure(new ProxyOptions()));
|
Should.NotThrow(() => pool.Configure(new ProxyOptions()));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Disposing_twice_is_safe()
|
||||||
|
{
|
||||||
|
// The container disposes each page once per registration — its own type and PageViewModel.
|
||||||
|
var (page, _, _) = Build("1.2.3.4");
|
||||||
|
|
||||||
|
page.Dispose();
|
||||||
|
|
||||||
|
Should.NotThrow(page.Dispose);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user