- Added methods to `IMediaStore` for loading, saving, and removing media rules, allowing users to manage rules for media items effectively.
- Updated `MediaFetcher` to utilize the new rule management system, integrating rule checks into the fetching process to handle geo-blocks and previously ruled items.
- Enhanced `ProxyPool` to support exclusion of proxies from specific countries during acquisition, improving the handling of geo-blocked content.
- Adjusted `FetchOptions` to include rules instead of tombstones, streamlining the decision-making process during media fetching.
- Updated UI components to support rule editing, providing users with a more interactive experience when managing media rules.
These changes improve the overall media collection process by allowing users to define rules for handling media items and enhancing the proxy management system for better content accessibility.
- Updated `IMediaSourceCatalog` to support user-added media sources, allowing dynamic editing and management of sources.
- Removed the `UrlListSource` class as its functionality is now integrated into the new catalog structure.
- Enhanced `CollectOptions` to default `RequireProxy` to true, ensuring stricter handling of proxy requirements.
- Improved error handling in `ParseError` to include a `Subject` field for better context on failures.
- Adjusted dependency injection to reflect changes in media source management, removing old source registrations.
- Introduced background proxy checks to ensure a more robust proxy pool management during collection processes.
These changes streamline the media collection process and improve the overall user experience by providing clearer error reporting and more flexible source management.
Every limit the fetcher was using was a constant. They are settings now, and
CollectOptions became the single place policy lives: AppSettings.ToCollectOptions
clamps them, and the HTTP layer's FetchOptions is projected from that. One
clamping site rather than two sets of ceilings drifting apart.
Clamping rather than validating, for the reason the proxy options already do it:
a hand-edited file must not stop the app from starting. A MaxItemBytes edited to
zero would otherwise refuse everything, and a zeroed concurrency would deadlock
the run outright - so both are pulled into range instead. An empty format filter
is read as "everything", because switching every format off is far more likely
to be a slip than an instruction to collect nothing.
The media root has an ordering problem - it is a setting that decides the paths
the container is built from - so the file is read once before the container
exists rather than making every path lazy for one value.
Purge is scoped to a source and lives on the Collect page, where the source is
already chosen. Content another source also holds survives, which is what the
index's reference count was for.
The showcase hint says out loud what a hard link means: editing the browsable
copy edits the original, and deleting it frees nothing until the last name goes.
That is surprising enough to belong in the UI rather than only in the code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Third step: the collector becomes wireable. Both catalogs coexist for exactly
this one step, so ParseViewModel and every existing test stay green while the
new domain is proven.
IMediaSource reuses the closed-generic trick ITextParser used, and for the same
reason - the container cannot resolve an open generic as IEnumerable<T>, so
adding a source stays a one-line registration. Its input is a MediaQuery rather
than text, because a source that walks a paginated listing needs an endpoint and
a cursor, not a string.
Sources discover; they do not download. That split is why UrlListSource lives in
the domain with no network at all, and why everything hard about fetching lives
in one place instead of once per source.
The catalog takes an explicit default id. Left to alphabetical order the landing
source would be the network one, so the app would open behind the proxy gate
before the user had asked for anything.
The runner decouples discovery from downloading with a bounded channel - a
listing of two hundred thousand items must not materialise because the workers
are slower than the source - and owns its workers, waiting for them even when
cancelled. Without that a stopped run keeps writing to the store after the page
has said it stopped.
The own-service listing is read leniently: the service on the other end is the
user's own and should not have to be rewritten to match a schema we invented, so
both a bare array of addresses and an object with items and a cursor work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Second half of the collector foundation. Still nothing in the app references
it; the pipeline is tested end to end against a deliberately badly behaved
loopback server before anything depends on it.
Types come from the bytes, never from the URL, the extension or Content-Type -
two of those three are chosen by whoever serves the file, and a host must not
get to pick the extension of a file written to the user's disk. Animation is a
separate question from kind: GIF89a proves nothing without a second image
descriptor, and a PNG is an APNG only if acTL precedes the first IDAT, so both
are walked properly rather than guessed.
Timeouts are split three ways because HttpClient.Timeout covers the whole
response: any value large enough for a 30 MB file is also large enough for a
dead connection to hang on. Connect, headers and a per-read idle deadline let
both be strict. Redirects are followed by hand since the shared proxy handler
disables them, which is what allows a hop cap, loop detection and refusing a
jump to a data: URL.
The lease verdict is a pure function, because ProxyLease's constructor is
internal to the domain and no test can fabricate one. Its rule is that the
verdict describes the transport, not the resource: a 404 is a working proxy,
and so is a 429 - blaming the proxy for an origin's rate limit would make the
pool rotate away from a good address in response to being asked to slow down.
Cancellation reports nothing at all.
Throttling exists to be obeyed. It is raised only by the host's own 429 and 503,
and never by rotating to another proxy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Leonid Pershin
