- Updated `IMediaSourceCatalog` to support user-added media sources, allowing dynamic editing and management of sources.
- Removed the `UrlListSource` class as its functionality is now integrated into the new catalog structure.
- Enhanced `CollectOptions` to default `RequireProxy` to true, ensuring stricter handling of proxy requirements.
- Improved error handling in `ParseError` to include a `Subject` field for better context on failures.
- Adjusted dependency injection to reflect changes in media source management, removing old source registrations.
- Introduced background proxy checks to ensure a more robust proxy pool management during collection processes.
These changes streamline the media collection process and improve the overall user experience by providing clearer error reporting and more flexible source management.
The scaffolding domain existed to prove the shell end to end before there was
anything real to put in it. There is now, so it goes - as CLAUDE.md promised it
would.
Gone: the two sample parsers, ITextParser, ParsedRecord, the parser catalog,
ParseViewModel and ParseView, their tests, and the settings key that remembered
which parser was last used. ParseError.LineNumber becomes Index, since for a
listing "line 42" was simply untrue, and the error keys move from Parse.Error.*
to Collect.Error.* now that parsing is not a concept here.
Kept: IParser<,>, ParseOutcome, ParseProgress and ParseError. The streaming
contract was always the general part - it was only ever the text-shaped closure
of it that was scaffolding.
Rendering the dashboard caught two keys that were referenced but never added
during the rename: the XAML was repointed and the resources were not. The parity
test could not see it, because it compares the two files against each other and
a key absent from both is consistent. That gap now has its own test, which reads
every {l:Loc} in the XAML and checks it resolves - a screenshot is too late and
too manual a way to find a missing string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Third step: the collector becomes wireable. Both catalogs coexist for exactly
this one step, so ParseViewModel and every existing test stay green while the
new domain is proven.
IMediaSource reuses the closed-generic trick ITextParser used, and for the same
reason - the container cannot resolve an open generic as IEnumerable<T>, so
adding a source stays a one-line registration. Its input is a MediaQuery rather
than text, because a source that walks a paginated listing needs an endpoint and
a cursor, not a string.
Sources discover; they do not download. That split is why UrlListSource lives in
the domain with no network at all, and why everything hard about fetching lives
in one place instead of once per source.
The catalog takes an explicit default id. Left to alphabetical order the landing
source would be the network one, so the app would open behind the proxy gate
before the user had asked for anything.
The runner decouples discovery from downloading with a bounded channel - a
listing of two hundred thousand items must not materialise because the workers
are slower than the source - and owns its workers, waiting for them even when
cancelled. Without that a stopped run keeps writing to the store after the page
has said it stopped.
The own-service listing is read leniently: the service on the other end is the
user's own and should not have to be rewritten to match a schema we invented, so
both a bare array of addresses and an object with items and a cursor work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pool now warms up from what the previous run learned instead of starting
cold every launch. Startup probes the remembered proxies first, stops as soon
as ProxyMinimumLive of them answer, and writes the survivors to
proxies.state.json after the warm-up and again on shutdown. Only proxies that
ever answered are stored: the feed republishes a few thousand dead addresses
every five minutes, and "was dead an hour ago" says almost nothing.
Remembered state is a hint, not a verdict. A restored proxy sorts first in the
warm-up queue but is not counted live until it answers in this session -
otherwise a launch a week later would report live proxies it had never spoken
to, the warm-up would skip the very entries it exists to re-check, and the
parser gate would open on week-old evidence.
That gate is the other half: a parser declaring RequiresNetwork will not run
while the pool has nothing live. The Parse page disables the run button and
shows a banner that leads to the Proxies page. Parsers that work on pasted text
are never gated - they have nothing to route, and blocking them would make the
app useless whenever the public lists are down. Two new settings cover the
escape hatch and the target: "allow network parsers without a proxy" and how
many live proxies to find at startup.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The parser will need to move between proxies, so this adds the module it will
sit on: pluggable sources, a pool that hands proxies out and learns from the
outcome, three rotation strategies, and a page to drive it.
Sources are IProxySource implementations. The public proxifly/free-proxy-list
feed is fetched as the combined all/data.json through jsDelivr and filtered
locally — one conditional request beats four per-protocol ones that can disagree
mid-publish — and cached for the five minutes upstream takes to regenerate. A
feed that is down keeps serving its last payload rather than emptying the pool.
The user's own list lives in proxies.custom.json beside the settings, takes a
pasted blob, and names the lines it could not parse instead of quietly dropping
them.
Both knobs the pool exposes are settings, as asked: rotation is Sticky (default,
the only one that keeps site sessions coherent), RoundRobin or WeightedRandom;
liveness is either a parallel sweep of the whole pool or a probe at hand-out
time. Free lists are a few percent alive, so skipping verification entirely
means mostly waiting on timeouts.
Two invariants worth keeping, both of which cost a bug to find:
Availability is decided by the quarantine, never by Health. Excluding everything
that has ever failed made the quarantine window dead code and discarded proxies
permanently on their first hiccup, which is exactly wrong for addresses that
flap constantly. Health only orders the candidates now.
A probe verdict does not touch the success/failure counters. Those are about
real requests, and letting a sweep over a few thousand proxies rewrite them
would drown the evidence weighted selection reads.
SOCKS needs no extra package — .NET resolves socks4/socks4a/socks5 in WebProxy —
but a proxifly record with "protocol": "https" is still an HTTP proxy reached
over http:// with CONNECT, not an https:// scheme.
115 new tests. Also fixes a pre-existing flake: a command gated on another
command's IsExecuting cannot be driven straight after its Execute() completes,
because IsExecuting is published on the output scheduler.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Greenfield skeleton for a parser desktop app. The domain is deliberately a
placeholder — IParser<TIn,TOut> plus two sample parsers — so the shell is
runnable and verifiable end to end before real logic lands.
Layers run one way: Core (no Avalonia, no IO) <- Infrastructure <- UI <- Desktop.
UI is a class library rather than the exe so headless tests build real views
without dragging in Program.cs, Serilog or the container.
Adaptive layout is built from what Avalonia actually offers, since it has no
AdaptiveTrigger or media queries: ResponsiveLayout observes Visual.Bounds and
projects a breakpoint onto both an attached property and :compact/:medium/
:expanded pseudoclasses, with 24px hysteresis so dragging a window edge cannot
make the layout flap. Pane state lives in the view model because a style setter
loses to a local value permanently; styles own only the visual variance.
Stack notes worth remembering: Avalonia.ReactiveUI is deprecated in favour of
ReactiveUI.Avalonia, and ReactiveUI 24 runs on the Primitives engine (RxVoid,
ISequencer, Signal<T>) and no longer self-initialises. Avalonia.Headless.XUnit
12.x requires xUnit v3. InvariantGlobalization must stay false or Semi.Avalonia
throws in its static constructor.
102 tests across three projects, including headless guards for the two failures
that are otherwise completely silent: a stylesheet whose selectors match nothing,
and a light palette too low-contrast for cards to read.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Leonid Pershin
