eb5061ee23821d47128e02af9f0a0173cd6eef88
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Leonid Pershin
|
eb5061ee23 |
Refactor media source handling and update collection options
- Updated `IMediaSourceCatalog` to support user-added media sources, allowing dynamic editing and management of sources. - Removed the `UrlListSource` class as its functionality is now integrated into the new catalog structure. - Enhanced `CollectOptions` to default `RequireProxy` to true, ensuring stricter handling of proxy requirements. - Improved error handling in `ParseError` to include a `Subject` field for better context on failures. - Adjusted dependency injection to reflect changes in media source management, removing old source registrations. - Introduced background proxy checks to ensure a more robust proxy pool management during collection processes. These changes streamline the media collection process and improve the overall user experience by providing clearer error reporting and more flexible source management. |
||
 Leonid PershinandClaude Opus 5
|
a4a0ea9a6b |
Keep collected media beside the executable, and let it be moved
Two things were wrong with where media lived. It defaulted to the user profile, which is the wrong home for the thing the application exists to accumulate: the collection grows without bound and belongs with the installation, so copying that folder takes the archive with it. And the setting for changing it existed but had no way to be set - the Settings page showed the path as read-only text. Media now defaults to a "media" folder next to the executable, and the Settings page has a box, a folder picker and a reset. Configuration stays in the profile, because that is genuinely per-user and the OS has an opinion about it. Writability is probed with a real file, not just a directory creation: creating a directory can succeed where writing into it does not, which is exactly what an install under Program Files looks like. On failure it falls back to the profile rather than refusing to start, and the effective path is shown in Settings so the fallback is visible instead of mysterious. A change applies on the next launch and says so. Paths are resolved before the container exists - the media root is read straight out of settings.json to build them - so applying it live would mean reconnecting the index, the blob store and the thumbnail cache underneath a possibly-running collection. Writing somewhere other than the box claims would be the worse failure. Existing files are not moved either; relocating an archive is its own operation with its own risks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
ceacec79e2 |
Show the collected content: thumbnails and a gallery
Until now a collected image was a row of text, and after a restart it was not
visible at all. The collect list gains a row thumbnail, and a Gallery page
browses the whole store with filters by source, format and address, paged at
120 tiles, with a built-in viewer showing the full size beside its provenance.
Thumbnails decode straight to the width they are drawn at. That is the whole
memory story: a 4000x3000 JPEG is about 48 MB once decoded, so decoding full
size and scaling afterwards runs out of memory long before the user finishes
scrolling. The cache is bounded and owns its bitmaps, which means its capacity
has to comfortably exceed a page - a bitmap evicted while still on screen would
be disposed out from under the renderer.
Paged rather than infinite-scrolled for the same reason: how much to decode is a
decision the page should make, not one the archive's size makes for it.
Video is not previewed and will not be. Extracting a first frame means FFmpeg,
which is a media stack in exchange for one picture per tile; those tiles show a
format badge instead. The refusal happens before touching the disk, because
attempting it would be an exception per tile.
Rendering the page caught the viewer overlay being see-through: it named a brush
that does not exist, and an unresolved DynamicResource fails silently - the
property just keeps its default. That is the second silent-reference bug to
reach a screenshot, so both kinds now have guards: one resolves every
{DynamicResource} in the XAML against both themes, the other checks every
{l:Loc} key exists. Both were confirmed to fail before being kept.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Leonid PershinandClaude Opus 5
|
f8744c930a |
Remove the demo text-parsing domain
The scaffolding domain existed to prove the shell end to end before there was
anything real to put in it. There is now, so it goes - as CLAUDE.md promised it
would.
Gone: the two sample parsers, ITextParser, ParsedRecord, the parser catalog,
ParseViewModel and ParseView, their tests, and the settings key that remembered
which parser was last used. ParseError.LineNumber becomes Index, since for a
listing "line 42" was simply untrue, and the error keys move from Parse.Error.*
to Collect.Error.* now that parsing is not a concept here.
Kept: IParser<,>, ParseOutcome, ParseProgress and ParseError. The streaming
contract was always the general part - it was only ever the text-shaped closure
of it that was scaffolding.
Rendering the dashboard caught two keys that were referenced but never added
during the rename: the XAML was repointed and the resources were not. The parity
test could not see it, because it compares the two files against each other and
a key absent from both is consistent. That gap now has its own test, which reads
every {l:Loc} in the XAML and checks it resolves - a screenshot is too late and
too manual a way to find a missing string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Leonid PershinandClaude Opus 5
|
fe62bcf53f |
Add collector settings and per-source purge
Every limit the fetcher was using was a constant. They are settings now, and CollectOptions became the single place policy lives: AppSettings.ToCollectOptions clamps them, and the HTTP layer's FetchOptions is projected from that. One clamping site rather than two sets of ceilings drifting apart. Clamping rather than validating, for the reason the proxy options already do it: a hand-edited file must not stop the app from starting. A MaxItemBytes edited to zero would otherwise refuse everything, and a zeroed concurrency would deadlock the run outright - so both are pulled into range instead. An empty format filter is read as "everything", because switching every format off is far more likely to be a slip than an instruction to collect nothing. The media root has an ordering problem - it is a setting that decides the paths the container is built from - so the file is read once before the container exists rather than making every path lazy for one value. Purge is scoped to a source and lives on the Collect page, where the source is already chosen. Content another source also holds survives, which is what the index's reference count was for. The showcase hint says out loud what a hard link means: editing the browsable copy edits the original, and deleting it frees nothing until the last name goes. That is surprising enough to belong in the UI rather than only in the code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
70fb3a1df3 |
Add the Collect page
The collector becomes usable. The page is a close copy of the parse page's shape - same proxy gate, same batched flush to the observable collection, same truncation cap that reports rather than truncates silently - because that shape was built for streaming outcomes and the collector produces exactly those. Two differences that are not cosmetic. The batch drops from 512 to 64: items arrive at network speed, roughly one a second, and a batch of five hundred would mean the list never visibly moved. And progress is explicitly indeterminate until a source finishes listing, because a paginated listing genuinely does not know its total until the last page - a bar pretending otherwise would be lying. The input swaps shape with the source: an endpoint source wants one address, a pasted-list source wants many lines. The proxy gate is unchanged in substance - only network sources are gated, and the banner keeps its x:Name because the headless tests find it that way. Rendering the page caught a defect the tests could not: IsVisible sat on the caption inside the header border rather than on the border, so hiding the text left its padding and divider behind as an empty bar above the input. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
6909884851 |
Add media sources and the collect runner, alongside the old parsers
Third step: the collector becomes wireable. Both catalogs coexist for exactly this one step, so ParseViewModel and every existing test stay green while the new domain is proven. IMediaSource reuses the closed-generic trick ITextParser used, and for the same reason - the container cannot resolve an open generic as IEnumerable<T>, so adding a source stays a one-line registration. Its input is a MediaQuery rather than text, because a source that walks a paginated listing needs an endpoint and a cursor, not a string. Sources discover; they do not download. That split is why UrlListSource lives in the domain with no network at all, and why everything hard about fetching lives in one place instead of once per source. The catalog takes an explicit default id. Left to alphabetical order the landing source would be the network one, so the app would open behind the proxy gate before the user had asked for anything. The runner decouples discovery from downloading with a bounded channel - a listing of two hundred thousand items must not materialise because the workers are slower than the source - and owns its workers, waiting for them even when cancelled. Without that a stopped run keeps writing to the store after the page has said it stopped. The own-service listing is read leniently: the service on the other end is the user's own and should not have to be rewritten to match a schema we invented, so both a bare array of addresses and an object with items and a cursor work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
1742c094e9 |
Add the download pipeline: sniffing, redirects, throttling, verdicts
Second half of the collector foundation. Still nothing in the app references it; the pipeline is tested end to end against a deliberately badly behaved loopback server before anything depends on it. Types come from the bytes, never from the URL, the extension or Content-Type - two of those three are chosen by whoever serves the file, and a host must not get to pick the extension of a file written to the user's disk. Animation is a separate question from kind: GIF89a proves nothing without a second image descriptor, and a PNG is an APNG only if acTL precedes the first IDAT, so both are walked properly rather than guessed. Timeouts are split three ways because HttpClient.Timeout covers the whole response: any value large enough for a 30 MB file is also large enough for a dead connection to hang on. Connect, headers and a per-read idle deadline let both be strict. Redirects are followed by hand since the shared proxy handler disables them, which is what allows a hop cap, loop detection and refusing a jump to a data: URL. The lease verdict is a pure function, because ProxyLease's constructor is internal to the domain and no test can fabricate one. Its rule is that the verdict describes the transport, not the resource: a 404 is a working proxy, and so is a 429 - blaming the proxy for an origin's rate limit would make the pool rotate away from a good address in response to being asked to slow down. Cancellation reports nothing at all. Throttling exists to be obeyed. It is raised only by the host's own 429 and 503, and never by rotating to another proxy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
181f974a37 |
Add the media store: content-addressed blobs, SQLite index, showcase
First half of replacing the stub text domain with a media collector. Nothing references this yet - the store is standalone and fully tested before anything depends on it. Blobs are addressed by SHA-256 and sharded two levels deep, so the same picture re-uploaded at a dozen addresses costs one file. Downloads stage in a sibling temp directory on the same volume and are promoted by rename, which is what keeps blobs/ free of truncated files: a crash leaves a stray .part that the next startup sweeps, never a half-image indistinguishable from a real one. The SQLite index holds provenance separately from content, so purging one source leaves blobs another source still references - that is what ref_count buys, and it is recomputed rather than incremented because the item upsert can replace a row pointing at a different blob. The seen_url journal deliberately outlives a purge: without that, the next run downloads again exactly what the user just deleted. Terminal outcomes are split from retryable ones so a flaky network does not permanently lose content. The showcase gives every item a dated, named path via hard links - a second name for one file, not a second file. Hard links are a filesystem privilege rather than a guarantee, so it degrades to copying and records which it achieved; the UI has to be able to admit that. Names suggested by the origin are treated as hostile: only the last path segment survives, Windows device names are pushed aside, and the extension comes from the sniffed kind, never from the remote. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
44fb0d3a5f |
Gate network parsers on a working proxy and remember what worked
The pool now warms up from what the previous run learned instead of starting cold every launch. Startup probes the remembered proxies first, stops as soon as ProxyMinimumLive of them answer, and writes the survivors to proxies.state.json after the warm-up and again on shutdown. Only proxies that ever answered are stored: the feed republishes a few thousand dead addresses every five minutes, and "was dead an hour ago" says almost nothing. Remembered state is a hint, not a verdict. A restored proxy sorts first in the warm-up queue but is not counted live until it answers in this session - otherwise a launch a week later would report live proxies it had never spoken to, the warm-up would skip the very entries it exists to re-check, and the parser gate would open on week-old evidence. That gate is the other half: a parser declaring RequiresNetwork will not run while the pool has nothing live. The Parse page disables the run button and shows a banner that leads to the Proxies page. Parsers that work on pasted text are never gated - they have nothing to route, and blocking them would make the app useless whenever the public lists are down. Two new settings cover the escape hatch and the target: "allow network parsers without a proxy" and how many live proxies to find at startup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
85656e70b0 |
Load the proxy pool at startup, and stop old settings files zeroing new defaults
The pool was empty until someone pressed Refresh, which also meant the parser
would have had nothing to work with. A ProxyPoolLoader now fills it once at
launch; startup does not await it, because blocking on a public list being
reachable would be the wrong trade, and it never throws. The proxy page joins
the same operation rather than starting a second download, so it reports the
outcome whether it is opened during the load or long after.
That change surfaced a worse bug underneath. The first run still loaded zero
entries with no error logged at all, which turned out to be the feed source
never being asked: options said UseFeed=False and Protocols=None. Neither is
reachable from the UI — both are default(T).
The cause is that AppSettings kept its defaults on property initialisers, and
the source-generated deserialiser does not run them. Reflection-based
deserialisation of "{}" keeps them; the generated context does not. So a
settings.json written before a setting existed came back with default(T) for it:
the proxy feed switched off, the protocol filter empty, the probe timeout zero
and the probe URL blank — and the app looked like the network had failed.
Defaults now live on primary constructor parameters, which STJ applies for
absent JSON members on both paths, so an older file upgrades cleanly. The
regression test writes a settings file from before the proxy settings existed
and asserts each one comes back at its default. ToProxyOptions also treats an
empty protocol filter as "all", since a hand-edited file that matches nothing is
the least useful possible reading of it.
Also quietens IHttpClientFactory to Warning: four Information lines per request
buried everything the app said, and a proxy sweep makes thousands of them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Leonid PershinandClaude Opus 5
|
8b552470b7 |
Localise the UI into Russian and English, switchable without a restart
Strings move into Localization/Strings.resx plus a Russian satellite. XAML uses
a {l:Loc Key} markup extension that yields a binding through the localizer's
indexer rather than a resolved string, so changing the language raises
PropertyChanged for the indexer and every caption in the app re-reads at once.
Resolving strings once at load would have been simpler and would have needed an
app restart to take effect.
Three places where this is more than a string swap:
Russian has three plural forms, so counted messages are assembled from .One /
.Few / .Many keys via Localizer.Plural instead of "{0} records" with an English
plural glued on. "1 запись", "3 записи", "7 записей".
Enum captions in pickers go through LocalizedOption<T>. A value converter would
resolve the caption once and never notice a language change; the wrapper keeps
identity on the enum value so the selection survives, while the label follows
the localizer. It needed a non-generic base for DataTemplate x:DataType, since
Avalonia 12 compiles bindings by default and cannot infer one for an open
generic.
Text originating in the domain would otherwise have stayed English under a
Russian UI — the screenshots showed exactly that. AvParser.Core still knows
nothing about languages: ParseError now carries a Code and Arguments, and the UI
translates Parse.Error.{Code} with a fallback to the English message. Parser
names work the same way (Parser.{id}.Name falling back to DisplayName), which
keeps "add a parser = one registration line" true — an untranslated parser shows
its own name rather than a missing-key marker.
Both .resx files are generated from one table so a key cannot exist in one and
be missing from the other, and the tests assert that, plus no blank translations
and identical {0} placeholder sets — a translation that drops a placeholder
throws at runtime rather than merely reading oddly. A headless test switches
language on a live shell and asserts the rendered text changes without the tree
being rebuilt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Leonid PershinandClaude Opus 5
|
9bf2ea5532 |
Add a proxy pool with rotation, liveness checks and a management page
The parser will need to move between proxies, so this adds the module it will sit on: pluggable sources, a pool that hands proxies out and learns from the outcome, three rotation strategies, and a page to drive it. Sources are IProxySource implementations. The public proxifly/free-proxy-list feed is fetched as the combined all/data.json through jsDelivr and filtered locally — one conditional request beats four per-protocol ones that can disagree mid-publish — and cached for the five minutes upstream takes to regenerate. A feed that is down keeps serving its last payload rather than emptying the pool. The user's own list lives in proxies.custom.json beside the settings, takes a pasted blob, and names the lines it could not parse instead of quietly dropping them. Both knobs the pool exposes are settings, as asked: rotation is Sticky (default, the only one that keeps site sessions coherent), RoundRobin or WeightedRandom; liveness is either a parallel sweep of the whole pool or a probe at hand-out time. Free lists are a few percent alive, so skipping verification entirely means mostly waiting on timeouts. Two invariants worth keeping, both of which cost a bug to find: Availability is decided by the quarantine, never by Health. Excluding everything that has ever failed made the quarantine window dead code and discarded proxies permanently on their first hiccup, which is exactly wrong for addresses that flap constantly. Health only orders the candidates now. A probe verdict does not touch the success/failure counters. Those are about real requests, and letting a sweep over a few thousand proxies rewrite them would drown the evidence weighted selection reads. SOCKS needs no extra package — .NET resolves socks4/socks4a/socks5 in WebProxy — but a proxifly record with "protocol": "https" is still an HTTP proxy reached over http:// with CONNECT, not an https:// scheme. 115 new tests. Also fixes a pre-existing flake: a command gated on another command's IsExecuting cannot be driven straight after its Execute() completes, because IsExecuting is published on the output scheduler. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
aeafe0af36 |
Add build and run scripts; drop the broken Avalonia.Diagnostics reference
A wrapper around `dotnet build` earns nothing, so build.ps1/build.sh are the full local gate instead — tool restore, package restore, format check, build, test — in the order that fails cheapest first, with a non-zero exit on failure. run.ps1/run.sh default to Debug and pass extra arguments through to the app. Both flavours ship because the Desktop head targets Windows, Linux and macOS. Writing them immediately paid for itself: the very first run failed restore on Avalonia.Diagnostics 12.1.1, which does not exist — the package stops at 11.3.x because Avalonia 12 moved the inspector into a separate tool with its own installation. The reference had survived because it sat behind Condition="'$(Configuration)' == 'Debug'", and `dotnet restore` evaluates with the default configuration while every build so far had passed -c Release. So `dotnet build -c Release` worked and a bare `dotnet restore` did not. Reference removed rather than replaced: AvaloniaUI.DiagnosticsSupport pulls in a separately installed tool, which is not a dependency to add to a skeleton without asking. README and CLAUDE.md no longer promise F12, and both traps are written down where the next person will hit them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Leonid PershinandClaude Opus 5
|
3db9d4dfc6 |
Scaffold AvParser: Avalonia 12 shell with adaptive layout
Greenfield skeleton for a parser desktop app. The domain is deliberately a placeholder — IParser<TIn,TOut> plus two sample parsers — so the shell is runnable and verifiable end to end before real logic lands. Layers run one way: Core (no Avalonia, no IO) <- Infrastructure <- UI <- Desktop. UI is a class library rather than the exe so headless tests build real views without dragging in Program.cs, Serilog or the container. Adaptive layout is built from what Avalonia actually offers, since it has no AdaptiveTrigger or media queries: ResponsiveLayout observes Visual.Bounds and projects a breakpoint onto both an attached property and :compact/:medium/ :expanded pseudoclasses, with 24px hysteresis so dragging a window edge cannot make the layout flap. Pane state lives in the view model because a style setter loses to a local value permanently; styles own only the visual variance. Stack notes worth remembering: Avalonia.ReactiveUI is deprecated in favour of ReactiveUI.Avalonia, and ReactiveUI 24 runs on the Primitives engine (RxVoid, ISequencer, Signal<T>) and no longer self-initialises. Avalonia.Headless.XUnit 12.x requires xUnit v3. InvariantGlobalization must stay false or Semi.Avalonia throws in its static constructor. 102 tests across three projects, including headless guards for the two failures that are otherwise completely silent: a stylesheet whose selectors match nothing, and a light palette too low-contrast for cards to read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |