Add media sources and the collect runner, alongside the old parsers
Third step: the collector becomes wireable. Both catalogs coexist for exactly this one step, so ParseViewModel and every existing test stay green while the new domain is proven. IMediaSource reuses the closed-generic trick ITextParser used, and for the same reason - the container cannot resolve an open generic as IEnumerable<T>, so adding a source stays a one-line registration. Its input is a MediaQuery rather than text, because a source that walks a paginated listing needs an endpoint and a cursor, not a string. Sources discover; they do not download. That split is why UrlListSource lives in the domain with no network at all, and why everything hard about fetching lives in one place instead of once per source. The catalog takes an explicit default id. Left to alphabetical order the landing source would be the network one, so the app would open behind the proxy gate before the user had asked for anything. The runner decouples discovery from downloading with a bounded channel - a listing of two hundred thousand items must not materialise because the workers are slower than the source - and owns its workers, waiting for them even when cancelled. Without that a stopped run keeps writing to the store after the page has said it stopped. The own-service listing is read leniently: the service on the other end is the user's own and should not have to be rewritten to match a schema we invented, so both a bare array of addresses and an object with items and a cursor work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1742c094e9
commit
6909884851
@@ -1,3 +1,5 @@
|
||||
using AvParser.Core.Collecting;
|
||||
using AvParser.Core.Collecting.Sources;
|
||||
using AvParser.Core.Parsing;
|
||||
using AvParser.Core.Parsing.Samples;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
@@ -22,6 +24,15 @@ public static class CoreServiceCollectionExtensions
|
||||
services.AddSingleton<ITextParser, KeyValueTextParser>();
|
||||
services.AddSingleton<IParserCatalog, ParserCatalog>();
|
||||
|
||||
services.AddSingleton<IMediaSource, UrlListSource>();
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
/// <summary>Id of the source the collector opens on.</summary>
|
||||
/// <remarks>
|
||||
/// Named rather than left to alphabetical order, which would land on the network source and
|
||||
/// open the page behind the proxy gate before the user has asked for anything.
|
||||
/// </remarks>
|
||||
public const string DefaultMediaSourceId = "url-list";
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user