Gate network parsers on a working proxy and remember what worked

The pool now warms up from what the previous run learned instead of starting
cold every launch. Startup probes the remembered proxies first, stops as soon
as ProxyMinimumLive of them answer, and writes the survivors to
proxies.state.json after the warm-up and again on shutdown. Only proxies that
ever answered are stored: the feed republishes a few thousand dead addresses
every five minutes, and "was dead an hour ago" says almost nothing.

Remembered state is a hint, not a verdict. A restored proxy sorts first in the
warm-up queue but is not counted live until it answers in this session -
otherwise a launch a week later would report live proxies it had never spoken
to, the warm-up would skip the very entries it exists to re-check, and the
parser gate would open on week-old evidence.

That gate is the other half: a parser declaring RequiresNetwork will not run
while the pool has nothing live. The Parse page disables the run button and
shows a banner that leads to the Proxies page. Parsers that work on pasted text
are never gated - they have nothing to route, and blocking them would make the
app useless whenever the public lists are down. Two new settings cover the
escape hatch and the target: "allow network parsers without a proxy" and how
many live proxies to find at startup.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Leonid Pershin
2026-08-13 19:11:51 +03:00
co-authored by Claude Opus 5
parent 85656e70b0
commit 44fb0d3a5f
31 changed files with 1349 additions and 41 deletions
@@ -6,12 +6,15 @@ namespace AvParser.Infrastructure.Tests;
public class ProxyPoolLoaderTests
{
private static ProxyPoolLoader Build(out CountingSource source, out ProxyPool pool)
private static ProxyPoolLoader Build(out CountingSource source, out ProxyPool pool) =>
Build(out source, out pool, new MemoryStateStore());
private static ProxyPoolLoader Build(out CountingSource source, out ProxyPool pool, IProxyStateStore stateStore)
{
source = new CountingSource();
pool = new ProxyPool([source], new NeverProbe(), new ProxyOptions());
return new ProxyPoolLoader(pool, NullLogger<ProxyPoolLoader>.Instance);
return new ProxyPoolLoader(pool, stateStore, NullLogger<ProxyPoolLoader>.Instance);
}
[Fact]
@@ -19,11 +22,42 @@ public class ProxyPoolLoaderTests
{
var loader = Build(out _, out var pool);
(await loader.EnsureLoadedAsync()).ShouldBe(2);
(await loader.EnsureLoadedAsync()).Total.ShouldBe(2);
pool.Entries.Count.ShouldBe(2);
loader.IsLoaded.ShouldBeTrue();
}
[Fact]
public async Task What_the_previous_run_learned_is_restored_onto_the_fresh_list()
{
// The feed republishes the same addresses every few minutes; the point of remembering is
// that a proxy known to work is not re-discovered from scratch on every launch.
var stateStore = new MemoryStateStore
{
State =
{
["http://1.2.3.4:8080"] = new ProxyStateRecord("http://1.2.3.4:8080", Alive: true, LatencyMs: 120),
},
};
var loader = Build(out _, out _, stateStore);
var result = await loader.EnsureLoadedAsync();
result.Restored.ShouldBe(1);
}
[Fact]
public async Task The_pool_state_is_written_back_after_a_load()
{
var stateStore = new MemoryStateStore();
var loader = Build(out _, out _, stateStore);
await loader.EnsureLoadedAsync();
stateStore.Saves.ShouldBeGreaterThan(0);
}
[Fact]
public async Task The_feed_is_fetched_once_however_many_callers_there_are()
{
@@ -52,10 +86,27 @@ public class ProxyPoolLoaderTests
public async Task A_source_that_throws_does_not_take_startup_down()
{
var pool = new ProxyPool([new ThrowingSource()], new NeverProbe(), new ProxyOptions());
var loader = new ProxyPoolLoader(pool, NullLogger<ProxyPoolLoader>.Instance);
var loader = new ProxyPoolLoader(pool, new MemoryStateStore(), NullLogger<ProxyPoolLoader>.Instance);
// The app has to start whether or not a public list is reachable.
(await loader.EnsureLoadedAsync()).ShouldBe(0);
(await loader.EnsureLoadedAsync()).Total.ShouldBe(0);
}
private sealed class MemoryStateStore : IProxyStateStore
{
public Dictionary<string, ProxyStateRecord> State { get; } = new(StringComparer.Ordinal);
public int Saves { get; private set; }
public Task<IReadOnlyDictionary<string, ProxyStateRecord>> LoadAsync(
CancellationToken cancellationToken = default
) => Task.FromResult<IReadOnlyDictionary<string, ProxyStateRecord>>(State);
public Task SaveAsync(IEnumerable<ProxyEntry> entries, CancellationToken cancellationToken = default)
{
Saves++;
return Task.CompletedTask;
}
}
private sealed class CountingSource : IProxySource