Gate network parsers on a working proxy and remember what worked

The pool now warms up from what the previous run learned instead of starting
cold every launch. Startup probes the remembered proxies first, stops as soon
as ProxyMinimumLive of them answer, and writes the survivors to
proxies.state.json after the warm-up and again on shutdown. Only proxies that
ever answered are stored: the feed republishes a few thousand dead addresses
every five minutes, and "was dead an hour ago" says almost nothing.

Remembered state is a hint, not a verdict. A restored proxy sorts first in the
warm-up queue but is not counted live until it answers in this session -
otherwise a launch a week later would report live proxies it had never spoken
to, the warm-up would skip the very entries it exists to re-check, and the
parser gate would open on week-old evidence.

That gate is the other half: a parser declaring RequiresNetwork will not run
while the pool has nothing live. The Parse page disables the run button and
shows a banner that leads to the Proxies page. Parsers that work on pasted text
are never gated - they have nothing to route, and blocking them would make the
app useless whenever the public lists are down. Two new settings cover the
escape hatch and the target: "allow network parsers without a proxy" and how
many live proxies to find at startup.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/AvParser.Core/Parsing/IParser.cs

@@ -21,6 +21,17 @@ public interface IParser<in TInput, TOutput>
     /// <summary>Cheap structural check — must not throw and must not do IO.</summary>
     bool CanParse(TInput input);
 
+    /// <summary>
+    /// Whether this parser makes network requests and therefore needs a working proxy.
+    /// </summary>
+    /// <remarks>
+    /// A default implementation rather than an abstract member, so adding a parser stays a
+    /// one-line change: a parser that works on text the user pasted opts out by saying nothing.
+    /// Parsers that fetch anything must set this, or they will run direct even when the user has
+    /// asked for proxy-only operation.
+    /// </remarks>
+    bool RequiresNetwork => false;
+
     /// <summary>Streams one outcome per logical record.</summary>
     IAsyncEnumerable<ParseOutcome<TOutput>> ParseAsync(
         TInput input,

src/AvParser.Core/Proxies/IProxyPool.cs

@@ -9,6 +9,9 @@ public interface IProxyPool
     /// <summary>Options currently in force.</summary>
     ProxyOptions Options { get; }
 
+    /// <summary>How many entries are known to work and are not sidelined right now.</summary>
+    int LiveCount { get; }
+
     /// <summary>Raised after the set of entries or their health changes.</summary>
     /// <remarks>
     /// A plain event rather than <c>IObservable</c> so the domain keeps no reactive dependency;
@@ -33,6 +36,21 @@ public interface IProxyPool
     /// <summary>Probes every entry in parallel and updates their health.</summary>
     /// <returns>How many answered.</returns>
     Task<int> SweepAsync(IProgress<ProxySweepProgress>? progress = null, CancellationToken cancellationToken = default);
+
+    /// <summary>
+    /// Probes best-known entries first and stops once <paramref name="targetLive"/> of them work.
+    /// </summary>
+    /// <returns>How many are live afterwards.</returns>
+    /// <remarks>
+    /// The startup counterpart to <see cref="SweepAsync"/>: proxies that answered on a previous
+    /// run are tried first, so a second launch usually confirms enough of them in a handful of
+    /// requests instead of re-probing a few thousand addresses.
+    /// </remarks>
+    Task<int> WarmUpAsync(
+        int targetLive,
+        IProgress<ProxySweepProgress>? progress = null,
+        CancellationToken cancellationToken = default
+    );
 }
 
 /// <summary>

src/AvParser.Core/Proxies/ProxyEntry.cs

@@ -181,6 +181,65 @@ public sealed class ProxyEntry
         }
     }
 
+    /// <summary>Whether a previous run saw this proxy answer.</summary>
+    /// <remarks>
+    /// Kept apart from <see cref="Health"/> so that "worked yesterday" can order the warm-up
+    /// without being mistaken for "works now". Only this session's verdict counts as live.
+    /// </remarks>
+    public bool WasAliveOnLastRun { get; private set; }
+
+    /// <summary>
+    /// Reinstates what a previous run learned about this proxy.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// Health comes back as <see cref="ProxyHealthState.Unknown"/> even for a proxy that answered
+    /// last time. Restoring it as <see cref="ProxyHealthState.Alive"/> would make the pool report
+    /// live proxies it has not spoken to — the warm-up would then skip them as already-confirmed,
+    /// and a launch a week later would open the parser gate on week-old evidence.
+    /// </para>
+    /// <para>
+    /// Restores no quarantine either: the window is wall-clock and a restart may be days later, so
+    /// carrying it over would sideline proxies for reasons that have long expired.
+    /// </para>
+    /// </remarks>
+    public void RestoreState(
+        bool wasAlive,
+        TimeSpan? latency,
+        int successCount,
+        int failureCount,
+        DateTimeOffset? lastCheckedUtc
+    )
+    {
+        lock (_gate)
+        {
+            Health = ProxyHealthState.Unknown;
+            WasAliveOnLastRun = wasAlive;
+            Latency = latency;
+            SuccessCount = Math.Max(0, successCount);
+            FailureCount = Math.Max(0, failureCount);
+            LastCheckedUtc = lastCheckedUtc;
+            ConsecutiveFailures = 0;
+            QuarantinedUntilUtc = null;
+        }
+    }
+
+    /// <summary>Best current belief about whether this proxy answers.</summary>
+    /// <remarks>
+    /// This session's verdict wins; with none, what the previous run saw carries forward. Unlike
+    /// <see cref="ProxyPool.LiveCount"/> this is a belief, not a confirmation — never gate on it.
+    /// </remarks>
+    public bool IsBelievedAlive =>
+        Health switch
+        {
+            ProxyHealthState.Alive => true,
+            ProxyHealthState.Dead => false,
+            _ => WasAliveOnLastRun,
+        };
+
+    /// <summary>Whether this proxy ever answered, and is therefore worth remembering.</summary>
+    public bool HasEverAnswered => SuccessCount > 0 || IsBelievedAlive;
+
     /// <inheritdoc />
     public override string ToString() => $"{Endpoint} [{Health}]";
 }

src/AvParser.Core/Proxies/ProxyOptions.cs

@@ -103,6 +103,25 @@ public sealed record ProxyOptions
     /// <summary>Whether the remote feed is consulted at all.</summary>
     public bool UseFeed { get; init; } = true;
 
+    /// <summary>
+    /// How many working proxies a warm-up aims for before it stops probing.
+    /// </summary>
+    /// <remarks>
+    /// A free list holds thousands of addresses of which a few percent work. Probing all of them
+    /// at every launch costs thousands of requests for information that goes stale in minutes;
+    /// stopping once there are enough to rotate through is the useful part of that work.
+    /// </remarks>
+    public int MinimumLiveProxies { get; init; } = 10;
+
+    /// <summary>
+    /// Whether work may run without a proxy when none is available.
+    /// </summary>
+    /// <remarks>
+    /// Only parsers that declare <c>RequiresNetwork</c> are affected; a parser working on pasted
+    /// text is never blocked.
+    /// </remarks>
+    public bool AllowDirectConnection { get; init; }
+
     /// <summary>Validates the options, throwing on values that would misbehave silently.</summary>
     /// <exception cref="ArgumentOutOfRangeException">A numeric option is out of range.</exception>
     public ProxyOptions Validated()

src/AvParser.Core/Proxies/ProxyPool.cs

@@ -64,6 +64,20 @@ public sealed class ProxyPool : IProxyPool
         }
     }
 
+    /// <inheritdoc />
+    public int LiveCount
+    {
+        get
+        {
+            var now = _time.GetUtcNow();
+
+            lock (_gate)
+            {
+                return _entries.Count(entry => entry.Health == ProxyHealthState.Alive && entry.IsAvailable(now));
+            }
+        }
+    }
+
     /// <inheritdoc />
     public event EventHandler? Changed;
 
@@ -268,6 +282,115 @@ public sealed class ProxyPool : IProxyPool
         return aliveCount;
     }
 
+    /// <inheritdoc />
+    public async Task<int> WarmUpAsync(
+        int targetLive,
+        IProgress<ProxySweepProgress>? progress = null,
+        CancellationToken cancellationToken = default
+    )
+    {
+        ArgumentOutOfRangeException.ThrowIfLessThan(targetLive, 1);
+
+        var options = Options;
+        var candidates = WarmUpOrder();
+        var total = candidates.Count;
+
+        if (total == 0 || LiveCount >= targetLive)
+        {
+            progress?.Report(new ProxySweepProgress(0, 0, LiveCount));
+            return LiveCount;
+        }
+
+        // Stops the remaining probes the moment the target is met. Linked so the caller's own
+        // cancellation still wins.
+        using var stop = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
+        using var limiter = new SemaphoreSlim(options.ProbeConcurrency, options.ProbeConcurrency);
+
+        var checkedCount = 0;
+
+        var work = candidates.Select(async entry =>
+        {
+            if (stop.IsCancellationRequested)
+            {
+                return;
+            }
+
+            try
+            {
+                await limiter.WaitAsync(stop.Token).ConfigureAwait(false);
+            }
+            catch (OperationCanceledException)
+            {
+                return;
+            }
+
+            try
+            {
+                var result = await _probe.ProbeAsync(entry.Endpoint, options, stop.Token).ConfigureAwait(false);
+                entry.RecordProbe(
+                    _time.GetUtcNow(),
+                    result.Alive,
+                    result.Latency,
+                    result.Error,
+                    quarantineOnFailure: options.BaseQuarantine
+                );
+            }
+            catch (OperationCanceledException)
+            {
+                // Either the target was reached or the caller gave up; neither is the proxy's fault.
+                return;
+            }
+            finally
+            {
+                limiter.Release();
+            }
+
+            var live = LiveCount;
+            progress?.Report(new ProxySweepProgress(Interlocked.Increment(ref checkedCount), total, live));
+
+            if (live >= targetLive)
+            {
+                await stop.CancelAsync().ConfigureAwait(false);
+            }
+        });
+
+        await Task.WhenAll(work).ConfigureAwait(false);
+
+        cancellationToken.ThrowIfCancellationRequested();
+
+        lock (_gate)
+        {
+            _strategy.Reset();
+        }
+
+        RaiseChanged();
+        return LiveCount;
+    }
+
+    /// <summary>
+    /// Orders candidates for a warm-up: what worked last time, then what looks most promising.
+    /// </summary>
+    /// <remarks>Public so the ordering can be asserted without running probes.</remarks>
+    public IReadOnlyList<ProxyEntry> WarmUpOrder()
+    {
+        var now = _time.GetUtcNow();
+
+        lock (_gate)
+        {
+            return
+            [
+                .. _entries
+                    .Where(entry => entry.IsAvailable(now))
+                    .OrderByDescending(entry => entry.Health == ProxyHealthState.Alive)
+                    .ThenByDescending(entry => entry.WasAliveOnLastRun)
+                    .ThenByDescending(entry => entry.SuccessCount > 0)
+                    .ThenBy(entry => entry.Latency ?? TimeSpan.MaxValue)
+                    .ThenByDescending(entry => entry.SuccessRate)
+                    .ThenByDescending(entry => entry.Endpoint.Score),
+            ];
+        }
+    }
+
     /// <summary>Records the outcome of a lease. Called by <see cref="ProxyLease"/>.</summary>
     internal void ReportOutcome(ProxyEntry entry, bool success, TimeSpan? latency, string? error)
     {

src/AvParser.Core/Settings/AppSettings.cs

@@ -45,6 +45,8 @@ public enum AppTheme
 /// <param name="ProxyProbeUrl">URL fetched to decide whether a proxy works.</param>
 /// <param name="ProxyProbeTimeoutSeconds">Per-proxy probe timeout, in seconds.</param>
 /// <param name="ProxyProbeConcurrency">How many probes run at once during a pool sweep.</param>
+/// <param name="ProxyMinimumLive">How many working proxies a startup warm-up aims for.</param>
+/// <param name="AllowDirectConnection">Whether network parsers may run without a proxy.</param>
 public sealed record AppSettings(
     AppTheme Theme = AppTheme.System,
     AppLanguage Language = AppLanguage.System,
@@ -59,7 +61,9 @@ public sealed record AppSettings(
     ProxyProtocolFilter ProxyProtocols = ProxyProtocolFilter.All,
     string ProxyProbeUrl = "http://www.gstatic.com/generate_204",
     int ProxyProbeTimeoutSeconds = 8,
-    int ProxyProbeConcurrency = 64
+    int ProxyProbeConcurrency = 64,
+    int ProxyMinimumLive = 10,
+    bool AllowDirectConnection = false
 )
 {
     /// <summary>Projects the proxy-related settings onto <see cref="ProxyOptions"/>.</summary>
@@ -83,6 +87,8 @@ public sealed record AppSettings(
             ProbeUrl = probeUrl,
             ProbeTimeout = TimeSpan.FromSeconds(Math.Clamp(ProxyProbeTimeoutSeconds, 1, 120)),
             ProbeConcurrency = Math.Clamp(ProxyProbeConcurrency, 1, 512),
+            MinimumLiveProxies = Math.Clamp(ProxyMinimumLive, 1, 500),
+            AllowDirectConnection = AllowDirectConnection,
         }.Validated();
     }
 }