Gate network parsers on a working proxy and remember what worked

The pool now warms up from what the previous run learned instead of starting
cold every launch. Startup probes the remembered proxies first, stops as soon
as ProxyMinimumLive of them answer, and writes the survivors to
proxies.state.json after the warm-up and again on shutdown. Only proxies that
ever answered are stored: the feed republishes a few thousand dead addresses
every five minutes, and "was dead an hour ago" says almost nothing.

Remembered state is a hint, not a verdict. A restored proxy sorts first in the
warm-up queue but is not counted live until it answers in this session -
otherwise a launch a week later would report live proxies it had never spoken
to, the warm-up would skip the very entries it exists to re-check, and the
parser gate would open on week-old evidence.

That gate is the other half: a parser declaring RequiresNetwork will not run
while the pool has nothing live. The Parse page disables the run button and
shows a banner that leads to the Proxies page. Parsers that work on pasted text
are never gated - they have nothing to route, and blocking them would make the
app useless whenever the public lists are down. Two new settings cover the
escape hatch and the target: "allow network parsers without a proxy" and how
many live proxies to find at startup.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Leonid Pershin
2026-08-13 19:11:51 +03:00
co-authored by Claude Opus 5
parent 85656e70b0
commit 44fb0d3a5f
31 changed files with 1349 additions and 41 deletions
+31
View File
@@ -106,6 +106,37 @@ dotnet csharpier check .
- **Лиза без вердикта нейтральна.** Отменённая операция — не вина прокси; считать это отказом - **Лиза без вердикта нейтральна.** Отменённая операция — не вина прокси; считать это отказом
значит карантинить здоровые прокси на каждый Cancel. значит карантинить здоровые прокси на каждый Cancel.
- **`Select` и `Next` — зарезервированные слова для CA1716.** Метод стратегии называется `Pick`. - **`Select` и `Next` — зарезервированные слова для CA1716.** Метод стратегии называется `Pick`.
- **`LiveCount` считает только `Alive` и не в карантине.** На нём висит гейт парсера, поэтому
«доступна» (карантин истёк) и «живая» здесь намеренно расходятся: гейт не должен открываться
от одного лишь истечения окна.
- **Прогрев обрывается по достижении цели, а не проходит список до конца.** `WarmUpAsync` линкует
CTS и гасит остаток, как только набралось `MinimumLiveProxies`. Порядок кандидатов —
`WarmUpOrder()`, он публичный ровно затем, чтобы порядок проверялся без прогона проб.
- **`RestoreState` не выставляет `Health = Alive`.** «Работала вчера» живёт в отдельном
`WasAliveOnLastRun` и влияет только на порядок прогрева. Если восстанавливать как `Alive`, пул
отрапортует живыми тех, с кем не разговаривал: прогрев сочтёт цель достигнутой и не проверит
никого, а гейт парсера откроется по данным недельной давности. Ловит
`A_remembered_proxy_is_not_reported_live_until_it_answers_again`.
- **`RestoreState` не восстанавливает карантин.** Окно — стенные часы, между запусками могли пройти
сутки; перенос окна сажал бы прокси за то, что давно истекло.
- **Сохраняются только `HasEverAnswered`** = `SuccessCount > 0 || IsBelievedAlive`, где
«believed» = вердикт этой сессии, а без него — прошлой. Мёртвые в фиде исчисляются тысячами и
переиздаются каждые пять минут. Важен именно перенос: прогрев обрывается рано, поэтому
большинство запомненных заканчивают сессию непроверенными — строгое `Health == Alive` стирало бы
накопленный список за пару запусков.
## Гейт парсера
`IParser.RequiresNetwork` — дефолтная реализация возвращает `false`, поэтому добавление парсера
остаётся однострочным. Парсер, который куда-то ходит, **обязан** её переопределить, иначе поедет
напрямую в обход настройки.
Гейт живёт в `ParseViewModel.RefreshProxyGate()` и складывается из трёх условий: парсер сетевой,
`AllowDirectConnection` выключен, `LiveCount == 0`. Он пересчитывается по событию пула (с
throttle 250 мс — пул дёргается на каждый исход лизы), при смене парсера и при смене настроек.
`ParseView.axaml` держит баннер под `x:Name="ProxyGateBanner"`; `ParseViewTests` рендерит его
по-настоящему, потому что мёртвый биндинг `IsVisible` не ломает ни одного VM-теста.
## Добавить строку в UI ## Добавить строку в UI
+26
View File
@@ -134,6 +134,32 @@ Avalonia матчит **точный** тип, а `ShellView` наследует
Упавшая прокси уходит в карантин с экспоненциальным окном (30 с → 15 мин), но **не** удаляется Упавшая прокси уходит в карантин с экспоненциальным окном (30 с → 15 мин), но **не** удаляется
навсегда: бесплатные прокси постоянно мигают, и жёсткий бан терял бы их безвозвратно. навсегда: бесплатные прокси постоянно мигают, и жёсткий бан терял бы их безвозвратно.
### Что запоминается между запусками
Пул грузится и прогревается сам при старте, нажимать «Обновить» не нужно. Прогрев идёт
**от известного хорошего**: сначала пробуются те, что отвечали в прошлый раз, затем самые быстрые
из них, и проверка обрывается, как только набралось `ProxyMinimumLive` живых (по умолчанию 10).
Иначе каждый запуск был бы полным свипом по паре тысяч адресов ради десятка рабочих.
Запомненное — это подсказка, а не зачёт: восстановленная прокси идёт первой в очередь на проверку,
но живой не считается, пока не ответит в этом запуске. Иначе запуск через неделю открывал бы гейт
парсера по недельной давности данным, а прогрев пропускал бы ровно те прокси, ради которых он есть.
Состояние лежит в `proxies.state.json` рядом с настройками и пишется после прогрева и на выходе.
Сохраняются **только те прокси, что когда-либо отвечали**: мёртвых в фиде тысячи, они переиздаются
каждые пять минут, и «было мертво час назад» не говорит почти ничего. Карантин не восстанавливается
— окно отсчитывается по стенным часам, а между запусками могли пройти сутки.
### Гейт «без прокси не работаем»
Парсер, который объявил `RequiresNetwork`, не запустится, пока в пуле нет ни одной живой прокси:
кнопка «Разобрать» гаснет, а на странице появляется баннер с переходом на страницу Proxies.
Гейт снимается настройкой **«Разрешить сетевым парсерам работать без прокси»**.
Парсеры, читающие вставленный пользователем текст, не блокируются никогда — им нечего
маршрутизировать, и блокировка делала бы приложение бесполезным всякий раз, когда публичные списки
лежат.
Использование из кода: Использование из кода:
```csharp ```csharp
+11
View File
@@ -21,6 +21,17 @@ public interface IParser<in TInput, TOutput>
/// <summary>Cheap structural check — must not throw and must not do IO.</summary> /// <summary>Cheap structural check — must not throw and must not do IO.</summary>
bool CanParse(TInput input); bool CanParse(TInput input);
/// <summary>
/// Whether this parser makes network requests and therefore needs a working proxy.
/// </summary>
/// <remarks>
/// A default implementation rather than an abstract member, so adding a parser stays a
/// one-line change: a parser that works on text the user pasted opts out by saying nothing.
/// Parsers that fetch anything must set this, or they will run direct even when the user has
/// asked for proxy-only operation.
/// </remarks>
bool RequiresNetwork => false;
/// <summary>Streams one outcome per logical record.</summary> /// <summary>Streams one outcome per logical record.</summary>
IAsyncEnumerable<ParseOutcome<TOutput>> ParseAsync( IAsyncEnumerable<ParseOutcome<TOutput>> ParseAsync(
TInput input, TInput input,
+18
View File
@@ -9,6 +9,9 @@ public interface IProxyPool
/// <summary>Options currently in force.</summary> /// <summary>Options currently in force.</summary>
ProxyOptions Options { get; } ProxyOptions Options { get; }
/// <summary>How many entries are known to work and are not sidelined right now.</summary>
int LiveCount { get; }
/// <summary>Raised after the set of entries or their health changes.</summary> /// <summary>Raised after the set of entries or their health changes.</summary>
/// <remarks> /// <remarks>
/// A plain event rather than <c>IObservable</c> so the domain keeps no reactive dependency; /// A plain event rather than <c>IObservable</c> so the domain keeps no reactive dependency;
@@ -33,6 +36,21 @@ public interface IProxyPool
/// <summary>Probes every entry in parallel and updates their health.</summary> /// <summary>Probes every entry in parallel and updates their health.</summary>
/// <returns>How many answered.</returns> /// <returns>How many answered.</returns>
Task<int> SweepAsync(IProgress<ProxySweepProgress>? progress = null, CancellationToken cancellationToken = default); Task<int> SweepAsync(IProgress<ProxySweepProgress>? progress = null, CancellationToken cancellationToken = default);
/// <summary>
/// Probes best-known entries first and stops once <paramref name="targetLive"/> of them work.
/// </summary>
/// <returns>How many are live afterwards.</returns>
/// <remarks>
/// The startup counterpart to <see cref="SweepAsync"/>: proxies that answered on a previous
/// run are tried first, so a second launch usually confirms enough of them in a handful of
/// requests instead of re-probing a few thousand addresses.
/// </remarks>
Task<int> WarmUpAsync(
int targetLive,
IProgress<ProxySweepProgress>? progress = null,
CancellationToken cancellationToken = default
);
} }
/// <summary> /// <summary>
+59
View File
@@ -181,6 +181,65 @@ public sealed class ProxyEntry
} }
} }
/// <summary>Whether a previous run saw this proxy answer.</summary>
/// <remarks>
/// Kept apart from <see cref="Health"/> so that "worked yesterday" can order the warm-up
/// without being mistaken for "works now". Only this session's verdict counts as live.
/// </remarks>
public bool WasAliveOnLastRun { get; private set; }
/// <summary>
/// Reinstates what a previous run learned about this proxy.
/// </summary>
/// <remarks>
/// <para>
/// Health comes back as <see cref="ProxyHealthState.Unknown"/> even for a proxy that answered
/// last time. Restoring it as <see cref="ProxyHealthState.Alive"/> would make the pool report
/// live proxies it has not spoken to — the warm-up would then skip them as already-confirmed,
/// and a launch a week later would open the parser gate on week-old evidence.
/// </para>
/// <para>
/// Restores no quarantine either: the window is wall-clock and a restart may be days later, so
/// carrying it over would sideline proxies for reasons that have long expired.
/// </para>
/// </remarks>
public void RestoreState(
bool wasAlive,
TimeSpan? latency,
int successCount,
int failureCount,
DateTimeOffset? lastCheckedUtc
)
{
lock (_gate)
{
Health = ProxyHealthState.Unknown;
WasAliveOnLastRun = wasAlive;
Latency = latency;
SuccessCount = Math.Max(0, successCount);
FailureCount = Math.Max(0, failureCount);
LastCheckedUtc = lastCheckedUtc;
ConsecutiveFailures = 0;
QuarantinedUntilUtc = null;
}
}
/// <summary>Best current belief about whether this proxy answers.</summary>
/// <remarks>
/// This session's verdict wins; with none, what the previous run saw carries forward. Unlike
/// <see cref="ProxyPool.LiveCount"/> this is a belief, not a confirmation — never gate on it.
/// </remarks>
public bool IsBelievedAlive =>
Health switch
{
ProxyHealthState.Alive => true,
ProxyHealthState.Dead => false,
_ => WasAliveOnLastRun,
};
/// <summary>Whether this proxy ever answered, and is therefore worth remembering.</summary>
public bool HasEverAnswered => SuccessCount > 0 || IsBelievedAlive;
/// <inheritdoc /> /// <inheritdoc />
public override string ToString() => $"{Endpoint} [{Health}]"; public override string ToString() => $"{Endpoint} [{Health}]";
} }
+19
View File
@@ -103,6 +103,25 @@ public sealed record ProxyOptions
/// <summary>Whether the remote feed is consulted at all.</summary> /// <summary>Whether the remote feed is consulted at all.</summary>
public bool UseFeed { get; init; } = true; public bool UseFeed { get; init; } = true;
/// <summary>
/// How many working proxies a warm-up aims for before it stops probing.
/// </summary>
/// <remarks>
/// A free list holds thousands of addresses of which a few percent work. Probing all of them
/// at every launch costs thousands of requests for information that goes stale in minutes;
/// stopping once there are enough to rotate through is the useful part of that work.
/// </remarks>
public int MinimumLiveProxies { get; init; } = 10;
/// <summary>
/// Whether work may run without a proxy when none is available.
/// </summary>
/// <remarks>
/// Only parsers that declare <c>RequiresNetwork</c> are affected; a parser working on pasted
/// text is never blocked.
/// </remarks>
public bool AllowDirectConnection { get; init; }
/// <summary>Validates the options, throwing on values that would misbehave silently.</summary> /// <summary>Validates the options, throwing on values that would misbehave silently.</summary>
/// <exception cref="ArgumentOutOfRangeException">A numeric option is out of range.</exception> /// <exception cref="ArgumentOutOfRangeException">A numeric option is out of range.</exception>
public ProxyOptions Validated() public ProxyOptions Validated()
+123
View File
@@ -64,6 +64,20 @@ public sealed class ProxyPool : IProxyPool
} }
} }
/// <inheritdoc />
public int LiveCount
{
get
{
var now = _time.GetUtcNow();
lock (_gate)
{
return _entries.Count(entry => entry.Health == ProxyHealthState.Alive && entry.IsAvailable(now));
}
}
}
/// <inheritdoc /> /// <inheritdoc />
public event EventHandler? Changed; public event EventHandler? Changed;
@@ -268,6 +282,115 @@ public sealed class ProxyPool : IProxyPool
return aliveCount; return aliveCount;
} }
/// <inheritdoc />
public async Task<int> WarmUpAsync(
int targetLive,
IProgress<ProxySweepProgress>? progress = null,
CancellationToken cancellationToken = default
)
{
ArgumentOutOfRangeException.ThrowIfLessThan(targetLive, 1);
var options = Options;
var candidates = WarmUpOrder();
var total = candidates.Count;
if (total == 0 || LiveCount >= targetLive)
{
progress?.Report(new ProxySweepProgress(0, 0, LiveCount));
return LiveCount;
}
// Stops the remaining probes the moment the target is met. Linked so the caller's own
// cancellation still wins.
using var stop = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
using var limiter = new SemaphoreSlim(options.ProbeConcurrency, options.ProbeConcurrency);
var checkedCount = 0;
var work = candidates.Select(async entry =>
{
if (stop.IsCancellationRequested)
{
return;
}
try
{
await limiter.WaitAsync(stop.Token).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
return;
}
try
{
var result = await _probe.ProbeAsync(entry.Endpoint, options, stop.Token).ConfigureAwait(false);
entry.RecordProbe(
_time.GetUtcNow(),
result.Alive,
result.Latency,
result.Error,
quarantineOnFailure: options.BaseQuarantine
);
}
catch (OperationCanceledException)
{
// Either the target was reached or the caller gave up; neither is the proxy's fault.
return;
}
finally
{
limiter.Release();
}
var live = LiveCount;
progress?.Report(new ProxySweepProgress(Interlocked.Increment(ref checkedCount), total, live));
if (live >= targetLive)
{
await stop.CancelAsync().ConfigureAwait(false);
}
});
await Task.WhenAll(work).ConfigureAwait(false);
cancellationToken.ThrowIfCancellationRequested();
lock (_gate)
{
_strategy.Reset();
}
RaiseChanged();
return LiveCount;
}
/// <summary>
/// Orders candidates for a warm-up: what worked last time, then what looks most promising.
/// </summary>
/// <remarks>Public so the ordering can be asserted without running probes.</remarks>
public IReadOnlyList<ProxyEntry> WarmUpOrder()
{
var now = _time.GetUtcNow();
lock (_gate)
{
return
[
.. _entries
.Where(entry => entry.IsAvailable(now))
.OrderByDescending(entry => entry.Health == ProxyHealthState.Alive)
.ThenByDescending(entry => entry.WasAliveOnLastRun)
.ThenByDescending(entry => entry.SuccessCount > 0)
.ThenBy(entry => entry.Latency ?? TimeSpan.MaxValue)
.ThenByDescending(entry => entry.SuccessRate)
.ThenByDescending(entry => entry.Endpoint.Score),
];
}
}
/// <summary>Records the outcome of a lease. Called by <see cref="ProxyLease"/>.</summary> /// <summary>Records the outcome of a lease. Called by <see cref="ProxyLease"/>.</summary>
internal void ReportOutcome(ProxyEntry entry, bool success, TimeSpan? latency, string? error) internal void ReportOutcome(ProxyEntry entry, bool success, TimeSpan? latency, string? error)
{ {
+7 -1
View File
@@ -45,6 +45,8 @@ public enum AppTheme
/// <param name="ProxyProbeUrl">URL fetched to decide whether a proxy works.</param> /// <param name="ProxyProbeUrl">URL fetched to decide whether a proxy works.</param>
/// <param name="ProxyProbeTimeoutSeconds">Per-proxy probe timeout, in seconds.</param> /// <param name="ProxyProbeTimeoutSeconds">Per-proxy probe timeout, in seconds.</param>
/// <param name="ProxyProbeConcurrency">How many probes run at once during a pool sweep.</param> /// <param name="ProxyProbeConcurrency">How many probes run at once during a pool sweep.</param>
/// <param name="ProxyMinimumLive">How many working proxies a startup warm-up aims for.</param>
/// <param name="AllowDirectConnection">Whether network parsers may run without a proxy.</param>
public sealed record AppSettings( public sealed record AppSettings(
AppTheme Theme = AppTheme.System, AppTheme Theme = AppTheme.System,
AppLanguage Language = AppLanguage.System, AppLanguage Language = AppLanguage.System,
@@ -59,7 +61,9 @@ public sealed record AppSettings(
ProxyProtocolFilter ProxyProtocols = ProxyProtocolFilter.All, ProxyProtocolFilter ProxyProtocols = ProxyProtocolFilter.All,
string ProxyProbeUrl = "http://www.gstatic.com/generate_204", string ProxyProbeUrl = "http://www.gstatic.com/generate_204",
int ProxyProbeTimeoutSeconds = 8, int ProxyProbeTimeoutSeconds = 8,
int ProxyProbeConcurrency = 64 int ProxyProbeConcurrency = 64,
int ProxyMinimumLive = 10,
bool AllowDirectConnection = false
) )
{ {
/// <summary>Projects the proxy-related settings onto <see cref="ProxyOptions"/>.</summary> /// <summary>Projects the proxy-related settings onto <see cref="ProxyOptions"/>.</summary>
@@ -83,6 +87,8 @@ public sealed record AppSettings(
ProbeUrl = probeUrl, ProbeUrl = probeUrl,
ProbeTimeout = TimeSpan.FromSeconds(Math.Clamp(ProxyProbeTimeoutSeconds, 1, 120)), ProbeTimeout = TimeSpan.FromSeconds(Math.Clamp(ProxyProbeTimeoutSeconds, 1, 120)),
ProbeConcurrency = Math.Clamp(ProxyProbeConcurrency, 1, 512), ProbeConcurrency = Math.Clamp(ProxyProbeConcurrency, 1, 512),
MinimumLiveProxies = Math.Clamp(ProxyMinimumLive, 1, 500),
AllowDirectConnection = AllowDirectConnection,
}.Validated(); }.Validated();
} }
} }
+12 -2
View File
@@ -49,19 +49,29 @@ public partial class App : Application
_ = _services.GetRequiredService<ILocalizationService>(); _ = _services.GetRequiredService<ILocalizationService>();
_ = _services.GetRequiredService<IThemeService>(); _ = _services.GetRequiredService<IThemeService>();
var proxies = _services.GetRequiredService<IProxyPoolLoader>();
if (ApplicationLifetime is IClassicDesktopStyleApplicationLifetime desktop) if (ApplicationLifetime is IClassicDesktopStyleApplicationLifetime desktop)
{ {
var settings = _services.GetRequiredService<ISettingsService>(); var settings = _services.GetRequiredService<ISettingsService>();
var window = CreateMainWindow(settings); var window = CreateMainWindow(settings);
desktop.MainWindow = window; desktop.MainWindow = window;
desktop.ShutdownRequested += (_, _) => settings.FlushAsync().GetAwaiter().GetResult(); desktop.ShutdownRequested += (_, _) =>
{
settings.FlushAsync().GetAwaiter().GetResult();
// Everything learned during the session — which proxies answered, how fast — is
// only in memory until now. Writing it here is what makes the next launch start
// from the ones that worked instead of re-probing a few thousand addresses.
proxies.SaveStateAsync().GetAwaiter().GetResult();
};
} }
// Start filling the proxy pool as soon as the window is up. Not awaited on purpose: the // Start filling the proxy pool as soon as the window is up. Not awaited on purpose: the
// load is a network round trip, and blocking startup on a public list being reachable // load is a network round trip, and blocking startup on a public list being reachable
// would be the wrong trade. It never throws, so there is nothing to observe. // would be the wrong trade. It never throws, so there is nothing to observe.
_ = _services.GetRequiredService<IProxyPoolLoader>().EnsureLoadedAsync(); _ = proxies.EnsureLoadedAsync();
base.OnFrameworkInitializationCompleted(); base.OnFrameworkInitializationCompleted();
} }
@@ -75,6 +75,7 @@ public static class InfrastructureServiceCollectionExtensions
)); ));
services.AddSingleton<IProxiedHttpClientFactory, ProxiedHttpClientFactory>(); services.AddSingleton<IProxiedHttpClientFactory, ProxiedHttpClientFactory>();
services.AddSingleton<IProxyStateStore, ProxyStateStore>();
services.AddSingleton<IProxyPoolLoader, ProxyPoolLoader>(); services.AddSingleton<IProxyPoolLoader, ProxyPoolLoader>();
return services; return services;
@@ -3,35 +3,45 @@ using Microsoft.Extensions.Logging;
namespace AvParser.Infrastructure.Proxies; namespace AvParser.Infrastructure.Proxies;
/// <summary>Fills the pool once at startup, so nothing has to be loaded by hand.</summary> /// <summary>Outcome of the startup load.</summary>
/// <param name="Total">Entries in the pool.</param>
/// <param name="Live">Entries confirmed to work.</param>
/// <param name="Restored">Entries recognised from the previous run.</param>
public readonly record struct ProxyPoolLoadResult(int Total, int Live, int Restored);
/// <summary>Fills and warms the pool once at startup, so nothing has to be loaded by hand.</summary>
public interface IProxyPoolLoader public interface IProxyPoolLoader
{ {
/// <summary> /// <summary>
/// Loads every source, once per process. Later callers get the same operation rather than a /// Loads every source, restores what the previous run learned, and probes until there are
/// second download. /// enough working proxies. Runs once per process; later callers get the same operation.
/// </summary> /// </summary>
/// <returns>How many proxies the pool holds afterwards.</returns>
/// <remarks>Never throws: a source that is down leaves the pool as it was.</remarks> /// <remarks>Never throws: a source that is down leaves the pool as it was.</remarks>
Task<int> EnsureLoadedAsync(); Task<ProxyPoolLoadResult> EnsureLoadedAsync();
/// <summary>Whether the initial load has finished.</summary> /// <summary>Whether the initial load has finished.</summary>
bool IsLoaded { get; } bool IsLoaded { get; }
/// <summary>Writes the current pool state so the next launch can start from it.</summary>
Task SaveStateAsync(CancellationToken cancellationToken = default);
} }
/// <inheritdoc cref="IProxyPoolLoader" /> /// <inheritdoc cref="IProxyPoolLoader" />
public sealed class ProxyPoolLoader(IProxyPool pool, ILogger<ProxyPoolLoader> logger) : IProxyPoolLoader public sealed class ProxyPoolLoader(IProxyPool pool, IProxyStateStore stateStore, ILogger<ProxyPoolLoader> logger)
: IProxyPoolLoader
{ {
private readonly IProxyPool _pool = pool ?? throw new ArgumentNullException(nameof(pool)); private readonly IProxyPool _pool = pool ?? throw new ArgumentNullException(nameof(pool));
private readonly IProxyStateStore _stateStore = stateStore ?? throw new ArgumentNullException(nameof(stateStore));
private readonly ILogger<ProxyPoolLoader> _logger = logger ?? throw new ArgumentNullException(nameof(logger)); private readonly ILogger<ProxyPoolLoader> _logger = logger ?? throw new ArgumentNullException(nameof(logger));
private readonly Lock _gate = new(); private readonly Lock _gate = new();
private Task<int>? _load; private Task<ProxyPoolLoadResult>? _load;
/// <inheritdoc /> /// <inheritdoc />
public bool IsLoaded => _load is { IsCompleted: true }; public bool IsLoaded => _load is { IsCompleted: true };
/// <inheritdoc /> /// <inheritdoc />
public Task<int> EnsureLoadedAsync() public Task<ProxyPoolLoadResult> EnsureLoadedAsync()
{ {
if (_load is { } started) if (_load is { } started)
{ {
@@ -48,28 +58,53 @@ public sealed class ProxyPoolLoader(IProxyPool pool, ILogger<ProxyPoolLoader> lo
} }
} }
private async Task<int> LoadAsync() /// <inheritdoc />
public Task SaveStateAsync(CancellationToken cancellationToken = default) =>
_stateStore.SaveAsync(_pool.Entries, cancellationToken);
private async Task<ProxyPoolLoadResult> LoadAsync()
{ {
var options = _pool.Options;
try try
{ {
// Worth logging: an empty pool is almost always a filter or a switched-off feed // Worth logging: an empty pool is almost always a filter or a switched-off feed
// rather than a network problem, and without this it looks identical to both. // rather than a network problem, and without this it looks identical to both.
_logger.LogInformation( _logger.LogInformation(
"Loading proxy pool (feed: {UseFeed}, protocols: {Protocols})", "Loading proxy pool (feed: {UseFeed}, protocols: {Protocols}, target live: {Target})",
_pool.Options.UseFeed, options.UseFeed,
_pool.Options.Protocols options.Protocols,
options.MinimumLiveProxies
); );
var count = await _pool.RefreshAsync(CancellationToken.None).ConfigureAwait(false); var total = await _pool.RefreshAsync(CancellationToken.None).ConfigureAwait(false);
_logger.LogInformation("Proxy pool loaded with {Count} entries", count);
return count; var state = await _stateStore.LoadAsync(CancellationToken.None).ConfigureAwait(false);
var restored = ProxyStateStore.Apply(_pool.Entries, state);
// Warming up before anything asks for a proxy is the whole point: the first request
// should not be the thing that discovers the list is 97% dead.
var live = await _pool
.WarmUpAsync(options.MinimumLiveProxies, null, CancellationToken.None)
.ConfigureAwait(false);
_logger.LogInformation(
"Proxy pool ready: {Live} live of {Total} ({Restored} remembered)",
live,
total,
restored
);
await SaveStateAsync(CancellationToken.None).ConfigureAwait(false);
return new ProxyPoolLoadResult(total, live, restored);
} }
catch (Exception ex) catch (Exception ex)
{ {
// Startup must not fail because a public list is unreachable; the user can retry from // Startup must not fail because a public list is unreachable; the user can retry from
// the Proxies page, and the app works without proxies in the meantime. // the Proxies page, and the app works without proxies in the meantime.
_logger.LogWarning(ex, "Could not load the proxy pool at startup"); _logger.LogWarning(ex, "Could not load the proxy pool at startup");
return _pool.Entries.Count; return new ProxyPoolLoadResult(_pool.Entries.Count, _pool.LiveCount, 0);
} }
} }
} }
@@ -0,0 +1,149 @@
using System.Text.Json;
using System.Text.Json.Serialization;
using AvParser.Core.Proxies;
using AvParser.Infrastructure.Storage;
using Microsoft.Extensions.Logging;
namespace AvParser.Infrastructure.Proxies;
/// <summary>What a previous run learned about one proxy.</summary>
/// <param name="Address">Address in <c>scheme://host:port</c> form.</param>
/// <param name="Alive">Whether the last check succeeded.</param>
/// <param name="LatencyMs">Round-trip time of the last successful check.</param>
/// <param name="SuccessCount">Successful uses.</param>
/// <param name="FailureCount">Failed uses.</param>
/// <param name="LastCheckedUtc">When the state was last updated.</param>
public sealed record ProxyStateRecord(
string Address,
bool Alive = false,
double? LatencyMs = null,
int SuccessCount = 0,
int FailureCount = 0,
DateTimeOffset? LastCheckedUtc = null
);
/// <summary>Source-generated serialiser metadata for the remembered pool state.</summary>
[JsonSourceGenerationOptions(WriteIndented = true, PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase)]
[JsonSerializable(typeof(List<ProxyStateRecord>))]
internal sealed partial class ProxyStateJsonContext : JsonSerializerContext;
/// <summary>Remembers which proxies worked, so the next launch does not start from nothing.</summary>
public interface IProxyStateStore
{
/// <summary>Reads the remembered state, keyed by <see cref="ProxyEndpoint.Key"/>.</summary>
Task<IReadOnlyDictionary<string, ProxyStateRecord>> LoadAsync(CancellationToken cancellationToken = default);
/// <summary>Writes the entries worth remembering.</summary>
Task SaveAsync(IEnumerable<ProxyEntry> entries, CancellationToken cancellationToken = default);
}
/// <inheritdoc cref="IProxyStateStore" />
/// <remarks>
/// Only entries that have ever answered are stored. Remembering the dead ones would mean carrying
/// a few thousand records to save re-testing addresses that the feed republishes every five
/// minutes anyway — and a proxy that was dead an hour ago tells you very little about now.
/// </remarks>
public sealed class ProxyStateStore(IAppPaths paths, ILogger<ProxyStateStore> logger) : IProxyStateStore
{
private readonly IAppPaths _paths = paths ?? throw new ArgumentNullException(nameof(paths));
private readonly ILogger<ProxyStateStore> _logger = logger ?? throw new ArgumentNullException(nameof(logger));
/// <inheritdoc />
public async Task<IReadOnlyDictionary<string, ProxyStateRecord>> LoadAsync(
CancellationToken cancellationToken = default
)
{
try
{
if (!File.Exists(_paths.ProxyStateFile))
{
return new Dictionary<string, ProxyStateRecord>(StringComparer.Ordinal);
}
var json = await File.ReadAllTextAsync(_paths.ProxyStateFile, cancellationToken).ConfigureAwait(false);
var records = JsonSerializer.Deserialize(json, ProxyStateJsonContext.Default.ListProxyStateRecord) ?? [];
var byKey = new Dictionary<string, ProxyStateRecord>(records.Count, StringComparer.Ordinal);
foreach (var record in records)
{
if (ProxyEndpoint.TryParse(record.Address, out var endpoint))
{
byKey[endpoint.Key] = record;
}
}
_logger.LogInformation("Restored {Count} remembered proxies", byKey.Count);
return byKey;
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or JsonException)
{
_logger.LogWarning(ex, "Could not read {Path}; starting without remembered proxies", _paths.ProxyStateFile);
return new Dictionary<string, ProxyStateRecord>(StringComparer.Ordinal);
}
}
/// <inheritdoc />
public async Task SaveAsync(IEnumerable<ProxyEntry> entries, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(entries);
var records = entries
.Where(entry => entry.HasEverAnswered)
.Select(entry => new ProxyStateRecord(
entry.Endpoint.ToString(),
entry.IsBelievedAlive,
entry.Latency?.TotalMilliseconds,
entry.SuccessCount,
entry.FailureCount,
entry.LastCheckedUtc
))
.ToList();
try
{
Directory.CreateDirectory(Path.GetDirectoryName(_paths.ProxyStateFile)!);
var temp = _paths.ProxyStateFile + ".tmp";
var json = JsonSerializer.Serialize(records, ProxyStateJsonContext.Default.ListProxyStateRecord);
await File.WriteAllTextAsync(temp, json, cancellationToken).ConfigureAwait(false);
File.Move(temp, _paths.ProxyStateFile, overwrite: true);
_logger.LogDebug("Remembered {Count} proxies", records.Count);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
_logger.LogWarning(ex, "Could not write {Path}", _paths.ProxyStateFile);
}
}
/// <summary>Applies remembered state onto the pool's entries.</summary>
/// <returns>How many entries were recognised.</returns>
public static int Apply(IEnumerable<ProxyEntry> entries, IReadOnlyDictionary<string, ProxyStateRecord> state)
{
ArgumentNullException.ThrowIfNull(entries);
ArgumentNullException.ThrowIfNull(state);
var applied = 0;
foreach (var entry in entries)
{
if (!state.TryGetValue(entry.Endpoint.Key, out var record))
{
continue;
}
entry.RestoreState(
record.Alive,
record.LatencyMs is { } ms ? TimeSpan.FromMilliseconds(ms) : null,
record.SuccessCount,
record.FailureCount,
record.LastCheckedUtc
);
applied++;
}
return applied;
}
}
@@ -17,6 +17,10 @@ public interface IAppPaths
/// <remarks>Kept separate from the settings file: it is a list the user edits and may want to back up or share.</remarks> /// <remarks>Kept separate from the settings file: it is a list the user edits and may want to back up or share.</remarks>
string CustomProxiesFile { get; } string CustomProxiesFile { get; }
/// <summary>Full path of the remembered proxy state.</summary>
/// <remarks>Separate from the custom list: this one is derived data the app rewrites itself.</remarks>
string ProxyStateFile { get; }
/// <summary>Directory holding rolling log files.</summary> /// <summary>Directory holding rolling log files.</summary>
string LogDirectory { get; } string LogDirectory { get; }
} }
@@ -50,6 +54,7 @@ public sealed class AppPaths : IAppPaths
DataDirectory = dataDirectory; DataDirectory = dataDirectory;
SettingsFile = Path.Combine(dataDirectory, "settings.json"); SettingsFile = Path.Combine(dataDirectory, "settings.json");
CustomProxiesFile = Path.Combine(dataDirectory, "proxies.custom.json"); CustomProxiesFile = Path.Combine(dataDirectory, "proxies.custom.json");
ProxyStateFile = Path.Combine(dataDirectory, "proxies.state.json");
LogDirectory = Path.Combine(dataDirectory, "logs"); LogDirectory = Path.Combine(dataDirectory, "logs");
} }
@@ -62,6 +67,9 @@ public sealed class AppPaths : IAppPaths
/// <inheritdoc /> /// <inheritdoc />
public string CustomProxiesFile { get; } public string CustomProxiesFile { get; }
/// <inheritdoc />
public string ProxyStateFile { get; }
/// <inheritdoc /> /// <inheritdoc />
public string LogDirectory { get; } public string LogDirectory { get; }
@@ -34,6 +34,8 @@ public static class UiServiceCollectionExtensions
services.AddSingleton<ParseViewModel>(static sp => new ParseViewModel( services.AddSingleton<ParseViewModel>(static sp => new ParseViewModel(
sp.GetRequiredService<IParserCatalog>(), sp.GetRequiredService<IParserCatalog>(),
sp.GetRequiredService<ISettingsService>(), sp.GetRequiredService<ISettingsService>(),
sp.GetRequiredService<IProxyPool>(),
sp,
sp.GetRequiredService<ILogger<ParseViewModel>>() sp.GetRequiredService<ILogger<ParseViewModel>>()
)); ));
services.AddSingleton<SettingsViewModel>(static sp => new SettingsViewModel( services.AddSingleton<SettingsViewModel>(static sp => new SettingsViewModel(
+21
View File
@@ -487,4 +487,25 @@
<data name="Parse.Error.EmptyKey" xml:space="preserve"> <data name="Parse.Error.EmptyKey" xml:space="preserve">
<value>Key is empty.</value> <value>Key is empty.</value>
</data> </data>
<data name="Proxies.Status.Ready" xml:space="preserve">
<value>{0} in the pool, {1} live.</value>
</data>
<data name="Parse.ProxyRequired" xml:space="preserve">
<value>This parser needs a working proxy, and none is available. Check the proxy list, or allow direct connections in settings.</value>
</data>
<data name="Parse.GoToProxies" xml:space="preserve">
<value>Open proxies</value>
</data>
<data name="Settings.AllowDirect" xml:space="preserve">
<value>Allow network parsers to run without a proxy</value>
</data>
<data name="Settings.AllowDirectHint" xml:space="preserve">
<value>When off, a parser that fetches anything refuses to start until at least one proxy answers. Parsers that only read pasted text are never blocked.</value>
</data>
<data name="Settings.MinimumLive" xml:space="preserve">
<value>LIVE PROXIES TO FIND AT STARTUP</value>
</data>
<data name="Settings.MinimumLiveHint" xml:space="preserve">
<value>Startup checks the ones that worked last time first and stops as soon as it has this many. Raising it makes the first launch slower.</value>
</data>
</root> </root>
@@ -487,4 +487,25 @@
<data name="Parse.Error.EmptyKey" xml:space="preserve"> <data name="Parse.Error.EmptyKey" xml:space="preserve">
<value>Пустой ключ.</value> <value>Пустой ключ.</value>
</data> </data>
<data name="Proxies.Status.Ready" xml:space="preserve">
<value>В пуле {0}, живых {1}.</value>
</data>
<data name="Parse.ProxyRequired" xml:space="preserve">
<value>Этому парсеру нужна рабочая прокси, а живых нет. Проверьте список прокси или разрешите прямое подключение в настройках.</value>
</data>
<data name="Parse.GoToProxies" xml:space="preserve">
<value>Открыть прокси</value>
</data>
<data name="Settings.AllowDirect" xml:space="preserve">
<value>Разрешить сетевым парсерам работать без прокси</value>
</data>
<data name="Settings.AllowDirectHint" xml:space="preserve">
<value>Когда выключено, парсер, который куда-то ходит, не запустится, пока не ответит хотя бы одна прокси. Парсеры, читающие вставленный текст, не блокируются никогда.</value>
</data>
<data name="Settings.MinimumLive" xml:space="preserve">
<value>СКОЛЬКО ЖИВЫХ ИСКАТЬ ПРИ СТАРТЕ</value>
</data>
<data name="Settings.MinimumLiveHint" xml:space="preserve">
<value>При старте сначала проверяются те, что работали в прошлый раз, и проверка прекращается, как только набралось столько. Больше значение — дольше первый запуск.</value>
</data>
</root> </root>
+83 -4
View File
@@ -3,12 +3,16 @@ using System.Diagnostics;
using System.Globalization; using System.Globalization;
using System.Text; using System.Text;
using AvParser.Core.Parsing; using AvParser.Core.Parsing;
using AvParser.Core.Proxies;
using AvParser.Core.Settings; using AvParser.Core.Settings;
using AvParser.UI.Localization; using AvParser.UI.Localization;
using AvParser.UI.Navigation;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using ReactiveUI; using ReactiveUI;
using ReactiveUI.Primitives; using ReactiveUI.Primitives;
using ReactiveUI.Primitives.Concurrency; using ReactiveUI.Primitives.Concurrency;
using ReactiveUI.Primitives.Signals;
using ReactiveUI.SourceGenerators; using ReactiveUI.SourceGenerators;
namespace AvParser.UI.ViewModels; namespace AvParser.UI.ViewModels;
@@ -18,7 +22,7 @@ namespace AvParser.UI.ViewModels;
/// This page exists to exercise the whole <see cref="IParser{TInput,TOutput}"/> contract — /// This page exists to exercise the whole <see cref="IParser{TInput,TOutput}"/> contract —
/// streaming, progress and cancellation — rather than to be a finished feature. /// streaming, progress and cancellation — rather than to be a finished feature.
/// </remarks> /// </remarks>
public partial class ParseViewModel : PageViewModel public partial class ParseViewModel : PageViewModel, IDisposable
{ {
/// <summary>Records buffered before being pushed to the UI collection in one go.</summary> /// <summary>Records buffered before being pushed to the UI collection in one go.</summary>
private const int BatchSize = 512; private const int BatchSize = 512;
@@ -31,10 +35,14 @@ public partial class ParseViewModel : PageViewModel
private readonly IParserCatalog _catalog; private readonly IParserCatalog _catalog;
private readonly ISettingsService _settings; private readonly ISettingsService _settings;
private readonly IProxyPool _proxyPool;
private readonly IServiceProvider _services;
private readonly ILogger<ParseViewModel> _logger; private readonly ILogger<ParseViewModel> _logger;
private readonly ISequencer _mainThread; private readonly ISequencer _mainThread;
private readonly ObservableAsPropertyHelper<bool> _isBusy; private readonly ObservableAsPropertyHelper<bool> _isBusy;
private readonly Signal<RxVoid> _proxyChanged = new();
private CancellationTokenSource? _cancellation; private CancellationTokenSource? _cancellation;
/// <summary>Text to parse.</summary> /// <summary>Text to parse.</summary>
@@ -53,9 +61,21 @@ public partial class ParseViewModel : PageViewModel
[Reactive] [Reactive]
public partial string? StatusMessage { get; set; } public partial string? StatusMessage { get; set; }
/// <summary>
/// Whether the selected parser needs the network but has no working proxy to use.
/// </summary>
/// <remarks>
/// Only network parsers are gated. A parser that works on text the user pasted has nothing to
/// route, and blocking it would make the app unusable whenever the public lists are down.
/// </remarks>
[Reactive]
public partial bool IsBlockedWithoutProxy { get; set; }
/// <summary>Creates the page.</summary> /// <summary>Creates the page.</summary>
/// <param name="catalog">Available parsers.</param> /// <param name="catalog">Available parsers.</param>
/// <param name="settings">Used to remember the selected parser.</param> /// <param name="settings">Used to remember the selected parser.</param>
/// <param name="proxyPool">Consulted for the live count that gates network parsers.</param>
/// <param name="services">Resolves the navigation service lazily, to keep pages acyclic.</param>
/// <param name="logger">Diagnostics.</param> /// <param name="logger">Diagnostics.</param>
/// <param name="mainThread"> /// <param name="mainThread">
/// Scheduler used to marshal collection and progress updates back to the UI thread. Tests /// Scheduler used to marshal collection and progress updates back to the UI thread. Tests
@@ -64,12 +84,16 @@ public partial class ParseViewModel : PageViewModel
public ParseViewModel( public ParseViewModel(
IParserCatalog catalog, IParserCatalog catalog,
ISettingsService settings, ISettingsService settings,
IProxyPool proxyPool,
IServiceProvider services,
ILogger<ParseViewModel> logger, ILogger<ParseViewModel> logger,
ISequencer? mainThread = null ISequencer? mainThread = null
) )
{ {
_catalog = catalog ?? throw new ArgumentNullException(nameof(catalog)); _catalog = catalog ?? throw new ArgumentNullException(nameof(catalog));
_settings = settings ?? throw new ArgumentNullException(nameof(settings)); _settings = settings ?? throw new ArgumentNullException(nameof(settings));
_proxyPool = proxyPool ?? throw new ArgumentNullException(nameof(proxyPool));
_services = services ?? throw new ArgumentNullException(nameof(services));
_logger = logger ?? throw new ArgumentNullException(nameof(logger)); _logger = logger ?? throw new ArgumentNullException(nameof(logger));
_mainThread = mainThread ?? RxSchedulers.MainThreadScheduler; _mainThread = mainThread ?? RxSchedulers.MainThreadScheduler;
@@ -77,11 +101,30 @@ public partial class ParseViewModel : PageViewModel
Parsers = [.. catalog.Parsers.Select(parser => new ParserViewModel(parser))]; Parsers = [.. catalog.Parsers.Select(parser => new ParserViewModel(parser))];
SelectedParser = Parsers.First(parser => parser.Id == catalog.FindOrDefault(settings.Current.LastParserId).Id); SelectedParser = Parsers.First(parser => parser.Id == catalog.FindOrDefault(settings.Current.LastParserId).Id);
var canParse = this.WhenAnyValue(x => x.InputText) // The pool changes on every lease outcome and on every probe, so coalesce before
.Select(static text => !string.IsNullOrWhiteSpace(text)) // re-evaluating whether the parser is allowed to run.
_proxyPool.Changed += OnProxyPoolChanged;
_proxyChanged
.Throttle(TimeSpan.FromMilliseconds(250), _mainThread)
.ObserveOn(_mainThread)
.Subscribe(_ => RefreshProxyGate());
RefreshProxyGate();
var canParse = this.WhenAnyValue(
x => x.InputText,
x => x.IsBlockedWithoutProxy,
static (text, blocked) => (text, blocked)
)
.Select(static state => !string.IsNullOrWhiteSpace(state.text) && !state.blocked)
.DistinctUntilChanged(); .DistinctUntilChanged();
ParseCommand = ReactiveCommand.CreateFromTask(RunParseAsync, canParse, _mainThread); ParseCommand = ReactiveCommand.CreateFromTask(RunParseAsync, canParse, _mainThread);
GoToProxiesCommand = ReactiveCommand.Create(
() => _services.GetRequiredService<INavigationService>().NavigateTo<ProxiesViewModel>(),
outputScheduler: _mainThread
);
_isBusy = ParseCommand.IsExecuting.ToProperty(this, nameof(IsBusy), false, _mainThread); _isBusy = ParseCommand.IsExecuting.ToProperty(this, nameof(IsBusy), false, _mainThread);
CancelCommand = ReactiveCommand.Create(() => _cancellation?.Cancel(), ParseCommand.IsExecuting, _mainThread); CancelCommand = ReactiveCommand.Create(() => _cancellation?.Cancel(), ParseCommand.IsExecuting, _mainThread);
@@ -111,9 +154,17 @@ public partial class ParseViewModel : PageViewModel
); );
// Remember the parser choice; the debounced settings service coalesces the writes. // Remember the parser choice; the debounced settings service coalesces the writes.
// Switching parser can also change whether the gate applies, since only network parsers
// are gated.
this.WhenAnyValue(x => x.SelectedParser) this.WhenAnyValue(x => x.SelectedParser)
.Where(static parser => parser is not null) .Where(static parser => parser is not null)
.Subscribe(parser => _settings.Update(current => current with { LastParserId = parser.Id })); .Subscribe(parser =>
{
_settings.Update(current => current with { LastParserId = parser.Id });
RefreshProxyGate();
});
_settings.Changes.Subscribe(_ => RefreshProxyGate());
// Errors surfacing from any command must not tear the process down. // Errors surfacing from any command must not tear the process down.
ParseCommand.ThrownExceptions.Subscribe(OnCommandFailed); ParseCommand.ThrownExceptions.Subscribe(OnCommandFailed);
@@ -152,6 +203,23 @@ public partial class ParseViewModel : PageViewModel
/// <summary>Fills the input with 50 000 rows, so progress and cancellation are observable.</summary> /// <summary>Fills the input with 50 000 rows, so progress and cancellation are observable.</summary>
public ReactiveCommand<RxVoid, string> GenerateLargeSampleCommand { get; } public ReactiveCommand<RxVoid, string> GenerateLargeSampleCommand { get; }
/// <summary>Takes the user to the page where the proxy problem can be fixed.</summary>
public ReactiveCommand<RxVoid, RxVoid> GoToProxiesCommand { get; }
/// <summary>Explains why parsing is blocked.</summary>
public string ProxyRequiredMessage => Localizer.Instance["Parse.ProxyRequired"];
/// <summary>Re-evaluates the proxy gate. Exposed so tests can drive it without waiting.</summary>
public void RefreshProxyGate()
{
var settings = _settings.Current;
IsBlockedWithoutProxy =
SelectedParser.Parser.RequiresNetwork && !settings.AllowDirectConnection && _proxyPool.LiveCount == 0;
}
private void OnProxyPoolChanged(object? sender, EventArgs e) => _proxyChanged.OnNext(RxVoid.Default);
private async Task RunParseAsync(CancellationToken commandToken) private async Task RunParseAsync(CancellationToken commandToken)
{ {
using var cancellation = CancellationTokenSource.CreateLinkedTokenSource(commandToken); using var cancellation = CancellationTokenSource.CreateLinkedTokenSource(commandToken);
@@ -286,11 +354,22 @@ public partial class ParseViewModel : PageViewModel
}); });
} }
/// <inheritdoc />
/// <remarks>The pool is a singleton and would otherwise keep this page alive for the process.</remarks>
public void Dispose()
{
_proxyPool.Changed -= OnProxyPoolChanged;
_proxyChanged.Dispose();
GC.SuppressFinalize(this);
}
/// <inheritdoc /> /// <inheritdoc />
protected override void OnLanguageChanged() protected override void OnLanguageChanged()
{ {
base.OnLanguageChanged(); base.OnLanguageChanged();
this.RaisePropertyChanged(nameof(ProxyRequiredMessage));
// The summary and the listed errors were both rendered in the previous language. // The summary and the listed errors were both rendered in the previous language.
StatusMessage = null; StatusMessage = null;
@@ -195,10 +195,11 @@ public partial class ProxiesViewModel : PageViewModel, IDisposable
private async Task ReportInitialLoadAsync(IProxyPoolLoader loader) private async Task ReportInitialLoadAsync(IProxyPoolLoader loader)
{ {
var count = await loader.EnsureLoadedAsync().ConfigureAwait(false); var result = await loader.EnsureLoadedAsync().ConfigureAwait(false);
var message = Localizer.Instance.Format( var message = Localizer.Instance.Format(
"Proxies.Status.PoolHolds", "Proxies.Status.Ready",
Localizer.Instance.Plural("Proxies.Count.Proxies", count) Localizer.Instance.Plural("Proxies.Count.Proxies", result.Total),
result.Live
); );
// Anything the user has done since — an add, a check — is more interesting than the // Anything the user has done since — an add, a check — is more interesting than the
@@ -58,6 +58,14 @@ public partial class SettingsViewModel : PageViewModel
[Reactive] [Reactive]
public partial int ProxyProbeConcurrency { get; set; } public partial int ProxyProbeConcurrency { get; set; }
/// <summary>How many working proxies the startup warm-up aims for.</summary>
[Reactive]
public partial int ProxyMinimumLive { get; set; }
/// <summary>Whether network parsers may run with no proxy available.</summary>
[Reactive]
public partial bool AllowDirectConnection { get; set; }
/// <summary>Creates the page.</summary> /// <summary>Creates the page.</summary>
public SettingsViewModel( public SettingsViewModel(
ISettingsService settings, ISettingsService settings,
@@ -92,6 +100,8 @@ public partial class SettingsViewModel : PageViewModel
ProxyProbeUrl = current.ProxyProbeUrl; ProxyProbeUrl = current.ProxyProbeUrl;
ProxyProbeTimeoutSeconds = current.ProxyProbeTimeoutSeconds; ProxyProbeTimeoutSeconds = current.ProxyProbeTimeoutSeconds;
ProxyProbeConcurrency = current.ProxyProbeConcurrency; ProxyProbeConcurrency = current.ProxyProbeConcurrency;
ProxyMinimumLive = current.ProxyMinimumLive;
AllowDirectConnection = current.AllowDirectConnection;
this.WhenAnyValue(x => x.SelectedTheme).ObserveOn(scheduler).Subscribe(option => _theme.Apply(option.Value)); this.WhenAnyValue(x => x.SelectedTheme).ObserveOn(scheduler).Subscribe(option => _theme.Apply(option.Value));
@@ -116,7 +126,9 @@ public partial class SettingsViewModel : PageViewModel
x => x.ProxyProbeUrl, x => x.ProxyProbeUrl,
x => x.ProxyProbeTimeoutSeconds, x => x.ProxyProbeTimeoutSeconds,
x => x.ProxyProbeConcurrency, x => x.ProxyProbeConcurrency,
(_, _, _, _, _, _) => RxVoid.Default x => x.ProxyMinimumLive,
x => x.AllowDirectConnection,
(_, _, _, _, _, _, _, _) => RxVoid.Default
) )
.Throttle(TimeSpan.FromMilliseconds(200), scheduler) .Throttle(TimeSpan.FromMilliseconds(200), scheduler)
.ObserveOn(scheduler) .ObserveOn(scheduler)
@@ -222,6 +234,8 @@ public partial class SettingsViewModel : PageViewModel
ProxyProbeUrl = ProxyProbeUrl, ProxyProbeUrl = ProxyProbeUrl,
ProxyProbeTimeoutSeconds = ProxyProbeTimeoutSeconds, ProxyProbeTimeoutSeconds = ProxyProbeTimeoutSeconds,
ProxyProbeConcurrency = ProxyProbeConcurrency, ProxyProbeConcurrency = ProxyProbeConcurrency,
ProxyMinimumLive = ProxyMinimumLive,
AllowDirectConnection = AllowDirectConnection,
}; };
return applied; return applied;
+39 -3
View File
@@ -8,7 +8,7 @@
x:Class="AvParser.UI.Views.ParseView" x:Class="AvParser.UI.Views.ParseView"
x:DataType="vm:ParseViewModel" x:DataType="vm:ParseViewModel"
> >
<Grid RowDefinitions="Auto,Auto,*"> <Grid RowDefinitions="Auto,Auto,Auto,*">
<!-- ===== Toolbar ===== --> <!-- ===== Toolbar ===== -->
<Border Grid.Row="0" Classes="card" Margin="0,0,0,12"> <Border Grid.Row="0" Classes="card" Margin="0,0,0,12">
<StackPanel Spacing="12"> <StackPanel Spacing="12">
@@ -65,8 +65,44 @@
</StackPanel> </StackPanel>
</Border> </Border>
<!-- ===== Blocked without a proxy ===== -->
<Border
x:Name="ProxyGateBanner"
Grid.Row="1"
Classes="card"
Margin="0,0,0,12"
Background="{DynamicResource AppDangerSoftBrush}"
BorderBrush="{DynamicResource AppDangerBrush}"
IsVisible="{Binding IsBlockedWithoutProxy}"
>
<DockPanel LastChildFill="True">
<Button
DockPanel.Dock="Right"
Classes="primary"
Margin="16,0,0,0"
VerticalAlignment="Center"
Command="{Binding GoToProxiesCommand}"
>
<StackPanel Orientation="Horizontal" Spacing="8">
<PathIcon Classes="glyph" Data="{DynamicResource IconShield}" />
<TextBlock Text="{l:Loc Parse.GoToProxies}" />
</StackPanel>
</Button>
<StackPanel Orientation="Horizontal" Spacing="10" VerticalAlignment="Center">
<PathIcon
Classes="glyph"
Data="{DynamicResource IconAlert}"
Foreground="{DynamicResource AppDangerBrush}"
VerticalAlignment="Center"
/>
<TextBlock Text="{Binding ProxyRequiredMessage}" TextWrapping="Wrap" VerticalAlignment="Center" />
</StackPanel>
</DockPanel>
</Border>
<!-- ===== Progress and status ===== --> <!-- ===== Progress and status ===== -->
<StackPanel Grid.Row="1" Spacing="8" Margin="0,0,0,12"> <StackPanel Grid.Row="2" Spacing="8" Margin="0,0,0,12">
<ProgressBar <ProgressBar
Minimum="0" Minimum="0"
Maximum="1" Maximum="1"
@@ -83,7 +119,7 @@
</StackPanel> </StackPanel>
<!-- ===== Input and results ===== --> <!-- ===== Input and results ===== -->
<Grid Grid.Row="2" ColumnDefinitions="*,8,1.4*"> <Grid Grid.Row="3" ColumnDefinitions="*,8,1.4*">
<Border Grid.Column="0" Classes="card" Padding="0"> <Border Grid.Column="0" Classes="card" Padding="0">
<DockPanel LastChildFill="True"> <DockPanel LastChildFill="True">
<Border <Border
+17
View File
@@ -128,6 +128,23 @@
/> />
</StackPanel> </StackPanel>
</Grid> </Grid>
<StackPanel Spacing="6">
<TextBlock Classes="caption" Text="{l:Loc Settings.MinimumLive}" />
<NumericUpDown
Value="{Binding ProxyMinimumLive}"
Minimum="1"
Maximum="500"
Increment="5"
FormatString="0"
/>
<TextBlock Classes="muted" Text="{l:Loc Settings.MinimumLiveHint}" TextWrapping="Wrap" />
</StackPanel>
<StackPanel Spacing="6">
<CheckBox IsChecked="{Binding AllowDirectConnection}" Content="{l:Loc Settings.AllowDirect}" />
<TextBlock Classes="muted" Text="{l:Loc Settings.AllowDirectHint}" TextWrapping="Wrap" />
</StackPanel>
</StackPanel> </StackPanel>
</Border> </Border>
@@ -0,0 +1,105 @@
using AvParser.Core.Proxies;
namespace AvParser.Core.Tests.Proxies;
public class ProxyPoolWarmUpTests
{
private static readonly ProxyOptions Options = new() { ProbeConcurrency = 1 };
private static ProxyPool Build(out FakeProxyProbe probe, out FakeTimeProvider clock, params string[] hosts)
{
var source = new FakeProxySource(ProxySourceKind.Feed);
source.Endpoints.AddRange(hosts.Select(host => ProxyFactory.Endpoint(host)));
probe = new FakeProxyProbe();
clock = new FakeTimeProvider();
return new ProxyPool([source], probe, Options, clock);
}
[Fact]
public async Task What_worked_last_time_is_tried_first()
{
// The whole point of remembering: a second launch should confirm the known-good ones
// rather than walking a list of a few thousand addresses from the top.
var pool = Build(out var probe, out var clock, "cold-a", "known-good", "cold-b");
await pool.RefreshAsync(TestContext.Current.CancellationToken);
pool.Entries.Single(entry => entry.Endpoint.Host == "known-good")
.RestoreState(wasAlive: true, TimeSpan.FromMilliseconds(30), 12, 1, clock.GetUtcNow());
pool.WarmUpOrder()[0].Endpoint.Host.ShouldBe("known-good");
// Asking for two: one is already known live, so the warm-up has a reason to probe at all.
probe.DefaultAlive = true;
await pool.WarmUpAsync(2, cancellationToken: TestContext.Current.CancellationToken);
probe.Probed[0].ShouldContain("known-good");
}
[Fact]
public async Task The_faster_of_two_remembered_proxies_goes_first()
{
var pool = Build(out _, out var clock, "slow", "fast");
await pool.RefreshAsync(TestContext.Current.CancellationToken);
pool.Entries.Single(entry => entry.Endpoint.Host == "slow")
.RestoreState(wasAlive: true, TimeSpan.FromMilliseconds(900), 3, 0, clock.GetUtcNow());
pool.Entries.Single(entry => entry.Endpoint.Host == "fast")
.RestoreState(wasAlive: true, TimeSpan.FromMilliseconds(40), 3, 0, clock.GetUtcNow());
pool.WarmUpOrder().Select(entry => entry.Endpoint.Host).ShouldBe(["fast", "slow"]);
}
[Fact]
public async Task Warming_up_stops_once_the_target_is_met()
{
// Probing all 200 when 2 were asked for would turn every launch into a full sweep.
var pool = Build(out var probe, out _, [.. Enumerable.Range(0, 200).Select(i => $"h{i}")]);
await pool.RefreshAsync(TestContext.Current.CancellationToken);
probe.DefaultAlive = true;
var live = await pool.WarmUpAsync(2, cancellationToken: TestContext.Current.CancellationToken);
live.ShouldBeGreaterThanOrEqualTo(2);
probe.ProbeCount.ShouldBeLessThan(200);
}
[Fact]
public async Task Warming_up_probes_nothing_when_enough_are_already_live()
{
var pool = Build(out var probe, out var clock, "a", "b");
await pool.RefreshAsync(TestContext.Current.CancellationToken);
foreach (var entry in pool.Entries)
{
entry.RecordSuccess(clock.GetUtcNow(), TimeSpan.FromMilliseconds(10));
}
(await pool.WarmUpAsync(2, cancellationToken: TestContext.Current.CancellationToken)).ShouldBe(2);
probe.ProbeCount.ShouldBe(0);
}
[Fact]
public async Task A_quarantined_proxy_is_not_warmed_up()
{
var pool = Build(out _, out var clock, "a", "b");
await pool.RefreshAsync(TestContext.Current.CancellationToken);
pool.Entries.Single(entry => entry.Endpoint.Host == "a")
.RecordFailure(clock.GetUtcNow(), TimeSpan.FromMinutes(5), TimeSpan.FromHours(1), 1);
pool.WarmUpOrder().Select(entry => entry.Endpoint.Host).ShouldBe(["b"]);
}
[Fact]
public async Task Nothing_live_reads_as_nothing_live()
{
var pool = Build(out var probe, out _, "a", "b");
await pool.RefreshAsync(TestContext.Current.CancellationToken);
probe.DefaultAlive = false;
(await pool.WarmUpAsync(1, cancellationToken: TestContext.Current.CancellationToken)).ShouldBe(0);
pool.LiveCount.ShouldBe(0);
}
}
@@ -6,12 +6,15 @@ namespace AvParser.Infrastructure.Tests;
public class ProxyPoolLoaderTests public class ProxyPoolLoaderTests
{ {
private static ProxyPoolLoader Build(out CountingSource source, out ProxyPool pool) private static ProxyPoolLoader Build(out CountingSource source, out ProxyPool pool) =>
Build(out source, out pool, new MemoryStateStore());
private static ProxyPoolLoader Build(out CountingSource source, out ProxyPool pool, IProxyStateStore stateStore)
{ {
source = new CountingSource(); source = new CountingSource();
pool = new ProxyPool([source], new NeverProbe(), new ProxyOptions()); pool = new ProxyPool([source], new NeverProbe(), new ProxyOptions());
return new ProxyPoolLoader(pool, NullLogger<ProxyPoolLoader>.Instance); return new ProxyPoolLoader(pool, stateStore, NullLogger<ProxyPoolLoader>.Instance);
} }
[Fact] [Fact]
@@ -19,11 +22,42 @@ public class ProxyPoolLoaderTests
{ {
var loader = Build(out _, out var pool); var loader = Build(out _, out var pool);
(await loader.EnsureLoadedAsync()).ShouldBe(2); (await loader.EnsureLoadedAsync()).Total.ShouldBe(2);
pool.Entries.Count.ShouldBe(2); pool.Entries.Count.ShouldBe(2);
loader.IsLoaded.ShouldBeTrue(); loader.IsLoaded.ShouldBeTrue();
} }
[Fact]
public async Task What_the_previous_run_learned_is_restored_onto_the_fresh_list()
{
// The feed republishes the same addresses every few minutes; the point of remembering is
// that a proxy known to work is not re-discovered from scratch on every launch.
var stateStore = new MemoryStateStore
{
State =
{
["http://1.2.3.4:8080"] = new ProxyStateRecord("http://1.2.3.4:8080", Alive: true, LatencyMs: 120),
},
};
var loader = Build(out _, out _, stateStore);
var result = await loader.EnsureLoadedAsync();
result.Restored.ShouldBe(1);
}
[Fact]
public async Task The_pool_state_is_written_back_after_a_load()
{
var stateStore = new MemoryStateStore();
var loader = Build(out _, out _, stateStore);
await loader.EnsureLoadedAsync();
stateStore.Saves.ShouldBeGreaterThan(0);
}
[Fact] [Fact]
public async Task The_feed_is_fetched_once_however_many_callers_there_are() public async Task The_feed_is_fetched_once_however_many_callers_there_are()
{ {
@@ -52,10 +86,27 @@ public class ProxyPoolLoaderTests
public async Task A_source_that_throws_does_not_take_startup_down() public async Task A_source_that_throws_does_not_take_startup_down()
{ {
var pool = new ProxyPool([new ThrowingSource()], new NeverProbe(), new ProxyOptions()); var pool = new ProxyPool([new ThrowingSource()], new NeverProbe(), new ProxyOptions());
var loader = new ProxyPoolLoader(pool, NullLogger<ProxyPoolLoader>.Instance); var loader = new ProxyPoolLoader(pool, new MemoryStateStore(), NullLogger<ProxyPoolLoader>.Instance);
// The app has to start whether or not a public list is reachable. // The app has to start whether or not a public list is reachable.
(await loader.EnsureLoadedAsync()).ShouldBe(0); (await loader.EnsureLoadedAsync()).Total.ShouldBe(0);
}
private sealed class MemoryStateStore : IProxyStateStore
{
public Dictionary<string, ProxyStateRecord> State { get; } = new(StringComparer.Ordinal);
public int Saves { get; private set; }
public Task<IReadOnlyDictionary<string, ProxyStateRecord>> LoadAsync(
CancellationToken cancellationToken = default
) => Task.FromResult<IReadOnlyDictionary<string, ProxyStateRecord>>(State);
public Task SaveAsync(IEnumerable<ProxyEntry> entries, CancellationToken cancellationToken = default)
{
Saves++;
return Task.CompletedTask;
}
} }
private sealed class CountingSource : IProxySource private sealed class CountingSource : IProxySource
@@ -0,0 +1,166 @@
using AvParser.Core.Proxies;
using AvParser.Infrastructure.Proxies;
using AvParser.Infrastructure.Storage;
using Microsoft.Extensions.Logging.Abstractions;
namespace AvParser.Infrastructure.Tests;
public sealed class ProxyStateStoreTests : IDisposable
{
private readonly string _directory = Path.Combine(
Path.GetTempPath(),
"AvParserTests",
Guid.NewGuid().ToString("N")
);
private ProxyStateStore Create() => new(new AppPaths(_directory), NullLogger<ProxyStateStore>.Instance);
public void Dispose()
{
if (Directory.Exists(_directory))
{
Directory.Delete(_directory, recursive: true);
}
}
private static ProxyEntry Entry(string address, ProxySourceKind kind = ProxySourceKind.Feed)
{
ProxyEndpoint.TryParse(address, out var endpoint).ShouldBeTrue();
return new ProxyEntry(endpoint!, kind);
}
[Fact]
public async Task An_absent_file_reads_as_nothing_remembered()
{
var store = Create();
(await store.LoadAsync(TestContext.Current.CancellationToken)).ShouldBeEmpty();
}
[Fact]
public async Task A_working_proxy_survives_a_round_trip()
{
var alive = Entry("http://1.2.3.4:8080");
alive.RecordSuccess(DateTimeOffset.UtcNow, TimeSpan.FromMilliseconds(250));
await Create().SaveAsync([alive], TestContext.Current.CancellationToken);
var state = await Create().LoadAsync(TestContext.Current.CancellationToken);
state.ShouldContainKey(alive.Endpoint.Key);
state[alive.Endpoint.Key].Alive.ShouldBeTrue();
state[alive.Endpoint.Key].LatencyMs.ShouldBe(250d);
}
[Fact]
public async Task Proxies_that_never_answered_are_not_remembered()
{
// The feed republishes a few thousand dead addresses every few minutes; carrying them over
// would bloat the file to save re-testing entries whose staleness tells us nothing.
var untried = Entry("http://1.2.3.4:8080");
var dead = Entry("http://5.6.7.8:3128");
dead.RecordProbe(DateTimeOffset.UtcNow, alive: false, latency: null, error: "timeout");
await Create().SaveAsync([untried, dead], TestContext.Current.CancellationToken);
(await Create().LoadAsync(TestContext.Current.CancellationToken)).ShouldBeEmpty();
}
[Fact]
public void Applying_remembered_state_restores_the_matching_entries_only()
{
var known = Entry("http://1.2.3.4:8080");
var unknown = Entry("http://9.9.9.9:8080");
var state = new Dictionary<string, ProxyStateRecord>(StringComparer.Ordinal)
{
[known.Endpoint.Key] = new(
known.Endpoint.ToString(),
Alive: true,
LatencyMs: 120,
SuccessCount: 7,
FailureCount: 2
),
};
ProxyStateStore.Apply([known, unknown], state).ShouldBe(1);
known.WasAliveOnLastRun.ShouldBeTrue();
known.Latency.ShouldBe(TimeSpan.FromMilliseconds(120));
known.SuccessCount.ShouldBe(7);
unknown.WasAliveOnLastRun.ShouldBeFalse();
}
[Fact]
public void A_remembered_proxy_is_not_reported_live_until_it_answers_again()
{
// Otherwise a launch a week later would open the parser gate on week-old evidence, and the
// warm-up would skip the very proxies it was supposed to re-check.
var entry = Entry("http://1.2.3.4:8080");
var state = new Dictionary<string, ProxyStateRecord>(StringComparer.Ordinal)
{
[entry.Endpoint.Key] = new(entry.Endpoint.ToString(), Alive: true),
};
ProxyStateStore.Apply([entry], state);
entry.Health.ShouldBe(ProxyHealthState.Unknown);
entry.IsBelievedAlive.ShouldBeTrue();
}
[Fact]
public async Task A_remembered_proxy_that_was_never_re_checked_is_still_remembered()
{
// The warm-up stops early, so most remembered entries end a session unprobed. Dropping
// them on save would erode the remembered set to nothing over a few launches.
var entry = Entry("http://1.2.3.4:8080");
ProxyStateStore.Apply(
[entry],
new Dictionary<string, ProxyStateRecord>(StringComparer.Ordinal)
{
[entry.Endpoint.Key] = new(entry.Endpoint.ToString(), Alive: true),
}
);
await Create().SaveAsync([entry], TestContext.Current.CancellationToken);
(await Create().LoadAsync(TestContext.Current.CancellationToken)).ShouldContainKey(entry.Endpoint.Key);
}
[Fact]
public async Task A_remembered_proxy_that_fails_its_re_check_is_forgotten()
{
var entry = Entry("http://1.2.3.4:8080");
ProxyStateStore.Apply(
[entry],
new Dictionary<string, ProxyStateRecord>(StringComparer.Ordinal)
{
[entry.Endpoint.Key] = new(entry.Endpoint.ToString(), Alive: true),
}
);
entry.RecordProbe(DateTimeOffset.UtcNow, alive: false, latency: null, error: "timeout");
await Create().SaveAsync([entry], TestContext.Current.CancellationToken);
(await Create().LoadAsync(TestContext.Current.CancellationToken)).ShouldBeEmpty();
}
[Fact]
public void A_remembered_proxy_is_never_restored_into_a_quarantine()
{
// The window is wall-clock; a restart may be days later, so an expired sideline must not
// be resurrected — the whole point of remembering is to start from the good ones.
var entry = Entry("http://1.2.3.4:8080");
entry.RecordFailure(DateTimeOffset.UtcNow, TimeSpan.FromMinutes(5), TimeSpan.FromHours(1), 1);
entry.IsQuarantined(DateTimeOffset.UtcNow).ShouldBeTrue();
var restored = new Dictionary<string, ProxyStateRecord>(StringComparer.Ordinal)
{
[entry.Endpoint.Key] = new(entry.Endpoint.ToString(), Alive: true),
};
ProxyStateStore.Apply([entry], restored);
entry.IsQuarantined(DateTimeOffset.UtcNow).ShouldBeFalse();
}
}
+46
View File
@@ -1,5 +1,6 @@
using AvParser.Core.Proxies; using AvParser.Core.Proxies;
using AvParser.Core.Settings; using AvParser.Core.Settings;
using AvParser.Infrastructure.Proxies;
using AvParser.UI.Services; using AvParser.UI.Services;
using AvParser.UI.ViewModels; using AvParser.UI.ViewModels;
using ReactiveUI.Primitives.Signals; using ReactiveUI.Primitives.Signals;
@@ -41,6 +42,22 @@ internal sealed class FakeThemeService(AppTheme initial = AppTheme.System) : ITh
public void Dispose() => _current.Dispose(); public void Dispose() => _current.Dispose();
} }
/// <summary>In-memory settings, so tests never touch the developer's real profile.</summary>
internal sealed class FakeSettingsService(AppSettings? initial = null) : ISettingsService, IDisposable
{
private readonly BehaviorSignal<AppSettings> _current = new(initial ?? new AppSettings());
public AppSettings Current => _current.Value;
public IObservable<AppSettings> Changes => _current;
public void Update(Func<AppSettings, AppSettings> mutate) => _current.OnNext(mutate(_current.Value));
public Task FlushAsync(CancellationToken cancellationToken = default) => Task.CompletedTask;
public void Dispose() => _current.Dispose();
}
/// <summary>An editable proxy list held in memory.</summary> /// <summary>An editable proxy list held in memory.</summary>
internal sealed class FakeMutableProxySource : IMutableProxySource internal sealed class FakeMutableProxySource : IMutableProxySource
{ {
@@ -80,3 +97,32 @@ internal sealed class FakeProxyProbe : IProxyProbe
CancellationToken cancellationToken = default CancellationToken cancellationToken = default
) => Task.FromResult(ProxyProbeResult.Failure("not probed in tests")); ) => Task.FromResult(ProxyProbeResult.Failure("not probed in tests"));
} }
/// <summary>A remembered-state store that keeps everything in memory.</summary>
internal sealed class FakeProxyStateStore : IProxyStateStore
{
public Dictionary<string, ProxyStateRecord> State { get; } = new(StringComparer.Ordinal);
public Task<IReadOnlyDictionary<string, ProxyStateRecord>> LoadAsync(
CancellationToken cancellationToken = default
) => Task.FromResult<IReadOnlyDictionary<string, ProxyStateRecord>>(State);
public Task SaveAsync(IEnumerable<ProxyEntry> entries, CancellationToken cancellationToken = default) =>
Task.CompletedTask;
}
/// <summary>
/// A loader that does nothing.
/// </summary>
/// <remarks>
/// The real one probes on startup, which would race these view tests: the load is fired from the
/// page constructor and would mark every fake proxy dead partway through an assertion.
/// </remarks>
internal sealed class FakeProxyPoolLoader : IProxyPoolLoader
{
public bool IsLoaded => true;
public Task<ProxyPoolLoadResult> EnsureLoadedAsync() => Task.FromResult(new ProxyPoolLoadResult(0, 0, 0));
public Task SaveStateAsync(CancellationToken cancellationToken = default) => Task.CompletedTask;
}
@@ -0,0 +1,121 @@
using Avalonia.Controls;
using Avalonia.Headless.XUnit;
using Avalonia.Threading;
using Avalonia.VisualTree;
using AvParser.Core.Parsing;
using AvParser.Core.Parsing.Samples;
using AvParser.Core.Proxies;
using AvParser.Core.Settings;
using AvParser.UI.ViewModels;
using AvParser.UI.Views;
using Microsoft.Extensions.Logging.Abstractions;
using ReactiveUI.Primitives.Concurrency;
namespace AvParser.UI.HeadlessTests;
public class ParseViewTests
{
private static (ParseView View, ParseViewModel ViewModel, Window Window) ShowPage(params ITextParser[] extra)
{
var catalog = new ParserCatalog([new DelimitedTextParser(), new KeyValueTextParser(), .. extra]);
var lastParser = extra.Length > 0 ? extra[0].Id : null;
var viewModel = new ParseViewModel(
catalog,
new FakeSettingsService(new AppSettings { LastParserId = lastParser }),
new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
new EmptyServiceProvider(),
NullLogger<ParseViewModel>.Instance,
ImmediateSequencer.Instance
);
var view = new ParseView { DataContext = viewModel };
var window = new Window
{
Width = 1400,
Height = 900,
Content = view,
};
window.Show();
Dispatcher.UIThread.RunJobs();
return (view, viewModel, window);
}
private static Border Banner(ParseView view) => view.FindControl<Border>("ProxyGateBanner").ShouldNotBeNull();
[AvaloniaFact]
public void No_banner_is_shown_for_a_parser_that_needs_no_network()
{
var (view, viewModel, _) = ShowPage();
viewModel.IsBlockedWithoutProxy.ShouldBeFalse();
Banner(view).IsVisible.ShouldBeFalse();
}
[AvaloniaFact]
public void A_blocked_network_parser_puts_the_banner_on_screen()
{
// Rendered rather than asserted on the view model: an IsVisible binding that never fires
// leaves the page silently unhelpful, which is exactly the failure this guards.
var (view, viewModel, _) = ShowPage(new NetworkParser());
Dispatcher.UIThread.RunJobs();
viewModel.IsBlockedWithoutProxy.ShouldBeTrue();
Banner(view).IsEffectivelyVisible.ShouldBeTrue();
}
[AvaloniaFact]
public void The_banner_offers_a_way_to_the_proxies_page()
{
var (view, viewModel, _) = ShowPage(new NetworkParser());
Dispatcher.UIThread.RunJobs();
var button = Banner(view).GetVisualDescendants().OfType<Button>().ShouldHaveSingleItem();
button.Command.ShouldBeSameAs(viewModel.GoToProxiesCommand);
}
[AvaloniaFact]
public void A_blocked_page_will_not_run_the_parser()
{
var (view, viewModel, _) = ShowPage(new NetworkParser());
viewModel.InputText = "anything";
Dispatcher.UIThread.RunJobs();
var run = view.GetVisualDescendants()
.OfType<Button>()
.First(candidate => ReferenceEquals(candidate.Command, viewModel.ParseCommand));
run.IsEffectivelyEnabled.ShouldBeFalse();
}
private sealed class EmptyServiceProvider : IServiceProvider
{
public object? GetService(Type serviceType) => null;
}
private sealed class NetworkParser : ITextParser
{
public string Id => "network";
public string DisplayName => "Network parser";
public string Description => "Fetches something";
public bool RequiresNetwork => true;
public bool CanParse(string input) => true;
public async IAsyncEnumerable<ParseOutcome<ParsedRecord>> ParseAsync(
string input,
IProgress<ParseProgress>? progress,
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken
)
{
await Task.Yield();
yield break;
}
}
}
@@ -24,7 +24,7 @@ public class ProxiesViewTests
var viewModel = new ProxiesViewModel( var viewModel = new ProxiesViewModel(
pool, pool,
custom, custom,
new ProxyPoolLoader(pool, NullLogger<ProxyPoolLoader>.Instance), new FakeProxyPoolLoader(),
NullLogger<ProxiesViewModel>.Instance, NullLogger<ProxiesViewModel>.Instance,
ImmediateSequencer.Instance ImmediateSequencer.Instance
); );
@@ -72,6 +72,8 @@ public class ViewLocatorTests
public string CustomProxiesFile => Path.Combine(DataDirectory, "proxies.custom.json"); public string CustomProxiesFile => Path.Combine(DataDirectory, "proxies.custom.json");
public string ProxyStateFile => Path.Combine(DataDirectory, "proxies.state.json");
public string LogDirectory => Path.Combine(DataDirectory, "logs"); public string LogDirectory => Path.Combine(DataDirectory, "logs");
} }
} }
@@ -1,7 +1,26 @@
using AvParser.Core.Proxies; using AvParser.Core.Proxies;
using AvParser.Infrastructure.Proxies;
namespace AvParser.UI.Tests.Fakes; namespace AvParser.UI.Tests.Fakes;
/// <summary>A remembered-state store that keeps everything in memory.</summary>
internal sealed class FakeProxyStateStore : IProxyStateStore
{
public Dictionary<string, ProxyStateRecord> State { get; } = new(StringComparer.Ordinal);
public int Saves { get; private set; }
public Task<IReadOnlyDictionary<string, ProxyStateRecord>> LoadAsync(
CancellationToken cancellationToken = default
) => Task.FromResult<IReadOnlyDictionary<string, ProxyStateRecord>>(State);
public Task SaveAsync(IEnumerable<ProxyEntry> entries, CancellationToken cancellationToken = default)
{
Saves++;
return Task.CompletedTask;
}
}
/// <summary>A feed whose contents the test supplies.</summary> /// <summary>A feed whose contents the test supplies.</summary>
internal sealed class FakeProxySource(IEnumerable<ProxyEndpoint>? endpoints = null) : IProxySource internal sealed class FakeProxySource(IEnumerable<ProxyEndpoint>? endpoints = null) : IProxySource
{ {
+113 -2
View File
@@ -1,5 +1,6 @@
using AvParser.Core.Parsing; using AvParser.Core.Parsing;
using AvParser.Core.Parsing.Samples; using AvParser.Core.Parsing.Samples;
using AvParser.Core.Proxies;
using AvParser.Core.Settings; using AvParser.Core.Settings;
using AvParser.UI.Tests.Fakes; using AvParser.UI.Tests.Fakes;
using AvParser.UI.ViewModels; using AvParser.UI.ViewModels;
@@ -11,15 +12,21 @@ namespace AvParser.UI.Tests;
public class ParseViewModelTests public class ParseViewModelTests
{ {
private static (ParseViewModel Page, FakeSettingsService Settings) Build(AppSettings? settings = null) private static (ParseViewModel Page, FakeSettingsService Settings) Build(
AppSettings? settings = null,
IProxyPool? proxyPool = null,
params ITextParser[] extraParsers
)
{ {
var catalog = new ParserCatalog([new DelimitedTextParser(), new KeyValueTextParser()]); var catalog = new ParserCatalog([new DelimitedTextParser(), new KeyValueTextParser(), .. extraParsers]);
var settingsService = new FakeSettingsService(settings); var settingsService = new FakeSettingsService(settings);
return ( return (
new ParseViewModel( new ParseViewModel(
catalog, catalog,
settingsService, settingsService,
proxyPool ?? new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
new EmptyServiceProvider(),
NullLogger<ParseViewModel>.Instance, NullLogger<ParseViewModel>.Instance,
ImmediateSequencer.Instance ImmediateSequencer.Instance
), ),
@@ -27,6 +34,36 @@ public class ParseViewModelTests
); );
} }
/// <summary>A container that resolves nothing; the page only reaches for navigation on click.</summary>
private sealed class EmptyServiceProvider : IServiceProvider
{
public object? GetService(Type serviceType) => null;
}
/// <summary>Stands in for the kind of parser the proxy gate exists for.</summary>
private sealed class NetworkParser : ITextParser
{
public string Id => "network";
public string DisplayName => "Network parser";
public string Description => "Fetches something";
public bool RequiresNetwork => true;
public bool CanParse(string input) => true;
public async IAsyncEnumerable<ParseOutcome<ParsedRecord>> ParseAsync(
string input,
IProgress<ParseProgress>? progress,
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken
)
{
await Task.Yield();
yield break;
}
}
private static Task RunAsync(ParseViewModel page) => page.ParseCommand.Execute().ToTask(); private static Task RunAsync(ParseViewModel page) => page.ParseCommand.Execute().ToTask();
/// <summary> /// <summary>
@@ -67,6 +104,80 @@ public class ParseViewModelTests
settings.Current.LastParserId.ShouldBe("key-value"); settings.Current.LastParserId.ShouldBe("key-value");
} }
[Fact]
public void A_local_parser_runs_even_with_no_proxy_at_all()
{
// Only network parsers are gated. Blocking a parser that works on pasted text would make
// the app unusable whenever the public lists are down, for no benefit at all.
var (page, _) = Build();
page.IsBlockedWithoutProxy.ShouldBeFalse();
}
[Fact]
public void A_network_parser_is_blocked_while_nothing_is_live()
{
var (page, _) = Build(
new AppSettings { LastParserId = "network" },
new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
new NetworkParser()
);
var canExecute = true;
using var subscription = page.ParseCommand.CanExecute.Subscribe(value => canExecute = value);
page.IsBlockedWithoutProxy.ShouldBeTrue();
page.InputText = "anything";
canExecute.ShouldBeFalse();
}
[Fact]
public async Task A_network_parser_runs_once_a_proxy_answers()
{
var endpoint = new ProxyEndpoint(ProxyProtocol.Http, "1.2.3.4", 8080);
var pool = new ProxyPool(
[new FakeProxySource([endpoint])],
new FakeProxyProbe().Set(endpoint, alive: true),
new ProxyOptions()
);
var (page, _) = Build(new AppSettings { LastParserId = "network" }, pool, new NetworkParser());
page.IsBlockedWithoutProxy.ShouldBeTrue();
await pool.RefreshAsync(TestContext.Current.CancellationToken);
await pool.WarmUpAsync(1, cancellationToken: TestContext.Current.CancellationToken);
page.RefreshProxyGate();
page.IsBlockedWithoutProxy.ShouldBeFalse();
}
[Fact]
public void Allowing_direct_connections_lifts_the_gate()
{
var (page, _) = Build(
new AppSettings { LastParserId = "network", AllowDirectConnection = true },
new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
new NetworkParser()
);
page.IsBlockedWithoutProxy.ShouldBeFalse();
}
[Fact]
public void Switching_away_from_a_network_parser_lifts_the_gate()
{
var (page, _) = Build(
new AppSettings { LastParserId = "network" },
new ProxyPool([], new FakeProxyProbe(), new ProxyOptions()),
new NetworkParser()
);
page.SelectedParser = page.Parsers.Single(parser => parser.Id == "delimited");
page.IsBlockedWithoutProxy.ShouldBeFalse();
}
[Theory] [Theory]
[InlineData("", false)] [InlineData("", false)]
[InlineData(" ", false)] [InlineData(" ", false)]
@@ -24,7 +24,7 @@ public class ProxiesViewModelTests
var page = new ProxiesViewModel( var page = new ProxiesViewModel(
pool, pool,
custom, custom,
new ProxyPoolLoader(pool, NullLogger<ProxyPoolLoader>.Instance), new ProxyPoolLoader(pool, new FakeProxyStateStore(), NullLogger<ProxyPoolLoader>.Instance),
NullLogger<ProxiesViewModel>.Instance, NullLogger<ProxiesViewModel>.Instance,
ImmediateSequencer.Instance ImmediateSequencer.Instance
); );
@@ -110,7 +110,7 @@ public class ProxiesViewModelTests
var page = new ProxiesViewModel( var page = new ProxiesViewModel(
pool, pool,
custom, custom,
new ProxyPoolLoader(pool, NullLogger<ProxyPoolLoader>.Instance), new ProxyPoolLoader(pool, new FakeProxyStateStore(), NullLogger<ProxyPoolLoader>.Instance),
NullLogger<ProxiesViewModel>.Instance, NullLogger<ProxiesViewModel>.Instance,
ImmediateSequencer.Instance ImmediateSequencer.Instance
); );
@@ -192,7 +192,7 @@ public class ProxiesViewModelTests
var page = new ProxiesViewModel( var page = new ProxiesViewModel(
pool, pool,
custom, custom,
new ProxyPoolLoader(pool, NullLogger<ProxyPoolLoader>.Instance), new ProxyPoolLoader(pool, new FakeProxyStateStore(), NullLogger<ProxyPoolLoader>.Instance),
NullLogger<ProxiesViewModel>.Instance, NullLogger<ProxiesViewModel>.Instance,
ImmediateSequencer.Instance ImmediateSequencer.Instance
); );