mrleo1nid/av-parser
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add a proxy pool with rotation, liveness checks and a management page

Browse Source 
The parser will need to move between proxies, so this adds the module it will
sit on: pluggable sources, a pool that hands proxies out and learns from the
outcome, three rotation strategies, and a page to drive it.

Sources are IProxySource implementations. The public proxifly/free-proxy-list
feed is fetched as the combined all/data.json through jsDelivr and filtered
locally — one conditional request beats four per-protocol ones that can disagree
mid-publish — and cached for the five minutes upstream takes to regenerate. A
feed that is down keeps serving its last payload rather than emptying the pool.
The user's own list lives in proxies.custom.json beside the settings, takes a
pasted blob, and names the lines it could not parse instead of quietly dropping
them.

Both knobs the pool exposes are settings, as asked: rotation is Sticky (default,
the only one that keeps site sessions coherent), RoundRobin or WeightedRandom;
liveness is either a parallel sweep of the whole pool or a probe at hand-out
time. Free lists are a few percent alive, so skipping verification entirely
means mostly waiting on timeouts.

Two invariants worth keeping, both of which cost a bug to find:

Availability is decided by the quarantine, never by Health. Excluding everything
that has ever failed made the quarantine window dead code and discarded proxies
permanently on their first hiccup, which is exactly wrong for addresses that
flap constantly. Health only orders the candidates now.

A probe verdict does not touch the success/failure counters. Those are about
real requests, and letting a sweep over a few thousand proxies rewrite them
would drown the evidence weighted selection reads.

SOCKS needs no extra package — .NET resolves socks4/socks4a/socks5 in WebProxy —
but a proxifly record with "protocol": "https" is still an HTTP proxy reached
over http:// with CONNECT, not an https:// scheme.

115 new tests. Also fixes a pre-existing flake: a command gated on another
command's IsExecuting cannot be driven straight after its Execute() completes,
because IsExecuting is published on the output scheduler.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Leonid Pershin
2026-08-13 17:22:30 +03:00
co-authored by Claude Opus 5
parent aeafe0af36
commit 9bf2ea5532
48 changed files with 4422 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/AvParser.UI.Tests/ProxiesViewModelTests.cs
+199
View File
@@ -0,0 +1,199 @@
using AvParser.Core.Proxies;
using AvParser.UI.Tests.Fakes;
using AvParser.UI.ViewModels;
using Microsoft.Extensions.Logging.Abstractions;
using ReactiveUI.Primitives.Concurrency;
namespace AvParser.UI.Tests;
public class ProxiesViewModelTests
{
private static (ProxiesViewModel Page, ProxyPool Pool, FakeMutableProxySource Custom) Build(params string[] hosts)
{
var custom = new FakeMutableProxySource();
var feed = new FakeProxySource(hosts.Select(host => Endpoint(host)));
var pool = new ProxyPool(
[feed, custom],
new FakeProxyProbe(),
new ProxyOptions(),
timeProvider: null,
random: new Random(1)
);
var page = new ProxiesViewModel(
pool,
custom,
NullLogger<ProxiesViewModel>.Instance,
ImmediateSequencer.Instance
);
return (page, pool, custom);
}
private static ProxyEndpoint Endpoint(string host, ProxyProtocol protocol = ProxyProtocol.Http) =>
new(protocol, host, 8080);
/// <summary>Runs a command to completion under the ambient test cancellation token.</summary>
private static Task Run<TResult>(ReactiveUI.ReactiveCommand<RxVoid, TResult> command) =>
command.Execute().ToTask(TestContext.Current.CancellationToken);
[Fact]
public void Starts_empty()
{
var (page, _, _) = Build();
page.Proxies.ShouldBeEmpty();
page.TotalCount.ShouldBe(0);
}
[Fact]
public async Task Refreshing_fills_the_list()
{
var (page, _, _) = Build("a", "b");
await Run(page.RefreshCommand);
page.Proxies.Count.ShouldBe(2);
page.TotalCount.ShouldBe(2);
page.StatusMessage.ShouldNotBeNull();
}
[Fact]
public async Task The_search_box_filters_by_address()
{
var (page, _, _) = Build("10.0.0.1", "10.0.0.2");
await Run(page.RefreshCommand);
page.SearchText = "10.0.0.2";
await Task.Delay(250, TestContext.Current.CancellationToken);
page.Proxies.ShouldHaveSingleItem().Address.ShouldContain("10.0.0.2");
}
[Fact]
public async Task The_protocol_filter_narrows_the_list()
{
var custom = new FakeMutableProxySource();
var feed = new FakeProxySource([Endpoint("http-one"), Endpoint("socks-one", ProxyProtocol.Socks5)]);
var pool = new ProxyPool([feed, custom], new FakeProxyProbe(), new ProxyOptions());
var page = new ProxiesViewModel(
pool,
custom,
NullLogger<ProxiesViewModel>.Instance,
ImmediateSequencer.Instance
);
await Run(page.RefreshCommand);
page.ProtocolFilter = ProxyProtocolFilter.Socks5;
await Task.Delay(250, TestContext.Current.CancellationToken);
page.Proxies.ShouldHaveSingleItem().Protocol.ShouldBe("SOCKS5");
}
[Fact]
public async Task Adding_custom_proxies_reports_what_it_could_not_parse()
{
var (page, _, custom) = Build();
page.NewProxies = "1.2.3.4:8080\nnot-a-proxy";
await Run(page.AddCustomCommand);
custom.Endpoints.Count.ShouldBe(1);
page.StatusMessage!.ShouldContain("not-a-proxy");
}
[Fact]
public async Task The_input_box_is_cleared_only_when_something_was_added()
{
var (page, _, _) = Build();
page.NewProxies = "not-a-proxy";
await Run(page.AddCustomCommand);
page.NewProxies.ShouldBe("not-a-proxy");
page.NewProxies = "1.2.3.4:8080";
await Run(page.AddCustomCommand);
page.NewProxies.ShouldBeEmpty();
}
[Fact]
public async Task Removing_is_offered_only_for_custom_entries()
{
var (page, _, _) = Build("feed-one");
await Run(page.RefreshCommand);
var canRemove = true;
using var subscription = page.RemoveSelectedCommand.CanExecute.Subscribe(value => canRemove = value);
page.SelectedProxy = page.Proxies.Single();
// Feed entries are republished upstream; removing one locally would be undone on the
// next refresh, so the command stays disabled.
canRemove.ShouldBeFalse();
}
[Fact]
public async Task A_custom_entry_can_be_removed()
{
var (page, _, custom) = Build();
page.NewProxies = "1.2.3.4:8080";
await Run(page.AddCustomCommand);
page.SelectedProxy = page.Proxies.Single();
page.SelectedProxy.IsCustom.ShouldBeTrue();
await Run(page.RemoveSelectedCommand);
custom.Endpoints.ShouldBeEmpty();
page.Proxies.ShouldBeEmpty();
}
[Fact]
public async Task Sweeping_updates_the_alive_count()
{
var custom = new FakeMutableProxySource();
var feed = new FakeProxySource([Endpoint("good"), Endpoint("bad")]);
var probe = new FakeProxyProbe();
probe.Set(Endpoint("good"), alive: true);
var pool = new ProxyPool([feed, custom], probe, new ProxyOptions());
var page = new ProxiesViewModel(
pool,
custom,
NullLogger<ProxiesViewModel>.Instance,
ImmediateSequencer.Instance
);
await Run(page.RefreshCommand);
await Run(page.SweepCommand);
page.AliveCount.ShouldBe(1);
page.IsSweeping.ShouldBeFalse();
page.StatusMessage!.ShouldContain("1 of 2");
}
[Fact]
public async Task Clearing_empties_the_custom_list()
{
var (page, _, custom) = Build();
page.NewProxies = "1.2.3.4:8080\n5.6.7.8:1080";
await Run(page.AddCustomCommand);
await Run(page.ClearCustomCommand);
custom.Endpoints.ShouldBeEmpty();
page.Proxies.ShouldBeEmpty();
}
[Fact]
public void Disposing_detaches_from_the_pool()
{
var (page, pool, _) = Build("a");
page.Dispose();
// The pool is a singleton; a page that stayed subscribed would be kept alive forever
// and would keep rebuilding its rows in the background.
Should.NotThrow(() => pool.Configure(new ProxyOptions()));
}
}