Batch RealDelay probes through an isolated Observatory core

[eliotcougar]

I would like to raise awareness to the issue with running bulk real-delay probes one xray-core per outbound inside the same process as the main core. Last time this proposal has been bundled together with some unrelated features. I tried to separate it cleanly, patched some edge cases (after I saw that guy with 500+ profiles in his app), cleaned-up the code in general, and collected some performance data. Hopefully, this time it will be considered when you have time for it.

Summary

This PR replaces the per-profile RealDelay hot path with a probe plan that runs compatible targets through one short-lived Xray core in a process separate from the VPN daemon.

It preserves progressive results, policy-group selection, cancellation, malformed-profile isolation, the existing RealDelay concurrency setting, subscription auto-test behavior, and fallback support for configurations that cannot be merged.

It depends on AndroidLibXrayLite draft PR 2dust/AndroidLibXrayLite#204.

Existing problem

The current RealDelay implementation creates one coroutine per profile on a limited dispatcher. For each ordinary profile it performs a raw TCP precheck, builds a complete configuration, creates/starts/closes a separate Xray core, and takes two GET samples.

For a large subscription this has four costs:

1. Hundreds of repeated configuration builds and Xray lifecycle operations.
2. A raw TCP precheck plus two HTTP requests per profile, even though the requested result is one RealDelay value.
3. One identity-less success notification per result, causing the UI to invalidate and rebuild the complete group repeatedly up to N^2 times.
4. Multiple temporary core.Instance values can overlap inside the same native process, including the process hosting the long-running VPN core.

Why overlapping Xray cores in one process are unsafe

The Xray user documentation describes operational Xray instances/processes as independent, and its architecture diagram covers a single Xray process. The embedded Go API has additional process-wide state.

XTLS's libXray embedding documentation explicitly warns that Xray-core keeps its system dialer DNS client and outbound manager in process-wide state. Creating a temporary ping/test core can replace that state; closing the temporary core does not restore the previous values; overlapping instances therefore need separate processes.

The exact xray-core revision pinned by this branch (5ca6f4b7d4dc) confirms it:

• Every core.New / core.NewWithContext calls internet.InitSystemDialer during instance construction.
• dnsClient and obm are package globals.
• InitSystemDialer overwrites them; the global outbound manager is also used for dialerProxy resolution.
• The effective system dialer is process-global, and its setter says the caller must ensure there is no race.
• Instance.Close closes the instance's features but does not restore the managers previously stored in those package globals.

A temporary test core can therefore redirect DNS or dialerProxy lookup through another instance's managers, leave the long-running core with replaced process state, or leave globals pointing at features that have already closed.

This PR changes CoreTestService from :RunSoLibV2RayDaemon to a disposable :Probe process. Interactive batches run one at a time, the process terminates after the batch, and a back-to-back replacement request is redelivered to a fresh process. Subscription update/testing also moves to :SubscriptionUpdate, separate from the VPN daemon.

Process isolation protects the live tunnel even when an unmergeable profile must use the legacy per-profile fallback. The optimized path goes further and avoids overlapping temporary cores by combining compatible targets into one Xray instance.

Proposed flow

selected profile GUIDs
        |
        v
cached profile/subscription lookup
        |
        v
minimal outbound dependency graphs
        |
        +---- malformed/custom/unsupported ----> bounded per-profile fallback
        |
        v
namespaced combined config + Burst Observatory
        |
        v
one Xray instance in :Probe / :SubscriptionUpdate
        |
        v
fixed native worker pool over physical targets
        |
        v
GUID + delay progressive results
        |
        v
500 ms UI coalescing + immediate final flush

Probe-plan construction

CoreConfigContextBuilder now has a probe-specific path that resolves only the outbound dependency graph required for testing. It skips unrelated routing/DNS/runtime sections and shares a lazily decoded profile/subscription snapshot across the complete batch.

For every compatible source configuration, ProbeConfigBuilder:

• namespaces every outbound and balancer tag to prevent collisions;
• rewrites streamSettings.sockopt.dialerProxy references to the namespaced dependency;
• validates every reference before adding any part of a source to the combined config;
• supports ordinary generated profiles and least-ping/least-load policy groups;
• expands policy groups into physical candidate targets;
• adds one Burst Observatory configured for HEAD, one sample, and a five-second timeout.

Custom Xray JSON, malformed dependency graphs, unsupported balancer strategies, and policy groups with unsupported fallback semantics are kept out of the combined config and tested through the individual compatibility path.

If Xray rejects a combined configuration despite per-source validation, the app recursively divides only the still-pending profile set. Valid subsets continue through shared cores; an irreducible single profile falls back individually. One bad profile therefore cannot discard the rest of the batch.

Concurrency and policy groups

The existing Concurrent RealDelay tests setting is passed to one native worker pool. Each worker performs one unchanged upstream BurstObservatory.Check for one outbound tag.

• A policy-group member consumes one worker slot, just like an ordinary profile.
• Total active checks within the batch never exceed the setting.
• Targets from different groups are interleaved so one large policy group cannot monopolize the queue.
• Every target receives exactly one sample, independent of least-load history/sampling settings.
• TCP-only testing retains a limited coroutine dispatcher because it does not create Xray instances.
• The shared-core phase and individual fallback phase are sequential.

The limit is therefore enforced over actual network probes, not merely over visible policy-group rows.

Progressive results and UI cost

AndroidLib emits a serialized callback when a target completes. A policy group may publish a new selected delay while other candidates remain, and receives a final completed=true update after its last candidate.

The app persists each result immediately and broadcasts a RealPingResult(guid, delay) DTO. Including the GUID removes the previous identity-less “something changed” event that forced a full group reload.

MainViewModel applies result DTOs directly to cached rows. It drains results through one serialized 500 ms loop:

• closely spaced completions are folded into one immutable list update;
• only rows whose delay changed are copied;
• a result arriving while a previous flush is waiting cannot become stranded;
• Finish waits for any scheduled drain and forces the final pending flush before completing the UI state.

The service-event buffer is enlarged for large bursts, while the persisted MMKV values remain authoritative. Progress notifications are throttled separately and count physical work units.

Cancellation and failure behavior

• Stopping/replacing an interactive test cancels the Android worker and the native controller.
• The controller context is passed to core.NewWithContext, so cancellation reaches active Observatory checks.
• A malformed profile is marked failed or isolated without failing unrelated profiles.
• Exceptions from an individual profile produce that profile's -1 result.
• Catastrophic batch failure marks remaining profiles failed and reports a non-success finish status.
• Automatic removal and sorting run only after a successful batch. Cancellation or fatal failure cannot delete profiles that were never tested.
• Back-to-back interactive requests are handed to fresh probe processes instead of reusing native process-global state.

End-to-end 500-profile benchmark

Conditions:

• Pixel 9 Pro x86_64 AVD, Android 17 / API 37, four virtual CPUs
• Play Store debug builds with the same signing certificate
• 500 deterministic SOCKS profiles in one subscription
• emulator-local SOCKS and /generate_204 fixtures with a fixed 100 ms response delay
• RealDelay concurrency 16
• one excluded warm-up and three measured runs per build
• CPU sampled every 250 ms; dumpsys meminfo sampled approximately once per second
• upstream app 739e303f + AndroidLib b2138986
• Observatory app b6fbe5e5 + AndroidLib 484a8771

Values are medians of three runs; parentheses are observed min-max ranges.

Metric	Current upstream	Observatory proposal	Change
First response	0.959 s (0.957-0.969)	1.688 s (1.687-1.765)	0.730 s later
Last probe response	8.430 s (8.404-8.494)	5.036 s (5.013-5.154)	40.3% faster
HTTP phase	7.410 s (7.407-7.465)	3.330 s (3.326-3.340)	55.1% shorter
Final UI state observed	10.635 s (10.588-10.651)	7.160 s (7.133-7.290)	32.7% sooner
Total app CPU time	21.22 s (20.83-21.27)	3.54 s (3.35-3.76)	83.3% less
Main-process CPU	13.11 s	2.34 s	82.2% less
Daemon/test-process CPU	8.16 s	1.19 s	85.4% less
Average app CPU	199.2%	49.4%	75.2% lower
Peak combined PSS	221,383 KiB	166,836 KiB	24.6% / 53.3 MiB less
Peak Java heap	42,296 KiB	33,476 KiB	20.9% less

All six measured runs completed all 500 targets without crashes, ANRs, probe failures, or connection timeouts. Upstream performed 500 TCP prechecks followed by 1,000 GET samples; the proposed path performed exactly 500 HEAD samples.

The 3.330-second measured HTTP phase is only about 6.6% above the 3.125-second theoretical floor for 500 targets, 16 workers, and 100 ms responses. Increasing concurrency is therefore not the useful next optimization.

The tradeoff is startup latency: building the complete dependency plan delays the first result by about 0.73 seconds. Caching or directly constructing minimal outbound fragments is the principled follow-up.

500 ms result-flush benchmark

The end-to-end comparison above used the earlier 50 ms coalescing window. A separate finish-aware emulator benchmark compared the pushed 50 ms version (d30745cc) with the same source using the final 500 ms interval.

Conditions: Pixel 5 x86_64 AVD, Android 11 / API 30, Play Store debug, 500 rows, concurrency waves of 16, three measured repetitions after warm-up. Every run included the production Finish event.

Result arrivals	UI updates, 50 → 500 ms	Process CPU	First partial update	Total completion
100 ms waves	32 → 7 (-78.1%)	118.3 → 72.0 ms (-39.2%)	54.3 → 502.7 ms	unchanged
250 ms waves	32 → 16 (-50.0%)	116.0 → 69.7 ms (-39.9%)	53.7 → 502.7 ms	unchanged
Single burst	1 → 1	noise-dominated	effectively immediate via Finish	unchanged

The deliberate UI tradeoff is up to roughly half a second before the first partial row update. The final batch completion is not extended because Finish forces the pending flush.

Benchmark boundaries

• These are controlled emulator results for healthy, batchable profiles, not universal device/network claims.
• Physical target count matters. A policy group can expand one visible row into many probes.
• Custom and unsupported configurations use individual cores and do not receive the full CPU/PSS benefit.
• Five-second dead targets at concurrency 16 imply approximately 5 s * ceil(targets / 16) network time: about 160 seconds for 500 targets if every worker times out.
• A large combined graph can use more memory than the measured ordinary SOCKS workload. Future chunking should use physical target count or serialized plan size, not an arbitrary visible-profile count.
• The full performance series used feature commits b6fbe5e5 / 484a8771. The final review tips (2f9b560b / 93273c7) retain that architecture and add lifecycle/cancellation fixes and cleanup.

Dependency and review setup

• AndroidLib PR: 2dust/AndroidLibXrayLite#204
• AndroidLib review tip: 93273c799e258e02905c466ac5d6230a9d158307
• v2rayNG review tip: 2f9b560b2cc08be371985103e9c682b60877979d

The generated libv2ray.aar is intentionally not committed. Until the AndroidLib PR is merged and the submodule/AAR is updated upstream, review builds must build the AAR from PR #204 and place it in V2rayNG/app/libs/libv2ray.aar.

Validation

I have been running the earlier version of this code without protections against broken configurations. I experienced no failures.

Completed against the final reviewed tips:

• focused ProbeConfigBuilderTest
• end-to-end native local HTTP test covering ordinary and policy-group targets
• verified HEAD, one request per target, exact concurrency ceiling, progressive completion accounting, and active cancellation
• native test under Go's race detector
• go test -race ./...
• go vet ./...
• four-ABI AndroidLib AAR build and generated API inspection
• :app:assemblePlaystoreDebug
• git diff --check

The focused Kotlin and Go test files are retained locally for continued development but intentionally are not part of either PR diff.