Repository navigation
Client: cut per-request allocation and CPU in the channel and session; measure the client in the load harness - #4720
Conversation
Profiling a 20-session load (read/write/browse/subscribe) against the reference server showed the request path dominated by lock spinning and per-node allocation. Paired A/B medians (workstation GC, 24 cores): Read 1 value +45% req/s (CPU/req -37%, alloc/req -39%, lock contention -94%), Read 100 values +40%, encrypted Read 100 +48%, Browse +33%, Write 100 +13%, data-change notifications -17% alloc. - ObjectPool: lock-free slot array; ConcurrentBag.Count took every per-thread lock on each Return of the pooled response source. - FairRequestQueue: drop redundant gate entries (fail-fast pre-checks, stopped check, owner count folded into selection) and coalesce wake pulses so the bounded signal channel's lock is only taken when no token is pending. - Isolation provider: compare peer addresses without allocating; build the unsecured verification context only when a classifier is set. - TCP listener: cache the certificate DER copies per channel instead of copying X509 RawData for every request. - EndpointBase: resolve the request ActivitySource once instead of walking the stack (GetCallingAssembly) per request. - Node managers: reuse one ServerSystemContext copy per request in the per-node callbacks, remove a discarded copy, use plain dictionaries for per-request operation caches, build the validation cache in one pass and fix zero-capacity error lists. - ResourceManager: lock-free translation lookups from an immutable snapshot rebuilt after writes (Browse serialised on it). - MonitoredItem: share the default DataChangeFilter instead of allocating one per sampled value. - Reference server: enable Server GC; documented in ServerScalability.md (doubles Read throughput under load vs workstation GC). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ServerLoadHarness runs a reference server in a child process and drives read/write/browse/subscription load against it, reporting client latency percentiles and the server's CPU, allocation, GC and lock-contention deltas, optionally recording the window with dotnet-trace. TraceAnalyzer aggregates those traces; ab.sh/ab-summary.ps1 run paired comparisons of two builds. Not part of UA.slnx. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read, Write, Browse and Call validate every node they touch, and the dispatcher discarded the NodeMetadata (248 B, plus copies of the role permission arrays) right after the check. Browse also read 13 attributes per reference target only to check its RolePermissions. The role-permission and access-restriction validators now run on an internal PermissionMetadata struct holding exactly the attributes they read; the NodeMetadata overloads build it and delegate, so the checks are unchanged. AsyncCustomNodeManager fills the struct on the stack from the same three attributes and namespace defaults that GetPermissionMetadataAsync reads. The dispatcher uses it when the metadata is not needed, and Browse's reference filter uses it instead of GetNodeMetadataAsync. Unresolved handles still take the NodeMetadata path with its fallbacks. Read 100 values: alloc/request 120.6 -> 95.7 KB, ~21k -> 26k req/s; Browse: 67.6 -> 51.8 KB, 26.7k -> 29.8k req/s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssion lookup locking SecureChannelContext.ClientChannelCertificate, ServerChannelCertificate and ChannelThumbprint are ByteString instead of byte[]. The TCP listener now shares one cached DER copy per channel with every request, and a mutable array handed to request handlers or IResourceIsolationClassifier code could have altered the evidence later requests are checked against. A ByteString is a read-only view, so the buffer is shared safely without copying. The constructor keeps its byte[] parameters and wraps them; a missing value is IsNull, so the session signature data (which skips a null certificate but hashes an empty one) is unchanged. Readers in Server, GDS, LDS and the Redundancy session mirror are migrated; the mirrored activation state keeps recording an empty, not null, certificate. MigrationGuide.md lists the mapping for application code. SessionManager.TryGetSessionContext runs at admission and at dispatch of every request and took the global bindings lock twice per call. The candidate snapshot now reads the concurrent session index and the activation-state table without the lock; the validating check under the lock still confirms the binding is the committed one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… translation snapshot ObjectPool: reuse, bounded retention, the per-processor cap, no shared rental under concurrent Get/Return, and no lost capacity after concurrent use. OperationContext.GetSystemContext: one copy per template, never another node manager's copy when templates alternate, and concurrent lookups bound to the same operation. ResourceManager: updates and new locales are visible after a snapshot was built, and concurrent readers see only complete translations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the OPC Foundation license header, drop the per-project Nullable, warnings-as-errors and analysis-level overrides and fix the diagnostics instead, remove comment section dividers, and record latency samples in a ConcurrentBag instead of locking List instances. The only suppression left is a documented CA2000 where the session takes ownership of a subscription. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d pool edge cases - StandardServer compared the session certificate with the channel certificate through Utils.IsEqual, which bound to the object overload once the channel certificate became a ByteString and always failed (BadSecurityChecksFailed on secure CreateSession and TransferSubscriptions). Compare the spans. - The isolation provider's ChannelSnapshot copies the certificate again: the context wraps a buffer the transport owns, and revalidation must detect evidence that changed after the owner was issued. - PeerMatches has the array fallback only on .NET Framework; on .NET the 16-byte stack buffer always holds the address. - Tests for a stopped FairRequestQueue (admission and readers) and for the ObjectPool capacity argument. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mple The sampling loop waited one interval after every sample, so the timer overshoot (about 1 ms on Linux, up to the 15.6 ms timer resolution on Windows) and the sampling time were added to every cycle: a 10 ms group took 63 % (Windows) to 90 % (Linux) of its samples, and the o6-automation opcua-benchmarks subscription test (100 Hz, every item >= 99 % of samples) failed with a single monitored item. Samples now have absolute deadlines one interval apart; a late wake-up shortens the next wait, and a stall longer than four intervals (at least 50 ms) restarts the schedule instead of bursting. The benchmark workload now passes up to 4096 items (about 410k samples/s). NodeState.ReadAttributes fills the result array in place instead of copying a list, removing an allocation per sampled permission check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The TCP listeners sent each response through TcpServerChannel.SendResponse, which waits for the channel gate synchronously on the thread-pool worker that finished the request. With many requests in flight per channel, workers parked on the gate while the receive loop that was handed the gate waited for a free pool thread, and the channel stalled until the pool grew: 10 sessions with 100 outstanding Reads each dropped to 0-1.7k req/s (0 with SignAndEncrypt). SendResponseAsync waits for the gate asynchronously; the TCP, Kestrel TCP and WebSocket listeners use it, and the latter two use the cached certificate copies. Same load: 121-131k req/s (89-91k encrypted). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hmark `--scenario counters` runs a server whose UInt64 counters advance in their read callback (sampling groups, configurable sampling rate) and checks per item, on the client and on the server, that at least 99 % of the expected samples arrived. `--queue` sizes MaxQueuedRequestCount for deep pipelines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…write queue
A server channel was faulted with BadTcpNotEnoughResources ("The channel
write queue is full") when a request arrived while more than 100 responses
were queued for writing, so a client pipelining more requests than that per
channel (e.g. the 1000 outstanding Reads per client of the o6-automation
benchmark) lost its channel. Server channels now stop reading while the queue
is full and resume once it has drained to 50; TCP flow control slows the
client instead. Requests already read when the queue filled stay bounded by
the request queue limits. Client channels keep reading, because responses
are what drain their writes.
10 clients x 1000 outstanding Reads: channels faulted before, now 0 errors at
about 45-54k req/s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In process the client drains responses fast enough that the server's write queue rarely fills, so the test checks that deep pipelines complete with the channels connected; the out-of-process harness exercises the full queue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of lag SamplingKeepsTheRequestedRateAsync measured the sampling rate against the wall clock and failed on a loaded macOS CI runner (78 % of the samples). The deadline arithmetic now lives in SamplingSchedule, and the tests drive it with simulated clocks: Windows- and Linux-like timer overshoot keeps the rate (where waiting a fixed interval drifts), a 200 ms pause is made up, a 3 s stall restarts the schedule once, and sampling slower than the interval does not run away. The schedule now tolerates a lag of four intervals or 500 ms (was 50 ms) before restarting, so a garbage collection or scheduling pause of that length is made up rather than lost; on a loaded machine this kept more samples at 4096 items (81 % vs 62-71 %). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaders The wait before a sample was converted with remaining * TicksPerSecond / frequency, which overflows a long for long intervals at the 1 GHz timestamp frequency of Unix (one hour is 3.6e19); Task.Delay then threw and ended the sampling loop. SamplingSchedule.ToTimeSpan scales through a double, covered by a one-hour test at 1 GHz. ResourceManagerSnapshotTests awaits its readers with a timeout instead of blocking on Task.WaitAll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The harness already ran the server in a child process; the parent process is the client, so the same counters taken there measure the client stack without the server. - client-process and client-per-op lines (allocated bytes, CPU, GC, lock contention) for the measurement window; - --trace-side client records the harness itself with dotnet-trace; - --client managed drives ManagedSession with the V2 subscription engine, --callback item uses the classic per-item Notification event; - classic sessions no longer ask the server for diagnostics, as an application would not; - ab.sh takes SCENARIOS/ROUNDS/OUT, tolerates CRLF scenario files and passes defaults before the scenario so a scenario can override them; ab-summary.ps1 reports client and server medians separately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Profiling the client process under load (paired A/B against the server branch, 20 sessions, workstation GC): Read 1 value +21% req/s with client alloc/request -46% and CPU/request -13%; 200 sessions +29%; SignAndEncrypt Read 1 value +28% (CPU -24%); Write 100 values client alloc -63%. The server shares the channel code and allocates 20-26% less per small request. - ActivitySource cached per instance in Session, SessionClientBatched, the classic Subscription and engine and the TCP/HTTPS client channels: StartActivity walked the stack (GetCallingAssembly) on every call. - Channel writes: an ordered queue drained by one thread-pool writer (local queue of the issuing thread) replaces the per-write ContinueWith/Unwrap chain and Task.Run; successful writes are reported inline, and the client channel only enters its gate for a failed write. - TcpByteTransport: no linked CancellationTokenSource for a non-cancellable token, single-chunk messages use the memory send, pooled async builders on .NET 6+, no closure per vectored send. - BinaryEncoder.GetNodeIdEncoding boxed every NodeId through Convert.ToUInt32(object) and Convert.ToByte(enum). - Symmetric chunk headers are written and read directly instead of through an encoder/decoder, stream and reader/writer per chunk; a short chunk still reports the decoder's error. - AES-CBC: one Aes per channel token and direction, with the one-shot EncryptCbc/DecryptCbc on .NET 6+, instead of a new instance (key import) per message. - ChannelAsyncOperation: a task-free completion wait for requests, cancellation by registration instead of WaitAsync, a shared timer callback and a shared box for default results. - Session keep alive: a response records the time instead of moving the timer (timer queue lock); the timer defers itself to one interval after the last response. Tests cover the write queue (order, one send at a time, off the caller, failures), the completion wait (cancellation, disposal, races), the AES cache, the transport send paths and the keep-alive deferral. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…//github.com/OPCFoundation/UA-.NETStandard into romanett/client-performance-profiling # Conflicts: # perf/ServerLoadHarness/Program.cs # src/Opc.Ua.Core/Stack/Tcp/UaSCBinaryChannel.cs
…formance-profiling # Conflicts: # perf/README.md # perf/ServerLoadHarness/Program.cs # perf/ab-summary.ps1 # perf/ab.sh # src/Opc.Ua.Core/Stack/Tcp/UaSCBinaryChannel.cs
There was a problem hiding this comment.
🟡 Changes recommended
AES disposal can race cache return, and managed-subscription setup can wait indefinitely after creation failures.
6 open findings
Prevent disposed token cache from being repopulated · New Bound setup readiness loop and handle monitored item errors · New Synchronize continuation registration deterministically · New Use a dedicated lock instead of the mutable list · New Describe ProcessStats counters as process-wide · New Seal the new public fixture class · New
What changed in this PR
Optimizes client/channel request paths and extends the performance harness with client-side measurements.
Changes:
- Reduces transport, encoding, tracing, and cryptographic allocations.
- Reworks request completion, write queuing, and keep-alive scheduling.
- Adds client benchmarks and regression tests.
| File | Description |
|---|---|
tests/Opc.Ua.Core.Tests/Stack/Transport/UaSCBinaryChannelWriteQueueTests.cs |
Tests ordered serialized writes. |
tests/Opc.Ua.Core.Tests/Stack/Transport/TcpByteTransportTests.cs |
Tests send paths and cancellation. |
tests/Opc.Ua.Core.Tests/Stack/Transport/ChannelAsyncOperationWaitTests.cs |
Tests task-free completion waits. |
tests/Opc.Ua.Core.Tests/Security/Certificates/CryptoUtilsAesCacheTests.cs |
Tests AES caching and concurrency. |
tests/Opc.Ua.Client.Tests/Session/SessionKeepAliveTests.cs |
Tests deferred keep-alive timing. |
src/Opc.Ua.Types/Encoders/BinaryEncoder.cs |
Removes NodeId boxing. |
src/Opc.Ua.Core/Stack/Tcp/UaSCBinaryClientChannel.cs |
Optimizes tracing and request waits. |
src/Opc.Ua.Core/Stack/Tcp/UaSCBinaryChannel.Symmetric.cs |
Uses direct header encoding and cached AES. |
src/Opc.Ua.Core/Stack/Tcp/UaSCBinaryChannel.cs |
Introduces the ordered write queue. |
src/Opc.Ua.Core/Stack/Tcp/TcpByteTransport.cs |
Optimizes socket send/receive paths. |
src/Opc.Ua.Core/Stack/Tcp/ChannelToken.cs |
Owns per-direction AES caches. |
src/Opc.Ua.Core/Stack/Tcp/ChannelAsyncOperation.cs |
Adds allocation-free completion waiting. |
src/Opc.Ua.Core/Stack/Https/HttpsTransportChannel.cs |
Caches its activity source. |
src/Opc.Ua.Core/Security/Certificates/CryptoUtils.cs |
Reuses AES-CBC instances. |
src/Opc.Ua.Client/Subscription/Classic/Subscription.cs |
Caches subscription activity sources. |
src/Opc.Ua.Client/Session/Subscription/ClassicSubscriptionEngine.cs |
Caches publish activity sources. |
src/Opc.Ua.Client/Session/SessionClientBatched.cs |
Caches batched-call activity sources. |
src/Opc.Ua.Client/Session/Session.cs |
Optimizes tracing and keep-alive scheduling. |
perf/ServerLoadHarness/Program.cs |
Measures clients and adds managed-session scenarios. |
perf/server-scenarios.txt |
Defines server benchmark scenarios. |
perf/README.md |
Documents client measurement support. |
perf/client-scenarios.txt |
Defines client benchmark scenarios. |
perf/ab.sh |
Makes A/B runs configurable. |
perf/ab-summary.ps1 |
Summarizes client and server medians. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Code coverage✅ Coverage gate passed.
Uncovered changed lines
Coverage is above the recorded baseline - consider ratcheting Thresholds live in |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4720 +/- ##
==========================================
+ Coverage 82.84% 82.86% +0.01%
==========================================
Files 2267 2268 +1
Lines 345968 346153 +185
Branches 60316 60358 +42
==========================================
+ Hits 286617 286824 +207
+ Misses 39253 39241 -12
+ Partials 20098 20088 -10
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Keep AES cache shutdown state separate from borrowed cipher ownership so returns racing token disposal release the keyed instances. Cover both encryption directions and concurrent return/disposal. Bound managed subscription setup to one minute, fail on item and subscription errors, and preserve cancellation. Add fake-clock readiness regressions. Make continuation registration deterministic, use a dedicated test lock, seal the AES fixture, and correct process-counter documentation. Validation: 112 targeted core tests and 11 readiness tests pass on each of net10.0 and net48. The harness builds without warnings and encrypted managed-subscription/classic-read smoke scenarios complete.
…path, write coalescing (#4737) Follow-ups from the "Not changed" list of #4720 (client per-request allocation and CPU): three changes that the paired A/B shows a gain for, plus a write-up of the fourth (discarded `ResponseHeader`), which is not changed. ## Results Paired A/B, each change against the commit before it: 4 alternating rounds, medians, 20 sessions, 24-core machine shared with other sessions. Client = the harness process, server = the reference server child. Allocated bytes per operation are deterministic; throughput and latency carry the run-to-run noise noted below. | Change | Scenario | Throughput | p50 | p90 | Client CPU / op | Client alloc / op | Server CPU / op | |---|---|---|---|---|---|---|---| | Receive staging | Read, 1 value | +4% | **−9%** | **−13%** | ≈ | ±0 | ≈ | | | Read, 1 value, SignAndEncrypt | +6% | −6% | −11% | ≈ | ±0 | ≈ | | | Read, 100 values | +2% | −4% | ≈ | ≈ | ±0 | ≈ | | | Data changes | ≈ | ≈ | ≈ | ≈ | ±0 | ≈ | | ManagedSession lock | Read, 1 value, ManagedSession | ≈ | ≈ | ≈ | ≈ | **−125 B** (2632 → 2507) | ≈ | | | Same, inflight 8 | ≈ | ≈ | ≈ | ≈ | −133 B | ≈ | | Write coalescing | Read, 1 value, 20 sessions on one shared channel, inflight 4 | **+13%** | **−13%** | −3% | −6% | ±0 | −4% | | | Same, inflight 1 | +20% ¹ | −8% | ¹ | −9% | ±0 | −13% | | | Read, 1 value, inflight 8 (one channel per session) | +2% | −4% | −6% | ≈ | ±0 | ≈ | | | Read, 100 values, inflight 8 | +3% | −4% | −8% | ≈ | ±0 | ≈ | | | Data changes | ≈ | ≈ | ≈ | ≈ | ±0 | ≈ | ≈: within run-to-run noise (about ±5% for CPU per op here). ¹ Rounds ranged from −42% to +32%; the inflight-4 runs were better in all four rounds. ## Changes - **`TcpByteTransport` receive**: `ReceiveChunkAsync` read the 8-byte chunk header and then the body, two `recv` calls per chunk. It now waits for data with a small staging buffer (8 KB, allocated on the first receive, not from the `BufferManager` budget). One receive returns the header and, when the chunk is small and already there, the whole chunk; bytes of following chunks are kept for the next call. A larger chunk is completed with a receive into the rented buffer, as before. - The receive buffer is still rented only after the header has been validated (message type, size against the negotiated limit), so an idle peer holds none of the budget. - A receive cancelled while its bytes are in the staging buffer keeps them for the next call; on .NET Framework the staging receive takes no token, so cancellation is observed once data arrives. - A receive that fails while reading the body of a chunk larger than the staging buffer has consumed part of the chunk; later receives then fail with `BadTcpInternalError` rather than read body bytes as a header. Before this PR such a failure lost the header and body bytes silently. The channel cancels receives only when it closes. - Server listener channels use the same transport. - **`ManagedSession` service calls**: each call was an async wrapper (await the reader lock, await the inner call), one more state machine box per request. - When services are available and no writer holds or waits for the lock, the reader lock is taken synchronously (`AsyncReaderWriterLock.TryEnterReader`). It is released by a cached continuation on the inner call's task, registered before the caller's await, so the lock is free again when the caller resumes. - Otherwise the call waits for the lock as before, so reconnect and failover still block new calls and drain the ones in flight. - The remaining 88 B per call is the task's continuation list (two continuations). - **Write coalescing** (`UaSCUaBinaryChannel`): when more writes for the same transport are queued behind the one being sent, the writer takes up to 16 of them and sends their segments in one vectored send. - Each write is then reported on its own, in order, with its size. A failed send is reported for every write in it, and their buffers are returned. - Writes still never start on the caller's stack, and one writer sends one batch at a time, so chunk order is unchanged. - Only byte-stream transports opt in (`TcpByteTransport`, also through the server's admission lease, via the internal `IUaSCChunkCoalescing`). WebSocket and capture decorators keep one send per chunk. ### Behaviour notes - Each TCP connection that has received holds an 8 KB staging array (pinned on .NET 5+), outside the `BufferManager` budget. The previous transport held an 8-byte header array. - `ManagedSession` service calls: if the inner call completes while the release continuation is being registered, the reader lock is released on the thread pool shortly after, instead of before the caller resumes. - Concurrent `ReceiveChunkAsync` calls on one `TcpByteTransport` now throw `InvalidOperationException`. The channel has a single receive loop, and interleaved receives on a stream were never valid. ## Not changed: discarded `ResponseHeader` Every generated structure's `Initialize()` constructs its structure-typed fields: `new ResponseHeader()` (112 B), `new RequestHeader()` (104 B). The decoders always create the instance (`new T()` or the activator) and then assign every field in `Decode`, so the default is thrown away. That is 4.6% of the client's 2.4 KB per Read and about 3% on the server, which discards a `RequestHeader` per decoded request and a `ResponseHeader` per response it constructs. `PublishResponse` also discards a default `NotificationMessage`. Options considered: - **Lazy getter** (`get => m_x ??= new X()`): `IsEqual`/`GetHashCode` compare fields, so a never-read default (null) would no longer equal an explicit default. Two threads reading an unset header could get different instances. Pooled instances reset through `Initialize()` anyway. - **Decode-only constructor** called by the activator: only the plain `Class`/`DerivedClass` templates assign every field in `Decode`, not optional-field or union types. A derived type's constructor must chain to a base built by an older generator in another assembly, which lacks it (the same problem `ForeignDerivedPooledExtensionClass` works around). It also needs a new public marker type in `Opc.Ua.Types`. - **Decode into the existing instance**: needs a new `IDecoder` member in all decoders. It also changes behaviour when a caller decodes into an object whose header it still references. Each touches every generated type for about 100 B per message, so this is left as is. ## Harness - `--client shared`: classic sessions whose secure channels come from one `ClientChannelManager`. - Created sequentially (`--connect-parallel`, default 1). Concurrent `Session.CreateAsync` calls on one manager fail with `BadSecureChannelClosed`, a separate bug. - 20 sessions on one channel reach about 45–70k req/s, against about 140k on 20 channels. - `ab.sh`: `CANDIDATE=` selects the candidate build. ## Testing - New: `TcpByteTransportTests`: chunks sent together, a chunk split across sends (inside the next header), a chunk larger than the staging buffer, cancellation keeping staged bytes. `UaSCBinaryChannelWriteQueueTests`: coalesced in order (1 + 16 + 16 + 7 sends), framing transport not coalesced, a failed batch reported for every write and later writes still sent. `AsyncReaderWriterLockTests`: `TryEnterReader` against a writer, the reader held until the operation completes and released on failure and on a completed operation, the awaiter resuming after the release. - Before the rebase onto master, net10.0, run sequentially: Core Stack+Security 4689 passed, 1 failed (`ClientBaseTests.ActivityTraceFlags_Metrics_ShouldRecordMetricsOnly`, a meter-listener flake fixed on master by #4708; passes 3/3 in isolation); Client 2768; Sessions 851 (379 skipped); Server 8398; Subscriptions 643. - After the rebase: Core Stack+Security 4811 passed; Client 2781; Stress Contract+Integration 45 passed; all 0 failed. Sessions, Server and Subscriptions were not rerun on the new base. - net48 build of `UA.slnx`: clean (0 warnings, 0 errors); `dotnet format whitespace` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ue-wide lock (#4750) # Server: cheaper symmetric crypto path, FairRequestQueue without a queue-wide lock Roadmap items E1 and 5b of the o6 benchmark gap analysis, one commit each. - **E1:** Windows one-shot `Aes.EncryptCbc`/`DecryptCbc` import the key (`BCryptImportKey`) on every message, although #4720 caches one `Aes` per channel token. The AES-GCM and ChaCha20-Poly1305 policies also created a new keyed instance per message. - **5b:** with `AsyncCustomNodeManager`, which is what the o6 benchmark's 2.0.1 worker uses, the `FairRequestQueue` gate became the largest waiting point. `TryEnqueue`, `TrySelect` and `TryDequeue` spun on one `Lock` for every request. No wire, configuration or public API changes. ## Results These are paired A/B runs against `origin/master`, five rounds per scenario. The two harness builds alternate per round, starting side swapped every round, on the same shared machine (Ryzen AI 9 HX 370, 12 cores / 24 threads, Windows 11). The tables show medians, with ranges in brackets where they matter. The server is configured like the o6 2.0.1 worker: `AsyncCustomNodeManager`, Server GC, queue sized clients × outstanding + 3. o6read uses the benchmark's native open62541 client workers; `read` uses the harness's .NET client. On this machine throughput moves between rounds by up to 2×, so CPU and bytes per request are the reliable signals. ### E1: symmetric crypto (measured on 0dff94b) | Scenario | req/s | Server CPU µs/req | Server alloc B/req | Client CPU µs/req | Client alloc B/req | |---|---|---|---|---|---| | read --nodes 1, Basic256Sha256 SignAndEncrypt, 24 cores | 115,300 → 103,600 (ranges overlap) | 64.6 → 60.9 (**−6 %**) | 5,698 → 5,486 | 44.9 → 41.7 (**−7 %**) | 2,672 → 2,456 | | o6read 10×100, encrypted, 4-CPU mask | 71,300 → 75,900 | 51.8 → 51.4 | 6,323 → 6,111 | – | – | | o6read 10×100, encrypted, 24 cores | 123,500 → 118,200 (ranges overlap) | 115.7 → 117.2 | 6,716 → 6,515 | – | – | | o6read 10×100, unencrypted, 4-CPU mask (control) | 79,100 → 70,600 | 48.0 → 48.4 | 6,088 → 6,090 | – | – | Isolated costs per call (one pinned core, 224-byte chunk, variants alternating in one process): | Operation | Before | After | |---|---|---| | AES-256-CBC encrypt | 778 ns, 128 B (one-shot) | 329 ns, 24 B (cached transform) | | AES-256-CBC decrypt | 682 ns, 128 B | 239 ns, 24 B | | AES-256-GCM seal | 1,022 ns, 72 B (new `AesGcm`) | 260 ns, 0 B (cached) | | HMAC-SHA256 | 306 ns (per-token instance, unchanged) | – | For comparison, `HMACSHA256.TryHashData` costs 458 ns and `IncrementalHash` 318 ns. **E1 does not close the 20–50 µs encrypted gap, and no crypto change could.** Pinned to four CPUs, encryption costs master only about 3–4 µs per request (51.8 vs 48.0 µs). The extra cost at 24 cores comes from threads spinning on locks while signing and encryption run inside the channel gate (roadmap A1). The ~7 µs of HMAC in the report's profile is a thread-time artefact. `HashProviderCng.AppendHashData` shows 40× more samples on the sign side, inside the gate, than on the verify side for the same chunk sizes. Threads are preempted there while spinners hold the CPUs. ### 5b: FairRequestQueue (unencrypted, E1 has no effect here) | Scenario | req/s | Server CPU µs/req | Server alloc B/req | Contention events/s | |---|---|---|---|---| | o6read 10×100, 24 cores | 149,000 → 150,200 | 110.8 [106–117] → 99.6 [96–109] (**−10 %**) | 6,404 → 6,367 | 4,533 → 2,125 | | o6read 10×1000, 24 cores | 152,200 → 161,200 | 106.7 → 96.3 (**−10 %**) | 6,421 → 6,358 | 4,131 → 1,921 | | read --nodes 1 --sessions 200, 24 cores | 126,100 → 141,200 (+12 %) | 53.6 → 51.1 (−5 %) | 5,353 → 5,294 | 4,078 → 455 (**−89 %**) | | o6read 10×100, 4-CPU mask | 85,600 → 83,900 | 45.7 [44.1–47.8] → 46.8 [44.4–48.9] (noise) | 6,089 → 6,060 | 281 → 176 | Traces of o6read 10×100 at 24 cores, one run per side: - **Lock spinning.** In the sampled CPU profile, frames above `Lock.TryEnterSlow` in the fair queue fell from **16.6 % of on-CPU samples** (`TryEnqueue` 5.8, `TrySelect` 5.8, `TryDequeue` 4.9) **to none**. - **Contention events in the window:** 15,331 → 5,879. - **What remains of the queue:** - the wake channel in `DequeueAsync`: idle workers parked on the coalesced signal, 1.3 % of on-CPU samples, kept unchanged; - `SessionManager.TryGetSessionContext`'s server-wide `m_bindingsLock`, which every request takes at admission and again at dispatch. It is now the largest spin point (~26 % of `Lock.TryEnterSlow` samples) and is a separate follow-up. The channel gate (A1) is another ~17 %. In the o6 configuration all ten sessions map to **one** owner, so per-owner locks would not have helped. An intermediate version with a lock per owner FIFO left 12.8 % of the samples on those locks. That is why the FIFO itself is lock-free. ## Changes ### Symmetric cipher cache (E1) - `SymmetricCipherCache` replaces `AesCache`; a `ChannelToken` keeps one per direction (`ServerCiphers`, `ClientCiphers`). Per slot it retains: - a CBC encryptor keyed with the token's key and IV; - a CBC decryptor keyed the same way; - the AES-GCM or ChaCha20-Poly1305 instance. - **How a CBC message is processed:** the cached `ICryptoTransform` transforms the whole message in place with `TransformBlock`. An empty `TransformFinalBlock` (padding `None`) then rewinds the chaining state to the IV without re-keying, on CNG, OpenSSL and .NET Framework CAPI alike. Cached and fresh ciphertext are tested equal. - **Concurrency and ownership:** - A cipher is taken out of its slot while in use, so concurrent senders never share one. - It is matched to the key and IV by reference, and replaced if the key material differs. - It is disposed instead of returned after a failed transform, or once the token has been disposed. - **Unchanged:** - The uncached public `CryptoUtils` overloads keep the one-shot path. - The order is still verify, then decrypt, then padding, so a tampered padding byte under a valid signature is still reported as padding, after the signature check. - HMAC is unchanged. ### FairRequestQueue (5b) - **No lock in admission or dispatch.** - Each owner has a lock-free FIFO: Michael–Scott, with the entries themselves as the nodes, so queueing allocates nothing. The `LinkedListNode` per request is gone. - A request is claimed exactly once, by a compare-and-swap from selection, cancellation or stop. Claimed entries are dropped lazily when they reach the head. - **Weighted round robin over a lock-free ready queue of turn tokens.** - A turn holds one token per queued request, up to the owner's weight. A weight of 1,000,000 therefore costs no more scheduling state than the backlog. - Taking a turn's last token queues the owner's next turn at once, behind the other owners. The turn state is one 64-bit word changed by CAS, so the service order is the token order even while workers race. Weights 1:3 under 8 concurrent workers stay within 2.5–3.5. - **Failed execution admission:** the head goes to a small sorted list under the owner's gate, served before the FIFO. Its owner is not retried before capacity changes, as before. - **Owner FIFO retirement:** an owner FIFO is retired by an atomic CAS of its count from 0 to a sentinel, and a retired FIFO can never be counted again. The next request of that owner starts a new FIFO, and the retired FIFO's leftover tokens are skipped. - **Unchanged:** - The count bound is reserved with `Interlocked` before linking. - Global FIFO mode still selects one at a time, so a blocked head cannot be overtaken. - Wake coalescing (#4707), parked-request decoupling and `ServerBase.RequestQueue`'s worker growth. ## Behaviour notes - **Cipher cache memory:** a token now keeps up to two CBC transforms (plus their `Aes`) or one AEAD instance per direction, instead of one `Aes`. They are released with the token. - **Weighted ordering under concurrency:** dispatch order across owners follows token order. Within an owner, admission order is kept, including requests that fail execution admission and go back to the front. - **Idle scheduling state:** cancelled requests leave stale turn tokens in the ready queue until the next dispatch pass. They are discarded at once when the last queued request is cancelled. - **net48:** the BouncyCastle-backed AEAD types are cached like the platform ones. ## Testing - **New and extended tests:** - `CryptoUtilsAesCacheTests`, for all CBC and AEAD policies (Basic128Rsa15, Basic256, Basic256Sha256, Aes128_Sha256_RsaOaep, Aes256_Sha256_RsaPss, ECC_nistP256, ECC_nistP256/P384/RSA_DH AesGcm, ChaChaPoly): - cached vs fresh ciphertext for 0–300-byte bodies; - IV or key change; - tampered signature, ciphertext, tag and header, each rejected with the cache recovering; - bad padding under a valid signature; - 256 concurrent senders; - token disposal during a transform, and a return racing disposal. - `FairRequestQueueTests`: - turn tokens bounded by the backlog, not the weight; - cancellation leaves no tokens behind; - 1:3 weights under 8 workers; - no lost wakeups (4 producers, 8 waiting workers, both modes); - cancellation racing dequeue completes or dispatches each request exactly once and returns every lease; - stop while workers wait and admissions race; - a 50-owner, 16-worker stress run with cancellations; - pipelined clients (1×200, 10×100, 64×1) with ample and with scarce execution capacity. - The last test stalls on an earlier version of this change that briefly marked an emptying owner as retired, when the race window is widened with an injected delay. That bug stalled o6read 10×100 every time in the harness; a dump showed 1,000 queued requests with no turn tokens. Without the delay the window is too narrow to hit at unit level. - All 90 FairRequestQueue tests pass 30/30 repeated runs on net10.0 and 5/5 on net48. - **Suites,** net10.0, after rebasing on 8b360da (#4748): | Suite | Passed | Failed | Skipped | |---|---|---|---| | Opc.Ua.Core.Tests | 6,357 | 0 | 88 | | Opc.Ua.Server.Tests | 8,483 | 0 | 5 | | Opc.Ua.Sessions.Tests (all secure-channel policies) | 851 | 0 | 379 | | Opc.Ua.Stress.Tests Contract + Integration | 45 | 0 | 0 | | Opc.Ua.Interop.Tests (1.5.378 peer, SignAndEncrypt Basic256Sha256, Aes256_Sha256_RsaPss, ECC_nistP256) | 218 | 0 | 29 (AEAD policies the 1.5.378 peer does not offer) | - **net48:** Opc.Ua.Core.Tests builds, and CryptoUtilsAesCacheTests plus FairRequestQueueTests pass. `dotnet format whitespace` is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>



Follows #4707 (replaces #4714, which closed when the #4707 branch was deleted). Profiling the client process under load showed per-request allocation spread over the channel write path, the response wait, per-chunk header encoders and an AES instance per message, plus a stack walk per service call. This PR removes them and extends the load harness to measure the client.
Results
Paired A/B against the #4707 head, the server work now on master (3 alternating rounds, medians, 20 sessions unless noted, workstation GC on both sides, 24-core machine). Client = the harness process, server = the reference server child.
¹ Throughput is bounded by the writer rate; latency is dominated by the 100 ms publishing interval. Per notification the client uses about 2 µs and 170–260 B.
² The client uses 0.2–0.4 cores in the subscription scenarios; the CPU per notification is within run-to-run noise there.
Client GC pause time in the window drops 38–66% in the Read and Write scenarios (17% for Browse). About 70% of the remaining client CPU per small request is kernel time in loopback socket send/receive.
Changes
Session,SessionClientBatched, the classicSubscriptionandClassicSubscriptionEngine, and the TCP/HTTPS client channels resolve theirActivitySourceonce.TelemetryExtensions.StartActivitycalledAssembly.GetCallingAssembly()(a stack walk) on every service call, ~8% of client CPU.UaSCUaBinaryChannel, client and server): an ordered queue drained by one thread-pool writer replaces the per-writeContinueWith/Unwrap/trailing continuation chain. The writer is queued to the issuing thread's local pool queue, like the task continuation was (queuing it globally cost ~15 µs of p50 in a first iteration). Successful writes are reported inline instead of throughTask.Run; failed writes still go through the pool. The client channel enters its gate only for a failed write instead of on every write completion.TcpByteTransport: no linkedCancellationTokenSourcewhen the caller's token cannot be cancelled (the channel always passesNone); a single-chunkBufferCollectionuses theReadOnlyMemorysend instead of the vectoredTask<int>send;PoolingAsyncValueTaskMethodBuilderon .NET 6+ for the receive/send state machines; no closure per vectored send.BinaryEncoder.GetNodeIdEncoding:Convert.ToUInt32(uint, IFormatProvider)bound to theobjectoverload andConvert.ToByte(enum, ...)boxed the enum, two boxes per encoded NodeId. 44% of the client allocation of a 100-value Read; the server encodes NodeIds the same way.BinaryPrimitivesinstead of aMemoryStream+BinaryEncoder/BinaryWriter(send) andBinaryDecoder/BinaryReader(receive) per chunk. A chunk too short for a field still gets the decoder'sBadDecodingError.ChannelTokenkeeps oneAesper direction, used with the one-shotEncryptCbc/DecryptCbcon .NET 6+ (net48 reuses the instance and creates the transform). Previously every message created anAes, imported the key (BCryptImportKey) and created a transform, ~15% of client CPU with SignAndEncrypt. The instance is taken out of the token while in use, so concurrent callers never share one. The publicCryptoUtilsoverloads are unchanged.ChannelAsyncOperation:SendRequestAsyncwaits on an internal task-free completion source instead ofEndAsync(TaskCompletionSource+Task+WaitAsyncpromise + state machine);EndAsyncregisters cancellation instead ofWaitAsync(ct); the timeout timer uses a static callback; completing withdefault(T)shares a box.Timer.Changeunder the session lock, taking the timer queue lock. It now records the time; the timer callback defers itself to one interval after the last response, which is when the per-response reset made it fire.Behaviour notes
TcpByteTransport.SendChunkAsync/ReceiveChunkAsyncreturn pooledValueTasks on .NET 6+: like anyValueTaskthey must be awaited once.HandleWriteCompletenow runs on the writer instead of a separate pool work item. Overrides outside the stack that block in it delay later writes on that channel.ExecutionContext(Activity.Currentis not set inside the transport send).Harness (
perf/)client-process:andclient-per-op:result lines;--trace-side clientrecords the harness itself.--client managed(ManagedSession, V2 engine) and--callback item(classic per-item events).DiagnosticInfoper notification).ab.sh:SCENARIOS/ROUNDS/OUT, CRLF tolerant, defaults before the scenario (a--sessions 200scenario silently ran with 20 before);client-scenarios.txt;ab-summary.ps1reports client and server medians.Monitor.Enter_Slowpathin the sampled CPU profile without matching contention events (46 in 10 s here), and allocation ticks over-attributing small objects allocated after large arrays.Testing
UaSCBinaryChannelWriteQueueTests(order, one send at a time, never on the caller's stack, failures reported and later writes still sent, concurrent writers),ChannelAsyncOperationWaitTests(late/early completion, faults, cancellation, disposal, continuation off the completing thread, 2000-iteration completion/cancellation race, timeout),CryptoUtilsAesCacheTests(cached = fresh cipher text across IVs, round trips, concurrent users, token disposal),TcpByteTransportTestssingle/multi-buffer and cancellable sends,SessionKeepAliveTestskeep-alive read exactly one interval after the last response.CertRotationContractTests,GateAndBypassTests,LeaseLifecycleTests,RetryBudgetEnforcementTestsreflection seam,CertRotationLiveTests), not caused by this PR.UA.slnxclean;dotnet format whitespaceclean.Not changed (follow-ups)
recvcalls per chunk (8-byte header, then body). Merging them needs read-ahead with carry-over, which the transport avoids so that idle peers do not hold receive buffers.ManagedSessionservice wrappers add an async state machine and a reader lock per call (~200 B).ResponseHeader(generator-wide, as noted in Server: cut request-path lock contention and allocation; add load harness #4707).MonitoredItemNotificationobjects and the V2DataValueChange[]per message dominate notification allocation (WithPoolNotificationspools the latter).🤖 Generated with Claude Code