Repository navigation
Server: cut request-path lock contention and allocation; add load harness - #4707
Merged
Merged
Conversation
Profiling a 20-session load (read/write/browse/subscribe) against the reference server showed the request path dominated by lock spinning and per-node allocation. Paired A/B medians (workstation GC, 24 cores): Read 1 value +45% req/s (CPU/req -37%, alloc/req -39%, lock contention -94%), Read 100 values +40%, encrypted Read 100 +48%, Browse +33%, Write 100 +13%, data-change notifications -17% alloc. - ObjectPool: lock-free slot array; ConcurrentBag.Count took every per-thread lock on each Return of the pooled response source. - FairRequestQueue: drop redundant gate entries (fail-fast pre-checks, stopped check, owner count folded into selection) and coalesce wake pulses so the bounded signal channel's lock is only taken when no token is pending. - Isolation provider: compare peer addresses without allocating; build the unsecured verification context only when a classifier is set. - TCP listener: cache the certificate DER copies per channel instead of copying X509 RawData for every request. - EndpointBase: resolve the request ActivitySource once instead of walking the stack (GetCallingAssembly) per request. - Node managers: reuse one ServerSystemContext copy per request in the per-node callbacks, remove a discarded copy, use plain dictionaries for per-request operation caches, build the validation cache in one pass and fix zero-capacity error lists. - ResourceManager: lock-free translation lookups from an immutable snapshot rebuilt after writes (Browse serialised on it). - MonitoredItem: share the default DataChangeFilter instead of allocating one per sampled value. - Reference server: enable Server GC; documented in ServerScalability.md (doubles Read throughput under load vs workstation GC). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ServerLoadHarness runs a reference server in a child process and drives read/write/browse/subscription load against it, reporting client latency percentiles and the server's CPU, allocation, GC and lock-contention deltas, optionally recording the window with dotnet-trace. TraceAnalyzer aggregates those traces; ab.sh/ab-summary.ps1 run paired comparisons of two builds. Not part of UA.slnx. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Cached mutable certificate bytes cross a deployment callback boundary, and the new tooling also violates repository-wide project and synchronization conventions.
12 open findings
What changed in this PR
Optimizes server request paths by reducing contention and allocations, enables Server GC, and adds performance tooling.
Changes:
- Introduces lock-free pooling, queue signaling, request-context caching, and translation snapshots.
- Reuses certificate/filter data and reduces node-manager allocations.
- Adds load generation, trace analysis, A/B scripts, and scalability documentation.
| File | Description |
|---|---|
src/Opc.Ua.Server/Subscription/MonitoredItem/MonitoredItem.cs |
Shares the default data-change filter. |
src/Opc.Ua.Server/Server/OperationContext.cs |
Caches per-request system contexts. |
src/Opc.Ua.Server/ResourceIsolation/DefaultServerResourceIsolationProvider.cs |
Reduces classification allocations. |
src/Opc.Ua.Server/NodeManager/ResourceManager.cs |
Adds immutable translation snapshots. |
src/Opc.Ua.Server/NodeManager/NodeManagerServiceDispatcher.cs |
Streamlines validation caches and list sizing. |
src/Opc.Ua.Server/NodeManager/CustomNodeManagerAsync.cs |
Uses ordinary operation dictionaries. |
src/Opc.Ua.Server/NodeManager/CustomNodeManager.cs |
Uses ordinary operation dictionaries. |
src/Opc.Ua.Server/NodeManager/AsyncCustomNodeManager.cs |
Reuses contexts and reduces cache allocations. |
src/Opc.Ua.Core/Types/Utils/ValueTask/ObjectPool.cs |
Replaces ConcurrentBag with lock-free slots. |
src/Opc.Ua.Core/Stack/Tcp/UaSCBinaryChannel.cs |
Uses cached certificate bytes. |
src/Opc.Ua.Core/Stack/Tcp/UaSCBinaryChannel.Asymmetric.cs |
Implements certificate-byte caching. |
src/Opc.Ua.Core/Stack/Tcp/TcpTransportListener.cs |
Reuses channel certificate bytes. |
src/Opc.Ua.Core/Stack/Server/FairRequestQueue.cs |
Reduces gate contention and coalesces wakes. |
src/Opc.Ua.Core/Stack/Server/EndpointBase.EndpointIncomingRequest.cs |
Uses the cached activity source. |
src/Opc.Ua.Core/Stack/Server/EndpointBase.cs |
Caches activity sources by telemetry context. |
samples/Reference/ConsoleReferenceServer/ConsoleReferenceServer.csproj |
Enables Server GC. |
perf/TraceAnalyzer/TraceAnalyzer.csproj |
Defines the trace-analysis tool. |
perf/TraceAnalyzer/Program.cs |
Aggregates CPU, allocation, and contention traces. |
perf/ServerLoadHarness/ServerLoadHarness.csproj |
Defines the load-harness project. |
perf/ServerLoadHarness/Program.cs |
Implements server and workload driver modes. |
perf/README.md |
Documents performance tooling. |
perf/ab.sh |
Runs paired benchmark scenarios. |
perf/ab-summary.ps1 |
Summarizes benchmark medians. |
perf/.gitignore |
Excludes generated performance artifacts. |
docs/ServerScalability.md |
Documents Server GC guidance. |
Directory.Packages.props |
Adds the TraceEvent package version. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Read, Write, Browse and Call validate every node they touch, and the dispatcher discarded the NodeMetadata (248 B, plus copies of the role permission arrays) right after the check. Browse also read 13 attributes per reference target only to check its RolePermissions. The role-permission and access-restriction validators now run on an internal PermissionMetadata struct holding exactly the attributes they read; the NodeMetadata overloads build it and delegate, so the checks are unchanged. AsyncCustomNodeManager fills the struct on the stack from the same three attributes and namespace defaults that GetPermissionMetadataAsync reads. The dispatcher uses it when the metadata is not needed, and Browse's reference filter uses it instead of GetNodeMetadataAsync. Unresolved handles still take the NodeMetadata path with its fallbacks. Read 100 values: alloc/request 120.6 -> 95.7 KB, ~21k -> 26k req/s; Browse: 67.6 -> 51.8 KB, 26.7k -> 29.8k req/s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ssion lookup locking SecureChannelContext.ClientChannelCertificate, ServerChannelCertificate and ChannelThumbprint are ByteString instead of byte[]. The TCP listener now shares one cached DER copy per channel with every request, and a mutable array handed to request handlers or IResourceIsolationClassifier code could have altered the evidence later requests are checked against. A ByteString is a read-only view, so the buffer is shared safely without copying. The constructor keeps its byte[] parameters and wraps them; a missing value is IsNull, so the session signature data (which skips a null certificate but hashes an empty one) is unchanged. Readers in Server, GDS, LDS and the Redundancy session mirror are migrated; the mirrored activation state keeps recording an empty, not null, certificate. MigrationGuide.md lists the mapping for application code. SessionManager.TryGetSessionContext runs at admission and at dispatch of every request and took the global bindings lock twice per call. The candidate snapshot now reads the concurrent session index and the activation-state table without the lock; the validating check under the lock still confirms the binding is the committed one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… translation snapshot ObjectPool: reuse, bounded retention, the per-processor cap, no shared rental under concurrent Get/Return, and no lost capacity after concurrent use. OperationContext.GetSystemContext: one copy per template, never another node manager's copy when templates alternate, and concurrent lookups bound to the same operation. ResourceManager: updates and new locales are visible after a snapshot was built, and concurrent readers see only complete translations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the OPC Foundation license header, drop the per-project Nullable, warnings-as-errors and analysis-level overrides and fix the diagnostics instead, remove comment section dividers, and record latency samples in a ConcurrentBag instead of locking List instances. The only suppression left is a documented CA2000 where the session takes ownership of a subscription. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Code coverage✅ Coverage gate passed.
Uncovered changed lines
Coverage is above the recorded baseline - consider ratcheting Thresholds live in |
…d pool edge cases - StandardServer compared the session certificate with the channel certificate through Utils.IsEqual, which bound to the object overload once the channel certificate became a ByteString and always failed (BadSecurityChecksFailed on secure CreateSession and TransferSubscriptions). Compare the spans. - The isolation provider's ChannelSnapshot copies the certificate again: the context wraps a buffer the transport owns, and revalidation must detect evidence that changed after the owner was issued. - PeerMatches has the array fallback only on .NET Framework; on .NET the 16-byte stack buffer always holds the address. - Tests for a stopped FairRequestQueue (admission and readers) and for the ObjectPool capacity argument. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
romanett
marked this pull request as draft
October 9, 2026 07:36
…mple The sampling loop waited one interval after every sample, so the timer overshoot (about 1 ms on Linux, up to the 15.6 ms timer resolution on Windows) and the sampling time were added to every cycle: a 10 ms group took 63 % (Windows) to 90 % (Linux) of its samples, and the o6-automation opcua-benchmarks subscription test (100 Hz, every item >= 99 % of samples) failed with a single monitored item. Samples now have absolute deadlines one interval apart; a late wake-up shortens the next wait, and a stall longer than four intervals (at least 50 ms) restarts the schedule instead of bursting. The benchmark workload now passes up to 4096 items (about 410k samples/s). NodeState.ReadAttributes fills the result array in place instead of copying a list, removing an allocation per sampled permission check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The TCP listeners sent each response through TcpServerChannel.SendResponse, which waits for the channel gate synchronously on the thread-pool worker that finished the request. With many requests in flight per channel, workers parked on the gate while the receive loop that was handed the gate waited for a free pool thread, and the channel stalled until the pool grew: 10 sessions with 100 outstanding Reads each dropped to 0-1.7k req/s (0 with SignAndEncrypt). SendResponseAsync waits for the gate asynchronously; the TCP, Kestrel TCP and WebSocket listeners use it, and the latter two use the cached certificate copies. Same load: 121-131k req/s (89-91k encrypted). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hmark `--scenario counters` runs a server whose UInt64 counters advance in their read callback (sampling groups, configurable sampling rate) and checks per item, on the client and on the server, that at least 99 % of the expected samples arrived. `--queue` sizes MaxQueuedRequestCount for deep pipelines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
marcschier
approved these changes
Oct 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…write queue
A server channel was faulted with BadTcpNotEnoughResources ("The channel
write queue is full") when a request arrived while more than 100 responses
were queued for writing, so a client pipelining more requests than that per
channel (e.g. the 1000 outstanding Reads per client of the o6-automation
benchmark) lost its channel. Server channels now stop reading while the queue
is full and resume once it has drained to 50; TCP flow control slows the
client instead. Requests already read when the queue filled stay bounded by
the request queue limits. Client channels keep reading, because responses
are what drain their writes.
10 clients x 1000 outstanding Reads: channels faulted before, now 0 errors at
about 45-54k req/s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In process the client drains responses fast enough that the server's write queue rarely fills, so the test checks that deep pipelines complete with the channels connected; the out-of-process harness exercises the full queue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of lag SamplingKeepsTheRequestedRateAsync measured the sampling rate against the wall clock and failed on a loaded macOS CI runner (78 % of the samples). The deadline arithmetic now lives in SamplingSchedule, and the tests drive it with simulated clocks: Windows- and Linux-like timer overshoot keeps the rate (where waiting a fixed interval drifts), a 200 ms pause is made up, a 3 s stall restarts the schedule once, and sampling slower than the interval does not run away. The schedule now tolerates a lag of four intervals or 500 ms (was 50 ms) before restarting, so a garbage collection or scheduling pause of that length is made up rather than lost; on a loaded machine this kept more samples at 4096 items (81 % vs 62-71 %). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new sampling-delay conversion overflows for the default one-hour interval on Unix, terminating the sampling loop.
2 open findings
12 resolved since last review
🧠 Review effort: Balanced
…eaders The wait before a sample was converted with remaining * TicksPerSecond / frequency, which overflows a long for long intervals at the 1 GHz timestamp frequency of Unix (one hour is 3.6e19); Task.Delay then threw and ended the sampling loop. SamplingSchedule.ToTimeSpan scales through a double, covered by a one-hour test at 1 GHz. ResourceManagerSnapshotTests awaits its readers with a timeout instead of blocking on Task.WaitAll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 of 7 tasks
marcschier
pushed a commit
that referenced
this pull request
Oct 9, 2026
…hs (#4715) ## Summary Allocation audit of the per-message / per-value paths (binary codec, built-in types, server request path, monitored items, client notification dispatch) looking for boxing and avoidable per-value copies. Measured with `GC.GetAllocatedBytesForCurrentThread` probe loops on net10.0 **and net48**, and with `dotnet-trace` allocation traces of the server load harness (#4707) for read/write/browse/subscription, server and client side — a value type in the "allocated types" list is a box. Wire output and error behaviour are unchanged. ### Fixed | Where | Cause | Frequency | Before → after | |---|---|---|---| | `Polyfills/System.IO.cs` (net48/netstandard2.0) | `BinaryWriter.Write(ReadOnlySpan<byte>)` polyfill copied with `ToArray()` — every string, array, ByteString, Guid write | per value | 32–64 B → 0; BrowseResponse×50 encode 15.3 → 10.9 KB | | `EnumHelper` (net48/netstandard2.0) | `EnumToInt32`/`Int32ToEnum`/`EnumToInt64` boxed outside `NET8_0_OR_GREATER` (every enum field, `Variant.From<TEnum>`, `EnumValue.From`) | per value | 24 B → 0 | | `BinaryDecoder.ReadString` + stream raw reads (net48) | temp `byte[]` per string, `BinaryReader.ReadBytes` | per value | BrowseResponse×50 decode 15.7 → 11.3 KB | | `WriteGuid` / `ReadGuid` (all) | `ToByteArray()` / `byte[16]` | per value (also Guid NodeIds) | 40 B → 0 each | | `VariantHelper.TryCastTo<TEnum>` (`DataValue.GetValue<TEnum>`), `EventRecordFieldReaders.GetEnum` | `Enum.ToObject` box | per call / per event field | 24 B → 0 | | `NodeManagerServiceDispatcher.GetManagerHandleAsync` | interface enumerator per validated node | per node | Write-100: 151.8 → 148.5 KB/request | ### Cold paths (third commit) - `Server.Auditing` write handler: `Convert.ToBoolean(object)` got a boxed `Variant` (not `IConvertible`) and threw `InvalidCastException` on any in-process write; now takes the boolean, `BadTypeMismatch` otherwise (new test). - `UpdateServerServiceLevel`: `Convert.ToByte(byte, IFormatProvider)` bound to the object overload. - Client publish-state / recovery-policy `HasFlag` (boxes on net48) → `(x & F) == F`. - `SequenceReader` big-endian float on net48/netstandard2.0 used `Convert.ToSingle(int)` (numeric conversion) instead of reinterpreting the bits. - JSON enum strings: one `Enum.GetName` + `string.Concat` instead of `IsDefined` + boxing `CoreUtils.Format`; same output (new test for the `EnumValue` overload). Left as is: `SamplingGroup` start-up `Convert.ToInt32(double, provider)` (rewritten by #4707), `SecureChannelRenewed` formatting (transport files in flight on the client perf branch), `Utils.IsEqual<T>` object path for structs (cold callers; changing it would mix `CompareTo` and `Equals` semantics), Machinery `HasFlag` (startup config). ### Not changed here - `BinaryEncoder.GetNodeIdEncoding` `Convert.ToUInt32(object)` / `Convert.ToByte(enum)` (72 B per NodeId encode, largest single item) and `ChannelAsyncOperation.Complete` boxing the `int` result are handled on the client performance branch. - By design: `Variant` / `NodeId` store `ArrayOf<T>`, Guid, ByteString identifiers boxed (`Variant.Copy` re-boxes RolePermissions during permission checks). ## Tests - New `BinaryCodecAllocationTests`: round trips in buffer and stream decoder mode, truncation errors, allocation per call (< 1 B/call over 20k calls; the counter can be off by an allocation quantum when other tests GC). - `Enum.ToObject` equivalence for `TryCastTo` and `EventRecordFieldReaders.GetEnum` across all underlying types. - net10.0 (third commit): Types 11465, affected Server 87, Client subscription 642 passed; net48 Types 11458 passed, `UA.slnx` 0 warnings. - net10.0: Types 11464, Core.Encoders 2947, Server 8386 passed; Core 6197 passed, 1 unrelated (`ActivityTraceFlags_Metrics_ShouldRecordMetricsOnly`, passes alone). - net48: Types 11457, Core.Encoders 2947, EventRecordFieldReaders 25 passed; `UA.slnx` builds with 0 warnings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
marcschier
added a commit
that referenced
this pull request
Oct 9, 2026
# Description
A server did not start in globalization-invariant mode
(`InvariantGlobalization`, or
`DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`), which Native AOT and
container images often enable. When the server loaded its default status
texts, `ResourceManager` created `CultureInfo("en-US")`. In this mode
the runtime throws `CultureNotFoundException` for every culture except
the invariant one, so the server crashed in
`ResourceManager.LoadDefaultText`.
`ResourceManager` now:
- keeps the locale id of each translation table as a string, and falls
back to its language subtag to find another region of the same language;
- when the runtime cannot create the culture, parses a locale id that
matches the grammar of RFC 5646 §2.1 into the recommended casing of its
subtags. It still rejects a language without a region, and rejects a tag
that does not match the grammar, as before;
- caches the resolved locale ids, so loading the default texts creates
each culture once.
`docs/NativeAoT.md` no longer warns against invariant mode for servers.
The new tests (`ResourceManagerInvariantGlobalizationTests`) reproduce
both invariant-mode behaviors of the runtime: no cultures at all, and
cultures that carry invariant data. They also cover malformed tags, and
well-formed tags with extended language, variant, extension, and private
use subtags.
The branch includes a merge of master: #4707 moved translation lookups
to a lock-free snapshot of the tables, and the snapshot now copies the
locale-id-based tables.
### Validation
| Command | Result |
| --- | --- |
| `dotnet test tests/Opc.Ua.Server.Tests -f net10.0
-p:CustomTestTarget=net10.0 --filter
"FullyQualifiedName~ResourceManager"` | 100 passed, including the 39
tests of the new fixture. Before the fix, 10 of its first 22 tests
failed; 7 of the 17 grammar cases failed against the first version of
the parser |
| `dotnet test tests/Opc.Ua.Server.Tests -f net48
-p:CustomTestTarget=net48 --filter "FullyQualifiedName~ResourceManager"`
| 100 passed |
| `dotnet test tests/Opc.Ua.Server.Tests -f net10.0
-p:CustomTestTarget=net10.0`, on 80e5163 (before the master merge) |
8408 passed, 0 failed, 5 skipped. The run exits 1 because of a
pre-existing certificate leak in `PushComplianceRegressionTests`, which
is unrelated to this change and fixed in #4719 |
| `dotnet build src/Opc.Ua.Server -c Release` (net10.0, net9.0, net8.0,
net48) | 0 warnings |
| MinimalBoilerServer started with
`DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1`, then MinimalClient (on
27c8478) | Before the fix: the server crashed with
`CultureNotFoundException`. After: it starts, and the client browses,
reads, and receives a subscription notification |
## Related Issues
- No tracking issue. Found while verifying the Native AOT documentation
(follow-up to #4555).
## Checklist
- [ ] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [x] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [x] I have added all necessary documentation.
- [x] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [x] I have addressed **all** PR feedback received.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: marcschier <11168470+marcschier@users.noreply.github.com>
marcschier
added a commit
that referenced
this pull request
Oct 10, 2026
…; measure the client in the load harness (#4720) Follows #4707 (replaces #4714, which closed when the #4707 branch was deleted). Profiling the client process under load showed per-request allocation spread over the channel write path, the response wait, per-chunk header encoders and an AES instance per message, plus a stack walk per service call. This PR removes them and extends the load harness to measure the client. ## Results Paired A/B against the #4707 head, the server work now on master (3 alternating rounds, medians, 20 sessions unless noted, workstation GC on both sides, 24-core machine). Client = the harness process, server = the reference server child. | Scenario | Throughput | p50 | p90 | Client CPU / op | Client alloc / op | Server alloc / op | |---|---|---|---|---|---|---| | Read, 1 value | **+21%** (141k → 170k req/s) | −9% | −11% | −13% | −46% (4.5 → 2.4 KB) | −20% | | Read, 1 value, ManagedSession | **+18%** | −9% | −12% | −18% | −44% | −20% | | Read, 1 value, SignAndEncrypt | **+28%** | −14% | −20% | −24% | −52% | −26% | | Read, 1 value, 200 sessions | **+29%** | −11% | −17% | −17% | −46% | −20% | | Read, 100 values | +5% | −2% | −3% | −6% | −44% (15.6 → 8.8 KB) | −2% | | Read, 100 values, SignAndEncrypt | +8% | −7% | −2% | −2% | −46% | −3% | | Write, 100 values | ±0 | ≈ | ≈ | −14% | **−63%** (10.8 → 4.0 KB) | −2% | | Browse (Server object) | **+12%** | −7% | −11% | −19% | −17% | −12% | | Data changes, classic fast callback | ±0 ¹ | ≈ | ≈ | −21% | −10% | −5% | | Data changes, classic item events | ±0 ¹ | ≈ | ≈ | ≈ ² | −7% | −5% | | Data changes, ManagedSession (V2) | ±0 ¹ | ≈ | ≈ | ±0 | −5% | −5% | | Data changes, SignAndEncrypt | ±0 ¹ | ≈ | ≈ | −22% | −14% | −7% | ¹ Throughput is bounded by the writer rate; latency is dominated by the 100 ms publishing interval. Per notification the client uses about 2 µs and 170–260 B. ² The client uses 0.2–0.4 cores in the subscription scenarios; the CPU per notification is within run-to-run noise there. Client GC pause time in the window drops 38–66% in the Read and Write scenarios (17% for Browse). About 70% of the remaining client CPU per small request is kernel time in loopback socket send/receive. ## Changes - **Activity source**: `Session`, `SessionClientBatched`, the classic `Subscription` and `ClassicSubscriptionEngine`, and the TCP/HTTPS client channels resolve their `ActivitySource` once. `TelemetryExtensions.StartActivity` called `Assembly.GetCallingAssembly()` (a stack walk) on every service call, ~8% of client CPU. - **Channel write queue** (`UaSCUaBinaryChannel`, client and server): an ordered queue drained by one thread-pool writer replaces the per-write `ContinueWith`/`Unwrap`/trailing continuation chain. The writer is queued to the issuing thread's local pool queue, like the task continuation was (queuing it globally cost ~15 µs of p50 in a first iteration). Successful writes are reported inline instead of through `Task.Run`; failed writes still go through the pool. The client channel enters its gate only for a failed write instead of on every write completion. - **`TcpByteTransport`**: no linked `CancellationTokenSource` when the caller's token cannot be cancelled (the channel always passes `None`); a single-chunk `BufferCollection` uses the `ReadOnlyMemory` send instead of the vectored `Task<int>` send; `PoolingAsyncValueTaskMethodBuilder` on .NET 6+ for the receive/send state machines; no closure per vectored send. - **`BinaryEncoder.GetNodeIdEncoding`**: `Convert.ToUInt32(uint, IFormatProvider)` bound to the `object` overload and `Convert.ToByte(enum, ...)` boxed the enum, two boxes per encoded NodeId. 44% of the client allocation of a 100-value Read; the server encodes NodeIds the same way. - **Symmetric chunk headers**: the 24 header bytes are written and read with `BinaryPrimitives` instead of a `MemoryStream` + `BinaryEncoder`/`BinaryWriter` (send) and `BinaryDecoder`/`BinaryReader` (receive) per chunk. A chunk too short for a field still gets the decoder's `BadDecodingError`. - **AES-CBC**: `ChannelToken` keeps one `Aes` per direction, used with the one-shot `EncryptCbc`/`DecryptCbc` on .NET 6+ (net48 reuses the instance and creates the transform). Previously every message created an `Aes`, imported the key (`BCryptImportKey`) and created a transform, ~15% of client CPU with SignAndEncrypt. The instance is taken out of the token while in use, so concurrent callers never share one. The public `CryptoUtils` overloads are unchanged. - **`ChannelAsyncOperation`**: `SendRequestAsync` waits on an internal task-free completion source instead of `EndAsync` (`TaskCompletionSource` + `Task` + `WaitAsync` promise + state machine); `EndAsync` registers cancellation instead of `WaitAsync(ct)`; the timeout timer uses a static callback; completing with `default(T)` shares a box. - **Session keep alive**: every good response called `Timer.Change` under the session lock, taking the timer queue lock. It now records the time; the timer callback defers itself to one interval after the last response, which is when the per-response reset made it fire. ### Behaviour notes - `TcpByteTransport.SendChunkAsync`/`ReceiveChunkAsync` return pooled `ValueTask`s on .NET 6+: like any `ValueTask` they must be awaited once. - A successful write's `HandleWriteComplete` now runs on the writer instead of a separate pool work item. Overrides outside the stack that block in it delay later writes on that channel. - The write itself no longer flows the caller's `ExecutionContext` (`Activity.Current` is not set inside the transport send). ## Harness (`perf/`) - `client-process:` and `client-per-op:` result lines; `--trace-side client` records the harness itself. - `--client managed` (ManagedSession, V2 engine) and `--callback item` (classic per-item events). - Classic fixture sessions no longer request diagnostics in every response (the server otherwise sends a `DiagnosticInfo` per notification). - `ab.sh`: `SCENARIOS`/`ROUNDS`/`OUT`, CRLF tolerant, defaults before the scenario (a `--sessions 200` scenario silently ran with 20 before); `client-scenarios.txt`; `ab-summary.ps1` reports client and server medians. - README notes two profile traps: `Monitor.Enter_Slowpath` in the sampled CPU profile without matching contention events (46 in 10 s here), and allocation ticks over-attributing small objects allocated after large arrays. ## Testing - New: `UaSCBinaryChannelWriteQueueTests` (order, one send at a time, never on the caller's stack, failures reported and later writes still sent, concurrent writers), `ChannelAsyncOperationWaitTests` (late/early completion, faults, cancellation, disposal, continuation off the completing thread, 2000-iteration completion/cancellation race, timeout), `CryptoUtilsAesCacheTests` (cached = fresh cipher text across IVs, round trips, concurrent users, token disposal), `TcpByteTransportTests` single/multi-buffer and cancellable sends, `SessionKeepAliveTests` keep-alive read exactly one interval after the last response. - After merging master: Core Stack 3571 passed, Client 2763 passed, 0 failed. - Before merging master, net10.0: Core Stack+Security 4680 passed; Client 2763; Sessions 849 (379 skipped); Server 8391; Subscriptions 643; all 0 failed. - Stress Contract+Integration: 5 failures, the same 5 on the unchanged #4707 head (`CertRotationContractTests`, `GateAndBypassTests`, `LeaseLifecycleTests`, `RetryBudgetEnforcementTests` reflection seam, `CertRotationLiveTests`), not caused by this PR. - net48 build of `UA.slnx` clean; `dotnet format whitespace` clean. ## Not changed (follow-ups) - Two `recv` calls per chunk (8-byte header, then body). Merging them needs read-ahead with carry-over, which the transport avoids so that idle peers do not hold receive buffers. - `ManagedSession` service wrappers add an async state machine and a reader lock per call (~200 B). - Generated responses allocate a discarded `ResponseHeader` (generator-wide, as noted in #4707). - Decoded `MonitoredItemNotification` objects and the V2 `DataValueChange[]` per message dominate notification allocation (`WithPoolNotifications` pools the latter). - Coalescing queued writes on one channel into one vectored send would cut syscalls under concurrent requests on a shared channel. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Marc <marcschier@hotmail.com> Co-authored-by: Marc Schier <marcschier@users.noreply.github.com>
romanett
added a commit
that referenced
this pull request
Oct 12, 2026
…that are actually reused, ready list instead of an O(N) scan (#4748) Roadmap item A5 of the o6 benchmark gap analysis. Every Publish response deep-cloned its NotificationMessage in `SentMessageQueue` (PR #4707 kept the clone because a retained message returns its pooled payload when the client acknowledges it, which can happen before the response was sent). This PR replaces the clone with a send-completion lease, makes the activator pool that recycles the notifications O(1) and sizes it for the server's load, builds messages without intermediate copies (the notification array is rented), and lets monitored items tell their subscription when they have something to report so the publish timer no longer walks every item. ## Results Paired A/B against master `0dff94bd3` (includes #4742 and #4743; alternating builds, 3 rounds, medians; 20 sessions unless noted, workstation GC, 24-core machine shared with other sessions). Server = the reference server child process. | Scenario | Server alloc / op | Server GC pause | Server CPU / op | Write→notify p50 / p99 | Throughput | |---|---|---|---|---|---| | `sub`, 5 subs × 100 items, pub 100 ms | **−57 %** (373 → 159 B) | **−57 %** | −7 % ¹ | −1 % / −2 % | +1 % ² | | `sub`, 5 subs × 100 items, pub 10 ms | **−57 %** (375 → 163 B) | **−52 %** | +9 % ¹ | −22 % / +1 % | ±0 ² | | `sub`, pub 100 ms, SignAndEncrypt | **−57 %** (380 → 163 B) | **−53 %** | +4 % ¹ | −11 % / **−12 %** | ±0 ² | | `sub`, 2 sessions × 1 sub × 1000 items, 20 changing, pub 50 ms (9 rounds) | −2 % ³ | ±0 | −23 % ³ | ±0 / −2 % | ±0 | | `counters`, 4000 items, 10 ms sampling, pub 1 s (per notification) | −2 % | −6 % | −6 % | – | ±0 | | `o6counters`, 2048 items (per sample) | −3 % ⁴ (111 → 108 B) | −5 % | −7 % | – | ±0, all PASS | | `o6counters`, 4096 items (per sample) | −3 % ⁴ | +14 % ⁵ | −6 % | – | ±0, all PASS | ¹ CPU per notification varies by ±15 % between rounds on this machine; in an earlier 8-round batch against `071660354` the same scenarios measured −14 % / −1 %. Read it as "no regression". Client CPU per notification moved −18 %…0 %, within its spread. ² Bounded by the writer rate. ³ The server is nearly idle (0.05–0.12 cores, 360 notifications/s; its allocation is the writer and keep-alives). The first 3 rounds measured +31 % CPU, 6 more rounds −23 % (medians 969 → 742 ms per window): noise-bound, no regression. ⁴ With #4743's sampling permission cache a sample costs 111 B; one Publish per second carries 2048/4096 notifications. ⁵ The activator pools no longer allocate their 1023-slot arrays up front; the heap is a few MB smaller, which shifts the GC's generation budgets so that some low-allocation runs make every collection gen1 (26 of 26 instead of 2). Pause totals differ by milliseconds per 10 s window (37 → 42 ms here, 21 → 20 ms at 2048 items). Allocating the arrays eagerly restored master's gen1 counts in a separate test. In a single CPU trace of `sub` at pub 10 ms, recycling acknowledged payloads (pool `Return`) went from 16 % of server on-CPU time (master, the full pool scanned per return) to 5 %. ## Changes - **Send completion instead of a deep clone** (`NotificationMessageLease`, `LeasedPublishResponse`, `SentMessageQueue`): the retransmission queue and every Publish response that carries a message each own a reference. The transports already call `IPooledEncodeable.Reuse()` on a response once it is encoded or dropped; the Publish response is now a `LeasedPublishResponse` whose `Reuse()` releases its reference. The pooled payload (`NotificationMessage`, `DataChangeNotification`/`EventNotificationList`, `MonitoredItemNotification`/`EventFieldList`) is recycled when the last owner releases it, so an acknowledgement that overtakes the send no longer recycles a payload that is still being encoded, and the queue and the response share one message. A reference that is never released only leaves the payload to the GC; every owner releases at most once. Keep-alive messages are owned by their response alone. Republish, the durable snapshot and the retransmission mirror still copy. - **Transports**: the HTTPS (binary and JSON), WebSocket (UA-SC, JSON, OpenAPI) and Kestrel TCP listeners and the WebAPI endpoint dispatcher now release responses after encoding, as the TCP listener does, so leased payloads are recycled there too. A response retained for a reconnect is released when it is sent; a send that throws leaves the response to the GC. - **Activator pool** (`BoundedObjectPool`): rent and return scanned the 1023 slots linearly; with the pool full every returned notification scanned all of them before being dropped (16 % of server CPU at 94k notifications/s). It is now a `ConcurrentQueue` behind the lock-free first slot with an `Interlocked` bound (a locked stack spun under contention when transport threads return whole messages). The queue is created on first use, so activators of types that are never recycled no longer allocate a 1023-slot array each. The client decoder rents through the same pool. - **Pool bound** (`PooledEncodeableType<T>.EnsureMaximumRetained`, internal): the bound can be raised, never lowered. The subscriptions raise the `MonitoredItemNotification` and `EventFieldList` pools to twice the number of monitored items in the process (a notification is recycled about two publishing cycles after it was built), between the default 1024 and 65536. At the cap the retained notifications take about 5 MB; the bound stays raised after items are deleted. - **Message construction** (`Subscription.InnerPublish`/`ConstructMessage`, `MonitoredItem.Publish`): the per-Publish `Queue<>`/`List<>` objects are reused by the subscription; messages are built at their final size (no `List`→`ArrayOf` copy, no `AddItem` reallocation); the `MonitoredItemNotification[]` of a message is rented from `ArrayPool` and the message carries an exact-length slice of it; the lease returns the array (cleared) with the payload, and messages dropped on queue overflow return theirs at once; only arrays the subscription rented are tracked, so restored, mirrored and copied messages never reach the array pool; the `DiagnosticInfo` list is only allocated when a notification has one; timestamps are stripped on the struct before the pooled notification is filled. - **Ready list** (`IReadySignalingMonitoredItem`, `Subscription.PromoteReadyMonitoredItems`): `MonitoredItem` acquires a ready signal (compare-exchange, a full fence after its flags are set) when it queues a value or event or a resend is requested, and hands itself to the subscription's intake list. The publish timer checks only these candidates instead of every item under the subscription lock. An item stays a candidate while it has something pending that can become reportable without a new value (sampling deadline, aggregate interval, samples of a SAMPLING item, trigger flag); after releasing the signal the subscription checks again, so a value queued concurrently is never missed. Created, restored, modified, monitoring-mode and triggering changes mark candidates, as does an item that leaves the publish list with something pending. ### Behaviour notes (custom subscription or monitored-item extensions) - **Publish responses share the retained message.** `PublishResponse.NotificationMessage` returned by `SubscriptionManager.PublishAsync` is the retained message itself, not a copy, and its payload objects return to the activator pools after the response was sent and acknowledged. Code that keeps or modifies the message of a Publish response (for example in a `StandardServer.PublishAsync` override or a request interceptor) must copy it first (`CoreUtils.Clone`). `Republish` still returns a copy. - **Custom transports**: a transport that does not call `Reuse()` on responses keeps working; the payload is then collected by the GC instead of being pooled. - **Custom `IMonitoredItem` implementations** (not derived from `MonitoredItem`) cannot signal readiness: a subscription that contains one scans all its items on every publishing cycle, as before. `MonitoredItem` subclasses signal through the base class. - **`DataChangeNotification.MonitoredItems` of a published message is a slice of a pooled array**: read it through `ArrayOf<T>` (`Span`, indexer, `Count`); code that unwraps the underlying array would see a longer array. - **Order within a message**: notifications of different monitored items appear in the order the items became ready rather than in the internal list order. OPC UA does not define an order across items; the order of the values of one item is unchanged. ## Harness (`perf/`) - `--write-nodes K` for `sub`: the writer changes only the first K monitored nodes, for large subscriptions in which few items change. - `server-scenarios.txt`: publish-path scenarios (`--pub 10`, 1000 items per subscription with 20 changing). ## Testing - New `NotificationEncodingLifetimeRegressionTests` cases: an acknowledgement before send completion does not recycle the payload still being encoded (the send completion does); Republish after the send completed returns the original content and stays independent of the recycled original; a failed send that releases the response twice, plus a copy of the response, recycles the payload exactly once and only after the queue released it; a keep-alive is owned by its response; the rented notification array stays intact until both acknowledgement and send completion (either order), the message encodes only its slice, and queue overflow returns the arrays of dropped messages. The existing 30-case matrix (publish / queued / republish / mirrors × acknowledge / evict / clear × data / events) now asserts that a shared payload survives every queue mutation until its response is released. - New `ReadyListRegressionTests`: the same scenarios on a subscription that uses the ready list and on one that scans (it also has an item that cannot signal) must publish the same notifications, for triggering (sampling items, backlog of the triggering item), monitoring-mode changes (Sampling↔Reporting, Disabled, disabled before publish), disabled publishing with keep-alives and ResendData, a value queued while the item was in the publish list, and the backlog of an item that left Reporting while in the publish list. Mutation check: removing the item's signal fails 5 of 6, removing the subscription-side marks fails the backlog case. Plus: a published item without pending values is not tracked; creating 1200 items raises the notification pool bounds to at least 2400. - `PooledEncodeableTypeTests`: the raised bound is respected and never lowered. - net10.0 on the rebased branch: Server 8481 passed (5 skipped); Sessions 851 (379 skipped); Client 2788; Subscriptions.Classic 57; Subscriptions.Durable 36 (11 skipped); Redundancy.Server 884, Redundancy.Client 134, Redundancy.Samples 17; Stress Contract+Integration 45; Subscriptions 657 (6 skipped); Types pool tests 165. All 0 failed. - net48 build of `Opc.Ua.Server.Tests` and `Opc.Ua.Bindings.Https` and `Opc.Ua.Types.Tests` clean; `dotnet format whitespace` clean. ## Not changed (follow-ups) - Republish still copies the message (`ISubscriptionManager.Republish` returns a `NotificationMessage`, so the response cannot carry a lease without an API change). - `EventFieldList[]` per event message could be rented like the data change array. - Sampling remains the larger share of allocation and CPU in the `counters`/`o6counters` scenarios. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
marcschier
pushed a commit
that referenced
this pull request
Oct 13, 2026
…ue-wide lock (#4750) # Server: cheaper symmetric crypto path, FairRequestQueue without a queue-wide lock Roadmap items E1 and 5b of the o6 benchmark gap analysis, one commit each. - **E1:** Windows one-shot `Aes.EncryptCbc`/`DecryptCbc` import the key (`BCryptImportKey`) on every message, although #4720 caches one `Aes` per channel token. The AES-GCM and ChaCha20-Poly1305 policies also created a new keyed instance per message. - **5b:** with `AsyncCustomNodeManager`, which is what the o6 benchmark's 2.0.1 worker uses, the `FairRequestQueue` gate became the largest waiting point. `TryEnqueue`, `TrySelect` and `TryDequeue` spun on one `Lock` for every request. No wire, configuration or public API changes. ## Results These are paired A/B runs against `origin/master`, five rounds per scenario. The two harness builds alternate per round, starting side swapped every round, on the same shared machine (Ryzen AI 9 HX 370, 12 cores / 24 threads, Windows 11). The tables show medians, with ranges in brackets where they matter. The server is configured like the o6 2.0.1 worker: `AsyncCustomNodeManager`, Server GC, queue sized clients × outstanding + 3. o6read uses the benchmark's native open62541 client workers; `read` uses the harness's .NET client. On this machine throughput moves between rounds by up to 2×, so CPU and bytes per request are the reliable signals. ### E1: symmetric crypto (measured on 0dff94b) | Scenario | req/s | Server CPU µs/req | Server alloc B/req | Client CPU µs/req | Client alloc B/req | |---|---|---|---|---|---| | read --nodes 1, Basic256Sha256 SignAndEncrypt, 24 cores | 115,300 → 103,600 (ranges overlap) | 64.6 → 60.9 (**−6 %**) | 5,698 → 5,486 | 44.9 → 41.7 (**−7 %**) | 2,672 → 2,456 | | o6read 10×100, encrypted, 4-CPU mask | 71,300 → 75,900 | 51.8 → 51.4 | 6,323 → 6,111 | – | – | | o6read 10×100, encrypted, 24 cores | 123,500 → 118,200 (ranges overlap) | 115.7 → 117.2 | 6,716 → 6,515 | – | – | | o6read 10×100, unencrypted, 4-CPU mask (control) | 79,100 → 70,600 | 48.0 → 48.4 | 6,088 → 6,090 | – | – | Isolated costs per call (one pinned core, 224-byte chunk, variants alternating in one process): | Operation | Before | After | |---|---|---| | AES-256-CBC encrypt | 778 ns, 128 B (one-shot) | 329 ns, 24 B (cached transform) | | AES-256-CBC decrypt | 682 ns, 128 B | 239 ns, 24 B | | AES-256-GCM seal | 1,022 ns, 72 B (new `AesGcm`) | 260 ns, 0 B (cached) | | HMAC-SHA256 | 306 ns (per-token instance, unchanged) | – | For comparison, `HMACSHA256.TryHashData` costs 458 ns and `IncrementalHash` 318 ns. **E1 does not close the 20–50 µs encrypted gap, and no crypto change could.** Pinned to four CPUs, encryption costs master only about 3–4 µs per request (51.8 vs 48.0 µs). The extra cost at 24 cores comes from threads spinning on locks while signing and encryption run inside the channel gate (roadmap A1). The ~7 µs of HMAC in the report's profile is a thread-time artefact. `HashProviderCng.AppendHashData` shows 40× more samples on the sign side, inside the gate, than on the verify side for the same chunk sizes. Threads are preempted there while spinners hold the CPUs. ### 5b: FairRequestQueue (unencrypted, E1 has no effect here) | Scenario | req/s | Server CPU µs/req | Server alloc B/req | Contention events/s | |---|---|---|---|---| | o6read 10×100, 24 cores | 149,000 → 150,200 | 110.8 [106–117] → 99.6 [96–109] (**−10 %**) | 6,404 → 6,367 | 4,533 → 2,125 | | o6read 10×1000, 24 cores | 152,200 → 161,200 | 106.7 → 96.3 (**−10 %**) | 6,421 → 6,358 | 4,131 → 1,921 | | read --nodes 1 --sessions 200, 24 cores | 126,100 → 141,200 (+12 %) | 53.6 → 51.1 (−5 %) | 5,353 → 5,294 | 4,078 → 455 (**−89 %**) | | o6read 10×100, 4-CPU mask | 85,600 → 83,900 | 45.7 [44.1–47.8] → 46.8 [44.4–48.9] (noise) | 6,089 → 6,060 | 281 → 176 | Traces of o6read 10×100 at 24 cores, one run per side: - **Lock spinning.** In the sampled CPU profile, frames above `Lock.TryEnterSlow` in the fair queue fell from **16.6 % of on-CPU samples** (`TryEnqueue` 5.8, `TrySelect` 5.8, `TryDequeue` 4.9) **to none**. - **Contention events in the window:** 15,331 → 5,879. - **What remains of the queue:** - the wake channel in `DequeueAsync`: idle workers parked on the coalesced signal, 1.3 % of on-CPU samples, kept unchanged; - `SessionManager.TryGetSessionContext`'s server-wide `m_bindingsLock`, which every request takes at admission and again at dispatch. It is now the largest spin point (~26 % of `Lock.TryEnterSlow` samples) and is a separate follow-up. The channel gate (A1) is another ~17 %. In the o6 configuration all ten sessions map to **one** owner, so per-owner locks would not have helped. An intermediate version with a lock per owner FIFO left 12.8 % of the samples on those locks. That is why the FIFO itself is lock-free. ## Changes ### Symmetric cipher cache (E1) - `SymmetricCipherCache` replaces `AesCache`; a `ChannelToken` keeps one per direction (`ServerCiphers`, `ClientCiphers`). Per slot it retains: - a CBC encryptor keyed with the token's key and IV; - a CBC decryptor keyed the same way; - the AES-GCM or ChaCha20-Poly1305 instance. - **How a CBC message is processed:** the cached `ICryptoTransform` transforms the whole message in place with `TransformBlock`. An empty `TransformFinalBlock` (padding `None`) then rewinds the chaining state to the IV without re-keying, on CNG, OpenSSL and .NET Framework CAPI alike. Cached and fresh ciphertext are tested equal. - **Concurrency and ownership:** - A cipher is taken out of its slot while in use, so concurrent senders never share one. - It is matched to the key and IV by reference, and replaced if the key material differs. - It is disposed instead of returned after a failed transform, or once the token has been disposed. - **Unchanged:** - The uncached public `CryptoUtils` overloads keep the one-shot path. - The order is still verify, then decrypt, then padding, so a tampered padding byte under a valid signature is still reported as padding, after the signature check. - HMAC is unchanged. ### FairRequestQueue (5b) - **No lock in admission or dispatch.** - Each owner has a lock-free FIFO: Michael–Scott, with the entries themselves as the nodes, so queueing allocates nothing. The `LinkedListNode` per request is gone. - A request is claimed exactly once, by a compare-and-swap from selection, cancellation or stop. Claimed entries are dropped lazily when they reach the head. - **Weighted round robin over a lock-free ready queue of turn tokens.** - A turn holds one token per queued request, up to the owner's weight. A weight of 1,000,000 therefore costs no more scheduling state than the backlog. - Taking a turn's last token queues the owner's next turn at once, behind the other owners. The turn state is one 64-bit word changed by CAS, so the service order is the token order even while workers race. Weights 1:3 under 8 concurrent workers stay within 2.5–3.5. - **Failed execution admission:** the head goes to a small sorted list under the owner's gate, served before the FIFO. Its owner is not retried before capacity changes, as before. - **Owner FIFO retirement:** an owner FIFO is retired by an atomic CAS of its count from 0 to a sentinel, and a retired FIFO can never be counted again. The next request of that owner starts a new FIFO, and the retired FIFO's leftover tokens are skipped. - **Unchanged:** - The count bound is reserved with `Interlocked` before linking. - Global FIFO mode still selects one at a time, so a blocked head cannot be overtaken. - Wake coalescing (#4707), parked-request decoupling and `ServerBase.RequestQueue`'s worker growth. ## Behaviour notes - **Cipher cache memory:** a token now keeps up to two CBC transforms (plus their `Aes`) or one AEAD instance per direction, instead of one `Aes`. They are released with the token. - **Weighted ordering under concurrency:** dispatch order across owners follows token order. Within an owner, admission order is kept, including requests that fail execution admission and go back to the front. - **Idle scheduling state:** cancelled requests leave stale turn tokens in the ready queue until the next dispatch pass. They are discarded at once when the last queued request is cancelled. - **net48:** the BouncyCastle-backed AEAD types are cached like the platform ones. ## Testing - **New and extended tests:** - `CryptoUtilsAesCacheTests`, for all CBC and AEAD policies (Basic128Rsa15, Basic256, Basic256Sha256, Aes128_Sha256_RsaOaep, Aes256_Sha256_RsaPss, ECC_nistP256, ECC_nistP256/P384/RSA_DH AesGcm, ChaChaPoly): - cached vs fresh ciphertext for 0–300-byte bodies; - IV or key change; - tampered signature, ciphertext, tag and header, each rejected with the cache recovering; - bad padding under a valid signature; - 256 concurrent senders; - token disposal during a transform, and a return racing disposal. - `FairRequestQueueTests`: - turn tokens bounded by the backlog, not the weight; - cancellation leaves no tokens behind; - 1:3 weights under 8 workers; - no lost wakeups (4 producers, 8 waiting workers, both modes); - cancellation racing dequeue completes or dispatches each request exactly once and returns every lease; - stop while workers wait and admissions race; - a 50-owner, 16-worker stress run with cancellations; - pipelined clients (1×200, 10×100, 64×1) with ample and with scarce execution capacity. - The last test stalls on an earlier version of this change that briefly marked an emptying owner as retired, when the race window is widened with an injected delay. That bug stalled o6read 10×100 every time in the harness; a dump showed 1,000 queued requests with no turn tokens. Without the delay the window is too narrow to hit at unit level. - All 90 FairRequestQueue tests pass 30/30 repeated runs on net10.0 and 5/5 on net48. - **Suites,** net10.0, after rebasing on 8b360da (#4748): | Suite | Passed | Failed | Skipped | |---|---|---|---| | Opc.Ua.Core.Tests | 6,357 | 0 | 88 | | Opc.Ua.Server.Tests | 8,483 | 0 | 5 | | Opc.Ua.Sessions.Tests (all secure-channel policies) | 851 | 0 | 379 | | Opc.Ua.Stress.Tests Contract + Integration | 45 | 0 | 0 | | Opc.Ua.Interop.Tests (1.5.378 peer, SignAndEncrypt Basic256Sha256, Aes256_Sha256_RsaPss, ECC_nistP256) | 218 | 0 | 29 (AEAD policies the 1.5.378 peer does not offer) | - **net48:** Opc.Ua.Core.Tests builds, and CryptoUtilsAesCacheTests plus FairRequestQueueTests pass. `dotnet format whitespace` is clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 13, 2026
romanett
added a commit
that referenced
this pull request
Oct 14, 2026
…r locks and allocations per request (#4782) Carved out of #4771 (roadmap item A1 of the o6 benchmark gap analysis): everything except running requests to completion on the receive loop. #4771 keeps run to completion, stacked on this PR, for a later release. Requests still go through the fair request queue and its workers, as on master. What changes is the channel around them: - the response is encoded without the channel gate; - the channel's single writer numbers, secures and sends; - requests take fewer locks and allocations on the channel. No wire, configuration or public API change. ## Results Paired A/B against master `6cfb8cf19` (with #4767). The builds alternate per round on the same machine, and the table shows medians of 5 rounds with the throughput range of the rounds in brackets. The machine was shared with other work, so the CPU and throughput ranges are wide; allocated bytes per request are deterministic. - **o6read** uses the native open62541 client workers of the o6-automation benchmark, in the o6 2.0.1 configuration: `AsyncCustomNodeManager`, Server GC, queue = clients × outstanding + 3. - **"4 cores"** pins the server to 4 cores. - **The other rows** use the harness's .NET client: 20 sessions unless noted, workstation GC, ReferenceServer. - **Contention** counts runtime lock-contention events in the measured window. | Scenario | req/s | Server CPU µs/req | Alloc B/req | Contention | |---|---|---|---|---| | o6read 10×100, 24 cores | 254,600 → 264,700 (+4 %) [228–292k → 255–270k] | 67.4 → 69.5 | 5,874 → 4,709 (**−20 %**) | 6,766 → 1,811 | | o6read 10×1000, 24 cores | 248,900 → 241,300 (−3 %) [231–258k → 233–260k] | 69.0 → 71.3 | 5,866 → 4,718 (**−20 %**) | 6,080 → 1,456 | | o6read 10×100 encrypted, 24 cores | 203,900 → 232,000 (**+14 %**) [111–226k → 204–264k] | 71.6 → 70.2 | 5,985 → 4,752 (**−21 %**) | 5,166 → 1,288 | | o6read 10×100 `--o6-async 0` (`CustomNodeManager2`), 24 cores | 235,600 → 243,900 (+4 %) | 40.1 → 38.8 | 6,702 → 5,886 (−12 %) | 148k → 124k | | o6read 1 client × 1000 (one channel) | 64,300 → 92,600 (**+44 %**) [52–69k → 76–100k] | 79.8 → 74.5 (−7 %) | 6,003 → 5,063 (−16 %) | 224 → 128 | | o6read 10×100, 4 cores | 156,600 → 140,400 (−10 %, noise: rounds 139–192k → 121–184k) | 25.0 → 28.3 | 5,435 → 4,688 (−14 %) | 97 → 184 | | o6read 10×1000, 4 cores | 107,100 → 125,200 (**+17 %**) [103–116k → 120–136k] | 36.5 → 31.7 (**−13 %**) | 9,966 → 4,767 (**−52 %**) | 312 → 160 | | o6read 10×100 encrypted, 4 cores | 103,300 → 126,300 (**+22 %**) [102–105k → 115–143k] | 37.9 → 31.3 (**−17 %**) | 5,498 → 4,736 (−14 %) | 46 → 29 | | o6read 10×100 `--o6-async 0`, 4 cores | 186,600 → 187,700 (+1 %) | 21.1 → 21.0 | 6,587 → 5,915 (−10 %) | 17.7k → 4.2k | | read --nodes 1, 20 sessions | 208,300 → 205,900 (−1 %) | 49.1 → 48.3 | 4,905 → 4,621 (−6 %) | p50/p99 0.072/1.18 → 0.072/1.26 ms | | read --nodes 1, 200 sessions | 216,400 → 208,600 (−4 %) | 42.0 → 43.2 | 4,790 → 4,518 (−6 %) | p50/p99 0.606/3.56 → 0.615/4.05 ms | | read --nodes 100 | 31,950 → 32,470 (+2 %) | 278 → 253 (−9 %) | 73,702 → 73,436 | p50/p99 0.290/2.51 → 0.283/2.48 ms | | browse | 38,380 → 40,460 (+5 %) | 162 → 161 | 39,305 → 38,981 | p50 0.349 → 0.330 ms | | sub (5 × 100 items, 100 ms) | 88,190 → 89,010 notif/s | 11.1 → 10.7 | 159 → 153 | p50/p99 20.7/64.3 → 20.8/66.0 ms | **Where it helps:** - Allocation per request falls in every scenario, by 20 % for o6read. - Lock contention falls by 70–80 % for o6read. - Throughput rises where the channel gate serialised the work: - one deep pipeline on one channel: +44 %; - encrypted: +14 % on 24 cores, +22 % on 4 cores; - 1000 in flight on 4 cores: +17 %. **Where it does not:** CPU per request at 24 cores is unchanged. The requests still go through the fair queue's workers, and the hand-off to a worker and back is what #4771 removes. The 4-core 10×100 row spread as much within each side as between the sides. ## Changes ### Sending - **Seal in the writer.** The producer encodes the response without the channel gate and queues it unsealed. The channel's single writer takes the queued responses in order under a per-channel send-order lock: it assigns sequence numbers, selects the current token, signs and encrypts, and sends the batch. Messages that number themselves (OpenSecureChannel, asymmetric faults, the client's requests) hold the same lock until they are queued. Token changes that can dispose a token take it as well. - **Reconnect.** The old transport is detached before the new token is activated. Unsealed responses still queued for it are then dropped without taking sequence numbers. The new transport is attached and the OpenSecureChannel response numbered under one hold of the send-order lock, so the response is the first message on the new transport. The channel opens again only after that. - **Failure past the sequence numbers.** A message whose security fails after its chunks took sequence numbers stops all numbering on that transport. The writer drops the messages producers numbered for it after the gap, and reports a transport error once it has left the send-order lock. That faults the channel (server) or reconnects (client). - **Numbered writes without the lock.** The writer takes writes that their producers numbered without the send-order lock, because they are queued in number order. A client channel queues only such writes, so its writer no longer waits for the lock that its request threads take. - **Writer on the pool.** The writer always starts on the thread pool. Running it on the producer's thread sent batches of one message. - **Buffers.** The encoded message is a struct in the write-queue entry. A small single-chunk message moves to a right-sized buffer only when writes are already queued, so deep pipelines no longer exhaust the bounded buffer pool (−52 % allocation at 10×1000 on 4 cores). ### Locks and allocations per request - **`ChannelGate`** takes the gate with one compare-and-exchange when it is free, and queues waiters under a lock only when it is not. `SemaphoreSlim` took its monitor on every wait and release, which the receive loop does for every message. The contract is unchanged: not re-entrant, FIFO handoff, cancellable waits, synchronous completion when free. The transport's send lock is a `ChannelGate` too. - **Namespace and server tables.** Every encoder and decoder reads `ServiceMessageContext.NamespaceUris`, `ServerUris` and `MappingVersion`. They are now read lock-free from one immutable snapshot that setters swap atomically. - **Allocation-free tracking.** Tracking a pending request (`PendingRequestScope`) and handing a decoded request out of the gate allocate nothing. - **Single-chunk decode.** A request in one chunk is decoded from that chunk's buffer. - **Fair-queue entry.** An entry coordinates its cancellation registration with an interlocked state (`RegistrationState`) instead of a lock object. ### Client fix `Session.CloseAsync` disposed the session synchronously, and `Dispose` waits for the async dispose of the session and its subscription engine. Closing 1000 sessions at once blocked one pool thread per session while that dispose needed pool threads, and the close took minutes in `ServerManySessionsLoadTestAsync(1000)`. `CloseAsync` now awaits `DisposeAsync`, and dispose awaits an engine that implements `IAsyncDisposable`, as `DefaultSubscriptionEngine` now does. ## Behaviour notes - **Wire order and tokens.** Sequence numbers stay strictly increasing in wire order, and the chunks of one message stay contiguous. Token renewal applies at a message boundary. After a reconnect, the OpenSecureChannel response is the first message on the new transport. - **Back-pressure** (#4707) is unchanged. The receive loop stops reading above 100 queued writes. - **Client.** `Session.CloseAsync` no longer blocks a thread on the dispose. ## Not in this PR The striped server-stop tokens of #4771 are dropped. Since #4767 a server request no longer registers with the stop token, because the deadline monitor polls it. ## Testing - `UaSCBinaryChannelSendOrderTests` covers: - wire order and random interleaving; - token renewal (decrypt and verify) for five policies; - buffers returned once; - a seal failure after numbering stops the channel, and a message numbered after the gap is not sent; - the replaced transport; - back-pressure; - a response that cannot be encoded is answered with a fault. - `TcpReconnectKeyAgreementTests.AResponseQueuedForTheDroppedTransportIsNotSecuredWithTheNewTokenAsync` (new) holds the old transport's sends during a reconnect. It fails on the previous order. The seal-failure test fails without the gap check. - Also run: `ChannelGateTests`, `TcpByteTransportTests`, `ServiceMessageContextMappingTests`. - Local, net10.0: - Opc.Ua.Core.Tests (Stack.Server, Transport, Tcp, ChannelGate): 2739 passed, 0 failed; - Opc.Ua.Configuration.Tests: 241 passed, 0 failed. net48 builds. `dotnet format whitespace --verify-no-changes` passes on the changed files. - CI: the `CI` workflow on this head is running; the commits were tested together with run to completion in #4771 (its last full CI run passed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Profiling the reference server under load showed most request-path CPU spent spinning on contended locks, plus several avoidable allocations per request or per node. This PR removes them, switches the reference server to Server GC, and adds the load harness used to measure it.
Results
Paired A/B against
master(3 alternating rounds, medians, 20 sessions, workstation GC on both sides, 24-core machine):¹ The faster server spends more time in the remaining per-node locks.
² Throughput is bounded by the writer rate in this scenario.
Separately, Server GC instead of workstation GC doubles Read-100 throughput (21k → 44k req/s) and cuts p90 from 2.1 ms to 0.6 ms, because workstation GC paused the server for about half the wall time under load. The working set grows from ~210 MB to ~430 MB.
Changes
ObjectPool<T>: lock-free slot array.ConcurrentBag.Counttook every per-thread lock on eachReturnof the pooled response value-task source, once per request.FairRequestQueue:DefaultServerResourceIsolationProvider:GetAddressBytes()allocations;X509Certificate2.RawDatareturns a fresh copy on every access, and the listener read it for every request.EndpointBase: resolves the requestActivitySourceonce instead ofAssembly.GetCallingAssembly()per request.ServerSystemContextcopy per request (cached on theOperationContext) for per-node callbacks: permission metadata, node metadata, view membership, method lookup;GetReferenceDescriptionAsync;Dictionaryinstead of aConcurrentDictionarywith 24 lock objects;ResourceManager: translation lookups read an immutable snapshot that is rebuilt after writes. Browse serialised on the global lock for every display name.MonitoredItem: shares the defaultDataChangeFilterinstead of allocating one per sampled value.ServerGarbageCollection=true; new "Garbage collector mode" section indocs/ServerScalability.md.perf/:ServerLoadHarness(out-of-process server plus load driver, server-side CPU/GC/alloc/contention deltas, optionaldotnet-tracecapture),TraceAnalyzer(TraceEvent-based CPU/allocation/contention aggregation), and paired A/B scripts. Not part ofUA.slnx.Behaviour notes for custom node managers
IDictionary<NodeId, NodeState>operation cache handed to overrides is now aDictionary. It was aConcurrentDictionary, but only one request flow uses it.ServerSystemContextcopy.Later commits on this PR
NodeMetadata(9068684c3): the role-permission and access-restriction validators run on an internalPermissionMetadatastruct;AsyncCustomNodeManagerfills it on the stack from the same attributes and namespace defaults. Browse's reference filter reads 3 attributes per target instead of 13. Read 100 values: 120.6 → 95.7 KB/request, ~21k → 26k req/s; Browse 67.6 → 51.8 KB, 26.7k → 29.8k req/s.936b28d76,dc638ab9f):SecureChannelContext.ClientChannelCertificate,ServerChannelCertificateandChannelThumbprintareByteString, so the cached per-channel certificate buffer is shared without a copy and cannot be altered by handlers or classifiers. Public API change, see the new section indocs/MigrationGuide.md. The isolation provider's revalidation snapshot still copies.936b28d76):SessionManager.TryGetSessionContexttakes the candidate snapshot without the bindings lock; the validating check under the lock is unchanged (2 instead of 4 global lock acquisitions per request).ObjectPool, the per-request system context cache, the translation snapshot,SecureChannelContextnull semantics and a stoppedFairRequestQueue;perf/follows the repository conventions.Testing
Opc.Ua.Server.Tests(net10.0): 8386 passed before theByteStringchange. After it, 13 secure-session and isolation tests failed; both causes are fixed indc638ab9fand those tests (25 with their variants) pass. A full rerun of the Server, Sessions, Core, Redundancy, LDS and GDS suites on the final commit is in progress.dotnet format whitespaceclean.o6-automation opcua-benchmarks (sampling, deep pipelines)
Replicated both tests of the o6-automation benchmark in the harness (
--scenario counters, andreadwith 10 clients × N outstanding), paired againstorigin/master:BadTcpNotEnoughResources)30b769d13): the sampling loop waited one interval after each sample, adding timer overshoot and sampling time to every cycle; samples now have absolute deadlines.cdac5c5ca):SendResponseblocked thread-pool workers on the channel gate; with many requests in flight the receive loop that was handed the gate waited for a pool thread and the channel stalled. The TCP, Kestrel TCP and WebSocket listeners now useSendResponseAsync.191cd2ba7): server channels stop reading while more than 100 responses are queued and resume at 50, instead of faulting the channel. Client channels keep reading. At 10,000 outstanding requests throughput is ~50k req/s with or without the limit engaged, so the remaining gap to the 100-per-client case is not the backpressure.Not changed
SentMessageQueuedeep-clones everyNotificationMessage. Retained messages return their pooled payloads on acknowledgement while the response copy may still be queued for send, and a client can acknowledge a sequence number before receiving it; the clone keeps a recycled payload from reaching another session's response. Removing it needs a send-completion hook from the transport.ResponseHeaderthat is replaced (~2-3 % of per-request allocation). Fixing it means making structure fields of every generated type lazily initialized, a generator-wide change to default values, equality and encoding.🤖 Generated with Claude Code