Skip to content
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
c4ba2df
Fix incomplete-message buffering and channel cleanup
marcschier Sep 23, 2026
5c764da
Merge branch 'master' into fix/incomplete-message-buffering
marcschier Sep 23, 2026
7f4b6c1
Merge branch 'master' into fix/incomplete-message-buffering
marcschier Sep 23, 2026
cab97bd
Combine targeted reassembly budgets with channel cleanup fixes
marcschier Sep 24, 2026
bd83155
Address review feedback on chunk reassembly and transport limits
marcschier Sep 24, 2026
cca7deb
Address PR review on abort handling and shared assembly deadlines
marcschier Sep 24, 2026
6319247
Complete session-binding and transport-admission foundations
marcschier Sep 24, 2026
d889b77
Start staged resource-isolation capacity planning
marcschier Sep 24, 2026
c4a058c
Give expiry regression sessions valid diagnostic identifiers
marcschier Sep 24, 2026
29f7070
Merge branch 'fix/incomplete-message-buffering' into feat/server-reso…
marcschier Sep 24, 2026
55088ab
Merge remote-tracking branch 'origin/master' into feat/server-resourc…
marcschier Sep 24, 2026
71e3227
Complete configurable runtime resource isolation
marcschier Sep 24, 2026
b34a343
Address isolation review feedback and CI regressions
marcschier Sep 25, 2026
979ee0d
Preserve session reassembly headroom and cache validated classification
marcschier Sep 25, 2026
2db8792
Merge remote-tracking branch 'origin/master' into feat/server-resourc…
marcschier Sep 25, 2026
699409a
Document continuity headroom and bounded OpenAPI request ownership
marcschier Sep 25, 2026
ecc0c1c
Keep byte continuity separate from reconnect identity reservations
marcschier Sep 26, 2026
884e006
Avoid isolation event ID collision after master merge
marcschier Sep 26, 2026
80dbf0b
Address isolation documentation and startup diagnostics feedback
marcschier Sep 26, 2026
600004a
Fix resource isolation review findings
Copilot Sep 26, 2026
52cbba9
Implement custom request parking and address isolation review feedback
marcschier Sep 27, 2026
e1a994a
Handle collection count changes during ArrayOf conversion
marcschier Sep 27, 2026
9e25e4e
Honor resource provider contracts independently of request ordering
marcschier Sep 27, 2026
05753ce
Synchronize recovery audit fake time with processed keepalives
marcschier Sep 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions docs/DependencyInjection.md
Original file line number Diff line number Diff line change
Expand Up @@ -797,6 +797,7 @@ properties (bindable from `IConfiguration` or set via the
| `ConfigureLoadedConfiguration` | Code-only callback | Override individual settings of the configuration loaded from `ConfigurationFile` / `ConfigurationStream`. |
| `ConfigureBuilder` | Code-only callback | Pre-security server-policy and server-option escape hatch, including max failed authentication attempts, sessions, channels, auditing, and HTTPS mutual TLS. |
| `ConfigureRateLimits` | Code-only callback | Tunes the default connection and session-establishment admission controls. |
| `ResourceIsolation` | `ConfigureResourceIsolation(...)` | Limits concurrent connections, retained data, and queued/running requests by caller; defaults to Balanced. Applied even when loading XML configuration. |

Configure incomplete-message capacity with
`builder.AddServer(...).WithChunkReassemblyBudget(maxBytes)`. This registers one
Expand All @@ -805,6 +806,42 @@ explicit budget, the server sizes one from `MaxMessageSize`; see
[incomplete messages](RateLimiting.md#incomplete-messages) for the defaults,
sessionless headroom, and direct-construction equivalent.

### Server resource isolation

`OpcUaServerOptions.ResourceIsolation` controls whether the running server
accepts a connection, retains message data, or queues/executes a request when
resources are limited. These capacity checks are additional to connection
rate limits and service authentication. Configure them with
`ConfigureResourceIsolation(...)` or the `OpcUa:Server:ResourceIsolation`
configuration section; see [server resource isolation](ResourceIsolation.md)
for direct, DI, and JSON examples.

**Balanced is the default.** Startup validates that finite configured totals
can hold its shared and reserved capacity; it neither removes reserves nor
raises totals to make an invalid configuration fit. This also applies when
loading an existing XML configuration. See the
[migration exception and fixes](MigrationGuide.md#transport-resource-limits).
SharedOnly is a supported shared-limit policy: without an explicitly supplied
provider, it skips the default isolation provider and its plan validation,
preserving existing rate, message, and shared reassembly limits, including
existing unlimited settings.

`AddResourceIsolationClassifier<T>()` registers a host-written
Comment thread
marcschier marked this conversation as resolved.
Outdated
`IResourceIsolationClassifier` for protected ingress or verified owner mapping.
Reserves alone do not give unknown connections protected access, and an
observed IP address is not proof of identity. TrustedReservations requires
both a classifier and provisioned `TrustedOwners`; see the
[dedicated-ingress example](ResourceIsolation.md#example-dedicated-trusted-ingress)
and its deployment requirements.

Direct servers expose `ResourceIsolationOptions`,
`ResourceIsolationClassifier`, and `ResourceIsolationProvider`. An explicitly
registered `IServerResourceIsolationProvider` takes precedence over the options,
including SharedOnly. The server does not validate it by constructing a default
plan or take ownership of its disposal; its creator or DI container owns it.
The provider author is responsible for matching the actual listener and queue
capacities. The server owns and disposes only the default provider it creates.

### Committed session bindings

Managed servers automatically supply their session manager's committed-binding
Expand Down
59 changes: 56 additions & 3 deletions docs/MigrationGuide.md
Original file line number Diff line number Diff line change
Expand Up @@ -827,13 +827,66 @@ implementation detail, and can no longer do so.

## Transport resource limits

Direct and hosted `StandardServer` instances use **Balanced resource isolation**
Comment thread
marcschier marked this conversation as resolved.
Outdated
by default. It limits resource use by caller, schedules decoded requests in
turns between callers, and keeps startup/reconnect capacity inside existing
totals. Unknown connections do not automatically qualify for those reserves;
protected access before authentication requires a deployment-enforced ingress
path and an `IResourceIsolationClassifier`.

**Configuration compatibility exception:** an existing configuration, including
a loaded XML file or test fixture, can now fail default-provider startup if it
has zero/unlimited resource bounds or insufficient room for the reserves and
shared capacity. Startup reports an explicit configuration error. It does not
silently remove Balanced's reserves or increase configured totals.

Choose the correction that matches the deployment rather than copying one
universal capacity setting:

| Configuration problem | Action |
| --- | --- |
| Zero/unlimited resource bounds are intentional | Select `ServerResourceIsolationMode.SharedOnly`. This is a current supported shared-limit mode, not deprecated functionality. |
| Finite bounds are intended | Set positive `ServerConfiguration.MaxSessionCount` / `MaxChannelCount` and `TransportQuotas.MaxMessageSize` / `MaxBufferSize`. Correct invalid test fixtures using the same rules. |
| Insufficient connection space | Ensure shared Connection capacity **after reservations** is at least `MaxSessionCount + 1`. For 100 Sessions, Balanced's default two connection reserves require at least 103 channels, not 101. Adjust `MaxChannelCount`, the intended `MaxSessionCount`, or explicit Connection-stage reserves. |
| Insufficient retained-message space | Adjust `WithChunkReassemblyBudget(maxBytes)` / `ServerBase.ChunkReassemblyBudget`, or reduce the allowed `MaxMessageSize` if appropriate. For a custom pool, set `MaxRetainedMessageBytes` to its proven upper bound, not a smaller value merely to pass validation. |
| Other stage or rate totals cannot hold the selected reserves | Review `Stages[].Capacity`, `BootstrapReserved`, `ReconnectReserved`, `ControlReserved`, `OwnerHardLimit`, and trusted-owner `Reservations`. Tune `ServerRateLimitOptions.MaxConcurrentSessionEstablishment`, `ConnectionsPerSecond`, or `ConnectionBurst` as applicable. A stage capacity override cannot enlarge its underlying total. |

Setting a stage reserve to zero explicitly removes that stage's protection.
If reserves are not wanted, FairShare provides caller-aware scheduling without
reserves, but still needs valid finite bounds. SharedOnly preserves existing
shared rate/message/reassembly behavior and unlimited settings because the
server installs **no default isolation provider** and skips its plan validation:

```csharp
// Direct construction, before startup:
server.ResourceIsolationOptions.Mode = ServerResourceIsolationMode.SharedOnly;

// Alternatively, configure a hosted server:
services.AddOpcUa()
.AddServer(options =>
options.ResourceIsolation.Mode = ServerResourceIsolationMode.SharedOnly);
```

An explicitly supplied `IServerResourceIsolationProvider` still takes precedence,
even in SharedOnly; the host owns its validation and lifetime. For hosted servers
loading XML, modify capacities in the XML or through
`OpcUaServerOptions.ConfigureLoadedConfiguration`, not `ConfigureBuilder`.
See [startup validation and sizing](ResourceIsolation.md#startup-validation-and-sizing)
and the [programmatic plan example](ResourceIsolation.md#inspect-the-plan-in-application-code)
to inspect the actual limits. The N+1 check concerns aggregate capacity; it does
not entitle each owner to the whole advertised Session maximum.

Applications migrating from 1.5.x have a server-wide budget for retained
intermediate-message buffers. With the reference server's 4 MiB maximum message
size, the default budget is **64 MiB**, and channels without an activated session
may fill only the lower **32 MiB**. A chunk that does not fit discards its partial
size, the default budget is **64 MiB**. Under SharedOnly, channels without an
activated Session may retain chunks only while total usage remains within the
lower **32 MiB**. Under the default isolation provider, class-based byte
Comment thread
marcschier marked this conversation as resolved.
Outdated
reservations replace that occupancy rule without increasing the 64 MiB total.
With a 65,536-byte buffer limit, Balanced reserves 16.25 MiB each for startup and
reconnect, leaving 31.5 MiB shared. A chunk that does not fit discards its partial
message and closes the channel with `BadTcpNotEnoughResources`. Final chunks,
single-chunk requests, response buffers, and client buffers are not charged to
this reassembly budget.
this reassembly budget; decoded-request limits still apply.
Comment thread
marcschier marked this conversation as resolved.
Outdated

For workloads with many simultaneous large requests, set
`WithChunkReassemblyBudget(maxBytes)` on the Dependency Injection (DI) server builder or assign
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Here is a list of available documentation for different topics:
* [Diagnostics](Diagnostics.md) — logging, telemetry, server audit events, server diagnostics nodes, and packet capture.
* [Performance Benchmarks](Benchmarks.md) — BenchmarkDotNet methodology, the 2.0 (`master`) vs 1.5.378 (`master378`) comparison, root-cause analysis of the encoder/decoder/session regressions and their real-world impact, the subscription-notification (pooled encodeable) micro-benchmarks, server session scalability (the 500-session capability, sizing and bottlenecks), and planned future work.
* [Server Session Scalability](ServerScalability.md) — why a single node tops out at ~2000 concurrent sessions, the establishment vs steady-state boundaries (socket backlog, the `BadTcpInternalError` retry-storm amplifier, the O(N²) diagnostics rescan, CreateSession crypto-under-lock, the RSA CPU wall, and the held-Publish worker-accounting coupling) with code references, and a prioritized admission-control / rate-limiting roadmap for moving beyond it.
* [Server resource isolation](ResourceIsolation.md) - Configurable shared/fair/balanced/trusted admission, protected resource floors, bounded caller accounting, and fair decoded-request scheduling.
* [Rate Limiting and Admission Control](RateLimiting.md) — the server's deterministic, configurable connection- and session-establishment limiters (on by default, `System.Threading.RateLimiting`-based, DI-pluggable), shared incomplete-message capacity and sessionless headroom, the `BadServerTooBusy` signalling, the diagnostics-independent server retry-after carriers (`ResponseHeader.additionalHeader`, HTTP `Retry-After`, UA-TCP ERR, and load-based `Server.ServiceLevel`), and the client's server-signal-aware adaptive reconnect backoff (`IReconnectPolicy.TryGetNextDelay`).
* Support for [WellKnownRoles & RoleBasedUserManagement](RoleBasedUserManagement.md).
* Pluggable [Identity Providers](IdentityProviders.md) — interfaces (`IClientIdentityProvider`, `IUserTokenAuthenticator`, `IAccessTokenProvider`, `ITokenIssuer`, `IIdentityClaims`) plus the OPC 10000-6 §6.5.2.2 `IssuerEndpointUrl` JSON parser for OAuth2 / OIDC / Entra / JWT flows.
Expand Down
76 changes: 64 additions & 12 deletions docs/RateLimiting.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,17 @@
# Rate limiting and admission control

The server applies deterministic, configurable admission control so it sheds a connection or session-establishment storm with a fast, standard "busy" signal instead of collapsing (see [Server Session Scalability](ServerScalability.md) for the underlying analysis). Clients react to that signal by backing off adaptively rather than retrying blindly. The primitives are built on `System.Threading.RateLimiting`, so the algorithm is pluggable and configurable through dependency injection.

Rate limiting is **on by default with conservative limits** sized so normal and bulk-but-well-behaved load is unaffected; only a storm is shed. Tune or disable it per deployment, or replace the whole limiter provider via DI.
Admission control means checking capacity before accepting more work. The server
applies configurable limits to reject excess connections or Session-establishment
requests early, reducing the work caused by a connection storm (see
[Server Session Scalability](ServerScalability.md) for the underlying analysis).
Clients respond to busy signals by backing off rather than retrying blindly.
The standard rate-limiting primitives use `System.Threading.RateLimiting`;
the provider can be replaced through dependency injection.

Rate limiting is **on by default with conservative limits**. Tune them for the
deployment: even legitimate clients can be refused during a sufficiently large
burst. Disabling rate limiting does not disable
[resource isolation](ResourceIsolation.md) or the incomplete-message budget.

## Server side

Expand All @@ -20,14 +29,15 @@ Rate limiting is **on by default with conservative limits** sized so normal and
| Hard session cap reached (`MaxSessionCount`) | `BadTooManySessions` |
| Connection shed at the listener | the connection is dropped; the client sees a transport error and, together with any subsequent `BadServerTooBusy`, backs off |
| Incomplete-message budget exhausted | `BadTcpNotEnoughResources` followed by channel closure; if the error cannot be sent, the client observes closure |
| Resource isolation refuses common HTTPS body processing | HTTP 503 (Service Unavailable), distinct from a rate limiter's HTTP 429 |

### Configuration

The deterministic limits live in `ServerRateLimitOptions`:

| Option | Default | Meaning |
| --- | --- | --- |
| `Enabled` | `true` | Master switch for all rate limiting. |
| `Enabled` | `true` | Switch for this provider's connection and Session-establishment rate limits, not resource isolation or message budgets. |
| `ListenBacklog` | 512 | Listener socket pending-connection backlog. |
| `ConnectionRateLimitEnabled` | `true` | Whether inbound connections are rate limited. |
| `ConnectionsPerSecond` | 500 | Sustained connection admission rate. |
Expand Down Expand Up @@ -69,8 +79,12 @@ server.RateLimitOptions = new ServerRateLimitOptions { ConnectionsPerSecond = 20
Raw TCP, Kestrel TCP, and UACP WebSocket listeners consume the
`IConnectionRateLimiter` supplied through `TransportListenerSettings`, injected
by `StandardServer.ConfigureTransportListenerSettings`. A custom server can
override that hook to supply its own limiter. The default
`TokenBucketConnectionRateLimiter` wraps a `System.Threading.RateLimiting.TokenBucketRateLimiter`.
override that hook to supply its own limiter. The ordinary shared
`TokenBucketConnectionRateLimiter` wraps a
`System.Threading.RateLimiting.TokenBucketRateLimiter`. With the default
isolation provider in Balanced or TrustedReservations, the default rate provider
instead uses `ResourceIsolationConnectionRateLimiter` to divide the same totals
between protected and shared traffic, as described below.
Kestrel and UACP WebSocket listeners also enforce their configured channel cap
across concurrent admissions and release capacity with the physical connection,
including reverse handoff. The raw listener retains its existing admission and
Expand All @@ -89,16 +103,18 @@ responses. A request that fits in one chunk can still be served while the
reassembly budget is full. A chunk that exceeds available capacity is rejected
without waiting; the partial message is discarded and its channel is closed.

The default is sixteen times `TransportQuotas.MaxMessageSize`, clamped to
The budget's default is sixteen times `TransportQuotas.MaxMessageSize`, clamped to
64 MiB through 1 GiB, then raised if necessary to accommodate four maximum-sized
messages. A configuration with unlimited message size receives a 1 GiB budget.
Both the stack's 2 MiB and the reference server's 4 MiB message limits select
**64 MiB**. `ChunkReassemblyBudget.GetDefaultMaxBytes(maxMessageSize)` returns
this value, and `ChunkReassemblyBudget.CreateDefault(configuration)` creates a
budget from an `EndpointConfiguration` using the same sizing policy.
budget from an `EndpointConfiguration` using the same sizing policy. The budget
can be used with unlimited message sizes in SharedOnly; the default Balanced
isolation provider separately requires finite message and buffer limits.

Channels without an activated session may reserve only while total usage stays
at or below `MaxBytesWithoutSession`, which defaults to half of `MaxBytes`.
In **SharedOnly**, channels without an activated session may reserve only while
total usage stays at or below `MaxBytesWithoutSession`, which defaults to half of `MaxBytes`.
Channels with activated sessions can use the remaining headroom. This is a
capacity policy, not an authentication boundary: an activated anonymous session
also qualifies. The two-argument constructor sets the sessionless threshold
Expand All @@ -109,8 +125,44 @@ The threshold is not reserved space for new clients. For example, with a
sessions prevents further sessionless intermediate chunks. Single-chunk
requests are still processed, but a large multi-chunk OpenSecureChannel,
CreateSession, or ActivateSession may be refused until occupancy drops.
Per-caller fairness and guaranteed bootstrap reservations require a separate
isolation policy; the shared budget alone does not provide them.
The shared budget alone provides neither request scheduling by caller nor
reserved startup capacity.

[Server resource isolation](ResourceIsolation.md) limits usage by caller and
can keep capacity reserved for verified startup/recovery work inside the same
totals. Direct and hosted `StandardServer` instances use **Balanced by default**.
With the default isolation provider active, its byte-class reservations replace
the sessionless occupancy rule above; the shared budget still enforces
`MaxBytes`. With a 4 MiB message limit and 65,536-byte buffer limit, Balanced
reserves 16.25 MiB each for startup and reconnect from a 64 MiB budget, leaving
31.5 MiB shared. These are conservative retained-array bounds, not payload sizes.

**A reserve does not make an unknown caller eligible to use it.** A deployment
needs trusted ingress classification for protected access before OPC UA
Comment thread
marcschier marked this conversation as resolved.
authentication; an IP address or previous successful connection is not proof
of identity. Unknown callers can be refused while reserved capacity is idle.
Comment thread
marcschier marked this conversation as resolved.
Outdated
SharedOnly is a supported mode retaining the shared occupancy rule, not a
Comment thread
marcschier marked this conversation as resolved.
Outdated
deprecated mode. It installs no default isolation provider, so existing
zero/unlimited settings do not undergo that provider's finite-plan validation.
Neither policy guarantees progress for all unknown callers or caps the whole
Comment thread
marcschier marked this conversation as resolved.
Outdated
process heap.

Balanced startup rejects insufficient capacities rather than silently removing
reserves or increasing totals. See the
[migration choices](MigrationGuide.md#transport-resource-limits) and
[plan inspection example](ResourceIsolation.md#inspect-the-plan-in-application-code)
for the actual fields and checks. Isolation remains independent of
`ServerRateLimitOptions.Enabled`.

When the default connection rate limiter is enabled alongside Balanced or
TrustedReservations, it also reserves one burst token and one token per second
for startup, reconnect, and each provisioned trusted owner, inside the existing
`ConnectionBurst` and `ConnectionsPerSecond` totals. At least one shared token
must remain in each total; startup rejects totals that cannot fit. Protected
traffic can also use available shared tokens, but ordinary traffic cannot use
the reserved tokens. Closing a connection does not refund a token. FairShare
and SharedOnly use the ordinary shared token bucket; an explicitly supplied
rate-limiter provider is not replaced or wrapped.
Comment thread
marcschier marked this conversation as resolved.
Outdated

Configure the hosted server through its fluent builder:

Expand Down
Loading
Loading