Skip to content

Raise ChannelLifetime for the CTT reference server and record the full CTT run of the merged CTT PRs - #4502

Merged
marcschier merged 14 commits into
masterfrom
romanett/ctt-test-run-regression-7b954e
Sep 24, 2026
Merged

marcschier merged 14 commits into
masterfrom
romanett/ctt-test-run-regression-7b954e

Conversation

@romanett

@romanett romanett commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Findings of a full CTT run (23 parts, 3,752 test cases, CTT 1.05.06 / scripts 1.05.513) against the merge of #4477, #4482, #4485 and #4486, which are all on master now, plus the CTT defects filed in Mantis.

No server code changes: the PR changes only the CTT configuration of the reference server sample and the plans/ documentation.

CTT configuration: raise ChannelLifetime to 120 s

CTT Security None 007.js and Security Basic256Sha256 005.js failed (also on master): they keep SecureChannels without a Session idle for about 51 s (60 s token, stretched by the CTT's CloseSession delay, C50) and expect the newest one to close with Good. TcpTransportListener.DetectInactiveChannels had already closed it after ChannelLifetime (30 s) without traffic (CloseSecureChannel().Result received BadInvalidState).

samples/Reference/ConsoleReferenceServer/Ctt.ReferenceServer.Config.xml now sets ChannelLifetime to 120000. The default of 30000 and the server's inactivity cleanup are unchanged.

An earlier revision of this PR changed the inactivity cleanup instead, keeping idle Session-less channels open until their token expired (Part 4 §5.6.2.1). That change was withdrawn because it lets an unauthenticated client hold a channel for up to the SecurityTokenLifetime (one hour by default) instead of 30 s. The difference to Part 4 §5.6.2.1 is recorded as an open server finding in plans/ctt-issues.md.

Validated on 2026-09-24 with the reverted code and the new configuration (full run, compared with the run of 2026-09-22 at test-case level):

  • Security None 007.js passes and Security Basic256Sha256 005.js has only the C50 CloseSession delay warnings (both failed with 30 s).
  • All other parts have the same error signatures as before; the remaining differences are "Response Timestamp shows a delay" warnings from a loaded machine (a back-to-back Session Services run with 30 s and 120 s on an idle machine is identical).
  • A & C Refresh2 Err_003.js/Err_004.js and Shelving initialize.js report BadSubscriptionIdInvalid in today's runs with both 30 s and 120 s on the same build, so they are unrelated to the configuration (both passed again in a run on the idle machine with 120 s).

Mantis

32 CTT defects filed on 2026-09-24 (11420–11454, script issues in CTT UA Scripts, native aggregate oracle issues in CTT UA Binary); 12 drafts were already covered by existing reports (including GDS issues 11406–11413 by node-opcua and 11102 for the opc.https/opc.wss endpoint selection); 5 are held back for further investigation. plans/ctt-issues.md links every entry.

CTT documentation (plans/ctt-issues.md, plans/ctt-testing.md)

  • ctt-issues.md lists every failing CTT test case with a short abstract and its Mantis issue; the details live in Mantis. It keeps the defects not filed yet, the open questions (U1–U4), the open server findings and the CTT project configuration notes, and no longer lists findings the server has fixed.
  • Notable CTT findings of this run: the aggregate oracle lacks the Part 13 Uncertain status and Calculated bit rules that Fix server/core validation, lifecycle and protocol regressions #4477 implements (C48/C49, 24 test cases); the CloseSession "delay in excess of 600ms" warnings are a CTT client artifact (C50: the server answers in 2–5 ms, the CTT stops the Session's SessionThread between stamping and sending); the CTT's FindTokenType rewrites its cached endpoints, so Security User Name Password 2 015.js reports duplicate PolicyIds (C36); A & C Comment passes or skips depending on the date (C39).
  • Open server finding: AddNodes after a child rename returns BadNodeIdExists (Fix server/core validation, lifecycle and protocol regressions #4477 × Fix AddNodes scaling and OpenSecureChannel revocation errors found by CTT Monitored Item, Node Management and security runs #4486 interaction, not covered by the CTT).
  • ctt-testing.md is a runbook for agents: setup, headless runs, selection generation, reading and comparing results, triage, recipes per conformance group (A & C, Session/Subscription, Monitored Items, security groups with an isolated PKI) and a full partitioned run of every group (about 80 minutes). It points to ctt-issues.md for the failures to expect.

Related Issues

Checklist

  • I have signed the CLA and read the CONTRIBUTING doc.
  • I have added tests that prove my fix is effective or that my feature works and increased code coverage.
  • I have added all necessary documentation.
  • I have verified that my changes do not introduce (new) build or analyzer warnings.
  • I ran all tests locally using the UA.slnx solution against at least .net framework and .net 10, and all passed.
  • I fixed all failing and flaky tests in the CI pipelines and all CodeQL warnings.
  • I have addressed all PR feedback received.

Validated locally with a full partitioned CTT run of the Release net10.0 reference server (reverted code, ChannelLifetime 120000).

🤖 Generated with Claude Code

@marcschier
marcschier marked this pull request as ready for review September 17, 2026 07:49
Copilot AI lite review requested due to automatic review settings September 17, 2026 07:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Cleanup does not account for a still-valid PreviousToken, which can prematurely close reconnecting channels.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates TCP inactivity cleanup to retain idle SecureChannels while valid tokens remain, adds regression coverage, and documents full CTT results.

Changes:

  • Added token-aware inactivity cleanup.
  • Added deterministic regression tests.
  • Documented CTT execution results and findings.
File summaries
File Description
TcpListenerChannel.cs Determines cleanup eligibility from channel and token state.
TcpTransportListener.cs Uses the new cleanup eligibility check.
TcpInactivityCleanupRegressionTests.cs Tests idle, stale, and token-valid channels.
plans/ctt-testing.md Records full CTT run procedures and results.
plans/ctt-issues.md Documents CTT findings and resolved server behavior.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/Opc.Ua.Core/Stack/Tcp/TcpListenerChannel.cs Outdated
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Code coverage

✅ Coverage gate passed.

Check Result Threshold
✅ Project line rate 87.47% (270642/309394 lines) >= 70.00%
✅ Project branch rate 77.78% >= 60.00%
ℹ️ Patch coverage no changed C# files -
ℹ️ Baseline delta (advisory) +13.87 pp 73.60% recorded

Coverage is above the recorded baseline - consider ratcheting coverage-thresholds.json.

Thresholds live in coverage-thresholds.json. Whole report before exclusions: line 86.57%, branch 76.93%.

@codecov

codecov Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.60%. Comparing base (23be306) to head (7c2ed6a).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4502      +/-   ##
==========================================
- Coverage   81.61%   81.60%   -0.01%     
==========================================
  Files        2069     2069              
  Lines      309394   309394              
  Branches    53699    53699              
==========================================
- Hits       252514   252495      -19     
- Misses      38576    38597      +21     
+ Partials    18304    18302       -2     
Flag Coverage Δ
actions 81.60% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 28 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mrsuciu

mrsuciu commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

"Behavior change: an idle open channel now stays until its token expires (at most SecurityTokenLifetime, one hour by default) instead of 30 s." This a bit concerning if it could be "exploited" by a rogue client,

Comment thread src/Opc.Ua.Core/Stack/Tcp/TcpListenerChannel.cs Outdated
@romanett
romanett marked this pull request as draft September 17, 2026 17:04
@romanett
romanett force-pushed the romanett/ctt-test-run-regression-7b954e branch from 36e55be to e026c5a Compare September 22, 2026 07:39
romanett and others added 9 commits September 22, 2026 09:39
Full CTT run (23 parts, 3,752 test cases) against origin/master + #4477,
- C48/C49: aggregate oracle lacks the Part 13 Uncertain status and
  Calculated bit rules that #4477 implements for Min/Max(ActualTime)
- C14 withdrawn: Auditing Connections pass since the event queue size
  fix (#4480)
- AddNodes BadNodeIdExists after a child rename (#4477 x #4486)
- Security None 007.js / Basic256Sha256 005.js fail on master too
  (CloseSession latency outlasts the idle channel lifetime)
- C10 also covers A & C Confirm Test_001.js depending on alarm phase
- coverage notes for UAFX, PubSub Publisher UADP and UserDefinedCU
- ctt-testing.md section 10: how to run every group in parts

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured with a request-arrival probe build of the reference server: the
server receives CloseSession ~520 ms after the CTT timestamps it and
completes it in 2-5 ms; the CTT stops the Session's SessionThread (~550 ms)
between stamping and sending (C50). Security None 007.js / Basic256Sha256
005.js fail because TcpTransportListener.DetectInactiveChannels closes an
idle SecureChannel after ChannelLifetime (30 s) while its SecurityToken is
valid for 60 s (Part 4 5.6.2.1); with ChannelLifetime 120000 the test passes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The server closes SecureChannels without traffic after ChannelLifetime.
CTT Security None 007.js and Security Basic256Sha256 005.js keep channels
without a Session idle for about 50 s and expect the newest one to still
be open; with 120000 both pass (were BadInvalidState on close).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TcpTransportListener.DetectInactiveChannels closed every channel that had
no traffic for ChannelLifetime (30 s by default), also open channels whose
security token was still valid. OPC 10000-4 5.6.2.1 keeps a SecureChannel
until it is closed or its last token expired; unused Session-less channels
are only to be evicted before reaching the channel limit. CTT Security
None 007.js and Security Basic256Sha256 005.js failed on the close of such
a channel (60 s token, about 50 s idle).

The inactivity cleanup now skips open channels whose current or renewed
token has not expired. Channels that never opened, faulted channels and
channels without a token keep the ChannelLifetime timeout, and the
oldest-unused eviction at MaxChannelCount is unchanged. An idle open
channel now stays until its token expires.

Reverts the ChannelLifetime workaround in Ctt.ReferenceServer.Config.xml.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reference server sends identical UserTokenPolicies (PolicyId 2, empty
SecurityPolicyUri) on all Basic256Sha256 endpoints; only the CTT's
UaEndpointDescription.FindTokenType rewrites the cached SignAndEncrypt
copies before Security User Name Password 2 015.js compares them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Opc.Ua.Sessions.Tests ClientTest.ConnectCloseSessionCloseChannelAsync
failed on CI (ubuntu, macOS): a client that detaches from a live Session
and goes silent expects the server to close the channel after the
ChannelLifetime inactivity timeout, although its token is still valid.

Only open channels without a Session are now kept while their token is
valid (OPC 10000-4 5.6.2.1 restricts closing unused Session-less
SecureChannels); a silent channel carrying a Session ends after the
inactivity timeout as before. CTT Security None 007.js and Security
Basic256Sha256 005.js still pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ReadSymmetricMessage accepts messages secured with PreviousToken until it
expires, which can be later than the current token when the lifetimes
differ. Keep a Session-less open channel while any of its current,
renewed or previous tokens is valid, and cover the lifetime mismatch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rerun of all 23 parts against origin/master plus the inactivity-cleanup
fix. Everything matches the run of 2026-09-15 except the Aggregates group,
which grew from 4,498 to 4,562 error messages: #4503 computes DeltaBounds
for Uncertain bounds, and a logged run shows server and oracle disagreeing
in both directions about which bounds are Bad and which Uncertain (U4).
Security None 007.js and Basic256Sha256 005.js pass with the default 30 s
ChannelLifetime, and Monitor Value Change V2 020.js passes again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Link the 32 issues filed on 2026-09-24 (11420-11454), the existing
reports that already cover 12 of the drafts, and the drafts held back
for further investigation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@romanett romanett changed the title Keep idle SecureChannels open while their token is valid and record the full CTT run of the merged CTT PRs Raise ChannelLifetime for the CTT reference server and record the full CTT run of the merged CTT PRs Sep 24, 2026
romanett and others added 2 commits September 24, 2026 19:00
Keeping idle Session-less SecureChannels open until their security token
expires lets an unauthenticated client hold a channel for up to the
SecurityTokenLifetime (one hour by default) instead of ChannelLifetime.
The CTT reference server configuration raises ChannelLifetime instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CTT Security None 007.js and Security Basic256Sha256 005.js keep
SecureChannels without a Session idle for about 51 s and expect the
newest one to still be open. With ChannelLifetime 120000 both pass
again; the default of 30000 and the server code are unchanged.

Record the validation run of 2026-09-24, the date dependency of C39 and
the %TEMP% cleanup that removed the isolated CTT PKI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@romanett
romanett requested a review from marcschier September 24, 2026 17:05
@romanett
romanett marked this pull request as ready for review September 24, 2026 17:05
@romanett
romanett requested a review from mrsuciu September 24, 2026 17:06
romanett and others added 2 commits September 24, 2026 19:27
List every failing CTT test case with a short abstract and the Mantis
link; the details live in Mantis. Keep the defects that are not filed
yet, the open questions, the open server findings and the project
configuration notes, and drop the findings the server has fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe setup, running, selections, result analysis, run comparison,
triage and the per-group recipes as current guidance, without references
to past runs, fixed issues or individual ctt-issues.md entries. Point to
ctt-issues.md for the known CTT defects to expect. The selection
generator now reads every profile set in a fixed order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@marcschier
marcschier enabled auto-merge (squash) September 24, 2026 18:13
@marcschier
marcschier merged commit 13f96fd into master Sep 24, 2026
167 checks passed
marcschier pushed a commit that referenced this pull request Sep 25, 2026
… and CTT configuration (#4545)

# Description

Resolves the open items of `plans/ctt-issues.md` (*Needs clarification*,
*Open server findings*, *CTT project configuration notes*). Each item
was checked against the OPC 10000 spec text. A server change was made
only where the server was wrong, and every server change has a
regression test. CTT script and oracle defects are filed in Mantis
instead.

### Server changes (`Opc.Ua.Server`)

- **NumberOfTransitions honours TreatUncertainAsBad.**
- Part 13 §5.4.3.24 does not count Bad values. §4.2.1.2 makes Uncertain
values "equivalent to Bad" when TreatUncertainAsBad is True, and the
aggregate definition doesn't override that.
- Uncertain values are therefore no longer counted, and no longer used
as the previous value, when the setting is True. The now unused
`TimeSlice.NonBadEarlyBound` is removed.
- **DurationInStateZero/NonZero status follows the variable's
interpolation.**
- §5.4.3.2.2: "A region is Uncertain if a region ends in a Bad or
Uncertain value and SlopedInterpolation is used."
- The calculator always built stepped status regions. So on a sloped
variable, the last region stayed Good while TimeAverage2 and Total2 over
the same interval were already Uncertain.
  - The duration value doesn't change.
- **AnnotationCount returns `Bad_NoData` and sets Partial.**
- Table 68 requires `Bad_NoData` for intervals before the start or after
the end of data. §5.3.3.2 requires the Partial bit for incomplete
intervals.
- `HistorianDispatcher` now derives the start and end of data from the
raw history (read with bounds) plus the annotation timestamps. It passes
them to a new `CalculateAnnotationCounts` overload; the old overload is
unchanged.
- Three existing dispatcher tests change their expectations. For
example, a node without any data now returns `Bad_NoData` instead of
Good zeros.

### Reference server sample and CTT configuration

- **`de-DE` ApplicationName.** The server advertises `de-DE` in
LocaleIdArray, but FindServers and GetEndpoints always returned the
en-US name. The sample now keys the ApplicationName and registers en-US
and de-DE translations (Part 4 §5.5.2.2 / §5.5.4.2 localeIds).
- **Subscription limit.** `Max Supported Subscriptions` in
`samples/UAReferenceServer.ctt.xml` and `MaxSubscriptionCount` in
`Ctt.ReferenceServer.Config.xml` go from 100 to 200. Subscription
Publish Min 05 `003.js` needs 190 subscriptions.
- **Alarm Cycle Time.** The CTT template's Alarm Cycle Time is 30 s. At
60 s it equalled `RequestedSessionTimeout`, so a Limit/Level CU that
starts a run finds its session expired (C40).

### Tests only

- **AddNodes duplicate BrowseName test:**
`AddNodeAsync_DuplicateBrowseNameUnderParent_ReturnsBadBrowseNameDuplicatedAsync`
never ran. Its items had no TypeDefinition, so its `Assume` always made
it inconclusive. The finding it belongs to (AddNodes after a rename
returning `BadNodeIdExists`) was already fixed by #4477.
- **RequestHandle echo:** a new test covers the ServiceFault for an
oversized argument in a chunked request (already fixed by #4488).
- **Value-based status:** a new test covers the aggregate status with
TreatUncertainAsBad = false (already fixed by #4503).

### Not changed

- **GDS QueryApplications/QueryServers still return records with
ServerCapability `NA`.** OPC 10000-12 §6.5.10/§6.5.11 says they "shall
not", but the CTT GDS scripts expect those records. Implementing the
exclusion turned 40 GDS test cases from pass to error, so it waits for
the script fix ([Mantis
11458](https://mantis.opcfoundation.org/view.php?id=11458)).
- **Idle SecureChannels are still closed after `ChannelLifetime`.**
Keeping them open would be security relevant. The CTT configuration's
`ChannelLifetime` of 120000 (#4502) covers the CTT case.

### Mantis issues filed

| Id | Issue |
| --- | --- |
| C51 | [11458](https://mantis.opcfoundation.org/view.php?id=11458) GDS
scripts expect NA records in query results |
| C52 | [11459](https://mantis.opcfoundation.org/view.php?id=11459) A &
C Refresh scripts leak subscriptions on the shared alarm session |
| C53 | [11460](https://mantis.opcfoundation.org/view.php?id=11460)
`StartThreadPublish.js` defaults misspelled properties |
| U2 | [11461](https://mantis.opcfoundation.org/view.php?id=11461)
Minimum2 oracle on reverse reads |
| U4 | [11462](https://mantis.opcfoundation.org/view.php?id=11462) Part
13 clarification: TreatUncertainAsBad and simple bounds (DeltaBounds) |

The A & C `BadSubscriptionIdInvalid` failures (Refresh, Refresh2,
Shelving) come from CPU load: with other processes at 100 % CPU, all
subscriptions of the CTT alarm session expire within one second. This
happens with and without this PR (3 of 3 loaded runs, 0 of 2 idle runs).

### CTT verification

CTT 1.05.06, scripts 1.05.513, compared at test-case level against a
build of the starting commit:

| Group | Before | After |
| --- | --- | --- |
| Aggregates | 4,562 error messages | 4,186. NumberOfTransitions: 12
test cases go from error to pass. AnnotationCount
`005-01.js`/`005-02.js` pass. NumberOfTransitions
`003-02.js`/`004-02.js` now fail because of C1 ([Mantis
11420](https://mantis.opcfoundation.org/view.php?id=11420)): they
request TreatUncertainAsBad = false, but the harness makes the server
use its default of true. |
| Discovery | 10 warnings | 0. Find Servers Filter `003.js`/`006.js` and
Get Endpoints `002.js` pass. |
| Subscription | – | Publish Min 05 `003.js` goes from warning to pass.
|
| Historical Access, GDS, Node Management | – | no new failures |

`plans/ctt-issues.md` is updated: resolved items leave the open
sections, and the new Mantis issues are recorded.

## Related Issues

- Related to #4479 (CTT configuration notes)
- Follows #4502

## Checklist

- [ ] I have signed the
[CLA](https://opcfoundation.org/license/cla/ContributorLicenseAgreementv1.0.pdf)
and read the
[CONTRIBUTING](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/CONTRIBUTING.md)
doc.
- [x] I have added tests that prove my fix is effective or that my
feature works and increased code coverage.
- [x] I have added all necessary documentation.
- [ ] I have verified that my changes do not introduce (new) build or
analyzer warnings.
- [ ] I ran **all** tests locally using the **UA.slnx** solution against
at least .net **framework** and .net **10**, and all passed.
- [ ] I fixed **all** failing and flaky tests in the CI pipelines and
**all** CodeQL warnings.
- [ ] I have addressed **all** PR feedback received.

Tests run locally on net10.0 after merging master:

| Project | Filter | Passed |
| --- | --- | --- |
| `Opc.Ua.Server.Tests` | Aggregate, Historian, NodeManagement, Publish,
Subscription | 1,419 |
| `Opc.Ua.Core.Tests` | TcpServerChannelBuffer | 65 |
| `Opc.Ua.History.Tests` | Aggregate, Annotation | 67 |
| `Opc.Ua.Sessions.Tests` | discovery locale tests | 8 |

Before the merge, the whole `Opc.Ua.Sessions.Tests` ClientTest fixture
passed (372 passed, 0 failed).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@marcschier
marcschier deleted the romanett/ctt-test-run-regression-7b954e branch September 29, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants