Skip to content

test(characterization): fix the public surface of memory, guardrails, skills, a2a and oauth before the v4 extraction (rail 3.8.55) - #16179

Merged
diegosouzapw merged 5 commits into
release/v3.8.52from
test/rail-characterization-b
Oct 10, 2026
Merged

diegosouzapw merged 5 commits into
release/v3.8.52from
test/rail-characterization-b

Conversation

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Summary

Rail 3.8.55, Task 12 ("characterization B"). Before the v4 extraction moves memory, guardrails, skills, a2a and oauth out of the core, the LTS needs tests that pin their public surface and observable behavior as it is today. This PR adds one characterization file per subsystem. There are no production changes.

Every file has three parts:

  • (a) Surface snapshot. Object.keys(await import(<barrel or externally consumed module>)).sort() compared with deepEqual against a literal list. If an export is added or removed, the test fails and the diff shows exactly what moved.
  • (b) Contracts for the 3–6 entry points that code outside the subsystem uses most.
  • (c) Invalid input. The current error behavior is pinned as it is now, without judging it.

When current behavior looks wrong, the test is titled characterization: <x> currently <y>. It asserts what happens today and the bug is not fixed here (see "Defects pinned" below).

What changed

File Tests Covers
tests/unit/characterization-memory.test.ts 20 barrel surface, sqlite auto-registration, store/recall (FTS) on a temp DB, UPSERT on (apiKeyId,key), token budget, access bump, list/delete, Zod/CHECK errors
tests/unit/characterization-guardrails.test.ts 19 barrel surface, 6 default guardrails + priorities, pii-masker with flags OFF (no mutation, same reference through the registry) / ON (masked copy), injection fixture in block/warn, registry short-circuit/skip/chain, resolveDisabledGuardrails
tests/unit/characterization-skills.test.ts 16 exports of the 15 skills modules consumed externally, builtin names, register/getSkill/version selection/global owner, loadFromDatabase, execute success/failure/history, sandbox runner via an injected container provider (no Docker)
tests/unit/characterization-a2a.test.ts 15 exports of the 5 a2a modules consumed externally, A2A_SKILL_HANDLERS, Agent Card v0.3 + v1.0 against the handler table, list-capabilities handler, executeA2ATaskWithState, task transitions/owner scoping, auth/owner helpers
tests/unit/characterization-oauth.test.ts 17 facade/registry/constants exports, 27 provider ids + flow types + aliases, resolvePublicCred wiring (checked by shape and identity, never as literals, per HR #11), claude PKCE auth URL + code exchange, Kiro social/OIDC refresh with a scripted fetch

DB-touching files import tests/_setup/isolateDataDir.ts and call resetDbInstance() in test.after. The env vars that change behavior (OMNIROUTE_API_KEY, REQUIRE_API_KEY, PII/injection flags, OAuth client-id overrides) are deleted at file start so results do not depend on the shell.

Defects pinned as characterization: (not fixed; issue candidates)

  1. memory: createMemory UPSERT with a Date expiresAt throws TypeError (the INSERT path calls toISOString(), the UPDATE path binds the raw Date).
  2. memory: sqliteBackend.search() without maxTokens throws a ZodError. It forwards maxTokens: undefined, which overrides the 2000 default. memoryManager.search swallows the error and returns [].
  3. a2a (consequence of 2): collectMemoryHits with the default sqlite backend always returns [], so A2A memory-hit observability never fires in production.
  4. memory: sqliteBackend.search ignores SearchConfig.limit.
  5. skills: resolveVersion with >=, <= or == never matches. charAt(0) is used as the operator, so the base version parses to NaN.
  6. skills: a timed-out execution is recorded as error, never timeout, even though SkillStatus.TIMEOUT and the DB CHECK both allow it.
  7. a2a: the v0.3 Agent Card (/.well-known/agent.json) serves hardcoded zh-CN name/descriptions, introduced by the localization commit feat(i18n): complete zh-CN localization for compression engines and dashboard UI #9038. The v1.0 card is English.
  8. guardrails (latent; no production caller passes customPatterns): a malformed custom regex makes the prompt-injection guard fail open, even in block mode.

Validation

Gate Command Result
memory test (isolated) node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test --test-force-exit tests/unit/characterization-memory.test.ts 20/20 pass
guardrails test (isolated) same runner, characterization-guardrails.test.ts 19/19 pass
skills test (isolated) same runner, characterization-skills.test.ts 16/16 pass
a2a test (isolated) same runner, characterization-a2a.test.ts 15/15 pass
oauth test (isolated) same runner, characterization-oauth.test.ts 17/17 pass
all 5 together same runner, tests/unit/characterization-*.test.ts 87/87 pass
ESLint npx eslint tests/unit/characterization-*.test.ts (+ lint-staged on commit) 0 problems
Typecheck npm run typecheck:core exit 0
Test discovery npm run check:test-discovery OK (files collected by the tests/unit/*.test.ts runner)
File size npm run check:file-size OK; largest new file is 523 lines (cap 1200)
Public creds npm run check:public-creds OK (no new literals)

The full unit suite and the build were not run, per the campaign rules (the devbox is saturated). CI covers both.

⚠️ base-red inherited: #15306

…cts before the v4 extraction

Pins the memory barrel exports, sqlite backend auto-registration, store/recall
(FTS on a temp DB), UPSERT semantics, token budget and current invalid-input
behavior. Three current defects are pinned as characterization cases (UPSERT
with a Date expiresAt throws, backend search without maxTokens throws a
ZodError, SearchConfig.limit is ignored).
…nd PII/injection contracts

Pins the guardrails barrel exports, the six default guardrails and their
priorities, pii-masker opt-in (no mutation with the flags off, masked copy
with PII_REDACTION_ENABLED), the prompt-injection fixture in block/warn modes,
registry short-circuit/skip/chaining and invalid-input behavior. A malformed
custom pattern currently makes the injection guard fail open; pinned as a
characterization case.
…ute lifecycle

Pins the exports of every skills module consumed outside src/lib/skills, the
builtin handler names, registry scoping (owner vs global, version selection),
loadFromDatabase, executor success/failure/history semantics and the sandbox
runner contract through an injected container provider. Pins two current
defects as characterization cases: two-char version constraints (>=, <=, ==)
never match, and a timeout is recorded as error instead of timeout.
… state machine

Pins the exports of the a2a modules consumed outside src/lib/a2a, both Agent
Cards (v0.3 and v1.0) against A2A_SKILL_HANDLERS, the list-capabilities
handler, executeA2ATaskWithState success/failure, task transitions and owner
scoping, and the auth/owner helpers. Pins two current defects as
characterization cases: the v0.3 card serves zh-CN strings, and
collectMemoryHits always returns [] with the default sqlite backend.
…ed wiring

Pins the oauth facade/registry/constants exports, every provider id with its
flow type and aliases, the resolvePublicCred() wiring of the embedded client
ids (asserted by shape and identity, never as literals), the claude PKCE
auth-url and code-exchange flow, and the Kiro social/OIDC refresh flows with a
scripted fetch, plus current invalid-input and upstream-failure behavior.
@diegosouzapw
diegosouzapw merged commit 9c55ea7 into release/v3.8.52 Oct 10, 2026
23 of 46 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

CI Coverage Report

  • Coverage job: skipped
  • PR test policy: success

Coverage artifact was not available for this run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant