Skip to content

test(characterization): fix the public surface of cloudAgent, evals, webhooks and gamification before the v4 extraction (rail 3.8.55) - #16192

Merged
diegosouzapw merged 4 commits into
release/v3.8.52from
test/rail-characterization-a
Oct 10, 2026
Merged

diegosouzapw merged 4 commits into
release/v3.8.52from
test/rail-characterization-a

Conversation

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Summary

Rail 3.8.55, Task 11 ("caracterização A"). In v4, cloudAgent, evals, webhooks and gamification move out of the core into @omniroute/mod-* modules. To prove that extraction byte-for-byte, the LTS first needs characterization tests that pin each subsystem's public surface and how it behaves today. This PR adds four of them. No production code changes.

What changed

Four new test files in tests/unit/ (the root, so the existing runner glob picks them up). There is one commit per subsystem:

File Tests Pins
characterization-cloudAgent.test.ts 27 Runtime exports of the barrel and all 11 modules; registry ids and base URLs; listSources / createTask / getStatus / approvePlan with a fake fetch (URL, headers, body, mapping); task DB round trip; serializeCloudAgentTask; CORS without Origin; credentials save/get/list/delete; Zod schemas
characterization-evals.test.ts 17 Exports of evalRunner / runtime / builtinSuites; the built-in suite catalogue; runSuite on a minimal suite graded by a fake judge (custom strategy); errored-case forcing (#13137); scorecards; custom DB suites; each evaluateCase strategy and its error path; runtime request building; pre-flight errors of runEvalSuiteAgainstTarget
characterization-webhooks.test.ts 17 Exports of webhookDispatcher + webhooks/**; the event catalogue; deliverWebhook against a real HTTP server on an ephemeral loopback port (payload, event headers, HMAC-SHA256 signature, 4xx final, 5xx retry/backoff, SSRF fail-closed without retries, connection refused); dispatchEvent routing by subscription and kind, delivery rows, last_status / failure_count, auto-disable at 10 failures; the Slack/Discord/Telegram builders
characterization-gamification.test.ts 19 Exports of the barrel and all 10 modules; XP_REWARDS; the 23-badge catalogue; the XP curve, level inversion and titles/tiers; the award pipeline on deterministic fixtures (fresh key → 11 XP + first-token; extending a 2-day streak → 27 XP + daily-user); recognition-only radar_supporter; anti-cheat; sharing; invites; servers

Surface snapshots are literal lists inside each test, so any change to them is a deliberate edit, never an automatic one. Quirks in today's behaviour are pinned, not fixed, and their test names start with characterization: … currently …:

  • isCloudAgentProvider uses in, so it also matches Object.prototype keys such as toString. It has no production caller.
  • CloudAgentBase.mapStatus checks pending before approval, so "pending approval" maps to queued.
  • CursorCloudAgent.listSources turns a non-OK upstream response into [].
  • updateCloudAgentTask silently drops columns that are not on its whitelist.
  • The custom eval strategy passes the judge's raw truthy value through as passed, without converting it to a boolean.
  • When every retry gets a 5xx, deliverWebhook returns status: 0, "Max retries exceeded" and the last upstream status is lost.
  • The Slack, Discord and Telegram builders throw TypeError on an unknown event (events are typed but never validated at runtime).
  • For non-finite XP, calculateLevel returns 1, while xpToNextLevel returns NaN or -Infinity.
  • An unknown gamification action still writes a leaderboard row with score 0.
  • transferTokens(NaN) gets past validation and returns the raw SQLite NOT NULL error. The route and the MCP tool both reject NaN through Zod, so this is not reachable from outside.
  • revokeInvite returns true for an id that does not exist.
  • evaluateBadges (no production caller) rejects with SyntaxError when any badge definition has malformed criteria.

Validation

Gate Command Result
cloudAgent test (isolated) node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts --import ./tests/_setup/isolateDataDir.ts --test --test-force-exit tests/unit/characterization-cloudAgent.test.ts 27/27 pass
evals test (isolated) same, characterization-evals.test.ts 17/17 pass
webhooks test (isolated) same, characterization-webhooks.test.ts 17/17 pass
gamification test (isolated) same, characterization-gamification.test.ts 19/19 pass
all four together same, --test-concurrency=4 tests/unit/characterization-*.test.ts 80/80 pass
ESLint npx eslint tests/unit/characterization-*.test.ts 0 problems
Prettier npx prettier --check tests/unit/characterization-*.test.ts clean
typecheck:core npm run typecheck:core exit 0
strict-null tsc on the new tests ad-hoc tsconfig extending tsconfig.typecheck-core.json with only the 4 files 0 errors in the new files (770 pre-existing errors in transitively-included src/ files, none in the new tests)
test discovery npm run check:test-discovery OK
file size npm run check:file-size OK (largest new file: 646 lines, testCap 1200)

The full unit suite and the build were not run, as the campaign preamble requires (the devbox is shared).

⚠️ base-red inherited: #15306

…rail 3.8.55)

Snapshot the runtime exports of src/lib/cloudAgent (barrel + every module)
and pin the observable contracts used by the /api/v1/agents routes:
registry, listSources/createTask/getStatus with a fake fetch, task
persistence, serialisation, credentials and Zod schemas. Current quirks
are pinned under 'characterization: ... currently ...' names.
…3.8.55)

Snapshot the runtime exports of src/lib/evals (evalRunner, runtime,
builtinSuites) and the built-in suite catalogue, then pin the grading
contract with a fake-judge suite (custom strategy), scorecards, custom
DB suites, runtime request building and pre-flight errors.
…racts (rail 3.8.55)

Snapshot the runtime exports of webhookDispatcher and src/lib/webhooks,
then pin delivery against a real HTTP server on an ephemeral loopback
port: JSON payload, event headers, HMAC-SHA256 signature, 4xx/5xx retry
policy, SSRF fail-closed, per-kind routing in dispatchEvent, delivery
bookkeeping, auto-disable and the Slack/Discord/Telegram builders.
…ontracts (rail 3.8.55)

Snapshot the runtime exports of src/lib/gamification (barrel + every
module), XP_REWARDS and the built-in badge catalogue, then pin the XP
curve/levels and the award pipeline for deterministic fixtures (points,
badges, streak bonus, leaderboard scopes), plus anti-cheat, sharing,
invites and servers. Current quirks are pinned, not fixed.
@github-actions

Copy link
Copy Markdown
Contributor

CI Coverage Report

  • Coverage job: skipped
  • PR test policy: success

Coverage artifact was not available for this run.

@diegosouzapw
diegosouzapw deleted the test/rail-characterization-a branch October 16, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant