Skip to content

methodology.md

Latest commit

 

History

History
881 lines (767 loc) · 51.4 KB

methodology.md

File metadata and controls

881 lines (767 loc) · 51.4 KB
Raw

Benchmark Methodology

The value of a benchmark is entirely in its fairness. This document defines the rules that make an AXIAM-vs-competitor run comparable, and the exact meaning of every metric we report.

1. Principles

  1. Identical logical workload. Every target receives the same sequence of logical operations (login, issue token, introspect, refresh, authz-check, …). Only the wire encoding differs, isolated in scenarios/lib/targets.js.
  2. Standard protocols only. We exercise OAuth2 (RFC 6749), OIDC, token introspection (RFC 7662), and JWKS (RFC 7517). No proprietary endpoints in the cross-target comparison. AXIAM-only extras (e.g. the gRPC AuthorizationService) are measured separately and clearly labelled as non-comparative.
  3. Equal resource envelope. Every target runs with the same CPU and memory caps (--cpus, mem_limit) so that "performance per resource" is meaningful. The caps are set in each targets/<name>/docker-compose.yml and overridable via BENCH_CPUS / BENCH_MEM.
  4. Same host, back-to-back. Targets are benchmarked sequentially on the same machine, never concurrently, so neither steals CPU/cache/IO from the other.
  5. Warm before measure. Each scenario runs a warm-up stage (excluded from metrics) before the measured stage, so JIT/connection-pool/cache effects do not pollute results.
  6. The load generator must not be the bottleneck. k6 runs on the host (not in the capped network), and we assert generator CPU headroom. If k6 saturates, the run is flagged invalid in the report.

2. The comparison matrix

result = f(target, security_profile, scenario)
  • target ∈ { axiam, keycloak, zitadel, … }
  • security_profile ∈ { p0-plaintext, p1-tls12, p2-tls13, p3-mtls } (see docs/security-profiles.md)
  • scenario ∈ the k6 scripts under scenarios/

Each cell produces one result record (JSON) under results/.

3. Scenarios

Scenario file Logical operation Protocol Comparative?
oauth2_password_login.js Resource-owner login → token (or session) HTTP/OAuth2 Yes
oauth2_client_credentials.js Machine-to-machine token issuance HTTP/OAuth2 Yes
token_introspection.js Validate an opaque/JWT token (RFC 7662) HTTP/OAuth2 Yes
token_refresh.js Refresh-token rotation HTTP/OAuth2 Yes
jwks_fetch.js Fetch signing keys (RFC 7517) HTTP Yes
userinfo.js OIDC /userinfo HTTP/OIDC Yes
authz_check_rest.js Authorization decision (REST) HTTP/REST AXIAM-only*
authz_batch_rest.js Batch authorization decision (REST) HTTP/REST AXIAM-only*
authz_check_grpc.js Low-latency authorization decision gRPC AXIAM-only*
authz_batch_grpc.js Batch authorization decision gRPC AXIAM-only*
zitadel_userinfo_grpc.js Identity read (AuthService/GetMyUser) gRPC Zitadel-only†

* Most competitors do not expose a directly equivalent authorization-decision endpoint (REST or gRPC, single or batch); these scenarios are reported separately as AXIAM capability metrics, not head-to-head numbers. The REST authz scenarios also serve as the wire baseline for SDK check_access/batch_check overhead (see sdk/HARNESS-SPEC.md). All four require a seeded resource + role grant and a logged-in user token; the authz scenarios log in as the bench user in setup().

† Zitadel's primary API surface is gRPC (maintainer requirement to benchmark it — see claude_dev/benchmark-improvement-plan.md D4), so zitadel_userinfo_grpc.js calls zitadel.auth.v1.AuthService/GetMyUser against the minimal vendored proto in scenarios/proto/zitadel/ (see that directory's README.md for exactly what was vendored/trimmed and why — short version: the real upstream .proto files at the pinned v4.15.2 tag were fetched, then hand-trimmed to just the one RPC and the message fields this benchmark decodes, to avoid pulling in Zitadel's full transitive proto graph for options/docs annotations that don't affect the wire format). Neither AXIAM nor Keycloak expose an equivalent gRPC identity RPC, so this scenario is wired into runner/run-benchmark.sh's Zitadel-only scenario list (ZITADEL_ONLY_SCENARIOS) and never appears for the other two targets. It pairs with Zitadel's own userinfo.js cell as a protocol-efficiency comparison (REST vs gRPC, same logical operation, same vendor) — see "Comparability: protocol-efficiency (gRPC vs REST, same vendor)" below. It is not a cross-vendor head-to-head number, and runner/report.py excludes it from the cross-vendor "Efficiency comparison" tables the same way it excludes the AXIAM-only authz scenarios above (both are listed in report.py's NON_COMPARATIVE_SCENARIOS).

A comparable gRPC "introspect" scenario was considered and deliberately not added: Zitadel's session.v2.SessionService (GetSession, CreateSession, …) operates on Zitadel's own session-ID/session-token identity model, not on an OAuth2 access/refresh token, so GetSession is not a genuine equivalent to token_introspection.js's RFC 7662 "is this token active" check — forcing that equivalence would benchmark a different logical operation under a misleading label, which §1's "identical logical workload" principle rules out. SessionService.CreateSession (the gRPC login counterpart) is task D5's concern, not D4's.

4. Load model

Each scenario uses a closed-loop ramping-VU model with three stages:

Stage Duration (default) Counted? Purpose
warm-up BENCH_WARMUP 30s No Fill caches/pools, hit steady state
measure BENCH_DURATION 120s Yes The reported numbers
cool-down 10s No Drain in-flight requests

VU count, ramp shape, and durations are environment-overridable (see scenarios/lib/config.js). The default targets a moderate sustained load; for saturation/sizing studies, raise BENCH_VUS until latency thresholds break and record the last passing point.

5. Metrics

Performance (from k6)

  • throughput — successful iterations per second over the measure stage (iterations rate, 2xx/expected-status only).
  • latency p50 / p95 / p99 — end-to-end request duration (http_req_duration, or grpc_req_duration), in milliseconds.
  • error_rate — fraction of iterations that failed a check or returned an unexpected status. A run with error_rate > BENCH_MAX_ERROR (default 1%) is marked invalid.
  • bench_http_proto → the report's http column — the wire protocol every measured response actually negotiated (k6's res.proto), recorded per response by scenarios/lib/metrics.js as 10/11/20/30 = HTTP/1.0 / 1.1 / 2.0 / 3, and decoded by report.py. It is a Trend, not a counter, so a cell whose min and max disagree renders as mixed(1.1,2.0) — such a cell spanned two protocols and is void as a controlled protocol comparison. — means the cell predates this metric or never got a response. Any security-cost table whose rows did not all negotiate the same protocol gets an explicit "protocol confound" warning, because its Δ then measures TLS and the protocol change together. Added in H6 after a TLS h1-vs-h2 conviction attempt turned out to have no record of which protocol its cells had used — do not interpret an h1/h2 comparison whose http column you have not read.

Resource (from the sampler)

Sampled every BENCH_SAMPLE_INTERVAL (default 1s) over the measure stage via docker stats on the target's containers (server + datastore + broker):

  • cpu_cores_avg / cpu_cores_p95 — CPU cores consumed (1.0 = one full core).
  • mem_mib_avg / mem_mib_p95 — resident memory in MiB.

report.py also keeps a per-container breakdown of cpu_avg/mem_avg (not just the whole-stack sum) and computes, per cell, a bottleneck column: the container(s) whose cpu_avg ≥ 0.95 × their configured CPU cap (read from meta.json's containers[].cpu_cap, falling back to the docker-compose.yml default for that role — server 2 CPU, DB 2 CPU, broker 1 CPU, TLS edge 1 CPU — if the meta predates that field), or none if nothing in the stack saturated. none is itself informative: it means the client, the network, or an un-pegged serialization point is the limiter, not raw CPU — see the "Appendix: per-container resource breakdown" section of the generated report for the full per-container table.

Host telemetry — CPU frequency, temperature, generator headroom

docker stats measures time-based core utilization: it cannot distinguish a core spinning at 3.9 GHz from one throttled to 2.2 GHz. On laptop hardware (see "Running on a laptop" in docs/security-profiles.md / the runbook) that gap matters, so resource/host-sampler.sh runs alongside the container sampler at the same cadence and writes <scenario>.host.csv (epoch_ms, cpu_mhz_avg, cpu_mhz_min, temp_c_max, host_cpu_util_pct, k6_cpu_cores), all from no-sudo /sys//proc reads:

  • mhz_avg — mean, over the window, of the per-sample mean CPU frequency across all online cores (/sys/…/cpufreq/scaling_cur_freq).
  • mhz_min / mhz_max ratio — the lowest single-core frequency seen anywhere in the window, divided by the window's peak mean frequency. A number near 1.0 means the clock stayed flat; a low number means at least one core spent time markedly slower than the pack.
  • temp_max — the hottest thermal zone's peak reading in the window (°C).
  • k6_cores_avg — CPU cores (not %) consumed by the k6 process(es) themselves during the window.

Interpretation rule (from the re-examination of the first full run, 2026-07-19): if repeated CPU-bound cells ~30 min apart agree closely (e.g. AXIAM introspection p0 vs p2: 2199 vs 2192 req/s) and mhz_avg stays flat across the run, cross-target/cross-profile comparisons are not distorted by throttling — a constant sustained-clock reduction depresses all absolute numbers uniformly, which is invisible to docker stats alone but would make every cell in the run conservative by the same factor, not selectively unfair to one target. report.py flags two conditions automatically:

  • clock_variance — this cell's mhz_avg sagged more than 15% below its own window's peak mhz_avg — the clock was not flat during this specific measurement, so treat its absolute numbers with more caution than a flagged-clean cell.
  • generator_saturated — k6_cores_avg > 0.8 × (host_cpus − stack_cap_cpus), i.e. k6 itself was eating most of the CPU headroom left after the target stack's caps — the load generator may be the bottleneck, not the target.

Both flags appear in the host_flags column of the "All results" table.

Efficiency (derived, the headline numbers)

  • throughput_per_core = throughput / cpu_cores_avg → requests per second per CPU core. Higher is better.
  • throughput_per_gib = throughput / (mem_mib_avg / 1024) → requests per second per GiB of RAM. Higher is better.
  • cpu_ms_per_request = (cpu_cores_avg * 1000) / throughput → CPU-milliseconds spent per request. Lower is better.

These derived numbers are the answer to "can AXIAM deliver competitor-level performance at a lower resource cost?" — compare throughput_per_core and cpu_ms_per_request across targets at equal latency.

report.py's efficiency tables also render a server-container-only variant of both numbers, computed against just the primary server/app container's CPU+mem (excluding the database, broker, and TLS edge containers). AXIAM's stack includes a broker (RabbitMQ) that Keycloak and Zitadel don't; the whole-stack numbers above fold that cost in silently, while the server-only variant isolates it so it stays visible rather than understating AXIAM's per-request server cost relative to a single-process competitor.

Comparability: protocol-efficiency (gRPC vs REST, same vendor)

A second, distinct comparability class alongside the fallback flag below: some scenarios exist specifically to compare two wire protocols against the same logical operation on the same vendor, not to compare vendors against each other. Today that's:

  • AXIAM: authz_check_rest.js/authz_batch_rest.js (REST) vs authz_check_grpc.js/authz_batch_grpc.js (gRPC) — the authorization decision, over both wire protocols AXIAM exposes it on.
  • Zitadel: userinfo.js (REST /oidc/v1/userinfo) vs zitadel_userinfo_grpc.js (gRPC AuthService/GetMyUser) — the identity read, over both wire protocols Zitadel exposes it on.

These are within-vendor pairs: "does gRPC cost less than REST for the same operation on the same server?" is a meaningful, fair comparison because nothing about the target, resource caps, or logical workload differs between the two rows — only the wire encoding does. They answer a materially different question than the cross-vendor "Efficiency comparison" tables ("is target A faster than target B?"), so report.py keeps them out of those tables entirely (NON_COMPARATIVE_SCENARIOS) rather than trying to average a gRPC cell into a REST-only cross-vendor group. To read a protocol-efficiency pair, compare the two scenarios' rows directly in the "All results" table for the same (target, profile) — e.g. zitadel_userinfo_grpc vs userinfo at zitadel / p0-plaintext.

Comparability flags (fallback operations)

Some logical ops can't always be measured for real on every target — e.g. Zitadel generally ships ROPC disabled, so its login() adapter falls back to client_credentials (see scenarios/lib/targets.js); token_refresh.js falls back the same way when a target issues no refresh token to rotate; and userinfo.js's setup() falls back to a client_credentials token only if minting a real user token fails. Every fallback increments the bench_fallback k6 counter for that iteration. A cell with bench_fallback > 0 is still valid (it passed the normal validity gates) but is annotated comparability: fallback-op, shown with fallback: yes in the full results table, and excluded from head-to-head efficiency/winner tables — a fallback op measures a different (usually cheaper) operation than its label, so ranking it against a real login/refresh would be comparing different things under the same name.

Comparability flags (protocol variant)

A third comparability class, orthogonal to both of the above: some scenarios are structurally guaranteed to measure a different underlying operation per target, even though every cell is a real, correct, non-fallback measurement of the operation its own label names. Today that's token_refresh.js: AXIAM ships no OAuth 2.1 ROPC/password grant (the grant is removed in OAuth 2.1), so it has no way to seed an OAuth2 refresh token from a password login — its cell measures a session refresh (POST /api/v1/auth/refresh, cookie + CSRF double-submit, axiam-auth's SessionRepository), while Keycloak's and Zitadel's cells measure the OAuth2 refresh_token grant. Both renew an access credential without re-authenticating — comparable at the product level — but never at the protocol level (see claude_dev/refresh-harness-diagnosis.md §6 for the full derivation).

Unlike fallback-op, this is a static, per-scenario property — it can't be derived from bench_fallback (a cell can read fallback_count: 0 and still be a protocol-variant cell), so report.py carries it as a fixed PROTOCOL_VARIANT_SCENARIOS set rather than a per-run classification. Unlike fallback-op, a protocol-variant cell is never excluded from head-to-head tables — the measured operation IS the labelled one, for every target — it is annotated comparability: protocol-variant (visible as protocol-variant in the "All results" table's fallback column, and as a per-target caveat in the "Efficiency comparison" section) so a reader sees the operations diverge before reading the numbers as "AXIAM is Nx Keycloak at refresh".

Security cost (derived across profiles)

For a fixed (target, scenario), the report computes the relative cost of each profile vs the p0-plaintext baseline:

  • tls_throughput_penalty = 1 - throughput(profile)/throughput(p0)
  • tls_latency_overhead_ms = p95(profile) - p95(p0)

This quantifies what each security tier costs, so an operator can choose a posture with eyes open.

6. Validity gates

A result record is flagged valid: false (and excluded from headline comparisons) if any hold:

  • error_rate > BENCH_MAX_ERROR
  • generator CPU saturation detected (k6 dropped iterations)
  • fewer than BENCH_MIN_SAMPLES resource samples captured
  • target health check failed during the measure stage

Invalid cells are still written to results/ (for debugging) but the report lists them in a separate "excluded" section.

7. Reproducibility

Every result record's meta.json embeds, per cell: target/profile/scenario, security profile (scheme, tls_min, client_auth), resource caps (caps), rate-limit posture, VU/duration config, and host CPU/RAM (host). It also records, for full reproducibility of what actually ran:

  • containers — for every container in the target's stack: name, role (server/db/mq/edge), image, image_digest (from docker inspect --format '{{.Config.Image}} {{index .RepoDigests 0}}'), and the cpu_cap it was measured against.
  • scenario_sha256 — hash of the exact scenarios/<name>.js file executed (sha256sum), so a report can be traced back to the harness code that produced it even after the scenario file later changes.
  • batch_size — BENCH_BATCH_SIZE (default 5), the batch-authz request size.
  • host_kernel, docker_version, cpu_model, cpu_governor — host facts (uname -r, docker version, /proc/cpuinfo, /sys/…/cpufreq/scaling_governor).
  • k6_cpu_cores_avg — the generator's own CPU consumption over the measure window (from the A6 host sampler), the basis for generator_saturated.

report.py tolerates older meta.json files that predate any of the above — missing fields degrade to sensible defaults (e.g. compose-file CPU cap defaults instead of a recorded cpu_cap) rather than crashing the report.

Two runs with the same embedded config on the same hardware should agree within run-to-run noise (typically <5% on throughput). Always report the median of N≥3 runs for any published figure — see §8 "Multiple runs — median-of-N (C1)" below for how bench-matrix/report.py do this for you automatically.

Sharing results. runner/seed.sh writes client secrets and the bench user's password to .seed/<target>.seed.env, which is gitignored and lives outside results/. results/ itself holds nothing secret. To hand off or publish a run, use just bench-pack: it archives only *.k6.json, *.res.csv, *.host.csv, *.meta.json, and report.md into results-<date>.tar.xz, and verifies (grepping the packed content for SECRET/PASSWORD) that nothing sensitive made it in before leaving the archive on disk.

8. Multiple runs — median-of-N (C1)

Single-run numbers hide run-to-run noise (thermal state, background load, scheduler jitter). just repeat=<N> … bench-matrix (default repeat := "3" in justfile) runs the entire target×profile×scenario matrix N times, each pass writing into its own results/run-<i>/ subtree using the exact same flat <target>/<profile>/<scenario>.* layout underneath — so every per-cell mechanism described elsewhere in this document (the seed-ok marker, run-benchmark.sh's meta/k6/resource/host outputs) is reused completely unchanged, just once per repeat.

report.py auto-detects which layout is present in the directory passed to --results:

  • results/run-<i>/ subdirectories present → median-of-N mode. For each (target, profile, scenario) cell, every metric is aggregated medianed independently per metric across that cell's valid runs — throughput, p50/p95/p99, cpu, mem (whole-stack and per-container), and the host telemetry columns. Derived numbers (thr/core, cpu_ms/req, …) are then recomputed from those medians, not separately medianed themselves.
    • A cell is only marked valid if ≥2 of its runs were individually valid — with 0 or 1 valid runs there's no meaningful median, so the aggregated cell is still shown (for visibility, using whatever data is available) but excluded from headline comparisons, same as any other invalid cell.
    • The report adds a runs(valid/n) column (e.g. 3/3) and a ±thr% column: the throughput spread across valid runs, (max−min)/2 expressed as a percentage of the median — a quick read on how noisy that cell was.
  • No run-*/ subdirectories (the classic single-pass layout, e.g. the existing 2026-07-19 results/ tree, or a manual bench-up/bench-seed/ bench-run workflow that never went through bench-matrix) → the report is generated exactly as before this change, with no runs/±thr% columns.

just repeat=1 … bench-matrix still works — it just produces a single results/run-1/ tree (report.py medians a single-element list, i.e. reports that one run's numbers, with runs(valid/n) = 1/1 or 0/1 and a cell marked invalid since 1 < 2 required valid runs).

9. Datastore sensitivity & fair DB tuning (C2)

Uncapped-DB sensitivity pass. just dbcaps=uncapped … (default dbcaps := "capped") raises the datastore's envelope from the standard 2 CPUs / 1024 MiB to 4 CPUs / 2048 MiB — BENCH_DB_CPUS/BENCH_DB_MEM were already read directly by every target's docker-compose.yml (surrealdb/postgres services' cpus:/mem_limit:); dbcaps just wires the two values through bench-up. Use it to check whether the datastore — not the server process — is the ceiling on a given scenario: run the same cell capped vs uncapped and see whether throughput moves. The chosen caps are recorded per-container in meta.json's containers[].cpu_cap / containers[].mem_cap_mib, read straight off the running container (docker inspect's HostConfig.NanoCpus/HostConfig.Memory) rather than trusted from the shell that ran bench-up — so a separate bench-run invocation still records whatever cap the DB container actually started with, and the "Appendix: per-container resource breakdown" table renders both cpu_cap and mem_cap(MiB) per cell.

Fair competitor DB tuning. Both Keycloak's and Zitadel's postgres service now start with minimal, uniform, non-durability tuning applied identically to both — shared_buffers=256MB, effective_cache_size=512MB, max_connections=200 via compose command: flags — sized sensibly for the standard 1 GiB cap rather than left at Postgres's stock defaults (which target a much larger box). This is a "same DB, sane settings" fix, not a thumb on the scale: both competitors get the exact same flags, and nothing about durability is touched (see below).

Durability parity note. Postgres (used by both Keycloak and Zitadel here) defaults to synchronous_commit = on: a transaction's WAL record is written and fsynced to disk before the client's COMMIT returns — durable-by-default. AXIAM's bench target (targets/axiam/docker-compose.yml) runs surrealdb/surrealdb:v3 with the SurrealKV storage engine (surrealkv:/data/axiam.db), no SURREAL_SYNC_DATA override — i.e. whatever the v3 image defaults to. SurrealKV itself exposes two per-transaction durability levels: Eventual (data written to the OS page cache, fsync deferred — SurrealKV's own stated default and "best performance" mode) and Immediate (fsync before commit() returns — the slower, durable mode). Publicly, SurrealDB has stated that 2.x did not enable disk sync by default, and that 3.x does (SURREAL_SYNC_DATA on by default), in direct response to community criticism that its earlier benchmark numbers were measured with every engine's writes sitting in the page cache rather than actually flushed to disk. Since the AXIAM bench compose pins :v3 and sets no override, it should inherit that "sync on" default — but this framework has not independently verified live fsync behavior against a running container (no strace/fsync-call instrumentation has been run; this environment currently has no live bench stack to check against). Until that's verified with an actual run:

  • Treat the Postgres-vs-SurrealKV durability comparison as not confirmed equivalent. If AXIAM's numbers were ever found to come from a configuration with fsync effectively off while Postgres's synchronous_commit = on stayed on, that would inflate AXIAM's write-heavy throughput (login, token issuance, refresh — anything that writes) unfairly relative to the competitors, and must be corrected before publishing head-to-head numbers on those scenarios.
  • This belongs on the public caveats list (PUBLIC_BENCH_ANALYSIS.md §4, "Other comparability caveats") as an open item, not silently assumed fine. PUBLIC_BENCH_ANALYSIS.md itself is regenerated as part of the plan's E4 task (after Phase C's re-run lands) rather than edited here — this methodology note is the source-of-truth statement E4 should carry forward verbatim into that caveats list.
  • We do not change either engine's durability settings to make AXIAM look faster — the caps/tuning above are the full extent of C2's changes.

10. Running on a laptop (C3)

This is not the intended long-term benchmark environment (see the plan's operating constraint — a server-class re-run is deferred until dedicated hardware is available), but until then, every run happens on a laptop with all the variance that implies: thermal throttling, power-management clock scaling, background processes, and battery-vs-AC behavior. This section is the runbook for controlling what can be controlled and measuring what can't (via the A6 host telemetry columns — mhz_avg, mhz_min/max, temp_max(C), k6_cores, host_flags in the "All results" table).

Before starting a matrix:

  1. Plug into AC. Battery power profiles throttle far more aggressively than plugged-in ones on most laptops, and some platforms cap turbo boost entirely on battery.
  2. Set the CPU governor to performance: 
    sudo cpupower frequency-set -g performance
    run-benchmark.sh reads /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor for meta.json's cpu_governor field regardless, but now also warns (not fails — some kernels/VMs don't expose a governor at all, in which case it reads unknown and the warning is skipped) at the start of every bench-run when the governor isn't performance.
  3. Optional stability mode — disable turbo boost entirely: 
    echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
    Turbo boost is itself a source of variance (it ramps and backs off based on thermal headroom that changes cell-to-cell). Disabling it trades a lower ceiling for a flatter one. Run the entire matrix in ONE mode — never mix turbo and no-turbo runs in the same dataset; a cell run with turbo on is not comparable to one run with it off, and nothing in the harness currently detects or flags a mixed-mode dataset, so this is on the operator to enforce.
  4. Close background applications — browsers, IDEs with language servers, sync clients, anything that periodically bursts CPU. k6 itself runs on the host (methodology §1.6), so anything competing with it or with the capped containers adds noise on both sides of the measurement.
  5. Raise the laptop for airflow (a stand, books, anything that isn't flush against a desk/lap) — closed intake vents are one of the most common causes of a laptop hitting its thermal ceiling under sustained load.

During the run — the idle gap between cells. run-benchmark.sh now pauses BENCH_CELL_PAUSE seconds (default 60s, 0 disables it) between scenarios within a bench-run invocation, so the previous cell's heat has a chance to dissipate and its connection pools/allocations settle before the next measurement starts — rather than every cell after the first starting from a warmer, more-loaded baseline than the one before it.

After the run — verify with telemetry, don't just trust the numbers. Check the "All results" table's host_flags column:

  • clock_variance — this cell's mean clock sagged >15% below its own window's peak; treat its absolute numbers with more caution.
  • generator_saturated — k6 itself was eating most of the host's non-stack CPU headroom; the load generator may have been the bottleneck, not the target.

If neither flag appears across the run and mhz_avg stays roughly flat scenario-to-scenario, per §5's interpretation rule the cross-target/ cross-profile comparisons are not distorted by throttling.

See also ../README.md for the quick-start commands this runbook assumes.

11. Production rate-limit posture (C4)

All AXIAM numbers elsewhere in a normal run use the neutralized rate-limit posture (rl := "neutralized" in justfile, the default) — AXIAM's per-IP limiter raised to effectively unlimited, so a single-source-IP k6 run measures endpoint capacity rather than the limiter (see §1's principles and the comment above AXIAM_BENCH_RL_POSTURE in targets/axiam/docker-compose.yml). Competitors ship no equivalent per-IP limiter, so neutralized is the only posture that is head-to-head-comparable.

To also publish what an operator actually gets out of the box, run the AXIAM matrix once with rl=prod — the server's production rate-limit defaults active. Because the results path (results/<target>/<profile>/<scenario>.*) doesn't encode posture, a prod run must not land in the same results/ tree as a neutralized run for the same target/profile/scenario (it would silently overwrite it). Direct it to its own tree instead:

BENCH_RESULTS_DIR="$PWD/results/axiam-prod-posture" \
  just target=axiam profile=p0-plaintext rl=prod bench-up
BENCH_RESULTS_DIR="$PWD/results/axiam-prod-posture" \
  just target=axiam bench-seed
BENCH_RESULTS_DIR="$PWD/results/axiam-prod-posture" \
  just target=axiam profile=p0-plaintext rl=prod bench-run
just target=axiam bench-down

python3 runner/report.py --results results/axiam-prod-posture

run-benchmark.sh stamps every cell's meta.json with the posture it actually ran under (rate_limits, read back from the running container via docker inspect rather than trusted from the invoking shell — see the comment above detect_rl_posture()), so report.py always knows which cells are prod. It renders them in a dedicated "AXIAM production rate-limit posture — NOT comparable to competitors" section, separate from the "All results" head-to-head tables, and posture_bucket() makes the "Efficiency comparison" tables refuse to render a comparison group that mixes postures (or contains an unknown/unstamped one) — verified by feeding report.py a synthetic prod-posture AXIAM cell alongside a neutralized Keycloak cell for the same (scenario, profile): the group is rejected with an explicit "Not comparable — mixed or unknown rate-limit posture" note instead of being silently averaged in.

Publish the prod-posture report alongside the neutralized matrix, not as a replacement for it — the framing is "AXIAM ships per-IP rate limits by default; Keycloak and Zitadel don't," turning what would otherwise read as a benchmark asterisk into a documented security-posture advantage.

12. Post-seed settle gate & cell-order rotation (G2)

Why this exists. Run 3's analysis (PRIVATE_BENCH_ANALYSIS.md §1, "THE run-3 discovery: a post-seed serialized-DB transient invalidates every 'first cell after seed'") found a ~5–7 minute window right after bench-seed in which the AXIAM stack serves everything at ~45 req/s, with SurrealDB pinned at ~1.0–1.3 cores (never its 2.0 cap) and the server nearly idle — the signature of a ~22 ms serialized unit of DB work queuing every request — before spontaneously recovering to normal throughput (~740 req/s) and DB CPU (~2.0 cores). Because run-benchmark.sh always ran the scenario glob in the same (alphabetical) order, authz_batch_grpc/authz_batch_rest were always cells 1–2 right after seed, in every run collected so far — so every historical batch-scenario number measured this transient, not the batch authz path. The same signature was caught independently on a completely different scenario (the B2 oauth2_client_credentials h1-isolation cell, also first-after-seed). G1's root-cause note (claude_dev/postseed-transient-investigation.md) has the bisection; this section documents the two harness countermeasures G2 added so the artifact can't silently corrupt a cell again, whatever the root cause turns out to be.

UPDATE (H2, 2026-07-28): the root cause is now known, and "post-seed transient" is not an accurate name for it. postseed-transient-investigation.md found the effect is a permanent, per-request cost, not a post-seed warm-up: six endpoints (POST /api/v1/authz/check, POST /oauth2/token, POST /oauth2/introspect, POST /oauth2/revoke, POST /api/v1/auth/login, and — a separate bug — GET /api/v1/users) perform one synchronous SurrealDB write (the shared rate-limit bucket UPSERT) before the handler runs, on every request, indefinitely. What varies by host is only how expensive that one write is: on the G-box (the run-3 host this section's numbers above describe) it is cheap enough (~22 ms) that the endpoints still clear hundreds of req/s once whatever makes it briefly pricier right after a fresh seed subsides; on the H2 investigation host it costs ~40 ms and never subsides — 16–21 ops/s at any concurrency, unaffected by idle time, restarts, pool size, or even an in-memory storage backend. The settle gate below is still exactly the right countermeasure (it protects against exactly this shape of clamp, whatever causes it, on whatever host is running), and everything in §12.1–§12.4 remains accurate to what it does — but do not read "post-seed window" as implying the effect is caused by seeding, or that it always clears. On a host where the write is expensive, the settle gate will correctly spend its full BENCH_SETTLE_TIMEOUT_SECS and stamp settle_timeout: true on every session that touches one of the six endpoints — that is the gate reporting a real, permanent property of that host, not a flaky probe (see postseed-transient-investigation.md §8.1).

12.1 Settle gate v2 (H1) — why the serial canary was blind

G2's original gate polled a single already-seeded canary request once per second and required BENCH_SETTLE_STABLE_SECS (30) consecutive ticks under BENCH_SETTLE_MAX_MS (100 ms) before releasing the first cell. Measured against a live run, it passed in ~34 seconds — deep inside the ~6-minute clamp G1 reproduced (§1 above; PRIVATE_BENCH_ANALYSIS.md §1). That is not a threshold-tuning bug; it is structural. The post-seed clamp is a concurrency ceiling — the stack sustains ~44 ops/s in-window vs ~730–750 ops/s settled — not a per-request latency problem. The clamp's own signature is a ~22 ms serialized unit of DB work queuing every request: at one request per second, that 22 ms is fast, whether the server behind it can actually sustain 44 ops/s or 730 ops/s. Asking one question a second can never distinguish "this server can only do 44 of these per second" from "this server can do 730 of these per second" — both answer a single lonely request in about the same ~22 ms. Only asking for more concurrent throughput than the clamp allows at once can reveal the difference, and a 1 rps canary structurally never does that. (The consequence, from the G run: G3 run-1 cells, G5's K=1/K=100 cells, G8's CC cells, and both G4/G8 CC comparison cells all measured inside this window despite the gate + a warm-up cell — see §1 above.)

H1 replaces the serial canary with a short concurrent burst probe, implemented in curl (no k6 dependency for the gate itself — k6 is reserved for the scenarios the gate protects):

  1. Fire BENCH_SETTLE_BURST_VUS (default 20) concurrent, closed-loop workers (no think time — this is deliberately a concurrency probe) against the same clamp-sensitive, target-appropriate endpoint the old canary used (axiam: POST /api/v1/authz/check, authenticated as the seeded bench user, cookie jar + CSRF token established once and shared; falls back to /health/JWKS if the seed didn't provide enough to authenticate — same fallback ladder as before) for BENCH_SETTLE_BURST_SECS (default 15 s).
  2. Require either BENCH_SETTLE_PROBE_THR (default 400 ops/s) or p50 latency under BENCH_SETTLE_PROBE_P50_MS (default 150 ms) under that concurrency. In-window measured ~44 ops/s, settled ~730–750 ops/s — 400 sits far from both, so there is no realistic false-pass/false-fail case at the threshold.
  3. If a probe attempt fails, retry after BENCH_SETTLE_RETRY_SECS (default 30 s) — a fresh burst, not a continuation of the failed one.
  4. If nothing clears the bar within BENCH_SETTLE_TIMEOUT_SECS (default 600 s / 10 min — comfortably above the ~5–7 min window G1 measured), the gate warns and proceeds anyway, exactly as before; it never treats "still not settled" as a hard failure.

BENCH_SETTLE_STABLE_SECS/BENCH_SETTLE_MAX_MS (the old serial-canary knobs) still parse without error if a script exports them, but the v2 gate no longer consults them.

Every cell this run produces — not just the first — records the wait in its meta.json:

"settle_wait_secs": 47,
"settle_timeout": false,
"settle_probe_thr": 400

(settle_wait_secs is the same value across every cell of one bench-run invocation, since the gate runs once per invocation; a median-of-N cell aggregated across results/run-*/ takes the max wait and any timeout across its repeats — the worst case, not an average, since a settle timeout on even one repeat means that repeat's data may still be suspect.) report.py surfaces settle_wait_secs/timeout in a dedicated "Appendix: post-seed settle gate" table, and adds a settle_timeout host_flags entry (alongside clock_variance/generator_saturated) to the main "All results" table for any cell whose gate hit the hard timeout. As of H1, report.py also refuses any such cell outright (excludes it from valid, listed under "Excluded (invalid) cells" with the reason spelled out, and printed as a loud WARN to stderr) rather than only flagging it — a contaminated cell silently entering a median/head-to-head table is exactly the data-quality failure this gate exists to prevent. All of the above degrades cleanly (simply absent/not triggered) on meta.json files that predate G2/H1, so older trees still render unchanged.

Set BENCH_SETTLE=0 to skip the gate entirely — useful for a quick manual bench-run against an already-settled (or already-known-warm) target where waiting is pure overhead.

12.2 Cell-order rotation

Independently of the settle gate (defense in depth — the gate should make this moot, but "first cell after seed" was silently wrong for months before anyone noticed), run-benchmark.sh now rotates the executed scenario order by the repeat's run index, so the corruption mechanism itself (always running the same scenario first) can't recur even if a future gate has a gap. bench-matrix (justfile) exports BENCH_RUN_INDEX=<i> for each of its repeat passes; run-benchmark.sh's rotate_scenarios() left-shifts the (already target/profile-filtered) scenario list by (BENCH_RUN_INDEX - 1) mod N:

  • run-1 executes scenarios in their natural (alphabetical) order,
  • run-2 starts one scenario further along,
  • run-3 two further, and so on, wrapping around.

This is a pure, deterministic function of the run index and scenario count — it changes only the order cells run in, never which scenarios run, and a manual single bench-run invocation (BENCH_RUN_INDEX unset, defaults to 1) is unaffected (natural order, same as before G2). Each cell records its position in the rotated order as cell_order_index in meta.json, so a report reader can always tell which cell was first (and therefore settle-gated) in a given run.

12.3 Self-describing labeled passes (A7-safe)

A sensitivity pass (e.g. AXIAM__DB__POOL_SIZE=4, a batch-strategy A/B, the decision cache on/off) previously left no trace in meta.json — only the results-directory name (sens-pool-4/, ...) said what the pass was (PRIVATE_BENCH_ANALYSIS.md §2.2). run-benchmark.sh now reads the AXIAM server container's actual environment (docker inspect, the same pattern detect_rl_posture() already used for rate_limits) and dumps every AXIAM__* variable it finds into each cell's meta.json under "axiam_env" — so pool size, batch strategy, decision cache, rate-limit posture, hash concurrency, or any other pass-through knob is identifiable from the metadata alone.

Secret redaction (A7 — shared archives contain no secret material) is mandatory and non-negotiable: the value of any AXIAM__* key whose name matches PASSWORD|SECRET|KEY|PEPPER|PEM|TOKEN (case-insensitive) is never written — only the key name (so the setting's presence stays visible) with a literal "<redacted>" placeholder:

"axiam_env": {
  "AXIAM__AUTHZ__BATCH_STRATEGY": "coalesced",
  "AXIAM__DB__POOL_SIZE": "4",
  "AXIAM__DB__PASSWORD": "<redacted>",
  "AXIAM__AUTH__JWT_PRIVATE_KEY_PEM": "<redacted>",
  "AXIAM__AUTH__PEPPER": "<redacted>",
  "AXIAM__AMQP__SIGNING_KEY": "<redacted>"
}

Because those key names legitimately contain PASSWORD/SECRET/KEY/PEM- shaped substrings even fully redacted, just bench-pack's leak check (justfile) now filters out every line already carrying the literal <redacted> placeholder before scanning packed content for SECRET/PASSWORD — an actual leaked value would never carry that placeholder, so this can't hide a real leak, only the expected, fully- redacted key names. report.py renders one representative cell's axiam_env per (target, profile) in a dedicated "Appendix: AXIAM env knobs (labeled passes)" section (every cell from the same bench-up shares one running server container, so the dump is identical across that target/profile's scenarios) — omitted entirely for results trees predating G2 or for non-AXIAM targets.

12.4 New environment knobs (G2/H1 summary)

Variable Default Meaning
BENCH_SETTLE 1 0 skips the post-seed settle gate entirely.
BENCH_SETTLE_BURST_VUS 20 Concurrent workers in each settle-gate burst probe (H1).
BENCH_SETTLE_BURST_SECS 15 Seconds each burst probe attempt runs.
BENCH_SETTLE_PROBE_THR 400 Pass threshold, ops/s, under the burst concurrency (OR'd with the p50 threshold below). Also recorded per cell as settle_probe_thr.
BENCH_SETTLE_PROBE_P50_MS 150 Pass threshold, p50 latency (ms) under the burst concurrency.
BENCH_SETTLE_RETRY_SECS 30 Gap between failed probe attempts.
BENCH_SETTLE_TIMEOUT_SECS 600 Hard cap on the settle wait; the gate then warns, proceeds, and records settle_timeout: true (as of H1, report.py also refuses any such cell).
BENCH_SETTLE_DRAIN_SECS 5 Pause after the LAST burst probe (pass or timeout) before the cell starts. Found live: a burst worker still mid-request when its window ends can still be executing server-side behind the shared single DB connection; without this drain, the very first request of the next cell (the scenario's own login) can land behind that straggler traffic and come back malformed.
BENCH_SETTLE_STABLE_SECS / BENCH_SETTLE_MAX_MS 30 / 100 Superseded G2 serial-canary knobs — still parse without error, no longer consulted by the v2 burst gate.
BENCH_RUN_INDEX 1 Drives cell-order rotation; set per-repeat by bench-matrix (BENCH_RUN_INDEX=$i), defaults to 1 (natural order) for a manual bench-run.

None of these are just recipe variables (target=, profile=, ...) — set them as plain environment variables, e.g.:

BENCH_SETTLE_BURST_VUS=30 BENCH_SETTLE_PROBE_THR=500 \
  just target=axiam profile=p0-plaintext bench-run

13. Dry runs — rehearsing the matrix (bench-dry-run)

A full bench-matrix is hours long, and most of what breaks it is not a performance problem but a client-contract problem: a seeded confidential client the target rejects, a p3-mtls client certificate k6 cannot open, a gRPC scenario dialling plaintext into a TLS listener, a scenario silently filtered out because OAuth2 was never configured. None of these announce themselves until that cell's turn comes round — potentially an hour or more into a run that then has to be restarted from the top.

A dry run rehearses the matrix in minutes. It is deliberately the same code path, not a simplified one: same profile env, same seed env, same filter_scenarios + cell-order rotation, same k6 run invocation, same resource/host samplers, same meta.json. Only the measured window shrinks, and only the grading changes:

just targets="axiam keycloak zitadel" profiles="p0-plaintext p2-tls13 p3-mtls" bench-dry-run
just target=axiam profile=p3-mtls dry=1 bench-run     # a single cell

What a dry run asserts

Each cell is graded from the same --summary-export JSON report.py reads, against the harness's own metrics (scenarios/lib/metrics.js):

Verdict Meaning
PASS Every operation completed with the status the scenario expected.
WARN It ran, but the cell would not measure what it claims — bench_fallback > 0 (a fallback op was measured instead of the labelled one), or the resource sampler wrote no rows (the real matrix would record no container CPU/mem for this cell).
SKIP Filtered out before running, with the reason. An AXIAM-only/Zitadel-only skip is expected; an OAuth2 skip is usually not — it means the confidential client was never seeded.
FAIL k6 wrote no summary (it died at init — bad options, unreadable certs, a setup() that threw), or no operation completed at all, or any operation failed. Rate-limit rejections (429 / RESOURCE_EXHAUSTED) are called out by name, since a wrong rl= posture looks identical to a broken client.

Bring-up and seeding are graded too: a target that never becomes ready, or a bench-seed whose post-seed smoke checks fail, is recorded as a FAIL row for that cell. Unlike bench-matrix, the sweep does not abort on the first failure — one pass produces the whole fix list. The exit status is non-zero if anything failed.

What a dry run is not

A dry run is never a measurement. It skips the post-seed settle gate (§12), so it measures squarely inside the transient window where p95 legitimately blows past the 2000 ms production gate — failing on that would be a false alarm about the one property a dry run does not check. So the latency gate is relaxed to BENCH_DRY_MAX_P95_MS while correctness stays strict: one failed check fails the cell.

Its artifacts are therefore fenced off three ways: they carry "dry_run": true in meta.json, they default to results/dry-run/, and both report.py (which skips any cell whose meta carries the flag) and just bench-pack (which prunes the whole subtree) exclude them.

Variable Default Meaning
BENCH_DRY_VUS 2 VUs per dry-run cell — enough to exercise the per-VU cookie jar / gRPC dial.
BENCH_DRY_WARMUP 2s Ramp stage.
BENCH_DRY_DURATION 5s Measured stage.
BENCH_DRY_COOLDOWN 1s Drain stage.
BENCH_DRY_MAX_P95_MS 30000 Relaxed p95 threshold — see above.
BENCH_DRY_RUN_TSV <results>/dry-run.tsv Verdict ledger; bench-dry-run points every cell at one file and renders SUMMARY.md from it.

BENCH_SETTLE and BENCH_CELL_PAUSE default to 0 in a dry run but are still honoured if set, so BENCH_SETTLE=1 ... bench-dry-run can rehearse the gate itself.

Refresh capacity is coupled to the login limit (A5/J4, run 5)

token_refresh.js pre-mints one session per VU in setup(), paced inside BENCH_LOGIN_PER_MIN, and each VU then chains rotations off its own session for the rest of the run. On a rotation failure the VU backs off rather than re-logging in.

That last part is the fix, and it is worth being precise about why. Before it, a VU that lost its refresh token immediately tried to log in again. Under rl=prod the login ceiling is 10/min per IP and the whole k6 fleet is one IP, so the re-login was throttled, the VU retried, and the cell's error rate filled up with login rejections. Run 5 reported the rl-prod refresh cell at 516/s with 4.4 % errors (run 4: 2.4 %) — a number that described the login limiter, not the refresh endpoint.

setup() refuses to start rather than produce that shape again:

  • if the pool cannot be filled, the run fails with a message saying the pre-mint was itself throttled — a half-filled pool would measure the same confusion more quietly;
  • if pacing the pre-mint would take longer than BENCH_MAX_SETUP_SECS (default 300 s), the run fails rather than spending ten minutes in setup and reporting a cell that was mostly warm-up.

BENCH_LOGIN_PER_MIN is read from the ceiling the running container actually has (same approach as detect_rl_posture), not from a second copy of the number in the runner. Neutralized cells get 0, which means no pacing, so nothing outside rl=prod changes.

The coupling itself is real and must not be hidden

Pre-minting removes a harness artifact. It does not remove the underlying product property, which is this: in a deployment with short sessions, refresh capacity is gated by the login limit, because every session that expires costs a login to replace. An operator sizing login_per_min is also, implicitly, sizing how many concurrent sessions their fleet can sustain.

That belongs in docs/deployment/rate-limit-sizing.md as a sizing note, not in a harness workaround. What the harness must never do is misattribute it — reporting login throttling as a refresh-endpoint error rate is how a real coupling becomes an imaginary bug.

J3 — the Keycloak login story is frozen

Keycloak's login figure stands at 51/s, profile p2, 2 GiB, 2 of 3 runs valid and is not being re-litigated. The 4 GiB rescue attempt made things worse rather than better (PRIVATE_BENCH_ANALYSIS.md §1.3), so raising the container memory cap is not the knob.

One thing would change this, and only one: a sweep of Keycloak's own JAVA_OPTS_KC_HEAP (the JVM heap Keycloak sizes itself with — not the container memory limit, which is what the 4 GiB attempt raised). If that sweep can ride along with a matrix pass at negligible cost, run it. If it needs its own pass, do not: publishing "51/s, 2 of 3 valid" with the failed rescue attempt described honestly is a better competitive claim than an unbounded hunt for a better number on a competitor's behalf.