Skip to content

README.md

Latest commit

 

History

History
319 lines (263 loc) · 16.8 KB

README.md

File metadata and controls

319 lines (263 loc) · 16.8 KB
Raw

AXIAM Benchmark Framework

A vendor-neutral, protocol-driven benchmark harness for comparing AXIAM against other open-source IAM systems (Keycloak, Zitadel, Authentik, Ory, …) across three dimensions:

  1. Performance — throughput (req/s) and latency (p50/p95/p99) under load.
  2. Resource efficiency — CPU and memory consumed to deliver that performance, so we can answer "can AXIAM match the competition with a smaller footprint?"
  3. Security posture — the same workload is replayed across a matrix of security profiles (from plaintext HTTP up to mTLS with client-certificate authentication and TLS 1.3-only), measuring the cost of stronger security.

It also includes per-SDK client-side benchmarks, so the client overhead of each official AXIAM SDK (Rust, TypeScript, Python, Java, C#, PHP, Go — each published from its own ilpanich/axiam-<lang>-sdk repository) can be measured against the raw protocol baseline. All 7 SDKs are implemented and all 7 bench harnesses are wired to their SDKs (see sdk/README.md); each builds against the sibling SDK checkout via a local path/replace/project reference until the alpha package lands on the public registry.

Why a custom framework?

There is no vendor-neutral standard benchmark for IAM systems. The de-facto reference is keycloak-benchmark (Gatling-based), but it is Keycloak-specific in its provisioning, dataset add-on, and endpoint assumptions.

AXIAM and every serious competitor speak the same wire standards — OAuth2 (RFC 6749), OIDC, token introspection (RFC 7662), JWKS (RFC 7517). So instead of re-implementing a vendor-coupled tool, this framework drives those standard flows through a thin per-target adapter layer (scenarios/lib/targets.js). Every target is hit with the identical logical workload; only the endpoint paths and request encodings differ. That keeps the comparison apples-to-apples.

The load generator is k6: a single static binary, scriptable in JavaScript, with native HTTP + gRPC support, built-in latency/throughput metrics, threshold gating, and machine-readable JSON output. It is deliberately lighter than a JVM-based generator (Gatling) so the load tool does not starve the system-under-test of the CPU we are trying to measure.

Directory layout

benchmarks/
├── README.md                 # this file
├── justfile                  # convenience commands (bench-up, bench-run, bench-report…)
├── run-improvement-tasks.sh  # per-task data collection for the pre-MVP plan (see below)
├── run-memory-experiment.sh  # B1/D9 allocator A/B (default malloc vs jemalloc)
├── docs/
│   ├── methodology.md        # how a fair run is defined; metric definitions
│   ├── security-profiles.md  # the TLS/cert profile matrix
│   └── interpreting-results.md
├── targets/                  # each system-under-test as a resource-capped compose file
│   ├── axiam/docker-compose.yml
│   ├── keycloak/docker-compose.yml
│   └── zitadel/docker-compose.yml
├── profiles/                 # security profiles (env files + registry)
│   ├── profiles.yaml
│   ├── p0-plaintext.env
│   ├── p1-tls12.env
│   ├── p2-tls13.env
│   └── p3-mtls.env
├── scenarios/                # k6 load scenarios (vendor-neutral)
│   ├── lib/{config,targets,metrics,auth}.js
│   ├── oauth2_password_login.js
│   ├── oauth2_client_credentials.js
│   ├── token_introspection.js
│   ├── token_refresh.js
│   ├── jwks_fetch.js
│   ├── userinfo.js
│   ├── authz_check_rest.js     # AXIAM-only; SDK check_access wire baseline
│   ├── authz_batch_rest.js     # AXIAM-only; SDK batch_check wire baseline
│   ├── authz_check_grpc.js
│   └── authz_batch_grpc.js
├── resource/                 # resource-consumption sampling
│   ├── sampler.sh            # docker stats → CSV
│   └── cadvisor-compose.yml  # optional richer telemetry
├── runner/
│   ├── run-benchmark.sh      # orchestrator: target × profile × scenario matrix
│   ├── seed.sh               # provision org/tenant/user/client per target
│   └── report.py             # aggregate raw results → comparative report
├── sdk/                      # per-language SDK client-side benchmarks (scaffolds)
│   ├── HARNESS-SPEC.md       # the JSON contract every SDK bench must emit
│   ├── run-all.sh
│   └── {rust,typescript,python,go,java,csharp,php}/
└── results/                  # run outputs (gitignored)

Quick start

# 0. Prerequisites: docker, docker compose, k6, python3, jq, openssl, bash.
cd benchmarks

# The AXIAM target needs a JWT keypair + DB/RabbitMQ creds. `bench-up` bootstraps
# throwaway local-only ones under docker/.secrets/ automatically (or reuses
# docker/.secrets/env if you provide real ones). By default it pulls the prebuilt
# server image ghcr.io/ilpanich/axiam/server:<version>; GHCR packages are private
# by default, so run `docker login ghcr.io` first (PAT with read:packages), set
# BENCH_AXIAM_IMAGE to an image you can pull, or build from source with build=1.

# NOTE: `just` variable overrides (target=…, profile=…) must come BEFORE the
# recipe name — placed after, `just` reads them as another recipe and errors
# with "justfile does not contain recipe `target=…`".

# 1. Bring up a target under a chosen security profile and seed it.
#    AXIAM uses the published ghcr image by default (no source build); pin a
#    different tag with BENCH_AXIAM_IMAGE, or force a local build with build=1.
just target=axiam profile=p2-tls13 bench-up            # prebuilt image
# just target=axiam profile=p2-tls13 build=1 bench-up  # local source build
just target=axiam bench-seed

# 2. Run the full scenario suite (load + resource sampling) for that target/profile.
just target=axiam profile=p2-tls13 bench-run

# 3. Repeat for a competitor.
just target=keycloak profile=p2-tls13 bench-up
just target=keycloak bench-seed
just target=keycloak profile=p2-tls13 bench-run

# 4. Generate a comparative report across everything in results/.
just bench-report

# 5. Tear down.
just target=axiam bench-down
just target=keycloak bench-down

Or run the entire matrix (all targets × all profiles × all scenarios) unattended. By default this repeats the whole matrix 3× (repeat := "3") into results/run-1/, results/run-2/, results/run-3/, and report.py medians each cell across the valid runs (see methodology §8); pass repeat=1 for a single quick pass instead:

just targets="axiam keycloak" profiles="p0-plaintext p2-tls13 p3-mtls" bench-matrix
# just repeat=1 targets="axiam keycloak" profiles="p0-plaintext p2-tls13 p3-mtls" bench-matrix

Rehearse the matrix first (bench-dry-run)

A full matrix is hours long, and a break in the k6 client contract — a seeded client the target rejects, a p3-mtls cert k6 cannot load, a gRPC dial into a TLS listener, an OAuth2 cell that gets silently skipped — does not announce itself until that cell's turn comes round. bench-dry-run walks the same target × profile grid with the same bring-up → seed → run → tear-down path and the same scenarios, but collapses each measured window to a few seconds and grades every cell on whether the k6 client could connect, send its request and get back the answer the scenario expects:

just targets="axiam keycloak zitadel" profiles="p0-plaintext p2-tls13 p3-mtls" bench-dry-run

It does not stop at the first broken cell — a failing bring-up, seed or scenario is recorded and the sweep carries on, so one pass gives you the whole fix list. Verdicts are PASS / WARN (ran, but the cell would not measure what it claims — e.g. a fallback op, or a sampler writing no rows) / SKIP (filtered out, with the reason) / FAIL. The exit status is non-zero if anything failed, and a table lands in results/dry-run/SUMMARY.md alongside per-cell *.dryrun.log files holding k6's own check breakdown.

Because it deliberately skips the post-seed settle gate, a dry run measures inside the transient window — so it relaxes the p95 latency gate to 30s while keeping correctness strict (a single failed check fails the cell). A dry run is never a measurement. Its artifacts carry "dry_run": true, live under results/dry-run/, and are excluded from both bench-report and bench-pack.

dry=1 applies the same treatment to a single cell:

just target=axiam profile=p3-mtls dry=1 bench-run

See docs/methodology.md for the rules that make a run comparable, and docs/security-profiles.md for the profile definitions.

Running the matrix on a laptop rather than dedicated hardware? See "Running on a laptop" in the methodology doc for the variance-control runbook (AC power, CPU governor, turbo-boost mode, cooldown pauses between cells) before trusting absolute numbers.

For a repeatable, noise-resistant run, use repeat=N (default 3) so bench-matrix runs the whole matrix N times and report.py medians each cell — see §8 "Multiple runs — median-of-N":

just repeat=3 targets="axiam keycloak" profiles="p0-plaintext p2-tls13" bench-matrix

Sharing a run (bench-pack) — what the archive contains

just bench-pack writes results-<date>.tar.xz. Its manifest is produced by runner/pack-filelist.sh — one script, so the archive and its regression test can never disagree:

Included (by extension, anywhere under results/) Why
*.json k6 summaries (*.k6.json) and per-cell run metadata (*.meta.json)
*.csv / *.tsv container resource samples (*.res.csv) and host telemetry (*.host.csv)
*.md the generated report.md, plus every investigation verdict — rl-prod-summary.md, sdk-report.md, targeted-run write-ups
*.log / *.txt investigation captures — nsenter.log socket snapshots, h5-revocation.log, docker-log excerpts

Excluded by construction (pruned, not filtered afterwards):

  • results/dry-run/** — a dry run writes real-looking artifacts for five-second unsettled windows. They are diagnostics, never measurements.
  • anything matching *seed* — client secrets and the bench user's password. These normally live in .seed/ (outside results/ entirely); the prune also covers the legacy results/<target>.seed.env path run-benchmark.sh still honours.

The include list is by extension on purpose. Run 5 shipped an archive that dropped every investigation artifact because the list named five exact filenames and could not anticipate the sixth (J13). Matching extensions means the next investigation's artifact is packed by default rather than found missing after the fact.

just bench-pack-selftest   # hermetic; no docker, no k6. Asserts both directions.

The self-test builds a fixture tree carrying one of every run-5 artifact shape, runs the real selection script over it, and fails if an investigation artifact is missing or if a dry-run/seed path survives. After packing, bench-pack additionally re-scans the finished archive for SECRET/PASSWORD content (ignoring the "<redacted>" key names axiam_env legitimately records).

Status of components

Component State
k6 protocol scenarios Implemented (HTTP); authz check + batch scenarios over both REST and gRPC
AXIAM target + seeding Implemented (prebuilt ghcr image by default, local build fallback); seeds org/tenant/admin via the gated bootstrap flow plus a resource/role/grant for authz checks
Keycloak / Zitadel targets Implemented (Keycloak 26.7.0, Zitadel v4.15.2)
Security profile matrix Implemented (p0–p3); mTLS requires per-target cert wiring; SDK benches cover p0–p2 (no SDK client-cert option yet)
Resource sampler + report Implemented (stdlib python, no external deps)
SDK client benchmarks All 7 wired to their SDKs (see sdk/README.md)
AMQP async-authz benchmarking Out of scope for v1.0-beta (see below)

Every SDK bench builds against its sibling ilpanich/axiam-<lang>-sdk checkout via a local path/replace/project reference until the alpha package is published — see each language's sdk/<lang>/TODO.md. sdk/HARNESS-SPEC.md documents the shared result contract every bench emits. As of run 3 no SDK bench has yet produced a validated status: "ok" record against a live target — that is tracked as task G10 below.

Targeted investigation runs

Two standalone scripts sit alongside the matrix for the open questions from run 3. They are separate from bench-matrix on purpose: each answers ONE question, writes its own verdict, and can be run in isolation.

Script Use it for
./run-improvement-tasks.sh <task> One subcommand per task in claude_dev/improvement-after-serious-benchmark.md that needs a live run. ./run-improvement-tasks.sh list prints the tasks with time estimates.
./run-memory-experiment.sh [a|b|both] The B1/D9 memory-retention A/B: builds a default-allocator and a jemalloc image, drives a login burst, watches RSS for 10 minutes per variant, and writes results/d9-summary.md.

Each run-improvement-tasks.sh task writes results/tasks/<task>/SUMMARY.md containing the measured numbers and the plan's acceptance criterion, so a task can be closed (or a follow-up opened) from its summary alone. Start with g1-timeline: run 3 found that for ~5–7 minutes after seeding, the AXIAM stack serves everything at ~45 req/s with the datastore pinned at ~1 core, which silently corrupted every cell that ran first after a seed — see PRIVATE_BENCH_ANALYSIS.md §1.

Seed-size sensitivity (bench-bulk-seed)

Every published AXIAM number so far was measured against a fixture of one tenant, two users, one resource and the ~100 built-in registry permissions. The obvious reader question — does the check path hold at 100 000 users and a four-deep resource tree? — had no answer in the archive (J12). It does now:

just target=axiam profile=p2-tls13 bench-up
just target=axiam bench-seed          # the functional fixture, via the REST API
just scale=10 bench-bulk-seed         # 10 000 users, 2 000 resources, depth 4
just target=axiam profile=p2-tls13 bench-run

runner/bulk-seed.sh writes SurrealQL directly into the datastore in batched transactions (--batch, default 1 000 statements per BEGIN/COMMIT) — provisioning 100 000 users through the REST API would mean 100 000 Argon2id hashes and is not a thing anyone waits for. Three properties make the resulting cell comparable:

  • The functional fixture is untouched. benchuser, bench-resource and bench-reader keep their ids, so the scenario runs the same logical query against a bigger index. That is the only comparison worth making.
  • The tree is deep, not just wide. --depth (default 4) and --fanout (6) build a balanced hierarchy, because a flat 10 000-resource fixture exercises the ancestor-walk code exactly as hard as a 1-resource one.
  • Some grants are denies. --deny-ratio (default 0.05) writes a fraction of grants as effect: deny, so the cell measures B1's deny-override path rather than only its no-denies short-circuit. Set 0 to measure the cheap path exclusively.

Bulk users cannot authenticate by construction — their password_hash is a sentinel string that is not an Argon2id encoded hash at all, so verification fails to parse. The fixture adds volume; it does not add usable credentials.

The resulting scale is recorded in .seed/axiam.bulk.env and lands in every cell's meta.json as seed_scale / seed_fixture, so a 10× measurement can never be mistaken for a base-fixture one. Absent file means scale 1.

just bench-bulk-verify prints the row counts without writing anything.

Out of scope (v1.0-beta)

AMQP async-authz benchmarking (server axiam-amqp + the Go/Python/TypeScript SDKs' AMQP modules) is deliberately deferred. k6 has no AMQP executor/protocol plugin, so measuring the async-authz-over-AMQP flow needs a custom load harness (publish decision requests, consume results, measure end-to-end latency and consumer throughput) rather than a k6 scenario. This is planned as a follow-up, not part of the current scenarios//sdk/ frameworks.