Skip to content

LocalGate: the tests.yml suites locally through Bobcat's Gate (Bobcat 1.0.0-rc.3) - #5726

Merged
jeremydmiller merged 2 commits into
masterfrom
local-gates/marten-local-gate
Oct 22, 2026
Merged

jeremydmiller merged 2 commits into
masterfrom
local-gates/marten-local-gate

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #5725

Stoat plan node: local-gates/marten-local-gate

./build.sh LocalGate runs the logical suites of the tests.yml matrix on one machine through the local commit-and-push Gate in Bobcat.Supervisor 1.0.0-rc.3 (bobcat#729). CI is unchanged.

What changed

Bobcat 1.0.0-rc.1 → 1.0.0-rc.3 (Directory.Packages.props)

  • In rc.2, Bobcat dropped its JasperFx dependency (bobcat#744), and the Bobcat.CritterStack vocabulary DaemonTests uses (CritterStackSpec, CritterStackScenario, EventStores) moved to WolverineFx.Bobcat. DaemonTests now references WolverineFx.Bobcat 6.51.2-alpha.1. That package is built on JasperFx/JasperFx.Events 2.91.0, the same version Marten builds against. It asks for Bobcat rc.2 and resolves rc.3.
  • build/build.csproj drops its JasperFx 2.90.2 VersionOverride. Bobcat.Supervisor no longer needs it.

The LocalGate target (build/LocalGate.cs)

  • Init and build happen once. LocalGate depends on Init once, preflights Postgres, and builds src/Marten.slnx -c Release once, covering both TFMs. In CI, each job runs compile-project, whose Init → Clean() wipes artifacts/test-ledger and rewrites AssemblyInfo.cs. The gate no longer repeats that once per unit.
  • Units. Each test project is one unit, on its newest TFM unless --framework all. On a first run the units go in CI's job-time order, longest first. After that, Bobcat orders them by .bobcat/gate-durations.json. One worker budget covers every unit.
  • Same supervisor as CI. The supervisor settings that runSupervised used to build inline now live in configureSupervisor (build/SupervisedTests.cs), and the gate passes it through .Supervise(...). That covers the opt-in [Trait("Retry","N")] policy (fresh-process retries), the retry budget, the stall report, the heartbeat and ledger-balanced lanes. EventSourcing and Linq keep their 4 workers. --disable-test-retry and --test-workers still apply.
  • Databases. Each unit gets its own lane databases through marten_testing_database, named marten_testing_<project>_<tfm>_w{n}. These never collide with each other, with CI-target lanes (marten_testing_w{n}) or with Weasel's lanes. Lane creation is serialized.
  • Exclusive tracks for fixed server-level names:
    • tenant-databases: Core, Daemon, MultiTenancy and Stress all create tenant1..6/database1.
    • shard-databases: MultiTenancy and TenantPartitionedEvents both use marten_shard_*.
    • default-database: EventSourcingTests' testing_projections hard-codes localhost:5432/marten_testing. Stress also holds it.
  • Connection-aware worker budget. The budget defaults to half the cores, but the gate lowers it to what the server's free max_connections can hold, at about 40 per worker, and logs the change. The first full run on the stock 100-connection container failed with hundreds of 53300: sorry, too many clients already. EventSourcingTests' four lanes alone peaked above 100. docker-compose.yml now sets max_connections=500. --gate-worker-budget overrides the budget.
  • Opt-in legs, not defaults (documented in CLAUDE.md):
    • --gate-include postgis|timescaledb|multihost|stress. Each one has a preflight. The MultiHost check logs in with the suite's credentials, so an unrelated container on 5440 fails it. The TimescaleDB check confirms the extension exists, because 5433 is also Wolverine's Postgres.
    • --gate-serializer Newtonsoft|SystemTextJson|split, where split uses CI's net9/net10 pairing.
    • --gate-database <conn>, used for the PG15 leg.
    • Also --gate-only, --gate-skip-build, --gate-timescale-database.
  • BOBCAT_RUN_TAG is the prefix, so each unit publishes as <tag>/<project>.<tfm>.
  • Ledgers. The retry ledger, run reports and folded test ledgers are written per unit after the gate finishes. Folded ledgers also go to .bobcat/test-ledger/, because Clean() empties artifacts/ (.bobcat/ is git-ignored).
  • Docs. The "Local gate" section of CLAUDE.md covers how to run the gate, each option and the measured wall-clock.

Gate report (this 18-core machine, net10.0, Postgres 17)

Final run on a 500-connection server (budget 9), built from the repo's own marten-postgresql image on port 5434 with max_connections=500, matching the new docker-compose.yml. I did not restart the shared 5432 container.

Gate: 19 unit(s), 9 worker(s) budgeted, 7m43s wall clock (13m11s one after the other)
  ✓ Marten.EntityFrameworkCore.Tests.net10.0       24s  60 passed
  ✓ ValueTypeTests.net10.0                         24s  343 passed
  ✓ ContainerScopedProjectionTests.net10.0        6.6s  80 passed
  ✓ Marten.PgVector.Tests.net10.0                  12s  105 passed
  ✓ PatchingTests.net10.0                         4.5s  125 passed, 1 skipped
  ✓ Marten.MemoryPack.Tests.net10.0               2.1s  14 passed
  ✓ Marten.AspNetCore.Testing.net10.0             3.4s  122 passed
  ✓ Marten.NodaTime.Testing.net10.0               1.8s  14 passed
  ✓ Marten.SourceGenerator.Tests.net10.0          1.9s  19 passed
  ✓ CompiledQueryTests.net10.0                    1.6s  9 passed
  ✓ ModularConfigTests.net10.0                    1.3s  5 passed
  ✓ Marten.Testing.net10.0                        1.1s  8 passed
  ✓ DocumentDbTests.net10.0                        44s  1150 passed, 1 skipped
  ✓ TenantPartitionedEventsTests.net10.0         1m33s  267 passed, 2 skipped
  ✓ EventSourcingTests.net10.0                   1m49s  2543 passed, 27 skipped, 4 worker(s)
  ✓ LinqTests.net10.0                              18s  1576 passed, 4 worker(s)
  ✓ DaemonTests.net10.0                          6m41s  378 passed
  ✓ MultiTenancyTests.net10.0                      23s  188 passed
  ✓ CoreTests.net10.0                              17s  820 passed, 8 skipped
Exit code 0: 19 clean, 0 passed on retry, 0 failed, 0 indeterminate, 0 not run.

End to end, ./build.sh LocalGate took 7m48s, including Init and the solution build. DaemonTests sets the wall clock. The 7m43s wall clock compares with 13m11s for the units run one after the other.

Run on the stock 100-connection marten-postgresql-1 on 5432 (the gate derived a budget of 2): 19/19 clean. Wall clock 11m27s against 14m10s one after the other; 11m30s end to end.

The MultiHost leg was not run locally because port 5440 is held by an unrelated container. TimescaleDB, PostGIS and the PG15 leg were also not exercised.

One intermittent failure, not caused by this change

LinqTests.Acceptance.using_explicit_sql_in_select_clauses.using_explicit_sql_to_scalar_value failed in 2 of 7 LinqTests runs inside the gate and passed in the other 5. It passed 3 times on a clean master worktree (4 lanes twice, 1 lane once).

The cause is in the test itself. It never saves its own users, so it passes only when no User named "Hank" is already in the shared IntegrationContext table. A probe on master showed that when a Hank does exist, ExplicitSql<string>("d.data -> 'FirstName'") returns the JSON-quoted "Hank", so the assertion fails. Whether it fails depends on which classes share its worker and the order they run in, not on the gate. I left it alone here.

Follow-up (not done here)

  • Move the tests.yml include: matrix to a generated build/gates.json and feed CI with fromJSON, so CI and LocalGate read one list.
  • Add markdownlint/cspell targets as a local twin of docs-prs.yml.

🤖 Generated with Claude Code

jeremydmiller and others added 2 commits October 22, 2026 12:43
rc.2 dropped Bobcat's JasperFx dependency (bobcat#744) and moved the Critter Stack
vocabulary DaemonTests uses (CritterStackSpec, CritterStackScenario, EventStores in
Bobcat.CritterStack) to WolverineFx.Bobcat, so DaemonTests takes WolverineFx.Bobcat
6.51.2-alpha.1, built on the same JasperFx/JasperFx.Events 2.91.0 Marten builds against.
The build project no longer needs its JasperFx 2.90.2 override.

Part of #5725.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
./build.sh LocalGate inits once, preflights Postgres, builds src/Marten.slnx -c Release
once, then runs every test project as a unit through Bobcat.Supervisor's Gate: newest
TFM unless --framework all, longest first, one worker budget across all units.

- Each unit keeps its CI target's supervisor configuration, now shared through
  configureSupervisor (opt-in Retry traits in a fresh process, stall report, heartbeat,
  ledger-balanced lanes) and its committed worker count.
- Each unit gets its own lane databases, marten_testing_<project>_<tfm>_w{n}, through
  marten_testing_database.
- Exclusive tracks keep suites that use fixed server-level names apart:
  tenant-databases (Core, Daemon, MultiTenancy, Stress), shard-databases
  (MultiTenancy, TenantPartitionedEvents) and default-database (EventSourcingTests'
  hard-coded localhost:5432/marten_testing).
- The budget is half the cores, lowered to what the server's free connections hold
  (about 40 per worker); a stock 100-connection Postgres ran out with 53300 under four
  EventSourcingTests lanes. docker-compose.yml now sets max_connections=500.
- Opt-in: --gate-include postgis|timescaledb|multihost|stress, --gate-serializer
  (Newtonsoft, SystemTextJson, split), --gate-database (the PG15 leg). Also
  --gate-only, --gate-skip-build, --gate-worker-budget.
- BOBCAT_RUN_TAG is the prefix each unit publishes under (<tag>/<project>.<tfm>).
- Durations and folded test ledgers persist in .bobcat/ (ignored), since Init's
  Clean() empties artifacts/.

CI is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit c899546 into master Oct 22, 2026
32 checks passed
@jeremydmiller
jeremydmiller deleted the local-gates/marten-local-gate branch October 22, 2026 18:03
@jeremydmiller jeremydmiller mentioned this pull request Oct 25, 2026
jeremydmiller added a commit to vpetrevski/marten that referenced this pull request Oct 26, 2026
Moves BobcatVersion from 1.0.0-rc.3 to 1.0.0-rc.4, the candidate before Bobcat 1.0.0 (JasperFx/bobcat#568).
rc.4 fixes the gate's default build passing -f to the whole solution (JasperFx/bobcat#774, found here
in JasperFx#5726), so Marten's GateDeclaration.Build workaround can go in a follow-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDKQhnBVcmVLCTkJbVeADk
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LocalGate: the full tests.yml matrix locally through Bobcat's Gate, one build, per-gate databases

1 participant