[jeremydmiller]

JasperFx/bobcat#562: Bobcat applied to src/DaemonTests, on Bobcat 1.0.0-alpha.1, ready for review. A comment on bobcat#562 planned to wait for 1.0.0-rc.1; Jeremy asked for it to be done now on alpha.1, so this pins 1.0.0-alpha.1 from nuget.org.

#5363 had already rendered two classes on 0.27.3 through decorated DaemonContext helpers. Nothing in DaemonTests used the Gherkin lane, so the 0.50.0 removal (bobcat#471) cost nothing here. This PR moves that work to 1.0, routes the projection waits through Bobcat's catch-up diagnosis, adds CritterStackSpec specifications, and renders three more daemon-behaviour classes.

What changed

Packages. Bobcat, Bobcat.Xunit, Bobcat.Generators (0.27.3) and Bobcat.Supervisor (0.60.4) all move to 1.0.0-alpha.1 through a single BobcatVersion property in Directory.Packages.props.

xUnit / MTP alignment. They can't line up repo-wide in this PR, so here is what was done:

	Before	After
DaemonTests, DaemonTests.ManualOnly	xunit.v3 3.2.2 / MTP 1.9	xunit.v3 4.0.2, xunit.runner.visualstudio 4.0.1 / MTP 2.5.1 (via VersionOverride)
The other 23 test projects	xunit.v3 3.2.2 / MTP 1.9	unchanged

• Bobcat.Xunit 1.0 requires xunit.v3 4.x on MTP 2.5.1, and the two xunit lines can't share a test project. Moving all 25 projects is its own change: 14 AssemblyInfo.cs files use CollectionBehavior(DisableTestParallelization = true), which is a compile error on 4.x. DaemonTests now uses [assembly: Xunit.v3.Parallelization(Mode = ParallelMode.None)].
• CoreTests had a ProjectReference to DaemonTests. It was used only for a transitive Lamar, and it would have dragged xunit 4 into CoreTests. I removed it and added Lamar directly. CoreTests builds on both TFMs and runs 826 total, 0 failed (net10.0).
• JasperFx. Bobcat 1.0 requires JasperFx ≥ 2.90.2; Marten ships against 2.90.1. DaemonTests resolves 2.90.2 transitively. build/ takes 2.90.2 via VersionOverride, because otherwise NU1605 blocks the supervisor. Marten's own JasperFx floor is unchanged. Moving it is your call.
• dotnet test on an MTP 2.x host needs global.json → "test": {"runner": "Microsoft.Testing.Platform"}, which is a repo-wide switch. I didn't make it. DaemonTests runs through ./build.sh test-daemon (the supervisor) or its built executable, which is how CI runs it already.

Rendered specifications.

• DaemonContext's helpers now use [Given] / [When] / [Then].
• [BobcatScenario] is gone, because Bobcat.Xunit now records every [BobcatFeature] class.
• <BobcatProjectAssertions> is on, so statement-level Shouldly checks in feature classes render as Then steps.
• Newly rendered: when_skipping_events_in_daemon, catching_up_mode_for_projections_and_subscriptions and rebuild_mode_start_resumes_after_a_failure, alongside the existing build_aggregate_projection and rebuilding_an_inline_projection.
• Test bodies changed only where the rendering needed it:
  • waiter-before-act shard waits now go through WaitForShard where they are awaited; WaitForShardState completes from the snapshot, per Unify aggregator lookup in ProjectionCollection to use the same aggre… #568;
  • marker comments added;
  • Marten's ShouldHaveTheSameElementsAs became Shouldly's ordered ShouldBe in the one rendered class;
  • one per-event loop of assertions became a single assertion.

CritterStackSpec. Specifications/async_daemon_specifications.cs specifies three daemon behaviours: catch-up after an append, delete on cancel, and rebuild through the running daemon. They run against a hosted app with AddAsyncDaemon(DaemonMode.Solo), and every read opens with Bobcat's own catch-up step.

Catch-up diagnostics. Two new DaemonContext helpers, WaitForCatchUp(readModel) and WaitForShard(daemon, shard, sequence, readModel). On a timeout they throw ProjectionCatchUpTimeoutException with the progress list from EventStores.ProjectionsBehindAsync, instead of a bare TimeoutException. catch_up_timeout_diagnostics.cs pins that diagnosis through both CritterStackScenario and DaemonContext.

Supervisor. The build compiles against Supervisor 1.0.0-alpha.1. Stall detection was already on for every suite: report-only at 5 minutes, with a 30s heartbeat. Across these runs it named nothing as stalled. The longest DaemonTests are the Bug_5567 hot/cold cases at about 60s each.

A rendered spec

From ./DaemonTests --filter-class …, written with BOBCAT_SPEC_CONSOLE_FILE (see JasperFx/bobcat#676):

Feature: The async daemon keeps read models current
════════════════════════════════════════════════════

  an async projection catches up with an appended event OK
  ────────────────────────────────────────────────────────────
    ✓ Given ShipmentSummary #id has already recorded ShipmentPacked(Id: #id, Items: 3) (9ms)
    ✓ When  the shipment is shipped (14ms)
    ✓ Then  ShipmentShipped is emitted
╭───┬─────────────────┬─────────┬────────╮
│ # │ event           │ values  │ Status │
├───┼─────────────────┼─────────┼────────┤
│ 1 │ ShipmentShipped │ Id: #id │   OK   │
╰───┴─────────────────┴─────────┴────────╯
    ✓ And   the ShipmentSummary read model has caught up (766ms)
    ✓ And   the ShipmentSummary matches (11ms)
╭───┬───────┬─────────┬────────╮
│ # │ Items │ Shipped │ Status │
├───┼───────┼─────────┼────────┤
│ 1 │ 3     │ True    │   OK   │
╰───┴───────┴─────────┴────────╯

Feature: Building an aggregate projection
══════════════════════════════════════════

  rebuild the projection OK
  ────────────────────────────────
    ✓ When  the projection daemon is running (120ms)
    ✓ Given the events are published (67ms)
    ✓ Then  the TripCustomName:All shard has reached sequence 231 (248ms)
    ✓ When  the TripCustomName projection is rebuilt (25ms)
    ✓ Then  every expected aggregate matches the actual (21ms)

Feature: Skipping events that fail to apply
════════════════════════════════════════════

  the shards should still be running OK
  ────────────────────────────────────────────
    ✓ Given the names are published to a running daemon, the bad ones failing to apply (314ms)
      ✓ When  the projection daemon is running (23ms)
      ✓ Then  the CollateNames:All shard has reached sequence 15 (40ms)
      ✓ And   the NamedDocuments:All shard has reached sequence 15 (6ms)
    ✓ Then  every shard is still running (2ms)
      ✓ And   shard.Status should be AgentStatus.Running (1ms)
      ✓ And   shard.Status should be AgentStatus.Running

A deliberately forced catch-up timeout

A throwaway spec, not committed, read a ShipmentSummary from a host with the daemon disabled and ProjectionTimeout = 5s:

  a read model with no daemon running FAILED
  ─────────────────────────────────────────────
    ✓ Given ShipmentSummary #id has already recorded ShipmentPacked(Id: #id, Items: 3) (361ms)
    ! Then  the ShipmentSummary read model has caught up (5172ms)
      ProjectionCatchUpTimeoutException — see below

    ProjectionCatchUpTimeoutException: Waiting for 'the ShipmentSummary read model has caught up' timed out
    after 5s, before reading the ShipmentSummary with id 93e943e8-…. Behind: 'ShipmentSummary:All' at 0 of 1.
    This is the environment (a daemon behind, stopped or not running), not evidence about the ShipmentSummary
    itself. The store said: TimeoutException: The projection shards (in total of 2) haven't been completely
    started within the timeout span

And through DaemonContext.WaitForShard, from the committed waiting_on_a_shard_that_never_starts_names_where_every_shard_stood:

Waiting for 'the TripCustomName:All shard has reached sequence 43' timed out after 2s, before reading the Trip.
Behind: 'TripCustomName:All' at 0 of 43. This is the environment (a daemon behind, stopped or not running), not
evidence about the Trip itself. The store said: TimeoutException: Shard TripCustomName:All did not reach
sequence number 43 in the time allowed

The diagnosis also found a real bug in this PR before review. The first two full gate runs failed the new specs with Behind: 'ShipmentSummary:All' at 0 of 4, which meant the hosted daemon had never run its shard. The cause was that the fixture cleaned event data after the Solo daemon started. That restarted the sequence underneath a shard that had already loaded its old progress, so the shard sat "ahead" of every new event. Isolated runs won that race and full runs lost it. A 3s delay before the clean reproduced it deterministically. The fixture now cleans before StartAsync.

Two things the first CI run found (second commit)

Bobcat's diagnosis named both of them on the first CI run:

• A Marten fix outside DaemonTests: WaitForNonStaleProjectionDataAsync now always throws the TimeoutException it documents.
  • Before, a timeout that landed inside its first round trip escaped as some other exception. That happens on a cold store's first storage check or a slow connect. It surfaced as a TaskCanceledException, an OperationCanceledException, or a MartenSchemaException/NpgsqlException wrapping one.
  • Bobcat only diagnoses TimeoutException. So on the slower CI runner, the forced timeout in catch_up_timeout_diagnostics showed up as a bare TaskCanceledException.
  • Both phases of the wait now report a cancellation by their own timeout as TimeoutException.
  • wait_for_non_stale_data_error_cases pins it on a cold store at 1, 20 and 100ms. Without the fix, all three fail.
  • This is product code, in a separate commit (e7f96db), if you'd rather take it on its own.
• Question for you: a rebuild through a running hosted daemon leaves the shard stopped.
  • On CI, two specs failed with 'ShipmentSummary:All' at 2 of 4.
  • The cause: JasperFx's rebuildProjectionCore stops the continuous agent, rebuilds with dedicated agents and disposes them, but never starts the continuous agent again. So every spec that ran after the rebuild spec on that host waited on a stopped shard. Locally it happened to run last.
  • The rebuild spec now has a host of its own.
  • Whether the Solo coordinator should resume the shard after RebuildProjectionAsync is your call. I left Marten and JasperFx behaviour as is.

What I ran

Local Postgres 16 (Host=localhost;Port=5432;Database=marten_testing;Username=postgres;password=postgres), .NET SDK 10.0.112. Only the net10.0 runtime was available here, so net9.0 was built but not run.

Run	Result
./build.sh test-daemon --framework net10.0 --configuration Release --disable-test-retry (1 lane)	374/374 passed, 0 retries, 7m09s; after the second commit 377/377
same, --test-workers 2 (a database per lane), twice	374/374 passed both times, 5m08s / 5m10s
the 33 tests in the converted classes, run directly	33/33, three runs in a row
CoreTests (net10.0), after losing its DaemonTests reference	826 total, 0 failed
dotnet build -c Release, DaemonTests / DaemonTests.ManualOnly / CoreTests / build	clean, both TFMs (no CI job builds ManualOnly)

No test was skipped or disabled, and no Retry trait was added.

For you to decide: two green local runs at two lanes saved about 2 minutes. I left CI at one worker and updated the comments in tests.yml and CLAUDE.md to record the measurement. If you want it, it's workers: 2 on TestDaemon plus a committed ledger from --update-test-ledger.

Upstream Bobcat issues

Filed on JasperFx/bobcat (by the coordinating session, from drafts written here). Each links back to bobcat#562.

1. BOBCAT_SPEC_CONSOLE=1 prints nothing when stdout is redirected, despite the docs bobcat#676: BOBCAT_SPEC_CONSOLE=1 prints nothing when stdout is redirected (xunit.v3 4.0.2 / MTP 2.5.1). It works in a TTY and with BOBCAT_SPEC_CONSOLE_FILE, but not when stdout is piped or captured, which is the CI case. This was already filed; this PR's evidence was added there as a comment.
2. The projection catch-up diagnosis is not reusable outside CritterStackScenario bobcat#685: the catch-up diagnosis isn't reusable outside CritterStackScenario. There's no public "wait and diagnose" helper, ProjectionCatchUpTimeoutException requires a read type even for shard waits, and AfterAsyncDaemonsCatchUpAsync throws a bare TimeoutException.
3. ProjectionsBehindAsync: say why the store "could not say", and what M in "at N of M" is bobcat#686: ProjectionsBehindAsync swallows why the store "could not say". It also doesn't document that of M is the highest sequence, not an event count.
4. A multi-line marker comment renders only its first line, mid-sentence, with no diagnostic bobcat#687: a multi-line marker comment renders only its first line, mid-sentence, with no diagnostic.
5. The console draws a "declared" marker step with the same ○ as a step that never ran bobcat#688: a "declared" marker step renders with the same ○ as a step that never ran, so a green scenario looks partly skipped.
6. Projected assertions in a loop render one step per iteration, and an untaken branch shows as "not run" bobcat#689: projected assertions in a loop render one step per iteration, and an untaken branch renders as "not run" in a green scenario.
7. Adopting Bobcat 1.0 in one project of a large repo: the xunit.v3 4.x and JasperFx floors bobcat#690: migration pain from 0.27.3 in a large repo. There's no guidance for adopting xunit 4 in one project, Bobcat.Supervisor carries Bobcat's JasperFx floor into a build tool, and global.json is needed for dotnet test.
8. BobcatProjectAssertions: a multi-line assertion subject breaks the build (unescaped newline in a generated string literal) bobcat#684, build-breaking: with BobcatProjectAssertions on, a multi-line assertion subject is written into a generated string literal with its newline unescaped, which produces 58 compile errors in generated files. Worked around here with a local.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GmWSVvZAZZSYUkzyLFpi9N

[jeremydmiller]

Linq (net9, PG15, Newtonsoft) failed at package restore, before any test ran:

build/build.csproj : error NU1102: Unable to find package Bobcat with version (>= 1.0.0-alpha.1)
  - Found 56 version(s) in nuget.org [ Nearest version: 0.60.4 ]

This is a stale nuget.org view on that one runner, not this change. Bobcat 1.0.0-alpha.1 was published and listed at 01:13 UTC, and nuget.org now returns 57 versions. Every other job restored the same build/ project and moved on to building and testing. No fix is needed in the PR. I'll re-run the failed job once, after the rest of the run finishes; GitHub won't re-run a single job while its run is still going. If it fails again I'll treat it as a real failure.

---

Generated by Claude Code