Skip to content

High-water allocation fence: read is_called so mark 0 is fenceable (#5091) #5092

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeremydmiller merged 2 commits into from
Jul 30, 2026
+349 −31
Merged

High-water allocation fence: read is_called so mark 0 is fenceable (#5091) #5092

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f4ee5c6
fix(#5091): fence the allocation history on allocated, not reserved, …
jeremydmiller Jul 30, 2026
a0e16c5
Adopt JasperFx 2.36.3 and Weasel 9.20.2 (the #5090 chain)
jeremydmiller Jul 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 33 additions & 9 deletions Directory.Packages.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,14 +157,21 @@
disposed daemon, ProjectionCoordinatorBase.PauseAsync logs ObjectDisposedException at Debug
instead of Error, and ProjectionCoordinatorBase.StopAsync calls the new abstract
ClearResolvedDaemons() seam after disposing daemons so subclass caches drop the disposed
instances (implemented here on ProjectionCoordinator + ExplicitProjectionCoordinator). -->
<PackageVersion Include="JasperFx" Version="2.36.2" />
<PackageVersion Include="JasperFx.Events" Version="2.36.2" />
<PackageVersion Include="JasperFx.Events.SourceGenerator" Version="2.36.2">
instances (implemented here on ProjectionCoordinator + ExplicitProjectionCoordinator).
JasperFx 2.36.3: jasperfx#592/#593 (marten#5090) — ProjectionCoordinatorBase.StartAsync now
cancels AND drains any existing leadership loop instead of disposing the old
CancellationTokenSource without cancelling it. Disposing a CTS does not cancel it and _runner
was overwritten, so a ResumeAsync not preceded by PauseAsync orphaned a live executeAsync for
the rest of the process — one that re-attained the leadership lock right after StopAsync
released it, and that won an advisory-lock handle after the lock had been disposed. Pairs
with Weasel 9.20.1 (weasel#396), which is what makes that stranded handle impossible. -->
<PackageVersion Include="JasperFx" Version="2.36.3" />
<PackageVersion Include="JasperFx.Events" Version="2.36.3" />
<PackageVersion Include="JasperFx.Events.SourceGenerator" Version="2.36.3">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageVersion>
<PackageVersion Include="JasperFx.SourceGenerator" Version="2.36.2" />
<PackageVersion Include="JasperFx.SourceGenerator" Version="2.36.3" />
<PackageVersion Include="Jil" Version="3.0.0-alpha2" />
<PackageVersion Include="Lamar" Version="7.1.1" />
<PackageVersion Include="Lamar.Microsoft.DependencyInjection" Version="15.0.0" />
Expand Down Expand Up @@ -215,7 +222,7 @@
<PackageVersion Include="Vogen" Version="7.0.0" />
<!-- 9.16.2: the hold at 9.2.1 (were newer builds actually being published?) is resolved —
Weasel.EntityFrameworkCore is on nuget.org, so it tracks the rest of the Weasel line. -->
<PackageVersion Include="Weasel.EntityFrameworkCore" Version="9.17.0" />
<PackageVersion Include="Weasel.EntityFrameworkCore" Version="9.20.2" />
<!-- Weasel.Postgresql 9.1.x: managed LIST partitions under multi-database (sharded) apply.
9.1.2 (marten#4706): managed-partition tables are no longer destructively rebuilt — the
out-of-band AddPartitionToAllTables path creates partitions while the generic schema diff
Expand Down Expand Up @@ -264,9 +271,26 @@
9.16.4 (weasel#356/#359): db-apply bounds its connection usage over many databases and extends the
pool-release/retry to SqlServer, MySql and Oracle.
9.17.0 (weasel#360): archive/tombstone/progression seams added to Weasel.Storage.Events. Additive for
Marten — bumped to stay current on the Weasel line (compile-verified). -->
<PackageVersion Include="Weasel.Postgresql" Version="9.17.0" />
<PackageVersion Include="Weasel.Storage" Version="9.17.0" />
Marten — bumped to stay current on the Weasel line (compile-verified).
9.20.1 (weasel#396/#397, marten#5090): AdvisoryLock can no longer strand a handle acquired after
DisposeAsync drained. TryAttainLockAsync checked _disposed only at method entry and then stored the
winning handle unconditionally, so an acquire in flight during disposal put its handle where nothing
would ever dispose it — with the default transaction-scoped leadership lock that is a backend left
'idle in transaction' on pg_try_advisory_xact_lock until the process exits, which the #4953 gap
detection then reads as a live pre-gap reserver and never advances past. The store now happens under
the same lock the drain latches under, a displaced handle (lock lost in monitored mode, re-attained)
is disposed rather than overwritten, and HasLock/ReleaseLockAsync move under that lock too.
9.20.2 (weasel#399/#400, found here): REQUIRED to move off 9.17.0. Weasel 9.18.0's computed-column
delta detection (weasel#373) routed every column through TableColumn.MatchesForDelta, where a bare
Equals(actual) binds to the protected NON-virtual Equals(TableColumn) overload and silently bypasses
subclass overrides of Equals(object). That override is exactly how RevisionColumn declares an
existing bigint mt_version acceptable for an integer-desired column instead of emitting a lossy
narrowing cast (#4614/#4742), so on 9.18.0–9.20.1 the column landed in Columns.Different, the table
was classified Update, and AssertDatabaseMatchesConfigurationAsync threw with an EMPTY change set —
Bug_4614_revision_column_int_for_IRevisioned's assert-check test fails on every version in that
range. 9.18/9.19/9.20 are otherwise additive for Marten and ride along (compile-verified). -->
<PackageVersion Include="Weasel.Postgresql" Version="9.20.2" />
<PackageVersion Include="Weasel.Storage" Version="9.20.2" />
<PackageVersion Include="WolverineFx.Marten" Version="4.2.0" />
<!-- The whole test suite is on xunit v3. VSTest stays the execution mode, so the
Nuke targets and the GitHub workflows are unaffected. -->
Expand Down
17 changes: 17 additions & 0 deletions docs/events/projections/async-daemon.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,23 @@ Some monitoring tools erroneously report this query as "load", however this quer
If this monitoring is undesirable for your scenario, you can opt-out by setting `options.Events.UseMonitoredAdvisoryLock` to false when configuring Marten.
:::

By default the `HotCold` leadership lock is transaction-scoped (`pg_try_advisory_xact_lock`), which
means the session holding it keeps a transaction open for as long as it is the leader. Set
`options.Events.UseAdvisoryLockTransaction` to false to use a session-scoped lock instead, which holds
no open transaction.

::: warning
Prefer `UseAdvisoryLockTransaction = false` when a **single process starts more than one
daemon-hosting `IHost` over its lifetime** — the usual shape of an xUnit integration suite that boots
and tears down a host per test class. Should any leadership lock session outlive its host, a
transaction-scoped lock leaves that session `idle in transaction` for the rest of the process, and the
daemon's high-water gap detection has to treat any transaction older than a sequence gap as a
potential in-flight append it must not skip past. One such session is enough to pin the high water
mark for every later daemon in that process, which surfaces as `WaitForNonStaleProjectionDataAsync`
timing out and a repeating `Daemon high water detection is holding before the sequence gap` log. A
session-scoped lock cannot cause that, because it holds no transaction to be seen.
:::

## Projection Distribution

If your Marten store is only using a single database, Marten will distribute projections by projection type. If your store is using
Expand Down
235 changes: 235 additions & 0 deletions src/DaemonTests/Bugs/Bug_5091_allocation_fence_at_mark_zero.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
using System;
using System.Threading;
using System.Threading.Tasks;
using DaemonTests.TestingSupport;
using JasperFx.Core;
using JasperFx.Events;
using Marten;
using Marten.Events.Daemon.HighWater;
using Marten.Storage;
using Marten.Testing;
using Marten.Testing.Harness;
using Microsoft.Extensions.Logging.Abstractions;
using Npgsql;
using Shouldly;
using Weasel.Postgresql;
using Xunit;

namespace DaemonTests.Bugs;

/// <summary>
/// #5091, follow-up to #4953/#5057. The allocation fence used to be built from the sequence's
/// RESERVED ceiling (<c>last_value</c>), and Postgres reports <c>last_value = 1</c> for a sequence
/// nothing has drawn from yet. No poll could ever report a value at or below a stuck mark of 0, so a
/// leading gap — the mark pinned at 0 under a hole at the very start of the sequence — could never be
/// fenced, and any permanently idle open transaction held it forever. That is the shape in #5090:
/// "Daemon high water detection is holding before the sequence gap above 0". Reading <c>is_called</c>
/// as well gives the highest ALLOCATED value, which is 0 for a pristine sequence, and mark 0 becomes
/// fenceable without weakening the proof.
/// </summary>
public class Bug_5091_allocation_fence_at_mark_zero: DaemonContext
{
private readonly ITestOutputHelper _output;

public Bug_5091_allocation_fence_at_mark_zero(ITestOutputHelper output): base(output)
{
_output = output;
}

private string Schema => theStore.Events.DatabaseSchemaName;

private async Task<NpgsqlConnection> openConnection()
{
var conn = new NpgsqlConnection(ConnectionSource.ConnectionString);
await conn.OpenAsync();
return conn;
}

// The #5090 zombie: a session that took a transaction-scoped advisory lock and will never run
// another statement. Its xact_start predates every gap, so the unfenced liveness probe counts it
// as a possible reserver forever.
private async Task<NpgsqlConnection> startIdleAdvisoryLockSession(long lockId)
{
var conn = await openConnection();
await conn.BeginTransactionAsync();
await conn.CreateCommand($"select pg_advisory_xact_lock({lockId})").ExecuteNonQueryAsync();
return conn;
}

private async Task appendEvents(int count)
{
await using var session = theStore.LightweightSession();
for (var i = 0; i < count; i++)
{
session.Events.StartStream(Guid.NewGuid(), new Bug5091GapEvent(Guid.NewGuid(), i + 1));
}

await session.SaveChangesAsync();
}

private async Task<long> scalar(string sql)
{
await using var conn = await openConnection();
var raw = await conn.CreateCommand(sql).ExecuteScalarAsync();
return raw is long l ? l : Convert.ToInt64(raw ?? 0L);
}

private HighWaterDetector buildDetector()
{
return new HighWaterDetector((MartenDatabase)theStore.Tenancy.Default.Database, theStore.Events,
NullLogger.Instance);
}

[Fact]
public async Task a_dead_leading_gap_skips_despite_an_idle_advisory_lock_session()
{
StoreOptions(opts =>
{
opts.Projections.StaleSequenceThreshold = 500.Milliseconds();
});
theStore.EnsureStorageExists(typeof(IEvent));

// The zombie is already parked before anything is appended — exactly the #5090 ordering,
// where the leaked session came from an earlier host in the same process.
var listener = await startIdleAdvisoryLockSession(5091001);
try
{
var detector = buildDetector();

// The fence-enabling reading: a poll over a pristine sequence. last_value is 1 here, but
// is_called is false, so nothing has been handed out and the allocated high is 0.
var baseline = await detector.Detect(CancellationToken.None);
baseline.CurrentMark.ShouldBe(0);

// seq 1 is reserved and rolled back: a permanently dead hole at the very start.
await using (var conn = await openConnection())
{
var tx = await conn.BeginTransactionAsync();
var seq = (long)(await conn.CreateCommand($"select nextval('{Schema}.mt_events_sequence')")
.ExecuteScalarAsync())!;
seq.ShouldBe(1);
await tx.RollbackAsync(TestContext.Current.CancellationToken);
}

await appendEvents(3); // 2..4 committed

// First sighting — the stale threshold is measured from here, so this one holds.
var first = await detector.DetectInSafeZone(CancellationToken.None);
first.CurrentMark.ShouldBe(0);

await Task.Delay(700, TestContext.Current.CancellationToken);

// Past the threshold. The only open transaction older than the gap is the idle advisory
// lock session, and it has provably executed nothing since before seq 1 was allocated, so
// it cannot be the reserver. The gap is dead: skip.
var second = await detector.DetectInSafeZone(CancellationToken.None);
_output.WriteLine($"Leading gap past threshold with idle listener: CurrentMark={second.CurrentMark}");
second.CurrentMark.ShouldBe(4);
second.IncludesSkipping.ShouldBeTrue();

var persisted = await scalar(
$"select coalesce(max(last_seq_id), 0) from {Schema}.mt_event_progression where name = 'HighWaterMark'");
persisted.ShouldBe(4);
}
finally
{
await listener.DisposeAsync();
}
}

[Fact]
public async Task a_live_reserver_of_the_leading_gap_still_holds_the_mark()
{
StoreOptions(opts =>
{
opts.Projections.StaleSequenceThreshold = 500.Milliseconds();
});
theStore.EnsureStorageExists(typeof(IEvent));

var listener = await startIdleAdvisoryLockSession(5091002);
try
{
var detector = buildDetector();
(await detector.Detect(CancellationToken.None)).CurrentMark.ShouldBe(0);

// seq 1 reserved by a transaction that is still alive. It called nextval AFTER the fence,
// which bumped its state_change, so the fence must keep it even though the idle listener
// is ruled out. Fencing mark 0 must not become a licence to skip live appends.
var conn = await openConnection();
var tx = await conn.BeginTransactionAsync();
try
{
var seq = (long)(await conn.CreateCommand($"select nextval('{Schema}.mt_events_sequence')")
.ExecuteScalarAsync())!;
seq.ShouldBe(1);

await appendEvents(3); // 2..4 committed

(await detector.DetectInSafeZone(CancellationToken.None)).CurrentMark.ShouldBe(0);

await Task.Delay(700, TestContext.Current.CancellationToken);
var held = await detector.DetectInSafeZone(CancellationToken.None);
_output.WriteLine($"Leading gap past threshold with LIVE reserver: CurrentMark={held.CurrentMark}");
held.CurrentMark.ShouldBe(0);

await tx.RollbackAsync(TestContext.Current.CancellationToken);
}
finally
{
await conn.DisposeAsync();
}

// The reserver died, so now the gap is provably dead.
await Task.Delay(200, TestContext.Current.CancellationToken);
var after = await detector.DetectInSafeZone(CancellationToken.None);
_output.WriteLine($"After reserver death: CurrentMark={after.CurrentMark}");
after.CurrentMark.ShouldBe(4);
after.IncludesSkipping.ShouldBeTrue();
}
finally
{
await listener.DisposeAsync();
}
}

[Fact]
public async Task a_detector_that_never_saw_the_pristine_sequence_still_holds_conservatively()
{
StoreOptions(opts =>
{
opts.Projections.StaleSequenceThreshold = 500.Milliseconds();
});
theStore.EnsureStorageExists(typeof(IEvent));

var listener = await startIdleAdvisoryLockSession(5091003);
try
{
// The gap forms before this detector ever polls — the fresh-host-over-an-existing-database
// case. There is no proof of when seq 1 was allocated, so the idle session stays a
// candidate reserver and the documented conservative hold remains (#5090's actual shape;
// its real fix is the Weasel handle strand, not this fence).
await using (var conn = await openConnection())
{
var tx = await conn.BeginTransactionAsync();
await conn.CreateCommand($"select nextval('{Schema}.mt_events_sequence')").ExecuteScalarAsync();
await tx.RollbackAsync(TestContext.Current.CancellationToken);
}

await appendEvents(3); // 2..4 committed

var detector = buildDetector();
(await detector.DetectInSafeZone(CancellationToken.None)).CurrentMark.ShouldBe(0);

await Task.Delay(700, TestContext.Current.CancellationToken);
var second = await detector.DetectInSafeZone(CancellationToken.None);
_output.WriteLine($"Fenceless leading gap past threshold: CurrentMark={second.CurrentMark}");
second.CurrentMark.ShouldBe(0);
}
finally
{
await listener.DisposeAsync();
}
}
}

public record Bug5091GapEvent(Guid Id, int Number);
Loading
Loading