Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 15 additions & 10 deletions docs/events/projections/async-daemon.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,18 +91,23 @@ As of right now, the daemon can run as one of two modes:
1. *HotCold* -- the daemon will use a built in [leader election](https://en.wikipedia.org/wiki/Leader_election) function individually for each
projection on each tenant database and **ensure that each projection is running on exactly one running process**.

::: tip
When running in `HotCold` mode, Marten will monitor the Postgres advisory lock by running a `SELECT pg_catalog.pg_sleep(60)` query to detect if the database restarts or fails-over.
Without this monitoring, Marten will not be aware of the lock loss and multiple async daemons can start running concurrently across multiple nodes, causing application failure.
By default the `HotCold` leadership lock is session-scoped (`pg_try_advisory_lock`), so the session
holding it keeps no transaction open while it is the leader. Set
`options.Events.UseAdvisoryLockTransaction` to true to use a transaction-scoped lock
(`pg_try_advisory_xact_lock`) instead. That is the better fit behind PgBouncer in transaction pooling
mode, which can hand a session-scoped lock's server connection to another client, but it means the
leader keeps a transaction open for as long as it holds leadership.

Some monitoring tools erroneously report this query as "load", however this query simply sleeps for 60 seconds and **does not** consume any database resources.
If this monitoring is undesirable for your scenario, you can opt-out by setting `options.Events.UseMonitoredAdvisoryLock` to false when configuring Marten.
:::
```cs
builder.Services.AddMarten(opts =>
{
opts.Connection(connectionString);

By default the `HotCold` leadership lock is transaction-scoped (`pg_try_advisory_xact_lock`), which
means the session holding it keeps a transaction open for as long as it is the leader. Set
`options.Events.UseAdvisoryLockTransaction` to false to use a session-scoped lock instead, which holds
no open transaction.
// Opt into the transaction-scoped leadership lock
opts.Events.UseAdvisoryLockTransaction = true;
})
.AddAsyncDaemon(DaemonMode.HotCold);
```

::: tip
Marten's gap detection recognizes its own leadership lock connections and never counts them as
Expand Down
898 changes: 464 additions & 434 deletions docs/migration-guide.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/scenarios/command_handler_workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ builder.Services.AddMarten(opts =>

It's pretty involved, but the key takeaway is that _if_ you are using lightweight sessions for a performance optimization
-- and you probably should even though that's not a Marten default! -- and _also_ using `FetchForWriting<T>()` with `Inline` projections, this optimizes your system to make fewer network round trips to the database and reuse the data
you already fetched when applying the `Inline` projection. **Marten 9 ships this flag at `true` by default** (see the [Marten 9 defaults section in the migration guide](../migration-guide.md#flipped-defaults-in-marten-9--read-this-section)) — the prior V8 default was `false`.
you already fetched when applying the `Inline` projection. **Marten 9 ships this flag at `true` by default** (see the [Marten 9 defaults section in the migration guide](../migration-guide.md#changed-defaults)) — the prior V8 default was `false`.

::: warning Aggregate mutations leak under `UseIdentityMapForAggregates = true`
This optimization assumes the **decider pattern**: your handler returns events, the inline projection rebuilds aggregate state from those events on save, and you do _not_ mutate fields on the `stream.Aggregate` instance returned by `FetchForWriting()`. The optimization works by stashing the fetched aggregate in the session's identity map and re-reading it from there when the inline projection applies the new events.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -192,8 +192,8 @@ public async Task restarted_daemon_advances_past_a_dead_gap_with_slow_sessions_p
/// #5125 — no restart and no third-party session needed. The store is already gapped before any
/// daemon has ever run (events appended then deleted; a rolled-back append leaves the same state),
/// so no fence can be established by construction. The only open transaction is the daemon's OWN
/// HotCold leadership lock, which is transaction-scoped by default and therefore sits
/// idle-in-transaction for its entire tenure — and gap detection counts it as a possible reserver
/// HotCold leadership lock, which is transaction-scoped here (UseAdvisoryLockTransaction) and therefore
/// sits idle-in-transaction for its entire tenure — and gap detection counts it as a possible reserver
/// of the very gap it is trying to skip.
/// </summary>
[Fact]
Expand Down Expand Up @@ -221,7 +221,7 @@ await conn.CreateCommand($"delete from {Schema}.mt_event_progression")
await PublishSingleThreaded();
var ceiling = await highestCommittedSequenceAsync();

using var host = await StartDaemonInHotColdMode();
using var host = await StartDaemonInHotColdMode(opts => opts.Events.UseAdvisoryLockTransaction = true);
await host.Daemon().Tracker.WaitForHighWaterMark(ceiling, 60.Seconds());

(await currentHighWaterMarkAsync()).ShouldBe(ceiling);
Expand Down
4 changes: 3 additions & 1 deletion src/DaemonTests/TestingSupport/DaemonContext.cs
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ internal async Task<ProjectionDaemon> StartDaemon(string tenantId)
return daemon;
}

internal async Task<IHost> StartDaemonInHotColdMode()
internal async Task<IHost> StartDaemonInHotColdMode(Action<StoreOptions> configure = null)
{
var host = await Host.CreateDefaultBuilder()
.UseLamar(services =>
Expand All @@ -96,6 +96,8 @@ internal async Task<IHost> StartDaemonInHotColdMode()
opts.Projections.DaemonLockId = lockId;

opts.Projections.Add(new TripProjectionWithCustomName(), ProjectionLifecycle.Async);

configure?.Invoke(opts);
}).AddAsyncDaemon(DaemonMode.HotCold);
}).StartAsync();

Expand Down
2 changes: 1 addition & 1 deletion src/Marten/Events/Daemon/HighWater/GapLivenessProbe.cs
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ from pg_stat_activity a
and a.state in ('idle in transaction', 'idle in transaction (aborted)')
and a.state_change < :fence
),
-- #5125: the daemon's OWN HotCold leadership lock. With the default transaction-scoped advisory
-- #5125: the daemon's OWN HotCold leadership lock. With a transaction-scoped advisory
-- lock it sits 'idle in transaction' for the whole of its leadership tenure, so it satisfies the
-- open-transaction clause forever — including on a store that was already gapped before any daemon
-- ran, where no allocation fence can ever be established and the quiescent CTE above is therefore
Expand Down
2 changes: 1 addition & 1 deletion src/Marten/Events/EventGraph.cs
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ public override IEvent BuildEvent(object eventData)
public bool UseMandatoryStreamTypeDeclaration { get; set; }
public bool UseMonitoredAdvisoryLock { get; set; } = true;

public bool UseAdvisoryLockTransaction { get; set; } = true;
public bool UseAdvisoryLockTransaction { get; set; }

public bool EnableAdvancedAsyncTracking { get; set; }
public bool EnableEventSkippingInProjectionsOrSubscriptions { get; set; }
Expand Down
2 changes: 1 addition & 1 deletion src/Marten/Events/IEventStoreOptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ public interface IEventStoreOptions
public bool UseMonitoredAdvisoryLock { get; set; }

/// <summary>
/// Uses a transaction-scoped advisory lock instead of a session-scoped one. This improves compatibility with PGBouncer and suppresses some irrelevant warning spam in Postgres logs. Enabled by default.
/// Uses a transaction-scoped advisory lock for HotCold leadership instead of a session-scoped one. This improves compatibility with PgBouncer in transaction pooling mode, but keeps a transaction open on the leader's connection for as long as it holds leadership. Disabled by default.
/// </summary>
public bool UseAdvisoryLockTransaction { get; set; }

Expand Down
2 changes: 1 addition & 1 deletion src/Marten/Events/IReadOnlyEventStoreOptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ public interface IReadOnlyEventStoreOptions
public bool UseMonitoredAdvisoryLock { get; set; }

/// <summary>
/// Uses a transaction-scoped advisory lock instead of a session-scoped one. This improves compatibility with PGBouncer and suppresses some irrelevant warning spam in Postgres logs. Enabled by default.
/// Uses a transaction-scoped advisory lock for HotCold leadership instead of a session-scoped one. This improves compatibility with PgBouncer in transaction pooling mode, but keeps a transaction open on the leader's connection for as long as it holds leadership. Disabled by default.
/// </summary>
public bool UseAdvisoryLockTransaction { get; set; }

Expand Down
Loading