Skip to content

Polecat-owned rolling range partitions for time-series document tables (#386) #387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeremydmiller merged 1 commit into from
Jul 30, 2026
Merged

Polecat-owned rolling range partitions for time-series document tables (#386) #387

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 20 additions & 9 deletions Directory.Packages.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,17 +50,19 @@
Func<IEvent, object?> — SOURCE BREAKING, which is why the bump and the call-site change
land together — plus reachable NaturalKeyBuilder / NaturalKeyFor() and loud failure on an
unbindable [NaturalKeySource] (polecat#369).
Also carries jasperfx#568/#572 daemon race fixes, picked up for free. -->
<PackageVersion Include="JasperFx" Version="2.36.2" />
<PackageVersion Include="JasperFx.Events" Version="2.36.2" />
Also carries jasperfx#568/#572 daemon race fixes, picked up for free.
JasperFx 2.36.3: the floor Weasel 9.21.0 depends on — bumped in lockstep with the
Weasel bump below so the matrix stays coherent rather than resolving transitively. -->
<PackageVersion Include="JasperFx" Version="2.36.3" />
<PackageVersion Include="JasperFx.Events" Version="2.36.3" />
<!-- Pin the sibling JasperFx packages for parity with Marten 9's matrix
even though Polecat doesn't currently reference them directly —
keeps the lockstep matrix coherent when transitive resolution
surfaces them through JasperFx / JasperFx.Events updates. The
RuntimeCompiler 5.x line is the active continuation of the 4.x
lineage; do not pin against the parallel stale 2.0.x series. -->
<PackageVersion Include="JasperFx.RuntimeCompiler" Version="5.0.0" />
<PackageVersion Include="JasperFx.SourceGenerator" Version="2.36.2" />
<PackageVersion Include="JasperFx.SourceGenerator" Version="2.36.3" />
<PackageVersion Include="Microsoft.Data.SqlClient" Version="7.0.0" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.0" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.0" />
Expand Down Expand Up @@ -124,16 +126,25 @@
AllowOrdinalSharing + explicit-ordinal add overloads (tenant bucketing),
MigrateAllTablesAsync new-table back-fill, and TenantPartitionAddResult /
TablePartitionStatus batch status reporting. Consumed by Polecat #335
(per-tenant partitioning parity for documents + streams). -->
<PackageVersion Include="Weasel.EntityFrameworkCore" Version="9.18.0" />
<PackageVersion Include="Weasel.SqlServer" Version="9.18.0" />
<PackageVersion Include="Weasel.Storage" Version="9.18.0" />
(per-tenant partitioning parity for documents + streams).
Weasel 9.21.0: weasel#401 — Weasel.SqlServer.Tables.Partitioning.ManagedRangePartitions
plus Weasel.Core.Partitioning's RollingWindowPolicy / PartitionPeriod: rolling
time-window RANGE partitions whose CreateDelta is purely additive (a window that has
rolled forward is never mistaken for drift) with NEXT USED + SPLIT RANGE roll-forward
and partition TRUNCATE + MERGE RANGE retention. Consumed by Polecat #386
(PartitionOn(...).ByRollingRange(...)). Also rides along from the intervening line:
weasel#391 (managed tenant bucketing actually shares one partition — #335's ordinal
sharing) and weasel#399 (TableColumn.MatchesForDelta compares through the virtual
Equals, so a subclassed column no longer churns the delta). -->
<PackageVersion Include="Weasel.EntityFrameworkCore" Version="9.21.0" />
<PackageVersion Include="Weasel.SqlServer" Version="9.21.0" />
<PackageVersion Include="Weasel.Storage" Version="9.21.0" />

<!-- Strongly typed IDs -->
<PackageVersion Include="StronglyTypedId" Version="1.0.0-beta08" />

<!-- Source generators (matched to JasperFx.Events above) -->
<PackageVersion Include="JasperFx.Events.SourceGenerator" Version="2.36.2" />
<PackageVersion Include="JasperFx.Events.SourceGenerator" Version="2.36.3" />

<!-- Build automation -->
<PackageVersion Include="Nuke.Common" Version="9.0.4" />
Expand Down
151 changes: 143 additions & 8 deletions docs/documents/partitioning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,20 @@
# Table Partitioning

Polecat can declaratively **RANGE-partition a document table** on a member you choose — the SQL Server
companion to Marten's `PartitionOn`. The classic use is a time-series retention table partitioned by
month, so that old data can eventually be pruned by dropping a partition instead of issuing a large
`DELETE`.
Polecat can **RANGE-partition a document table** on a member you choose — the SQL Server companion to
Marten's `PartitionOn`. The classic use is a time-series retention table partitioned by month, so that old
data is reclaimed by retiring a partition instead of issuing a large `DELETE`. Three strategies are
available:

| Strategy | Who owns the partitions |
| -------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| `ByRange(...)` / `PartitionByRange(...)` | you declare a fixed boundary list; Polecat rolls additions forward in place |
| [`ByRollingRange(...)`](#rolling-time-windows) | Polecat, from a rolling-window policy — provisioning *and* retention |
| [`ByExternallyManagedRange(...)`](#externally-managed-range-partitions) | something outside Polecat, which Polecat then never touches |

::: tip
This is built on SQL Server partition **functions** and **schemes** rather than the child-table model
PostgreSQL/Marten uses, so the migration story differs. It requires `Weasel.SqlServer` 9.3.0 or later.
PostgreSQL/Marten uses, so the migration story differs. Declarative range partitioning requires
`Weasel.SqlServer` 9.3.0 or later; rolling time windows require 9.21.0 or later.
:::

## Partitioning by a date member
Expand Down Expand Up @@ -57,14 +64,142 @@ opts.Schema.For<MetricsSample>()
Schema migration adds the new partition with no data movement. Removing a boundary or changing the
column/type is reported as a rebuild rather than performed silently.

## Rolling time windows

Declaring every boundary up front only works while the set of partitions is *fixed*. Real time-series
storage needs it to **move**: provision next month, retire last year. Rather than hand-writing that DDL on
a schedule forever, describe the window and let Polecat own it:

```csharp
var store = DocumentStore.For(opts =>
{
opts.Connection(connectionString);

// Keep 12 months of history, provision 3 months ahead. Polecat splits in the partitions at the
// leading edge and retires the aged ones at the trailing edge — no application-authored DDL.
opts.Schema.For<MetricsSample>()
.PartitionOn(x => x.BucketEnd)
.ByRollingRange(PartitionPeriod.Month, periodsAhead: 3, periodsBehind: 12);
});
```

`PartitionPeriod` (from `Weasel.Core.Partitioning`) supports `Hour`, `Day`, `Week`, `Month`, and `Year`, and
the whole window is computed in UTC. The partition member must be a `DateTime` or `DateTimeOffset` — a rolling window is a function of the
clock, so anything else is rejected at *configuration* time with a message that names the member, rather
than surfacing as an opaque partition-function error during the first migration.

There are also overloads taking a `RollingWindowPolicy` directly, or a pre-built `ManagedRangePartitions`.
Pass the *same* manager instance to several document types to roll all of their tables forward in one pass:

```csharp
using Weasel.SqlServer.Tables.Partitioning; // ManagedRangePartitions
using Weasel.Core.Partitioning; // RollingWindowPolicy, PartitionPeriod

var manager = new ManagedRangePartitions(
RollingWindowPolicy.Monthly(periodsAhead: 3, periodsBehind: 12),
column: "bucket_end", sqlDataType: "datetimeoffset");

opts.Schema.For<MetricsSample>().PartitionOn(x => x.BucketEnd).ByRollingRange(manager);
opts.Schema.For<TraceSample>().PartitionOn(x => x.BucketEnd).ByRollingRange(manager);
```

The manager is also where you set `Filegroup` if the partitions should not go to `PRIMARY`, and it takes a
`TimeProvider` so the window can be rolled forward deterministically in tests instead of waiting on the
calendar.

### How the two halves are driven

| | Driven by | Why |
| -------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **Provision** the leading edge | ordinary schema migration | with a rolling-window manager attached the partition delta is purely additive, so a rolled-forward window is a `SPLIT` |
| **Retire** the trailing edge | startup pass + `Advanced.*` | migration never removes data, so the retention half has to be driven separately |

The window is a pure function of the policy and the clock, which is what makes this safe: a window that has
rolled forward differs from the database by exactly one new boundary at the leading edge and one aged
boundary at the trailing edge. A boundary the declaration no longer names is a period that has **aged out** —
the normal steady state of a rolling window, not drift — so `CreateDelta` reports `Additive` or `None`, never
`Rebuild`. (A column or type change still rebuilds, as it must.)

Polecat runs the maintenance pass — roll forward, then retire everything below the retention floor — at
startup, alongside the schema changes it already applies:

```csharp
builder.Services.AddPolecat(opts =>
{
// ... the ByRollingRange() configuration above
}).ApplyAllDatabaseChangesOnStartup();
```

Applying changes on startup is how a host says "Polecat owns this schema", and retiring a partition is
emphatically a schema change — so the pass is gated on the same opt-in as the migration itself.

### Retention is a partition operation, not a `DELETE`

Retiring a period is `TRUNCATE TABLE ... WITH (PARTITIONS (n))` followed by
`ALTER PARTITION FUNCTION ... MERGE RANGE`, **in that order**. `MERGE RANGE` on a partition that still holds
rows does not reclaim anything — it *moves* those rows into the neighbouring partition, which is the
opposite of the point. Truncating first deallocates the partition's pages in O(1), and the merge that
follows is then metadata-only against an empty partition. If the truncate fails, the boundary is
deliberately left in place.

Only boundaries the policy itself would have produced are ever retired, so a hand-added boundary — or one
left over from a different period size — is left strictly alone.

::: warning
Retiring a period removes its rows. That is the point — it is what makes reclaim O(1) instead of a mass
`DELETE` — but choose `periodsBehind` to match the retention policy you actually want.
:::

If a process is long-lived enough to outrun the number of periods you provision ahead — an hourly window
especially — run the pass yourself on whatever cadence the period size demands:

```csharp
// Roll every rolling-window table forward to its current window and retire the partitions that have
// aged past their retention floor. Idempotent, and safe to run from several nodes at once.
await store.Advanced.ApplyRollingPartitionsAsync(token);

// ...or run just one half
await store.Advanced.RollPartitionsForwardAsync(token); // additive only, never removes data
await store.Advanced.DropAgedRollingPartitionsAsync(token); // retention only
```

Each returns Weasel `TablePartitionStatus[]` — one entry per managed table, so a partial failure surfaces
per table rather than taking the whole pass down.

::: tip
A SQL Server RANGE function always spans `(-infinity, +infinity)`, so the outermost partitions absorb any
row written outside the provisioned window. Unlike PostgreSQL there is no "no partition of relation" error
to guard against and no `DEFAULT` overflow partition to declare.
:::

## Externally-managed range partitions

If something genuinely outside Polecat owns the partitions, use the externally-managed variant. Polecat
creates the partition function, scheme, and table once with the supplied initial boundaries and then never
reconciles the partitioning again, so runtime `SPLIT`/`SWITCH`/`MERGE` from elsewhere survives a later
schema apply:

```csharp
opts.Schema.For<MetricsSample>()
.PartitionOn(x => x.BucketEnd)
.ByExternallyManagedRange(
new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero),
new DateTimeOffset(2026, 2, 1, 0, 0, 0, TimeSpan.Zero));
```

For an ordinary time-series retention table, prefer [rolling time windows](#rolling-time-windows). Opting
out of Weasel means opting out of its **ordering and dependency management**, not just its DDL generation —
a hand-rolled rebuild that re-creates a partitioned table can easily do so before a helper object its
indexes depend on exists, and the resulting failure looks like a broken index rather than what it is.

## Limitations

- Supported for **single-tenant** document tables only; combining member RANGE partitioning with
conjoined multi-tenancy throws at start-up. Conjoined document tables can instead use the managed
per-tenant partitioning below.
- Dropping aged partitions for retention (`SWITCH`/`MERGE RANGE`) and externally-managed partition
rolling are not yet wired into the document API — for now, manage those out of band, or keep pruning
with a predicate delete.
- One partition scheme per table is a SQL Server constraint, so a document type cannot combine member
RANGE partitioning (declared, rolling, or externally managed) with the store's managed per-tenant
partitioning.

## Managed per-tenant partitioning (#335)

Expand Down
Loading
Loading