Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/features/governance/budget-and-limits.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -183,18 +183,18 @@ It sets the alignment **mode**, not a shared reset instant. An owner holds one w
Alignment only applies to durations that have a calendar boundary: `1d`, `1w`, `1M`, `1Q` and `1Y`. A **sub-day** window such as a `1h` request limit stays on its rolling schedule even when its owner is aligned, and the flag changes nothing for it.
</Note>

Turning calendar alignment **on** for an existing owner never moves `last_reset` backwards, so an already-open window is not re-dated. What happens next is decided **per window**, against that window's own boundary:
Turning calendar alignment **on** for an existing owner **keeps its accumulated usage**. Each open window is adopted onto the calendar grid instead of being reset: its start moves forward to the boundary it now follows, its usage carries over untouched, and its first aligned reset happens at the **next** boundary.

- **Window opened after its most recent boundary** - nothing resets now. It keeps its start and its accumulated usage, and its first aligned reset happens at its next boundary.
- **Window opened before its most recent boundary** - it is **already due** under the new rule, so the next reset evaluation resets it and **clears its current usage**: spend for a budget, the token or request counter for a rate limit. For a monthly window switched on partway through a month, this is the common case.
So a monthly budget with $42 of spend, aligned on 5 February, keeps the $42 and resets on 1 March. Nothing is cleared at the switch, and nothing is forgiven.

A reset evaluation is not only the background sweep. The same due-or-not check runs again on the request path, just before a request's usage is recorded, so an already-due window is cleared by the next request even if no sweep has fired yet.
Two details follow from each window being adopted on its own terms:

Because each window is judged separately, one switch can leave a `1d` limit untouched while clearing a `1M` budget on the same owner.
- **Windows do not move together.** A `1M` budget adopts the month boundary while a `1d` counter beside it adopts midnight.
- **Sub-day windows are not adopted at all**, because they have no boundary to adopt. A `1h` counter keeps its rolling schedule, as described above.

<Warning>
Enabling alignment on an owner with accumulated usage can therefore clear that usage almost immediately, and which of its windows are affected depends on each one's own duration and start. There is generally no moment when every window is freshly reset, so to preserve them all set `calendar_aligned` at **create** time and avoid the switch-over entirely. To protect one specific window, enable alignment just after **that** window resets.
</Warning>
<Note>
The boundary only ever moves **forward**. A window that already opened after its most recent boundary is current, so it is left exactly as it is. This forward-only rule is what keeps every node in a cluster agreeing on which window is open.
</Note>

### Quarterly budgets and fiscal quarters

Expand Down
34 changes: 34 additions & 0 deletions framework/configstore/tables/budget.go
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,40 @@ func (b *TableBudget) WindowStart(now time.Time) time.Time {
return RollingWindowStart(anchor, duration, now)
}

// AdoptCalendarAlignment re-anchors an already-open window onto the calendar grid
// it has just been told to follow, and reports whether it moved.
//
// Called once, on the transition from rolling to calendar-aligned. Without it the
// switch is destructive: the reset path resets whenever WindowStart(now) is after
// LastReset, and for a freshly aligned budget WindowStart is the most recent
// boundary, so any window that opened before that boundary is instantly due and
// its usage is cleared. That is most of the month for a monthly budget.
//
// Moving LastReset up to the boundary makes the window current instead of overdue,
// so nothing resets now and the first aligned reset lands on the next boundary.
// The move is forward-only, which is the one direction every persistence path
// allows: rewinding would re-open a window the cluster already treated as spent.
// CurrentUsage is deliberately untouched - the operator asked to change a
// schedule, not to forgive spend.
//
// Returns false, changing nothing, when the budget is not aligned, when its
// duration has no calendar boundary (sub-day windows stay rolling), or when the
// window already opened at or after the boundary and is therefore current.
func (b *TableBudget) AdoptCalendarAlignment(now time.Time) bool {
if b == nil || !b.IsCalendarAligned || !IsCalendarAlignableDuration(b.ResetDuration) {
return false
}
// Read through WindowStart rather than GetCalendarPeriodStart directly so the
// boundary adopted here can never drift from the one the reset path compares
// against.
start := b.WindowStart(now)
if !start.After(b.LastReset) {
return false
}
b.LastReset = start
return true
}

// StampCalendarAlignment copies an owner's calendar-alignment flag onto the
// budgets and rate limit it owns.
//
Expand Down
142 changes: 142 additions & 0 deletions framework/configstore/tables/budgetwindow_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -607,3 +607,145 @@ func TestQuarterlyBudgetResetIsNotPerpetuallyDue(t *testing.T) {
fmt.Sprintf("quarter start %s: budget became due an hour into its quarter", quarterStart))
}
}

// TestAdoptCalendarAlignmentPreservesTheOpenWindow pins the switch-over contract:
// turning calendar alignment on must not cost the owner its accumulated usage.
//
// Without adoption, a window opened before the current boundary is already due
// the moment the flag flips, because budgetResetTarget resets whenever
// WindowStart(now) is after LastReset. Adoption moves LastReset forward onto the
// boundary instead, which the forward-only guard permits, so the window becomes
// current rather than overdue and its usage survives to the next real boundary.
func TestAdoptCalendarAlignmentPreservesTheOpenWindow(t *testing.T) {
now := time.Date(2026, time.February, 5, 12, 0, 0, 0, time.UTC)
monthStart := time.Date(2026, time.February, 1, 0, 0, 0, 0, time.UTC)

newBudget := func(duration string, lastReset time.Time) *TableBudget {
return &TableBudget{
ID: "adopt-budget",
MaxLimit: 100,
CurrentUsage: 42,
ResetDuration: duration,
IsCalendarAligned: true,
CreatedAt: lastReset,
LastReset: lastReset,
}
}

t.Run("a window opened before the boundary is adopted onto it", func(t *testing.T) {
budget := newBudget("1M", time.Date(2026, time.January, 10, 9, 0, 0, 0, time.UTC))

assert.True(t, budget.AdoptCalendarAlignment(now), "the window moved, so adoption reports a change")
assert.True(t, budget.LastReset.Equal(monthStart), "the window is re-anchored on the boundary it now follows")
assert.Equal(t, 42.0, budget.CurrentUsage, "adoption must never spend the operator's usage")
assert.False(t, budget.WindowStart(now).After(budget.LastReset),
"and the budget must no longer read as due, which is the whole point")
})

t.Run("a window opened after the boundary is left alone", func(t *testing.T) {
lastReset := time.Date(2026, time.February, 3, 9, 0, 0, 0, time.UTC)
budget := newBudget("1M", lastReset)

assert.False(t, budget.AdoptCalendarAlignment(now), "nothing to move")
assert.True(t, budget.LastReset.Equal(lastReset), "an already-current window keeps its own start")
})

t.Run("the boundary is never moved backwards", func(t *testing.T) {
lastReset := time.Date(2026, time.February, 20, 9, 0, 0, 0, time.UTC)
budget := newBudget("1M", lastReset)

assert.False(t, budget.AdoptCalendarAlignment(now))
assert.True(t, budget.LastReset.Equal(lastReset),
"rewinding would re-open a window the cluster already agreed was spent")
})

t.Run("a sub-day window is not adopted", func(t *testing.T) {
lastReset := time.Date(2026, time.February, 5, 9, 30, 0, 0, time.UTC)
budget := newBudget("1h", lastReset)

assert.False(t, budget.AdoptCalendarAlignment(now),
"an hourly window has no calendar boundary, so alignment leaves it rolling")
assert.True(t, budget.LastReset.Equal(lastReset))
})

t.Run("an unaligned budget is not adopted", func(t *testing.T) {
lastReset := time.Date(2026, time.January, 10, 9, 0, 0, 0, time.UTC)
budget := newBudget("1M", lastReset)
budget.IsCalendarAligned = false

assert.False(t, budget.AdoptCalendarAlignment(now),
"adoption belongs to the switch-over, not to every config write")
assert.True(t, budget.LastReset.Equal(lastReset))
})

t.Run("a quarterly budget adopts its fiscal boundary", func(t *testing.T) {
budget := newBudget("1Q", time.Date(2025, time.December, 10, 9, 0, 0, 0, time.UTC))
budget.ResetConfig = &BudgetResetConfig{QuarterStartMonth: int(time.February)}

assert.True(t, budget.AdoptCalendarAlignment(now))
assert.True(t, budget.LastReset.Equal(time.Date(2026, time.February, 1, 0, 0, 0, 0, time.UTC)),
"a February fiscal year opens Q1 on Feb 1, not on the calendar quarter")
})
}

// TestAdoptCalendarAlignmentOnRateLimits mirrors the budget contract for the two
// counters a rate limit carries.
//
// Token and request counters keep independent durations and independent
// LastReset values, so adoption is decided per counter. rateLimitResetTarget
// applies the same "boundary after last reset" rule as budgets, which means the
// same switch-over would otherwise clear a counter the operator was still using.
func TestAdoptCalendarAlignmentOnRateLimits(t *testing.T) {
now := time.Date(2026, time.February, 5, 12, 0, 0, 0, time.UTC)
monthStart := time.Date(2026, time.February, 1, 0, 0, 0, 0, time.UTC)
dayStart := time.Date(2026, time.February, 5, 0, 0, 0, 0, time.UTC)
stale := time.Date(2026, time.January, 10, 9, 0, 0, 0, time.UTC)

newRateLimit := func(tokenDuration, requestDuration string) *TableRateLimit {
return &TableRateLimit{
ID: "adopt-rate-limit",
TokenResetDuration: &tokenDuration,
TokenCurrentUsage: 900,
TokenLastReset: stale,
RequestResetDuration: &requestDuration,
RequestCurrentUsage: 17,
RequestLastReset: stale,
IsCalendarAligned: true,
}
}

t.Run("each counter adopts its own boundary", func(t *testing.T) {
rl := newRateLimit("1M", "1d")

assert.True(t, rl.AdoptCalendarAlignment(now), "both counters moved")
assert.True(t, rl.TokenLastReset.Equal(monthStart), "the monthly counter adopts the month boundary")
assert.True(t, rl.RequestLastReset.Equal(dayStart), "the daily counter adopts midnight, not the month boundary")
assert.Equal(t, int64(900), rl.TokenCurrentUsage, "adoption must not clear a counter")
assert.Equal(t, int64(17), rl.RequestCurrentUsage)
})

t.Run("a sub-day counter is left rolling while its sibling adopts", func(t *testing.T) {
rl := newRateLimit("1M", "1h")

assert.True(t, rl.AdoptCalendarAlignment(now), "the monthly counter still moves")
assert.True(t, rl.TokenLastReset.Equal(monthStart))
assert.True(t, rl.RequestLastReset.Equal(stale),
"an hourly counter has no calendar boundary, so it keeps its rolling anchor")
})

t.Run("an unaligned rate limit is not adopted", func(t *testing.T) {
rl := newRateLimit("1M", "1d")
rl.IsCalendarAligned = false

assert.False(t, rl.AdoptCalendarAlignment(now))
assert.True(t, rl.TokenLastReset.Equal(stale))
assert.True(t, rl.RequestLastReset.Equal(stale))
})

t.Run("counters with no duration are skipped", func(t *testing.T) {
rl := &TableRateLimit{ID: "no-durations", IsCalendarAligned: true, TokenLastReset: stale, RequestLastReset: stale}

assert.False(t, rl.AdoptCalendarAlignment(now))
assert.True(t, rl.TokenLastReset.Equal(stale))
})
}
33 changes: 33 additions & 0 deletions framework/configstore/tables/ratelimit.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,36 @@ func (rl *TableRateLimit) BeforeSave(tx *gorm.DB) error {

return nil
}

// AdoptCalendarAlignment re-anchors this rate limit's open counters onto the
// calendar grid they have just been told to follow, and reports whether either
// moved. Counterpart to TableBudget.AdoptCalendarAlignment; the reasoning for
// moving the boundary rather than resetting is documented there.
//
// The two counters are decided independently: they carry their own durations and
// their own LastReset values, so a monthly token limit can adopt the month
// boundary while an hourly request limit stays rolling in the same call.
// Usage is never cleared - the operator changed a schedule, not a quota.
func (rl *TableRateLimit) AdoptCalendarAlignment(now time.Time) bool {
if rl == nil || !rl.IsCalendarAligned {
return false
}
adopt := func(duration *string, lastReset *time.Time) bool {
if duration == nil || !IsCalendarAlignableDuration(*duration) {
return false
}
// Rate limits carry no quarter definition, so the fiscal start is not
// applicable here; see QuarterStartNotApplicable.
start := GetCalendarPeriodStart(*duration, now, QuarterStartNotApplicable)
if !start.After(*lastReset) {
return false
}
*lastReset = start
return true
}
// Both counters are evaluated; || would short-circuit and leave the second
// one on a stale boundary whenever the first adopted.
tokenMoved := adopt(rl.TokenResetDuration, &rl.TokenLastReset)
requestMoved := adopt(rl.RequestResetDuration, &rl.RequestLastReset)
return tokenMoved || requestMoved
}
10 changes: 7 additions & 3 deletions plugins/governance/budgetcycle_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -857,9 +857,13 @@ func TestDumpBudgetsCannotUndoOperatorReset(t *testing.T) {
// creates the budget at test time, so LastReset is always inside the current
// period. Both cases are pinned below so the difference is visible.
//
// Closing the gap needs either an activation marker or a dedicated write path,
// because UpdateBudget copies last_reset back from the stored row on every config
// write. Tracked as follow-up work; until then the docs describe this behaviour.
// The gap is closed by adoption: the owner handlers now call
// AdoptCalendarAlignmentInMemory on the switch-over, which moves each open window
// forward onto its boundary so the sweep below never finds it overdue. These
// assertions stay as they are on purpose - budgetResetTarget itself is unchanged,
// and it is precisely its "reset whatever is overdue" rule that makes adoption
// necessary. Read them as the reason the switch-over needs a step, not as a
// description of what an operator sees.
func TestEnablingCalendarAlignmentCanResetAtTheBoundaryAlreadyPassed(t *testing.T) {
store := newStandaloneStore(t)
now := time.Date(2026, time.February, 5, 12, 0, 0, 0, time.UTC)
Expand Down
70 changes: 70 additions & 0 deletions plugins/governance/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,13 @@ type GovernanceStore interface {
// ResetBudgetUsageInMemory clears one budget's usage on operator request,
// leaving its reset boundary untouched.
ResetBudgetUsageInMemory(ctx context.Context, budgetID string) (*configstoreTables.TableBudget, bool)
// AdoptCalendarAlignmentInMemory re-anchors a budget onto the calendar boundary
// it has just been told to follow, preserving its usage. Reports whether the
// boundary moved.
AdoptCalendarAlignmentInMemory(ctx context.Context, budgetID string, now time.Time) bool
// AdoptRateLimitCalendarAlignmentInMemory is the rate-limit counterpart,
// applying the rule to the token and request counters independently.
AdoptRateLimitCalendarAlignmentInMemory(ctx context.Context, rateLimitID string, now time.Time) bool
// DB sync for expired items
ResetExpiredRateLimits(ctx context.Context, resetRateLimits []*configstoreTables.TableRateLimit) error
ResetExpiredBudgets(ctx context.Context, resetBudgets []*configstoreTables.TableBudget) error
Expand Down Expand Up @@ -2127,6 +2134,69 @@ func (gs *LocalGovernanceStore) ResetBudgetUsageInMemory(ctx context.Context, bu
return reset, true
}

// AdoptCalendarAlignmentInMemory re-anchors a budget onto the calendar grid when
// its owner has just switched alignment on, and reports whether it moved.
//
// Called on the false-to-true transition only. Without it the switch is
// destructive: budgetResetTarget resets whenever WindowStart(now) is after
// LastReset, so a window that opened before the current boundary is instantly due
// and the next sweep clears its usage. See TableBudget.AdoptCalendarAlignment.
//
// The usage carried into the swap is the one observed inside the CAS loop, not a
// value read beforehand: a request bumping this budget between a read and the
// write would otherwise have its spend dropped. No reset generation is bumped
// here - nothing was reset, so an in-flight dump's rows are still accurate.
func (gs *LocalGovernanceStore) AdoptCalendarAlignmentInMemory(ctx context.Context, budgetID string, now time.Time) bool {
for {
raw, exists := gs.budgets.Load(budgetID)
if !exists || raw == nil {
return false
}
old, ok := raw.(*configstoreTables.TableBudget)
if !ok || old == nil {
return false
}
clone := *old
// The owner's flag has already been applied upstream, but the in-memory
// copy predates it, so stamp it here or adoption declines its own work.
clone.IsCalendarAligned = true
if !clone.AdoptCalendarAlignment(now) {
return false
}
clone.RefreshOverrideCyclesRemaining()
if gs.budgets.CompareAndSwap(budgetID, raw, &clone) {
gs.logger.Debug(fmt.Sprintf("Adopted budget %s onto its calendar boundary %s (usage %.2f preserved)",
budgetID, clone.LastReset, clone.CurrentUsage))
return true
}
}
}

// AdoptRateLimitCalendarAlignmentInMemory is the rate-limit counterpart, applying
// the same switch-over rule to the token and request counters independently.
func (gs *LocalGovernanceStore) AdoptRateLimitCalendarAlignmentInMemory(ctx context.Context, rateLimitID string, now time.Time) bool {
for {
raw, exists := gs.rateLimits.Load(rateLimitID)
if !exists || raw == nil {
return false
}
old, ok := raw.(*configstoreTables.TableRateLimit)
if !ok || old == nil {
return false
}
clone := *old
clone.IsCalendarAligned = true
if !clone.AdoptCalendarAlignment(now) {
return false
}
if gs.rateLimits.CompareAndSwap(rateLimitID, raw, &clone) {
gs.logger.Debug(fmt.Sprintf("Adopted rate limit %s onto its calendar boundaries (token %s, request %s)",
rateLimitID, clone.TokenLastReset, clone.RequestLastReset))
return true
}
}
}

// ResetExpiredBudgetsInMemory checks and resets budgets that have exceeded their reset duration.
// With no budgetIDs it scans every budget; with IDs it only checks those budgets.
// refreshReferences controls whether embedded owner references (VK, team, customer) are updated
Expand Down
Loading
Loading