Repository navigation
docs(flare-workflow): design-spec for Restate-style loop durability #491
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| # Design-spec: should `StepMode::Loop` adopt Restate's per-step durability primitive? | ||
|
|
||
| > Item #115. Written recommendation, no code change — the gap identified here | ||
| > is real but not urgent (item #112 already ships correctly without it via a | ||
| > domain-level workaround). Recorded as a design doc so the concrete shape and | ||
| > cost estimate exist for whoever picks this up next. | ||
|
|
||
| Scope: the one gap named in the dispatch — `execute_loop` journals a single | ||
| `JournalEntry::StepRun` after the whole loop exits, not per iteration — not a | ||
| general "should flare-workflow become Restate" question. | ||
|
|
||
| ## The gap, precisely | ||
|
|
||
| `src/engine.rs::execute_loop` (lines ~1182-1302): | ||
|
|
||
| - `for iter in 1..=*max_iterations` runs the step executor once per | ||
| iteration, persisting `ctx.data`/`input`/`output`/`variables` into | ||
| `WorkflowState` via `state_store.update()` after every iteration (this part | ||
| is already durable — it's item #112's basis for putting counters in | ||
| `ctx.data`). | ||
| - But `state_store.append_journal(..., JournalEntry::StepRun { .. })` is only | ||
| called **once**, after the `for` loop exits (success) or on failure inside | ||
| the loop. There is no per-iteration journal entry. | ||
| - Recovery (`execute_workflow`, lines 483-533) memoizes a step by looking for | ||
| the *last* `JournalEntry::StepRun` with a matching `step_id`. If none | ||
| exists (crash mid-loop, loop never finished), the step is **not** memoized | ||
| and `execute_loop` re-enters from `iter = 1` with a fresh `max_iterations` | ||
| budget on resume — even though N-1 iterations already ran and their | ||
| `ctx.data` side effects already landed in `WorkflowState`. | ||
|
|
||
| Net effect: a crash on iteration 7 of a 10-iteration loop doesn't replay | ||
| iterations 1-6 for free from the journal — it silently restarts the loop's | ||
| iteration *counter* (though whatever `ctx.data` mutations survived stay | ||
| persisted, since those went through `state_store.update()`, not the journal). | ||
| This is a real but narrow gap: it affects only iteration-count/budget | ||
| bookkeeping across a crash, not data loss of `ctx.data` itself. | ||
|
|
||
| ## How Restate actually solves this (verified against docs.restate.dev) | ||
|
|
||
| Restate does **not** special-case loops. Its durable-execution model | ||
| (`docs.restate.dev/concepts/durable_execution`, `.../guides/request-lifecycle`) | ||
| is: every `ctx.run()` (and `ctx.sleep`/`ctx.call`/awakeable) call is journaled | ||
| as an individual entry with a stored result. On failure, **the entire handler | ||
| function is replayed from the top**; each time execution reaches a `ctx.*` | ||
| call, the SDK checks the journal by call order — if a result is already | ||
| recorded, it's returned instantly without re-running the closure; once replay | ||
| reaches the first entry with no recorded result, real (live) execution | ||
| resumes from there. | ||
|
|
||
| Restate's own "Evaluation Feedback Loop" pattern | ||
| (`docs.restate.dev/ai/patterns/workflow-evaluator`) is structurally identical | ||
| to `StepMode::Loop`: a bounded `for` loop calling `ctx.run()` once per | ||
| iteration until a pass condition or `max_iterations`. Their docs state | ||
| plainly: *"Each generate and evaluate call is persisted in the journal. If | ||
| the process crashes after a successful generation but before evaluation, the | ||
| generated code is replayed from the journal without calling the LLM again."* | ||
| This is achieved with **zero special loop handling** — it's an emergent | ||
| property of the general whole-function-replay engine (every `ctx.run()` call | ||
| anywhere in the handler is journaled the same way, loop or not). | ||
|
|
||
| So Restate's actual primitive here is the general one: full deterministic | ||
| replay of arbitrary handler code, journaling every side-effecting call as it | ||
| happens. That general mechanism is precisely item #112's "fork (b)" — the | ||
| big, explicitly-out-of-scope rebuild (arbitrary imperative code, deterministic | ||
| replay of a whole function/script). | ||
|
|
||
| ## Answering the three questions | ||
|
|
||
| **1. Does Restate have a primitive that closes this gap without requiring | ||
| fork (b)?** | ||
|
|
||
| Not as a distinct, narrower primitive — Restate's own solution to this exact | ||
| shape of problem *is* fork (b), just applied at small scale. There's no | ||
| separate "loop checkpoint" API in Restate distinct from `ctx.run()` + replay. | ||
|
|
||
| But `flare-workflow` doesn't need to import Restate's *general* mechanism to | ||
| close this *specific* gap, because `execute_loop`'s shape is far more | ||
| constrained than an arbitrary Restate handler: | ||
|
|
||
| - The loop body is always exactly one call: `step.executor.execute()`. There | ||
| is no other branching, no other side-effecting calls interleaved between | ||
| iterations, no arbitrary control flow to rediscover. | ||
| - The exit conditions (`until` substring match, `max_iterations`) are static | ||
| data from `StepDefinition`, already known before the loop starts — nothing | ||
| about *why* the loop is shaped this way needs to be re-derived by replaying | ||
| code. | ||
| - Because of that, `flare-workflow` can get the *same resume behavior* | ||
| Restate gets from full replay-and-skip, but without an interpreter/replay | ||
| step at all: just read the journal once at loop entry, compute how many | ||
| iterations are already durably recorded, and start the Rust `for` loop's | ||
| counter from `resume_from` instead of `1`. This is the CompletableEntry | ||
| idea (`JournalEntry`'s own doc comment already invokes it) applied one | ||
| level deeper — into a single step's internal iteration count — rather than | ||
| Restate's general per-call, whole-function version of it. | ||
|
|
||
| So: yes, a *scoped-down instance* of Restate's underlying idea (per-operation | ||
| journaling, resume-from-first-incomplete-entry) closes this cleanly. What | ||
| doesn't transfer is Restate's actual mechanism (whole-function replay) — | ||
| that part is legitimately fork (b) and stays out of scope. | ||
|
|
||
| **2. Concrete shape of the change, and cost estimate** | ||
|
|
||
| Cheap, bounded addition — not fork (b) in disguise. Confirmed against the | ||
| actual schema: the SQLite `journal` table (`sqlite_store.rs`) is already | ||
| generic (`run_id, seq, entry_type TEXT, payload TEXT`) — a new `JournalEntry` | ||
| variant needs **no migration**, just a new Rust enum arm. | ||
|
|
||
| - `types.rs`: add `JournalEntry::LoopIteration { step_id: StepId, iteration: | ||
| u32, output: Vec<u8> }` as a **new, separate** variant — not reusing | ||
| `StepRun`. Keeping it separate matters: `execute_workflow`'s DAG-level | ||
| memoization (lines 500-514) pattern-matches only on `StepRun`/`Sleep` to | ||
| decide a step is complete; a `LoopIteration` entry must never be mistaken | ||
| for that terminal signal, or a mid-loop crash would wrongly memoize the | ||
| step as done. Add matching `is_completed`/`entry_type` arms (~10 LOC). | ||
| - `execute_loop`: at the top, read the run's journal, filter | ||
| `LoopIteration` entries for this `step.id`, take the max `iteration` found | ||
| (default 0), seed `current_output`/`executed` from it, and change the loop | ||
| bound from `for iter in 1..=*max_iterations` to | ||
| `for iter in (resume_from + 1)..=*max_iterations`. After each successful | ||
| iteration's existing `state_store.update()` call, add one | ||
| `state_store.append_journal(run_id, JournalEntry::LoopIteration { .. })` | ||
| call. Leave the failure path and the final terminal `StepRun` append (which | ||
| still marks the step complete for DAG purposes) untouched. ~30-40 LOC. | ||
|
Comment on lines
+115
to
+123
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win Check the resumed checkpoint before running another iteration. If iteration 🤖 Prompt for AI Agents |
||
| - No `StateStore` trait changes — `append_journal`/`journal` already exist | ||
| and are exactly what's needed. | ||
| - New test: crash-mid-loop resume — run N iterations, drop the run before | ||
| the terminal `StepRun` is written (or simulate via directly appending | ||
| `LoopIteration` entries then re-invoking `execute_loop`), assert it resumes | ||
| from `N+1` and doesn't re-run 1..N. Natural home: alongside the existing | ||
| loop coverage in `tests/semantics_test.rs`. | ||
|
|
||
| Total: roughly 50-70 LOC plus one test. No schema migration, no new trait | ||
| methods, no changes outside `flare-workflow`. | ||
|
|
||
| Known trade-off, same class of risk the engine already accepts elsewhere: | ||
| if a crash lands between the per-iteration `state_store.update()` and the | ||
| new `append_journal` call, that one iteration's step executor may re-run on | ||
| resume (its `ctx.data` effects already landed, but the checkpoint didn't). | ||
| This is the identical window `execute_step_with_retry` already has today | ||
| between its own `state_store.update()` and `append_journal` calls (lines | ||
| 1033-1072) — not a new risk class, just the existing single-step guarantee | ||
| applied per-iteration instead of per-step. | ||
|
Comment on lines
+135
to
+142
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift Make the state update and loop checkpoint recover as one consistent unit. The proposed order updates Use an atomic state/checkpoint operation, or persist and restore a state snapshot that matches each checkpoint. Add failure injection at this exact boundary. 🤖 Prompt for AI Agents |
||
|
|
||
| **3. If no clean primitive existed** — moot; #2 gives a concrete, cheap path. | ||
| The domain-level workaround from #112 (counters in `ctx.data`) remains | ||
| correct and doesn't need to change — this closes the engine-level gap | ||
| *underneath* it, it doesn't replace it. Domains that already rely on | ||
| `ctx.data` counters keep working unchanged; they'd simply also start | ||
| resuming their iteration budget correctly across a crash instead of | ||
| restarting it. | ||
|
|
||
| ## Recommendation | ||
|
|
||
| Worth doing as a small, self-contained follow-up (not urgent, not blocking | ||
| anything): add `JournalEntry::LoopIteration` and journal each successful | ||
| iteration in `execute_loop`, resuming the iteration counter from the journal | ||
| on re-entry. This directly closes the per-iteration durability gap, costs | ||
| ~50-70 LOC with no schema migration and no trait changes, and stays entirely | ||
| within the existing "journal a completable operation, resume from the first | ||
| incomplete one" pattern the engine already uses for `StepRun`/`Sleep`/ | ||
| `WaitEvent` — it does not require adopting Restate's general whole-function | ||
| deterministic-replay engine (fork (b)), because `execute_loop`'s shape is | ||
| static and known ahead of time rather than something that needs rediscovery | ||
| via replay. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
Repository: getappz/agentflare
Length of output: 50374
🏁 Script executed:
Repository: getappz/agentflare
Length of output: 50374
🌐 Web query:
serde_json externally tagged enum deserialization unknown variant fails by default💡 Result:
In Serde, deserializing an externally tagged enum with an unknown variant fails by default, and there is no built-in attribute to automatically ignore unknown variants or map them to a "catch-all" variant for this representation [1][2][3]. This behavior occurs because the externally tagged representation requires the deserializer to know the variant ahead of time to parse the associated content correctly [4]. Because Serde's default implementation strictly enforces the schema defined by the enum variants, any tag not present in the definition is treated as an error [1][2]. Common workarounds include: 1. Custom Deserializer: Implement the Deserialize trait manually for your enum. Inside the deserialize method, you can deserialize the input into a serde_json::Value (or similar intermediate representation), inspect the tag, and manually handle unknown variants by returning a custom "Unknown" variant or skipping the data [3]. 2. Untagged "Wrapper" Enum: You can wrap your target enum in an untagged enum that tries to deserialize into your primary enum first, and if that fails, deserializes into a catch-all type (like serde_json::Value or a map) [1]. 3. Third-party Crates or Patterns: For specific use cases (like collections of enums), developers often use patterns that filter out invalid items during the collection deserialization process (e.g., using a wrapper struct that deserializes into an Option and then filtering the results) [5]. Note that the #[serde(other)] attribute is specifically designed for internally or adjacently tagged enums and does not function as a catch-all for externally tagged enums [6][7]. Using it on an externally tagged enum may compile, but it will not provide the desired fallback behavior [7].
Citations:
#[serde(other)]and externally tagged enum serde-rs/serde#2010🏁 Script executed:
Repository: getappz/agentflare
Length of output: 2293
🏁 Script executed:
Repository: getappz/agentflare
Length of output: 2227
Restrict “no migration” to the SQL schema and define mixed-version behavior.
The current
InMemoryStoreandSqliteStoreneed noStateStoreor table-schema change. However,SqliteStore::journal()deserializes every payload as the currentJournalEntry; an older worker rejects a persistedLoopIterationas an unknown enum variant and cannot recover the run. Replace “no migration” with “no SQL schema migration” and specify a compatible rollout or journal-format fallback.🤖 Prompt for AI Agents