Skip to content

fix(transactions): scope storage fault injection - #10545

Merged
ReubenBond merged 1 commit into
dotnet:mainfrom
ReubenBond:rb-fix-transaction-fault-recovery
Aug 12, 2026
Merged

ReubenBond merged 1 commit into
dotnet:mainfrom
ReubenBond:rb-fix-transaction-fault-recovery

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Aug 12, 2026 •

Copy link
Copy Markdown
Member

Fixes #9551

A controlled BeforePrepareAndCommit storage fault could be armed while an older cleanup write was still pending. That unrelated batch consumed the one-shot fault, so queue restore completed without correlation to the new transaction and the recovery observer waited until its watchdog.

Snapshot transaction IDs from commit records and prepared states before each provider write, then require BeforePrepareAndCommit faults to match the armed transaction. Prepare and confirm retain next-write behavior because those provider batches are not always identifiable. Azure and DynamoDB wrappers share the same behavior, with regression coverage for unrelated and unidentified batches, all storage fault types, and one-shot consumption.

All 10 Azure fault-injection matrix cases pass on both net8.0 and net10.0 with Azurite.

Microsoft Reviewers: Open in CodeFlow

Prevent cleanup writes from consuming transaction-specific prepare-and-commit faults by matching identifiable storage batches to the armed transaction.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 12, 2026 03:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses flaky transaction fault-injection behavior in the Transactions TestKit by ensuring one-shot storage faults armed for BeforePrepareAndCommit are scoped to the intended transaction, preventing unrelated/older storage writes from consuming the fault.

Changes:

  • Added transaction-id-aware arming and injection APIs for controlled storage fault injectors, with optional transaction matching.
  • Updated Azure Table and DynamoDB fault-injection storage wrappers to snapshot transaction IDs from commit records/prepared state and pass them into injector hooks.
  • Added regression tests validating scoped faults ignore unrelated batches and unscoped faults still apply to unidentified batches.
Show a summary per file
File Description
test/Transactions/Orleans.Transactions.Tests/FaultInjectionControlTests.cs Adds regression tests for scoped vs unscoped storage fault behavior.
src/Orleans.Transactions.TestKit.Base/FaultInjection/ControlledInjection/SimpleAzureStorageExceptionInjector.cs Implements transaction-scoped matching and one-shot consumption protection for storage faults.
src/Orleans.Transactions.TestKit.Base/FaultInjection/ControlledInjection/IControlledFaultInjector.cs Introduces internal scoped injector contract plus helper/extension methods to arm and pass transaction IDs.
src/Orleans.Transactions.TestKit.Base/FaultInjection/ControlledInjection/FaultInjectionTransactionReource.cs Uses the new arming helper and enables transaction-match requirement for BeforePrepareAndCommit.
src/Orleans.Transactions.TestKit.Base/FaultInjection/ControlledInjection/FaultInjectionDynamoDBTransactionStateStorage.cs Captures transaction IDs and routes them into injector Before/AfterStore hooks.
src/Orleans.Transactions.TestKit.Base/FaultInjection/ControlledInjection/FaultInjectionAzureTableTransactionStateStorage.cs Captures transaction IDs and routes them into injector Before/AfterStore hooks.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 6/6 changed files
  • Comments generated: 0
  • Review effort level: Lite

@ReubenBond
ReubenBond merged commit c2a7530 into dotnet:main Aug 12, 2026
136 of 138 checks passed
@ReubenBond
ReubenBond deleted the rb-fix-transaction-fault-recovery branch August 12, 2026 14:05
This was referenced Aug 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky Test: Orleans.Transactions.Azure.Tests.TransactionFaultInjectionTests.MultiGrainWriteTransaction_FaultInjection

2 participants