Skip to content

feat(disruptions): immediate + scheduled (N分後) red-team fire [ADR-037] - #1687

Merged
susumutomita merged 3 commits into
mainfrom
feat/disruption-scheduled-fire
Jun 3, 2026
Merged

susumutomita merged 3 commits into
mainfrom
feat/disruption-scheduled-fire

Conversation

@susumutomita

@susumutomita susumutomita commented Jun 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

Implements the scheduled-fire slice of ADR-037: an operator can fire a red-team disruption immediately or scheduled N minutes later. First of the {scheduled, recurring, while-undefended} timing modes.

Depends on #1686 (ADR-037 design). Merge #1686 first — this branch references [ADR-037] throughout and is the implementation of that ADR's scheduled mode.

Mechanism (zero new infra)

  • Fire schema gains timing (immediate | scheduled) + afterMinutes (1–1440), with cross-field refines.
  • The fire handler stamps afterMinutes into the published DisruptionFired detail and records scheduledFor in the audit row.
  • The executor, on afterMinutes>0, reuses the existing aws-scheduler one-shot (the revert mechanism) to re-invoke itself with {mode:"inject", detail} at T+N. The executor's scheduler role + CreateSchedule already exist for revert → no IAM/CDK change.
  • Operator UI: an immediate/scheduled SegmentedControl + minutes input (pre-filled from the disruption's defaultAfterMinutes); the fire audit table gains a "Scheduled for" column.

Idempotency (parity with the immediate path)

The immediate path is dedupe-safe because claimExecution runs in the same invocation that injects (EventBridge redelivery → duplicate). aws-scheduler is also at-least-once, so the deferred inject claims a distinct inject-phase key (EXEC#{requestId}#{teamId}#INJECT) at injection time → scheduler redelivery is fenced exactly like the immediate path.

Test plan

  • infra: disruption-execute / route / schedule-revert / executor-store / fire / routes / contract — 85 disruption tests green (defer, scheduled-inject, inject-phase claim + duplicate, parse guard, scheduledFor, distinct PK).
  • app-admin: DisruptionsPanel 18 tests (timing toggle, out-of-range disable, default pre-fill, scheduledFor column shows time vs "-"); 953 tests, coverage 100%.
  • make harness green, tsc green, biome clean.

Review trail

  • /review: approve (correct, conventions-clean, no security concerns; clean generalization of the scheduler mechanism).
  • /security-review: no findings — afterMinutes double-validated + bounded; scheduler payload server-built; schedule-name sanitized; tenant ownership/ExternalId path unchanged.
  • /simplify: diff is de-duplicating (extracts sendOneShot/oneShotAt/FireModal/teamPicker); altitude review found + fixed the inject-phase idempotency gap (commit 2).

Regression analysis

  • Immediate fire is byte-identical to before: afterMinutes absent → published detail / audit unchanged → immediate inject. Regression-pinned ("immediate omits afterMinutes").
  • Executor: afterMinutes-less fired events take the unchanged inject path; the new mode:"inject" route branch only matches the new scheduler payload.
  • Idempotency: event-phase and inject-phase claims use distinct PKs (no interference); fire-time claim still fences duplicate fired events to one schedule.
  • always-ends (ADR-029): each injection (immediate or deferred) still schedules a revert.

Physical impact

  • UPDATE: DisruptionExecutor + EventApi Lambda function code (bundled assets); frontend application-admin-console bundle.
  • NO-OP: CFn resource topology, IAM policies, DynamoDB capacity — unchanged (the inject-phase claim is a 1-WCU conditional Put to the existing Disruptions table; scheduler perms pre-exist for revert).

Relates #1417

Summary by CodeRabbit

Release Notes

  • New Features
    • Scheduled disruption injection: Configure execution delays (1–1440 minutes) for deferred fault injection
    • Audit logs now display scheduled execution timestamps alongside immediate fires
    • Immediate vs. scheduled timing modes with pre-filled default delays from disruption catalog
    • New column in disruption audit table showing scheduled execution times
    • Validation and error messaging for scheduling delay inputs

susumutomita and others added 3 commits June 3, 2026 21:35
ユーザー要件「障害を即座に実行と、スケジュール実行両方できるように」を実装。
ADR-037 timing モデルの第一弾 (scheduled)。

機構 (新 infra ゼロ):
- fire schema に timing(immediate|scheduled) + afterMinutes(1..1440) を追加 (cross-field refine)
- fire handler は afterMinutes を published Detail に乗せ、 audit に scheduledFor を記録
- executor は afterMinutes>0 の時、 既存 aws-scheduler one-shot を転用して inject を T+N に
  遅延予約 (mode:"inject")。 claim は fired-event 時に取得済 → 遅延 inject は再claim しない
- executor の scheduler role / CreateSchedule 権限は revert 用に既存 → IAM 変更なし
- operator UI に即座/N分後トグル + 分入力 (defaultAfterMinutes で pre-fill)

リファクタ: fire modal を FireModal に抽出 (form state を内包、 panel を薄く保つ)。

## Test plan
- infra: disruption-execute / route / schedule-revert / fire / routes / contract = 全 green
- app-admin: DisruptionsPanel 16 tests、 全 953 tests green、 coverage 100% 維持
- make harness green、 tsc green、 biome clean

## Regression analysis
- immediate fire は従来と完全同一 (afterMinutes 未指定 → published Detail / audit 不変、 即注入)。
- executor: afterMinutes 無しの fired event は従来どおり即注入。
- 冪等性不変: claim は fired-event 時に取得 → EventBridge 再配送は遅延予約より前に弾く。
- always-ends (ADR-029): 各注入に revert 予約は不変。

## Physical impact
- UPDATE: DisruptionExecutor / EventApi Lambda の関数コード (bundled asset) のみ。
- NO-OP: CFn リソース構成 / IAM ポリシー / DynamoDB capacity に差分なし。

Relates #1417
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…DR-037]

/simplify (altitude) finding: the scheduled-fire deferred-inject path dropped the
immediate path's idempotency guarantee. The immediate path is safe because
claimExecution runs in the SAME invocation that injects (EventBridge redelivery →
duplicate → no double-inject). The deferred mode:"inject" path skipped the claim, but
aws-scheduler is ALSO at-least-once: a Lambda error after sendDispatch → scheduler
retry → double inject.

Fix (parity with the established mechanism): executeScheduledInject claims a distinct
inject-phase key (EXEC#{requestId}#{teamId}#INJECT) at injection time, so scheduler
redelivery is deduped exactly like the immediate path dedupes EventBridge redelivery.
The fire-time event-phase claim is unchanged.

## Test plan
- disruption-execute / route / executor-store: 85 disruption tests green, tsc green,
  biome clean, make harness green

## Regression analysis
- Immediate path unchanged (event-phase claim). Deferred path now matches immediate
  path's at-least-once posture; event/inject claims use distinct PKs (no interference).

## Physical impact
- UPDATE: DisruptionExecutor Lambda code only. NO-OP: CFn / IAM / DynamoDB capacity.

Relates #1417
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/review follow-up: the scheduled-fire injection time was stored (audit row scheduledFor)
but not surfaced in the operator audit table — an operator could see a scheduled fire
in history but not WHEN it lands. Add a "Scheduled for" column (ja/en); immediate fires
render "-".

## Test plan
- DisruptionsPanel: audit-rows test covers both a scheduled row (shows the time) and an
  immediate row (shows "-"). app-admin coverage 100% (953 tests), tsc + biome + harness green.

## Regression analysis
- Display-only addition to the audit table. No API / data-shape change (scheduledFor was
  already on DisruptionAuditRow). Immediate fires unaffected (render "-").

## Physical impact
- NO-OP: frontend-only; no CFn / IAM / DynamoDB change.

Relates #1417
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 3, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR implements ADR-037 scheduled disruption injection, enabling Red Team chaos fault injection to be deferred by a configurable delay. The feature spans request validation, fire handler persistence, executor routing with phase-aware idempotency, AWS Scheduler integration, and a frontend fire modal with timing selection UI.

Changes

Scheduled Disruption Injection

Layer / File(s) Summary
Request contract and validation schema
apps/application-admin-console/src/api/disruptions-client.ts, infrastructure/lib/problem-deploy/handlers/event-handler/types.ts, infrastructure/lib/problem-deploy/handlers/event-handler/disruption-types.ts, infrastructure/lib/problem-deploy/handlers/event-handler/routes/disruptions.ts
DisruptionTiming type and optional timing/afterMinutes fields added to fire requests. Backend Zod schema validates afterMinutes (1–1440 min range) is present only when timing is "scheduled". DisruptionAuditRow gains optional scheduledFor timestamp for scheduled injection tracking.
Fire request processing and audit persistence
infrastructure/lib/problem-deploy/handlers/event-handler/disruption-fire.ts, infrastructure/test/problem-deploy/disruption-fire.test.ts
Event handler computes scheduledFor from afterMinutes and persists it into audit rows. EventBridge Detail conditionally includes afterMinutes only for scheduled fires. Tests verify immediate fires omit afterMinutes and scheduled fires include both EventBridge and audit scheduledFor fields.
Executor routing and phase-aware idempotency
infrastructure/lib/problem-deploy/handlers/disruption-executor-handler/route.ts, infrastructure/lib/problem-deploy/handlers/disruption-executor-handler/executor-store.ts, infrastructure/test/problem-deploy/disruption-route.test.ts, infrastructure/test/problem-deploy/disruption-executor-store.test.ts
Executor Lambda router adds explicit mode: "inject" path for AWS Scheduler redelivery alongside EventBridge direct inject and revert routes. parseDisruptionFiredDetail parses optional afterMinutes (positive finite only). claimExecution now accepts optional phase parameter ("event" default, "inject") to use distinct DynamoDB idempotency keys (appending #INJECT for inject phase) and prevent duplicate scheduled injections.
Immediate vs scheduled execution orchestration
infrastructure/lib/problem-deploy/handlers/disruption-executor-handler/execute.ts, infrastructure/test/problem-deploy/disruption-execute.test.ts
Extracted injectAndScheduleRevert helper unifies inject+revert workflows. executeDisruptionAction claims "event" phase; if afterMinutes is set, schedules delayed inject and returns { kind: "scheduled" }, otherwise executes immediate injection. New exported executeScheduledInject claims "inject" phase for redelivered deferred injections and executes the inject+revert helper.
AWS Scheduler integration for delayed injection
infrastructure/lib/problem-deploy/handlers/disruption-executor-handler/schedule-revert.ts, infrastructure/lib/problem-deploy/handlers/disruption-executor-handler/index.ts, infrastructure/test/problem-deploy/disruption-schedule-revert.test.ts
Refactored scheduling: sanitizeScheduleName, oneShotAt, and sendOneShot helpers centralize AWS Scheduler configuration. New injectScheduleName and scheduleInject create one-shot schedules for deferred inject, computing UTC fire times and bumping detail.firedAt for re-invocation. Lambda DI wires scheduleInject dependency into executor.
Frontend fire modal and timing configuration
apps/application-admin-console/src/pages/event-detail/DisruptionsPanel.tsx, apps/application-admin-console/src/i18n/locales/en.json, apps/application-admin-console/src/i18n/locales/ja.json, apps/application-admin-console/test/pages/event-detail/DisruptionsPanel.test.tsx
DisruptionsPanel refactored to extract fire UI into FireModal component managing scope/team/timing form state and buildFireRequest helper. Timing validation enforces min/max afterMinutes bounds. Audit table gained scheduledFor column. i18n strings added for timing labels, descriptions, validation messages, and scheduled fire notifications in English and Japanese. Tests cover default immediate behavior, scheduled timing submission, defaultAfterMinutes pre-fill, and out-of-range validation.

Sequence Diagram(s)

See the diagram embedded in the hidden review stack artifact showing immediate vs scheduled injection flows through the executor Lambda and AWS Scheduler.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

  • susumutomita/TenkaCloud#1646: Extends the same disruption executor Lambda orchestration paths established in the ADR-031 cross-account foundation, with ADR-037 layering delayed injection scheduling on top.
  • susumutomita/TenkaCloud#889: Introduced the original Red Team disruption "fire" flow; this PR extends it with timing/afterMinutes inputs and scheduledFor audit persistence through the existing handler and type surfaces.

Poem

🐰 A chaos engineer's dream, now deferred,
Schedule your disruptions, let time be your word.
Scheduled and scoped, with AWS grace,
Faults flow in minutes, not running in haste. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 68.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main feature: support for immediate and scheduled (N minutes later) red-team disruption firing per ADR-037.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/disruption-scheduled-fire

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@susumutomita
susumutomita enabled auto-merge (squash) June 3, 2026 13:54
@susumutomita
susumutomita merged commit fae81da into main Jun 3, 2026
7 of 8 checks passed
@susumutomita
susumutomita deleted the feat/disruption-scheduled-fire branch June 3, 2026 13:58
@codecov

codecov Bot commented Jun 3, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.71%. Comparing base (23c7062) to head (f3a1981).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1687      +/-   ##
==========================================
+ Coverage   92.68%   92.71%   +0.02%     
==========================================
  Files         431      431              
  Lines       11792    11834      +42     
  Branches     3620     3641      +21     
==========================================
+ Hits        10930    10972      +42     
  Misses        295      295              
  Partials      567      567              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant