Skip to content

fix(problem-deploy): make disruptions work in Lite mode (same-account injection) - #1711

Merged
susumutomita merged 1 commit into
mainfrom
fix/1710-lite-mode-disruptions
Jun 4, 2026
Merged

susumutomita merged 1 commit into
mainfrom
fix/1710-lite-mode-disruptions

Conversation

@susumutomita

Copy link
Copy Markdown
Owner

Summary

Lite mode で「障害を注入したのにフロントエンドがダウンしない」(fire は記録されるが注入が silently no-op) を修正します。Closes #1710。原因は 2 つの連動した問題でした。

  1. resolveDeployment が cross-account フィールドを必須にしていた: competitorRoleArn + externalIdParameterName が両方無いと undefined(→ no_deployment → no-op)。Lite は same-account deploy でどちらも持たないため、Lite の fire は常に no-op。→ 両フィールドを optional にし、COMPLETE な same-account row は target を返す。executor は既存の assumeCompetitorRole の same-account 経路(双方 absent → undefined creds)で Lambda 自身の credentials を使う。

  2. executor Lambda の role に ssm:SendCommand が無かった: 注入は assumed の競技者 AdministratorAccess role 経由という設計(ADR-031)。Lite には assume 先が無いので、ssm:SendCommand を Lambda 自身の role に、同一アカウントの instance + 標準 shell document(AWS-RunShellScript / SSM-SessionManagerRunShell)に scope して付与。SaaS は従来通り assumed role で注入するため本 grant は不使用。

lambda-invoke / cfn-stack-update kind の Lite 対応は同様の own-role grant が必要(follow-up)。ssm-run-command(サンプル red team)は本 PR で動作。

反映

problem-deploy backend の再デプロイで executor Lambda が更新されます。

Test plan

  • disruption-executor-store.test.ts: cross-account フィールド無しの Lite COMPLETE row が same-account target を返すことを pin(旧 "skip" テストを置換)。
  • disruption-executor-lambda.test.ts: 自前 role に ssm:SendCommand が付き同一アカウント instance + AWS-RunShellScript に scope、Invoke/UpdateStack は依然無いことを pin。
  • 既存 executor / contract テスト green。make harness / make before-commit(synth 含む)green。

Regression analysis

  • SaaS (cross-account): 両フィールドが揃えば従来通り target を返し、注入は assumed Admin role 経由のまま(own role の SendCommand 不使用)。挙動不変。
  • IAM scope: ssm:SendCommand は same-account(${stack.account})の instance + 標準 shell document に限定。Lite は単一テナント=organizer 自身の account のためブラスト半径は閉じる。

Physical impact

  • AWS::IAM::Policy(DisruptionExecutor Lambda role)に ssm:SendCommand ステートメント 追加 = UPDATE(in-place)。Lambda code 更新 = UPDATE。新規/削除リソースなし。

… injection)

Closes #1710

In Lite mode the disruption fire was recorded but injection silently no-op'd. Two
coupled causes:

1. resolveDeployment required competitorRoleArn + externalIdParameterName (cross-
   account fields) and returned undefined otherwise. Lite deployments are same-account
   and carry neither, so every Lite fire resolved to no_deployment. Those fields are
   now optional; a COMPLETE same-account row resolves to a target and the executor uses
   its own credentials via the existing same-account branch of assumeCompetitorRole.

2. The executor Lambda role had no ssm:SendCommand (injection rode the assumed
   competitor AdministratorAccess role). Lite has no role to assume, so grant
   ssm:SendCommand on the Lambda's own role, scoped to same-account instances + the
   standard shell document. SaaS still injects via the assumed role.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@susumutomita, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 25 minutes and 23 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 96943f1e-bbe4-453b-99e4-c5da078ec800

📥 Commits

Reviewing files that changed from the base of the PR and between f399603 and ab28102.

📒 Files selected for processing (5)
  • infrastructure/lib/problem-deploy/disruption-executor-lambda.ts
  • infrastructure/lib/problem-deploy/handlers/disruption-executor-handler/execute.ts
  • infrastructure/lib/problem-deploy/handlers/disruption-executor-handler/executor-store.ts
  • infrastructure/test/problem-deploy/disruption-executor-lambda.test.ts
  • infrastructure/test/problem-deploy/disruption-executor-store.test.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/1710-lite-mode-disruptions

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov

codecov Bot commented Jun 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.75%. Comparing base (f399603) to head (ab28102).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1711   +/-   ##
=======================================
  Coverage   92.75%   92.75%           
=======================================
  Files         434      434           
  Lines       11896    11899    +3     
  Branches     3654     3655    +1     
=======================================
+ Hits        11034    11037    +3     
  Misses        295      295           
  Partials      567      567           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@susumutomita
susumutomita merged commit d7533e6 into main Jun 4, 2026
10 checks passed
@susumutomita
susumutomita deleted the fix/1710-lite-mode-disruptions branch June 4, 2026 11:11
susumutomita added a commit that referenced this pull request Jun 4, 2026
…rows (#1712)

#1711 made resolveDeployment return same-account targets but assumed Lite rows
carry neither competitorRoleArn nor externalIdParameterName. In reality the
deploy-handler persists competitorRoleArn to the deployment row (deploy.ts:177)
while externalIdParameterName only rides the deploy event detail
(deploy.ts:259-261). So every COMPLETE row is asymmetric: role set, externalId
absent.

resolveDeployment then returned a target with a lone competitorRoleArn, which
assumeCompetitorRole rejects via its both-or-neither guard ("must be provided
together"). Lite disruptions still failed end-to-end — now throwing instead of
the previous no_deployment no-op, but injecting nothing either way.

Honor the same both-or-neither contract in resolveDeployment: emit the
cross-account fields only when both are present; otherwise resolve a
same-account target and let the executor inject with its own credentials (the
ssm:SendCommand grant added in #1711).

Verified against the live Lite deploy (hello-world-battle / frontend-down): the
COMPLETE row carries competitorRoleArn=TenkaCloud-local-deploy-Role and no
externalIdParameterName; the target EC2 (i-046414a66bfb65dd7) is SSM-managed and
Online, so once the command is sent the injection lands.

Closes #1710

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(problem-deploy): disruptions silently no-op in Lite mode (executor requires cross-account ExternalId)

1 participant