feat(ci): acknowledge rerun-test commands

.github/workflows/comment-ci-command.yml

@@ -16,6 +16,7 @@ jobs:
       github.event.comment.user.type == 'User'
     outputs:
       capability: ${{ steps.authorize.outputs.capability }}
+      success_reaction: ${{ steps.authorize.outputs.success_reaction }}
     runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:
@@ -37,11 +38,13 @@ jobs:
           CI_COMMAND_PREFLIGHT: "true"
         run: python3 .github/workflows/scripts/comment_ci_command.py
 
-      - name: Reject an unknown capability
+      - name: Reject unknown routing outputs
         if: >-
-          steps.authorize.outputs.capability != 'none' &&
-          steps.authorize.outputs.capability != 'issues' &&
-          steps.authorize.outputs.capability != 'actions'
+          (steps.authorize.outputs.capability != 'none' &&
+           steps.authorize.outputs.capability != 'issues' &&
+           steps.authorize.outputs.capability != 'actions') ||
+          (steps.authorize.outputs.success_reaction != 'none' &&
+           steps.authorize.outputs.success_reaction != '+1')
         run: exit 1
 
       - name: Mint the issues-scoped App token
@@ -96,3 +99,33 @@ jobs:
           CI_COMMAND_API_TOKEN: ${{ steps.actions-token.outputs.token }}
           CI_COMMAND_POLICY_PATH: .github/workflows/policies/comment-command-access.json
         run: python3 .github/workflows/scripts/comment_ci_command.py
+
+  acknowledge-command:
+    needs: [handle-command, actions-command]
+    if: >-
+      needs.actions-command.result == 'success' &&
+      needs.handle-command.outputs.success_reaction == '+1'
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Check out the trusted handler
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        with:
+          ref: ${{ github.sha }}
+          persist-credentials: false
+          sparse-checkout: .github/workflows/scripts/comment_ci_command.py
+          sparse-checkout-cone-mode: false
+
+      - name: Mint the reaction-scoped App token
+        id: reaction-token
+        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1
+        with:
+          client-id: ${{ vars.CI_COMMAND_APP_CLIENT_ID }}
+          private-key: ${{ secrets.CI_COMMAND_APP_PRIVATE_KEY }}
+          permission-issues: write
+
+      - name: Acknowledge the successful command
+        env:
+          CI_COMMAND_API_TOKEN: ${{ steps.reaction-token.outputs.token }}
+          CI_COMMAND_ACKNOWLEDGE: "true"
+        run: python3 .github/workflows/scripts/comment_ci_command.py

.github/workflows/scripts/comment_ci_command.py

@@ -89,6 +89,7 @@ class CommandSpec(NamedTuple):
     allows_user_ids: bool
     audit_key: str
     audit_value: object
+    success_reaction: str
 
 
 class CommandContext(NamedTuple):
@@ -286,7 +287,7 @@ def __init__(self, token):
             raise CommentCommandError("GitHub API token is missing")
         self.token = token
 
-    def _request(self, path, *, method="GET", payload=None, expected_status=None, expect_json=True):
+    def _request(self, path, *, method="GET", payload=None, expected_statuses=None, expect_json=True):
         data = None
         if payload is not None:
             data = json.dumps(payload, separators=(",", ":")).encode("utf-8")
@@ -303,10 +304,9 @@ def _request(self, path, *, method="GET", payload=None, expected_status=None, ex
         )
         try:
             with urllib.request.urlopen(request, timeout=15) as response:
-                if expected_status is not None and response.status != expected_status:
-                    raise CommentCommandError(
-                        f"GitHub API returned HTTP {response.status}; expected {expected_status}"
-                    )
+                if expected_statuses is not None and response.status not in expected_statuses:
+                    expected = " or ".join(str(status) for status in expected_statuses)
+                    raise CommentCommandError(f"GitHub API returned HTTP {response.status}; expected {expected}")
                 body = response.read()
                 if not expect_json:
                     if body.strip():
@@ -401,7 +401,7 @@ def rerun_failed_jobs(self, run_id):
         self._request(
             f"/repos/{REPOSITORY}/actions/runs/{run_id}/rerun-failed-jobs",
             method="POST",
-            expected_status=201,
+            expected_statuses=(201,),
             expect_json=False,
         )
 
@@ -411,10 +411,20 @@ def create_workflow_dispatch(self, workflow_file, ref, inputs):
             f"/repos/{REPOSITORY}/actions/workflows/{encoded_workflow}/dispatches",
             method="POST",
             payload={"ref": ref, "inputs": inputs},
-            expected_status=204,
+            expected_statuses=(204,),
             expect_json=False,
         )
 
+    def add_comment_reaction(self, comment_id, content):
+        result = self._request(
+            f"/repos/{REPOSITORY}/issues/comments/{comment_id}/reactions",
+            method="POST",
+            payload={"content": content},
+            expected_statuses=(200, 201),
+        )
+        if not isinstance(result, dict) or result.get("content") != content:
+            raise CommentCommandError("GitHub API did not confirm the comment reaction")
+
 
 def _validate_live_pull(pull, pull_number):
     if not isinstance(pull, dict) or pull.get("number") != pull_number:
@@ -847,6 +857,7 @@ def _run_file_value(request):
         True,
         "label",
         _add_label_resource,
+        "none",
     ),
     ClearLabels: CommandSpec(
         "clear_labels",
@@ -858,6 +869,7 @@ def _run_file_value(request):
         False,
         "command",
         _clear_command_value,
+        "none",
     ),
     RerunFailedCI: CommandSpec(
         "rerun_failed_ci",
@@ -869,6 +881,7 @@ def _run_file_value(request):
         False,
         "command",
         _rerun_command_value,
+        "none",
     ),
     RunTestFile: CommandSpec(
         "run_test_file",
@@ -880,6 +893,7 @@ def _run_file_value(request):
         False,
         "test_file",
         _run_file_value,
+        "+1",
     ),
 }
 
@@ -933,26 +947,47 @@ def authorize_policy(event, policy, api):
     return pull_number, actor_id, request, spec
 
 
-def _write_capability(capability):
+def acknowledge_event(event, api):
+    pull_number, actor_id, _, request = parse_event(event)
+    spec = _command_spec(request)
+    if spec.success_reaction == "none":
+        raise CommentCommandError("command does not define a success reaction")
+    comment_id = _positive_int(event["comment"].get("id"), "comment ID")
+    api.add_comment_reaction(comment_id, spec.success_reaction)
+    return {
+        "actor_id": actor_id,
+        "decision": "ALLOW_SUCCESS_REACTION_CONFIRMED",
+        "pull_number": pull_number,
+        "reaction": spec.success_reaction,
+    }
+
+
+def _write_routing(capability, success_reaction):
     if capability not in {"none", "issues", "actions"}:
         raise CommentCommandError("command registry selected an unknown capability")
+    if success_reaction not in {"none", "+1"}:
+        raise CommentCommandError("command registry selected an unknown success reaction")
     output_path = os.environ.get("GITHUB_OUTPUT")
     if not output_path:
         return
     try:
         with Path(output_path).open("a", encoding="utf-8") as output:
             output.write(f"capability={capability}\n")
+            output.write(f"success_reaction={success_reaction}\n")
     except OSError as error:
         raise CommentCommandError(f"cannot write GITHUB_OUTPUT: {error}") from error
 
 
 def main():
     try:
         event = load_json(os.environ["GITHUB_EVENT_PATH"])
-        if os.environ.get("CI_COMMAND_PREFLIGHT") == "true":
+        if os.environ.get("CI_COMMAND_ACKNOWLEDGE") == "true":
+            api = GitHubAPI(os.environ["CI_COMMAND_API_TOKEN"])
+            result = acknowledge_event(event, api)
+        elif os.environ.get("CI_COMMAND_PREFLIGHT") == "true":
             pull_number, actor_id, _, request = parse_event(event)
             if request is None:
-                _write_capability("none")
+                _write_routing("none", "none")
                 print(
                     json.dumps(
                         {
@@ -965,14 +1000,12 @@ def main():
                     )
                 )
                 return 0
-
-        policy = load_policy(os.environ["CI_COMMAND_POLICY_PATH"])
-        api = GitHubAPI(os.environ["CI_COMMAND_API_TOKEN"])
-        if os.environ.get("CI_COMMAND_PREFLIGHT") == "true":
+            policy = load_policy(os.environ["CI_COMMAND_POLICY_PATH"])
+            api = GitHubAPI(os.environ["CI_COMMAND_API_TOKEN"])
             pull_number, actor_id, request, spec = authorize_policy(event, policy, api)
             authorization = {"actor_id": actor_id, "pull_number": pull_number}
             authorization[spec.audit_key] = spec.audit_value(request)
-            _write_capability(spec.capability)
+            _write_routing(spec.capability, spec.success_reaction)
             print(
                 json.dumps(
                     authorization,
@@ -981,7 +1014,10 @@ def main():
                 )
             )
             return 0
-        result = process_event(event, policy, api)
+        else:
+            policy = load_policy(os.environ["CI_COMMAND_POLICY_PATH"])
+            api = GitHubAPI(os.environ["CI_COMMAND_API_TOKEN"])
+            result = process_event(event, policy, api)
     except CommentCommandError as error:
         print(f"::error::{error}")
         return 1

docs/ci/01-label.md

@@ -65,6 +65,8 @@ GitHub can omit `pull_requests` from fork workflow-run payloads. For a fork, the
 
 Post `/rerun-test <test-file>` as the entire comment to run one registered test file on the PR's current head, e.g. `/rerun-test tests/e2e/precision/test_hf_attention_cp_relayout.py`. Despite the name, this dispatches a fresh workflow and does not require the test to have run previously. The handler accepts only a repo-relative path under the registry scan roots (`tests/e2e`, `tests/fast`, `tests/fast-gpu`, `tests/ci`), then dispatches the fixed default-branch `.github/workflows/run-ci-file.yml` with the PR number, exact head SHA, and file path as inputs.
 
+After GitHub confirms the workflow dispatch, the App reacts to the original command with 👍. The reaction acknowledges that the request was accepted; the final pass or failure remains in the `Rerun Test` Actions run and is not posted back as a PR comment.
+
 An explicit file request is the selection: domain labels and the nightly cadence gate do not apply, while a symlinked or `disabled` test, an unregistered path, a ROCm-only registration, or a file with more than one CPU/CUDA registration fails the resolve job instead of silently running nothing.
 
 The dispatched workflow and its reusable workflows come from the reviewed default-branch commit. A trusted resolver reads the separately checked-out PR snapshot without importing it and maps the requested registration to a fixed CPU/CUDA suite, runner, image, and timeout. The execution job then checks out that same exact SHA and uses the requested path as its command entrypoint: bounded `pytest` for CPU or bounded `python3` for CUDA. It does not depend on a runner implementation in the PR snapshot, so the command also works for PRs created before this command exists.
@@ -75,13 +77,13 @@ A file run honors the PR body's `ci-megatron-pr` and `ci-sglang-pr` pins; CUDA f
 
 The workflow is disabled by default. Workflow owners may set the repository variable `CI_COMMAND_APP_ENABLED=true` only after completing these steps:
 
-1. Create a GitHub App, install it only on `radixark/miles`, and grant `Pull requests: read`, `Issues: write`, and `Actions: write`; do not grant `Contents: write`. Each request mints only one capability-specific token: label commands request `Issues: write`, while rerun and file-run commands request `Actions: write`.
+1. Create a GitHub App, install it only on `radixark/miles`, and grant `Pull requests: read`, `Issues: write`, and `Actions: write`; do not grant `Contents: write`. Mutation tokens remain capability-specific: label commands request `Issues: write`, while rerun and file-run commands request `Actions: write`. After a successful `/rerun-test` dispatch, a separate token requests only `Issues: write` to add the 👍 reaction.
 2. Store the App client ID in the repository variable `CI_COMMAND_APP_CLIENT_ID` and its private key in the repository secret `CI_COMMAND_APP_PRIVATE_KEY`.
 3. Protect the final bytes under `.github/workflows/` that implement the command gateway—its workflows, handler, and policy: require code-owner review, enable stale-review dismissal or last-push approval, and explicitly accept administrators who can still bypass the rule as external trust roots.
 4. In the target repository, compare manually adding a test label with adding the same label through the App. Confirm that both trigger the expected CUDA, ROCm, and held-run approval consumers.
    Then run `/clear-labels`; confirm that it removes only the CI control labels and does not start another CUDA, ROCm, or held-run approval workflow.
    Then create a disposable failed run on the current PR head and confirm that `/rerun-failed-ci` reruns only its failed jobs and dependent jobs.
-   Finally, after `run-ci-file.yml` exists on the default branch, post `/rerun-test` with one registered CUDA test file and confirm the dispatched run uses that file as the command entrypoint from the recorded PR head SHA on its registered suite's runner.
+   Finally, after `run-ci-file.yml` exists on the default branch, post `/rerun-test` with one registered CUDA test file and confirm the dispatched run uses that file as the command entrypoint from the recorded PR head SHA on its registered suite's runner and the original command receives a 👍 reaction only after dispatch succeeds.
 
 The handler evaluates the caller against the checked-in access policy before minting the App token and again before label mutation begins. An explicit `add_label_access.user_ids` match uses the numeric comment-author identity bound to the event; otherwise the handler checks the caller's live repository permission.
 Actions command comments for one PR are serialized in GitHub's queued concurrency mode. Up to GitHub's 100-pending limit, commands wait instead of replacing an older pending comment.
@@ -91,7 +93,7 @@ The comment handler runs only fixed, reviewed code from the default branch and n
 
 Initial authorization, PR, policy, or run-state errors fail before any mutation. A recheck error before a later rerun request stops that request, while any earlier accepted reruns remain applied.
 
-If the additive label `POST`, a label `DELETE`, a failed-job rerun `POST`, or a workflow-dispatch `POST` was sent but its response timed out, was malformed, or could not be confirmed, GitHub may already have applied the change. `/clear-labels` and `/rerun-failed-ci` can issue multiple requests and are not atomic: if a later request fails, earlier changes remain applied. The handler does not retry or roll back automatically; inspect the PR's current labels or Actions runs before deciding whether to retry.
+If the additive label `POST`, a label `DELETE`, a failed-job rerun `POST`, a workflow-dispatch `POST`, or the later reaction `POST` was sent but its response timed out, was malformed, or could not be confirmed, GitHub may already have applied the change. A reaction failure after dispatch does not cancel the file run. `/clear-labels` and `/rerun-failed-ci` can issue multiple requests and are not atomic: if a later request fails, earlier changes remain applied. The handler does not retry or roll back automatically; inspect the PR's current labels or Actions runs before deciding whether to retry.
 
 GitHub reruns failed jobs and their dependent jobs with the original run's `GITHUB_SHA`, `GITHUB_REF`, event payload, and triggering actor privileges. A rerun therefore does not inherit the commenter's or App token's privileges; consumers of the original event payload do not see labels changed afterward. GitHub permits reruns for up to 30 days after the original run and limits a workflow run to 50 attempts.