Skip to content

feat(ci): on-demand tmux real-user testing for PRs #5203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yiliang114 merged 11 commits into from
Jun 21, 2026
Merged

feat(ci): on-demand tmux real-user testing for PRs #5203

Changes from 4 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
ea7a3e2
feat(ci): add on-demand tmux real-user testing to triage workflow
yiliang114 Jun 16, 2026
8d3e271
fix(ci): address tmux-testing review findings
yiliang114 Jun 17, 2026
aced4c1
refactor(ci): collapse tmux gating into one decision output
yiliang114 Jun 17, 2026
0aa0de9
fix(ci): wait out UNKNOWN mergeability before deciding
yiliang114 Jun 17, 2026
c31e77a
Merge branch 'main' into feat/triage-tmux-testing
yiliang114 Jun 17, 2026
1a8bac1
fix(ci): harden tmux publish comment and mergeability/exit-code handling
yiliang114 Jun 17, 2026
3637708
ci: harden tmux testing workflow
yiliang114 Jun 18, 2026
8f42c00
Merge remote-tracking branch 'origin/main' into feat/triage-tmux-testing
yiliang114 Jun 20, 2026
9231155
fix(ci): address tmux testing review follow-ups
yiliang114 Jun 20, 2026
01985fa
fix(ci): report tmux prepare failures
yiliang114 Jun 21, 2026
7dc28f8
fix(ci): use latest qwen CLI for tmux testing
yiliang114 Jun 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
349 changes: 340 additions & 9 deletions .github/workflows/qwen-triage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,17 @@ on:
inputs:
number:
description: 'Issue or PR number to triage'
required: true
required: false
type: 'number'
tmux_pr:
description: 'PR number to run tmux real-user testing on (instead of triage)'
required: false
type: 'number'
no_comment:
description: 'tmux dry-run: run the test but do not post a comment on the PR'
required: false
default: false
type: 'boolean'

permissions:
contents: 'read'
Expand All @@ -21,16 +30,26 @@ permissions:

jobs:
authorize:
# Gate PR and /triage-comment triggers on the principal having write+
# permission: the PR author for PR events, the commenter for /triage
# comments. Replaces the old eligibility checks based on same-repo PRs and
# comment author_association, so fork PRs by trusted authors are triaged.
# `issues` and `workflow_dispatch` triggers do not need this gate.
# Gate the principal on having write+ permission before any agent runs:
# - pull_request_target / `/triage` comment -> gates triage (read-only),
# keyed on the PR author / the commenter respectively.
# - `/tmux` comment / `tmux_pr` dispatch -> gates real-user testing, which
# EXECUTES the PR author's code, so it is keyed on the PR author (whose
# code runs), not the commenter/dispatcher (see principal resolution).
# Replaces the old eligibility checks based on same-repo PRs and comment
# author_association, so fork PRs by trusted authors are covered.
# The `issues` and `workflow_dispatch`-with-`number` (triage) triggers need
# no gate: triage is read-only and dispatch already requires write to
# invoke. But `tmux_pr` dispatch runs the *PR author's* code, not the
# dispatcher's, so it IS gated here on the PR author's permission.
if: |-
github.repository == 'QwenLM/qwen-code' &&
(github.event_name == 'pull_request_target' ||
(github.event_name == 'issue_comment' &&
startsWith(github.event.comment.body, '@qwen-code /triage')))
(startsWith(github.event.comment.body, '@qwen-code /triage') ||
startsWith(github.event.comment.body, '@qwen-code /tmux'))) ||
(github.event_name == 'workflow_dispatch' &&
github.event.inputs.tmux_pr != ''))
runs-on: 'ubuntu-latest'
timeout-minutes: 5
permissions:
Expand All @@ -50,11 +69,28 @@ jobs:
EVENT_NAME: '${{ github.event_name }}'
PR_AUTHOR: '${{ github.event.pull_request.user.login }}'
COMMENT_USER: '${{ github.event.comment.user.login }}'
ISSUE_AUTHOR: '${{ github.event.issue.user.login }}'
COMMENT_BODY: '${{ github.event.comment.body }}'
TMUX_PR: '${{ github.event.inputs.tmux_pr }}'
run: |-
set -euo pipefail
case "$EVENT_NAME" in
pull_request_target) principal="$PR_AUTHOR" ;;
issue_comment) principal="$COMMENT_USER" ;;
issue_comment)
# /tmux executes the PR AUTHOR's code, so gate on the author's
# permission (whose code runs), not the commenter's. /triage only
# reads content, so the commenter's permission gates it.
case "$COMMENT_BODY" in
'@qwen-code /tmux'*) principal="$ISSUE_AUTHOR" ;;
*) principal="$COMMENT_USER" ;;
esac
;;
workflow_dispatch)
# Only the tmux_pr dispatch reaches authorize. It runs the PR
# author's code, so resolve and gate on that author (not the
# dispatcher). Empty/unresolvable author fails closed below.
principal="$(gh pr view "$TMUX_PR" --repo "$GITHUB_REPOSITORY" --json author --jq '.author.login' 2>/dev/null || true)"
;;
*) principal="" ;;
esac
if [ -z "$principal" ]; then
Expand Down Expand Up @@ -124,7 +160,9 @@ jobs:
always() &&
github.repository == 'QwenLM/qwen-code' && (
github.event_name == 'issues' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'workflow_dispatch' &&
github.event.inputs.number != '' &&
github.event.inputs.tmux_pr == '') ||
(
((github.event_name == 'pull_request_target' &&
github.event.pull_request.draft == false) ||
Expand Down Expand Up @@ -175,3 +213,296 @@ jobs:
"sandbox": false
}
prompt: '/triage ${{ steps.resolve.outputs.number }} --repo ${{ github.repository }}'

# On-demand real-user testing: a write-permission user comments
# `@qwen-code /tmux` on a PR to launch the changed app in a tmux TUI and
# exercise the affected flow. EXECUTES untrusted PR code, so: gated on the PR
# AUTHOR (whose code runs) having write via the authorize job, runs read-only
# with NO GitHub token in the agent env, and keeps credentials out of .git.
tmux-testing:
needs: ['authorize']
if: >-
always() &&
github.repository == 'QwenLM/qwen-code' &&
(
(github.event_name == 'issue_comment' &&
github.event.issue.pull_request &&
startsWith(github.event.comment.body, '@qwen-code /tmux') &&
needs.authorize.outputs.should_run == 'true') ||
(github.event_name == 'workflow_dispatch' &&
github.event.inputs.tmux_pr != '' &&
needs.authorize.outputs.should_run == 'true')
)
# One real-user test per PR at a time. Concurrent /tmux runs would share the
# same self-hosted runner workspace and git worktrees and clobber each other,
# so serialize them (cancel-in-progress: false lets the in-flight test finish
# rather than killing it mid-run).
concurrency:
group: '${{ github.workflow }}-tmux-${{ github.event.issue.number || github.event.inputs.tmux_pr }}'
Comment thread
yiliang114 marked this conversation as resolved.
Resolved
cancel-in-progress: false
timeout-minutes: 25
runs-on: ['self-hosted', 'linux', 'x64', 'ecs-qwen']
permissions:
contents: 'read'
outputs:
pr_number: '${{ steps.pr.outputs.pr_number }}'
# steps.run sets the verdict for an actual test; steps.pr sets 'n/a' when
# the PR has no TUI surface. Both empty -> stayed silent (skip case).
verdict: '${{ steps.run.outputs.verdict || steps.pr.outputs.verdict }}'
steps:
- name: 'Resolve PR and check state'
id: 'pr'
env:
GH_TOKEN: '${{ secrets.CI_BOT_PAT }}'
PR_NUMBER: '${{ github.event.issue.number || github.event.inputs.tmux_pr }}'
run: |-
set -euo pipefail
echo "pr_number=${PR_NUMBER}" >> "$GITHUB_OUTPUT"
# One API call per attempt for everything the job gates on (state +
# file list). Right after a /tmux comment GitHub may not have computed
# mergeability yet (mergeable=UNKNOWN), and refs/pull/N/merge is only
# current once it has — so give it a few seconds to settle before
# deciding, rather than checking out a stale/missing ref.
for attempt in 1 2 3 4 5; do
data="$(gh pr view "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --json state,isDraft,mergeable,files)"
mergeable="$(jq -r '.mergeable' <<< "$data")"
[ "$mergeable" != "UNKNOWN" ] && break
echo "::notice::Mergeability for PR #${PR_NUMBER} not computed yet; retry ${attempt}/5."
sleep 3
done
state="$(jq -r '.state' <<< "$data")"
is_draft="$(jq -r '.isDraft' <<< "$data")"
# decision drives every step below: skip (nothing to do, stay silent) |
# na (no TUI surface to exercise) | run (drive the app).
if [ "$state" != "OPEN" ] || [ "$is_draft" = "true" ]; then
echo "::notice::Skipping tmux testing: PR #${PR_NUMBER} state=${state} draft=${is_draft}."
echo "decision=skip" >> "$GITHUB_OUTPUT"
exit 0
fi
# The checkout below uses refs/pull/N/merge, which GitHub only keeps
# current while the PR merges cleanly. For a conflicting PR the ref is
# stale or missing, so skip rather than test the wrong tree.
if [ "$mergeable" = "CONFLICTING" ]; then
echo "::notice::Skipping tmux testing: PR #${PR_NUMBER} has merge conflicts; refs/pull/${PR_NUMBER}/merge is unavailable."
echo "decision=skip" >> "$GITHUB_OUTPUT"
exit 0
fi
# Only drive the app for PRs that touch a user-facing/TUI surface;
# otherwise a real-user test has nothing to exercise (e.g. CI-only PRs).
if jq -r '.files[].path' <<< "$data" | grep -qE 'packages/cli/src/ui/|packages/cli/.*\.tsx$|windowTitle|packages/web-shell/client/'; then
echo "decision=run" >> "$GITHUB_OUTPUT"
else
echo "::notice::PR #${PR_NUMBER} touches no TUI surface; tmux testing is not applicable."
echo "verdict=n/a" >> "$GITHUB_OUTPUT"
echo "decision=na" >> "$GITHUB_OUTPUT"
fi

- name: 'Clean stale review worktrees'
if: "steps.pr.outputs.decision == 'run'"
run: |-
set -uo pipefail
[ -e .git ] || exit 0
rm -rf .qwen/tmp/review-pr-* 2>/dev/null || true
git worktree prune -v || true

- name: 'Checkout PR merge ref'
if: "steps.pr.outputs.decision == 'run'"
uses: 'actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd' # v6.0.2
with:
# Untrusted PR code — keep the token out of .git/config.
persist-credentials: false
ref: 'refs/pull/${{ steps.pr.outputs.pr_number }}/merge'
fetch-depth: 1

- name: 'Run tmux real-user testing'
Comment thread
yiliang114 marked this conversation as resolved.
Resolved
if: "steps.pr.outputs.decision == 'run'"
id: 'run'
# NOTE: no GitHub token here — this step runs untrusted PR code. Only
# the model credentials are exposed (same as the review agent). The
# GITHUB_TOKEN / GH_TOKEN are pinned empty so nothing in the inherited
# environment can hand the agent a repo-scoped credential.
env:
GITHUB_TOKEN: ''
GH_TOKEN: ''
OPENAI_API_KEY: '${{ secrets.REVIEW_OPENAI_API_KEY }}'
OPENAI_BASE_URL: '${{ secrets.REVIEW_OPENAI_BASE_URL }}'
OPENAI_MODEL: '${{ vars.QWEN_PR_REVIEW_MODEL }}'
PR_NUMBER: '${{ steps.pr.outputs.pr_number }}'
REPOSITORY: '${{ github.repository }}'
run: |-
set -euo pipefail
if ! command -v qwen >/dev/null 2>&1; then
echo "::error::qwen CLI not found on runner"
exit 1
fi

# Bypass the runner proxy before launching qwen: the proxy cuts the
# SSE stream to the model host, and qwen reads HTTP(S)_PROXY directly
# without honoring NO_PROXY. This drops the proxy for the whole agent
# run, including the build it drives — the ecs-qwen runner reaches the
# npm registry without the HTTP proxy (validated end-to-end). Unlike
# the review job we add no gh/git proxy wrappers: this job carries no
# GitHub token, so the agent does no authenticated gh/git network ops.
# shellcheck disable=SC2016
configure_qwen_network() {

@yiliang114 yiliang114 Jun 21, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am leaving this inline rather than extracting a shared action in this PR. The two call sites still differ in env names and trust boundary, and making a composite/shared helper would touch the existing review workflow as well. Worth doing as a follow-up once this path settles, but I do not want to expand this PR blast radius.

local openai_host
if ! command -v node >/dev/null 2>&1; then
echo "::error::node is required to parse OPENAI_BASE_URL"
exit 1
fi
openai_host="$(node -e 'console.log(new URL(process.env.OPENAI_BASE_URL).hostname)')"
if [ -z "$openai_host" ]; then
echo "::error::Could not parse a hostname from OPENAI_BASE_URL"
exit 1
fi
export NO_PROXY="${NO_PROXY:+$NO_PROXY,}${openai_host}"
export no_proxy="${no_proxy:+$no_proxy,}${openai_host}"
unset HTTPS_PROXY https_proxy HTTP_PROXY http_proxy
echo "qwen_http_proxy=disabled (host=${openai_host})"
}
configure_qwen_network

QWEN_CMD=(qwen --auth-type openai --approval-mode yolo)
if [ -n "${OPENAI_MODEL:-}" ]; then
QWEN_CMD+=(--model "$OPENAI_MODEL")
fi

mkdir -p "$RUNNER_TEMP/tmux-results"
set +e
timeout --kill-after=10s 20m "${QWEN_CMD[@]}" \
--prompt "/tmux-real-user-testing ${PR_NUMBER} --repo ${REPOSITORY}" \
--output-format stream-json \
| tee "$RUNNER_TEMP/tmux-results/output.jsonl"
EXIT_CODE=${PIPESTATUS[0]}
set -e

# Collect the skill's narrative artifacts (report.md, readable logs)
# from the workspace tmp/ into the upload dir.
find tmp -maxdepth 2 -type d -name '*-tmux-*' -exec cp -r {} "$RUNNER_TEMP/tmux-results/" \; 2>/dev/null || true

if [ "$EXIT_CODE" -eq 124 ]; then
VERDICT='timeout'
elif [ "$EXIT_CODE" -ne 0 ]; then
VERDICT='fail'
else
VERDICT='pass'
fi
echo "verdict=$VERDICT" >> "$GITHUB_OUTPUT"
echo "tmux verdict: $VERDICT (exit $EXIT_CODE)" >> "$GITHUB_STEP_SUMMARY"

- name: 'Upload tmux results'
if: "always() && steps.pr.outputs.decision == 'run'"
uses: 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' # v4.6.2
with:
name: 'tmux-results-${{ steps.pr.outputs.pr_number }}'
path: '${{ runner.temp }}/tmux-results/'
retention-days: 7

- name: 'Clean up runner workspace'
# Mirror the stale-worktree cleanup at the start of the job, but at the
# end and on every outcome. Without it the checked-out PR tree, the
# skill's tmp/*-tmux-* dirs, and any worktrees accumulate on the
# persistent self-hosted runner across runs.
if: "always() && steps.pr.outputs.decision == 'run'"
run: |-
set -uo pipefail
[ -e .git ] || exit 0
rm -rf .qwen/tmp/review-pr-* 2>/dev/null || true
find tmp -maxdepth 2 -type d -name '*-tmux-*' -exec rm -rf {} + 2>/dev/null || true
git worktree prune -v || true

# Post the tmux verdict back to the PR. Runs on a clean GitHub-hosted runner
# with the write PAT and never checks out PR code, so the write credential is
# isolated from the untrusted-code execution in tmux-testing above.
publish-tmux:
needs: ['tmux-testing']
# Post when there is a verdict to report, OR when tmux-testing failed for
# infrastructure reasons (checkout/runner/setup error) so the requester gets
# an explicit signal instead of a silent void. The empty-verdict success
# case (PR closed/draft/conflicting/no-TUI handled upstream) stays silent.
if: >-
always() && github.event.inputs.no_comment != 'true' &&
(needs.tmux-testing.result == 'failure' ||

@yiliang114 yiliang114 Jun 21, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving skipped out intentionally. A skipped tmux-testing job is the normal no-op path for non-/tmux, unauthorized, or otherwise non-runnable triggers; publishing on skipped would turn unrelated comments/events into PR noise. The explicit surfaced outcomes are failure, cancelled, and successful runs with a real verdict.

(needs.tmux-testing.result == 'success' && needs.tmux-testing.outputs.verdict != ''))
runs-on: 'ubuntu-latest'
permissions:
pull-requests: 'write'
steps:
- name: 'Download tmux results'
uses: 'actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c' # v5.0.0
with:
name: 'tmux-results-${{ needs.tmux-testing.outputs.pr_number }}'
path: 'tmux-results'
continue-on-error: true

- name: 'Post tmux result comment'
env:
GH_TOKEN: '${{ secrets.CI_BOT_PAT }}'
PR_NUMBER: '${{ needs.tmux-testing.outputs.pr_number }}'
VERDICT: '${{ needs.tmux-testing.outputs.verdict }}'
TMUX_RESULT: '${{ needs.tmux-testing.result }}'
RUN_URL: '${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}'
# shellcheck disable=SC2016
run: |-
set -euo pipefail
if [ -z "${PR_NUMBER:-}" ]; then
echo "::warning::No PR number resolved; cannot post a tmux result comment."
exit 0
fi
BODY_FILE="${RUNNER_TEMP:-/tmp}/tmux-comment.md"

# Embed a file inside a collapsed <details> as a fenced code block.
# Fenced (not rendered) on purpose: PR-driven test output must not
# render markdown, fire @mentions, or break the comment layout.
emit_block() {
local summary="$1" file="$2" max="$3" content longest width fence
[ -n "$file" ] && [ -f "$file" ] || return 0
content="$(head -c "$max" "$file")"
if [ "$(wc -c < "$file")" -gt "$max" ]; then
content="${content}"$'\n\n...truncated -- full log in the run artifacts.'
fi
# The content is untrusted PR output and may contain ``` runs that
# would close a 3-backtick fence early and let the rest render as
# markdown. Open/close with a fence one backtick longer than the
# longest backtick run in the content (min 3).
longest="$(printf '%s' "$content" | grep -oE '`+' | awk '{ if (length > m) m = length } END { print m + 0 }')"
width=$(( longest >= 3 ? longest + 1 : 3 ))
fence="$(printf '%*s' "$width" '' | tr ' ' '`')"
printf '<details>\n<summary>%s</summary>\n\n%stext\n' "$summary" "$fence"
printf '%s\n' "$content"
printf '%s\n\n</details>\n\n' "$fence"
}
Comment thread
yiliang114 marked this conversation as resolved.
Resolved

if [ "${TMUX_RESULT:-}" != "success" ] || [ -z "${VERDICT:-}" ]; then
# tmux-testing did not finish (infrastructure error): report it so the
# requester is not left with silence.
{
printf '%s\n\n' '<!-- qwen-triage:tmux -->'
printf '**tmux real-user testing: infrastructure failure** - [workflow run](%s)\n\n' "$RUN_URL"
printf 'The testing job did not complete (checkout, runner, or setup error) and produced no verdict. See the workflow run for details.\n\n'
printf '%s\n' '— _Qwen Code · tmux real-user testing_'
} > "$BODY_FILE"
else
REPORT="$(find tmux-results -name 'report.md' 2>/dev/null | head -1 || true)"
TRANSCRIPT="$(find tmux-results -name 'tmux-readable-full.log' 2>/dev/null | head -1 || true)"
{
printf '%s\n\n' '<!-- qwen-triage:tmux -->'
printf '**tmux real-user testing: %s** - [workflow run](%s)\n\n' "$VERDICT" "$RUN_URL"
printf 'Launched the changed app in a real tmux session and exercised the affected flow.\n\n'

@wenshao wenshao Jun 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] The Run step deliberately distinguishes timeout (exit 124) and infra-error (SIGKILL 137 / SIGSEGV 139 — OOM, crash, forced timeout) from a genuine fail, with a comment that these are "not a test outcome." But both still produce result=success + a non-empty, non-n/a verdict, so they pass every branch above and land in this final else — which prints "Launched the changed app in a real tmux session and exercised the affected flow" plus the (for an OOM/timeout, typically empty or truncated) report and transcript blocks.

So a run that crashed/OOM'd/timed out before exercising anything is reported with a body asserting the flow was exercised — reintroducing at the rendering layer the exact "verdict misleads" outcome the infra-error classification was added to prevent. Consider an explicit branch for timeout/infra-error before this else whose body doesn't claim the flow ran (e.g. "the app launched but testing did not complete — it timed out or the process was killed; this is not a pass/fail result").

— claude-opus-4-8[1m] via Qwen Code /qreview

emit_block 'E2E test report' "$REPORT" 20000
emit_block 'Full tmux transcript' "$TRANSCRIPT" 30000
printf '%s\n' '— _Qwen Code · tmux real-user testing_'
} > "$BODY_FILE"
fi

# Dedup: update an existing tmux comment if one is already present.
EXISTING="$(
gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" --paginate -F per_page=100 \
| jq -sr '[.[][] | select(.body | contains("<!-- qwen-triage:tmux -->"))] | last | .id // empty'
)" || EXISTING=""
if [ -n "$EXISTING" ]; then
gh api -X PATCH "repos/$GITHUB_REPOSITORY/issues/comments/$EXISTING" -F body=@"$BODY_FILE" >/dev/null
else
gh api "repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER/comments" -F body=@"$BODY_FILE" >/dev/null
fi
echo "Posted tmux result to PR #${PR_NUMBER} (verdict=${VERDICT})." >> "$GITHUB_STEP_SUMMARY"
Loading