Repository navigation
feat(skills): add agent reproduction workflows #4118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 7 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
184a416
chore(skills): add codex reproduce workflows
DragonnZhang 693ca1e
feat(agent-reproduce): implement agent reproduction workflow and supp…
DragonnZhang 34facd5
feat(skills): capture reference agent state diffs
DragonnZhang 8e112d3
docs(skills): fix copy-paste script paths to use .qwen/skills/
DragonnZhang 98442df
docs(skills): document ssl_insecure flag in run_with_mitm.sh
DragonnZhang 8c4917d
fix(skills): harden reproduction capture helpers
DragonnZhang c8255c4
fix(skills): address reproduction review feedback
DragonnZhang b58fb22
fix(skills): match sensitive paths by segment instead of regex split
DragonnZhang 622ced8
fix(skills): guard normalize_trace against non-object JSONL lines
DragonnZhang 4d356e8
fix(skills): redact AWS and Google API keys in mitm env.txt
dragon 186ab92
fix(skills): preserve localhost in NO_PROXY for mitm wrapper
dragon 29fdae3
fix(skills): use portable sed char classes for token-name redaction
dragon 6db64b9
fix(skills): detect Gemini systemInstruction in normalize_trace
dragon 924f393
fix(skills): report per-request message-count + trailing message diffs
dragon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| --- | ||
| name: agent-reproduce-align | ||
| description: Use after a Codex or Claude Code feature has been implemented in Qwen Code to run the selected reference agent and Qwen Code under the same scenario, capture HTTP and terminal traces, compare request bodies, tool/function schemas, outputs, and iterate until the reproduced behavior is close enough. | ||
| --- | ||
|
|
||
| # Agent Reproduce Align | ||
|
|
||
| ## Purpose | ||
|
|
||
| Use this skill when Qwen Code already has a candidate implementation and needs evidence-based parity with a selected reference agent: `codex` or `claude-code`. The goal is not byte-for-byte equality; it is matching the observable contract that matters for the feature. | ||
|
|
||
| Default target repo: the current working directory. Use a user-specified path only when the user explicitly provides one. | ||
|
|
||
| ## Reference Agent Selection | ||
|
|
||
| Use the same reference agent selected during `$agent-reproduce-feature`. If the earlier choice is unavailable, ask once and record the answer in the scenario or run notes. | ||
|
|
||
| ## Workflow | ||
|
|
||
| 1. Re-state the parity target: | ||
| - feature name and trigger | ||
| - selected reference agent | ||
| - one baseline prompt or interaction script | ||
| - acceptable differences | ||
| - must-match fields | ||
| 2. Run the reference agent and Qwen Code in separate capture directories with the same scenario. | ||
| 3. Capture the selected reference agent's local state before and after the | ||
| reference run when state may affect parity. | ||
| 4. Normalize traces with `scripts/normalize_trace.py`. | ||
| 5. Compare normalized traces with `scripts/compare_traces.py`. | ||
| 6. Inspect differences in this order: | ||
| - reference-agent state changes that explain behavior | ||
| - missing tool/function names | ||
| - schema shape and required fields | ||
| - model settings and response mode | ||
| - prompt role/order differences that affect behavior | ||
| - terminal-visible output and exit status | ||
| 7. Patch Qwen Code, rerun the smallest failing scenario, and repeat. | ||
| 8. Preserve only redacted minimal fixtures in the repo. | ||
|
|
||
| Read `references/alignment-workflow.md` before the first comparison pass. | ||
|
|
||
| ## Common Commands | ||
|
|
||
| Normalize: | ||
|
|
||
| ```sh | ||
| .qwen/skills/agent-reproduce-align/scripts/normalize_trace.py \ | ||
| .repro-runs/reference/http.jsonl \ | ||
| > .repro-runs/reference/normalized.json | ||
| ``` | ||
|
|
||
| Compare: | ||
|
|
||
| ```sh | ||
| .qwen/skills/agent-reproduce-align/scripts/compare_traces.py \ | ||
| .repro-runs/reference/normalized.json \ | ||
| .repro-runs/qwen/normalized.json | ||
| ``` | ||
|
|
||
| Run a paired shell scenario: | ||
|
|
||
| ```sh | ||
| REPRO_REFERENCE_AGENT=codex \ | ||
| .qwen/skills/agent-reproduce-align/scripts/run_pair_capture.sh \ | ||
| .repro-runs/slash-help \ | ||
| "codex exec '/help'" \ | ||
| "npm test -- --runInBand" | ||
| ``` | ||
|
|
||
| For Claude Code, set `REPRO_REFERENCE_AGENT=claude-code` and replace the first | ||
| command with the discovered Claude Code command. When `REPRO_REFERENCE_AGENT` | ||
| is set, the paired runner writes `reference/state-before`, | ||
| `reference/state-after`, and `reference/state-diff`. Use the paired runner only | ||
| when shell quoting is simple. For interactive slash commands, run the two | ||
| captures manually with tmux so each side can receive the same keystrokes. Use | ||
| `REPRO_REFERENCE_STATE_ROOT=/tmp/some-root` only for tests or custom state | ||
| directories. | ||
|
|
||
| ## Comparison Rules | ||
|
|
||
| - Compare contracts before wording. Exact prompt text is usually implementation detail. | ||
| - Treat absent schemas, wrong required fields, or wrong argument names as high-signal failures. | ||
| - Treat output ordering as significant only when the user-visible workflow depends on it. | ||
| - Do not chase provider-specific endpoints, model names, IDs, timestamps, token counts, or ephemeral headers unless the feature depends on them. | ||
| - Do not chase every local state write. Treat state diffs as explanatory | ||
| evidence unless the feature contract requires a particular config, memory, or | ||
| permission side effect. | ||
| - Stop when Qwen Code passes the user-visible scenario and the remaining trace differences are documented as intentional. | ||
|
|
||
| ## Done Criteria | ||
|
|
||
| - Reference-agent and Qwen Code traces for the same scenario exist locally. | ||
| - Reference-agent state diff exists or state capture is documented as | ||
| irrelevant for the scenario. | ||
| - The normalized comparison has no unexplained must-match differences. | ||
| - Qwen Code tests or smoke commands cover the fixed behavior. | ||
| - Any remaining mismatch is written down in the task notes or Qwen Code docs when it affects users. |
84 changes: 84 additions & 0 deletions
84
.qwen/skills/agent-reproduce-align/references/alignment-workflow.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| # Alignment Workflow Reference | ||
|
|
||
| The alignment phase starts after Qwen Code has a candidate implementation. Use it to create a tight loop: run the selected reference agent and Qwen Code, compare traces, patch the target, and rerun only the failing scenario. | ||
|
|
||
| ## Trace Inputs | ||
|
|
||
| Expected raw capture layout: | ||
|
|
||
| ```text | ||
| .repro-runs/<scenario>/ | ||
| reference/ | ||
| http.jsonl | ||
| command.stdout | ||
| command.stderr | ||
| command.exit | ||
| state-before/state-manifest.json | ||
| state-after/state-manifest.json | ||
| state-diff/state-diff.md | ||
| qwen/ | ||
| http.jsonl | ||
| command.stdout | ||
| command.stderr | ||
| command.exit | ||
| ``` | ||
|
|
||
| Use capture scripts from `$agent-reproduce-feature` for raw capture, or use | ||
| `run_pair_capture.sh` for simple non-interactive shell scenarios. Set | ||
| `REPRO_REFERENCE_AGENT=codex` or `REPRO_REFERENCE_AGENT=claude-code` with the | ||
| paired runner to capture reference-agent state automatically. | ||
|
|
||
| ## Normalization | ||
|
|
||
| `normalize_trace.py` reads mitm JSONL output and emits stable JSON: | ||
|
|
||
| - request method and URL path | ||
| - JSON request body summary | ||
| - message role order and brief content hashes | ||
| - tool/function names | ||
| - schema required fields | ||
| - response status code | ||
|
|
||
| It intentionally drops: | ||
|
|
||
| - timestamps | ||
| - authorization and cookie headers | ||
| - provider request IDs | ||
| - full message text unless needed for a hash | ||
|
|
||
| ## Diff Triage | ||
|
|
||
| High priority: | ||
|
|
||
| - missing request entirely | ||
| - wrong endpoint family | ||
| - missing tool/function schema | ||
| - incompatible required fields or enum values | ||
| - slash command not routed to the same behavior class | ||
| - state changes that prove the feature writes config, memory, permissions, or | ||
| another user-visible local store | ||
|
|
||
| Medium priority: | ||
|
|
||
| - prompt role ordering differences | ||
| - terminal output phrasing differences | ||
| - streaming versus non-streaming if users can observe it | ||
| - unexplained state changes that plausibly affect future runs | ||
|
|
||
| Low priority: | ||
|
|
||
| - timestamps, IDs, token counts | ||
| - harmless wording differences | ||
| - extra target-side metadata ignored by the provider | ||
|
|
||
| ## Iteration Loop | ||
|
|
||
| 1. Pick the highest-priority unexplained mismatch. | ||
| 2. Patch only the likely owner module in Qwen Code. | ||
| 3. Run the focused test/smoke path. | ||
| 4. Capture only the affected scenario again. | ||
| 5. Refresh the reference state diff if the suspected mismatch involves local | ||
| state. | ||
| 6. Normalize and compare again. | ||
|
|
||
| Stop when the target behavior is compatible and remaining differences are either irrelevant or explicitly documented. |
135 changes: 135 additions & 0 deletions
135
.qwen/skills/agent-reproduce-align/scripts/compare_traces.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| #!/usr/bin/env python3 | ||
| """Compare normalized reproduction traces and print actionable differences.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import json | ||
| import sys | ||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
|
|
||
| def load(path: Path) -> dict[str, Any]: | ||
| return json.loads(path.read_text(encoding="utf-8")) | ||
|
|
||
|
|
||
| def tool_index(request: dict[str, Any]) -> dict[str, dict[str, Any]]: | ||
| return { | ||
|
DragonnZhang marked this conversation as resolved.
Resolved
|
||
| tool.get("name") or f"<unnamed-{idx}>": tool | ||
| for idx, tool in enumerate(request.get("tools") or []) | ||
| } | ||
|
|
||
|
|
||
| def tool_name_counts(request: dict[str, Any]) -> dict[str, int]: | ||
| counts: dict[str, int] = {} | ||
| for idx, tool in enumerate(request.get("tools") or []): | ||
| name = tool.get("name") or f"<unnamed-{idx}>" | ||
| counts[name] = counts.get(name, 0) + 1 | ||
| return counts | ||
|
|
||
|
|
||
| def compare_request(idx: int, left: dict[str, Any], right: dict[str, Any]) -> list[str]: | ||
| diffs: list[str] = [] | ||
| prefix = f"request[{idx}]" | ||
| for key in ( | ||
|
DragonnZhang marked this conversation as resolved.
Resolved
DragonnZhang marked this conversation as resolved.
Resolved
|
||
| "method", | ||
| "url_path", | ||
| "body_keys", | ||
| "body_values", | ||
| "model", | ||
| "stream", | ||
| "response_status", | ||
| ): | ||
| if left.get(key) != right.get(key): | ||
| diffs.append(f"{prefix}.{key}: {left.get(key)!r} != {right.get(key)!r}") | ||
|
|
||
|
DragonnZhang marked this conversation as resolved.
Resolved
DragonnZhang marked this conversation as resolved.
Resolved
|
||
| left_roles = [item.get("role") for item in left.get("messages") or []] | ||
| right_roles = [item.get("role") for item in right.get("messages") or []] | ||
| if left_roles != right_roles: | ||
|
DragonnZhang marked this conversation as resolved.
Resolved
|
||
| diffs.append(f"{prefix}.message_roles: {left_roles!r} != {right_roles!r}") | ||
| for msg_idx, (left_msg, right_msg) in enumerate( | ||
| zip(left.get("messages") or [], right.get("messages") or []) | ||
| ): | ||
| if left_msg.get("content_hash") != right_msg.get("content_hash"): | ||
| diffs.append( | ||
| f"{prefix}.messages[{msg_idx}].content_hash: " | ||
| f"{left_msg.get('content_hash')!r} != " | ||
| f"{right_msg.get('content_hash')!r}" | ||
| ) | ||
|
|
||
| left_tool_list = left.get("tools") or [] | ||
| right_tool_list = right.get("tools") or [] | ||
| if len(left_tool_list) != len(right_tool_list): | ||
| diffs.append( | ||
| f"{prefix}.tools_count: {len(left_tool_list)} != {len(right_tool_list)}" | ||
| ) | ||
| if tool_name_counts(left) != tool_name_counts(right): | ||
| diffs.append( | ||
| f"{prefix}.tool_name_counts: " | ||
| f"{tool_name_counts(left)!r} != {tool_name_counts(right)!r}" | ||
| ) | ||
| left_tools = tool_index(left) | ||
| right_tools = tool_index(right) | ||
| missing = sorted(set(left_tools) - set(right_tools)) | ||
| extra = sorted(set(right_tools) - set(left_tools)) | ||
| if missing: | ||
| diffs.append(f"{prefix}.tools_missing_in_right: {missing}") | ||
| if extra: | ||
| diffs.append(f"{prefix}.tools_extra_in_right: {extra}") | ||
|
|
||
|
DragonnZhang marked this conversation as resolved.
Resolved
DragonnZhang marked this conversation as resolved.
Resolved
|
||
| for name in sorted(set(left_tools) & set(right_tools)): | ||
| for key in ("type", "description_hash", "required", "properties", "schema"): | ||
| if left_tools[name].get(key) != right_tools[name].get(key): | ||
| diffs.append( | ||
| f"{prefix}.tool[{name}].{key}: " | ||
| f"{left_tools[name].get(key)!r} != {right_tools[name].get(key)!r}" | ||
| ) | ||
| return diffs | ||
|
|
||
|
|
||
| def main() -> int: | ||
| parser = argparse.ArgumentParser() | ||
| parser.add_argument("left", type=Path, help="Reference normalized trace") | ||
| parser.add_argument("right", type=Path, help="Target normalized trace, usually Qwen Code") | ||
| args = parser.parse_args() | ||
|
|
||
| try: | ||
| left = load(args.left) | ||
| right = load(args.right) | ||
| except (OSError, json.JSONDecodeError) as exc: | ||
| print(f"Failed to load normalized trace: {exc}", file=sys.stderr) | ||
| return 2 | ||
|
|
||
| diffs: list[str] = [] | ||
|
DragonnZhang marked this conversation as resolved.
Resolved
|
||
|
|
||
| if left.get("request_count") != right.get("request_count"): | ||
| diffs.append( | ||
| f"request_count: {left.get('request_count')!r} != {right.get('request_count')!r}" | ||
| ) | ||
|
|
||
| for idx, (left_req, right_req) in enumerate( | ||
| zip(left.get("requests") or [], right.get("requests") or []) | ||
| ): | ||
| diffs.extend(compare_request(idx, left_req, right_req)) | ||
| left_requests = left.get("requests") or [] | ||
| right_requests = right.get("requests") or [] | ||
| if len(left_requests) > len(right_requests): | ||
| for idx, request in enumerate(left_requests[len(right_requests) :], len(right_requests)): | ||
| diffs.append(f"request[{idx}].missing_in_right: {request!r}") | ||
| elif len(right_requests) > len(left_requests): | ||
| for idx, request in enumerate(right_requests[len(left_requests) :], len(left_requests)): | ||
| diffs.append(f"request[{idx}].extra_in_right: {request!r}") | ||
|
|
||
| if not diffs: | ||
| print("No normalized trace differences found.") | ||
| return 0 | ||
|
|
||
| print("Normalized trace differences:") | ||
| for diff in diffs: | ||
| print(f"- {diff}") | ||
|
DragonnZhang marked this conversation as resolved.
Resolved
|
||
| return 1 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.