feat: add Midscene report analysis skill

README.md

@@ -16,6 +16,7 @@
   - iOS (controlled via WebDriverAgent): [`skills/ios-automation`](skills/ios-automation)
   - HarmonyOS (controlled via HDC): [`skills/harmony-automation`](skills/harmony-automation)
   - Vitest + Midscene E2E (scaffold, convert, and manage AI-powered E2E tests for Web, Android, iOS): [`skills/vitest-midscene-e2e`](skills/vitest-midscene-e2e)
+  - Midscene report analysis (validate passed and failed results, identify false-pass/false-fail outcomes, and trace root causes): [`skills/midscene-report-analysis`](skills/midscene-report-analysis)
 
 
 ## Safety Warning
@@ -102,6 +103,10 @@ Use Midscene computer skill to open the Keynote app and create a new presentatio
 Use Midscene browser skill to open the Google search page and search for "Midscene".
 ```
 
+```
+Use Midscene report analysis skill to analyze /absolute/path/to/midscene-report.html.
+```
+
 ## Issues
 
 For bug reports, feature requests, and discussions, please visit the main Midscene repository: https://github.com/web-infra-dev/midscene/issues

skills/midscene-report-analysis/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: midscene-report-analysis
+description: |
+  Analyze complete Midscene report URLs or local HTML files to validate recorded
+  pass/fail results, identify false-pass and false-fail results, diagnose failures
+  and incomplete executions, and trace evidenced root causes.
+
+  Powered by Midscene.js (https://midscenejs.com)
+allowed-tools:
+  - Bash
+---
+
+# Analyze Midscene Reports
+
+## Workflow
+
+### 1. Inspect the report
+
+Require a complete report: an HTTP(S) URL, absolute HTML path, or directory containing `index.html`. Reject screenshots, summaries, or isolated records.
+
+Inspect before analysis:
+
+~~~bash
+/Users/bytedance/midscene/packages/cli/bin/midscene report-tool --action inspect --htmlPath /absolute/path/report.html [--outputDir <directory>]
+~~~
+
+Continue only when inspect returns `report`, `reportStatus`, `localReport`, and `markdownFiles`.
+
+### 2. Review, analyze, and render
+
+1. **Review the complete report.** Read every file in `markdownFiles` to EOF and inspect every screenshot link in task order. For each task, compare model narration, executed action, machine result, and resulting UI; track abnormalities and recovery. Resolve omissions from `localReport`; search cannot replace full review. Before step 2, report exact files-read/total and screenshot-links-viewed/total counts, counting repeated links in context, and proceed only at full coverage.
+
+   - **Verify model narration.** Treat model-authored text as claims, not evidence. Validate it against executed actions, machine results, and screenshots; prefer direct records on conflict.
+   - **Check scroll coverage.** Before declaring an element absent, verify the relevant scroll container across every applicable horizontal and vertical bound.
+
+2. **Define the result conditions.** Derive the required conditions and judgment point from the test instruction governing the recorded result. Use earlier or parallel executions only as context or prerequisite evidence, not as separate outcomes.
+
+3. **Read the applicable rules.** Select by `reportStatus`:
+   - `fail`: read [failed-result-analysis.md](references/failed-result-analysis.md) completely. Read [cause-taxonomy.md](references/cause-taxonomy.md) completely before setting `causeCategories` for any result assessment.
+   - `pass`: read [passed-result-analysis.md](references/passed-result-analysis.md) completely. Read [cause-taxonomy.md](references/cause-taxonomy.md) completely before setting `causeCategories` for any result assessment.
+   - `incomplete`: read [incomplete-execution-analysis.md](references/incomplete-execution-analysis.md) completely. Read [cause-taxonomy.md](references/cause-taxonomy.md) completely before setting `causeCategories`.
+
+4. **Assess the report globally.** Using the selected workflow, relate every recorded abnormality to the required conditions. Decide whether it was causal, contributory, or unrelated, and whether its effects recovered. Then assess the recorded result and attribute causes with independent confidence and an evidence-based reason for every category where applicable. For a fail or pass report, finalize `resultAssessment` here.
+
+5. **Get the Schema and create the result.** Only after steps 1–4, call `analysis-template` for every report to obtain its `analysisResultPath` and applicable `schema`.
+
+   For a fail or pass report, pass the finalized assessment:
+
+   ~~~bash
+   /Users/bytedance/midscene/packages/cli/bin/midscene report-tool --action analysis-template --reportStatus <fail|pass> --resultAssessment <resultAssessment> --htmlPath <localReport> [--outputDir <inspect-artifact-directory>]
+   ~~~
+
+   For an incomplete report, omit `--resultAssessment`:
+
+   ~~~bash
+   /Users/bytedance/midscene/packages/cli/bin/midscene report-tool --action analysis-template --reportStatus incomplete --htmlPath <localReport> [--outputDir <inspect-artifact-directory>]
+   ~~~
+
+   Use the returned `schema` to create each result JSON at `analysisResultPath`; do not edit the Markdown. Use `localReport` for reading, but set the result's `report` to inspect's `report`.
+
+6. **Choose the output by report count.** Count the reports supplied in the user's current request.
+
+   - **Exactly one report:** Render the completed JSON:
+
+     ~~~bash
+     /Users/bytedance/midscene/packages/cli/bin/midscene report-tool --action render-analysis --analysisResultPath <analysisResultPath>
+     ~~~
+
+     Return the renderer output unchanged. Do not add an analysis-JSON link, edit the rendered Markdown or input report, or change unrelated external state.
+
+   - **More than one report:** Render every completed result JSON so each case has its own Markdown file:
+
+     ~~~bash
+     /Users/bytedance/midscene/packages/cli/bin/midscene report-tool --action render-analysis --analysisResultPath <analysisResultPath>
+     ~~~
+
+     Keep every renderer-generated Markdown file and its absolute path. Return one compact aggregate summary containing:
+     1. total report count and counts by `reportStatus` and, where applicable, `resultAssessment`;
+     2. the main cross-report findings, grouped by recurring issue;
+     3. one concise row per report with its sequence number, status, assessment (or `incomplete`), cause category/confidence pairs (or `unattributed`), one-sentence conclusion, the original clickable report URL from inspect's `report` field, and a clickable absolute link to that case's renderer-generated Markdown file.
+
+     Save the same aggregate summary Markdown to `<batch-artifact-directory>/batch-analysis-summary.md` before returning it. Use the user-specified output directory as `batch-artifact-directory`; otherwise create one task-scoped local directory shared by all reports in the batch. Do not overwrite an existing unrelated file. Return the summary in the response and add a clickable absolute link to the saved Markdown file.

skills/midscene-report-analysis/references/cause-taxonomy.md

@@ -0,0 +1,27 @@
+# Cause taxonomy
+
+Write each `causeCategories` entry as `{ "category": <category>, "confidence": <high|medium|low>, "reason": <evidence-based explanation> }`. Include a category only when observed report facts support it; inability to exclude a category is not enough. Preserve exact machine errors, describe the observed mechanism in free text, stop at the observable layer, and do not invent second-level enums.
+
+Assign confidence independently for every category:
+
+- `high`: direct, corroborated evidence locates the cause in this category.
+- `medium`: the report supports the attribution as the best inference, but does not directly expose the full mechanism.
+- `low`: observed symptoms support the category as a plausible cause, but the report cannot distinguish it from another supported owner.
+
+Multiple entries may be joint causes or alternative possible causes. Explain which relationship applies in the conclusion or evidence. Low confidence does not permit listing arbitrary categories that are merely unexcluded.
+
+Write a non-empty `reason` for every entry. Tie it to report facts, justify the selected confidence, and state whether the category is a joint cause or an alternative possible cause when multiple entries are present.
+
+| Category | Assign when |
+| --- | --- |
+| `model_reasoning` | The test is sufficiently specified and observable, but the model makes an incorrect interpretation, decision, action, or assertion. Examples: selects the wrong visible element; skips a required step despite sufficient context; claims success without observing the required fact; scrolls the wrong axis, direction, or container; declares a target absent or a task impossible before exhausting the applicable vertical and horizontal bounds. Do not infer this category from a failed model task alone. |
+| `test_design` | The test lacks information, preconditions, or observable criteria required for valid execution or judgment. Examples: omits a required account or permission; refers to stale or nonexistent test data; asks to verify an effect that the report cannot observe. |
+| `midscene_runtime` | Evidence locates the failure in Midscene's execution or browser-control layer. Examples: the CDP connection is lost; the browser target or frame detaches; the action bridge fails to execute a command. |
+| `tested_system` | An otherwise valid test is blocked by the tested application's behavior or state. Examples: an application-owned API or data lookup fails and dependent controls become empty or unusable; a correctly established permission or business state is rejected or lost; a correct action receives no visible application response. |
+| `external_dependency` | Evidence locates the failure outside the model, test, Midscene runtime, and tested application. Examples: the model service returns a terminal 429 or 5xx; DNS or a proxy prevents connectivity; external authentication or environment provisioning fails. Never use this category for uncertainty. |
+
+Resolve common boundaries from evidence:
+
+- Missing login, permission, data, or state: use `test_design` for an omitted precondition, `tested_system` for a correctly established state that is rejected or lost, and `external_dependency` for an independent authentication or provisioning failure. When the report proves only that expected data or an option is absent and lacks the specification, API response, permission state, or test-data baseline needed to distinguish ownership, record the supported alternatives with `low` confidence and explain that they are alternatives rather than joint causes.
+- Malformed action protocol: use `model_reasoning` when the model output itself violates the required action format; use `midscene_runtime` when valid model output is parsed or executed incorrectly.
+- Action followed by timeout: use `model_reasoning` for a wrong target, `midscene_runtime` for a proven browser-control failure, and `tested_system` for a correct action followed by a visibly unresponsive application. Attribute only owners supported by positive evidence.

skills/midscene-report-analysis/references/failed-result-analysis.md

@@ -0,0 +1,22 @@
+# Midscene failed-result analysis
+
+## Goal
+
+Determine whether the recorded failure is false (`false_fail`). If the task truly failed, identify why and where it failed.
+
+## Result assessments
+
+Assess each required condition at its judgment point:
+
+| Result assessment | Use when |
+| --- | --- |
+| `true_fail` | At least one required condition is proven unmet. |
+| `false_fail` | Every required condition is proven met despite the recorded failure. |
+| `unverifiable` | Evidence is insufficient to determine whether the conditions were met. |
+| `inconclusive` | Reliable evidence conflicts or remains inherently ambiguous. |
+
+`false_fail` requires direct evidence that all required conditions succeeded.
+
+## Attribution
+
+Attribute the issue underlying the assessment: the actual failure for `true_fail`, the erroneous failure record for `false_fail`, or the evidence gap or conflict for `unverifiable` and `inconclusive`. Use [cause-taxonomy.md](cause-taxonomy.md), assign confidence and an evidence-based reason to every category, and explain whether multiple entries are joint causes or alternative possible causes.

skills/midscene-report-analysis/references/incomplete-execution-analysis.md

@@ -0,0 +1,22 @@
+# Midscene incomplete-execution analysis
+
+## Goal
+
+After completing the report-wide analysis, answer two questions independently:
+
+1. What report-internal issue was observed before termination?
+2. Why did the final task remain non-terminal?
+
+Always answer both questions. When no internal issue was observed or the exact interruption reason is not established, state that explicitly instead of omitting the field. Do not call the report passed or failed.
+
+## Observed issue
+
+In `observedIssue`, describe the evidenced internal issue that materially delayed, derailed, or prevented progress. If no such issue was observed, state that directly. If the evidence is conflicting or cannot distinguish the issue, describe that bounded uncertainty without guessing. A recovered issue may still be included when it changed the execution path.
+
+## Interruption
+
+In `interruptionReason`, explain why the final task remained non-terminal. State a specific cause only when the report establishes it; otherwise state that the exact cause is not recorded. Do not use an earlier issue as the interruption reason unless evidence shows that it remained active or propagated to the last recorded step. A lone non-terminal status is not a cause.
+
+## Attribution
+
+Attribute causes supported by observed report facts. A category may describe the observed issue or a directly linked interruption. Assign confidence and an evidence-based reason to every category and explain whether multiple entries are joint or alternative; a lone non-terminal status does not support a possible interruption cause.

skills/midscene-report-analysis/references/passed-result-analysis.md

@@ -0,0 +1,18 @@
+# Midscene passed-result analysis
+
+## Goal
+
+Determine whether the recorded pass is false (`false_pass`).
+
+## Result assessments
+
+Assess each required condition at its judgment point:
+
+| Result assessment | Use when |
+| --- | --- |
+| `true_pass` | Every required condition is supported. |
+| `false_pass` | At least one required condition is proven unmet. |
+| `unverifiable` | Evidence is insufficient for at least one required condition. |
+| `inconclusive` | Reliable evidence conflicts or remains inherently ambiguous. |
+
+Judge `resultAssessment` separately from cause attribution. Attribute the evidenced issue relevant to the assessment: a recovered issue for `true_pass`, the incorrect pass for `false_pass`, the evidence gap for `unverifiable`, or the evidence conflict for `inconclusive`. Assign confidence and an evidence-based reason to every category, explain whether multiple entries are joint or alternative, and omit unrelated or evidence-free possibilities.