Update Vally to 0.14.0

[PureWeen]

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Description

Updates the pinned @microsoft/vally-cli version used by skill validation and live soak workflows from 0.12.0 to the latest stable release, 0.14.0.

Also updates direct lint command examples and makes eval-spec comments version-neutral where the described scoring behavior remains unchanged.

Validation

• Installed Vally 0.14.0 through .github/scripts/SetupVallyRuntime.sh
• Strict-linted all 16 Vally evaluation specifications with Vally 0.14.0

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 37907

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 37907"

[azure-pipelines]

Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR updates the pinned @microsoft/vally-cli version used by GitHub Actions skill validation and live soak workflows from 0.12.0 to 0.14.0, and refreshes related eval-spec commentary/docs to be version-neutral where behavior is expected to remain unchanged.

Changes:

• Bump VALLY_VERSION to 0.14.0 in the skill validation and soak workflows.
• Remove hard-coded “Vally 0.12 …” wording in eval-spec comments, keeping the explanations version-neutral.
• Update docs/examples that show the npx @microsoft/vally-cli@... lint command to use 0.14.0.

Show a summary per file

File	Description
.github/workflows/skill-validation.yml	Updates VALLY_VERSION to 0.14.0 and refreshes Node requirement comment.
.github/workflows/skill-evaluation-soak.yml	Updates VALLY_VERSION to 0.14.0 for the scheduled soak workflow.
.github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/try-fix/tests/eval.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/pr-review/tests/eval.gh-auth.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/evaluate-pr-tests/tests/eval.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/code-review/tests/soak.capability.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/code-review/tests/hermeticity.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/code-review/tests/eval.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/code-review/tests/eval.inline-findings.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/analyze-sessions/tests/eval.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/skills/analyze-sessions/SKILL.md	Updates the example lint command to @microsoft/vally-cli@0.14.0.
.github/skills/analyze-sessions/references/design-rationale.md	Updates referenced lint command to @microsoft/vally-cli@0.14.0.
.github/skills/agentic-labeler/tests/eval.vally.yaml	Makes scoring commentary version-neutral (removes “0.12” mention).
.github/scripts/SetupVallyRuntime.sh	Updates a comment to remove the version-specific “0.12” phrasing.

Review details

• Files reviewed: 15/15 changed files
• Comments generated: 2
• Review effort level: Lite

[github-actions]

Skill Validation Results

@PureWeen — new skill validation results are available based on this last commit: 617beb1.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

⚠️ Skill Validation Results — 617beb1 · Update Vally to 0.14.0 · 2026-08-27T18:38:27Z

✅ Static Checks Passed

Skills: 24 | Eval specs linted: 16

Full lint output

── .github/skills/agentic-labeler/tests/eval.vally.yaml
npm warn EBADENGINE Unsupported engine {
npm warn EBADENGINE   package: '@microsoft/vally-cli@0.12.0',
npm warn EBADENGINE   required: { npm: '>=11.11.1', node: '>=22.0.0' },
npm warn EBADENGINE   current: { node: 'v22.23.2', npm: '10.9.8' }
npm warn EBADENGINE }
npm warn EBADENGINE Unsupported engine {
npm warn EBADENGINE   package: '@microsoft/vally@0.12.0',
npm warn EBADENGINE   required: { npm: '>=11.11.1', node: '>=22.0.0' },
npm warn EBADENGINE   current: { node: 'v22.23.2', npm: '10.9.8' }
npm warn EBADENGINE }
npm warn EBADENGINE Unsupported engine {
npm warn EBADENGINE   package: '@microsoft/vally-server@0.12.0',
npm warn EBADENGINE   required: { npm: '>=11.11.1', node: '>=22.0.0' },
npm warn EBADENGINE   current: { node: 'v22.23.2', npm: '10.9.8' }
npm warn EBADENGINE }
Vally collects pseudonymous usage telemetry (command name, version, outcome, duration, persistent device identifier (when available), and coarse OS/runtime info) to improve the product. No prompts, datasets, file paths, or arguments are collected.
Opt out any time by setting VALLY_TELEMETRY_OPTOUT=1 or DO_NOT_TRACK=1.
Learn more: https://aka.ms/vally/telemetry
✔ .github/skills/agentic-labeler/tests/eval.vally.yaml is valid
── .github/skills/analyze-sessions/tests/eval.vally.yaml
✔ .github/skills/analyze-sessions/tests/eval.vally.yaml is valid
── .github/skills/ci-fix/tests/eval.ownership.vally.yaml
✔ .github/skills/ci-fix/tests/eval.ownership.vally.yaml is valid
── .github/skills/ci-fix/tests/eval.vally.yaml
✔ .github/skills/ci-fix/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/eval.inline-findings.vally.yaml
✔ .github/skills/code-review/tests/eval.inline-findings.vally.yaml is valid
── .github/skills/code-review/tests/eval.producer-trace.vally.yaml
✔ .github/skills/code-review/tests/eval.producer-trace.vally.yaml is valid
── .github/skills/code-review/tests/eval.trim-aot.vally.yaml
✔ .github/skills/code-review/tests/eval.trim-aot.vally.yaml is valid
── .github/skills/code-review/tests/eval.vally.yaml
✔ .github/skills/code-review/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/hermeticity.vally.yaml
✔ .github/skills/code-review/tests/hermeticity.vally.yaml is valid
── .github/skills/code-review/tests/soak.capability.vally.yaml
✔ .github/skills/code-review/tests/soak.capability.vally.yaml is valid
── .github/skills/evaluate-pr-tests/tests/eval.vally.yaml
✔ .github/skills/evaluate-pr-tests/tests/eval.vally.yaml is valid
── .github/skills/pr-review/tests/eval.gh-auth.vally.yaml
✔ .github/skills/pr-review/tests/eval.gh-auth.vally.yaml is valid
── .github/skills/try-fix/tests/eval.restore.vally.yaml
✔ .github/skills/try-fix/tests/eval.restore.vally.yaml is valid
── .github/skills/try-fix/tests/eval.vally.yaml
✔ .github/skills/try-fix/tests/eval.vally.yaml is valid
── .github/skills/verify-tests-fail-without-fix/tests/eval.protocol.vally.yaml
✔ .github/skills/verify-tests-fail-without-fix/tests/eval.protocol.vally.yaml is valid
── .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml
✔ .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml is valid

⚠️ LLM Evaluation Incomplete

No valid JUnit suite results were found for: agentic-labeler, analyze-sessions, ci-fix, code-review, evaluate-pr-tests, pr-review, try-fix, verify-tests-fail-without-fix.

🔍 Full results and investigation steps