feat: add azd CLI evaluation and testing framework#7202

Open

spboyer wants to merge 21 commits intomainfrom

feat/eval-framework

Commits on Apr 2, 2026

feat: add azd CLI evaluation and testing framework

spboyer
and
Copilot
committed
docs: add authentication and secrets section to eval README

spboyer
and
Copilot
committed
docs: add comprehensive how-to guides for creating evals, graders, and tests

spboyer
and
Copilot
committed
fix: resolve CI failures in eval unit tests and cspell

spboyer
and
Copilot
committed
fix: stop command-sequencing tests from overriding AZD_CONFIG_DIR

spboyer
and
Copilot
committed
docs: expand auth section with subscription config and no-popup guarantee

spboyer
and
Copilot
committed
refactor: address review feedback from @jongio and Copilot

spboyer
and
Copilot
committed
fix: address round 2 review feedback from @jongio

spboyer
and
Copilot
committed
Fix review: mock targets, env remove, CI grader tests, HTTPError handling

spboyer
and
Copilot
committed
fix: skip waza validation when CLI not installed

spboyer
and
Copilot
committed
Update cli/azd/test/eval/tasks/environment/switch-env.yaml

spboyer
and
jongio
committed
Update cli/azd/test/eval/README.md

spboyer
and
jongio
committed
fix: address remaining PR review feedback

spboyer
and
Copilot
committed
chore: add 40-minute timeout to eval-e2e lifecycle step

spboyer
and
Copilot
committed
Address review feedback on eval framework

spboyer
and
Copilot
committed
Remove continue-on-error masking in eval-waza.yml

spboyer
and
Copilot
committed
Fix error handling in graders and cleanup step

spboyer
and
Copilot
committed
Address review feedback on eval framework

spboyer
and
Copilot
committed
Address review feedback on eval framework (round 2)

spboyer
and
Copilot
committed
Re-trigger CI (flaky TestInitializer_PromptIfNonEmpty)

spboyer
and
Copilot
committed
Use tag-for-deletion pattern in eval cleanup step

spboyer
and
Copilot
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add azd CLI evaluation and testing framework#7202

feat: add azd CLI evaluation and testing framework#7202
spboyer wants to merge 21 commits intomainfrom
feat/eval-framework

Commits on Apr 2, 2026

feat: add azd CLI evaluation and testing framework

docs: add authentication and secrets section to eval README

docs: add comprehensive how-to guides for creating evals, graders, and tests

fix: resolve CI failures in eval unit tests and cspell

fix: stop command-sequencing tests from overriding AZD_CONFIG_DIR

docs: expand auth section with subscription config and no-popup guarantee

refactor: address review feedback from @jongio and Copilot

fix: address round 2 review feedback from @jongio

Fix review: mock targets, env remove, CI grader tests, HTTPError handling

fix: skip waza validation when CLI not installed

Update cli/azd/test/eval/tasks/environment/switch-env.yaml

Update cli/azd/test/eval/README.md

fix: address remaining PR review feedback

chore: add 40-minute timeout to eval-e2e lifecycle step

Address review feedback on eval framework

Remove continue-on-error masking in eval-waza.yml

Fix error handling in graders and cleanup step

Address review feedback on eval framework

Address review feedback on eval framework (round 2)

Re-trigger CI (flaky TestInitializer_PromptIfNonEmpty)

Use tag-for-deletion pattern in eval cleanup step