Skip to content

feat(aeh): MLflow logging, tool/MCP checks, and advisory regression - #61

Merged
GuyZivRH merged 12 commits into
mainfrom
feat/aeh-mlflow-tools-monitoring
Aug 3, 2026
Merged

feat(aeh): MLflow logging, tool/MCP checks, and advisory regression#61
GuyZivRH merged 12 commits into
mainfrom
feat/aeh-mlflow-tools-monitoring

Conversation

@GuyZivRH

@GuyZivRH GuyZivRH commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Wire Agent Eval Harness (AEH) observability and advisory checks into CI/monitoring: MLflow logging after evaluate, tool/MCP input validation, optional non-blocking regression and harness inventory, plus shared-namespace MLflow deploy defaults.

Changes

  • Add post-evaluate MLflow logging (scripts/log_aeh_mlflow.py) with log-results, push-feedback, and optional sync-dataset (schema mapping inferred from input.yaml:prompt when present)
  • Fall back to a minimal logger when upstream log_results.py skips because mlflow is missing; ensure mlflow is importable in the evaluate step
  • Harden AEH scoring: avoid null judges mapping to reward 1.0, bake LiteLLM judge model into task packages, set reward score_range to [0, 1] for LLM judges
  • Preserve Harbor tool/MCP interception artifacts; validate inputs.tools / handlers in validate.py
  • Add advisory (non-blocking) AEH regression (score.py regression) and harness inventory (eval-check) via new scripts and evaluate/test task steps
  • Expose pipeline params: enable-aeh-regression-check, aeh-baseline-run-id, enable-aeh-eval-check (CI); pass AEH-related params through monitoring pipelines
  • Add MLflow OpenShift manifests under config/mlflow/ and retarget defaults to the shared ab-eval-flow namespace
  • Sample aeh-hello-world updates (eval config, skill package) plus unit tests for the new paths

Test plan

  • python -m pytest — 1100 passed
  • Manual: AEH evaluate run logs MLflow results (and push-feedback / sync-dataset when mapping available)
  • Manual: advisory regression / eval-check emit reports without failing the PipelineRun
  • Optional: set aeh-baseline-run-id and confirm regression compares against that run

Related

  • Builds on AEH engine integration (APPENG-5300)
  • Harbor/AEH MLflow traces work upstream (agent-eval-harness)

GuyZivRH added a commit that referenced this pull request Aug 2, 2026
Format with ruff, resolve pairwise MLflow run dirs, tighten MLflow host
allowlist, document optional Route, and only set score_range on injected rewards.
@GuyZivRH
GuyZivRH force-pushed the feat/aeh-mlflow-tools-monitoring branch from 39eb77e to 145be7b Compare August 2, 2026 08:40
@GuyZivRH GuyZivRH changed the title Feat/aeh mlflow tools monitoring feat(aeh): MLflow logging, tool/MCP checks, and advisory regression Aug 2, 2026
GuyZivRH added a commit that referenced this pull request Aug 2, 2026
Bake mlflow into AEH image (keep /tmp pip fallback), harden MLflow no-op
detection, default eval-check off, portable PVC, ops docs, missing job-dir rewrite.
GuyZivRH added 12 commits August 3, 2026 13:51
…rams

Wire optional post-evaluate MLflow logging, preserve Harbor tool
interception artifacts, validate inputs.tools/handlers, and expose AEH
params on monitoring pipelines. Add a dedicated MLflow deploy for
guy-ziv-evalflow and sample mlflow.experiment on aeh-hello-world.
Upstream log_results.py exits 0 if mlflow is missing, which skipped our
minimal logger. Detect that case, require an importable mlflow package,
and install via python -m pip in the evaluate step.
Prevent null judges from defaulting to reward 1.0, bake LiteLLM
claude-sonnet into task packages, add the hello-world skill package,
and rewrite absolute harbor_job_dir for AEH MLflow traces.
AEH defaults weighted rewards to Likert [1, 5], which mapped [0, 1]
judge scores of 1.0 to overall reward 0.0.
Replace personal guy-ziv-evalflow namespace hardcoding in MLflow
manifests and trigger_test_runs.sh defaults with the shared cluster
namespace used by the rest of config/.
Wire non-blocking AEH score regression and harness inventory into evaluate/test,
and expand MLflow logging to push-feedback plus optional sync-dataset.
Format with ruff, resolve pairwise MLflow run dirs, tighten MLflow host
allowlist, document optional Route, and only set score_range on injected rewards.
Bake mlflow into AEH image (keep /tmp pip fallback), harden MLflow no-op
detection, default eval-check off, portable PVC, ops docs, missing job-dir rewrite.
Rely on upstream message markers only so exit-0 success output is not
misclassified when mlflow is absent from the test environment.
Force --experiment to the PipelineRun id so control/treatment share one
experiment as sibling runs; patch AEH config skill to the reports folder.
Pin notes to AEH 1.30.1; prefer full mlflow over skinny so Harbor trace
export APIs are available in the evaluate image.
@GuyZivRH
GuyZivRH force-pushed the feat/aeh-mlflow-tools-monitoring branch from 1f02dc7 to d24c5f8 Compare August 3, 2026 10:52
@GuyZivRH
GuyZivRH merged commit ec050e9 into main Aug 3, 2026
1 check passed
@GuyZivRH
GuyZivRH deleted the feat/aeh-mlflow-tools-monitoring branch August 3, 2026 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant