feat(aeh): MLflow logging, tool/MCP checks, and advisory regression - #61
Merged
Conversation
GuyZivRH
added a commit
that referenced
this pull request
Aug 2, 2026
Format with ruff, resolve pairwise MLflow run dirs, tighten MLflow host allowlist, document optional Route, and only set score_range on injected rewards.
GuyZivRH
force-pushed
the
feat/aeh-mlflow-tools-monitoring
branch
from
August 2, 2026 08:40
39eb77e to
145be7b
Compare
GuyZivRH
added a commit
that referenced
this pull request
Aug 2, 2026
Bake mlflow into AEH image (keep /tmp pip fallback), harden MLflow no-op detection, default eval-check off, portable PVC, ops docs, missing job-dir rewrite.
…rams Wire optional post-evaluate MLflow logging, preserve Harbor tool interception artifacts, validate inputs.tools/handlers, and expose AEH params on monitoring pipelines. Add a dedicated MLflow deploy for guy-ziv-evalflow and sample mlflow.experiment on aeh-hello-world.
Upstream log_results.py exits 0 if mlflow is missing, which skipped our minimal logger. Detect that case, require an importable mlflow package, and install via python -m pip in the evaluate step.
Prevent null judges from defaulting to reward 1.0, bake LiteLLM claude-sonnet into task packages, add the hello-world skill package, and rewrite absolute harbor_job_dir for AEH MLflow traces.
AEH defaults weighted rewards to Likert [1, 5], which mapped [0, 1] judge scores of 1.0 to overall reward 0.0.
Replace personal guy-ziv-evalflow namespace hardcoding in MLflow manifests and trigger_test_runs.sh defaults with the shared cluster namespace used by the rest of config/.
Wire non-blocking AEH score regression and harness inventory into evaluate/test, and expand MLflow logging to push-feedback plus optional sync-dataset.
Format with ruff, resolve pairwise MLflow run dirs, tighten MLflow host allowlist, document optional Route, and only set score_range on injected rewards.
Bake mlflow into AEH image (keep /tmp pip fallback), harden MLflow no-op detection, default eval-check off, portable PVC, ops docs, missing job-dir rewrite.
Rely on upstream message markers only so exit-0 success output is not misclassified when mlflow is absent from the test environment.
Force --experiment to the PipelineRun id so control/treatment share one experiment as sibling runs; patch AEH config skill to the reports folder.
Pin notes to AEH 1.30.1; prefer full mlflow over skinny so Harbor trace export APIs are available in the evaluate image.
GuyZivRH
force-pushed
the
feat/aeh-mlflow-tools-monitoring
branch
from
August 3, 2026 10:52
1f02dc7 to
d24c5f8
Compare
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wire Agent Eval Harness (AEH) observability and advisory checks into CI/monitoring: MLflow logging after evaluate, tool/MCP input validation, optional non-blocking regression and harness inventory, plus shared-namespace MLflow deploy defaults.
Changes
scripts/log_aeh_mlflow.py) withlog-results,push-feedback, and optionalsync-dataset(schema mapping inferred frominput.yaml:promptwhen present)log_results.pyskips becausemlflowis missing; ensuremlflowis importable in the evaluate stepscore_rangeto[0, 1]for LLM judgesinputs.tools/ handlers invalidate.pyscore.py regression) and harness inventory (eval-check) via new scripts and evaluate/test task stepsenable-aeh-regression-check,aeh-baseline-run-id,enable-aeh-eval-check(CI); pass AEH-related params through monitoring pipelinesconfig/mlflow/and retarget defaults to the sharedab-eval-flownamespaceaeh-hello-worldupdates (eval config, skill package) plus unit tests for the new pathsTest plan
python -m pytest— 1100 passedaeh-baseline-run-idand confirm regression compares against that runRelated
APPENG-5300)