fix(evaluator): guard HarborRunner submissions to be subprocess only - #1062
Merged
Conversation
Contributor
|
ngoncharenko
force-pushed
the
ngoncharenko/aalgo-343-k8s-harbor-guard
branch
2 times, most recently
from
August 5, 2026 00:33
e0cc663 to
8526e5f
Compare
ngoncharenko
marked this pull request as ready for review
August 5, 2026 00:33
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughHarbor agent-evaluation compilation now resolves execution profiles through the Jobs service, routes supported subprocess profiles, rejects incompatible profiles, and reports temporary dependency outages as HTTP 503 responses. Unit and integration tests cover these paths. ChangesHarbor compilation and dependency errors
Sequence Diagram(s)sequenceDiagram
participant AgentEvalJob
participant JobsClient
participant ExecutionProfile
AgentEvalJob->>JobsClient: Retrieve Harbor execution profile
JobsClient-->>AgentEvalJob: Return execution profile
AgentEvalJob->>ExecutionProfile: Validate backend and commands
ExecutionProfile-->>AgentEvalJob: Provide subprocess command
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/api_factory.py`:
- Around line 692-696: Update the PlatformJobDependencyUnavailableError handler
in
packages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/api_factory.py:692-696
to raise status.HTTP_503_SERVICE_UNAVAILABLE, and document HTTPException(503) at
packages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/api_factory.py:672-673.
Rename the related test and assert status code 503 in
packages/nmp_common/tests/api_factory/test_api_factory.py:1748-1773.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b4446be3-474b-47a3-8126-c3679b4dac19
📒 Files selected for processing (7)
packages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/api_factory.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/exceptions.pypackages/nmp_common/tests/api_factory/test_api_factory.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.pyplugins/nemo-evaluator/tests/integration/conftest.pyplugins/nemo-evaluator/tests/integration/test_agent_evaluate_job.pyplugins/nemo-evaluator/tests/test_agent_evaluate.py
Contributor
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/evaluator/agent-eval/harbor-runner.mdx`:
- Around line 25-27: Update the Harbor runner documentation to scope the
subprocess executor requirement exclusively to Evaluator plugin submissions
using `nemo evaluator agent-evaluate submit`. Clarify that standalone SDK runs
through `run_harbor_eval()` and `HarborAgentTaskRunner` do not require
`NMP_JOBS_ENABLE_SUBPROCESS_EXECUTOR` or a NeMo Platform restart.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e8860f33-1bd4-415e-89c0-f23f8f155a32
📒 Files selected for processing (1)
docs/evaluator/agent-eval/harbor-runner.mdx
ngoncharenko
changed the title
ngoncharenko
force-pushed
the
ngoncharenko/aalgo-343-k8s-harbor-guard
branch
from
August 5, 2026 22:35
9df81ef to
4eb9f9d
Compare
Contributor
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
arpitsardhana
approved these changes
Aug 5, 2026
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
ngoncharenko
force-pushed
the
ngoncharenko/aalgo-343-k8s-harbor-guard
branch
from
August 6, 2026 00:16
7b362c5 to
d8804c6
Compare
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Remote Harbor submissions now resolve the selected Jobs execution profile before scheduling. Supported profiles get an explicit subprocess executor; Docker, Kubernetes, and missing subprocess profiles fail with HTTP 422 before a job record is created.
Previously, the guard checked Jobs' merged profile list while executor translation only considered explicitly configured profiles. A default Docker runtime could therefore pass the guard and still run Harbor on Docker. Local Harbor and non-Harbor paths are unchanged.
Tracked by AALGO-343 and AALGO-474.
Review focus
plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py— profile resolution, explicit subprocess routing, and backend rejectionplugins/nemo-evaluator/tests/test_agent_evaluate.py— supported/unsupported profile coverage and dependency failure handlingplugins/nemo-evaluator/tests/integration/test_agent_evaluate_job.py— Docker-backed rejection and the no-job-record assertionpackages/nemo_platform_plugin/src/nemo_platform_plugin/jobs/api_factory.pyandexceptions.py— HTTP 422 mapping for profile lookup failuresValidation
ty, and diff checks — passedSummary by CodeRabbit
New Features
Bug Fixes
Documentation