[matthewgrossman]

Summary

Adds e2e test coverage for the platform jobs system running on the subprocess executor, enables mock inference provider for e2e tests, and fixes a bug where subprocess job logs had epoch-zero timestamps causing wrong ordering.

Part of the e2e test migration tracked by AIRCORE-715.

Changes

e2e/conftest.py

• Removed --service-group all from the nemo services run args. Plain nemo services run starts all services and controllers via the default fallback. Passing --service-group all explicitly populated selected_services, which skipped the fallback that also starts controllers — meaning the jobs scheduler never ran and jobs stayed in created forever.
• Added pytest_configure hook that sets NMP_INFERENCE_GATEWAY_MOCK_PROVIDER_PREFIX=igw-mock- and clears the Configuration cache, so add_mock_provider() works in e2e tests.

e2e/test_jobs.py — 13 tests (8 passing, 5 skipped), ported from Platform-Deploy

• test_basic_platform_job_lifecycle — create, run, complete, check logs
• test_job_logs_across_multiple_batches — OTLP log capture across batches with ordering assertion
• test_job_config_is_readable — job reads config from $NEMO_JOB_STEP_CONFIG_FILE_PATH
• test_job_passing_data_between_steps — multi-step job with persistent storage
• test_job_using_secret_environment_variable — secret injected as env var
• test_job_with_expected_failure — non-zero exit → error status
• test_job_cancel_immediately — cancel right after creation
• test_job_cancel_once_active — wait for active, then cancel via SIGTERM
• test_job_pause_resume — skipped: subprocess backend has no SIGSTOP/SIGCONT handling (AIRCORE-721)
• test_job_pause_and_cancel — skipped: same reason
• test_job_using_additional_volume — skipped: Docker-only (container volume mounts)
• test_job_invalid_image_format — skipped: image is discarded by cpu→subprocess translation

e2e/test_inference.py — 10 tests for mock inference provider

• Provider CRUD, chat completions via provider/model/OpenAI routes, streaming, model listing, error simulation, per-model sequential responses, virtual model creation

subprocess_runtime.py — bug fix

• SubprocessOtelLogger.emit() was not passing a timestamp to the OpenTelemetry logger. This caused timeUnixNano=0 in the protobuf payload (protobuf defaults None to 0 for uint64). All logs got epoch-zero timestamps, so ORDER BY timestamp produced arbitrary ordering.
• Fix: pass timestamp=time_ns().

otlp/endpoints.py — defensive fix

• If timeUnixNano is 0, fall back to observedTimeUnixNano (which OpenTelemetry auto-populates). Protects against any other OTLP emitter that omits the timestamp.

How jobs work in subprocess mode

Tests submit jobs with CPUExecutionProviderSpec (same as Docker). The jobs service's translate_cpu_container_steps_to_subprocess() automatically rewrites cpu/default steps to subprocess/default — discarding the container image and running the command directly on the host. This means the same test code works across both backends.

Test plan

• uv run --frozen pytest e2e/ -v --run-e2e — 22 passed, 5 skipped
• uv run --frozen pytest services/core/jobs/tests/controllers/test_subprocess_runtime.py -v — 8 passed
• Verified log ordering is correct after timestamp fix
• Verified test_health_ready passes (was failing when --service-group all was used due to controllers not starting)

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests

  • Added comprehensive end-to-end test suite for Inference Gateway covering chat completions, streaming, model listing, and error handling.
  • Added extensive end-to-end test suite for job management covering lifecycle, cancellation, data passing between steps, and failure handling.

• Chores

  • Improved logging infrastructure with nanosecond-precision timestamps and enhanced timestamp parsing fallback logic.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	18658/24692	75.6%	62.0%
Integration Tests	11978/23469	51.0%	26.2%

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds two complementary E2E test suites with supporting infrastructure: inference gateway tests validating mock provider routing and features, and job execution tests covering subprocess-backed job lifecycle, data handling, and state management.

Changes

Inference Gateway E2E Tests (Mock Provider Mode)

Layer / File(s)	Summary
Mock provider environment configuration e2e/conftest.py	pytest_configure hook activates mock inference provider mode by setting NMP_INFERENCE_GATEWAY_MOCK_PROVIDER_PREFIX and clearing configuration cache.
Provider CRUD and basic routing e2e/test_inference.py	Test module setup with _unique_name() helper; provider creation/deletion flow; routing tests for /v1/chat/completions by provider name, model entity name, and OpenAI-compatible endpoint.
Advanced gateway scenarios e2e/test_inference.py	Streaming SSE output, OpenAI /v1/models endpoint, error simulation (HTTP 500 with InternalServerError), sequential per-model responses with clamping, and auto-provisioned VirtualModel validation.

Job Execution E2E Tests (Subprocess Backend)

Layer / File(s)	Summary
Service infrastructure and logging improvements e2e/conftest.py, services/core/jobs/.../subprocess_runtime.py, services/core/files/.../otlp/endpoints.py	Service startup removes --service-group all flag; subprocess runtime adds nanosecond-precision timestamps via time_ns(); OTLP fallback uses observedTimeUnixNano when timeUnixNano is zero/missing.
Job lifecycle and log handling e2e/test_jobs.py	Module setup with _job_diagnostic_message() for assertion context; basic job lifecycle (create → complete → verify command output); multi-batch log ordering test with delayed echo output.
Job data and secrets handling e2e/test_jobs.py	Config file readability via NEMO_JOB_STEP_CONFIG_FILE_PATH; persistent data passing between steps via job storage; secrets attached as environment variables with verification in logs.
Job state transitions and failure handling e2e/test_jobs.py	Expected failure handling (non-zero exit → error status); immediate cancellation; cancellation during active state; all assert final cancelled or error status with log validation.
Docker-backend-only and image validation tests e2e/test_jobs.py	Three skipped pause/resume/volume tests due to subprocess backend limitations (SIGSTOP/SIGCONT, container mounts); parametrized invalid image format test validating job failure (image validation bypassed in subprocess translation).

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#125: Modifies e2e/conftest.py service launcher configuration and E2E test harness behavior.

Suggested reviewers

• benmccown
• mckornfield
• svvarom

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title 'feat(e2e): Add jobs e2e tests w/ fixes, mock inference provider' directly maps to the main changes: adding e2e tests for jobs, introducing a mock inference provider, and fixing subprocess log timestamps.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch subprocess-jobs-e2e/mgrossman

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@e2e/test_jobs.py`:
- Around line 216-218: The test currently fetches step_logs once via
sdk.jobs.get_logs(workspace, name=job.name) and asserts immediately that the
combined messages contain "Data from first step", which races OTLP flush/ingest;
change this to poll/retry fetching step_logs until the expected message appears
or a timeout elapses (e.g., loop with small sleep and overall timeout), using
sdk.jobs.get_logs(workspace, name=job.name) and job.name as the identifier, then
assert after the poll completes or fail with a clear message if timed out.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fa3aff01-46ff-4be4-9c4a-3406e8420069

📥 Commits

Reviewing files that changed from the base of the PR and between bd33431 and aeaa580.

📒 Files selected for processing (5)

• e2e/conftest.py
• e2e/test_inference.py
• e2e/test_jobs.py
• services/core/files/src/nmp/core/files/api/v2/otlp/endpoints.py
• services/core/jobs/src/nmp/core/jobs/controllers/backends/subprocess_runtime.py

[mckornfield]

I reviewed a PR ryan did as well that added the test_jobs file, might be worth coordinating