feat(build): Add CI test that builds+tests the wheels

[matthewgrossman]

What

Adds a CI job that builds and tests every shipping wheel — nemo-platform and nemo-platform-plugin — across every supported Python (3.11, 3.12, 3.13). Each row builds the wheel with the exact same code path the release workflow uses, installs it into an isolated env, and exercises it end-to-end. Today CI only runs against the editable workspace install, so wheel-install breakages (missing policy.wasm, vendor-metadata drift, entry-point typos, namespace-package gotchas, broken Requires-Dist, generated-extra mismatches) only surface on manual post-release smoke checks.

The build itself is factored into a new composite action, .github/actions/build-nemo-platform-wheel, which is also called by release-bundle.yaml's build-sdks matrix. CI test wheels and published wheels now come out of one code path.

Shape

policy-wasm  ──▶  wheel-test  ──▶  wheel-test-aggregate
                  matrix(package × python)        (required-check pin)
                  6 rows

• wheel-test matrix is package: [nemo-platform, nemo-platform-plugin] × python-version: [3.11, 3.12, 3.13] = 6 rows. fail-fast: false so a single-row failure doesn't hide the others' logs.
• wheel-test-aggregate is a single-job umbrella that succeeds iff every matrix row succeeded. Branch protection should pin this check name (Wheel build + test); the matrix can grow without anyone touching protection settings.
• Composite action owns the build path (uv setup, conditional Studio asset compile, version stamping via stamp_sdk_version.py, uv build --wheel, single-wheel guard, rm -rf safety check on caller-provided out-dir). Stamping always runs regardless of caller, with cadence: nightly + sentinel epoch 19700101000000 for CI tests so the test wheel version is well-formed PEP 440 dev but obviously synthetic.

Per-package test surface

• nemo-platform: uv tool install ./*.whl[services], then script/test-nemo-cli.sh boots the platform via nemo services run (no --services filter — every bundled service module loads, ~17 services + 4 controllers) and polls nemo workspaces list -f json until both default and system (the platform-seeded workspaces) appear, with SIGTERM-then-SIGKILL teardown of the whole process group via bash job control (set -m).
• nemo-platform-plugin: library-only (no [project.scripts]), so uv venv + uv pip install + python -c "import nemo_platform_plugin; import nemo_platform_plugin.cli; import nemo_platform_plugin.commands". That still catches the bug classes a wheel-test is supposed to catch (broken pyproject.toml, broken Requires-Dist, hatch_build.py regressions, namespace-package gotchas).

script/test-nemo-cli.sh is install-method agnostic — anywhere nemo is on PATH (uv tool install from a wheel, uv sync in dev, pipx, distro package), the script can run and prove the CLI works.

What this does NOT do (yet)

• Not yet a required check. The aggregator name Wheel build + test is set up to be the single pin point; flipping branch-protection-required is a one-line admin change once this has been green for a few working days.
• Doesn't gate release-stable.yaml. That's the second admin step: extend the stable-release dispatch to fail if Wheel build + test didn't pass on the source SHA.
• Platform-Deploy migration is separate. Platform-Deploy/.github/workflows/release-sdk.yaml still has its own inline build path that drifts from this action; migrating it to consume build-nemo-platform-wheel@<sha> is a Platform-Deploy PR.
• No vendor-drift handling. The test deliberately runs against committed metadata; vendor-drift detection belongs in lint.

Coderabbit feedback addressed

• rm -rf "${abs_out_dir}" in the action now rejects empty// paths and requires the resolved path to live inside source-root or $GITHUB_WORKSPACE.
• Readiness probe now requires both default and system workspaces (the entities service seeds them back-to-back, so missing one is a real signal that init partially failed).
• github.run_started_at reference removed in favor of the sentinel epoch (the context property doesn't exist).
• SHA-pinning of actions/checkout@v6 etc. and persist-credentials: false deferred — every other job in ci.yaml uses major-version tags, so policy-pinning the wheel-test job alone would be inconsistent. Belongs in a separate repo-wide PR.

Verification

• All YAML files parse with yaml.safe_load.
• script/test-nemo-cli.sh is shellcheck and bash -n clean.
• Pre-commit clean on all changed files.
• Most recent CI run on this branch: 7 jobs green (6 matrix rows + aggregator).

Tracking

• Linear: AIRCORE-682
• Plan: docs/plans/aircore-682-wheel-test.md

Signed-off-by: Matthew Grossman mgrossman@nvidia.com

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a composite GitHub Action to stamp and build a single nemo-platform wheel, integrates it into CI for nightly wheel testing, refactors the release workflow to use the action, and adds a CLI integration test script.

Changes

Wheel build action and CI/release integration

Layer / File(s)	Summary
Composite wheel build action .github/actions/build-nemo-platform-wheel/action.yaml	New composite action defines inputs/outputs, validates cadence/release inputs, conditionally installs pnpm/Node for nemo-platform, runs stamp_sdk_version.py, builds the wheel via uv build --wheel, verifies exactly one .whl exists, and exports wheel-path and wheel-version.
Release workflow migration .github/workflows/release-bundle.yaml	Refactors build-sdks job to call the composite action instead of inline setup/stamp/build steps; artifact upload now uses the action's wheel-path output.
CI wheel-test job .github/workflows/ci.yaml	New job downloads policy.wasm, synthesizes nightly timestamp, builds the wheel via the composite action, installs the built wheel into an isolated env using uv tool install, runs script/test-nemo-cli.sh from outside the workspace with cleaned env, and uploads dist/*.whl as an artifact.
CLI integration test script script/test-nemo-cli.sh	Bash script that verifies nemo on PATH and version, runs import-time subcommand smoke tests, starts nemo services run in background, polls up to 60s for workspace readiness, captures logs, and performs trap-based cleanup.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	Title accurately summarizes the main change: adding CI infrastructure (composite action, test job, script) to build and test wheels in isolation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch mgrossman/aircore-682-ci-build-wheel-install-in-fresh-env-smoke-test-nemo-platform

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/actions/build-nemo-platform-wheel/action.yaml:
- Around line 171-178: The script currently does an unsafe rm -rf on
"${abs_out_dir}" which can delete arbitrary absolute paths; before running rm
-rf validate and guard abs_out_dir by (1) rejecting empty or "/" values and
exiting non‑zero, and (2) ensuring abs_out_dir is inside the SOURCE_ROOT path
(e.g., check that abs_out_dir starts with "$(cd "${SOURCE_ROOT}" && pwd)/" or
equivalent) so caller-provided OUT_DIR cannot point outside source tree; perform
these checks after computing abs_out_dir and only then run rm -rf
"${abs_out_dir}".

In @.github/workflows/ci.yaml:
- Around line 213-216: In the wheel-test job update the checkout step (uses:
actions/checkout@v6) to disable credential persistence by adding with:
persist-credentials: false and pin the action to its full commit SHA; likewise
replace uses: actions/download-artifact@v8 and uses: actions/upload-artifact@v6
with their respective full commit SHAs to avoid floating tags—locate the
checkout, download-artifact, and upload-artifact steps by those exact uses
strings and update them accordingly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 729eeeef-86e9-4daa-bd13-59882a20c6ef

📥 Commits

Reviewing files that changed from the base of the PR and between ba0bbfc and 610a04e.

📒 Files selected for processing (4)

• .github/actions/build-nemo-platform-wheel/action.yaml
• .github/workflows/ci.yaml
• .github/workflows/release-bundle.yaml
• script/test-nemo-cli.sh

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	18244/24191	75.4%	61.9%
Integration Tests	11664/22973	50.8%	25.9%

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yaml:
- Around line 223-231: The workflow uses a non-existent context property
github.run_started_at when setting RUN_STARTED_AT; replace RUN_STARTED_AT: ${{
github.run_started_at }} with a valid context such as RUN_STARTED_AT: ${{
github.event.workflow_run.run_started_at }} (or another appropriate event field
for your trigger) so the subsequent ts="$(date -u -d "${RUN_STARTED_AT}"
+%Y%m%d%H%M%S)" call in the run step gets a real ISO-8601 timestamp; update the
env assignment and verify stamp_sdk_version.py invocation still receives the
expected 14-digit string.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 14ac5ae2-d993-4d6d-9f49-87297d99bfaa

📥 Commits

Reviewing files that changed from the base of the PR and between 610a04e and 14c826e.

📒 Files selected for processing (1)

• .github/workflows/ci.yaml

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@script/test-nemo-cli.sh`:
- Around line 81-85: The readiness loop currently only checks for the "default"
workspace via the jq filter '.data[]? | select(.name == "default")' which can
yield false positives; update the check used in the until condition (the call to
`nemo workspaces list -f json` and its jq predicate) to assert that both
"default" and "system" workspaces exist (e.g., change the jq expression to test
presence of both names or to count/match both entries) so the loop only succeeds
when both seeded workspaces are present.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3a33d091-2d7a-4304-a3ea-c719825c70d2

📥 Commits

Reviewing files that changed from the base of the PR and between 83d71dd and 58f4054.

📒 Files selected for processing (1)

• script/test-nemo-cli.sh

[mckornfield]

two minors and a major, but probably fine (can stamp after)