Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
ad8e948
fix(hermes): keep lazy installer probe offline
apurvvkumaria Aug 14, 2026
04683ce
fix(ci): build PR images from local base
prekshivyas Aug 14, 2026
2845e61
fix(ci): scope local base builds
prekshivyas Aug 14, 2026
8d76b4b
test(ci): cover registry build ordering
cv Aug 14, 2026
1afbc32
fix(ci): publish local-base images with Docker driver
cv Aug 14, 2026
545537e
fix(ci): select builder for digest publication
cv Aug 14, 2026
26f4705
fix(hermes): clear base dependency probe state
cv Aug 14, 2026
5bdfcfb
Merge branch 'main' into codex/fix-protected-hermes-offline-cache
prekshivyas Aug 14, 2026
58df15e
Merge branch 'main' into codex/fix-protected-hermes-offline-cache
prekshivyas Aug 14, 2026
70f1213
fix(hermes): reset offline probe target
cv Aug 14, 2026
eb2227d
fix(hermes): make denial probe status explicit
cv Aug 14, 2026
af8bbf2
fix(hermes): reset offline lazy probe state
prekshivyas Aug 14, 2026
747c60f
Merge branch 'main' into codex/fix-protected-hermes-offline-cache
prekshivyas Aug 14, 2026
4a95356
test(ci): refresh onboard decision budget
prekshivyas Aug 14, 2026
8690295
fix(ci): import local base into buildx
cv Aug 14, 2026
05ae55f
test(ci): execute OCI base resolver contract
prekshivyas Aug 14, 2026
f802c9b
Merge remote-tracking branch 'origin/main' into review/pr-9158-feedback
cv Aug 14, 2026
7c07af4
test(sandbox): keep Ollama stop checks within timeout
cv Aug 14, 2026
e55b5e5
test(onboard): preload portable lock modules
prekshivyas Aug 14, 2026
a769d59
fix(hermes): verify Hindsight probe artifacts
prekshivyas Aug 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 36 additions & 1 deletion .github/workflows/managed-images.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,7 @@ jobs:
done

- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0

- name: Set up Node.js
Expand Down Expand Up @@ -425,6 +426,7 @@ jobs:
--tag "$LOCAL_BASE_REFERENCE" \
.
printf 'ref=%s\n' "$LOCAL_BASE_REFERENCE" >> "$GITHUB_OUTPUT"
printf 'local=true\n' >> "$GITHUB_OUTPUT"
printf '### %s PR base\n\nLocally built from `%s` at `%s`.\n' \
"$DISPLAY_NAME" "$BASE_DOCKERFILE" "$CANDIDATE_SHA" \
>> "$GITHUB_STEP_SUMMARY"
Expand Down Expand Up @@ -467,6 +469,7 @@ jobs:
exit 1
fi
printf 'ref=%s\n' "$reference" >> "$GITHUB_OUTPUT"
printf 'local=false\n' >> "$GITHUB_OUTPUT"
printf '### %s PR base\n\n`%s`\n' "$DISPLAY_NAME" "$reference" \
>> "$GITHUB_STEP_SUMMARY"

Expand All @@ -483,9 +486,41 @@ jobs:
--build-arg "NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION=1" \
--build-arg "NEMOCLAW_MANAGED_IMAGE_RUNTIME_USER=root"

- name: Build PR managed image locally
- name: Build PR managed image from local base
if: steps.base.outputs.local == 'true'
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
shell: bash
env:
BASE_IMAGE: ${{ steps.base.outputs.ref }}
CANDIDATE_SHA: ${{ github.event.pull_request.head.sha }}
DOCKERFILE: ${{ matrix.dockerfile }}
IMAGE_REFERENCE: ${{ matrix.image }}:${{ github.event.pull_request.head.sha }}
run: |
set -euo pipefail
# The base resolver loads a changed base into Docker's local image
# store. Keep this consumer on that same store: a Buildx container
# builder otherwise treats the local-only reference as Docker Hub.
docker build \
--platform linux/amd64 \
--file "$DOCKERFILE" \
--tag "$IMAGE_REFERENCE" \
--label "org.opencontainers.image.source=https://github.com/${GITHUB_REPOSITORY}" \
--label "org.opencontainers.image.revision=${CANDIDATE_SHA}" \
--label "io.nvidia.nemoclaw.agent=${{ matrix.agent }}" \
--label "io.nvidia.nemoclaw.managed-image.contract=1" \
--label "io.nvidia.nemoclaw.managed-image.platform=linux/amd64" \
--label "io.nvidia.nemoclaw.managed-image.startup-profile=1" \
--label "io.nvidia.nemoclaw.managed-image.capabilities=1" \
--label "io.nvidia.nemoclaw.managed-image.cohort=ghrun-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" \
--build-arg "BASE_IMAGE=${BASE_IMAGE}" \
--build-arg "NEMOCLAW_MANAGED_IMAGE_CAPABILITY_UNION=1" \
--build-arg "NEMOCLAW_MANAGED_IMAGE_RUNTIME_USER=root" \
.

- name: Build PR managed image from registry base
if: steps.base.outputs.local != 'true'
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
with:
builder: ${{ steps.base.outputs.local == 'true' && 'default' || steps.buildx.outputs.name }}
context: .
file: ${{ matrix.dockerfile }}
platforms: linux/amd64
Expand Down
22 changes: 17 additions & 5 deletions agents/hermes/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -1153,9 +1153,10 @@ RUN set -eu; \
# uv creates the Hermes virtual environment without pip. Seed pip from Python's
# bundled ensurepip payload while this layer is still root, then verify the
# sandbox can read but cannot modify the virtual environment used by the lazy
# installer. The Hermes lazy installer still installs the approved dependency
# as the sandbox user.
RUN /opt/hermes/.venv/bin/python -I -m ensurepip --upgrade --default-pip \
# installer. Dockerfile.base separately installs and imports the official
# Hindsight client. This layer uses a build-only wheel with the same package
# identity to isolate the lazy-installer and durable-target contract from PyPI.
RUN --network=none /opt/hermes/.venv/bin/python -I -m ensurepip --upgrade --default-pip \
&& /opt/hermes/.venv/bin/python -I -m pip --version \
&& chmod 644 /opt/hermes/.venv/.lock \
&& test "$(stat -c '%U:%G %a' /opt/hermes/.venv/.lock)" = "root:root 644" \
Expand Down Expand Up @@ -1195,25 +1196,36 @@ RUN /opt/hermes/.venv/bin/python -I -m ensurepip --upgrade --default-pip \
if ln -sf /usr/bin/false /opt/hermes/.venv/bin/python 2>/dev/null; then exit 1; fi' \
&& test ! -e /opt/hermes/.venv/lib/.nemoclaw-sandbox-write-probe \
&& test "$(stat -c '%U:%G %a' /sandbox/.hermes/lazy-packages)" = "sandbox:sandbox 750" \
&& test -z "$(find /sandbox/.hermes/lazy-packages -mindepth 1 -print -quit)" \
&& install -d -o root -g root -m 0755 /tmp/nemoclaw-hindsight-probe \
&& /opt/hermes/.venv/bin/python -I -c \
"import zipfile; entries = {'hindsight_client/__init__.py': 'NEMOCLAW_BUILD_PROBE_FIXTURE = True\n', 'hindsight_client-0.6.1.dist-info/METADATA': 'Metadata-Version: 2.4\nName: hindsight-client\nVersion: 0.6.1\nSummary: NemoClaw offline lazy-installer build probe\n\n', 'hindsight_client-0.6.1.dist-info/WHEEL': 'Wheel-Version: 1.0\nGenerator: NemoClaw build probe\nRoot-Is-Purelib: true\nTag: py3-none-any\n', 'hindsight_client-0.6.1.dist-info/RECORD': 'hindsight_client/__init__.py,,\nhindsight_client-0.6.1.dist-info/METADATA,,\nhindsight_client-0.6.1.dist-info/WHEEL,,\nhindsight_client-0.6.1.dist-info/RECORD,,\n'}; wheel = zipfile.ZipFile('/tmp/nemoclaw-hindsight-probe/hindsight_client-0.6.1-py3-none-any.whl', 'w', compression=zipfile.ZIP_DEFLATED); [wheel.writestr(name, data) for name, data in entries.items()]; wheel.close()" \
&& chown root:root /tmp/nemoclaw-hindsight-probe/hindsight_client-0.6.1-py3-none-any.whl \
&& chmod 0444 /tmp/nemoclaw-hindsight-probe/hindsight_client-0.6.1-py3-none-any.whl \
&& HOME=/sandbox \
PIP_FIND_LINKS=/tmp/nemoclaw-hindsight-probe \
PIP_NO_INDEX=1 \
UV_CACHE_DIR=/sandbox/.hermes/cache/uv \
UV_FIND_LINKS=/tmp/nemoclaw-hindsight-probe \
UV_NO_CACHE=1 \
UV_OFFLINE=1 \
HERMES_HOME=/sandbox/.hermes \
HERMES_LAZY_INSTALL_TARGET=/sandbox/.hermes/lazy-packages \
/usr/bin/setpriv --reuid=sandbox --regid=sandbox --init-groups -- \
/opt/hermes/.venv/bin/python -I -c \
"from tools.lazy_deps import ensure; ensure('memory.hindsight', prompt=False)" \
&& rm -rf /tmp/nemoclaw-hindsight-probe \
&& HERMES_LAZY_INSTALL_TARGET=/sandbox/.hermes/lazy-packages \
/usr/bin/setpriv --reuid=sandbox --regid=sandbox --init-groups -- \
/opt/hermes/.venv/bin/python -I -c \
"from tools.lazy_deps import activate_durable_lazy_target; activate_durable_lazy_target(); import importlib.metadata as m; assert m.version('hindsight-client') == '0.6.1'" \
"from tools.lazy_deps import activate_durable_lazy_target; activate_durable_lazy_target(); import hindsight_client, importlib.metadata as m; from pathlib import Path; target = Path('/sandbox/.hermes/lazy-packages').resolve(); assert m.version('hindsight-client') == '0.6.1'; assert hindsight_client.NEMOCLAW_BUILD_PROBE_FIXTURE is True; assert Path(hindsight_client.__file__).resolve().is_relative_to(target)" \
&& lazy_plan='{"version":1,"readOnlyRoots":["lazy-packages"],"confidentialRoots":[],"readOnlyPrefixes":[],"confidentialPrefixes":[],"writableSubpaths":[]}' \
&& /usr/local/lib/nemoclaw/state-dir-guard.py lock \
--config-dir /sandbox/.hermes --plan-json "$lazy_plan" \
&& /usr/bin/setpriv --reuid=gateway --regid=gateway --init-groups -- \
env HERMES_LAZY_INSTALL_TARGET=/sandbox/.hermes/lazy-packages \
/opt/hermes/.venv/bin/python -I -c \
"from tools.lazy_deps import activate_durable_lazy_target; activate_durable_lazy_target(); import hindsight_client" \
"from tools.lazy_deps import activate_durable_lazy_target; activate_durable_lazy_target(); import hindsight_client; from pathlib import Path; assert hindsight_client.NEMOCLAW_BUILD_PROBE_FIXTURE is True; assert Path(hindsight_client.__file__).resolve().is_relative_to(Path('/sandbox/.hermes/lazy-packages').resolve())" \
&& if /usr/bin/setpriv --reuid=gateway --regid=gateway --init-groups -- \
sh -c ': > /sandbox/.hermes/lazy-packages/.nemoclaw-gateway-write-probe' 2>/dev/null; then \
echo "ERROR: gateway can modify locked Hermes lazy packages" >&2; exit 1; \
Expand Down
19 changes: 19 additions & 0 deletions agents/hermes/Dockerfile.base
Original file line number Diff line number Diff line change
Expand Up @@ -555,6 +555,25 @@ RUN printf '%s\n' \
"import multipart; assert multipart.__version__ == '0.0.32', multipart.__version__" \
&& rm -rf /root/.cache/uv

# Keep official-package compatibility separate from the final image's offline
# lazy-installer contract. Install the reviewed Hindsight client into a
# temporary durable target, import it against the exact base environment, and
# remove the probe before publishing the base image.
RUN install -d -o sandbox -g sandbox -m 0750 \
/tmp/nemoclaw-hindsight-client-probe \
/tmp/nemoclaw-hindsight-client-cache \
&& HOME=/sandbox \
UV_CACHE_DIR=/tmp/nemoclaw-hindsight-client-cache \
UV_NO_CACHE=1 \
HERMES_HOME=/sandbox/.hermes \
HERMES_LAZY_INSTALL_TARGET=/tmp/nemoclaw-hindsight-client-probe \
/usr/bin/setpriv --reuid=sandbox --regid=sandbox --init-groups -- \
/opt/hermes/.venv/bin/python -I -c \
"from tools.lazy_deps import ensure; ensure('memory.hindsight', prompt=False); import hindsight_client, importlib.metadata as m; from pathlib import Path; target = Path('/tmp/nemoclaw-hindsight-client-probe').resolve(); assert m.version('hindsight-client') == '0.6.1'; assert Path(hindsight_client.__file__).resolve().is_relative_to(target)" \
&& rm -rf \
/tmp/nemoclaw-hindsight-client-probe \
/tmp/nemoclaw-hindsight-client-cache

ENV PATH="/usr/local/bin:/opt/hermes/.venv/bin:${PATH}" \
HERMES_TUI_DIR="/opt/hermes/ui-tui" \
HERMES_WEB_DIST="/opt/hermes/hermes_cli/web_dist"
Expand Down
13 changes: 12 additions & 1 deletion test/hermes-dependency-review.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,10 @@ describe("Hermes 0.19.0 dependency review", () => {
expect(dockerfile).toContain(
"from tools.lazy_deps import ensure; ensure('memory.hindsight', prompt=False)",
);
expect(dockerfileBase).toContain(
"HERMES_LAZY_INSTALL_TARGET=/tmp/nemoclaw-hindsight-client-probe",
);
expect(dockerfileBase).toContain("import hindsight_client, importlib.metadata as m");
expect(dockerfile).toContain("state-dir-guard.py lock");
expect(dockerfile).toContain("--reuid=gateway --regid=gateway --init-groups");
expect(dockerfile).toContain("gateway can modify locked Hermes lazy packages");
Expand Down Expand Up @@ -243,7 +247,7 @@ describe("Hermes 0.19.0 dependency review", () => {
expect(review).toContain("exact uv `0.11.33`");
});

it("seeds root-owned pip before the sandbox user installs a lazy dependency", () => {
it("keeps the sandbox lazy-installer probe offline", () => {
const instructions = dockerfileInstructions(dockerfile);
const lazyInstallLayer = instructions.find(
(instruction) =>
Expand Down Expand Up @@ -292,6 +296,7 @@ describe("Hermes 0.19.0 dependency review", () => {
`if ln -sf /usr/bin/false /opt/hermes/.venv/bin/python 2>/dev/null; then exit 1; fi`,
`test ! -e /opt/hermes/.venv/lib/.nemoclaw-sandbox-write-probe`,
`test "$(stat -c '%U:%G %a' /sandbox/.hermes/lazy-packages)" = "sandbox:sandbox 750"`,
`test -z "$(find /sandbox/.hermes/lazy-packages -mindepth 1 -print -quit)"`,
"from tools.lazy_deps import ensure; ensure('memory.hindsight', prompt=False)",
];
let previousIndex = -1;
Expand All @@ -304,6 +309,12 @@ describe("Hermes 0.19.0 dependency review", () => {
previousIndex = contractIndex;
}
expect(layer).toContain("-perm /022");
expect(layer).toContain("--network=none");
expect(layer).toContain("PIP_NO_INDEX=1");
expect(layer).toContain("UV_FIND_LINKS=/tmp/nemoclaw-hindsight-probe");
expect(layer).toContain("UV_OFFLINE=1");
expect(layer).toContain("NEMOCLAW_BUILD_PROBE_FIXTURE");
expect(layer).not.toContain("https://");
expect(layer).not.toContain(`test -z "$(find -P /opt/hermes/.venv`);
expect(layer).not.toContain(`printf ''`);

Expand Down
4 changes: 3 additions & 1 deletion test/managed-image-pr-base-resolution.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,9 @@ exit 90
env: environment,
});
expect(result.status, result.stderr).toBe(0);
expect(fs.readFileSync(output, "utf8")).toBe("ref=nemoclaw-managed-pr/openclaw-base:test\n");
expect(fs.readFileSync(output, "utf8")).toBe(
"ref=nemoclaw-managed-pr/openclaw-base:test\nlocal=true\n",
);
expect(fs.readFileSync(dockerLog, "utf8")).toContain(
"buildx build --platform linux/amd64 --load --file Dockerfile.base --tag nemoclaw-managed-pr/openclaw-base:test .",
);
Expand Down
18 changes: 15 additions & 3 deletions test/managed-image-publication-workflow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -408,7 +408,8 @@
const matrix = prBuilder.strategy?.matrix?.include ?? [];
const steps = prBuilder.steps ?? [];
const permissionDrift = step(prBuilder, "Reproduce reviewed discovery permission drift");
const build = step(prBuilder, "Build PR managed image locally");
const localBaseBuild = step(prBuilder, "Build PR managed image from local base");
const registryBaseBuild = step(prBuilder, "Build PR managed image from registry base");
const contract = step(prBuilder, "Validate exact PR managed image contract");

expect(workflow.on?.pull_request?.paths).toEqual(
Expand Down Expand Up @@ -469,6 +470,7 @@
expect(prBuilder.permissions).toEqual({ contents: "read", packages: "write" });
expect(step(prBuilder, "Checkout").with?.["persist-credentials"]).toBe(false);
expect(step(prBuilder, "Checkout").with?.ref).toBe("${{ github.event.pull_request.head.sha }}");
expect(step(prBuilder, "Set up Docker Buildx").id).toBe("buildx");
const matrixByAgent = new Map(matrix.map((entry) => [entry.agent, entry]));
expect([...matrixByAgent.keys()].sort()).toEqual([
"hermes",
Expand All @@ -484,7 +486,8 @@
expect(steps.indexOf(permissionDrift)).toBeGreaterThan(
steps.indexOf(step(prBuilder, "Checkout")),
);
expect(steps.indexOf(permissionDrift)).toBeLessThan(steps.indexOf(build));
expect(steps.indexOf(permissionDrift)).toBeLessThan(steps.indexOf(localBaseBuild));
Comment thread
cv marked this conversation as resolved.
Resolved
expect(steps.indexOf(permissionDrift)).toBeLessThan(steps.indexOf(registryBaseBuild));

for (const action of steps.filter((candidate) => candidate.uses)) {
expect(action.uses, action.name).toMatch(fullShaAction);
Expand All @@ -503,7 +506,15 @@
expect(resolveBase).toContain('reference="${BASE_REPOSITORY}@${digest}"');
expect(resolveBase).toContain('actual="sha256:$(sha256sum "$exact_raw"');

expect(build.with).toMatchObject({
expect(localBaseBuild.if).toBe("steps.base.outputs.local == 'true'");
expect(registryBaseBuild.if).toBe("steps.base.outputs.local != 'true'");
const localBuild = required(localBaseBuild.run, "PR managed image local build is missing");
expect(localBuild).toContain("docker build");
expect(localBuild).toContain("--platform linux/amd64");
expect(localBuild).toContain('--build-arg "BASE_IMAGE=${BASE_IMAGE}"');
expect(localBuild).toContain('--tag "$IMAGE_REFERENCE"');
expect(localBuild).not.toContain("docker buildx build");
expect(registryBaseBuild.with).toMatchObject({
platforms: "linux/amd64",
load: true,
push: false,
Expand Down Expand Up @@ -633,7 +644,8 @@
const sameRepository = "github.event.pull_request.head.repo.full_name == github.repository";
expect(login.if).toBe(sameRepository);
expect(publish.if).toBe(sameRepository);
expect(publish.with).toMatchObject({

Check failure on line 647 in test/managed-image-publication-workflow.test.ts

View workflow job for this annotation

GitHub Actions / cli-test-shards (5)

[integration] test/managed-image-publication-workflow.test.ts > complete managed-image publication workflow > builds and exercises every shipped agent from an exact PR image before merge (#7744)

AssertionError: expected { context: '.', …(8) } to match object { …(5) } (5 matching properties omitted from actual) - Expected + Received { - "builder": "${{ steps.base.outputs.local == 'true' && 'default' || steps.buildx.outputs.name }}", "outputs": "type=image,name=${{ matrix.repository }},push-by-digest=true,name-canonical=true,push=true", "platforms": "linux/amd64", "provenance": false, "sbom": false, } ❯ test/managed-image-publication-workflow.test.ts:647:26
builder: "${{ steps.base.outputs.local == 'true' && 'default' || steps.buildx.outputs.name }}",
platforms: "linux/amd64",
outputs:
"type=image,name=${{ matrix.repository }},push-by-digest=true,name-canonical=true,push=true",
Expand Down
Loading