Skip to content

chore(registry): add sandboxed bin discovery - #11668

Closed
jdx wants to merge 1 commit into
mainfrom
agent/registry-bin-sandbox-discovery
Closed

jdx wants to merge 1 commit into
mainfrom
agent/registry-bin-sandbox-discovery

Conversation

@jdx

@jdx jdx commented Aug 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • add a maintainer runner that installs registry tools one per disposable Docker/Podman container and records the exact version, resolved backend, and executable names from mise bin-paths
  • support parallel full-registry runs, stable sharding, atomic resumable output, per-tool logs, failure categories, and executable collision reporting
  • document the bulk workflow and cover normalization, output parsing, sandbox arguments, resume/sharding, and collision aggregation with unit tests

Sandbox model

Each tool runs with a read-only root filesystem, tmpfs-backed mise state, dropped capabilities, no-new-privileges, and explicit CPU, memory, PID, and timeout limits. No host directory or host credential environment is passed into the tool container. Network access remains enabled because installation requires it.

This follows #11666, which added the registry bin metadata consumed by the eventual bulk update.

Validation

  • python3 -m unittest xtasks/test_discover_registry_bins.py
  • mise run lint-fix
  • end-to-end Docker discovery for actionlint (single bin) and age (four bins)
  • verified completed tools are skipped on resume

AI-assisted — Tool: Codex; model: OpenAI/GPT-5; version: unavailable.

Summary by CodeRabbit

  • New Features

    • Added a tool for discovering registry executables in isolated Docker or Podman environments.
    • Supports concurrent discovery, filtering, sharding, retries, resumable runs, resource limits, failure classification, and collision reporting.
    • Produces structured results and per-tool logs.
  • Documentation

    • Added instructions for running discovery for individual tools or the full registry.
  • Tests

    • Added coverage for parsing, isolation, resumption, result persistence, and executable-name collision detection.

@coderabbitai

coderabbitai Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Changes

Registry binary discovery

Layer / File(s) Summary
Sandboxed discovery CLI
xtasks/discover_registry_bins.py, docs/contributing.md
Adds containerized registry executable discovery with resource controls, structured results, logging, retries, sharding, resumption, and collision reporting. Contributor documentation describes the commands and options.
Discovery behavior validation
xtasks/test_discover_registry_bins.py
Tests normalization, parsing, version extraction, container isolation, shard resumption, result persistence, and collision detection.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

  • jdx/mise#11666: Adds and consumes registry bins metadata for shim auto-installation, which this CLI discovers.

Poem

A rabbit bounds through containers bright,
Finding each tool in sandboxed light.
Results are saved, collisions shown,
Failed shards wait to be retried alone.
“Hop!” says the rabbit, “the registry’s known!”

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the pull request's sandboxed registry binary discovery feature.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jdx
jdx force-pushed the agent/registry-bin-sandbox-discovery branch from 51aed12 to 42c2c32 Compare August 3, 2026 17:08
@jdx jdx changed the title agent/registry bin sandbox discovery chore(registry): add sandboxed bin discovery Aug 3, 2026
@github-actions

github-actions Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Instruction counts

benchmark trend instructions Δ wall (min) Δ
env █▁▇▆▇▇▅▄ 80,115,758 → 80,083,782 -0.04% 17.38 → 17.74ms +2.07%
hook-env █▁▇▇█▇▄▇ 82,244,496 → 82,330,078 +0.10% 18.56 → 18.31ms -1.38%
ls █▁▇▆▇█▄▄ 73,082,175 → 73,080,581 -0.00% 15.93 → 16.57ms +4.00%
registry ▂▃█▇█▃▁▁ 49,085,585 → 49,065,627 -0.04% 11.85 → 12.13ms +2.31%
startup ▂▇▇███▂▁ 15,748,518 → 15,740,840 -0.05% 9.40 → 9.18ms -2.31%

No instruction-count regression above 1%.

Only instruction counts gate. Wall clock is shown for context — on identical hardware it moves 4-20% run to run.

Measured by tak — instruction-counted CLI benchmarks, stored in this repository's git notes.

42c2c32d7c4e vs d32c02750540 · measured on the runner, not pushed to the history.

@greptile-apps

greptile-apps Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds a sandboxed maintainer utility for discovering executable names supplied by registry tools, together with contributor documentation and focused unit tests.

  • Builds or accepts a container image containing mise and runs each selected registry tool in an isolated, resource-limited container.
  • Supports resumable output, sharding, parallel discovery, failure logs, and executable-collision reporting.
  • Tests output parsing, executable normalization, sandbox arguments, resume behavior, and artifact generation.

Confidence Score: 4/5

The PR should not merge until the discovery image either uses a Linux binary matching the selected platform or rejects incompatible host builds.

The documented default copies a host-native debug executable unchanged into a Linux/amd64 image, so maintainers on macOS, Windows, or ARM64 Linux cannot run the new discovery workflow successfully.

Files Needing Attention: xtasks/discover_registry_bins.py and docs/contributing.md

Important Files Changed

Filename Overview
xtasks/discover_registry_bins.py Implements the discovery workflow, but its host-native binary and fixed Linux/amd64 defaults make the documented command fail on common non-Linux or non-amd64 development hosts.
xtasks/test_discover_registry_bins.py Adds focused unit coverage for parsing, normalization, sandbox flags, resumability, and collision artifacts.
docs/contributing.md Documents discovery and sharding workflows, but the default invocation does not explain the required Linux/amd64 binary compatibility.

Fix All in Claude Code

Reviews (1): Last reviewed commit: "chore(registry): add sandboxed bin disco..." | Re-trigger Greptile

Comment on lines +67 to +68
parser.add_argument("--mise-bin", type=Path, default=Path("target/debug/mise"))
parser.add_argument("--platform", default="linux/amd64")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Container binary platform mismatch

When a maintainer runs the documented command on macOS, Windows, or ARM64 Linux, the script copies the host-native target/debug/mise into a Linux/amd64 image, causing registry listing or discovery containers to fail with an executable-format error.

Fix in Claude Code

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
xtasks/discover_registry_bins.py (1)

213-224: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Name the registry-listing container so a timeout can clean it up.

discover_tool passes --name and force-removes the container after subprocess.TimeoutExpired. list_registry_tools does not. A timeout here kills only the engine client, so the container keeps running and --rm never triggers.

♻️ Proposed cleanup for the listing container
 def list_registry_tools(options: SandboxOptions, timeout: int) -> list[str]:
-    command = container_base_args(options) + [
-        "/usr/local/bin/mise",
-        "registry",
-        "--json",
-        "--hide-aliased",
-    ]
-    result = subprocess.run(command, capture_output=True, text=True, timeout=timeout, check=False)
+    container_name = f"mise-bin-discovery-registry-{os.getpid()}"
+    command = container_base_args(options)
+    command[-1:-1] = ["--name", container_name]
+    command.extend(["/usr/local/bin/mise", "registry", "--json", "--hide-aliased"])
+    try:
+        result = subprocess.run(
+            command, capture_output=True, text=True, timeout=timeout, check=False
+        )
+    except subprocess.TimeoutExpired:
+        subprocess.run(
+            [options.engine, "rm", "--force", container_name],
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            check=False,
+        )
+        raise
     if result.returncode != 0:
         raise RuntimeError(f"failed to list registry tools: {result.stderr.strip()}")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@xtasks/discover_registry_bins.py` around lines 213 - 224, Update
list_registry_tools to assign a unique container name and pass it through the
command arguments, matching discover_tool’s --name behavior. Ensure the timeout
cleanup path can use that name to force-remove the listing container after
subprocess.TimeoutExpired, while preserving the existing registry parsing and
error handling.
xtasks/test_discover_registry_bins.py (1)

45-60: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding tests for classify_failure and the per-tool command assembly.

discover_tool inserts --name and DISCOVERY_TOOL with command[-1:-1]. That slice must keep the image as the last argument before /bin/sh -c. A regression there is silent. classify_failure maps marker presence to an error category, and it is untested.

🧪 Proposed additional tests
    def test_classify_failure_categories(self):
        tool = discovery.MARKERS["tool"]
        versions = discovery.MARKERS["versions"]
        self.assertEqual(discovery.classify_failure(1, "", True), "timeout")
        self.assertEqual(discovery.classify_failure(None, "", False), "container")
        self.assertEqual(discovery.classify_failure(1, "noise", False), "install")
        self.assertEqual(discovery.classify_failure(1, tool, False), "tool_metadata")
        self.assertEqual(
            discovery.classify_failure(1, f"{tool}\n{versions}", False), "version_metadata"
        )

    def test_discover_tool_keeps_image_before_entrypoint(self):
        options = discovery.SandboxOptions(
            engine="docker",
            image="test-image",
            platform="linux/amd64",
            memory="1g",
            cpus="1",
            pids=64,
            state_size="2g",
        )
        command = discovery.container_base_args(options)
        command[-1:-1] = ["--name", "n", "--env", "DISCOVERY_TOOL=tool"]
        self.assertEqual(command[-1], "test-image")
        self.assertIn("DISCOVERY_TOOL=tool", command)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@xtasks/test_discover_registry_bins.py` around lines 45 - 60, Add focused
tests for classify_failure covering timeout, container, install, tool_metadata,
and version_metadata outcomes based on exit status and marker presence. Also
test discover_tool’s command assembly to verify --name and DISCOVERY_TOOL
insertion preserves the image as the final argument before the entrypoint.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/contributing.md`:
- Around line 815-833: Update the documentation before the first
discover_registry_bins.py example to include the required cargo build command
that creates target/debug/mise. Revise the shard guidance to state that each
shard writes its own output and that maintainers must combine those outputs
manually, without implying the script provides a merge command.

In `@xtasks/discover_registry_bins.py`:
- Around line 257-264: Update installed_version() to select the record marked as
active when multiple installed records are returned, while preserving the
existing validation for missing or invalid versions. If no active record exists,
fall back to the sole installed record; continue raising the existing error when
neither selection is unambiguous.

---

Nitpick comments:
In `@xtasks/discover_registry_bins.py`:
- Around line 213-224: Update list_registry_tools to assign a unique container
name and pass it through the command arguments, matching discover_tool’s --name
behavior. Ensure the timeout cleanup path can use that name to force-remove the
listing container after subprocess.TimeoutExpired, while preserving the existing
registry parsing and error handling.

In `@xtasks/test_discover_registry_bins.py`:
- Around line 45-60: Add focused tests for classify_failure covering timeout,
container, install, tool_metadata, and version_metadata outcomes based on exit
status and marker presence. Also test discover_tool’s command assembly to verify
--name and DISCOVERY_TOOL insertion preserves the image as the final argument
before the entrypoint.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 0bf7b44a-88a7-40c0-a696-7b0f05d1c633

📥 Commits

Reviewing files that changed from the base of the PR and between d32c027 and 42c2c32.

📒 Files selected for processing (3)
  • docs/contributing.md
  • xtasks/discover_registry_bins.py
  • xtasks/test_discover_registry_bins.py

Comment thread docs/contributing.md
Comment on lines +815 to +833
```sh
python3 xtasks/discover_registry_bins.py actionlint age
```

The script builds `target/debug/mise` into a local Docker or Podman image and
installs each tool in its own disposable, resource-limited container. It does
not mount host directories or pass host credentials into those containers.
Successful results include the resolved backend, exact version, and executable
names reported by `mise bin-paths`; failures retain a per-tool log and are
retried when the same command is resumed.

To process the complete registry, use `--all`. Large runs can be split into
stable shards and merged afterward by running each shard with its own output:

```sh
python3 xtasks/discover_registry_bins.py --all --jobs 8 \
--shard-count 4 --shard-index 0 \
--output registry-bin-discovery-0.json
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document the required build step, and clarify that shard merging is manual.

The script defaults --mise-bin to target/debug/mise and raises "mise binary not found" when that file is absent. Add the build command before the first example. Also state that shard outputs are combined by the maintainer, because the script has no merge command.

📝 Proposed documentation update
 Maintainers can collect the executable names installed by registry tools with
 the sandboxed discovery script:
 
 ```sh
+cargo build
 python3 xtasks/discover_registry_bins.py actionlint age

@@
-To process the complete registry, use --all. Large runs can be split into
-stable shards and merged afterward by running each shard with its own output:
+To process the complete registry, use --all. Large runs can be split into
+stable shards by running each shard with its own output file. Combine the
+shard outputs manually after all shards finish:

</details>

<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

```suggestion

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/contributing.md` around lines 815 - 833, Update the documentation before
the first discover_registry_bins.py example to include the required cargo build
command that creates target/debug/mise. Revise the shard guidance to state that
each shard writes its own output and that maintainers must combine those outputs
manually, without implying the script provides a merge command.

Comment on lines +257 to +264
def installed_version(tool: str, versions: Any) -> str:
records = versions.get(tool) if isinstance(versions, dict) else versions
if not isinstance(records, list) or len(records) != 1:
raise ValueError(f"expected one installed version for {tool}")
version = records[0].get("version")
if not isinstance(version, str) or not version:
raise ValueError(f"missing concrete installed version for {tool}")
return version

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify the JSON schemas consumed by the discovery script.
set -eu

# `mise bin-paths --json` entries: does each entry expose `name`?
rg -nP -C5 'bin[_-]paths' --glob '!**/*.lock' src | head -100

# `mise tool --json`: does the payload expose `backend`?
ast-grep run --lang rust --pattern 'struct $NAME { $$$ }' src/cli/tool.rs || true
rg -nP -C3 '\bbackend\b' src/cli/tool.rs | head -40

# `mise ls --json --installed`: record fields (version, active, installed) and dict-vs-array shape.
rg -nP -C4 'active|installed|requested_version' src/cli/ls.rs | head -60

# `mise registry --json --hide-aliased`: does each entry expose `short`?
rg -nP -C4 'short|hide_aliased' src/cli/registry.rs | head -40

Repository: jdx/mise

Length of output: 12195


🏁 Script executed:

#!/bin/bash
set -eu

echo "== locate files =="
git ls-files | rg 'xtasks/discover_registry_bins\.py|src/cli/ls|src/cli/tool|src/cli/bin_paths|src/cli/registry' || true

echo "== discover_registry_bins relevant lines =="
sed -n '1,420p' xtasks/discover_registry_bins.py

echo "== ls json structs and renderers =="
rg -n -C6 'struct .*Json|impl .*Json|serde_json|active|installed|version|current_source|requested|ToolVersion' src/cli/ls.rs src/cli/ls_json.rs src/cli/tool.rs src/cli/bin_paths.rs src/cli/registry.rs 2>/dev/null || true

echo "== json model definitions across source =="
rg -n -C5 'struct .*Json|struct .*Output|Serialize|Deserialize|json' src | rg -n -C5 'active|requested|installed|ToolVersion|BinPath|RegistryToolOutput|ToolInfo|output_json' | head -200

Repository: jdx/mise

Length of output: 50364


Choose the active installed version instead of rejecting multiple installed records.

mise use --pin "${DISCOVERY_TOOL}@latest" can leave both the version created by test-tool and the pinned version installed. installed_version() currently raises when len(records) != 1, and discover_tool() wraps that in error_category: "output". Select the single active record when present, and fall back to the only installed record.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@xtasks/discover_registry_bins.py` around lines 257 - 264, Update
installed_version() to select the record marked as active when multiple
installed records are returned, while preserving the existing validation for
missing or invalid versions. If no active record exists, fall back to the sole
installed record; continue raising the existing error when neither selection is
unambiguous.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant