fix(secrets): isolate plugin bootstrap before hydration

[alexgunsberg]

Problem

An external secret-source plugin can be configured in secrets.sources, but dotenv loading runs before ordinary plugin discovery. The first process pass therefore treats the source as unknown and skips its credentials.

Fix

Add a restricted bootstrap phase before secret hydration:

• scan only enabled native plugins under the explicit Hermes home
• import only manifests that declare matching provides_secret_sources, with syntax-only compatibility for legacy direct-registration plugins
• use a restricted context that exposes only register_secret_source
• leave normal process-wide plugin discovery untouched and uncached
• support explicit secrets.sources and enabled secrets.<source> sections
• redact bootstrap failures and remain fail-open
• apply the same semantics to profile-scoped hydration

This is a targeted alternative to #81960. That PR re-pulls secrets after full general plugin discovery; this PR hydrates declared source plugins before unrelated plugins import, so plugins that need those credentials during register() do not cache a first-process failure.

Regression proof

On unmodified upstream, the added tests fail because the source is unknown on first load. On this branch they verify:

• unrelated enabled plugins are not imported before hydration
• an explicit non-process Hermes home is honored
• omitted secrets.sources still works through enabled source config
• repeated load is idempotent
• bootstrap and fetch exceptions are fail-open and redact sentinel values
• benign config reads do not import or fetch plugins

Verification

• scripts/run_tests.sh related plugin/env/config suite: 148 passed
• package bootstrap → full discovery regression: passes here; fails on the prior head with the slash command missing
• live unannotated Proton Pass plugin probe: 5 secrets applied, no unknown-source warning
• Ruff: passed
• git diff --check: passed

[Bartok9]

Collab review (Proton Pass / first-process hazard)

I pulled pr-82607, read the full diff against #81960, and ran:

python3 -m pytest tests/test_external_secret_source_plugin_startup.py -q
# 6 passed in ~1.6s

Verdict

Prefer this PR’s shape over #81960’s post-discovery re-pull for the hazard you called out: full PluginManager import before secrets are hydrated can cache a register() failure when an unrelated plugin reads a plugin-sourced env var. Targeted bootstrap (manifest filter / AST legacy probe → restricted context → apply_all → ordinary discovery later, _discovered untouched) is the right seam.

Live probe I added on top of your suite (not in-tree yet): secret-source plugin hydrates VAULT_TOKEN, then full discover_plugins() loads a consumer plugin whose register() requires that env — passes on this branch. That is the real agent-to-agent analogue of the Proton Pass failure mode.

What already looks solid

• Explicit hermes_home= scoping (only scans that home’s plugins/ + that home’s plugins.enabled)
• Unrelated enabled plugins not imported during bootstrap (your marker test)
• Omitted secrets.sources still discovers via enabled secrets.<name> sections
• Fail-open + redacted bootstrap/fetch errors (no exception text interpolation)
• Benign config read does not import plugins
• Fetch-exception redaction in registry.py

Gaps I’d close before merge (happy to send a tiny follow-up PR if you want)

1. Silent re-register on full discovery (noise, not correctness)
Bootstrap already registers the source. Full discovery re-imports the module (same sys.modules name) and calls register() again with a full PluginContext. register_source then logs:

Secret source '…' already registered; ignoring duplicate

On a clean Proton Pass boot that warning is a false alarm. Minimal fix in PluginContext.register_secret_source (or register_source when the existing entry is the same name and replace=False): treat “already registered” as success / debug, not warning.

2. Missing regression: consumer plugin after hydration
Your suite proves the source plugin path and “unrelated not imported early,” but not “later full discovery register() sees credentials.” Suggest adding something like:

def test_full_discovery_consumer_plugin_sees_hydrated_secret(tmp_path):
    _write_secret_source_plugin(
        tmp_path,
        fetch_body=(
            "        return FetchResult(secrets={'STARTUP_TEST_API_KEY': 'ready'})"
        ),
    )
    consumer = tmp_path / "plugins" / "consumer-plugin"
    consumer.mkdir()
    (consumer / "plugin.yaml").write_text(
        "name: consumer-plugin\nkind: standalone\nversion: 1.0.0\n",
        encoding="utf-8",
    )
    (consumer / "__init__.py").write_text(
        "import os\n"
        "from pathlib import Path\n"
        "def register(ctx):\n"
        "    assert os.environ.get('STARTUP_TEST_API_KEY') == 'ready'\n"
        "    Path(os.environ['HERMES_HOME'], 'consumer-ok').write_text('ok')\n",
        encoding="utf-8",
    )
    # enable both plugins + secrets section …
    # load_hermes_dotenv → discover_plugins → assert consumer-ok

That locks the ordering contract against a future “optimize” that moves bootstrap after general import again.

3. Optional: only treat known source-shaped keys as configured
_discover_configured_secret_source_plugins currently treats every secrets.* dict value as a source name. Today that’s probably fine; if secrets: ever gains non-source nested maps, bootstrap would scan for unknown names. Low urgency — flag only.

4. vs #81960
Keep origin tracking / docs polish from #81960 if useful, but do not land post-discovery re-pull instead of this bootstrap for the first-process consumer hazard. Re-pull can remain a belt for child processes; it is not a substitute for hydrate-before-unrelated-import.

Docs nit

PluginContext.register_secret_source NOTE now says discovery runs when secrets.sources contains an unregistered name — accurate for the list path; also mention enabled secrets.<source> sections (your omitted-list test).

---

I’m not opening a competing PR. If you want the consumer test + quiet re-register as a patch on your branch, say the word and I’ll push a PR against alexgunsberg:fix/plugin-secret-source-startup or paste a patch. @alexgunsberg

[Bartok9]

Collab review (Proton Pass / first-process hazard)

I pulled pr-82607, read the full diff against #81960, and ran:

python3 -m pytest tests/test_external_secret_source_plugin_startup.py -q
# 6 passed in ~1.6s

Verdict

Prefer this PR’s shape over #81960’s post-discovery re-pull for the hazard you called out: full PluginManager import before secrets are hydrated can cache a register() failure when an unrelated plugin reads a plugin-sourced env var. Targeted bootstrap (manifest filter / AST legacy probe → restricted context → apply_all → ordinary discovery later, _discovered untouched) is the right seam.

Live probe I added on top of your suite (not in-tree yet): secret-source plugin hydrates VAULT_TOKEN, then full discover_plugins() loads a consumer plugin whose register() requires that env — passes on this branch. That is the real agent-to-agent analogue of the Proton Pass failure mode.

What already looks solid

• Explicit hermes_home= scoping (only scans that home’s plugins/ + that home’s plugins.enabled)
• Unrelated enabled plugins not imported during bootstrap (your marker test)
• Omitted secrets.sources still discovers via enabled secrets.<name> sections
• Fail-open + redacted bootstrap/fetch errors (no exception text interpolation)
• Benign config read does not import plugins
• Fetch-exception redaction in registry.py

Gaps I’d close before merge (happy to send a tiny follow-up PR if you want)

1. Silent re-register on full discovery (noise, not correctness)
Bootstrap already registers the source. Full discovery re-imports the module (same sys.modules name) and calls register() again with a full PluginContext. register_source then logs:

Secret source '…' already registered; ignoring duplicate

On a clean Proton Pass boot that warning is a false alarm. Minimal fix in PluginContext.register_secret_source (or register_source when the existing entry is the same name and replace=False): treat “already registered” as success / debug, not warning.

2. Missing regression: consumer plugin after hydration
Your suite proves the source plugin path and “unrelated not imported early,” but not “later full discovery register() sees credentials.” Suggest adding something like:

def test_full_discovery_consumer_plugin_sees_hydrated_secret(tmp_path):
    _write_secret_source_plugin(
        tmp_path,
        fetch_body=(
            "        return FetchResult(secrets={'STARTUP_TEST_API_KEY': 'ready'})"
        ),
    )
    consumer = tmp_path / "plugins" / "consumer-plugin"
    consumer.mkdir()
    (consumer / "plugin.yaml").write_text(
        "name: consumer-plugin\nkind: standalone\nversion: 1.0.0\n",
        encoding="utf-8",
    )
    (consumer / "__init__.py").write_text(
        "import os\n"
        "from pathlib import Path\n"
        "def register(ctx):\n"
        "    assert os.environ.get('STARTUP_TEST_API_KEY') == 'ready'\n"
        "    Path(os.environ['HERMES_HOME'], 'consumer-ok').write_text('ok')\n",
        encoding="utf-8",
    )
    # enable both plugins + secrets section …
    # load_hermes_dotenv → discover_plugins → assert consumer-ok

That locks the ordering contract against a future “optimize” that moves bootstrap after general import again.

3. Optional: only treat known source-shaped keys as configured
_discover_configured_secret_source_plugins currently treats every secrets.* dict value as a source name. Today that’s probably fine; if secrets: ever gains non-source nested maps, bootstrap would scan for unknown names. Low urgency — flag only.

4. vs #81960
Keep origin tracking / docs polish from #81960 if useful, but do not land post-discovery re-pull instead of this bootstrap for the first-process consumer hazard. Re-pull can remain a belt for child processes; it is not a substitute for hydrate-before-unrelated-import.

Docs nit

PluginContext.register_secret_source NOTE now says discovery runs when secrets.sources contains an unregistered name — accurate for the list path; also mention enabled secrets.<source> sections (your omitted-list test).

---

I’m not opening a competing PR. If you want the consumer test + quiet re-register as a patch on your branch, say the word and I’ll push a PR against alexgunsberg:fix/plugin-secret-source-startup or paste a patch. @alexgunsberg

[Bartok9]

Thanks for the follow-up push (9365aa7 — isolate secret bootstrap imports). Re-checked the delta against the earlier review:

Closed

1. Quiet re-register — PluginContext.register_secret_source now short-circuits when the source was already registered in bootstrap (_secret_source_bootstrap_paths + debug “retained bootstrap…”). test_package_plugin_registers_non_secret_capabilities_after_bootstrap asserts "already registered" not in result.stderr. Good.
2. Canonical sys.modules isolation — bootstrap loads under a temp module name, restores pre-bootstrap canonical cache in finally, so full discovery can still attach non-secret capabilities. That’s the right fix for package plugins; stronger than what I had sketched.

Still optional (not blocking from my side)

• A direct “consumer plugin register() sees hydrated env” regression (ordering contract). The package non-secret capability test covers a related path; the env-visible consumer case would still lock the Proton Pass / agent-to-agent hazard explicitly.
• Docs NOTE on register_secret_source still only mentions secrets.sources; worth a half-line that enabled secrets.<source> sections also trigger bootstrap (matches test_omitted_sources_list_uses_enabled_source_section).

No competing PR from me. LGTM on the bootstrap isolation direction — happy to re-run the suite if you want another pair of eyes after any further polish.

[Bartok9]

@alexgunsberg — quick human note on top of the technical review:

This is really strong work. The targeted bootstrap-before-hydration design is the right fix for the Proton Pass / first-process hazard, and the follow-up isolation commit tightened it further. We want to see this land.

We’re staying on the thread (not dropping until it’s merged). If anything would help — rebase onto main, a small patch on your branch, another suite run, CI triage, wording nits — just say the word and we’ll jump on it. No pressure and no competing PR from us; happy to support your PR until it’s published.

Thanks for doing this carefully.

[Bartok9]

Weekly collab check-in

Still watching this until merge — no pressure, no competing PR.

Status snapshot (2026-08-10):

• Branch tip still 9365aa7 (bootstrap isolation) — looks good from our last pass
• Compare vs main: ~100 commits behind / 3 ahead (diverged) — merge box may stay BLOCKED until rebase
• Related fix(plugins): bootstrap plugin secret sources after discovery #81960 still open; we continue to prefer this bootstrap-before-hydration shape for the first-process / Proton Pass hazard

Happy to help if useful:

1. Rebase assist onto current main (diff review / conflict triage notes — we won’t force-push your fork without an invite)
2. Optional consumer-after-hydrate regression test patch (paste or PR against your branch if you want it)
3. Suite re-run / CI triage after rebase

Just ping @Bartok9 anytime. Thanks again for the careful work here.

[Bartok9]

Weekly collab check-in

Still watching until merge — no pressure, no competing PR.

Status snapshot (2026-08-17):

• Branch tip still 9365aa7 (bootstrap isolation) — unchanged since 2026-08-09
• GitHub merge state: dirty / not mergeable (conflicts with current main) — rebase needed before the merge box clears
• Related fix(plugins): bootstrap plugin secret sources after discovery #81960 (post-discovery re-pull) merged 2026-08-12. We still prefer this PR’s bootstrap-before-hydration shape for the first-process / Proton Pass consumer hazard; re-pull is a useful belt for later processes, not a full substitute for hydrate-before-unrelated-import
• No new review decision / CI rollup signal on this PR from our side since last check

Happy to help if useful:

1. Rebase assist onto current main (conflict triage notes / review of the rebased diff — we won’t force-push your fork without an invite; maintainer_can_modify is on if you’d rather have maintainers land the rebase)
2. Optional consumer-after-hydrate regression test patch (paste or PR against your branch)
3. Suite re-run / CI triage after rebase

Just ping @Bartok9 anytime. Thanks again for the careful work here.