Skip to content

fix(web): a benched lazy-install must not disable an importable SDK - #80022

Open
rodrigogs wants to merge 1 commit into
NousResearch:mainfrom
rodrigogs:pr/parallel-sdk-lazy-install
Open

fix(web): a benched lazy-install must not disable an importable SDK#80022
rodrigogs wants to merge 1 commit into
NousResearch:mainfrom
rodrigogs:pr/parallel-sdk-lazy-install

Conversation

@rodrigogs

Copy link
Copy Markdown

Problem

_ensure_parallel_sdk_installed in plugins/web/parallel/provider.py documented one contract and implemented the opposite. Its docstring promised to swallow a benign availability error from the lazy-deps helper and let the subsequent from parallel import ... be the real gate on whether the SDK is usable. The handlers were:

except ImportError:
    pass                          # never fired
except Exception as exc:
    raise ImportError(str(exc))   # fired for exactly the intended case

The root cause is a wrong exception class. tools.lazy_deps.ensure signals an unusable feature by raising FeatureUnavailable, which subclasses RuntimeError — not ImportError. So:

  • the narrow except ImportError arm never matched the error it was written for;
  • the broad except Exception arm caught FeatureUnavailable and converted "cannot install right now" into a hard ImportError;
  • the import that was supposed to make the decision was never reached.

The practical consequence: on any host with security.allow_lazy_installs=false, the Parallel provider was dead whether or not parallel-web was actually installed. Declining to install a package was treated as the package being absent. A correctly provisioned host that had parallel-web present but lazy installs disabled by policy got the same failure as a host missing the dependency entirely.

Fix

Try the install; if it is impossible, then check whether it was needed at all. Only a genuinely missing package raises, and the raised error keeps the actionable install hint:

except FeatureUnavailable as exc:
    if not _parallel_sdk_importable():
        raise ImportError(str(exc)) from exc
    # Already importable; nothing needed installing.
except Exception as exc:
    raise ImportError(str(exc)) from exc

Unrelated faults (an OSError from the installer, say) are real problems and still surface to the caller as ImportError rather than being silently swallowed. An ImportError from importing the lazy-deps helper itself returns early and lets the import decide, which is what the original docstring described.

Why sys.modules is checked before find_spec

if "parallel" in sys.modules:
    return True
try:
    return importlib.util.find_spec("parallel") is not None
except (ImportError, ValueError):
    return False

Two reasons, and the order matters:

  1. An already-imported module is importable by definition; consulting sys.modules first is both correct and cheaper than a filesystem scan.
  2. importlib.util.find_spec raises ValueError on a module present in sys.modules whose __spec__ is None. That is not a hypothetical: it is exactly how the existing test suite injects a stub SDK — a bare types.ModuleType("parallel") has no __spec__. Calling find_spec first would raise ValueError on precisely the inputs the check needs to answer True for. ValueError is caught as well, so a malformed entry degrades to "not importable" instead of escaping as an unexpected exception type.

This also explains the pre-existing failure in tests/tools/test_web_tools_config.py::TestParallelClientConfig: the suite installs a stub parallel module in sys.modules, but production code raised before ever looking at it.

Test evidence

tests/tools/test_parallel_sdk_ensure.py (new) covers the three distinct paths, since the bug was that one exception class was being conflated with another:

  • test_importable_sdk_survives_disabled_lazy_installsFeatureUnavailable raised while the package imports fine: not an error.
  • test_missing_sdk_still_reports_the_install_hint — genuinely absent package still raises ImportError carrying parallel-web.
  • test_unrelated_failure_is_still_surfaced — an OSError from the installer is not swallowed.
$ ./venv/bin/pytest tests/tools/test_parallel_sdk_ensure.py tests/tools/test_web_tools_config.py -q
41 passed in 2.36s

$ ./venv/bin/ruff check plugins/web/parallel/provider.py
All checks passed!

Reverting only provider.py to its current main content, with the new tests in place, fails as expected — confirming the production change is what the tests are pinning rather than the tests passing incidentally:

FAILED tests/tools/test_parallel_sdk_ensure.py::test_importable_sdk_survives_disabled_lazy_installs
FAILED tests/tools/test_web_tools_config.py::TestParallelClientConfig::test_creates_client_with_key
FAILED tests/tools/test_web_tools_config.py::TestParallelClientConfig::test_singleton_returns_same_instance
3 failed, 38 passed

Note that the two TestParallelClientConfig failures are pre-existing on main and are fixed by this change; they were the symptom that led to the bug.

Context

Found alongside #79839 and #79840 while investigating a single incident; the three are independent fixes in different subsystems and can be reviewed and merged in any order. This branch is based on current main (ff3793fdf) and touches two files: plugins/web/parallel/provider.py and the new test module.

@alt-glitch alt-glitch added type/bug Something isn't working comp/plugins Plugin system and bundled plugins tool/web Web search and extraction area/config Config system, migrations, profiles P3 Low — cosmetic, nice to have sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 6, 2026
@rodrigogs
rodrigogs force-pushed the pr/parallel-sdk-lazy-install branch 4 times, most recently from 3a8c89e to 37f6173 Compare August 15, 2026 20:23
@rodrigogs
rodrigogs force-pushed the pr/parallel-sdk-lazy-install branch 3 times, most recently from 26124b0 to 31521a2 Compare August 17, 2026 21:50
@rodrigogs

Copy link
Copy Markdown
Author

Hi — heads-up that CI has never actually run on this PR: every workflow run (CI + Docker Build) since it was opened ends in action_required with zero jobs, and the run page says 'This workflow is awaiting approval from a maintainer'. The branch is mergeable with no conflicts and doesn't touch any workflow files. Could a maintainer approve the workflow runs (or enable CI for outside collaborators)? Happy to address anything the checks find.

@rodrigogs
rodrigogs force-pushed the pr/parallel-sdk-lazy-install branch 2 times, most recently from 81f62b9 to a47b18a Compare August 23, 2026 23:03
@rodrigogs

Copy link
Copy Markdown
Author

Rebased onto current upstream/main (0a171fffe); new head a47b18a6601f.

Conflict-free, and nothing upstream disturbed this PR — including the defect itself:
plugins/web/parallel/provider.py::_ensure_parallel_sdk_installed on current main still contains the exact
except ImportError: pass / except Exception: raise ImportError(str(exc)) pair this PR fixes, and
tools/lazy_deps.FeatureUnavailable is still a RuntimeError subclass with the same signature.

Verification on the new head: tests/tools/test_parallel_sdk_ensure.py3 passed; every other module
that imports the Parallel provider or lazy_deps (test_web_tools_config.py, test_web_providers.py,
test_web_keyless_fallback.py) → 102 passed.

One deliberate non-change, kept so the diff stays identical to what was reviewed: provider.py splits the
stdlib imports into two groups (import importlib.util / import sys, blank line, import logging /
import os). The repo's ruff config selects only PLW1514, so no isort rule enforces grouping and lint is
clean — but if you would rather the blank line went away, it is a one-line follow-up.

This repository does not run CI on pull requests from forks, so the checks tab stays empty and protect-main's required All required checks pass context never reports — which is why this PR shows mergeable: true with mergeStateStatus: BLOCKED. Approving the workflow run (or landing it on the strength of the local evidence) is all that is left from my side.

`_ensure_parallel_sdk_installed` promised, in its own docstring, to swallow a
benign availability error and let `from parallel import ...` be the real gate.
The handlers did the opposite:

  except ImportError:  pass                 # never fired
  except Exception:    raise ImportError()  # fired for the intended case

`lazy_deps.ensure` signals an unusable feature with `FeatureUnavailable`, which
subclasses RuntimeError, not ImportError. So the narrow arm never matched, the
broad arm turned "cannot install" into a hard failure, and the import that was
supposed to decide was never reached. On any host with
`security.allow_lazy_installs=false` the Parallel provider was unusable whether
or not `parallel-web` was actually present.

Catch FeatureUnavailable explicitly and only fail when the package is genuinely
absent, keeping the actionable install hint. `_parallel_sdk_importable` checks
`sys.modules` before `find_spec`, because an already-imported module is
importable by definition and `find_spec` raises ValueError on entries with no
`__spec__` — which is how the existing tests inject a stub SDK. Unrelated faults
still surface as ImportError for the caller.

This is why tests/tools/test_web_tools_config.py::TestParallelClientConfig failed
on such a host: the suite installs a stub `parallel` module in `sys.modules`, but
production raised before ever looking at it.
@rodrigogs
rodrigogs force-pushed the pr/parallel-sdk-lazy-install branch from a47b18a to 7671a6a Compare August 24, 2026 15:06
@rodrigogs

Copy link
Copy Markdown
Author

Force-pushed a metadata-only fix so the contributor attribution check can pass.

.github/workflows/contributor-check.yml exits 1 on any commit-author email that has no file under
contributors/emails/, and ci.yaml calls it, so it feeds the required All required checks pass context.
This branch carried a placeholder author identity from a misconfigured local git config, which the check
would have rejected the moment a maintainer approved the workflow runs.

Every commit's author is now Rodrigo Gomes <2362425+rodrigogs@users.noreply.github.com> — the account's
GitHub noreply address, which the check auto-resolves via its +…@users.noreply.github.com rule, so no
mapping file is needed. Author dates are preserved, and the tree is byte-identical: git diff <old-head> <new-head> is empty, so nothing about the change under review moved and the verification I posted earlier
still stands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/web Web search and extraction type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants