Skip to content

feat: ship standalone in-process providers and YOLOX local mode - #3

Merged
Timverhoogt merged 8 commits into
mainfrom
feat/in-process-providers
Jul 23, 2026
Merged

Timverhoogt merged 8 commits into
mainfrom
feat/in-process-providers

Conversation

@Timverhoogt

@Timverhoogt Timverhoogt commented Jul 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • removes the separate provider broker and caregiver-auth deployment path
  • adds fixed-policy in-process OpenAI and private no-key local modes
  • replaces unreliable SSDLite detection with pinned Apache-2.0 YOLOX-Nano
  • handles multi-instance rooms with cross-view class stability and exact-box revalidation
  • separates wake-race and physical Stop cleanup bounds
  • prepares the truthful 0.2.0 standalone release

Verification

  • 114 tests passed
  • Ruff passed
  • wheel + sdist package/secret/model/license boundary checks passed
  • official Pollen reachy-mini-app-assistant check passed on the CM4, including clean install, entry-point registration and uninstall
  • exact wheel bd30ae8a4d4cb5fa1327d66bf323421d277fb16c045a6b99a503eead0c94aa7a physically accepted on Reachy Mini Wireless
  • local clue → typed guess → reveal → Stop completed; camera off, folded, motors disabled

Hardware scope

  • Reachy Mini Wireless: physically accepted
  • redesigned Lite-host profile: implemented and automated, explicitly documented as awaiting same-version physical acceptance

Summary by CodeRabbit

  • New Features

    • Added standalone version 0.2.0 support for Lite and Wireless deployment profiles.
    • Added direct OpenAI vision mode and offline local detection and speech mode.
    • Added guided local model and voice asset installation with verification.
    • Added clearer provider, compute, and local setup controls in the app.
  • Safety

    • Camera access now requires explicit opt-in for each game.
    • Improved fail-closed behavior, cancellation, moderation, and Stop handling.
  • Documentation

    • Updated release, architecture, setup, security, privacy, and language guidance.

@coderabbitai

coderabbitai Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@Timverhoogt, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 36 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fae99b23-e8eb-4890-af43-7eb2a2f5b9e6

📥 Commits

Reviewing files that changed from the base of the PR and between d76aab0 and a136d8c.

📒 Files selected for processing (12)
  • README.md
  • SECURITY.md
  • docs/ARCHITECTURE.md
  • reachy_mini_i_spy/config.py
  • reachy_mini_i_spy/local_assets.py
  • reachy_mini_i_spy/local_provider.py
  • reachy_mini_i_spy/main.py
  • reachy_mini_i_spy/static/index.html
  • tests/test_config.py
  • tests/test_local_assets.py
  • tests/test_local_provider.py
  • tests/test_public_positioning.py
📝 Walkthrough

Walkthrough

This PR removes the standalone provider broker (FastAPI app, systemd service, deploy docs, caregiver auth guard, readiness script, and related tests) and replaces it with an in-process fixed-policy OpenAI provider plus a new offline local ONNX/YOLOX detection and Sherpa-ONNX TTS mode. Configuration, API routes, frontend UI, runtime cancellation/stop handling, packaging, and documentation are all updated to reflect the 0.2.0 standalone release with provider selection (openai/local) instead of broker credentials.

Changes

Standalone provider redesign

Layer / File(s) Summary
Broker removal
hermes_broker/*, deploy/*, reachy_mini_i_spy/auth.py, scripts/verify_broker_readiness.py, tests/test_auth.py, tests/test_broker.py, MANIFEST.in
Deletes broker app/service/docs, caregiver auth guard, readiness script, and broker tests; removes broker prune entries.
Config model
reachy_mini_i_spy/config.py, tests/test_config.py
AppConfig replaces provider_url/broker_token/device_id with provider/api_key, updates validation, env defaults, and legacy-file handling.
In-process provider client
reachy_mini_i_spy/provider.py, tests/test_provider.py
Direct OpenAI calls with strict validation and local fallback replace broker request/session mechanics.
Local model/voice assets
reachy_mini_i_spy/local_assets.py, reachy_mini_i_spy/models/*, tests/test_local_assets.py
Adds pinned asset registry, safe download/extraction, verification, and atomic install for detector/voice models.
LocalProvider detection/TTS
reachy_mini_i_spy/local_provider.py, reachy_mini_i_spy/game.py, tests/test_local_provider.py
Implements ONNX detection, target selection/moderation, guess judging, and offline TTS; adds configurable confidence threshold and opt-in wording.
Runtime cancellation/stop
reachy_mini_i_spy/runtime.py, tests/test_runtime.py
Removes remote-cancel scheduling, changes default stop reason to "user", adds timing constants and diagnostic mapping.
API routes/settings
reachy_mini_i_spy/main.py
Removes caregiver/CSRF gating, adds deployment profile detection, simplifies settings schema.
Frontend UI
reachy_mini_i_spy/static/index.html, reachy_mini_i_spy/static/main.js, index.html
Replaces broker settings panel with vision settings (provider/API key/local install), removes CSRF flow.
Packaging/artifact validation
pyproject.toml, scripts/check_artifacts.py
Bumps version to 0.2.0, adds ONNX/TTS deps, enforces required model assets and checksums.
Documentation
CHANGELOG.md, README.md, SECURITY.md, docs/*, tests/test_public_positioning.py, tests/test_gate_adversarial.py
Updates docs/tests to describe 0.2.0 standalone provider redesign and camera opt-in wording.

Estimated code review effort: 4 (Complex) | ~75 minutes

Sequence Diagram(s)

sequenceDiagram
  participant UI
  participant Main as main.py
  participant Provider as ProviderClient
  participant OpenAI as OpenAI API
  participant Local as LocalProvider

  UI->>Main: POST /api/game/start
  Main->>Main: check load_config().configured
  Main->>Provider: select_target(frames)
  alt provider == openai
    Provider->>OpenAI: POST chat/completions
    OpenAI-->>Provider: JSON target
  else provider == local
    Provider->>Local: select_target(frames)
    Local-->>Provider: Target
  end
  Provider-->>Main: validated Target
  Main-->>UI: game state
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 7.03% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: standalone in-process providers plus YOLOX local mode.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/in-process-providers

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🧹 Nitpick comments (3)
reachy_mini_i_spy/static/index.html (1)

69-69: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Optional: stale wording in helper text.

"provider URL" no longer applies (the broker/URL was removed), and the hardcoded "$0.10/month in credits" figure will likely drift out of date. Consider trimming to just the fixed-policy statement.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reachy_mini_i_spy/static/index.html` at line 69, Update the helper text
paragraph near the fixed policy statement to remove the stale provider URL
reference and the time-sensitive Hugging Face credit amount, retaining only the
accurate statement about fixed models, prompts, and safety bounds.
reachy_mini_i_spy/local_assets.py (1)

98-108: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor: hardcoded download_bytes can drift from the registry.

local_assets_status() reports a fixed 34_483_424 regardless of the actual VOICE_ASSETS contents. If a voice asset is added/changed, this estimate silently goes stale.

♻️ Suggested fix
-        "download_bytes": 34_483_424,
+        "download_bytes": sum(0 if _voice_ready(a) else _ARCHIVE_SIZE_ESTIMATE for a in VOICE_ASSETS.values()),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reachy_mini_i_spy/local_assets.py` around lines 98 - 108, Update
local_assets_status() so download_bytes is calculated from the current
VOICE_ASSETS registry rather than a hardcoded constant, summing each registered
voice asset’s expected download size using the existing asset metadata.
reachy_mini_i_spy/local_provider.py (1)

271-315: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Local target's minimum_confidence gate is effectively a no-op.

_detect() (line 191) already discards any detection with score < self.SCORE_THRESHOLD, so every candidate passed into select_target() already satisfies score >= SCORE_THRESHOLD. Passing minimum_confidence=self.SCORE_THRESHOLD (line 313) to validate_target therefore can never reject on confidence — the check is vacuously true. This also means the local provider's effective confidence floor (0.30) is much lower than the cloud provider's fixed 0.78 floor; worth confirming that's the deliberate design rather than a leftover placeholder.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@reachy_mini_i_spy/local_provider.py` around lines 271 - 315, Review the
confidence-threshold flow between _detect and select_target: validate_target’s
minimum_confidence check is redundant because _detect already filters detections
below SCORE_THRESHOLD. Remove the vacuous validation argument or otherwise
enforce the intended local confidence floor explicitly, and confirm the local
threshold is deliberately distinct from the cloud provider’s 0.78 floor. Keep
target validation and selection behavior unchanged apart from correcting the
effective confidence policy.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/ARCHITECTURE.md`:
- Around line 68-75: Update the local-mode description near the “local path”
statement to clarify that it replaces both cloud vision and cloud speech
synthesis backends with ONNX object detection and sherpa-onnx, while leaving the
safety state machine unchanged. Preserve the listed safety, determinism,
language-table, stop/generation, and licensing details.

In `@reachy_mini_i_spy/config.py`:
- Around line 29-38: The public_dict method currently performs
local_assets_status verification on every status/settings request, including
OpenAI mode. Update public_dict to avoid computing local_assets when provider is
not "local", or cache detector verification using the detector file’s size and
modification time; preserve the existing local_assets payload behavior for
local-provider configurations.

In `@reachy_mini_i_spy/local_provider.py`:
- Around line 15-16: Add the sherpa-onnx core/runtime dependency at version
1.13.4 to the relevant CI installation step before jobs import
local_provider.py. Ensure the CI platform installs the package that provides
libonnxruntime.so, while preserving the existing onnxruntime and sherpa_onnx
imports.
- Around line 396-401: Update the callback passed in the TTS engine’s generate
call to return a non-zero value while synthesis should continue and return zero
when self._cancelled is set. Preserve the existing cancellation behavior by
inverting the _cancelled.is_set() result.

In `@reachy_mini_i_spy/main.py`:
- Around line 97-118: Restrict the `settings_app` server binding to localhost
instead of `0.0.0.0`, updating the `custom_app_url` configuration used when
creating the app. Preserve the existing settings routes and ensure the app
remains reachable from the device itself while no longer accepting
network-interface connections.

In `@README.md`:
- Line 59: Update the README paragraph describing Hugging Face hosted inference
to remove the mutable “$0.10/month” credit figure, or replace it with a dated,
authoritative reference. Apply the same correction to the matching inline note
while preserving the explanation that free users require an HF token and hosted
inference is not used as fallback.

In `@SECURITY.md`:
- Line 5: Update the “Supported version” section in SECURITY.md to explicitly
identify 0.2.0 as the supported release and retain the applicable acceptance
profile; if 0.2.0 is not yet accepted, clearly label the policy as pre-release
instead of using an anonymous in-development release description.

In `@tests/test_public_positioning.py`:
- Line 39: Replace the stale "provider broker" assertion in the public
positioning test with an assertion matching the new provider-selection or
local-mode label rendered in index.html, while preserving the surrounding
page-content checks.

---

Nitpick comments:
In `@reachy_mini_i_spy/local_assets.py`:
- Around line 98-108: Update local_assets_status() so download_bytes is
calculated from the current VOICE_ASSETS registry rather than a hardcoded
constant, summing each registered voice asset’s expected download size using the
existing asset metadata.

In `@reachy_mini_i_spy/local_provider.py`:
- Around line 271-315: Review the confidence-threshold flow between _detect and
select_target: validate_target’s minimum_confidence check is redundant because
_detect already filters detections below SCORE_THRESHOLD. Remove the vacuous
validation argument or otherwise enforce the intended local confidence floor
explicitly, and confirm the local threshold is deliberately distinct from the
cloud provider’s 0.78 floor. Keep target validation and selection behavior
unchanged apart from correcting the effective confidence policy.

In `@reachy_mini_i_spy/static/index.html`:
- Line 69: Update the helper text paragraph near the fixed policy statement to
remove the stale provider URL reference and the time-sensitive Hugging Face
credit amount, retaining only the accurate statement about fixed models,
prompts, and safety bounds.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fd27127d-4616-464c-a1ad-66f7d86a3e83

📥 Commits

Reviewing files that changed from the base of the PR and between 2be830d and d76aab0.

📒 Files selected for processing (39)
  • CHANGELOG.md
  • MANIFEST.in
  • README.md
  • SECURITY.md
  • deploy/README.md
  • deploy/hermes-ispy-broker.service
  • docs/ADDING_A_LANGUAGE.md
  • docs/ARCHITECTURE.md
  • docs/RELEASE_PROVENANCE.md
  • docs/SAFETY_CONTRACT.md
  • hermes_broker/__init__.py
  • hermes_broker/app.py
  • index.html
  • pyproject.toml
  • reachy_mini_i_spy/auth.py
  • reachy_mini_i_spy/config.py
  • reachy_mini_i_spy/game.py
  • reachy_mini_i_spy/local_assets.py
  • reachy_mini_i_spy/local_provider.py
  • reachy_mini_i_spy/main.py
  • reachy_mini_i_spy/models/README.md
  • reachy_mini_i_spy/models/YOLOX_LICENSE.txt
  • reachy_mini_i_spy/models/__init__.py
  • reachy_mini_i_spy/models/yolox_nano.onnx
  • reachy_mini_i_spy/provider.py
  • reachy_mini_i_spy/runtime.py
  • reachy_mini_i_spy/static/index.html
  • reachy_mini_i_spy/static/main.js
  • scripts/check_artifacts.py
  • scripts/verify_broker_readiness.py
  • tests/test_auth.py
  • tests/test_broker.py
  • tests/test_config.py
  • tests/test_gate_adversarial.py
  • tests/test_local_assets.py
  • tests/test_local_provider.py
  • tests/test_provider.py
  • tests/test_public_positioning.py
  • tests/test_runtime.py
💤 Files with no reviewable changes (9)
  • hermes_broker/init.py
  • scripts/verify_broker_readiness.py
  • deploy/README.md
  • tests/test_auth.py
  • reachy_mini_i_spy/auth.py
  • MANIFEST.in
  • deploy/hermes-ispy-broker.service
  • hermes_broker/app.py
  • tests/test_broker.py

Comment thread docs/ARCHITECTURE.md Outdated
Comment thread reachy_mini_i_spy/config.py Outdated
Comment thread reachy_mini_i_spy/local_provider.py Outdated
Comment thread reachy_mini_i_spy/local_provider.py
Comment thread reachy_mini_i_spy/main.py
Comment thread README.md Outdated
Comment thread SECURITY.md Outdated
Comment thread tests/test_public_positioning.py Outdated
@Timverhoogt

Copy link
Copy Markdown
Owner Author

Review remediation complete on a136d8c:

  • lazy-loaded Sherpa to fix x86 native import portability
  • corrected Sherpa continuation/cancellation callback semantics
  • bound the custom settings app to loopback
  • avoided local model checksum work in OpenAI status mode
  • derived remaining download bytes from pinned per-asset metadata
  • updated stale release/security/architecture/HF copy and public-positioning tests
  • retained the local validate_target(... minimum_confidence=SCORE_THRESHOLD) check intentionally as defense-in-depth if detector filtering changes

Verification: 117 local tests, GitHub CI green, artifact checks green, official Pollen clean install/entry-point/uninstall validator green, exact Wireless artifact acceptance green.

@Timverhoogt
Timverhoogt merged commit 73b01db into main Jul 23, 2026
2 checks passed
@Timverhoogt
Timverhoogt deleted the feat/in-process-providers branch July 23, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant