Skip to content

fix(photon): label upstream CatchUpEvents failures - #50071

Closed
SidUParis wants to merge 1 commit into
NousResearch:mainfrom
SidUParis:fix/photon-stream-cloud-diagnostics
Closed

fix(photon): label upstream CatchUpEvents failures#50071
SidUParis wants to merge 1 commit into
NousResearch:mainfrom
SidUParis:fix/photon-stream-cloud-diagnostics

Conversation

@SidUParis

Copy link
Copy Markdown
Contributor

Summary

  • label Photon Spectrum CatchUpEvents internal server errors as upstream of Hermes
  • keep the existing re-subscribe behavior, but add an operator hint that local allowlists/pairing are not the cause
  • add a regression assertion so the diagnostic does not disappear

Background

While dogfooding Photon iMessage through Hermes, the gateway and sidecar were healthy (/healthz returned {"ok": true}), but inbound delivery stopped and the sidecar repeatedly logged:

/photon.imessage.v1.EventService/CatchUpEvents UNKNOWN: Unknown server error occurred

Restarting the gateway/sidecar re-established the stream and replies resumed. This failure happens before Hermes receives an inbound event, so chasing PHOTON_ALLOWED_USERS or pairing state is misleading. The PR makes that distinction explicit in logs for future operators.

Tests

  • node --check plugins/platforms/photon/sidecar/index.mjs
  • python -m pytest tests/plugins/platforms/photon/test_spectrum_patch.py -q

Copilot AI review requested due to automatic review settings June 21, 2026 08:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves operational diagnostics for the Photon iMessage sidecar by distinguishing Photon Spectrum CatchUpEvents internal server errors as upstream of Hermes, helping operators avoid chasing local allowlist/pairing issues when inbound delivery stalls.

Changes:

  • Add inboundStreamErrorMessage() in the Photon sidecar to enrich inbound stream error logs when CatchUpEvents failures occur.
  • Update the inbound stream retry loop to use the new diagnostic message.
  • Add a regression test to ensure the diagnostic strings remain present in the sidecar source.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
plugins/platforms/photon/sidecar/index.mjs Adds error classification for CatchUpEvents failures and routes the retry-loop logging through it.
tests/plugins/platforms/photon/test_spectrum_patch.py Adds a regression assertion to prevent the new diagnostic from disappearing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +363 to +366
out +=
" | Photon Spectrum CatchUpEvents returned an internal server error; " +
"this is upstream of Hermes, so inbound iMessages may not be delivered " +
"until Photon recovers or the stream is re-established.";
Comment on lines +23 to +26
assert "function inboundStreamErrorMessage" in index
assert "EventService/CatchUpEvents" in index
assert "this is upstream of Hermes" in index
assert "PHOTON_ALLOWED_USERS" in index
@alt-glitch alt-glitch added type/bug Something isn't working comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have labels Jun 21, 2026
@huntsyea

Copy link
Copy Markdown
Contributor

We saw this exact diagnostic gap on a live gateway today: the local sidecar was alive and listening, but CatchUpEvents was failing upstream.

Local state during failure:

Photon sidecar: running, 127.0.0.1:8789 listening
spectrum-ts: 5.2.0
@spectrum-ts/imessage: 5.2.0
Photon credentials: stored

Errors included both:

/photon.imessage.v1.EventService/CatchUpEvents PERMISSION_DENIED: [upstream] Authentication failed. grpcCode: 7
/photon.imessage.v1.EventService/CatchUpEvents UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection termination. grpcCode: 14

The added operator hint is useful because the failure is upstream of Hermes receiving an event. Without that label, it is easy to waste time checking PHOTON_ALLOWED_USERS, pairing state, or local sidecar health even though none of those can cause CatchUpEvents to fail before inbound delivery.

Cherry-picked locally with the existing fatal-inbound recovery logic preserved; focused checks passed:

node --check plugins/platforms/photon/sidecar/index.mjs
python3 -m pytest tests/plugins/platforms/photon/test_spectrum_patch.py -q -o 'addopts='
# 4 passed

T02200059 pushed a commit to T02200059/hermes-agent that referenced this pull request Jun 24, 2026
pai-scaffolde pushed a commit to pai-scaffolde/hermes-agent that referenced this pull request Jun 28, 2026
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants