Skip to content

feat(line): add opt-in smart reply modality - #40933

Open
jethac wants to merge 8 commits into
NousResearch:mainfrom
jethac:feat/line-smart-modality-policy
Open

feat(line): add opt-in smart reply modality#40933
jethac wants to merge 8 commits into
NousResearch:mainfrom
jethac:feat/line-smart-modality-policy

Conversation

@jethac

@jethac jethac commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds opt-in LINE smart reply modality routing behind smart_modality / LINE_SMART_MODALITY.
  • Sends native voice replies for LINE voice/audio input and suppresses final text after successful voice delivery.
  • Keeps text input as text and lets photo replies inherit the last text/voice modality for that LINE chat participant.
  • Introduces the adapter reply delivery policy seam (reply_delivery_policy) that the modality routing rides on, with exception isolation and multiplex-safe adapter resolution.
  • Carries regression tests for typed media-cache routing (_download_media dispatch), which main currently lacks.

Self-contained — one approval lands the whole former stack

This PR was previously presented as the top of a 3-PR stack (#35785#40931 → this). Both former base PRs are now closed in favor of this one: their entire diffs were already carried here as patch-equivalent commits, so the three open PRs presented ~triple review surface for one change. History on the review feedback:

Rebased onto current main (2026-08-21); the stale AUTHOR_MAP hunk was dropped (that dict is frozen — the contributor mapping now lives in contributors/emails/).

Related open work

#87397 proposes a per-platform suppress_text_when_voice config flag solving an overlapping problem in gateway/platforms/base.py. The policy seam here is the more general mechanism (per-adapter, exception-isolated, modality-aware); flagging so a maintainer can pick one mechanism deliberately rather than both landing.

Test Plan

  • python -m py_compile gateway/run.py gateway/platforms/base.py plugins/platforms/line/adapter.py
  • python -m pytest tests/gateway/test_reply_delivery_policy.py tests/gateway/test_line_smart_modality.py tests/gateway/test_line_plugin.py tests/gateway/test_platform_base.py -q (141 passed on the rebased head)

@alt-glitch alt-glitch added type/feature New feature or request comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have labels Jun 7, 2026
@jethac
jethac force-pushed the feat/line-smart-modality-policy branch from 1e5f706 to 5683949 Compare June 7, 2026 02:55
@jethac
jethac force-pushed the feat/line-smart-modality-policy branch from 5683949 to 4e5a81a Compare June 23, 2026 10:22
@jethac

jethac commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this feature PR onto the refreshed adapter policy branch (#40931).

Local verification:

uv run --with pytest --with pytest-asyncio --with aiohttp python -m pytest \
  tests/gateway/test_line_smart_modality.py \
  tests/gateway/test_line_plugin.py \
  tests/gateway/test_reply_delivery_policy.py \
  tests/gateway/test_voice_command.py::TestAutoVoiceReply \
  tests/gateway/test_media_download_retry.py \
  -q -o 'addopts='
# 138 passed

uv run --with pytest --with pytest-asyncio --with aiohttp python -m pytest \
  tests/gateway/test_voice_command.py \
  -q -o 'addopts='
# 163 passed, 21 skipped

@jethac
jethac force-pushed the feat/line-smart-modality-policy branch 2 times, most recently from a659756 to 40aa476 Compare June 23, 2026 10:39
@jethac
jethac force-pushed the feat/line-smart-modality-policy branch 7 times, most recently from 3797890 to 3bc90ec Compare June 28, 2026 13:46
@jethac
jethac force-pushed the feat/line-smart-modality-policy branch from 3bc90ec to da82cca Compare July 2, 2026 06:16

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the LINE modality work. The feature is still absent from current main, but two delivery/documentation issues need resolution before salvage.

Problems

  • gateway/run.py:11401 suppresses only the normal final send. Current main marks already_sent after the final streamed content is confirmed delivered (gateway/run.py:20065-20073), and a None response means the adapter does not send another message (gateway/platforms/base.py:4894-4914). Thus, with streaming enabled, the voice reply can be followed by text that was already streamed.
  • plugins/platforms/line/plugin.yaml:67 adds LINE_SMART_MODALITY, but neither website/docs/user-guide/messaging/line.md:159-177 nor website/docs/reference/environment-variables.md:549-569 documents it.

Suggested changes

  • Gate/disable text streaming for smart voice-delivery turns before content is emitted, and add an integration test for the streaming-enabled path.
  • Add the setting and its LINE_PUBLIC_URL audio-delivery prerequisite to both LINE documentation tables.

Automated hermes-sweeper review.

Comment thread gateway/run.py Outdated
Comment thread plugins/platforms/line/plugin.yaml
@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 14, 2026
@jethac
jethac force-pushed the feat/line-smart-modality-policy branch from da82cca to ce65e86 Compare July 19, 2026 01:19
@jethac

jethac commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review — both points fixed; specifics in the inline replies, including two known limits of the streaming gate I'd rather disclose up front (interim-preview path, rapid-fire modality race). Rebased onto current main atop #40931. CI green, mergeable.

@jethac
jethac force-pushed the feat/line-smart-modality-policy branch 3 times, most recently from 9b117c7 to 384f6a5 Compare August 1, 2026 01:28
@jethac
jethac force-pushed the feat/line-smart-modality-policy branch from 384f6a5 to 5ab0632 Compare August 12, 2026 05:22
jethac and others added 2 commits August 21, 2026 17:37
Rebase note: upstream 73e193c ("fix(line): normalize inbound media
types and cache routing") independently shipped a superset of this
branch's adapter fix — typed cache helper dispatch, fileName threading,
and a (path, media_type) return from _download_media. The adapter
changes are therefore dropped in favor of upstream's version.

What remains from the original commit: the mocked regression tests
covering _download_media routing for all four LINE content types,
cache/fetch failure fallbacks, and fileName threading from the message
event — adapted to upstream's keyword-only ``filename=`` parameter and
tuple return, and to media_types now carrying MIME types.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebase note (onto f75b577): upstream's voice.auto_tts sync
(_should_auto_tts_for_chat, "voice accompanies text replies unless the
chat explicitly set off", with unset voice_mode None distinct from
"off") and the NousResearch#60671 streaming-TTS skip landed in these same paths
since this branch's base. Adapted the graft to preserve both: the base
adapter's default reply_delivery_policy and the runner's legacy
fallback now include the auto_tts term, _reply_delivery_policy passes
voice_mode through raw (None when unset) instead of defaulting to
"off", and the runner call site keeps upstream's streaming-TTS guard
while threading _voice_reply_sent/suppress-text through it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

[rebase note 2026-08-01: re-resolved against upstream e444d16 — adopted upstream's tightened voice.auto_tts precedence (fallback only when chat has no explicit voice mode, `voice_mode is None`, upstream changed from `!= "off"`) in both the base adapter's default reply_delivery_policy and the runner's legacy fallback]
jethac and others added 6 commits August 21, 2026 17:37
self.adapters is the default profile's adapter map. Resolve the
reply-delivery-policy and inbound-observe adapter through
_adapter_for_source(event.source) so multiplex secondary profiles
consult their own adapter's policy (aligns with 8a9bc38).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Catch exceptions from the adapter reply_delivery_policy callback and
fall back to the legacy voice-mode gate so a buggy adapter policy can
never disrupt final reply delivery. Mirrors the isolation pattern used
by _observe_inbound_message.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With streaming enabled, the final text is streamed and marked
already_sent before the reply-delivery policy runs, so returning None
after the voice reply could not retract it: users got a voice reply
plus the duplicate streamed text. Add a
BasePlatformAdapter.voice_reply_replaces_text pre-flight hook,
implement it for LINE smart modality from the observed inbound
modality, and consult it in both gateway streaming setups BEFORE any
content is emitted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

[rebase note 2026-07-30] Upstream extracted the inline run_sync closure into
TurnRunner.run_sync; the streaming gate that lived in the old closure is
ported to TurnRunner.run_sync (via self._runner._suppress_text_streaming_for_voice,
source from ctx.source). No behavior change intended.
Add the opt-in smart reply modality toggle to the LINE user-guide
env-var table and the environment-variables reference, noting that
successful audio delivery requires LINE_PUBLIC_URL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jethac
jethac force-pushed the feat/line-smart-modality-policy branch from 5ab0632 to 69cf023 Compare August 21, 2026 08:39
@jethac

jethac commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Consolidation update — this PR is now the single vehicle for the former #35785#40931#40933 stack.

  • test(line): regression coverage for typed media-cache routing #35785 and refactor(gateway): add adapter reply delivery policy #40931 are closed in favor of this PR. Verified via git cherry: their entire diffs were already carried here as patch-equivalent commits, so the three open PRs were ~triple review surface for one change. One approval here lands everything.
  • All review feedback across the stack is fixed and the threads are resolved — adapter resolution via _adapter_for_source (multiplex-safe) + exception-isolated policy callback with legacy-gate fallback (the refactor(gateway): add adapter reply delivery policy #40931 points), and the streaming pre-flight gate + LINE_SMART_MODALITY docs (this PR's points).
  • Rebased onto current main today (head 69cf023405): all cherry-picks applied clean, the stale AUTHOR_MAP hunk is dropped (frozen dict; the mapping lives in contributors/emails/ now), and the touched-file suite is green locally (141 passed).

One thing worth a maintainer's deliberate pick: #87397 proposes a per-platform suppress_text_when_voice config flag that solves an overlapping problem in the same file. The policy seam here is the more general mechanism; happy to reconcile in either direction, but one of the two should be chosen before both bitrot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants