feat(web): mobile-first chat — native UI, voice dictation, installable PWA by alexzehe · Pull Request #50140 · NousResearch/hermes-agent

alexzehe · 2026-06-21T12:27:35Z

Summary

Makes the dashboard a first-class experience on phones: on small screens the chat renders as a native messaging UI instead of the terminal-in-the-browser, and the dashboard is now an installable PWA. Desktop is unchanged.

This came out of a real deployment — a family running Hermes on a home Mac, reached over Tailscale from their phones — so the focus was making it genuinely pleasant to use day-to-day from an iPhone.

What's included

Native mobile chat — web/src/components/MobileChat.tsx, rendered on viewports ≤640px (desktop keeps the xterm/PTY terminal):

Message bubbles with streamed markdown; collapsible reasoning shown above the answer
Inline, ordered tool cards — each tool.start closes the current text segment so the final answer renders after the tools; args/results are JSON-stringified (no more [object Object])
Image & PDF attachments, resume-from-history (honors ?resume=), interrupt/stop, inline approvals
Driven directly by the tui_gateway JSON-RPC WebSocket (session.create/prompt.submit + streamed events)

Voice dictation — the mic records on the device and POSTs to the existing POST /api/audio/transcribe endpoint (local faster-whisper by default); the transcript fills the composer.

Claude-style UI — time-based greeting empty state; composer card (attach · model pill · mic · send) with an auto-growing input.

Installable PWA — web manifest, app icons (reusing apps/desktop/assets/icon.png), Apple meta tags, and a minimal pass-through service worker (no app-code caching, so updates are never served stale). PWA static assets (/icons, /manifest.webmanifest, /sw.js) are added to the dashboard-auth public allowlist so the OS can fetch the home-screen icon during "Add to Home Screen" (that fetch carries no session cookie).

Standalone / iOS polish — env(safe-area-inset-top) on the mobile header, 100lvh in standalone mode to fill the screen, full-bleed chat, hide the chat page-header on phones, theme-colored page canvas, locked pinch-zoom.

Backend — pdf.attach falls back to PyMuPDF (fitz) when pdftoppm/poppler is unavailable (e.g. macOS without Homebrew).

Scope / safety

All behavior changes are gated to small screens / standalone PWA mode — desktop and the terminal chat are untouched.
No new required dependencies for the core path: dictation reuses the existing transcribe endpoint; the PDF fallback only triggers when poppler is missing.

Testing

tsc + vite build pass on top of current main.
Exercised end-to-end on a real deployment: native chat (streaming, tool calls, image OCR, PDF reading), dictation (local Whisper), resume-from-history, and the installed PWA on iOS Safari (standalone / Add to Home Screen).

Screenshots

Native mobile chat on iOS — ordered inline tool cards, streamed reasoning, and the composer (attach · model pill · mic · send):

…e PWA Replaces the terminal-in-the-browser with a native chat experience on phones and makes the dashboard installable as a PWA. Desktop is unchanged. - MobileChat: on screens <=640px, render a native chat (message bubbles, streamed markdown, collapsible reasoning above the answer, inline ordered tool cards with readable args, image/PDF attachments, resume-from-history) driven by the tui_gateway JSON-RPC WebSocket, instead of the xterm/PTY terminal. Desktop keeps the terminal. - Voice dictation: the mic records on the device and POSTs to the existing /api/audio/transcribe endpoint (local faster-whisper by default); the transcript fills the composer. - Claude-style empty state (time-based greeting) + composer (attach, model pill, mic, send) with auto-growing input. - Installable PWA: web manifest, app icons, Apple meta tags, and a minimal pass-through service worker (no app-code caching, so updates aren't stale). PWA static assets are added to the dashboard-auth public allowlist so the OS can fetch the home-screen icon during install (that fetch carries no cookie). - Standalone/iOS polish: safe-area top inset on the mobile header, 100lvh in standalone to fill the screen, full-bleed chat, hide the chat page-header on phones, theme-color the page canvas, lock pinch-zoom. - pdf.attach: fall back to PyMuPDF (fitz) when poppler/pdftoppm is unavailable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

teknium1

Thank you for the substantial mobile dashboard contribution. The native-chat goal is still unmet on current main, but this version needs correctness work before it can safely replace the phone PTY surface.

Problems

web/src/components/MobileChat.tsx:334 calls approval.respond without session_id; current tui_gateway/server.py:10215-10219 requires a session before resolving the approval.
The phone branch mounts MobileChat without the selected profile. Current web/src/pages/ChatPage.tsx:299 scopes dashboard chat through scopedProfile, while MobileChat accepts but never receives that prop.
MobileChat.tsx:218-223 clears its live session on reconnect; with no initial ?resume=, MobileChat.tsx:209-212 creates a fresh session instead of recovering the active conversation.
Root-absolute PWA URLs at web/index.html:10-11 and manifest.webmanifest:5,12 are not rewritten by the prefix handling in hermes_cli/web_server.py:15790-15798.

Suggested changes

Carry session_id for approvals, propagate scopedProfile, and adopt a reconnect strategy that preserves the durable session identity.
Make PWA URLs prefix-aware and add focused protocol, profile, reconnect, and prefix tests.

Automated hermes-sweeper review.

teknium1 · 2026-07-14T22:28:46Z

+
+  const stop = useCallback(() => { if (sidRef.current) sendRpc("session.interrupt", { session_id: sidRef.current }); }, [sendRpc]);
+  const respondApproval = useCallback((item: ApprovalItem, choice: "approve" | "deny") => {
+    sendRpc("approval.respond", { approval_id: item.approvalId, choice });


approval.respond requires the live session_id: current tui_gateway/server.py:10215-10219 calls _sess(params, rid) before resolving the approval. Include session_id: sidRef.current, otherwise every inline approval returns a session error.

teknium1 · 2026-07-14T22:28:46Z

+      if (unmounting.current) return;
+      sidRef.current = ""; curText.current = null; curReason.current = null;
+      setRunning(false); setConn("closed");
+      reconnectTimer.current = setTimeout(() => void connect(), 1500);


This reconnect drops sidRef, and the gateway-ready path creates a new session whenever the original resume prop is absent. A transient phone-network disconnect therefore abandons a newly created conversation; preserve and resume its durable session identity.

teknium1 · 2026-07-14T22:28:46Z

+    // Key by the resume target so picking a different chat from history
+    // remounts MobileChat and resumes that specific session (not the first).
+    return <MobileChat key={resumeParam ?? "new"} resume={resumeParam} />;
+  }


Pass the selected scopedProfile here. Main scopes the terminal chat via scopedProfile (ChatPage.tsx:299,914), but this branch creates/resumes the native chat without it, so a selected management profile is ignored on phone-sized screens.

teknium1 · 2026-07-14T22:28:46Z

-      content="width=device-width, initial-scale=1.0, viewport-fit=cover"
+      content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, viewport-fit=cover"
    />
+    <link rel="manifest" href="/manifest.webmanifest" />


This root-absolute manifest URL bypasses an X-Forwarded-Prefix deployment. Current _serve_index rewrites /assets, favicon, fonts, and ds-assets only (hermes_cli/web_server.py:15790-15798); make this URL prefix-aware or extend that rewrite with coverage.

alexzehe marked this pull request as ready for review June 21, 2026 12:33

alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/tui Terminal UI (ui-tui/ + tui_gateway/) comp/cli CLI entry point, hermes_cli/, setup wizard labels Jun 21, 2026

This was referenced Jul 2, 2026

feat(web): add mobile PWA install support #56799

Open

feat(dashboard): installable PWA + fix Skills row overflow on mobile #58532

Open

This was referenced Jul 8, 2026

feat(web): add installable dashboard metadata #60873

Closed

fix(web): harden desktop webapp chat and mobile controls #60829

Closed

teknium1 reviewed Jul 14, 2026

View reviewed changes

alt-glitch mentioned this pull request Jul 24, 2026

feat(web): Android PWA and comprehensive mobile dashboard optimization #70397

Closed

alt-glitch mentioned this pull request Aug 2, 2026

feat(web): add installable mobile dashboard PWA #77086

Open

jakewvincent mentioned this pull request Aug 19, 2026

[Feature]: RFC: phone-native web chat using canonical Hermes sessions #89661

Open

1 task

Navigation Menu

Uh oh!

feat(web): mobile-first chat — native UI, voice dictation, installable PWA - #50140

feat(web): mobile-first chat — native UI, voice dictation, installable PWA#50140
alexzehe wants to merge 1 commit into
NousResearch:mainfrom
alexzehe:pr-mobile-pwa

alexzehe commented Jun 21, 2026 •

edited

Loading

Uh oh!

teknium1 left a comment

Uh oh!

teknium1 Jul 14, 2026

Uh oh!

teknium1 Jul 14, 2026

Uh oh!

teknium1 Jul 14, 2026

Uh oh!

teknium1 Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

alexzehe commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Scope / safety

Testing

Screenshots

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

teknium1 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alexzehe commented Jun 21, 2026 •

edited

Loading