Skip to content

feat(voice): recognise this project's names - #102

Merged
ANT1OFF merged 2 commits into
mainfrom
feat/voice-vocab
Nov 2, 2026
Merged

ANT1OFF merged 2 commits into
mainfrom
feat/voice-vocab

Conversation

@ANT1OFF

@ANT1OFF ANT1OFF commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Dictation now spells the names of the project you're working in the way the code does, plus your own word list. It's layer 1 of the plan, a pass after the commit. Layer 2, sherpa-onnx hotwords on Parakeet, was measured and not kept (see below).

What changes

  • Spelling after commit: apps/web/src/dictation/vocabulary.ts is pure. It runs on each committed utterance, after feat(voice): spoken symbols, formatting and filler clean-up #99's postprocess() (committedText → applyVocabulary), never on the live partial.
    • Case and spacing variants: "use dictation" → useDictation, "dictation dot TS" or dictation.ts → dictation.ts, "feat slash voice vocab" → feat/voice-vocab.
    • The user's "sounds like" spellings are replaced exactly.
    • A conservative near-miss match for long names ("Nematron" → Nemotron, "feet slash voice vocad" → feat/voice-vocab).
  • Precision rules:
    • A run made only of common words ("set state", "list file names", "plan clip") is never turned into a project name unless "dot" or "slash" is said in it.
    • A single common word is never replaced unless the user lists it as a spelling.
    • Two names of the same kind that sound alike cancel out. Symbols beat file stems, which beat folders, branches and titles.
    • A near miss is taken only for names of 8 or more letters: within 1 edit, or 2 at 12 letters and over. It must be the unique nearest, and never when the difference is an ending ("Parakeets" stays plural).
  • Project vocabulary (fs.vocabulary RPC, packages/server/src/fs/vocabulary.ts):
    • Sources: tracked file names and stems (git ls-files), folder names, top-level exports found by a capped git grep -o over TS/JS/Rust/Python, local branches, and words from the project's recent thread titles.
    • Only distinctive names are kept (distinctiveTerm).
    • Files and symbols from the last 200 commits come first.
    • It is fetched only once dictation is prewarmed or started, never on opening a thread.
  • Settings, Voice page → "Vocabulary" group: a "Use this project's names" switch (on by default) and "Your words", a list with "sounds like" spellings that saves on each change. The "Vocabulary hints" copy no longer says hints are unsupported.

Limits

  • Caps: 3,000 project terms in all, with at most 1,500 symbols, 1,200 file names, 300 folders, 150 branches and 150 title words. Up to 200 thread titles are read. The user's list holds 500 words with 8 spellings each.
  • Git work: each git call is bounded at 5 s; the export grep's output is capped at 2 MB.
  • Cache: kept per checkout and read again when HEAD or the index's mtime changes, or after 10 minutes. 8 checkouts are cached at once.
  • Locality: everything stays local. Nothing logs a term or transcript text.

Hotwords: measured, not kept

Clips: Parakeet TDT v2 int8 offline, through sherpa-onnx-node 2.0.0, with 2 threads like the product. 12 synthetic say clips × 2 voices (Samantha, Daniel): 10 name repo identifiers and 2 are controls of only common words.

Hotword setup: they only apply with modelingUnit: "bpe" and a bpe.vocab, which these exports don't ship. One was generated from tokens.txt with scores of −id. Pre-tokenized phrases are rejected ("Encode hotwords failed").

Metrics:

  • WER is over the spoken words, with case and separators normalised.
  • "Names" counts target identifiers that appear verbatim in the final text.
  • "False" counts control clips that layer 1 changed.
Condition WER raw WER + layer 1 Names raw Names + layer 1 False on controls
greedy (today) 10.9 % 6.1 % 4/24 22/24 0
modified beam search, no hotwords 10.4 % 6.1 % 6/24 22/24 0
beam + hotwords (2,835 project terms), score 0.5 6.5 % 5.7 % 8/24 22/24 0
beam + hotwords (2,835), score 1.0 6.1 % 5.2 % 8/24 22/24 0
beam + hotwords (2,835), score 1.5 8.3 % 7.8 % 8/24 21/24 0
beam + hotwords (the 12 targets only), score 1.0 5.7 % 5.7 % 8/24 22/24 0

Why hotwords weren't kept:

  • Most of the gain is already layer 1's. Layer 1 alone takes names from 4/24 to 22/24 and WER from 10.9 % to 6.1 %.
  • Hotwords add about two words in roughly 230 at best (6.1 % → 5.2 %), on TTS audio, with no gain in names.
  • They are score-sensitive. At 1.5 they cut the end off an utterance in both voices ("voice input section dot tsx" → "voice input sects"). A lost word is worse than a misspelling.
  • They change today's decoding. Accurate mode would have to switch from greedy to beam search, with a vocab file whose scores are invented.
  • Latency wasn't measured reliably: the machine's load average was about 30 while the bench ran (200–565 ms per clip, all conditions overlapping).

The two names layer 1 still misses:

  • "inaccurate.ts" for accurate.ts: "inaccurate" is a real word.
  • "Sherponx" / "sherpa onks" for sherpa-onnx: 2 edits on a 10-letter name.

Merge with #98 and #99

Checks

  • bun run typecheck, bun run lint, bun run fmt:check, bun run check:scaffold, bun run check:docs: pass.
  • bun run test over dictation, speech process and module, server fs and the fs handler, contracts rpc, settings and vocabulary, and web settings and composer: 97 files, 979 passed, 1 skipped.
  • playwright test e2e/dictation.spec.ts: 21 passed. That includes the new test: a committed utterance spells retry.test.ts and the user's "para keet" → Parakeet, and with "Use this project's names" off only the user's word is spelled.
  • No full bun run check of my own: the machine is shared.

Known limits:

  • Synthetic success only: no live microphone test.
  • Project names apply only in a thread's composer. The /new and /new-task drafts get the user's list alone.

Screenshots

Settings → Voice input with a word list, the whole page (dark)

Before After
settings-voice-vocabulary-dark before settings-voice-vocabulary-dark after

Settings → Voice input with a word list, the whole page (light)

Before After
settings-voice-vocabulary-light before settings-voice-vocabulary-light after

Committed dictation spells the thread's project's names the way the code
does ("use dictation dot ts" -> use-dictation.ts) and the user's own word
list with sounds-like spellings. fs.vocabulary builds the project terms
from git (tracked names, exported symbols, branches, thread titles),
capped at 3000 and cached by commit and index. Hotwords were measured on
Parakeet and not kept.
@ANT1OFF
ANT1OFF merged commit d306e35 into main Nov 2, 2026
38 of 41 checks passed
@ANT1OFF
ANT1OFF deleted the feat/voice-vocab branch November 2, 2026 22:44
ANT1OFF added a commit that referenced this pull request Nov 3, 2026
Desktop release metadata proposal only. Merging this PR does not publish
or tag Gram; follow docs/RELEASING.md to release.
---


## [1.5.0](v1.4.0...v1.5.0)
(2026-11-03)


### Features

* **voice:** a normal Voice setting with a push-to-talk key picker
([#98](#98))
([a49acdf](a49acdf))
* **voice:** recognise this project's names
([#102](#102))
([d306e35](d306e35))
* **voice:** spoken symbols, formatting and filler clean-up
([#99](#99))
([912b5e5](912b5e5))
* **voice:** voice input on Linux with a TTS-free sherpa-onnx build
([#100](#100))
([4606b96](4606b96))


### Bug Fixes

* **release:** stop the release notes saying voice is missing on Linux
([#107](#107))
([fbde4f2](fbde4f2))
* **worktrees:** sweep archived merged and no-PR worker worktrees
([#103](#103))
([b9ed8d4](b9ed8d4))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: gram-release-please[bot] <351321620+gram-release-please[bot]@users.noreply.github.com>
Co-authored-by: Johan <32440065+ANT1OFF@users.noreply.github.com>
ANT1OFF added a commit that referenced this pull request Nov 3, 2026
## What

When a worker or an orchestrator is at rest waiting on something, the
sidebar now says what, at a glance: an hourglass and a short line under
its name, and the whole list on hover. The Board's chips and ⌘J's
Workers list read the same selector, so the three surfaces say the same
words.

Presentation only. Nothing in `packages/` changes: everything comes from
the ledger the web already gets (PRs, wakes, watched runs, limit waits)
and the threads store.

### One "waiting on" model
`apps/web/src/orchestration/waiting-on.ts` turns a calm worker's or
orchestrator's waits into:
- `label`: the row's line, e.g. `Waiting · checks on #69`
- `reasons`: every wait, each with a kind, a short form and a full line
- `hint`: the hover, i.e. the label with `+N` when more apply, then
every wait's line

| Kind | Label | Comes from |
| --- | --- | --- |
| checks | `Waiting · checks on #69`, `Waiting · local check #69` |
required checks pending; Gram's local check running |
| review | `Waiting · review on #69` | the review bot reviewing the head
|
| merge | `Waiting · merge of #69`, `Waiting · #69 held` | an armed
`merge_when_ready`; a hold on an agent's work |
| dependency | `Waiting · #102 to merge`, `Waiting · acme/api#7 to
merge`, `Waiting · #645 to move` | `depends_on`; a wake on another PR |
| run | `Waiting · Release run` | `run_monitor` |
| condition | `Waiting · main to move`, `Waiting · thread to finish`;
any other kind as its chip reads it (`wakeConditionLabel`) | #108's
condition wakes; the hover names the thread and says `wakeConditionWhen`
|
| timer | `Waiting · until 08:10`, `Waiting · until Wed 09:00` | a timed
wake |
| limit | `Paused · limit until 08:10`, `Paused · retry at 07:05` | a
usage-limit or rate-limit wait |
| held | `Held · by you` | the user holds the worker |
| frozen | (reserved) | nothing produces it yet; the merge freeze will |

**Which wait the label names.** What holds everything else comes first
(a limit, a frozen base, your hold). Then what clears soonest by its
nature (CI, the review bot, Gram's merge), then the longer waits
(another PR, a run, a condition). A timed wake comes last, since it's
mostly the fallback beside the others. Within a kind, the soonest wins.

**Waiting is never "needs you".** A wait is only said of a calm tone
(waiting, idle, or finished with a run still watched), never beside
work, an ask or a failure. Nothing here touches the Needs you list or
counts.

### Surfaces
- **Sidebar worker rows**: the hourglass line replaces the card's now
line ("checks running", "you hold it") while the worker waits. The hover
card lists every wait.
- **Orchestrator line**: a second line under the workspace name for its
own pending wakes, or a limit it waits out. The line's hover adds the
list. It's hidden while the workspace is paused or parked.
- **Board**: a chip's line is the same label, and its hover the same
list.
- **⌘J**: a row's now line is the same label, and its hover the same
list.
- The waiting tone's label is now "Waiting" rather than "Waiting on
checks", since it covers more than checks.

### Behaviour change
A worker waiting out a usage limit (`limitWait` in the ledger) now reads
as waiting on automation, with the hollow dot. Before, its activity
ignored the limit, so it showed idle (or attention).

### Labels kept to four words
The Board keeps a chip's line to four words. Every label fits, with the
` · ` separator counted as punctuation; `board-rail.spec.ts` now counts
it that way.

### Condition kinds still to come
`conditionShort` special-cases `branchMoved` and `threadFinished` only.
Any other kind falls back to contracts' `wakeConditionLabel`, and its
hover line to `wakeConditionWhen`. So #110 (`freezeLifted`), #111
(`ciStuck`, machine load and disk) and #112 (`releasePublished`,
`versionChanged`) show up without a change here. Whichever lands last
adds any icon, words or priority of its own. The `frozen` kind is
reserved for #110's frozen-main state.

## Screenshots
From `pr-shots` on synthetic mock data. The "kinds" scene
(`simulateWaitingOn`) adds a timed wake, a watched release run, a usage
limit, an armed merge, a wake on another PR and one on `main` moving,
and the orchestrator resting on two wakes of its own (a thread to
finish, ranked first, and a timed one).


**The sidebar over `casefiles`: workers waiting on CI, the review bot
and a dependency say what they wait on** (dark)

_After only: the base timed out staging it; the light pair below has the
before._

![sidebar-waiting-dark
after](https://github.com/user-attachments/assets/8cc9172d-160d-4f3a-8544-e82574065fc2)

**The sidebar over `casefiles`: workers waiting on CI, the review bot
and a dependency say what they wait on** (light)

| Before | After |
| --- | --- |
| ![sidebar-waiting-light
before](https://github.com/user-attachments/assets/8e4579f5-a2dc-4621-8fe9-01d367de281b)
| ![sidebar-waiting-light
after](https://github.com/user-attachments/assets/29058cf0-2c0c-499a-8b52-56cd887be7b5)
|

**Every kind of wait in the sidebar, and the orchestrator resting on a
wake of its own** (dark)

_After only: the scene's mock scenario is new in this PR._

![sidebar-waiting-kinds-dark
after](https://github.com/user-attachments/assets/0beada9b-6c10-4254-a7d9-98bfe7fe50ea)

**Every kind of wait in the sidebar, and the orchestrator resting on a
wake of its own** (light)

_After only: the scene's mock scenario is new in this PR._

![sidebar-waiting-kinds-light
after](https://github.com/user-attachments/assets/34353279-1bc1-4030-a5f8-35354028e9e0)

**A waiting worker's hover card lists every wait, the one its row names
first** (dark)

_After only: the scene's mock scenario is new in this PR._

![sidebar-waiting-hover-dark
after](https://github.com/user-attachments/assets/605089f6-e232-47e4-933b-d626fd010bf2)

**The same waits on the Board's chips, in the sidebar's words** (dark)

_After only: the scene's mock scenario is new in this PR._

![board-waiting-kinds-dark
after](https://github.com/user-attachments/assets/7f7e95fd-cd77-473d-93cc-8741f318a85f)

**The same waits on the Board's chips, in the sidebar's words** (light)

_After only: the scene's mock scenario is new in this PR._

![board-waiting-kinds-light
after](https://github.com/user-attachments/assets/8325dda3-5043-4bd4-813e-52b88bb33567)

**⌘J's Workers list says the same** (dark)

_After only: the scene's mock scenario is new in this PR._

![workers-list-waiting-dark
after](https://github.com/user-attachments/assets/c91f9952-74e7-4c0d-98b5-2f834c58522c)

**The same `casefiles` workers on the Board: the chips agree with
Quiet's dots** (dark)

| Before | After |
| --- | --- |
| ![quiet-busy-board-dark
before](https://github.com/user-attachments/assets/ba5d04eb-5035-4fbb-8a44-2f65f8deb895)
| ![quiet-busy-board-dark
after](https://github.com/user-attachments/assets/0a43c050-93f4-4562-bb81-e46e2b1297ac)
|

**The same `casefiles` workers on the Board: the chips agree with
Quiet's dots** (light)

| Before | After |
| --- | --- |
| ![quiet-busy-board-light
before](https://github.com/user-attachments/assets/d317ead1-7c3c-4404-8976-a7bda03e991d)
| ![quiet-busy-board-light
after](https://github.com/user-attachments/assets/1200bee4-f0c0-4feb-bb26-b0d73beef83f)
|


## Verification
On `fc8e877`, rebased onto `main` after #108:
- `bun run typecheck`, `bun run lint`, `bun run fmt:check`, `bun run
check:scaffold`: pass
- `vitest run src/orchestration src/views src/sidebar src/thread
src/shell src/api`: 1228 passed (on `767c1af`); `waiting-on.test.ts`: 30
passed on the head
- Affected e2e, `--workers=2` (`overview-sidebar board-parity board-rail
board-virtual feature-sheet orchestration workers-list quiet-view
overview-extras workspace-density reaching-workers worker-questions
one-thread-budgets ask-stack`): 135 passed, 3 failed. Two were tests to
update for this change (the orchestrator's waits read in rank order;
⌘J's four-word cap counting the ` · `), fixed in `fc8e877`. The third,
`overview-sidebar.spec.ts:851` (a Needs you count of 9 for 8), is
untouched by this and passed alone. All three passed alone on the head.
- Specs updated, not deleted: `board-rail` and `workers-list` count
words without the separator; `overview-sidebar` reads a row's name with
its wait, and tells the agent's glyph from the hourglass.
- Not run: the full `bun run check`, left to the merge gate.

## Known limits
- `frozen` has a label and a rank but no source until the merge freeze
lands.
- A thread-finished condition shows "thread to finish" on the row; the
thread's title is in the hover only, to keep the line short.
- Synthetic mock data only; not run against a live provider or real
workspace.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant