Voice Command

[MariaLiama]

Description of the new feature/enhancement

Voice Command to control the terminal

Proposed technical implementation details (optional)

[JushBJJ]

If microsoft approves this, we should probably brainstorm ideas on how this would work.

[zadjii-msft]

I honestly don't see how this would effectively useful, however, I'll acknowledge that this is a valid feature request.

There's definitely a 0% chance that this gets implemented by our team outside of a hackathon. I will throw this into our mega-thread for extenstion ideas though, because I could definitely imagine some other third-party developer trying their hand at this feature, and the Terminal should be able to support them.

[WSLUser]

Technically this could be useful for those with disabilities (other than those who can't speak or speak well). Next thing you know the Console/Terminal is supports virtual keyboards with key presses being entered via blinks of an eye.

[carlos-zamora]

Implementation-wise, I guess you would have to do the following...

• have voice commands to invoke keybinding actions
• use UIA to navigate/read the contents of a terminal buffer
• (the hard part) Input:
  • use speech-to-text to "type" commands into a terminal
  • bonus points if you can figure out mouse interactivity haha
  • figure out some way to send key-chords directly to the terminal (i.e.: ctrl-r)

Fun thought experiment. It would definitely be a fun extension to implement too.

[DHowett-MSFT]

Yeah, this is an interesting extension idea. Thanks! Triaged, backlog, requires Area-Extensibility.

[rjperrella]

Microsoft already has voice recognition technology built into Windows (since Vista if memory serves). It already knows how to do the "hard parts" that @carlos-zamora mentioned in his comment. Is there not an API you can interface with to make voice recognition work better? At the moment, for example, it seems the voice recognition software is unable to "see" what was entered in, so it can't make selections.

[speechchemistry]

I currently use typing mode with NATO phonetics (e.g. load Windows Speech Recognition (WSR), and say "Start typing" then say "Lima Sierra Enter") and you will see what is in the current directory. I realise that sounds a bit tedious but in the long run it's faster than the normal WSR dictation mode because of the improved accuracy. Microsoft has hardly touched WSR since Windows 7 though so it has not kept up with the accuracy of Dragon. Unfortunately Dragon does not fully work on Windows Terminal - how come? Putty works ok with Dragon v15 "spell mode" and it's unlikley that Nuance have optimised it for such a niche app. So is Putty using a more standard form of text input than Windows Terminal? It would be so useful for me with my disability if Windows Terminal was fully compatible with Dragon Speech Recognition.

[charliecalvert]

Voice recognition does work in the terminal. It would be useful to see a list of valid commands. So far I know " press" and "go to beginning/end of line". What else is available?

[JushBJJ]

How do you enable it? @charliecalvert

[charliecalvert]

@JushBJJ Make sure you have the Speech Recognition dialog running and set to Listening. This ancient video has the basics.

[carlos-zamora]

Voice Access seems relevant to this.

• https://beebom.com/what-is-voice-access-windows-11-how-use/#:~:text=1%20Open%20the%20Settings%20app%20using%20the%20Windows,the%20offline%20language%20pack%20to%20get%20started.%20
• https://blogs.windows.com/windows-insider/2021/12/08/announcing-windows-11-insider-preview-build-22518/

We should make sure this has a pleasant experience with Windows Terminal. It really just uses the UIA API, so nothing crazy should happen (knock on wood).

[zadjii-msft]

Note: Discussion:
Can we close this now that Voice Access is GA?

[rjperrella]

Just tried it out on Terminal. Seems to work. I think you can close it.

[charliecalvert]

You'd say voice access is GA, can you tell me what GA stands for in this context?

…

On Fri, Mar 24, 2023 at 12:32 PM Ron Perrella ***@***.***> wrote: Just tried it out on Terminal. Seems to work. I think you can close it. — Reply to this email directly, view it on GitHub <#5434 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN2IFTHSLPHE43LVKDDYW3W5XZEBANCNFSM4MM33PJA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zadjii-msft]

Oh I was using "GA" as "Generally available", i.e., no longer just in Insiders builds.

I'm gonna call this one - I'd bet that Voice Access is better than anything we'd be able to build ourselves. And if it isn't, then I'm sure that's feedback that they'd appreciate to help make VA even better.

Thanks all!