VoiceOver unable to interact with buttons #8178
Comments
|
Not an Apple user, so I cannot test that directly. But I just checked the coding and the menu icon button gets a label set that ends up as the aria label:
Do other buttons such as the back and next button work (e.g. moving a Lovelace view left or right)? |
spacegaier
added
the
Accessibility
|
There's a mix of buttons that work and those that do not. Works:
Does not work:
The ones that do not work somewhat universally show highlighted state when interacted with (in fact, the sidebar looks like it changes the activated one visually but it does not do the action). |
|
To make sure we are talking about the same ones: Those do work (moving a view left or right)?
|
|
Oh, no, this one does work: The left/right arrows in your screenshot do not show up as distinct buttons, they're part of the view itself. Activating it just goes to that one, and you can't move them. |
zacwest
changed the title
|
Is anyone working on this? This current limitation is a significant show stopper to use Home Assistant for blind people. |
|
Given that no one is currently assigned to the issue here, probably not (unless someone is without mentioning it here). @zacwest FYI: The entries in the sidebar are not buttons, but just links. They seem to be missing an Also in the first post you mentioned the "hamburg menu" icon, but as I wrote that one does have a label. However, in your second post you then mentioned the 3 dot icon (e.g. to activate UI edit mode or go to the more-info). Did you mean that one also in the original post? |
|
The main menu button doesn't read out as a label in VoiceOver. I'm guessing there's some disconnect between the aria labels and the UI elements themselves when it comes to Safari. |
|
Since a video is worth a bajillion words, this is what VoiceOver reads out for the things in the header, at least. RPReplay_Final1615764290.movThe clicking noises that I do repeatedly are the "activate" action, which normally would perform the button or whatnot. |
|
You do get the ripple, so it does get something... Weird that some do, and some don't work, since they mostly all work the same 🤔 |
|
I think overall very few buttons work, but they all visually look like they are handling the action. |
|
You can also reproduce this using the iOS simulator and Xcode's Accessibility Inspector. You can launch both in Xcode via Xcode > Open Developer Tool.
|
|
Hey all, is there any development here? It would be fantastic if this issue could be resolved quickly. It is literally the only thing stopping me from moving a completely working openHAB setup over to HA. Home Assistant is at the moment effectively not usable by the visually impaired and blind community, which is especially sad as they are a niche but very important user group... I apologize for not contributing to the ticket myself. I do not own an iPhone or have any relevant experience in iOS / web development. The only thing I can do is raise awareness for the importance of this ticket and offer testing support. Thanks and all the best! |
|
@ThomDietrich this issue is specifically for iPhone/iPad, if you don't any of those devices, your issue is probably a different one? Could you maybe create a separate issue with your problems and the devices/browsers/tools you use? |
|
Hey @bramkragten, sorry for the misunderstanding. For background see my related issues #8643 and home-assistant/iOS#1538 Long story short: My brother is blind and depends on his smart home daily. He uses his iPhone and the excellent VoiceOver functionality to interact with every app, just as any of us do. The Home Assistant app is currently inaccessible for VoiceOver users. That effectively blocks us from the planned transition to HA and hence why this issue is so severely important for me, him, and the visually impaired community which could otherwise consider the live-changing advantages of a smart home (exagerating a bit, or am I While I have you: There are a few minor additional issues with accessibility which I captured in home-assistant/iOS#1538 (the one here being the major blocking one). If Nabu Casa wants to address the topic of accessibility by the blind more strategically and eventually celebrate that as another achievement, I would be happy to connect you to a few visually impaired smart home users. |
|
Hello @zacwest and @spacegaier, did you by any chance find the time to look deeper into this? @bramkragten does a Nabu Casa internal format exist to address this topic more strategically. Thanks all! |
|
This PR of mine might be related since I want to ensure that all icon buttons (so buttons without any text) have a title assigned so that tooltips are shown. I will also try to test that with voice commands to ensure the aria labels gets properly set as well. |
|
Great to see! Let us know what the test reveals :) |
|
@spacegaier one more thought. button titles and aria labels are among the common issues with accessibility but in the case of this ticket the buttons can not be "pressed", being the way bigger issue. The video posted by @zacwest demonstrates that quite nicely. Based on your PR description it doesn't solve that... |
|
Hey all, to keep this ticket alive... By now we transitioned my brothers home to Home Assistant and we "hacked" everything so he is able to use his iPhone app to do the most important things. The app is still a pain to use with the screenreader. @bramkragten did you by any chance mention the topic of "smart home for the visually impaired" in any internal discussion? I would be thankful to point me in the right direction to advocate for the topic. @spacegaier @zacwest if you would be interested, I am willing to put a bounty on the resolution of this issue. |
|
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. |
|
iOS 15 is a little nicer here - the button names are read. However, you still cannot activate the buttons with voiceover - nothing happens. |
|
I think I read somewhere that VoiceOver can only activate if the buttons have an |
|
I agree this issue is a major one and has been since the launch of the iOS app. I'd like to try and debug this, but I cannot do it on my phone. Could a developer please respond and let me know if it is enough to examine the code on my desktop PC, or if the mobile view of buttons is altered significantly. If so, an explanation of how (and where in the code) would be super helpful. |
|
We know nearly all buttons cannot be pressed with VoiceOver in either browser or app. Can anyone verify whether or not buttons can be activated on Android with Talkback? I don't have a device readily setup for me to test. |
|
Hey @steverep triggered by the recent comments by @bartbunting and yourself, I reached out to some of the core team via Discord and we had a great video call last Thursday. I've now provided the guys with videos showing the biggest issues and @bramkragten is carrying this forward. I will leave further coordination to him but he mentioned that he is looking for test users, so stay tuned!! All the best, th |
|
I just randomly found this issue, which sounds similar but on Android: #10617 |
Yep, that's probably the same issue, but I'd leave both open and not assume one fix will close both. I also think I might have found the problem, but I need to figure out a way to test it on iOS. |
|
If you have a Mac, you can use the simulator instructions in the iOS repo -- or run the simulator with the website -- a previous comment of mine in this thread includes how to use Accessibility Inspector with it. If you do not, push up a branch and I can test it. |
|
So after a lot of sniffing through the DOM and accessibility tree, my diagnosis for why most buttons can't be clicked with screen readers on mobile is that they are clicking a button that does absolutely nothing. Take the biggest offending components: Instead, both of these custom elements are listening for clicks outside their shadow boundaries and performing the actual actions, and they are not keyboard navigable which is a WCAG 2.1.1 failure. Why these buttons work using desktop screen readers and not mobile is a bit of a mystery, except that in general desktop screen readers and accessibility APIs are much smarter at catching inaccessibility like this and working around it. Weirdly, on iOS at least, some buttons can actually be clicked by using a triple tap with VoiceOver, which simulates a "long press". I have no idea why this would work. So I think the obvious solution is to remove those click events from the custom elements and pass them to the buttons inside their shadow trees. This would also be the cleanest solution and least confusing in the final markup. If that were an insurmountable feat, I could probably rig up a much dirtier solution playing with lots of attributes, but I'd hide in shame afterwards. Unfortunately, I don't have a Mac to test on iOS, and I haven't had the chance to setup a Firefox remote test on an Android, but I feel fairly confident that moving those click listeners would do the trick. Note this should also fix #10617 for Android. |
|
I've been working on this bug for over a week and I need some help from some experienced HA developers please 😅 . First, I now realize my last comment was a bit uninformed on the finer points of event bubbling and delegation. I'm mainly a scientific/engineering programmer who dabbles head first into web development, but I'm a fast learner 👨🎓 . In any case, here's the important parts of what I've learned:
What I really need is for someone with both an iPhone and Mac to use the Accessibility Inspector. Go to any Finally, any suggestions on other packages or HA code to investigate as potentially interfering here would be appreciated. |
|
Hi Steve,
Firstly I just like to say thank you very much for the effort you’re
putting into this it’s very much appreciated.
I’m not sure that I can help all that much. I have both an iPhone and a
Mac. Although currently no Xcode installed. If no one else is able to
provide the trace you need I’m happy to install Xcode
And have a crack at getting the diagnostics you need.
Having said that I don’t have very much time so it might be a slow process
given the things I don’t know.
If no one steps up with a more readily available environment I’m willing to
give it ago because this bug needs to die.
Turn around might be a little bit slow. But then again this bugs been
around for I think multiple years.
Once again thanks for the work you’re putting into this.
Regards
Bart
On 22 Jan 2022, at 7:12 am, Steve Repsher ***@***.***> wrote:
I've been working on this bug for over a week and I need some help from
some experienced HA developers please 😅 .
First, I now realize my last comment was a bit uninformed on the finer
points of event bubbling and delegation. I'm mainly a
scientific/engineering programmer who dabbles head first into web
development, but I'm a fast learner 👨🎓 .
In any case, here's the important parts of what I've learned:
1. A replica of the ha-icon-button component on CodePen
<https://codepen.io/steverep/pen/BawvXpL> works just fine with VoiceOver
on iOS, which tells me there's nothing fundamentally wrong with the code
(or CodePen is injecting something to fix the issue)
2. Adding a simple click event listener directly to the <button> element
also does nothing, which tells me the click event is either being
interrupted immediately or never fired
3. Upgrading to lit 2.1.1 makes no difference (current frontend is held
at 2.0.2)
What I really need is for someone with both an iPhone and Mac to use
the Accessibility
Inspector
<https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/OSXAXTestingApps.html>.
Go to any ha-icon-button component, and look at the data in the
accessibility tree for it and all of its descendants. If you can dump that
and attach it here it might be a big help. And also simulate a click with
VoiceOver on the button to check for errors obviously.
Finally, any suggestions on other packages or HA code to investigate as
potentially interfering here would be appreciated.
—
Reply to this email directly, view it on GitHub
<#8178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADFSPV4CVC72ASDZSIMX23UXG42RANCNFSM4WFZU44Q>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@bramkragten maybe you can help here? |
|
Bingo 🥳. I started tracing click events in the Firefox debugger on my desktop and noticed all of them were being run through a handler in a polymer module I'm working on a PR for the fix and also filing an issue with polymer. |
|
That’s great news!
Thanks for your persistence. This bug has been a major problem for a long
time.
It’s great to finally hear there is a way to move forward.
Regards
Bart
On 25 Jan 2022, at 4:34 am, Steve Repsher ***@***.***> wrote:
Bingo 🥳. I started tracing click events in the Firefox debugger on my
desktop and noticed all of them were being run through a handler in a polymer
module gestures.js
<https://github.com/Polymer/polymer/blob/faae3b1f82ecf3273374641f53a80828d3eb69c9/lib/utils/gestures.js>.
Snooping around there led me to an obscure and undocumented setting called
cancelSyntheticClickEvents, which is true by default. Flipping it to false
solves this issue. The handler is either ignorant of or misreading screen
reader clicks and just killing the events before they go anywhere.
I'm working on a PR for the fix and also filing an issue with polymer.
—
Reply to this email directly, view it on GitHub
<#8178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADFSPT3ZQBIDD6RUVHAIOTUXWES3ANCNFSM4WFZU44Q>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
I can confirm this resolves the vast majority of the button issues, thanks. There are still some issues with the sidebar and more info screens which I'll open another issue for. |
|
@ThomDietrich mentioned before that this was a show stopper for him. I understand that some might exclusively use IOS. So I don't want this to seem to be down playing the importance of this. But I do want to acknowledge the great work that the front end developers of HomeAssistant as this is a really useable interface on Windows with Windows screen readers. I don't have a system that I can install pre-release software on at the moment unfortunately. But as soon as this is released, I will most definitly test. I recorded a video this evening to show HomeAssistant developers how a blind screen reader user interacts with the HomeAssistant UI. Please let me know if I can be of any assistance. I am a big fan of what this project does. |
|
To all screen reader users, Update now to 2022.2.0 to get this fix, but do not update to the 2022.2.1 or later release when it comes. The fix has already been reverted because of #11520, so the mobile apps will become unusable again. @bramkragten please make sure a warning is also put in the release notes. |
|
Hi everyone,
I can confirm that version 2020.2.0 works Great with VoiceOver. It’s
awesome to be able to use the iOS app finally.
Sad to hear it has caused other unexpected side-effects and had to be
reverted. Hoping these can be worked around and provide a solution that
works for everyone. For now I’ll stick to this version.
Although this is probably not the best forum for this I’m not sure how else
to get the information out. There is a mailing list called accessible Home
Assistant accessibleHome ***@***.*** which is a very low traffic
list set up for the discussion of accessibility issues related to Home
Assistant if anyone is interested.
Regards
Bart
On 4 Feb 2022, at 5:11 am, Steve Repsher ***@***.***> wrote:
To all screen reader users,
Update now to 2022.2.0 to get this fix, but *do not* update to the 2022.2.1
or later release when it comes. The fix has already been reverted because
of #11520 <#11520>, so the
mobile apps will become unusable again.
@bramkragten <https://github.com/bramkragten> please make sure a warning is
also put in the release notes.
—
Reply to this email directly, view it on GitHub
<#8178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADFSPVINEPFAWQDCICWOWLUZLANLANCNFSM4WFZU44Q>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
#11693 will fix this again |
|
Upcoming version 2022.3 (now in beta) will have this fixed again as @bramkragten mentioned, so it'll be safe to upgrade. Screen reader users following this will probably quickly notice issue #11502 when opening dialogs. See my comments for the work around and subscribe to that issue for updates. |
Checklist
The problem
The hamburger/menu button has 2 major issues:
Expected behavior
With VoiceOver enabled, I can interact with the menu button.
Steps to reproduce
Environment
State of relevant entities
n/a
Problem-relevant frontend configuration
Occurs on a vanilla install.
Additional information
Message me up on Discord and I can walk you through testing VoiceOver in the iOS simulator or on a physical device.
The text was updated successfully, but these errors were encountered: