Accessibility support for targeted platforms needs to be considered

[zersiax]

Accessibility is more and more becoming an integral part of the design process. For this UI toolkit to be considered for any serious appDev work, this rather vital part of overall UX needs to be considered.
Full disclosure: I am both a blind computer user and a blind developer. In both respects, I would not be able to use the toolkit if this has not been given the attention it, even by law in some places, requires.

[grokys]

I agree - I consider this a must-have!

[jkoritzinsky]

I've done some research on this and here's what I've found:

This should probably hook into the windowing platform (apis are part of those areas).

Here's the APIs we'll probably have to use for each platform:

Win32: Microsoft UI Automation (System.Windows.Automation). We'll have to create server-side providers. Here's some resources:

• https://msdn.microsoft.com/en-us/library/ms747327.aspx
• https://msdn.microsoft.com/en-us/library/ms748277.aspx
• https://msdn.microsoft.com/en-us/library/ms746691.aspx

GTK: The ATK system

Linux in general: AT-SPI

OSX: The NSAccessibility protocols https://developer.apple.com/library/mac/documentation/Accessibility/Conceptual/AccessibilityMacOSX/ImplementingAccessibilityforCustomControls.html#//apple_ref/doc/uid/TP40001078-CH256-SW1

iOS: The UIAccessibilty protocols https://developer.apple.com/library/ios/documentation/UIKit/Reference/UIAccessibility_Protocol/index.html

Android: https://developer.android.com/guide/topics/ui/accessibility/apps.html#custom-views

I suggest that we make a native OSX windowing platform at some point to make accessibility integration easier.

Also it looks like we'll have to make proxy objects (similar to WPF automation peers) to interface with the accessibility systems.

[Roemer]

I am also interested in this. I've written a Windows-based UI automation library FlaUI and would like to also support Avalonia applications. One way would be (as you mentioned) to implement the corresponding protocols for each system. I suspect this is a very time-consuming task.
An alternative would be to create an Avalonia extension which exposes a service (for example a kestrel http service) which allows to communicate with the Avalonia application.
Someone already made a PoC with my library and Avalonia: https://github.com/mterwoord/avalonia-automation
The plus would be that this should be working on all systems with only one implementation, the downside would be that other native automation librarys/applications would not work unless they also implement this custom protocol. What is the general direction Avalonia is going here? Would that Kestrel extension be a way to continue with this?

[kekekeks]

I was considering to implement UIA, since Mono already has ATK bridge for it. Not sure what to do about OSX though.

[Roemer]

You mean that one: https://github.com/mono/uia2atk ? I have the feeling that is kind of abandoned and not completed yet.

[jkoritzinsky]

I'm planning on doing a bridge for each platform and having an ARIA like system that the implementations hook into to get their data.

[jkoritzinsky]

For Win32, we should consider hooking into the Win32 Caret APIs so accessibility tools like Magnifier can correctly follow text input on the screen.

[YBAZAN]

Anything new about this point?

[grokys]

Hi @YBAZAN - yes I've started work on it on the https://github.com/AvaloniaUI/Avalonia/compare/feature/ui-automation branch.

There's basic support on win32 but I've hit a roadblock there, is there anyone on this thread who has implemented a UI automation server on win32? Or X11/OSX for that matter.

My current problem is that although inspect.exe and Accessibility Insights can follow the focused control, Windows Narrator seems unable to - it picks up initial focus but pressing Tab to move focus results in it just highlighting the whole window.

[grokys]

I've opened an early WIP PR for this feature: #5177

It'd really be appreciated if someone with more knowledge of this area could give us feedback and/or technical input into this because it's not something any of the core maintainers are particularly familiar with.

[robloo]

@grokys Commented here

We started implementing automation peers in #5177 but I don't really like WPF/UWP's API that much because it uses GetX methods instead of properties meaning that we can't use INotifyPropertyChanged for change notifications. Should we break our rule of following WPF/UWP APIs in this case?

Personally, I would be hesitant to change this. Mostly because I don't fully understand the automation peer system that well. However, I have a feeling it will be more trouble than it's worth changing it right now. For development speed it might be better to adopt what works now (more code sharing as well) and then when automation is better understood in a future version of Avalonia, accept a breaking change to improve this. Aside from switching to properties perhaps other undiscovered improvements could be made as well.

Alternatively, I don't think there is anything preventing supporting both GetX and a corresponding property? This would allow changing to only properties to be done gradually. Internally, within the automation peers, it could still forward to the properties from the methods. The framework could then be modified to watch for INotifyPropertyChanged if that simplifies things.

[maxkatz6]

@grokys @robloo a bit weird proposal, but it might work:
Use properties for accessibility classes, and create source generator that will create extensions methods in the WPF/UWP compatibility layer.

[robloo]

Use properties for accessibility classes, and create source generator that will create extensions methods in the WPF/UWP compatibility layer.

Good idea, I am leaning more towards a hybrid approach is better supporting both. Then there is no breaking changes for the code already written dependent on GetX methods, but the framework can take advantage of properties and the simplifications INotifyPropertyChanged provides. Code generation to add the GetX methods would simplify a lot and reduce development time.

[grokys]

I posted this as a comment on the WIP accessibility PR, but reposting here for visibility:

As I keep working on this the more I hate the WPF/UWP AutomationPeer API... You end up having to build three trees: Logical Tree -> AutomationPeer tree -> OS Automation Node Tree and keep them synchronized. For what end?

The more I look at the OSX NSAccessibility API the more I feel jealous. Compare the workflow in WPF and OSX to make a control an accessible button.

WPF:

• Create a MyButtonAutomationPeer
• Implement the IInvokeProvider interface on the peer
• Override OnCreateAutomationPeer on MyButton to return a MyButtonAutomationPeer

OSX:

• Implement NSAccessibilityButton on MyButton

Does anyone have any experience of these APIs? Is there an advantage to the WPF/UWP way of doing it?

@zersiax as you opened this issue you may have an opinion?

[grokys]

Here's a excerpt from a unit test for WPF's automation peers. It's completely broken from an API standpoint. The only reason it works is a bunch of REALLY NASTY internal hacks to make stuff like this work:

// Here's our button.
var button = new Button();

// Use UIADecorator as standard decorators don't have automation peers
var root1 = new UIADecorator { Child = button };
var root2 = new UIADecorator();

// We get the automation peer for the button
var buttonPeer = UIElementAutomationPeer.CreatePeerForElement(button);

// WTF, the button peer parent is null, even though the button has a parent!
Assert.Null(buttonPeer.GetParent());

// That's because its parent automation peer hasn't been created yet.
var root1Peer = UIElementAutomationPeer.CreatePeerForElement(root1);

// The button parent is still null, even though the button has a parent and the parent
// automation peer has been created.
Assert.Null(buttonPeer.GetParent());

// We have to call GetChildren on the root to make the button peer have a parent.
root1Peer.GetChildren();
Assert.IsAssignableFrom<UIElementAutomationPeer>(buttonPeer.GetParent());

// OK fair enough; we can say "if the parent is null we need to walk the tree calling
// GetChildren in order to build the UIA tree". But... what if we now reparent the
// button?
root1.Child = null;
root2.Child = button;

// The parent peer is still root1! So we have no way to know at any point in time whether
// our automation tree is correct.
var parentPeer = Assert.IsAssignableFrom<UIElementAutomationPeer>(buttonPeer.GetParent());
Assert.Same(root1, parentPeer.Owner);

[zersiax]

Honestly I'd chip in but I know very little about the way this is handled on the WPF/WinForms/UWP level, and I'm actually not sure why we need to do all this on that level.
https://en.wikipedia.org/wiki/Microsoft_UI_Automation#:~:text=Microsoft%20UI%20Automation%20(UIA)%20is,successor%20to%20Microsoft%20Active%20Accessibility.

UIA is what in the end delivers accessibility on Windows. Most standard WPF controls as well as UWP controls have sensible defaults in exposing UIA properties to my knowledge:
https://www.codemag.com/Article/0810102/A-Pragmatic-Approach-to-WPF-Accessibility
https://docs.microsoft.com/en-us/windows/uwp/design/accessibility/basic-accessibility-information

Is there no way to abstract over these already existing patterns and use what MS already gives for free?
As for OS X, same difference. The standard controls already have sensible accessibility defaults, reinventing the wheel to talk to the underlying APIs rather than trying to reuse what's already there seems like it'd be way harder than it needs to be. But then, I have no idea how Avalonia's internals work and if what I am proposing is even doable.

[grokys]

Yeah Microsoft UI Automation (UIA) is the API on Windows. As you say WPF and UWP have support for interfacing with these APIs. Because Avalonia works on the same level as WPF/UWP we can't reuse their implementations as such, we have to implement it ourselves with backends for UIA on Windows, NSAccessibility on OSX and AT-SPI on Linux.

However as well as interfacing with the OS-level APIs there also has to be a user-facing API in Avalonia which abstracts over these platform APIs. I have got a decent way along porting WPF's user-facing API implementation to Avalonia but it's so full of hacks I'm reconsidering whether we should go with that API and instead use something more similar to NSAccessibility on OSX.

I was hoping that you had experience developing for UIA and maybe the other Accessibility frameworks on the other OSs so I could get some input, but no problem if not ;) I think I'll experiment with a few ideas and see which feels best. I'm not sure at this point that following the WPF API is even going to be that advantageous from a porting standpoint because to be honest, it seems like most people don't bother/need to customize the accessibility behavior for their controls, or if they do that customization is reasonably trivial.

[robloo]

It might be easiest to start talking about the layers here. In WPF:

1. Controls (Composed of both Visual and Logical tree elements)
2. AutomationPeers implementing WPF accessibility API (an abstraction layer)
3. System accessibility API (UIA)

In MacOS some optimizations can be done and the controls perhaps directly implement the system accessibility API. That makes sense but loosing the abstraction ties your hands in some other ways. In MacOS it doesn't matter much as Apple tightly controls the entire system (more than Windows). In Avalonia however you MUST follow the WPF abstraction layers simply because it runs cross-platform.

The high-level controls in Avalonia, each will need to implement a cross-platform ready 'Avalonia Accessibility API' whether that is implemented directly by the control or in an AutomationPeer class is irrelevant. The functionality required by the two distinct layers is still there even if you attempt to integrate the two. I would further argue that integrating the code for these first two layers directly in the control classes is not good for maintainability and separation of concerns, (otherwise would recommend partial classes)). Finally, the Avalonia accessibility API will connect with the system API.

Also don't forget that accessibility in Avalonia must have knowledge of not only the logical tree - but the visual tree as well most likely. Some sort of adorners will need to be shown when controls have focus for accessibility purposes. This means I don't see how you will get around having to keep the various trees in-sync. That said, synchronization shouldn't be too difficult as it should only go one way App -> system through the various layers (Edit: of course input goes both ways).

[grokys]

Thanks for the input @robloo!

With your feedback and a heavy heart I've decided to try to just port the WPF/UWP API for the moment because I don't feel like I have enough experience in this area to come up with a decent alternative API. Having said that, I'll respond to a few points you made:

The high-level controls in Avalonia, each will need to implement a cross-platform ready 'Avalonia Accessibility API' whether that is implemented directly by the control or in an AutomationPeer class is irrelevant. The functionality required by the two distinct layers is still there even if you attempt to integrate the two.

Yes, but I think you're underestimating the suckiness of the AutomationPeer API and how it interacts with UIA ;) A few examples:

• It goes to great lengths to lazily create the needed parts of the automation tree (see below) but then enumerates properties by querying the automation peer with the IDs of every property in existence, which stands at about 300 if no custom ones are added. And each of these properties are queried on a background thread meaning each one of the 300 separate property reads needs to be marshalled to the UI thread. For each node in the tree.
• Peers are created lazily and parented when another peer returns the peer from GetChildrenCore. Except a re-read of the child peers is only triggered via UIA, which only reads the children of the node it's currently interested in. Meaning that at any time you have absolutely no idea if the state of the automation peer tree makes any sense whatsoever (it usually doesn't, unless you go through every node in the tree to update it)
• Which, strangely enough! WPF does! It causes an update of the child nodes on every layout pass, which means the child nodes of every peer are completely recalculated by allocating a new list and the doing a diff with the previous child state, which in 99% of cases hasn't changed. At least it can stay on the UI thread.

I could continue but I'd just be ranting.

Also don't forget that accessibility in Avalonia must have knowledge of not only the logical tree - but the visual tree as well most likely.

WPF's automation peers work on the visual, not logical tree by default.

That said, synchronization shouldn't be too difficult as it should only go one way App -> system through the various layers

It wouldn't be difficult if you could guarantee whether any particular layer was up-to-date by any other means than recreating the whole tree! ;)

Anyway I've been going back and forth too long on this, so lets implement the WPF API, try to make it a little more sane, and see how it fares with the other platform's accessibility APIs.

[robloo]

Yes, but I think you're underestimating the suckiness of the AutomationPeer API

Yea, I don't have a lot of experience with the inner workings but I'm quite certain things can be improved. When testing automation in apps a lot of bugs do tend to appear.

Anyway I've been going back and forth too long on this, so lets implement the WPF API, try to make it a little more sane, and see how it fares with the other platform's accessibility APIs.

I think this is probably the safest strategy to get started. Once accessibility is implemented for macOS and Windows I'm sure there will be obvious ways to improve the API. Even some of the ideas you had for properties would be helpful. I'm certainly not arguing that the WPF API should remain unchanged and was only stating that the overall architecture should remain more-or-less equivalent.

Cross-platform accessibility API's are quite rare. Perhaps only 1-2 frameworks have done it. Avalonia will be breaking a lot of new ground here I think. This means there are a lot of different ideas to learn from and any improvements to the way WPF did things should be greatly encouraged! It's one of those things that's extremely difficult to design ahead of time though. Changing things as they are implemented should be expected.

[ShrutiJaiswal1494]

Hi,
I have pulled the Automation PR and built a local version of Avalonia to do some Accessibility tests on Windows. There are few errors that I am getting while using the Windows Accessibility tool. Any help on how to resolve these errors will be really appreciated.

Error 1: for TreeView Control

When running the Accessibility tool on TreeView Control I was getting the below error

I am not able to figure out how to set different names for the List or TreeViwItem. I tried the below code but that will set the same name for each TreeViewItem.

<UserControl.Styles>
    <Style Selector="TreeViewItem">
      <Setter Property="AutomationProperties.LabeledBy" Value="{Binding #TreeViewItemName}"/>
    </Style>
  </UserControl.Styles>
<Textbox Name="TreeViewItemName" Text="Tree Item" />

Below is the code for the TreeView Control

	<TreeView Name="TreeView" Items="{Binding Items}" SelectedItem="{Binding SelectedItem, Mode=TwoWay}">
            <TreeView.DataTemplates>
              <TreeDataTemplate DataType="models:ItemNode" ItemsSource="{Binding Children}">
                <StackPanel Orientation="Horizontal">
                    <TextBlock Name="Name" Text="{Binding Name}" />
                    <TextBlock Name="Id" Text="{Binding Id}" />
                    <TextBlock Name="Address" Text="{Binding Address}" />
                    <TextBlock Name="Phone" Text="{Binding PhoneNo}" />
                </StackPanel>
              </TreeDataTemplate>
            </TreeView.DataTemplates>
          </TreeView>

Is there a way to set the AutomationProperties.LabeledBy for each item in the tree?

Error 2: for Textbox Control

Seems like the Textbox control is missing support for Text Pattern.

Error 3: for ComboBox Control

For Combox the support for ExpandCollapse Pattern is missing.

For errors 2 and 3, Are these expected? Also, is there a way to get rid of these errors.?

[grokys]

Hi @ShrutiJaiswal1494 - thanks for checking it out!

Yeah as you've noticed that PR is still a work-in-progress and none of us really have the domain expertise needed to properly evaluate what's still needed. You comments are helpful however:

Issue 1

<Setter Property="AutomationProperties.LabeledBy" Value="{Binding #TreeViewItemName}"/>

I think this may be a problem with control scopes. It's not clear from your example where TreeViewItemName exists.

However I suspect the real problem you're encountering is that there's not yet any automation peer for TreeView/TreeViewItem implemented... If this were implemented it could get the Name from the Header property. As I say: WIP ;)

Issue 2

Yeah, in this case the TextBox does have an automation peer but looks like it's not yet implementing the text pattern.

Issue 3

This one looks like a bug. That pattern should be implemented by ComboBoxAutomationPeer as it implements IExpandCollapseProvider.

[ShrutiJaiswal1494]

Hey @grokys ,

Thanks for the reply.

For the TreeView error I have the textblock before the TreeView Control. However, when I tried moving the textblock inside the TreeView It was not able to set the Automation.LabeledBy property.

Code that works with Textblock outside TreeView (This will set the same name for each tree view item)

<UserControl xmlns="https://github.com/avaloniaui"
             xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
             xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
             xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
             xmlns:models="clr-namespace:People.UI.Models"
             mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450">
<UserControl.Styles>
    <Style Selector="TreeViewItem">
      <Setter Property="AutomationProperties.LabeledBy" Value="{Binding #TreeViewItemName}"/>
    </Style>
  </UserControl.Styles>
<DockPanel>
<Textblock Name="TreeViewItemName" Text="Tree Item" />
<TreeView Name="TreeView" Items="{Binding Items}" SelectedItem="{Binding SelectedItem, Mode=TwoWay}">
	<TreeView.DataTemplates>
              <TreeDataTemplate DataType="models:ItemNode" ItemsSource="{Binding Children}">
                <StackPanel Orientation="Horizontal">
                    <TextBlock Name="Name" Text="{Binding Name}" />
                    <TextBlock Name="Id" Text="{Binding Id}" />
                    <TextBlock Name="Address" Text="{Binding Address}" />
                    <TextBlock Name="Phone" Text="{Binding PhoneNo}" />
                </StackPanel>
              </TreeDataTemplate>
	</TreeView.DataTemplates>
</TreeView>
<DockPanel/>
<UserControl/>

Output in Accessibility Tool

Code with Textblock inside TreeView (the intention was to have different names for TreeView items)

<UserControl xmlns="https://github.com/avaloniaui"
             xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
             xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
             xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
             xmlns:models="clr-namespace:People.UI.Models"
             mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450">
<UserControl.Styles>
    <Style Selector="TreeViewItem">
      <Setter Property="AutomationProperties.LabeledBy" Value="{Binding #TreeViewItemName}"/>
    </Style>
  </UserControl.Styles>
<DockPanel>
<TreeView Name="TreeView" Items="{Binding Items}" SelectedItem="{Binding SelectedItem, Mode=TwoWay}">
	<TreeView.DataTemplates>
              <TreeDataTemplate DataType="models:ItemNode" ItemsSource="{Binding Children}">
                <StackPanel Orientation="Horizontal">
                    <Textblock Name="TreeViewItemName" Text="{Binding Id}" />
                    <TextBlock Name="Name" Text="{Binding Name}" />
                    <TextBlock Name="Id" Text="{Binding Id}" />
                    <TextBlock Name="Address" Text="{Binding Address}" />
                    <TextBlock Name="Phone" Text="{Binding PhoneNo}" />
                </StackPanel>
              </TreeDataTemplate>
	</TreeView.DataTemplates>
</TreeView>
<DockPanel/>
<UserControl/>

Output

[grokys]

Hmm, this looks like you might be hitting a name scope problem... Would you be able to try the same thing on WPF and see if it works there first of all? That way we can see if it's an Avalonia bug or expected behavior. Sorry, I'm rather busy right now so won't have time to check myself for a while.

[ShrutiJaiswal1494]

Hey @grokys,
I tried running the Accessibility tool on a WPF application with a TreeView. In Wpf the tool does not complain about the Name property being null for the items.

TreeView Xaml Code in WPF

<Window x:Class="Names.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:local="clr-namespace:Names"
        mc:Ignorable="d"
        Title="MainWindow" Height="450" Width="800">
    <Window.DataContext>
        <local:MainWindowViewModel/>
    </Window.DataContext>
    <DockPanel>
        <TextBlock Name="TreeViewItemName" Text="Tree Item" Visibility="Hidden"/>
        <TreeView AutomationProperties.Name="EmployeeTree" ItemsSource="{Binding EmployeeList}">
            <TreeView.ItemTemplate>
                <HierarchicalDataTemplate ItemsSource="{Binding Team}">
                    <TextBlock Text="{Binding Name}" />
                </HierarchicalDataTemplate>
            </TreeView.ItemTemplate>
        </TreeView>
    </DockPanel>
</Window>

Output for TreeView in WPF

Output for TreeView in Avalonia

[grokys]

#5177 is ready for review as an MVP of this feature.

There's still a lot missing but the PR is already huge, so barring any crashers, I'd like to propose getting it merged and working on improvements in subsequent PRs.

[Roemer]

Really cool work on that end. Is the windows part based on standard Windows UI Automation? At a first glance it seems like it, so it should be fully compatible to be automated with FlaUI. Really looking forward to give it a try.

[akash07k]

What's the accessibility state of Avalonia UI as of now?
Thinking to develop a important application with it if it is accessible.
Please help

[webczat]

what's the status? know the issue is closed but I mean general accessibility status as of now.

[maxkatz6]

Windows and macOS are supported. More platforms can be supported, but needs community help with them.

[webczat]

linux is as always not... I can mostly provide some help or connections to people who could explain the a11y stack but probably couldn't take implementation.

[kekekeks]

linux is as always not

Well, it's somewhat related to AT-SPI2 not having any usable documentation nor usable debugging tools

[webczat]

maybe I can agree with no usable documentation, although I think the main source is the xml definitions of dbus interfaces.
When it goes to testing tools, really? I mean, there is accerciser which shows accessibility trees, which I'd count as debug tool for at-spi. Is there anything else that would be useful?

[maxkatz6]

We don't have any free hands to implement Linux automation atm. But we can provide help with implementation, if somebody wants.
First good step would be to create an issue with possible APIs that can be used to implement automation. It would also works as an issue to track this feature.

[webczat]

to do the issue I need some info from you. does avalonia base directly on x/wayland, or does it base on an existing toolkit at least when it goes to creating top level windows? If so, which toolkit that is? the answer to which automation tools need to be used depend on that, because you usually cannot externally override another toolkit's accessibility implementation to add your own support behind it's back.

[maxkatz6]

@webczat
Avalonia is based directly on X11 with Avalonia.X11 backend. Pure Wayland backend is planned and was drafted - #11546, but was never completed, as core team doesn't have it in priority yet.

FreeDesktop/DBus extensions are actively used as well.

From my understanding, platform implementation needs to wrap Avalonia AutomationPeer of the window (automation root).
That's exactly how MacOS implementation works (including native part).
And same for Windows. In both cases platform implementation redirects automation calls to the wrapped AutomationPeer, as well as listens for events from AutomationPeer.

[webczat]

i don't know anything about avalonia's architecture, note I am not only blind but also a daily linux user, which means learning avalonia while not being able to sensibly test anything I write wouldn't make a lot of sense. I can only comment on the linux side of the puzzle. :) I'm not currently planning to take the development myself, but it is likely valuable to make an issue and describe what I know of implementation requirements, if that helps. Will do it in a second.

[webczat]

I created #14275 to track this and to give all the info I know or which I have deduced without implementing this myself.