Selenium.axe works in production but gives errors in locally hosted environments

[juprog]

Hi,
I hope someone can maybe provide a hint of what is happening, and what could be the cause for the following.
first time here, and first time trying to create a proof of concept to integrate running accessibility tests.

I managed to create some tests , select elements, select iframes, and create reports against those. These are running fine against our production servers, but they will not run against any of our internally hosted environments.

Note: it works for the main login page. That page has no frames. But as soon as I log in, and load any of the internal application pages , they all have iframes, and that is where it seems to fail.

Here is the error that I receive against any of the logged in pages (where the equivalent in production just works):

javascript error: Failed to execute 'getComputedStyle' on 'Window': parameter 1 is not of type 'Element'.
(Session info: chrome=91.0.4472.114)

at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)

at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary2 parameters) at OpenQA.Selenium.Remote.RemoteTargetLocator.Frame(IWebElement frameElement) at Selenium.Axe.WebDriverInjectorExtensions.InjectIntoFrames(IWebDriver driver, String script, IList1 parents)
at Selenium.Axe.WebDriverInjectorExtensions.Inject(IWebDriver driver, IAxeScriptProvider scriptProvider)
at Selenium.Axe.AxeBuilder..ctor(IWebDriver webDriver, AxeBuilderOptions options)
at Selenium.Axe.AxeBuilder..ctor(IWebDriver webDriver)
at SeleniumAxe_POC.Testclass.POC_TestAxe()
Void UnpackAndThrowOnError(OpenQA.Selenium.Remote.Response)

So it seems that pages served from our internal servers, are different, and when the InjectIntoframes occurs, that is where it breaks?

IIS setting? something to do with security, headers? I am not too sure what is at play here?

thanks,
julie

[TroyWalshProf]

I am guessing that the issue is related to this bug:
dequelabs/axe-core#3002
and this one:
#154

Upshot is that --allow-file-access-from-files argument should get you up and running.
If that doesn't work "--disable-web-security" may also do the trick, but that is a last resort.

[TroyWalshProf]

Note* When I finally get around to completing #151 you will be able to simply ignore the iFrames

[juprog]

I am guessing that the issue is related to this bug:
dequelabs/axe-core#3002
and this one:
#154

Upshot is that --allow-file-access-from-files argument should get you up and running.
If that doesn't work "--disable-web-security" may also do the trick, but that is a last resort.

Allow file access from files sounds "risky" to me...
I think I had tried the --disable-web-security flag ,in an experiment using axe-core in webdriverjs and that had not worked. It could be me, just learning all these things.

So using selenium-axe in my selenium tests...
How do I use those flags , I am not sure where or how to use/pass them... with the AxeBuilder? how do I pass these down to axe core? (I looked at documentation , but I may have missed it?)

could it also be something like this , in relation to allowed-origins : they do not recommend using the option <unsafe_all_origins> which is what I would try? (again - not sure HOW to relate that into selenium-axe...
axe.configure({
allowedOrigins: ['<same_origin>', 'https://deque.com']
});

Thanks for the quick replies!

[juprog]

oh, are the flags (that you mentioned) passed to the chromedriver options ? before using axe... ?

tried both, individually, and then together, and that did not seem to change the behavior/outcome.
--allow-file-access-from-files
--disable-web-security

[TroyWalshProf]

Those are chromedriver options.
See:

SeleniumAxeDotnet/Selenium.Axe/Selenium.Axe.Test/IntegrationTests.cs

Line 363 in bb850c1

options.AddArgument("--allow-file-access-from-files");

[TroyWalshProf]

@juprog - Did the Chrome option address your issue?

[juprog]

I isolated something in trying to find a solution. I do not know what it means.

just before the .analyze, I do a driver.SwitchTo().DefaultContent();
and that seems to cause the error : javascript error: Failed to execute 'getComputedStyle' on 'Window': parameter 1 is not of type 'Element'. when I run tests against a local/internally deployed host server

if I comment out the line driver.SwitchTo().DefaultContent(); , then I no longer get the above issue. Instead I get a
stale element reference: element is not attached to the page document
and it still fails to analyze.

so I am not sure how to proceed. before the analyze(), I need to switch into nested frames, to go click on a button, to load the next part. once on the next part, I usually have to navigate back up to the top, then through the frames to the various elements. All of it works with selenium (the reference is never stale, the items are found, manipulated) -- but the axe stuff fails...
results = new AxeBuilder(driver)
.Include("#IDofElement")
.WithOutputFile("TestResults.json"))
.Analyze();

the exact same code works against our production server/app. I am not sure what to do with this.

[TroyWalshProf]

Could this be an HTTP vs HTTPs issue?

[juprog]

thanks for the suggestion. I also I tried that , still nothing. Internally, I tried using IPs, then names, with/without https (with proper ssl certificate)... still seemed to behave the same way.

It may turn out to be something silly... but it just feels like I do not know enough, or am missing something...

[TroyWalshProf]

What happens if you run your test against Firefox?
Does the Axe Chrome extension have the same issue?

[juprog]

this is getting confusing even for me after trying variations on variations, so here is a summary

I can run Chrome Axe Extension on any page, or subframe, or subelement inside a subframe
** sometimes I need to re-execute 2 or 3 times, but it does work

• works on the app in production
• works on the app deployed to localhost, and other internal webservers (http, https)

I can run Firefox Axe Extensions on full page only (they do not have the option for partial page elements)

• works in production
• works localhost, internal webservers (http, https)

I can automate the UI of our application with Selenium webdriver by itself (without the Selenium-axe)

• that works everywhere : against production, internal, localhost
• all navigation and element manipulation works (I login , I navigate frames, and subframes, I find and click buttons, etc)

Running my project written with Selenium-axe against 2 of our production servers - works fine (I like it!)

• I run it with Chrome webdriver and Firefox webdriver and they both work. I can do the .analyze with output, and I can do the CreateAxeHTMLReport . I can go into any subframe, pick elements, and return targeted results .

Running the exact same project containing selenium-axe with only 1 change ( target url in driver.Navigate().GotoURL is targeting the same app, but deployed on internal IIS webservers)

• works ONLY on the first page, which is a login form (that page has no frames, no other injected 3rd party scripts)
• does not work on anything PAST the login page : in chrome webdriver
• does not run on anything PAST the login page : using firefox webdriver
  **I tried disabling flags for security and allow file access, no difference.

So, it does seem that it is related to frames (or combination of internal+externalframes sources) because I can target at least the main page deployed to an internal webserver and run that.

[juprog]

status update: after a couple sleepless nights... it does works locally with the firefox driver.

I am not sure if I missed it in all my tests, or if there were other issues that I was having before . But now, the tests work locally/Internally only in firefox. I still have the original issue in Chrome.

in production : any browser works.

I wish I understood the "why" of why it will not work in Chrome, but right now, as long as I can run it internally and produce reports, that works for me.

[TroyWalshProf]

Maybe this is the result of a race condition.
Would try adding a sleep (just for testing) before running the check.
If this is the issue you may want to look using something like Polly or wait for specific elements to be displayed.

[TroyWalshProf]

@juprog - Did waiting address this?

[juprog]

no. no amount of waiting has seemed to work. see below for more.