Fix searching elements inside WebViews for iOS 13.3 #273

fr0l · 2020-01-10T15:39:52Z

Fixes issue with searching and interacting with elements inside WebViews for iOS 13.3

Swizzle defaultParameters of XCAXClient_iOS to set maxDepth to a meaningful level of recursion.

Original implementation of defaultParams returns this values

{
  maxArrayCount = 2147483647;
  maxChildren = 2147483647;
  maxDepth = 2147483647;
  traverseFromParentsToChildren = 1;
}

It is quite possible, that it is enough to change only the maxDepth parameter. I'm still experimenting.

Please do not merge yet, Just for verification if it does not break stuff.

fr0l · 2020-01-10T16:17:33Z

Possibly fixes appium/appium#13776

imurchie · 2020-01-10T16:46:36Z

Thanks for this!

I'm not sure it will fix the actual going into webviews (as in appium/appium#13776) but it will hopefully fix the accessing of webview contents within the native context (as in appium/appium#13821). This latter has always been a crapshoot, changing from version to version of both iOS and Xcode.

fr0l · 2020-01-10T17:51:12Z

@imurchie i didn't quite get about

actual going into webviews

Basically what happens with the fix — is now XCTest can traverse views and subviews (including WebViews) of an XCUIApplication and then return XCUIElementQuery (s) and XCUIElement (s) in a meaningful time.

If someone would have very deep element hierarchy, then probably increasing maxDepth would help (but it also will increase chance of a timed out operation.

Most probably we could make the maxDepth parameter configurable (i.e. add it to FBConfiguration) to make it easy to do experiment.

What was happening before — a WebView process would start to consume 100% CPU and method _XCT_requestSnapshotForElement: attributes: parameters: reply: (which is highly likely used under the hood of XCUIElementQuery while getting snapshots XCElementSnapshot) would just timeout somewhere on the way, but a WebView would still consume 100% CPU.

Currently I've tested this fix with several applications using XCTest (the lowest level framework from Appium's point of view)

Hopefully it would not require any changes in Appium tools at all (apart from recompiling WebDriverAgent)

imurchie · 2020-01-10T17:56:07Z

I mean that the way webview contents are generally better handled is to switch to the webview context and interact with the HTML there, rather than interacting through the native layer. The SafariViewController issue is attempting to do that, and it is unlikely that any changes in WebDriverAgent will affect that.

See https://github.com/appium/appium/blob/master/docs/en/writing-running-appium/web/hybrid.md

rakurati7 · 2020-01-10T20:39:52Z

I tried your new change with new WebDriverAgent and it's working great, thanks. However I still find that few websites like yahoo.com, msn.com, m.weibo.cn webview content is not accessible.

For reference here is the actual issue that I posted

mykola-mokhnach · 2020-01-11T09:53:43Z

I agree it would be nice to have the depth value configurable. We also had exactly the same issue in Android and workarounded it by setting the maximum recursion depth to 70.

fr0l · 2020-01-11T10:09:41Z

@mykola-mokhnach i will update the PR in 10-15 min with configurable value

fr0l · 2020-01-11T13:18:18Z

@mykola-mokhnach @imurchie I've modified the PR with allowing to change configuration.

fr0l · 2020-01-11T13:27:10Z

@rakurati7
I've tested the default value of maxDepth to 50 to cover case of web resources like 'yahoo.com' to still be able to find elements.

If you need bigger level of maxDepth, then you need to modify the snapshotTimeout parameter to somewhat like 40 seconds.

mykola-mokhnach · 2020-01-11T14:15:24Z

If you need bigger level of maxDepth, then you need to modify the snapshotTimeout parameter to somewhat like 40 seconds.

It would be nice to describe this somewhere in the troubleshooting documentation at https://github.com/appium/appium/tree/master/docs/en/writing-running-appium/ios

WebDriverAgentLib/Utilities/FBConfiguration.m

fr0l · 2020-01-11T15:41:14Z

If you need bigger level of maxDepth, then you need to modify the snapshotTimeout parameter to somewhat like 40 seconds.

It would be nice to describe this somewhere in the troubleshooting documentation at https://github.com/appium/appium/tree/master/docs/en/writing-running-appium/ios

I'll try to do that

fr0l · 2020-01-11T15:47:04Z

@mykola-mokhnach please could you take a look if it is ok now

WebDriverAgentLib/Utilities/FBConfiguration.m

mykola-mokhnach · 2020-01-11T17:26:37Z

WebDriverAgentLib/Utilities/FBConfiguration.m

@@ -411,4 +434,20 @@ + (BOOL)reduceMotionEnabled
  }
  return NO;
 }
+
+/**
+ Returns defaul parameters for traversing elements tree from parents to children while requesting XCElementSnapshot.


fr0l · 2020-01-11T18:04:55Z

fixed docs

rakurati7 · 2020-01-11T18:42:33Z

@rakurati7
I've tested the default value of maxDepth to 50 to cover case of web resources like 'yahoo.com' to still be able to find elements.

If you need bigger level of maxDepth, then you need to modify the snapshotTimeout parameter to somewhat like 40 seconds.

QQ, So with your new commit I have to build a new WDA, right? and the parameters maxDepth and snapshotTimeout should be passed as a Webdriver capabilities?. Please clarify, thanks.

fr0l · 2020-01-11T22:37:50Z

@rakurati7 yes, you will need to build WDA. But the parameters snapshotTimeout and maxDepth can be set using /appium/settings API.

For example something like this:

This will get session id:

DEFAULT_SESSION_ID=$(curl -s http://localhost:8100/status | awk -F'[:]' '/sessionId/ { gsub(/[" ,]/, ""); print $2 }')

This will get current settings:

curl -s http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This will change settings:

curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 100, "snapshotTimeout" : 60 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This is example search of an element with a name 'Search'

curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element

This is example response for found element:

{
  "value" : {
    "ELEMENT" : "0F000000-0000-0000-65DE-000000000000",
    "element-6066-11e4-a52e-4f735466cecf" : "0F000000-0000-0000-65DE-000000000000"
  },
  "sessionId" : "5CB69058-971C-4A2C-B270-C667AA2439B9"
}%

This is changing settings to lower values:

curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 10, "snapshotTimeout" : 15 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This is example search of an element with a name 'Search' again

curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element

But this time, no element is found:

{
  "value" : {
    "error" : "no such element",
    "message" : "unable to find an element using 'name', value 'Search'",
.....

KazuCocoa

🎉
It would be nice to add the snapshotMaxDepth in https://github.com/appium/appium/blob/master/docs/en/advanced-concepts/settings.md latere

jlipps · 2020-01-12T04:19:41Z

thanks for this valuable contribution @fr0l!

rakurati7 · 2020-01-12T06:01:42Z

@rakurati7 yes, you will need to build WDA. But the parameters snapshotTimeout and maxDepth can be set using /appium/settings API.

For example something like this:

This will get session id:

DEFAULT_SESSION_ID=$(curl -s http://localhost:8100/status | awk -F'[:]' '/sessionId/ { gsub(/[" ,]/, ""); print $2 }')

This will get current settings:

curl -s http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This will change settings:

curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 100, "snapshotTimeout" : 60 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This is example search of an element with a name 'Search'

curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element

This is example response for found element:

{
  "value" : {
    "ELEMENT" : "0F000000-0000-0000-65DE-000000000000",
    "element-6066-11e4-a52e-4f735466cecf" : "0F000000-0000-0000-65DE-000000000000"
  },
  "sessionId" : "5CB69058-971C-4A2C-B270-C667AA2439B9"
}%

This is changing settings to lower values:

curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 10, "snapshotTimeout" : 15 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This is example search of an element with a name 'Search' again

curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element

But this time, no element is found:

{
  "value" : {
    "error" : "no such element",
    "message" : "unable to find an element using 'name', value 'Search'",
.....

Thank you, I am referring to this document and I found Appium settings can be set using 'settings[settingsKey]'. So with this commit can I set the following capabilities in Appium inspector.
'settings[snapshotTimeout]' : 40
'settings[maxDepth]' : 60

WebDriverAgentLib/Utilities/FBConfiguration.m

WebDriverAgentLib/Utilities/FBXCAXClientProxy.m

fr0l · 2020-01-12T14:05:10Z

@mykola-mokhnach thanks for awesome code review!

WebDriverAgentLib/Utilities/FBConfiguration.m

mykola-mokhnach · 2020-01-12T16:23:30Z

you're welcome. Thanks for your contributions

WebDriverAgentLib/Utilities/FBConfiguration.m

rakurati7 · 2020-01-13T05:05:47Z

@rakurati7 yes, you will need to build WDA. But the parameters snapshotTimeout and maxDepth can be set using /appium/settings API.
For example something like this:
This will get session id:
DEFAULT_SESSION_ID=$(curl -s http://localhost:8100/status | awk -F'[:]' '/sessionId/ { gsub(/[" ,]/, ""); print $2 }')
This will get current settings:
curl -s http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot
This will change settings:
curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 100, "snapshotTimeout" : 60 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot
This is example search of an element with a name 'Search'
curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element
This is example response for found element:
{
  "value" : {
    "ELEMENT" : "0F000000-0000-0000-65DE-000000000000",
    "element-6066-11e4-a52e-4f735466cecf" : "0F000000-0000-0000-65DE-000000000000"
  },
  "sessionId" : "5CB69058-971C-4A2C-B270-C667AA2439B9"
}%
This is changing settings to lower values:
curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 10, "snapshotTimeout" : 15 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot
This is example search of an element with a name 'Search' again
curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element
But this time, no element is found:
{
  "value" : {
    "error" : "no such element",
    "message" : "unable to find an element using 'name', value 'Search'",
.....
Thank you, I am referring to this document and I found Appium settings can be set using 'settings[settingsKey]'. So with this commit can I set the following capabilities in Appium inspector.
'settings[snapshotTimeout]' : 40
'settings[maxDepth]' : 60

@fr0l Could you please clear me this?

rakurati7 · 2020-01-20T15:39:55Z

@rakurati7 yes, you will need to build WDA. But the parameters snapshotTimeout and maxDepth can be set using /appium/settings API.
For example something like this:
This will get session id:
DEFAULT_SESSION_ID=$(curl -s http://localhost:8100/status | awk -F'[:]' '/sessionId/ { gsub(/[" ,]/, ""); print $2 }')
This will get current settings:
curl -s http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot
This will change settings:
curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 100, "snapshotTimeout" : 60 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot
This is example search of an element with a name 'Search'
curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element
This is example response for found element:
{
  "value" : {
    "ELEMENT" : "0F000000-0000-0000-65DE-000000000000",
    "element-6066-11e4-a52e-4f735466cecf" : "0F000000-0000-0000-65DE-000000000000"
  },
  "sessionId" : "5CB69058-971C-4A2C-B270-C667AA2439B9"
}%
This is changing settings to lower values:
curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 10, "snapshotTimeout" : 15 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot
This is example search of an element with a name 'Search' again
curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element
But this time, no element is found:
{
  "value" : {
    "error" : "no such element",
    "message" : "unable to find an element using 'name', value 'Search'",
.....
Thank you, I am referring to this document and I found Appium settings can be set using 'settings[settingsKey]'. So with this commit can I set the following capabilities in Appium inspector.
'settings[snapshotTimeout]' : 40
'settings[maxDepth]' : 60

@rakurati7 yes, you will need to build WDA. But the parameters snapshotTimeout and maxDepth can be set using /appium/settings API.

For example something like this:

This will get session id:

DEFAULT_SESSION_ID=$(curl -s http://localhost:8100/status | awk -F'[:]' '/sessionId/ { gsub(/[" ,]/, ""); print $2 }')

This will get current settings:

curl -s http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This will change settings:

curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 100, "snapshotTimeout" : 60 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This is example search of an element with a name 'Search'

curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element

This is example response for found element:

{
  "value" : {
    "ELEMENT" : "0F000000-0000-0000-65DE-000000000000",
    "element-6066-11e4-a52e-4f735466cecf" : "0F000000-0000-0000-65DE-000000000000"
  },
  "sessionId" : "5CB69058-971C-4A2C-B270-C667AA2439B9"
}%

This is changing settings to lower values:

curl -s -X POST -d '{ "settings" : { "snapshotMaxDepth" : 10, "snapshotTimeout" : 15 } }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/appium/settings | grep snapshot

This is example search of an element with a name 'Search' again

curl -s -X POST -d '{ "using" : "name", "value" : "Search"  }' http://localhost:8100/session/${DEFAULT_SESSION_ID}/element

But this time, no element is found:

{
  "value" : {
    "error" : "no such element",
    "message" : "unable to find an element using 'name', value 'Search'",
.....

@fr0l I tried the new appium@beta with new WDA. I still not able to find elements inside webview for few websites like yahoo.com, nytimes.com. I am passing the following settings as capabilities in appium inspector
"settings[snapshotTimeout]": 60,
"settings[snapshotMaxDepth]": 100

Original issue here

fr0l force-pushed the xcode_11_3_web_view branch from d06b9f3 to d870120 Compare January 10, 2020 16:16

imurchie mentioned this pull request Jan 10, 2020

After updating to new Appium, it fails to recognise Webview content in both iOS 12 & 13 devices appium/appium#13821

Closed

fr0l force-pushed the xcode_11_3_web_view branch 5 times, most recently from 2b54cbb to 024f047 Compare January 11, 2020 13:16

mykola-mokhnach reviewed Jan 11, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

mykola-mokhnach reviewed Jan 11, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

mykola-mokhnach reviewed Jan 11, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

fr0l force-pushed the xcode_11_3_web_view branch 2 times, most recently from 622aae8 to e9f253c Compare January 11, 2020 15:35

mykola-mokhnach reviewed Jan 11, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

mykola-mokhnach reviewed Jan 11, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

mykola-mokhnach reviewed Jan 11, 2020

View reviewed changes

mykola-mokhnach approved these changes Jan 11, 2020

View reviewed changes

fr0l force-pushed the xcode_11_3_web_view branch from e9f253c to b4ec652 Compare January 11, 2020 18:04

KazuCocoa approved these changes Jan 12, 2020

View reviewed changes

mykola-mokhnach reviewed Jan 12, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

fr0l force-pushed the xcode_11_3_web_view branch from b4ec652 to 06379ab Compare January 12, 2020 13:12

mykola-mokhnach reviewed Jan 12, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

mykola-mokhnach reviewed Jan 12, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Show resolved Hide resolved

mykola-mokhnach reviewed Jan 12, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBXCAXClientProxy.m Outdated Show resolved Hide resolved

fr0l force-pushed the xcode_11_3_web_view branch from 06379ab to 77cd2e1 Compare January 12, 2020 14:04

mykola-mokhnach reviewed Jan 12, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

fr0l force-pushed the xcode_11_3_web_view branch from 77cd2e1 to cc6eba2 Compare January 12, 2020 17:14

mykola-mokhnach reviewed Jan 12, 2020

View reviewed changes

WebDriverAgentLib/Utilities/FBConfiguration.m Outdated Show resolved Hide resolved

Fix searching elements inside WebViews for iOS 13.3

ed7fbe1

fr0l force-pushed the xcode_11_3_web_view branch from cc6eba2 to ed7fbe1 Compare January 12, 2020 20:48

mykola-mokhnach merged commit 79f40f6 into appium:master Jan 13, 2020

amedvedjev mentioned this pull request Jan 13, 2020

[iOS 13.3] Unable switch context to SafariViewController appium/appium#13776

Closed

mykola-mokhnach mentioned this pull request Jan 13, 2020

iOS 13.3: Failure to Start Safari Test Session when Certain Pages are Loaded appium/appium#13824

Closed

mykola-mokhnach mentioned this pull request Jan 24, 2020

[perfomance][appium-xcuitest-driver][webview]Cannot find elements inside webview with JS on iOS13 appium/appium#13870

Closed

mykola-mokhnach mentioned this pull request Feb 20, 2020

Appium fails to fetch the content of the webview for some of the webpages. appium/appium#13967

Closed

KazuCocoa mentioned this pull request Mar 5, 2020

changelog: add 1.17.0 CHANGELOG appium/appium#14015

Merged

mykola-mokhnach mentioned this pull request Mar 30, 2020

Anyone got the solution for this > Cannot get 'xml' source of the current application #13831 appium/appium#14126

Closed

fr0l deleted the xcode_11_3_web_view branch July 8, 2020 02:34

renovate bot mentioned this pull request Jun 6, 2021

Update dependency appium to v1.22.0 tadashi0713/webdriverio-appium-on-docker-android#15

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix searching elements inside WebViews for iOS 13.3 #273

Fix searching elements inside WebViews for iOS 13.3 #273

fr0l commented Jan 10, 2020 •

edited

Loading

fr0l commented Jan 10, 2020

imurchie commented Jan 10, 2020

fr0l commented Jan 10, 2020

imurchie commented Jan 10, 2020 •

edited

Loading

rakurati7 commented Jan 10, 2020

mykola-mokhnach commented Jan 11, 2020

fr0l commented Jan 11, 2020 •

edited

Loading

fr0l commented Jan 11, 2020

fr0l commented Jan 11, 2020

mykola-mokhnach commented Jan 11, 2020

fr0l commented Jan 11, 2020

fr0l commented Jan 11, 2020

mykola-mokhnach Jan 11, 2020

fr0l commented Jan 11, 2020

rakurati7 commented Jan 11, 2020

fr0l commented Jan 11, 2020 •

edited

Loading

KazuCocoa left a comment

jlipps commented Jan 12, 2020

rakurati7 commented Jan 12, 2020

fr0l commented Jan 12, 2020

mykola-mokhnach commented Jan 12, 2020

rakurati7 commented Jan 13, 2020

rakurati7 commented Jan 20, 2020

Fix searching elements inside WebViews for iOS 13.3 #273

Fix searching elements inside WebViews for iOS 13.3 #273

Conversation

fr0l commented Jan 10, 2020 • edited Loading

fr0l commented Jan 10, 2020

imurchie commented Jan 10, 2020

fr0l commented Jan 10, 2020

imurchie commented Jan 10, 2020 • edited Loading

rakurati7 commented Jan 10, 2020

mykola-mokhnach commented Jan 11, 2020

fr0l commented Jan 11, 2020 • edited Loading

fr0l commented Jan 11, 2020

fr0l commented Jan 11, 2020

mykola-mokhnach commented Jan 11, 2020

fr0l commented Jan 11, 2020

fr0l commented Jan 11, 2020

mykola-mokhnach Jan 11, 2020

Choose a reason for hiding this comment

fr0l commented Jan 11, 2020

rakurati7 commented Jan 11, 2020

fr0l commented Jan 11, 2020 • edited Loading

KazuCocoa left a comment

Choose a reason for hiding this comment

jlipps commented Jan 12, 2020

rakurati7 commented Jan 12, 2020

fr0l commented Jan 12, 2020

mykola-mokhnach commented Jan 12, 2020

rakurati7 commented Jan 13, 2020

rakurati7 commented Jan 20, 2020

fr0l commented Jan 10, 2020 •

edited

Loading

imurchie commented Jan 10, 2020 •

edited

Loading

fr0l commented Jan 11, 2020 •

edited

Loading

fr0l commented Jan 11, 2020 •

edited

Loading