Look into android latest (7.1) failure #3

vweevers · 2018-02-13T22:51:29Z

No description provided.

jhiesey · 2018-02-22T08:26:26Z

Haven't looked too deep, but my guess is this is related to the following caveat:
"Android 7.0 only comes on the GoogleAPI flavor, so make sure that you have your desired capability set to deviceName: Android GoogleAPI Emulator." seen in the release notes

Upstream issue: airtap/browsers#3

jhiesey · 2018-02-27T03:10:19Z

OK, so it works if we set deviceName: Android GoogleAPI Emulator, but that specific string is never returned from the sauce labs rest api (https://saucelabs.com/rest/v1/info/browsers/webdriver) as a long_name value.

On the other hand, more specific names like 'Samsung Galaxy S8 GoogleAPI Emulator' are returned. We could choose one of the available names arbitrarily, or hardcode 'Android GoogleAPI Emulator' somewhere (not ideal). Thoughts?

ralphtheninja · 2018-02-27T03:14:52Z

In which way is this broken?

jhiesey · 2018-02-27T03:17:39Z

Any test that uses android 7 fails:

/Users/jhiesey/Projects/node/airtap/node_modules/wd/lib/webdriver.js:134
        err = new Error('The environment you requested was unavailable.' + jsonData.value ? ('\n' + jsonData.value.message) : '');
                                                                                    ^

TypeError: Cannot read property 'value' of undefined
    at /Users/jhiesey/Projects/node/airtap/node_modules/wd/lib/webdriver.js:134:85
    at Request._callback (/Users/jhiesey/Projects/node/airtap/node_modules/wd/lib/http-utils.js:88:7)
    at Request.self.callback (/Users/jhiesey/Projects/node/airtap/node_modules/request/request.js:186:22)
    at Request.emit (events.js:160:13)
    at Request.<anonymous> (/Users/jhiesey/Projects/node/airtap/node_modules/request/request.js:1081:10)
    at Request.emit (events.js:160:13)
    at IncomingMessage.<anonymous> (/Users/jhiesey/Projects/node/airtap/node_modules/request/request.js:1001:12)
    at Object.onceWrapper (events.js:255:19)
    at IncomingMessage.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)

The exception thrown in webdriver itself when trying to generate the error message is a separate issue.

ralphtheninja · 2018-02-27T04:29:05Z

Some digging (and annoying multiple property translations) just to get some more information on what it looks like now.

When asking the sauce labs api we get the following meta data:

{
  "short_version": "7.1",
  "long_name": "Samsung Galaxy Tab A 10 GoogleAPI Emulator",
  "api_name": "android",
  "long_version": "7.1.",
  "device": "GalaxyTabA10GoogleAPI",
  "latest_stable_version": "",
  "automation_backend": "webdriver",
  "os": "Linux"
}

Current functionality is to pick .api_name, which gets translated to .name here:

https://github.com/airtap/airtap/blob/d0dd74a8326602d0bdc4d40f9d3f23baee824f49/lib/get-saucelabs-browsers.js#L42-L56

Which then gets translated to .browser here:

https://github.com/airtap/airtap/blob/master/lib/airtap.js#L55

Which then gets translated to .browserName here:

https://github.com/airtap/airtap/blob/master/lib/sauce-browser.js#L59

ralphtheninja · 2018-02-27T04:30:25Z

I'm thinking the ideal solution is to be able to use the first info struct we get from their api and just pass that back to sauce labs when requesting a test and get rid of all these translations in the first place. But that would require interaction with sauce labs and get their support to improve on this.

ralphtheninja · 2018-02-27T10:25:08Z

Some more information. When using wd.init(initOpts) we pass in a DesiredCapabilities which has browserName, version and platform, which doesn't give us much to go on. If we can't do browserName='android' and version='7.0' it seems to me we're out of luck.

Please correct me if I'm wrong.

Also, the status on the JsonWireProtocol for Selenium is OBSOLETE. Not sure what that exactly means but I'm guessing sort of oldish and soon-to-be-deprecated-or-killed? Maybe want to take a look into the W3C specification. All we're doing anyway is picking a browser and telling it to load a page and no fancy DOM stuff, so seems to me it should be fairly easy to switch to something more up to date.

ralphtheninja · 2018-02-27T10:26:34Z

Some more information. When using wd.init(initOpts) we pass in a DesiredCapabilities which has browserName, version and platform, which doesn't give us much to go on. If we can't do browserName='android' and version='7.0' it seems to me we're out of luck.

If we can't get that ^ to work it seems to me that the problem lies at Sauce Labs.

ralphtheninja · 2018-02-27T10:45:48Z

Oh apparently JsonWireProtocol is the way to go. I think obsolete above just means that the api docs are old. Not really any use of switching from wd to webdriverio then

ralphtheninja · 2018-03-08T12:55:22Z

admc/wd#522 was submitted to fix the jsonData error.

feross · 2018-05-25T06:26:30Z

I just noticed the jsonData error starting to happen with iPhone latest as well :-(

browsers:
  - name: iphone
    version: latest

feross · 2018-05-25T06:39:10Z

If I manually update wd to greater than 1.6.1 to fix the jsonData bug, then I get this error which is still preventing iPhone 11.3 tests from running:

airtap browser error ([email protected]: [init({"name":"arch","browserName":"iphone","version":"11.3","platform":"Mac 10.13"})] The environment you requested was unavailable.: Appium 1.8.0 does not support Mac 10.13. Please check our platform configurator (https://saucelabs.com/docs/platforms)), restarting +165ms

What does this mean?

Appium 1.8.0 does not support Mac 10.13. Please check our platform configurator (https://saucelabs.com/docs/platforms)

Any ideas?

vweevers · 2018-05-25T06:46:15Z

I had issues with the same browser yesterday: airtap/airtap#152 (comment). But after a few retries it did succeed.

feross · 2018-05-25T07:03:09Z

I just merged airtap/airtap#165 which upgrades wd so at least the jsonData bug is fixed on master.

This still doesn't fix Android 7. Still getting:

airtap browser error ([email protected]: [init({"name":"arch","browserName":"android","version":"7.1","platform":"Linux"})] The environment you requested was unavailable.: Unsupported OS/browser/version/device combo: OS: 'unspecified', Browser: 'android', Version: '7.1.', Device: 'Android Emulator'), restarting +186ms

feross · 2018-05-25T07:03:45Z

@vweevers Retrying does not help for the iPhone error I'm getting. It seems similar to the Android 7 issue (i.e. it's a browser configuration issue):

Appium 1.8.0 does not support Mac 10.13. Please check our platform configurator (https://saucelabs.com/docs/platforms)

vweevers · 2018-05-25T09:06:22Z

Seeing the same thing now on memdown. To verify that appium is the problem, maybe we can try SAUCE_APPIUM_VERSION=X airtap test.js? Where X is an older version (I don't know which version though).

See https://github.com/airtap/airtap/blob/67246f20b52fcf465331b0adfbde18edc06bde0e/lib/sauce-browser.js#L59-L61

vweevers · 2018-05-25T09:20:49Z

Tried SAUCE_APPIUM_VERSION=1.7.2. Same error:

[init({"build":"432","name":"memdown","browserName":"iphone","version":"11.3","platform":"Mac 
10.13","appium-version":"1.7.2","tunnel-identifier":"432.1"})] The environment you requested was 
unavailable.: Appium 1.7.2 does not support Mac 10.13. Please check our platform configurator 
(https://saucelabs.com/docs/platforms)

vweevers · 2018-05-25T10:01:36Z

Another potential problem is that sauce-browsers / airtap only dedups browsers by OS, not device. The Sauce Labs API returns multiple iphones with the same version and os:

 {
  "short_version": "11.3",
  "long_name": "iPhone 5s Simulator",
  "api_name": "iphone",
  "long_version": "11.3.",
  "device": "iPhone 5s",
  "latest_stable_version": "",
  "automation_backend": "webdriver",
  "os": "Mac 10.13"
}, {
  "short_version": "11.3",
  "long_name": "iPhone X Simulator",
  "api_name": "iphone",
  "long_version": "11.3.",
  "device": "iPhone X",
  "latest_stable_version": "",
  "automation_backend": "webdriver",
  "os": "Mac 10.13"
}

This can cause airtap to queue the same browser more than once:

airtap:sauce:iphone:11.3 browser conf: {"name":"memdown","build":"432","browser":"iphone","version":"11.3","platform":"Mac 10.13"}
airtap:sauce:iphone:11.3 browser conf: {"name":"memdown","build":"432","browser":"iphone","version":"11.3","platform":"Mac 10.13"}

ralphtheninja · 2018-05-25T10:12:48Z

Another potential problem is that sauce-browsers / airtap only dedups browsers by OS, not device.

@vweevers Lets dedup this in airtap.

ralphtheninja · 2018-05-25T10:15:07Z

Also, I think we should do something about the translations I mentioned above.

vweevers · 2018-05-25T10:15:59Z

Also, I think we should do something about the translations I mentioned above.

Long-term, definitely. But for lack of unit tests, we should be careful to change too much at once.

vweevers · 2018-05-25T10:16:13Z

@feross in the mean time you can downgrade to iphone 11.2:

  - name: iphone
    version: 11.2
    platform: Mac 10.12

vweevers · 2018-05-25T10:20:30Z

(beware though, because of the dedup issue, this will start 12 identical jobs!)

vweevers · 2018-05-25T12:20:58Z

in the mean time you can downgrade to iphone 11.2

Scratch that. iphone 11.2 and ipad are now also failing.

lpinca · 2018-05-27T06:13:07Z

Another potential problem is that sauce-browsers / airtap only dedups browsers by OS, not device.

I think this is by design, if you don't specify platform, browsers with the same version are deduplicated. On the other hand if you specify platform you want all browsers for that platform to be tested. For example

- name: iphone
    version: 11.2
    platform: Mac 10.12

asks to test all browsers whose name matches iphone, have version 11.2 and platform Mac 10.12, not just one.

vweevers · 2018-05-27T06:28:09Z

@lpinca you're not wrong. For the time being though, we have to deduplicate (in airtap) because airtap currently ignores the device property. So the input browsers are distinct but when we send it to Sauce Labs they're all the same.

ralphtheninja · 2018-06-03T23:28:27Z

@vweevers It's basically this key that should be unique, right?

https://github.com/airtap/airtap/blob/c760dac878943dbf66745097edf254ac67eda49c/bin/airtap.js#L150-L151

Damnit! Sometimes I can't get permalinks to work.

ralphtheninja · 2018-06-03T23:32:23Z

No, nevermind.

ralphtheninja · 2018-06-03T23:38:50Z

Well, actually, yes 😆

These property translations are painful to me. In other words we should dedupe on .os and .api_name properties in the original data we get from sauce-browsers.

They translate into:

.os -> .platform
.api_name -> .browser

But isn't the the final .version property important as well? So maybe it's the data to zuul.browser() that should be unique:

https://github.com/airtap/airtap/blob/c760dac878943dbf66745097edf254ac67eda49c/bin/airtap.js#L153-L157

vweevers · 2018-06-04T07:56:12Z

Sometimes I can't get permalinks to work.

Those don't work cross-repo, sadly.

Let's continue discussion at airtap/airtap#168?

ralphtheninja · 2018-06-04T08:06:03Z

Those don't work cross-repo, sadly.

Makes sense now!

jhiesey · 2018-09-24T21:37:46Z

I'm still getting something similar:

Error: [email protected]: [init({"build":"280","name":"[secure]","browserName":"android","version":"7.0","platform":"Linux","tunnel-identifier":"280.1"})] The environment you requested was unavailable.: Misconfigured -- Unsupported OS/browser/version/device combo: OS: 'unspecified', Browser: 'android', Version: '7.0.', Device: 'Android Emulator'
    at /home/travis/build/jhiesey/[secure]/node_modules/wd/lib/webdriver.js:140:15
    at Request._callback (/home/travis/build/jhiesey/[secure]/node_modules/wd/lib/http-utils.js:89:7)
    at Request.self.callback (/home/travis/build/jhiesey/[secure]/node_modules/request/request.js:186:22)
    at Request.emit (events.js:182:13)
    at Request.<anonymous> (/home/travis/build/jhiesey/[secure]/node_modules/request/request.js:1163:10)
    at Request.emit (events.js:182:13)
    at IncomingMessage.<anonymous> (/home/travis/build/jhiesey/[secure]/node_modules/request/request.js:1085:12)
    at Object.onceWrapper (events.js:273:13)
    at IncomingMessage.emit (events.js:187:15)
    at endReadableNT (_stream_readable.js:1092:12)

Is this the same issue?

vweevers · 2018-09-24T21:47:24Z

@jhiesey if you're on latest airtap, then yes. It seems we haven't gotten to the root cause yet. We've updated wd (airtap/airtap#165) and fixed duplicates (airtap/airtap#181), but android 7 is something else

jhiesey · 2018-09-24T21:49:23Z

Yup I'm using v0.1.0

I might take a look into this soonish if nobody else does.

vweevers · 2020-01-23T15:27:54Z

Finally figured this out for real!

For Android and iOS, we and Sauce Labs use the appium automation backend. However, sauce-browsers fetches /rest/v1/info/platforms/webdriver rather than /rest/v1/info/platforms/all. The latter also includes appium-only browsers, including the one we need for android 7+: "Android GoogleAPI Emulator" (which is hardcoded in airtap atm).

vweevers · 2020-01-23T15:32:23Z

Plus, we should be sending it to Sauce Labs like so:

caps['deviceName'] = 'Android GoogleAPI Emulator';
caps['browserName'] = 'Chrome';
caps['platformVersion'] = '7.1';
caps['platformName'] = 'Android';

Rather than what we do now (which Sauce Labs in fact translates into the above):

caps['deviceName'] = 'Android GoogleAPI Emulator';
caps['browserName'] = 'android';
caps['version'] = '7.1';
caps['platform'] = 'Linux';

vweevers · 2020-01-25T13:14:02Z

Wrote https://github.com/airtap/sauce-browsers to handle this (will replace sauce-browsers).

vweevers referenced this issue in webtorrent/magnet-uri Feb 21, 2018

remove broken android test

571e642

jhiesey added a commit to jhiesey/stream-http that referenced this issue Feb 22, 2018

Disable android 7 tests due to airtap issue

d628649

Upstream issue: airtap/browsers#3

jhiesey mentioned this issue Feb 22, 2018

Re-enable tests disabled for airtap jhiesey/stream-http#83

Open

ralphtheninja mentioned this issue May 24, 2018

shaky tests on travis airtap/airtap#152

Closed

2 tasks

ralphtheninja mentioned this issue May 25, 2018

Dedup browsers based on OS and device airtap/airtap#168

Closed

vweevers mentioned this issue May 25, 2018

Downgrade broken iphone latest (11.3) to 11.2 Level/memdown#162

Closed

vweevers mentioned this issue Dec 29, 2018

Fix android 7 airtap/airtap#235

Merged

vweevers closed this as completed in airtap/airtap#235 Dec 29, 2018

Look into android latest (7.1) failure #3

Look into android latest (7.1) failure #3

Comments

vweevers commented Feb 13, 2018

jhiesey commented Feb 22, 2018

jhiesey commented Feb 27, 2018

ralphtheninja commented Feb 27, 2018

jhiesey commented Feb 27, 2018

ralphtheninja commented Feb 27, 2018

ralphtheninja commented Feb 27, 2018

ralphtheninja commented Feb 27, 2018

ralphtheninja commented Feb 27, 2018 • edited Loading

ralphtheninja commented Feb 27, 2018 • edited Loading

ralphtheninja commented Mar 8, 2018

feross commented May 25, 2018 • edited Loading

feross commented May 25, 2018 • edited Loading

vweevers commented May 25, 2018

feross commented May 25, 2018

feross commented May 25, 2018

vweevers commented May 25, 2018 • edited Loading

vweevers commented May 25, 2018

vweevers commented May 25, 2018

ralphtheninja commented May 25, 2018

ralphtheninja commented May 25, 2018

vweevers commented May 25, 2018 • edited Loading

vweevers commented May 25, 2018

vweevers commented May 25, 2018

vweevers commented May 25, 2018

lpinca commented May 27, 2018 • edited Loading

vweevers commented May 27, 2018 • edited Loading

ralphtheninja commented Jun 3, 2018 • edited Loading

ralphtheninja commented Jun 3, 2018

ralphtheninja commented Jun 3, 2018 • edited Loading

vweevers commented Jun 4, 2018

ralphtheninja commented Jun 4, 2018

jhiesey commented Sep 24, 2018

vweevers commented Sep 24, 2018

jhiesey commented Sep 24, 2018

vweevers commented Jan 23, 2020

vweevers commented Jan 23, 2020

vweevers commented Jan 25, 2020

ralphtheninja commented Feb 27, 2018 •

edited

Loading

ralphtheninja commented Feb 27, 2018 •

edited

Loading

feross commented May 25, 2018 •

edited

Loading

feross commented May 25, 2018 •

edited

Loading

vweevers commented May 25, 2018 •

edited

Loading

vweevers commented May 25, 2018 •

edited

Loading

lpinca commented May 27, 2018 •

edited

Loading

vweevers commented May 27, 2018 •

edited

Loading

ralphtheninja commented Jun 3, 2018 •

edited

Loading

ralphtheninja commented Jun 3, 2018 •

edited

Loading