tests: test on Linux, macOS and Windows #11538

DanielRuf · 2018-10-14T19:08:40Z

Description

Test on Linux, macOS and Windows. See https://blog.travis-ci.com/2018-10-11-windows-early-release

Types of changes

Documentation
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (anything that would change an existing functionality)
Maintenance (refactor, code cleaning, development tools...)

Checklist

I have read and follow the CONTRIBUTING.md document.
The pull request title and template are correctly filled.
The pull request targets the right branch (develop or develop-v...).
My commits are correctly titled and contain all relevant information.
I have updated the documentation accordingly to my changes (if relevant).
I have added tests to cover my changes (if relevant).

DanielRuf · 2018-10-15T18:42:31Z

@ncoden not sure if this should be unencrypted https://travis-ci.org/zurb/foundation-sites/jobs/441389318#L441

ncoden

With this new configuration, we are starting tests with:

Headless Chrome with Node lts/boron on windows, linux and macOS
Headless Chrome with Node lts/carbon on windows, linux and macOS
Headless Chrome with Node stable on windows, linux and macOS
18 browsers via Browserstack with Node stable on windows, linux and macOS

So tests are run 63 times. Are you sure this is what you want to do ?

DanielRuf · 2018-10-15T19:03:16Z

18 browsers via Browserstack with Node stable on windows, linux and macOS

This is still WIP.
We might change this one to run only on one host.

ncoden · 2018-10-15T19:10:20Z

I don't understand how it would be useful (or useful enough to slow tests) to be tested on several versions of Node on Linux, macOS and Window.

We specify the Foundation package as compatible with node >= 6 so it makes sense to test several versions of Node. But it is up to the Node maintainers to test it on all plateform to ensure it behave the same.

There is some developers tools that embed binaries and can be plateform-specific. But does it worth it to test all this on every pull request ?

DanielRuf · 2018-10-15T19:25:32Z

I don't understand how it would be useful (or useful enough to slow tests) to be tested on several versions of Node on Linux, macOS and Window.

Because there are differences between the platforms and dependencies.

DanielRuf · 2018-10-15T19:26:02Z

But it is up to the Node maintainers to test it on all plateform to ensure it behave the same.

Do not forget the dependencies.

ncoden · 2018-10-15T19:28:28Z

Ok I see. Plateform-specific dependencies (like Sass).

In that case, what about:

Headless Chrome with Node lts/boron on Linux
Headless Chrome with Node lts/carbon on Linux
Headless Chrome with Node lts/dubnium on Linux
Headless Chrome with Node stable on Linux
Headless Chrome with Node stable on macOS
Headless Chrome with Node stable on Window
18 browsers via Browserstack with Node stable on Linux

Also I would prefer to stay with lts versions instead of 8/10. It's more explicit that we only support long-term versions (6.14.x and not 6.0.x).

ncoden · 2018-10-15T20:16:30Z

@DanielRuf The npm installation is failing on Windows: https://travis-ci.org/zurb/foundation-sites/jobs/441821489#L296

DanielRuf · 2018-10-15T20:30:37Z

I know, this is an issue with nvs and the Windows support is still in Beta.

Also all major Node versions should be checked on all platforms. Time does not matter for the testsuite.

DanielRuf · 2018-10-15T20:32:18Z

Also lts/* does not work with nvs in many cases (was fixed now) but 6, 8 and 10 is better to understand as new LTS will get other names so it makes not much sense to use lts/*.

DanielRuf · 2018-10-15T20:33:41Z

Also I would prefer to stay with lts versions instead of 8/10

These are LTS branches / releases. Every even release branch will become LTS, every odd one is shortliving (5, 7, 9).

DanielRuf · 2018-10-15T20:35:00Z

6 will use the latest release of Node.js 6. Also keep in mind that this is just for CI which has always the latest branch releases.

DanielRuf · 2018-10-15T20:38:22Z

Also see https://travis-ci.community/t/windows-instances-hanging-before-install/250/9

And it seems the Windows builds leak probably secure env vars.

DanielRuf · 2018-10-16T12:18:24Z

See jasongin/nvs#66 (basically npm is broken on Windows in specific cases).

ncoden · 2018-10-20T21:29:13Z

@DanielRuf Let me know when you gets it working on Window 😉 (and good luck!)

ncoden · 2018-10-20T21:52:15Z

@DanielRuf We could use Travis "Stages" too. https://docs.travis-ci.com/user/build-stages

Like in the first stage Tests:

Unit tests (Linux + Node stable)
Unit tests (Linux + Node 10)
Unit tests (Linux + Node 8)

Then if it successes, in a second stage Additional Tests:

BrowserStack (Linux + Node stable)
Unit tests (Linux + Node 6)
Unit tests (macOS + Node stable)
Unit tests (macOS + Node 10)
Unit tests (macOS + Node 8)
Unit tests (macOS + Node 6)
Unit tests (Windows + Node stable)
Unit tests (Windows + Node 10)
Unit tests (Windows + Node 8)
Unit tests (Windows + Node 6)

Travis doens't run more than 3 job in parallel so it would not slow down tests and stop the Travis build sooner if any test fails. I moved BrowserStack to first position because it takes 5 times as long to finish (so others tests will run in parallel). What do you think?

DanielRuf · 2018-10-21T09:18:01Z

Currently just the puppeteer issue exists (the dll issue), the remaining steps work now.

ncoden · 2018-10-28T19:24:37Z

@DanielRuf What's the status of this?

DanielRuf · 2018-10-28T19:32:22Z

@DanielRuf What's the status of this?

Still the same, WIP.

DanielRuf · 2018-10-28T19:33:17Z

The Puppeteer dependency is not compatible as you can see in the log of the failing build.

ncoden · 2018-11-18T17:07:22Z

Tests fails because a process continue to run in Windows. It may be Yarn. https://travis-ci.community/t/build-doesnt-finish-after-completing-tests/288/

See https://travis-ci.community/t/build-doesnt-finish-after-completing-tests/288

ncoden

@DanielRuf I resolved conflicts, fixed the Yarn issue that prevented the build to finish and added cache for it. LGTM 👍. Say a word and it's merged 😉

DanielRuf · 2018-11-20T09:04:05Z

.travis.yml

+  - 6
+  - 8
+  - 10
+  - 11


Let's fail early and use node instead of 11 or what do you think @ncoden?

We have 1 year to think about this 😄

Well, 12 is scheduled for 2019-04-23 ;-)

… for v6.5.0 d0f6c3b tests: test on Linux, macOS and Windows 8273493 tests: use Node.js 6, 8 and 10 78b516f tests: run Browswestack tests only on one platform 6234087 ci: use Yarn e457b19 chore: downgrade mocha-headless-chrome for now b453e55 fix: downgrade doiuse to support Node.js 6 0c92518 fix: disable Yarn GPG so it doesn't prevent the Travis build to finish 3f3c331 chore: cache Yarn in Travis Co-Authored-By: Daniel Ruf <[email protected]> Co-Authored-By: Nicolas Coden <[email protected]> Signed-off-by: Nicolas Coden <[email protected]>

DanielRuf added the 🚨tests label Oct 14, 2018

ncoden reviewed Oct 15, 2018

View reviewed changes

ncoden added the WIP label Oct 16, 2018

DanielRuf and others added 6 commits October 19, 2018 13:31

tests: test on Linux, macOS and Windows

d0f6c3b

tests: use Node.js 6, 8 and 10

8273493

tests: run Browswestack tests only on one platform

78b516f

ci: use Yarn

6234087

chore: downgrade mocha-headless-chrome for now

e457b19

fix: downgrade doiuse to support Node.js 6

b453e55

Merge branch 'develop' into DanielRuf/tests/test-linux-macos-windows

9fa1c99

ncoden added 2 commits November 18, 2018 19:14

fix: disable Yarn GPG so it doesn't prevent the Travis build to finish

0c92518

See https://travis-ci.community/t/build-doesnt-finish-after-completing-tests/288

chore: cache Yarn in Travis

3f3c331

ncoden approved these changes Nov 18, 2018

View reviewed changes

ncoden merged commit 4e8ad74 into foundation:develop Nov 20, 2018

DanielRuf commented Nov 20, 2018

View reviewed changes

DanielRuf deleted the tests/test-linux-macos-windows branch November 20, 2018 11:27

ncoden mentioned this pull request Jan 12, 2019

🚀 Release v6.5.2 #11647

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: test on Linux, macOS and Windows #11538

tests: test on Linux, macOS and Windows #11538

DanielRuf commented Oct 14, 2018 •

edited by ncoden

Loading

DanielRuf commented Oct 15, 2018

ncoden left a comment •

edited

Loading

DanielRuf commented Oct 15, 2018 •

edited

Loading

ncoden commented Oct 15, 2018 •

edited

Loading

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

ncoden commented Oct 15, 2018 •

edited

Loading

ncoden commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 16, 2018

ncoden commented Oct 20, 2018

ncoden commented Oct 20, 2018 •

edited

Loading

DanielRuf commented Oct 21, 2018

ncoden commented Oct 28, 2018

DanielRuf commented Oct 28, 2018

DanielRuf commented Oct 28, 2018

ncoden commented Nov 18, 2018

ncoden left a comment •

edited

Loading

DanielRuf Nov 20, 2018

ncoden Nov 20, 2018 •

edited

Loading

DanielRuf Nov 20, 2018

tests: test on Linux, macOS and Windows #11538

tests: test on Linux, macOS and Windows #11538

Conversation

DanielRuf commented Oct 14, 2018 • edited by ncoden Loading

Description

Types of changes

Checklist

DanielRuf commented Oct 15, 2018

ncoden left a comment • edited Loading

Choose a reason for hiding this comment

DanielRuf commented Oct 15, 2018 • edited Loading

ncoden commented Oct 15, 2018 • edited Loading

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

ncoden commented Oct 15, 2018 • edited Loading

ncoden commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 15, 2018

DanielRuf commented Oct 16, 2018

ncoden commented Oct 20, 2018

ncoden commented Oct 20, 2018 • edited Loading

DanielRuf commented Oct 21, 2018

ncoden commented Oct 28, 2018

DanielRuf commented Oct 28, 2018

DanielRuf commented Oct 28, 2018

ncoden commented Nov 18, 2018

ncoden left a comment • edited Loading

Choose a reason for hiding this comment

DanielRuf Nov 20, 2018

Choose a reason for hiding this comment

ncoden Nov 20, 2018 • edited Loading

Choose a reason for hiding this comment

DanielRuf Nov 20, 2018

Choose a reason for hiding this comment

DanielRuf commented Oct 14, 2018 •

edited by ncoden

Loading

ncoden left a comment •

edited

Loading

DanielRuf commented Oct 15, 2018 •

edited

Loading

ncoden commented Oct 15, 2018 •

edited

Loading

ncoden commented Oct 15, 2018 •

edited

Loading

ncoden commented Oct 20, 2018 •

edited

Loading

ncoden left a comment •

edited

Loading

ncoden Nov 20, 2018 •

edited

Loading