Adjust the CI to work with monorepo

[mlewand]

Provide a description of the task

CI customization will need to be adapted to work with the monorepo.

Couple of questions already:

• we need to adjust current continuous integration flow
  • what tests should be run when a PR is submitted / branch pushed? affected packages or entire monorepo test suite?
  • what test should be run once branch is merged to master? affected packages or entire monorepo test suite?
  • should there be cron runs?

📃 Other details

This is a subtask of #6466.

[Reinmar]

I'd also consider moving to a different CI service as Travis definitely isn't ideal. I have no experience here, but I hear about CircleCI and GH workflows.

Some links:

• https://github.com/marketplace/category/continuous-integration
• https://help.github.com/en/actions/building-and-testing-code-with-continuous-integration/setting-up-continuous-integration-using-github-actions

[Reinmar]

CircleCI's free tier has 1000 minutes monthly: https://github.com/marketplace/circleci. However, I'm sure that we can consider paid options if need be (AFAICS, we'd need the startup plan which isn't that pricy). Especially that not all the code that we have is open sourced anyway.

You can ping @czerwonkabartosz with some questions.

[czerwonkabartosz]

Feel free to ask questions about CircleCI. I will try to answer somehow or @morganfree will answer because he already knows the CircleCI better than me (knows the CircleCI API and documentation by heart) ;)

[pomek]

My proposals about CI flow:

what tests should be run when a PR is submitted / branch pushed? affected packages or entire monorepo test suite?

For PR build – check the coverage of changed packages, execute tests in changed packages, build docs.

For non-master commits: – execute tests in changed packages. At some point, you will create a PR which triggers "PR build" that checks more.

what test should be run once branch is merged to master? affected packages or entire monorepo test suite?

Master commits (push or merge) – all tests with code coverage. Also, docs should be checked.

should there be cron runs?

We're using Cron for publishing nightly docs. So I guess we want to stay with it.

---

Except for the above, we need to remember:

• a validator for checking paths (relatives inside a package, "packages imports" across packages)
• do we need yarn run docs:api --validate-only if we're going to build full documentation?
• anything else?

---

Issues:

Code coverage. Since we had problems with memory when running code coverage for the entire repository, we decided to check the cc of a single package in its repository. Now we're going to have monorepo and we need to find a way how to generate CC.

Should we take a risk and generate cc for few packages at the same time (yarn run test -f core,engine,... -c)? At some point, it will crash.

Or call n-times the same command with different values:

yarn run test -f core -c
yarn run test -f engine -c
yarn run test -f ... -c

Unfortunately, I will take some time.

Another problem is our "cc-service" that does not handle monorepo. Coveralls requires a repository on GH in order to start collecting CC.

Maybe Codecov could resolve our problem but after looking on docs, it's a little weird we need to put 50x packages in the config file. OTOH, I didn't go too deep in this topic. Maybe I'm wrong.

[mlewand]

@pomek your proposal looks good to me 👍 although I'd make the cases as similar as possible so non-master builds imho should also run check coverage of changed packages, rather than tests alone.

As for the issues part, we have verified there is no longer an issue with memory when doing code coverage tests. Seems that our recent unit tests memory leaks or tooling upgrade might have fixed that.

I see no problem in manually maintaining codecov file with our subpackages explicitly listed, as long as this is documented :D

[Reinmar]

I moved this back to it31 as a great majority of the work was done already. If you want to have a followup in it32, please report a new ticket.