Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider reducing/moving Azdo jobs from PR to nightly CI #4123

Closed
DonJayamanne opened this issue Jan 23, 2019 · 2 comments
Closed

Consider reducing/moving Azdo jobs from PR to nightly CI #4123

DonJayamanne opened this issue Jan 23, 2019 · 2 comments
Labels
debt Covers everything internal: CI, testing, refactoring of the codebase, etc.

Comments

@DonJayamanne
Copy link

DonJayamanne commented Jan 23, 2019

This will greatly speed up Azdo CI.
Waiting for CI to pass takes a while when there are number of PRs.
This will only get worse over time, with more engineers working on PRs in parallel.

E.g.

  • We don't need to run on both Linux & macOS, running just one of them should suffice.
    • This brings it down from 15 to ~8 jobs
  • Remove pip install test requirements from Prebuild job
    • Save another 29s
  • Move Unit tests from prebuild job
    • Save around 2m
  • Leave only linux debugger tests
    • Remove windows debugger tests
    • Save another job

Note, this applies only to PRs

@ghost ghost added the triage-needed Needs assignment to the proper sub-team label Jan 23, 2019
@DonJayamanne DonJayamanne added the important Issue identified as high-priority label Jan 24, 2019
@ghost ghost removed the triage-needed Needs assignment to the proper sub-team label Jan 24, 2019
@DonJayamanne DonJayamanne added debt Covers everything internal: CI, testing, refactoring of the codebase, etc. triage-needed Needs assignment to the proper sub-team labels Jan 24, 2019
@ghost ghost removed the triage-needed Needs assignment to the proper sub-team label Jan 24, 2019
@d3r3kk
Copy link

d3r3kk commented Jan 24, 2019

I think speeding up PR validation step is worthwhile as well, and if the rest of the team agrees too, I think we can certainly use Linux steps in lieu of macOS for PR validation.

Note that we are currently calling out nightly failures in our daily standup in order to ensure we don't miss anything. We are actively looking for better ways to make the nightly CI build more discoverable and are open to ideas (we cannot put the badge at the top of our README.md, yet).

The nightly CI will still run macOS tests in full along with all other tests.

I would add the following detail:

  • Move Unit tests from prebuild job
    • Move them into the same job that runs testSingleWorkspace
    • This adds 2m to the testSingleWorkspace job that runs in parallel (contrasting the prebuild job which blocks all other phases until it completes).
    • This improves our test coverage on both Linux, Windows, (and macOS for nightlies).

@brettcannon brettcannon added feature-* needs proposal Need to make some design decisions labels Feb 19, 2019
@DonJayamanne DonJayamanne mentioned this issue Mar 5, 2019
8 tasks
@luabud luabud removed the important Issue identified as high-priority label Mar 12, 2019
@DonJayamanne DonJayamanne removed this from the 2019, week 11 - March Sprint 6 milestone Mar 13, 2019
@d3r3kk
Copy link

d3r3kk commented Mar 15, 2019

This is complete as of #4775. It is fairly trivial to add/remove jobs from the matrix now. #4775 left the minimum set of builds working per PR to get us up and running, add test runs back to the matrix to improve coverage without adding a lot of time to the build.

@d3r3kk d3r3kk closed this as completed Mar 15, 2019
@ghost ghost removed the needs proposal Need to make some design decisions label Mar 15, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Apr 13, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
debt Covers everything internal: CI, testing, refactoring of the codebase, etc.
Projects
None yet
Development

No branches or pull requests

5 participants