[CI] Better utilization of CI resources and other CI improvements

[hcho3]

Run-away cloud cost of our Jenkins CI server has been a pressing issue (#5176). It is hosted on AWS, which charges by the hour. #5884 finally created the mechanism to enforce a daily budget via throttling.

We now have a dashboard page to keep track of daily spending: https://xgboost-ci.net/dashboard/

Now it is time to extract savings and ensure that we are using limited CI resources on where it matters.

State of the CI: The free credits from AWS ran out this month, so we now have to start drawing from the Open Collective account, which currently has 10531.16 USD. If we limit ourselves to spending 33 USD per day, the balance will last 287 days.

• Make all tests conditional on the presence of a GitHub comment (e.g. run tests). Right now, tests run automatically, and there are many cases where automatically starting tests is wasteful.
• Skip tests for all draft pull requests.
• Allow "rolling over" left-over allowance from previous day. For example, if the daily budget is 33 USD and we spent only 10 USD yesterday, we should be able to spend up to 56 USD (33 USD + 23 USD roll over). The spending pattern is quite spiky:
  
• Migrate some tests to free services. I suspect this may have limited impact, since any tests using GPUs need to run in Jenkins.
• Pin point which tests cost the most. I was surprised to learn that Windows jobs cost more than 50% of the expenses, even though we run more tests in Linux:
  
  • [CI] Reduce load on Windows CI pipeline #5892 reduces the number of tests for Windows target by half.
  • Further improvements and savings in Jenkins pipeline #5904 saves the cost of Windows test pipeline by up to 66%.
• Related: write a summary report of the AWS expenses for the last 6 months.
• Speed up C++ builds. It takes ~ 10 min on Linux and ~ 15 min on Windows to build XGBoost with GPU support.
• Try to get more funding, which is easier said than done.

Other CI improvements, outside of Jenkins

• Migrate from AppVeyor to GitHub actions. AppVeyor tests are often a bottleneck because it only runs one test at a time.
• Remove CPU-only tests from Jenkins CI pipeline. This is especially important for Windows targets, since Windows instances tend to cost more.

[terrytangyuan]

Thanks for surfacing and outlining the issues. Would a build be triggered by anyone that comments run tests or is it limited to committers only?

[hcho3]

@terrytangyuan I think only committers should have a right to start tests.

• We can migrate some basic checks off of the Jenkins server, so that they are always run. For example, lint checks should always run.
• Or we could let anyone start tests for the first time, and then subsequent tests would require a go-ahead from a committer.

[terrytangyuan]

The second option sounds better to me. Even though link checks are fast but if a contributor pushes many commits it could be a problem.

[hcho3]

Actually two options can go together. Lint checks can run in GitHub Actions and would not be subject to the EC2 quota. So we can let contributors run as many lint checks as they'd like. More substantial tests should require a committer's approval when they are run second time.

[terrytangyuan]

Got it. Yea all checks on GitHub Actions can always run.

[hcho3]

I raised the daily limit to 50 USD, to allow more jobs to run each day. We don't want to slow down our development speed too much. I'm really hoping to reduce the cost of each test job, so that we can bring the daily rate back down.

[hcho3]

Summary of expenses this year, per OS:

Platform	Windows (Amazon VPC)($)	Linux/UNIX($)	No Platform($)	Total cost ($)
Platform Total	7823.18	3397.2535665664	898.3373072795	12118.770873845900
2020-01-01	1145.258	439.38216705	185.986642061	1770.626809111
2020-02-01	1154.0820000000000	487.434750354	128.50315517710000	1770.0199055311
2020-03-01	1052.63	431.9893799469	134.3857972443	1619.0051771912
2020-04-01	1588.134	693.1112277207	130.5596068873	2411.804834608
2020-05-01	1122.826	468.5888252106	130.633854798	1722.0486800086
2020-06-01	1147.7060000000000	554.182259014	118.1775170761	1820.0657760901000
2020-07-01	612.544	322.5649572702	70.0907340357	1005.1996913059

Windows jobs cost whopping 64.5% of the total, and Linux cost only 28.0% of the total.

[hcho3]

@dmlc/xgboost-committer Good news: #5904 saves the cost of Windows test pipeline by up to 66%.

[hcho3]

It's been a week since we enacted a daily budget, and so far we've managed to rein in the cost:

[hcho3]

I just came up with a more robust method: Change the permissions of the Jenkins manager node. Ordinarily, it is given the right to launch new EC2 instances via an IAM policy. To restrict provision of new instances, it suffices to attach the following policy JSON:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:RunInstances",
                "ec2:StartInstances"
            ],
            "Effect": "Deny",
            "Resource": "*"
        }
    ]
}

This Deny policy will override the pre-existing permission and the manager would no longer be able to launch any new EC2 worker EC2 instances. (It can however terminate existing EC2 instances.)

Update. The Cost Watcher Lambda function now also controls whether the Jenkins manager (master) can launch EC2 workers or not: hcho3/xgboost-devops@e7402fe

[hcho3]

#5904 has been very effective in reducing the cost of the Windows CI pipeline:

[terrytangyuan]

Great work!

[hcho3]

There were two mistakes that slowed down GPU tests to ~ 40 minutes:

• Badly configured Docker caching, causing wasteful re-building of Docker containers. (A rebuild takes ~25 mins) Fixed by [CI] Use separate Docker cache for each CUDA version #6305.
• Resource contention caused by two test jobs running concurrently on a single GPU. Contention slowed down test_shap test to 11 min. Solved by limiting the number of concurrent jobs to 1 per GPU worker (Slow test: TestGPUPredict::test_shap took 11 min. to complete #6313). Now test_shap test takes < 20 sec. to complete

Now GPU test suite completes in 15 min.

[hcho3]

Completed in #8142. Now we require manual approval for running tests with pull requests