Performance measuring utilities for DefinitelyTyped.
This is not currently designed for public consumption, so docs, tests, and usage help are very light. The utility is used only in Azure Pipelines for DefinitelyTyped and TypeScript. There are three main functions that run in CI:
- Every night, see what packages on DefinitelyTyped haven’t been benchmarked with the current version of TypeScript since their last change, and benchmark them. The CLI entrypoint
getPackagesToBenchmark
queries the Cosmos DB instance to see what packages need to be benchmarked, then generates a file specifying what to run. In another phase, a large number of build agents read that file with thebenchmark
entrypoint and benchmark the packages in parallel. After a new TypeScript release, it takes several weeks of running this nightly to cover all of DefinitelyTyped. - On every PR to DefinitelyTyped, pull the benchmarks for changed packages. If they haven’t been run or aren’t up-to-date, run them on
master
. Then, benchmark the PR, and compare. Post the summary of the results in a comment on the PR. This is done with thecompare
CLI entrypoint. - Every night, find existing benchmarks that have been run with the latest TypeScript version, run them with
typescript@master
, and compare. This is done with thecompareTypeScript
CLI entrypoint. Results are uploaded to thetypeScriptComparisons
collection in the database.
The package registers a bin
called definitely-not-slow
upon installation:
npx definitely-not-slow compare --package react/v16 --upload false
This utility isn’t published to npm; rather, a GitHub workflow produces branches that include the compiled code:
npm install github:andrewbranch/definitely-not-slow#production # Published on every tag
npm install github:andrewbranch/definitely-not-slow#beta # Published on every push to master
It can be useful to queue a Pipelines build manually against a real DefinitelyTyped PR with the variables COMMENT=false
and DNS_VERSION=beta
(definitely-not-slow—bad acronym, sorry) to test changes to definitely-not-slow on real code.