[build_ts_refs] improve caches, allow building a subset of projects#107981
Merged
spalger merged 9 commits intoelastic:masterfrom Aug 11, 2021
Merged
Conversation
…entionally-expire-extracted-caches
spalger
added
release_note:skip
Contributor
|
Pinging @elastic/kibana-operations (Team:Operations) |
Contributor
Author
|
Sorry for the massive number of teams pinged, this PR edits a ton of TS config files so I'm not going to wait for all reviewers as we know these updates are going to work. |
Contributor
💚 Build Succeeded
Metrics [docs]Async chunks
Page load bundle
History
To update your PR or re-run it, just comment with: |
ppisljar
approved these changes
Aug 11, 2021
Contributor
ppisljar
left a comment
ppisljar
left a comment
There was a problem hiding this comment.
appservices changes lgtm
botelastic
bot
added
Feature:Drilldowns
Contributor
|
Pinging @elastic/apm-ui (Team:apm) |
Contributor
|
Pinging @elastic/fleet (Team:Fleet) |
tylersmalley
approved these changes
Aug 11, 2021
spalger
added a commit
to spalger/kibana
that referenced
this pull request
Aug 11, 2021
…lastic#107981) * [build_ts_refs] improve caches, allow building a subset of projects * cleanup project def script and update refs in type check script * rename browser_bazel config to avoid kebab-case * remove execInProjects() helper * list references for tsconfig.types.json for api-extractor workload * disable composite features of tsconfig.types.json for api-extractor * set declaration: true to avoid weird debug error * fix jest tests Co-authored-by: spalger <spalger@users.noreply.github.com> # Conflicts: # .gitignore # examples/hello_world/tsconfig.json # tsconfig.json # tsconfig.refs.json
spalger
pushed a commit
that referenced
this pull request
Aug 11, 2021
…107981) (#108139) * [build_ts_refs] improve caches, allow building a subset of projects * cleanup project def script and update refs in type check script * rename browser_bazel config to avoid kebab-case * remove execInProjects() helper * list references for tsconfig.types.json for api-extractor workload * disable composite features of tsconfig.types.json for api-extractor * set declaration: true to avoid weird debug error * fix jest tests Co-authored-by: spalger <spalger@users.noreply.github.com> # Conflicts: # .gitignore # examples/hello_world/tsconfig.json # tsconfig.json # tsconfig.refs.json Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Aug 11, 2021
…-png-pdf-report-type * 'master' of github.com:elastic/kibana: (101 commits) [ML] APM Latency Correlations: Field/value candidates prioritization (elastic#107370) [Reporting] Add lenience to a test on the order of asserted logs (elastic#108135) [Lens] fix do not submit invalid query in filtered metric (elastic#107542) skip flaky test (elastic#108043) fix newly introduced type error (elastic#107593) [Reporting] server side code clean up (elastic#106940) [build_ts_refs] improve caches, allow building a subset of projects (elastic#107981) [APM] Add new ftr_e2e to kibana CI and remove current e2e tests. (elastic#107593) add manage rules link to alerts dropdown (elastic#107950) [ML] Enable Index data visualizer document count chart to update time range query (elastic#106438) [Security Solutions][Detection Engine] Fixes "undefined" crash for author field by adding a migration for it (elastic#107230) [Actions UI] Fixed Jira Api token label. (elastic#107776) [Alerting UI] Fixed display permissions for edit/delete buttons when user has read only access. (elastic#107996) [Maps] fix code owners (elastic#108106) Update EMS landing page url (elastic#108102) Do not render page header for loading domains (elastic#108078) Update dependency @elastic/charts to v33.2.2 (elastic#107939) [APM] Display throughput as tps (instead of tpm) when bucket size < 60 seconds (elastic#107850) [Fleet] Fix all category count (elastic#108089) [Security Solution][Bug] - Disable alert table RBAC until fields sorted (elastic#108034) ... # Conflicts: # x-pack/plugins/reporting/server/export_types/common/generate_png.ts # x-pack/plugins/reporting/server/lib/screenshots/index.ts # x-pack/plugins/reporting/server/lib/screenshots/observable.test.ts # x-pack/plugins/reporting/server/lib/screenshots/observable.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
node scripts/build_ts_refshas a major flaw in the way it handles caches: we extract caches to the output directory whenever we're building on a new mergeBase and in doing so set the modified times of the output files to the moment the cache is extracted, buttscwill only build types for a project if the output files are older than the input files.To fix this we need to make sure that the output files do not have newer timestamps than the input files, as long as the input files have changes since the cache was created. To accomplish this we use
git diff --name-only {sha-cache-was-made-on}to get the list of files which have updated, map those to TS projects, and then set the modified time of the files from those caches tonew Date(0). Doing this forces TS to use the.tsbuildinfofile to determine which files have changed and how to update the types, which is a little more than twice as fast as running a fresh build without caches in my limited testing.This PR also migrates all TS projects to be "composite" and "incremental", which means that all references outside of the project are pointing to another composite project, which builds its own types, and build info is written to a
.tsbuildinfofile to support incrementally updating types based on specific file changes. This structure was implemented in most of the TS projects in the repo but not completed until this PR. Now any project that isn't built by bazel extendstsconfig.base.jsonand is composite+incremental. Any project that is built in bazel uses thetsconfig.bazel.jsonbase (or thetsconfig.browser_bazel.jsonvariant) and is not composite or incremental.This PR integrates #107804 without requiring users to commit the updated file, and instead allows users to put
// SELF MANAGEDat the top of the file to control which projects we build types for withscripts/build_ts_refs. Some teams have had success limiting the project scope, and it seems like we should be able to support that usage pattern better this way.Finally, this PR reworks
scripts/type_checkto only check non-composite projects asbuild_ts_refsis executed before it runs and validates the types of all composite projects, sotype_checkonly needs to check the non-composite projects. Also updated the script to not uselistrso that it renders better.