Skip to content

[build] Upload CDN assets#173159

Merged
jbudz merged 9 commits intoelastic:mainfrom
jbudz:cdn-upload
Jan 4, 2024
Merged

[build] Upload CDN assets#173159
jbudz merged 9 commits intoelastic:mainfrom
jbudz:cdn-upload

Conversation

@jbudz
Copy link
Contributor

@jbudz jbudz commented Dec 12, 2023

This uploads CDN assets to a GCS bucket on commit and after all tests have passed. This will run on pull requests with ci:project-deploy-* and ci:build-serverless-image labels, and on main. Assets will include the first 12 digits of the commit sha as a base path.

@jbudz jbudz added Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// Team:Operations Kibana-Operations Team release_note:skip Skip the PR/issue when compiling release notes labels Dec 12, 2023
@jbudz jbudz requested a review from a team as a code owner December 12, 2023 13:55
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@jbudz jbudz requested a review from a team December 12, 2023 13:56
Copy link
Contributor

@TinaHeiligers TinaHeiligers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM although I'm relying on CI's approval here

@jbudz
Copy link
Contributor Author

jbudz commented Dec 13, 2023

buildkite test this

@jbudz
Copy link
Contributor Author

jbudz commented Dec 13, 2023

@rudolf @maneta are we ready to proceed?

CDN_ASSETS_FOLDER=$(mktemp -d)
tar -xf "kibana-$BASE_VERSION-cdn-assets.tar.gz" -C "$CDN_ASSETS_FOLDER" --strip=1

gsutil -m cp -r "$CDN_ASSETS_FOLDER/*" "gs://$GCS_SA_CDN_QA_BUCKET/$GIT_ABBREV_COMMIT"
Copy link
Contributor

@rudolf rudolf Dec 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would we construct this path to point to the static assets from index.html? We already use a build number https://github.com/elastic/kibana/blob/main/packages/core/rendering/core-rendering-server-internal/src/bootstrap/bootstrap_renderer.ts#L82

Is this GIT_ABBREV_COMMIT hash included in any of the RawPackageInfo fields https://github.com/elastic/kibana/blob/main/packages/kbn-config/src/env.ts#L45-L48

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build number doesn't contain all assets and in a branching scenario can cause conflicts.

% tree -L 3
.
├── 70129
│   └── bundles
│       ├── core
│       ├── kbn-monaco
│       ├── kbn-ui-shared-deps-npm
│       ├── kbn-ui-shared-deps-src
│       └── plugin
└── ui
    ├── favicons
    │   ├── favicon.distribution.ico
    │   ├── favicon.distribution.png
    │   ├── favicon.distribution.svg
    │   ├── favicon.ico
    │   ├── favicon.png
    │   └── favicon.svg
    ├── fonts
    │   ├── inter
    │   └── roboto_mono
    ├── legacy_dark_theme.css
    ├── legacy_dark_theme.min.css
    ├── legacy_light_theme.css
    └── legacy_light_theme.min.css

13 directories, 10 files

Is this GIT_ABBREV_COMMIT hash included in any of the RawPackageInfo fields https://github.com/elastic/kibana/blob/main/packages/kbn-config/src/env.ts#L45-L48

It is, build.sha{0,12}. It's also available to the controller as the id used to reference a build.

Copy link
Contributor

@Ikuni17 Ikuni17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add the label(s) to upload the assets?

Copy link
Contributor

@rudolf rudolf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"deploying" assets from main is useful for testing but could cause the bucket to fill up. We'd want to just check in a month or so how much storage space this consumes and perhaps add some kind of clean up.

@jbudz jbudz added the ci:project-deploy-elasticsearch Create an Elasticsearch Serverless project label Jan 3, 2024
@jbudz
Copy link
Contributor Author

jbudz commented Jan 4, 2024

Can we add the label(s) to upload the assets?

Tested at https://buildkite.com/elastic/kibana-pull-request/builds/185740#018cd137-3289-4436-b145-a76601626d6a. The build failed on the project deploy step, but it's unrelated - we're not using the CDN yet. I'm looking into it on another thread.

An example asset is https://kibana-cdn.gcp.qa.cld.elstc.co/db0dcc65ad07/70433/bundles/plugin/infra/1.0.0/4bc2b11b721c9874ba512ea725c670a6.png

@jbudz jbudz removed the ci:project-deploy-elasticsearch Create an Elasticsearch Serverless project label Jan 4, 2024
@kibana-ci
Copy link

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jbudz jbudz merged commit 615e3f5 into elastic:main Jan 4, 2024
@kibanamachine kibanamachine added the backport:skip This PR does not require backporting label Jan 4, 2024
delanni pushed a commit that referenced this pull request Jan 8, 2024
In #173159 we authenticated with
another service account, and were no longer operating under the expected
config. This was causing `gcloud secrets` to access the wrong project
and throw errors.

This revokes the service account after we're done uploading CDN assets
so we can switch back to the default service account.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
delanni pushed a commit to delanni/kibana that referenced this pull request Jan 11, 2024
In elastic#173159 we authenticated with
another service account, and were no longer operating under the expected
config. This was causing `gcloud secrets` to access the wrong project
and throw errors.

This revokes the service account after we're done uploading CDN assets
so we can switch back to the default service account.

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// Team:Operations Kibana-Operations Team v8.13.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants