Skip to content

[CI] Fix moon cache usage #259760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
delanni merged 10 commits into from
Mar 27, 2026
Merged

[CI] Fix moon cache usage #259760

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
f176b0d
try erasing prior cache before archiving
delanni Mar 26, 2026
ab4e988
further processing and investigation to moon cache
delanni Mar 26, 2026
dc3ba0b
retrieving moon cache from the image doesn't seem to make sense - let…
delanni Mar 26, 2026
43e1640
use self-referential deps to extract-version-dependencies
delanni Mar 26, 2026
6b96b04
separate extract version deps from webpack build step
delanni Mar 26, 2026
0908280
re-enable cache
delanni Mar 26, 2026
3a5c0c6
also, prevent caching dist plugins
delanni Mar 26, 2026
ed987dc
remove investigative logging
delanni Mar 26, 2026
feeef53
Merge branch 'main' into fix-moon-cache-usage
delanni Mar 27, 2026
614e6b4
re-instate remote cache
delanni Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .buildkite/scripts/bootstrap.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,6 @@ if [[ "$(pwd)" != *"/local-ssd/"* && "$(pwd)" != "/dev/shm"* ]]; then
mkdir -p ./.moon/cache
echo "Extracting moon-cache.tar.gz to ./.moon/cache"
tar -xzf ~/moon-cache.tar.gz -C ./
elif [[ -d ~/.kibana-moon-cache ]]; then
Copy link
Copy Markdown
Member Author

@delanni delanni Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was trying to retrieve a warmed moon cache from the VM image build.

It seems that cache cannot be reused, probably because some absolute paths are no longer valid on the executors compared to the VM image builder's context. Since it is not useful any longer, it just adds extra weight to the .moon/cache prior to archiving it per-build, that also could have been the reason for failing bootstraps

echo "Using ~/.moon/cache as a starting point"
mkdir -p ./.moon/cache
mv ~/.kibana-moon-cache/* ./.moon/cache
fi
fi

Expand Down
1 change: 0 additions & 1 deletion .buildkite/scripts/common/setup_job_env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,6 @@ if [[ "${CI:-}" =~ ^(1|true)$ ]]; then
MOON_REMOTE_CACHE_TOKEN=$(vault_get moon-remote-cache token)
export MOON_REMOTE_CACHE_TOKEN
fi
export MOON_CACHE=off

PIPELINE_PRE_COMMAND=${PIPELINE_PRE_COMMAND:-".buildkite/scripts/lifecycle/pipelines/$BUILDKITE_PIPELINE_SLUG/pre_command.sh"}
if [[ -f "$PIPELINE_PRE_COMMAND" ]]; then
Expand Down
1 change: 1 addition & 0 deletions .buildkite/scripts/steps/store_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ source .buildkite/scripts/common/util.sh

export MOON_CACHE=write
.buildkite/scripts/bootstrap.sh

echo "--- Archive moon cache"
if [[ ! -d .moon/cache ]]; then
echo "No moon cache directory found, skipping archive"
Expand Down
16 changes: 8 additions & 8 deletions .moon/workspace.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,14 @@ pipeline:

# https://moonrepo.dev/docs/guides/remote-cache
# Auth credentials will be missing on local environments, but that's OK.
# remote:
# host: "https://kibana-bazel-remote-h5qd3jkxkq-uc.a.run.app"
# api: http
# cache:
# localReadOnly: true
# auth:
# headers:
# Authorization: "Basic ${MOON_REMOTE_CACHE_TOKEN}"
remote:
host: "https://kibana-bazel-remote-h5qd3jkxkq-uc.a.run.app"
api: http
cache:
localReadOnly: true
auth:
headers:
Authorization: "Basic ${MOON_REMOTE_CACHE_TOKEN}"


telemetry: false
5 changes: 4 additions & 1 deletion packages/kbn-plugin-helpers/src/tasks/build_webpack_packages.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,10 @@ export async function buildWebpackPackages({ log, quiet, dist }: TaskContext) {

const args = ['kbn', 'build-shared'];
if (quiet) args.push('--quiet');
if (dist) args.push('--dist');
if (dist) {
Copy link
Copy Markdown
Member Author

@delanni delanni Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This avoids reusing a bad cache

args.push('--dist');
args.push('--no-cache');
}

await execa('yarn', args, { cwd: REPO_ROOT, stdio });
}
4 changes: 2 additions & 2 deletions src/dev/kbn_pm/src/commands/bootstrap/bootstrap_command.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,8 +112,8 @@ export const command = {

await Promise.all([
time('prepare webpack bundles for packages', async () => {
log.info('extract relevant versions for packages and pre-build webpack bundles');
await moonRun([':extract-version-dependencies', ':build-webpack'], {
log.info('pre-build webpack bundles');
await moonRun([':build-webpack'], {
Copy link
Copy Markdown
Member Author

@delanni delanni Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the extract-version-dependencies tasks are already marked as dependencies of the webpack build. I've experimented with this, and it seems we don't benefit from the cache until the 2nd rerun locally if we run both tasks like this, but we do if we just rely on :build-webpack pulling in it's dependencies

pipe: !quiet,
quiet,
noCache: forceInstall,
Expand Down
4 changes: 2 additions & 2 deletions src/platform/packages/private/kbn-ui-shared-deps-npm/moon.extend.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ tasks:
extract-version-dependencies:
command: node
args:
- $workspaceRoot/scripts/extract_version_dependencies.js
- scripts/extract_version_dependencies.js
- src/platform/packages/private/kbn-ui-shared-deps-npm/version_dependencies.txt
- --collect
- '@babel/runtime'
Expand Down Expand Up @@ -110,7 +110,7 @@ tasks:
args:
- build
deps:
- extract-version-dependencies
- ~:extract-version-dependencies
- '^:build-webpack'
inputs:
- '@group(src)'
Expand Down
4 changes: 2 additions & 2 deletions src/platform/packages/private/kbn-ui-shared-deps-npm/moon.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ tasks:
extract-version-dependencies:
command: node
args:
- $workspaceRoot/scripts/extract_version_dependencies.js
- scripts/extract_version_dependencies.js
- src/platform/packages/private/kbn-ui-shared-deps-npm/version_dependencies.txt
- '--collect'
- '@babel/runtime'
Expand Down Expand Up @@ -139,7 +139,7 @@ tasks:
args:
- build
deps:
- extract-version-dependencies
- ~:extract-version-dependencies
- ^:build-webpack
inputs:
- '@group(src)'
Expand Down
4 changes: 2 additions & 2 deletions src/platform/packages/private/kbn-ui-shared-deps-src/moon.extend.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ tasks:
extract-version-dependencies:
command: node
args:
- $workspaceRoot/scripts/extract_version_dependencies.js
- scripts/extract_version_dependencies.js
- src/platform/packages/private/kbn-ui-shared-deps-src/version_dependencies.txt
- --collect
- '@babel/plugin-transform-numeric-separator'
Expand Down Expand Up @@ -46,7 +46,7 @@ tasks:
args:
- build
deps:
- extract-version-dependencies
- ~:extract-version-dependencies
- '^:build-webpack'
inputs:
- '@group(src)'
Expand Down
4 changes: 2 additions & 2 deletions src/platform/packages/private/kbn-ui-shared-deps-src/moon.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ tasks:
extract-version-dependencies:
command: node
args:
- $workspaceRoot/scripts/extract_version_dependencies.js
- scripts/extract_version_dependencies.js
- src/platform/packages/private/kbn-ui-shared-deps-src/version_dependencies.txt
- '--collect'
- '@babel/plugin-transform-numeric-separator'
Expand Down Expand Up @@ -75,7 +75,7 @@ tasks:
args:
- build
deps:
- extract-version-dependencies
- ~:extract-version-dependencies
- ^:build-webpack
inputs:
- '@group(src)'
Expand Down
4 changes: 2 additions & 2 deletions src/platform/packages/shared/kbn-monaco/moon.extend.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ tasks:
extract-version-dependencies:
command: node
args:
- $workspaceRoot/scripts/extract_version_dependencies.js
- scripts/extract_version_dependencies.js
- src/platform/packages/shared/kbn-monaco/version_dependencies.txt
- --collect
- babel-loader
Expand All @@ -34,7 +34,7 @@ tasks:
args:
- build
deps:
- extract-version-dependencies
- ~:extract-version-dependencies
inputs:
- '@group(src)'
- package.json
Expand Down
4 changes: 2 additions & 2 deletions src/platform/packages/shared/kbn-monaco/moon.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ tasks:
extract-version-dependencies:
command: node
args:
- $workspaceRoot/scripts/extract_version_dependencies.js
- scripts/extract_version_dependencies.js
- src/platform/packages/shared/kbn-monaco/version_dependencies.txt
- '--collect'
- babel-loader
Expand All @@ -89,7 +89,7 @@ tasks:
args:
- build
deps:
- extract-version-dependencies
- ~:extract-version-dependencies
inputs:
- '@group(src)'
- package.json
Expand Down
Loading