Skip to content

POC yarn-offline-mirror#21849

Closed
jbudz wants to merge 3 commits intoelastic:masterfrom
jbudz:offline-mirror
Closed

POC yarn-offline-mirror#21849
jbudz wants to merge 3 commits intoelastic:masterfrom
jbudz:offline-mirror

Conversation

@jbudz
Copy link
Copy Markdown
Contributor

@jbudz jbudz commented Aug 9, 2018

This uses the yarn-offline-mirror feature to cache our dependencies. Roughly, the setup involves a separate repo that has archives for all of our dependencies. That repo is cloned, and yarn installs using the packages in that folder.

There's several options for caching, including our own mirror, but I think this gives us the most flexibility. When we branch kibana to target a release we can update and branch the cache. We'll have a clear archive of dependencies needed to run a build for that branch. We can add tags if necessary for patch releases.

A cache server would need regular, long term backups to be consistent in addition to protecting the availability and integrity of the packages. I'd push for avoiding that complexity if we can.

Closes #15874

@elastic elastic deleted a comment from elasticmachine Aug 9, 2018
@elastic elastic deleted a comment from elasticmachine Aug 9, 2018
@elastic elastic deleted a comment from elasticmachine Aug 9, 2018
@elastic elastic deleted a comment from elasticmachine Aug 9, 2018
@jbudz jbudz added discuss Team:Operations Kibana-Operations Team labels Aug 9, 2018
@@ -0,0 +1 @@
yarn-offline-mirror "../kibana-extra/kibana-offline-mirror"
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably should replace this with a ci only config

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use yarn config set yarn-offline-mirror during the CI setup

}

checkout_sibling "elasticsearch" "${PARENT_DIR}/elasticsearch" "USE_EXISTING_ES"
checkout_sibling "kibana-offline-mirror" "${KIBANA_EXTRA_DIR}/kibana-offline-mirror" "USE_EXISTING_KIBANA_OFFLINE_MIRROR"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why put it in the kibana-extras directory and not a Kibana sibling, like the elasticsearch checkout?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no reason other than our previous convention. given the kibana prefix i think it makes sense to move it up one

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

const offline = options.offline === true;
const extraArgs = [
frozenLockfile ? '--frozen-lockfile' : '',
offline ? '--offline' : '',
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does --offline require that we update the cache repo with new dependencies before they will pass CI? Should we perhaps use --prefer-offline?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does, good call. do we use prefer-offline here, and keep it at offline for builds? the idea being that we do want to fail somewhere if we're not cached?

the more it i think about it -offline sounds like maintenance nightmare. what if we based it on the release flag?
--release ? '--offline' : '--prefer-offline'

@jbudz
Copy link
Copy Markdown
Contributor Author

jbudz commented Aug 10, 2018

More ideas:

  1. I had a thought the other day to throw all build dependencies that are static in the cache. it would help with Mirror phantomjs #18471, less fresh downloads, unique ip address less potential for failures. node, yarn, phantom, chromium, etc. postinstall scripts in yarn will still fetch things like geckodriver so i'm not sure on the approach for that.
  2. force commit signing https://help.github.com/articles/enabling-required-commit-signing/
  3. git-lfs https://git-lfs.github.com/

@jbudz jbudz mentioned this pull request Aug 20, 2018
@jbudz
Copy link
Copy Markdown
Contributor Author

jbudz commented Nov 19, 2018

#25840

@jbudz jbudz closed this Nov 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

discuss Team:Operations Kibana-Operations Team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants