Skip to content

POC yarn-offline-mirror #21849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jbudz wants to merge 3 commits into from
Closed

POC yarn-offline-mirror #21849

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2803127
yarnrc
jbudz Aug 9, 2018
7ae0b46
add yarn-offline-mirror
jbudz Aug 9, 2018
7c0c977
readd cacheDir
jbudz Aug 9, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .yarnrc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
yarn-offline-mirror "../kibana-extra/kibana-offline-mirror"
Copy link
Copy Markdown
Contributor Author

@jbudz jbudz Aug 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably should replace this with a ci only config

Copy link
Copy Markdown
Member

@tylersmalley tylersmalley Aug 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use yarn config set yarn-offline-mirror during the CI setup

9 changes: 5 additions & 4 deletions packages/kbn-pm/dist/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26150,6 +26150,9 @@ var subscribeTo = function (result) {
}
};
}
else if (result && typeof result[__WEBPACK_IMPORTED_MODULE_9__symbol_observable__["a" /* observable */]] === 'function') {
return Object(__WEBPACK_IMPORTED_MODULE_4__subscribeToObservable__["a" /* subscribeToObservable */])(result);
}
else if (Object(__WEBPACK_IMPORTED_MODULE_5__isArrayLike__["a" /* isArrayLike */])(result)) {
return Object(__WEBPACK_IMPORTED_MODULE_1__subscribeToArray__["a" /* subscribeToArray */])(result);
}
Expand All @@ -26159,9 +26162,6 @@ var subscribeTo = function (result) {
else if (result && typeof result[__WEBPACK_IMPORTED_MODULE_8__symbol_iterator__["a" /* iterator */]] === 'function') {
return Object(__WEBPACK_IMPORTED_MODULE_3__subscribeToIterable__["a" /* subscribeToIterable */])(result);
}
else if (result && typeof result[__WEBPACK_IMPORTED_MODULE_9__symbol_observable__["a" /* observable */]] === 'function') {
return Object(__WEBPACK_IMPORTED_MODULE_4__subscribeToObservable__["a" /* subscribeToObservable */])(result);
}
else {
var value = Object(__WEBPACK_IMPORTED_MODULE_7__isObject__["a" /* isObject */])(result) ? 'an invalid object' : "'" + result + "'";
var msg = "You provided " + value + " where a stream was expected."
Expand Down Expand Up @@ -28354,7 +28354,8 @@ const BootstrapCommand = exports.BootstrapCommand = {
return _asyncToGenerator(function* () {
const batchedProjects = (0, _projects.topologicallyBatchProjects)(projects, projectGraph);
const frozenLockfile = options['frozen-lockfile'] === true;
const extraArgs = frozenLockfile ? ['--frozen-lockfile'] : [];
const offline = options.offline === true;
const extraArgs = [frozenLockfile ? '--frozen-lockfile' : '', offline ? '--offline' : ''].filter(Boolean);
_log.log.write(_chalk2.default.bold('\nRunning installs in topological order:'));
for (const batch of batchedProjects) {
for (const project of batch) {
Expand Down
6 changes: 5 additions & 1 deletion packages/kbn-pm/src/commands/bootstrap.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,11 @@ export const BootstrapCommand: ICommand = {
const batchedProjects = topologicallyBatchProjects(projects, projectGraph);

const frozenLockfile = options['frozen-lockfile'] === true;
const extraArgs = frozenLockfile ? ['--frozen-lockfile'] : [];
const offline = options.offline === true;
const extraArgs = [
frozenLockfile ? '--frozen-lockfile' : '',
offline ? '--offline' : '',
Copy link
Copy Markdown
Contributor

@spalger spalger Aug 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does --offline require that we update the cache repo with new dependencies before they will pass CI? Should we perhaps use --prefer-offline?

Copy link
Copy Markdown
Contributor Author

@jbudz jbudz Aug 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does, good call. do we use prefer-offline here, and keep it at offline for builds? the idea being that we do want to fail somewhere if we're not cached?

the more it i think about it -offline sounds like maintenance nightmare. what if we based it on the release flag?
--release ? '--offline' : '--prefer-offline'

].filter(Boolean);

log.write(chalk.bold('\nRunning installs in topological order:'));

Expand Down
2 changes: 1 addition & 1 deletion src/dev/build/tasks/bootstrap_task.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,6 @@ export const BootstrapTask = {
description: 'Running `yarn kbn bootstrap` to make sure all dependencies are up-to-date',

async run(config, log) {
await exec(log, 'yarn', ['kbn', 'bootstrap', '--skip-kibana-extra']);
await exec(log, 'yarn', ['kbn', 'bootstrap', '--skip-kibana-extra', '--offline']);
},
};
2 changes: 1 addition & 1 deletion src/dev/build/tasks/install_dependencies_task.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ export const InstallDependenciesTask = {

// We're using --no-bin-links to support systems that don't have symlinks.
// This is commonly seen in shared folders on virtual machines
const args = ['--production', '--ignore-optional', '--pure-lockfile', '--no-bin-links'];
const args = ['--production', '--ignore-optional', '--pure-lockfile', '--no-bin-links', '--offline'];

await exec(log, 'yarn', args, {
cwd: build.resolvePath(),
Expand Down
4 changes: 3 additions & 1 deletion src/dev/ci_setup/git_setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export KIBANA_DIR="$(cd "$(dirname "$SCRIPT")/../../.."; pwd)"

# PARENT_DIR is the directory where we will checkout es and x-pack-es
export PARENT_DIR="$(cd "$KIBANA_DIR"/..; pwd)"
export KIBANA_EXTRA_DIR="$PARENT_DIR"/"kibana-extra"

function checkout_sibling {
project=$1
Expand All @@ -31,7 +32,7 @@ function checkout_sibling {
useExisting="$(eval "echo "\$$useExistingParamName"")"

if [ -z ${useExisting:+x} ]; then
if [ -d "$targetDir" ]; then
if [ -d "$targetDir" ] && [ -n "$(ls "$targetDir")" ]; then
echo "I expected a clean workspace but an '${project}' sibling directory already exists in [$PARENT_DIR]!"
echo
echo "Either define '${useExistingParamName}' or remove the existing '${project}' sibling."
Expand Down Expand Up @@ -99,3 +100,4 @@ function checkout_sibling {
}

checkout_sibling "elasticsearch" "${PARENT_DIR}/elasticsearch" "USE_EXISTING_ES"
checkout_sibling "kibana-offline-mirror" "${KIBANA_EXTRA_DIR}/kibana-offline-mirror" "USE_EXISTING_KIBANA_OFFLINE_MIRROR"
Copy link
Copy Markdown
Member

@tylersmalley tylersmalley Aug 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why put it in the kibana-extras directory and not a Kibana sibling, like the elasticsearch checkout?

Copy link
Copy Markdown
Contributor Author

@jbudz jbudz Aug 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no reason other than our previous convention. given the kibana prefix i think it makes sense to move it up one

17 changes: 17 additions & 0 deletions src/dev/ci_setup/kibana_setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
###
### install dependencies
###
echo " -- installing node.js dependencies"
yarn kbn bootstrap --offline


###
### verify no git modifications
###

GIT_CHANGES="$(git ls-files --modified)"
if [ "$GIT_CHANGES" ]; then
echo -e "\n${RED}ERROR: 'yarn kbn bootstrap' caused changes to the following files:${C_RESET}\n"
echo -e "$GIT_CHANGES\n"
exit 1
fi
19 changes: 1 addition & 18 deletions src/dev/ci_setup/setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,22 +89,5 @@ yarnGlobalDir="$(yarn global bin)"
export PATH="$PATH:$yarnGlobalDir"
hash -r


###
### install dependencies
###
echo " -- installing node.js dependencies"
### configure
yarn config set cache-folder "$cacheDir/yarn"
yarn kbn bootstrap


###
### verify no git modifications
###

GIT_CHANGES="$(git ls-files --modified)"
if [ "$GIT_CHANGES" ]; then
echo -e "\n${RED}ERROR: 'yarn kbn bootstrap' caused changes to the following files:${C_RESET}\n"
echo -e "$GIT_CHANGES\n"
exit 1
fi
1 change: 1 addition & 0 deletions test/scripts/jenkins_selenium.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ set -e
source "$(dirname $0)/../../src/dev/ci_setup/setup.sh"
source "$(dirname $0)/../../src/dev/ci_setup/git_setup.sh"
source "$(dirname $0)/../../src/dev/ci_setup/java_setup.sh"
source "$(dirname $0)/../../src/dev/ci_setup/kibana_setup.sh"

node scripts/build --release --debug --oss;

Expand Down
1 change: 1 addition & 0 deletions test/scripts/jenkins_unit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ set -e
source "$(dirname $0)/../../src/dev/ci_setup/setup.sh"
source "$(dirname $0)/../../src/dev/ci_setup/git_setup.sh"
source "$(dirname $0)/../../src/dev/ci_setup/java_setup.sh"
source "$(dirname $0)/../../src/dev/ci_setup/kibana_setup.sh"

export TEST_ES_FROM=source
xvfb-run "$(FORCE_COLOR=0 yarn bin)/grunt" jenkins:unit --from=source;
2 changes: 1 addition & 1 deletion test/scripts/jenkins_xpack.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ set -o pipefail
source "$(dirname $0)/../../src/dev/ci_setup/setup.sh"
source "$(dirname $0)/../../src/dev/ci_setup/git_setup.sh"
source "$(dirname $0)/../../src/dev/ci_setup/java_setup.sh"

source "$(dirname $0)/../../src/dev/ci_setup/kibana_setup.sh"

export XPACK_DIR="$(cd "$(dirname "$0")/../../x-pack"; pwd)"
echo "-> XPACK_DIR ${XPACK_DIR}"
Expand Down