Add grunt-task to build ES from source #8866

thomasneirynck · 2016-10-27T22:32:07Z

Build ES from source in Kibana. This requires ES and Kibana repositories to be on the same level in the directory structure.

Run the grunt-task with: $> npm run es-start

The task ensures Kibana and Elasticsearch share the same major version.

epixa · 2016-10-27T22:48:37Z

tasks/es-start.js

+ *
+ * @param grunt
+ */
+module.exports = function (grunt) {


export default

w33ble

Some small changes to the commands used

w33ble · 2016-10-27T22:49:43Z

tasks/es-start.js

+        throw new Error('Could not build ElasticSearch');
+      }
+
+      const distrib = await runCommand(`find ${esRepo}distribution/tar/build/distributions/elasticsearch*.tar.gz`);


Should probably use ${esDest} here

w33ble · 2016-10-27T22:52:22Z

tasks/es-start.js

+
+      console.log('Building Elasticsearch. This may take a while if this is the first time.');
+      try {
+        await runCommand('cd ../elasticsearch && gradle assemble -x test');


I think you want gradle clean assemble here, which I believe will clean up old builds that may exist there from past build of previous snapshot versions.

If we only plan to build and run we can also use gradle run

tsullivan · 2016-10-27T22:54:06Z

tasks/es-start.js

+
+      console.log('Building Elasticsearch. This may take a while if this is the first time.');
+      try {
+        await runCommand('cd ../elasticsearch && gradle assemble -x test');


This command will build the docs as well, which takes a long time and isn't really needed. It will also build the zip distribution, which isn't used in this scirpt. Maybe change to:

gradle clean :distribution:tar:assemble

That will skip running the tests as well, which it looks like you want to do

w33ble · 2016-10-27T22:54:49Z

The way this is written, it won't work on Windows (no find or tar commands). It can be updated to work in windows as well, most likely by using node for those operations instead. I'm not sure how gradle works there though...

w33ble · 2016-10-27T23:02:19Z

If I'm not mistaken, this will rebuild ES fresh every time, which takes a while, and will get annoying for development use if that's the intended use for this. If would be much better if it would check for an existing build and you could use a flag to force it to do a rebuild for you.

Also, it may be nice to check the version file in the repo you try building from to make sure it matches what Kibana wants, otherwise if the repo is a bit stale, the user could end up waiting for a build that won't even work.

tsullivan · 2016-10-27T23:05:51Z

If would be much better if it would check for an existing build and you could use a flag to force it to do a rebuild for you.

I would recommend copying the build artifact to the root level of the elasticsearch repo before untarring it, and then in your check on whether to do a build, look for the copied artifact.

tsullivan · 2016-10-27T23:06:49Z

tasks/es-start.js

+        const message = `Kibana and Elasticsearch must be on
+         the same major branch: Kibana=${kibanaMajor}, Elasticsearch=${elasticsearchMajor}`;
+        throw new Error(message);
+      }


Thoughts on whether it should also do a git pull from the upstream repo?

thomasneirynck · 2016-10-28T03:47:18Z

A few quick initial comments already:

Windows wasn't on my radar for now. At this point, we're not wholly sure if this workflow is desirable, so I was looking to get a MVP in first that we could use and see how it goes for a while. But point taken, it would benefit form being generically usable, not just on a unix.

I would not do a pull automatically. This way, users have the choice which commit from ES to run against. I would only enforce a match between major versions as there's no reasonable expectation of compatibility there.

I think gradle should be intelligent enough not to do a rebuild when none of the source has changed. So there is not really a danger of having to recompile each time (not a gradle expert, but that's my experience with maven/ant at least). I'll double check ...

w33ble · 2016-10-28T04:40:13Z

I think gradle should be intelligent enough not to do a rebuild when none of the source has changed.

If that's how it works, fine. I don't know if it does, never tried building more than once on the same commit.

I would not do a pull automatically.

+1, you may want to run a specific version, or lag behind master a bit, for obvious reasons. Just checking the version in the repo/build should be sufficient. Maybe showing the build date on startup if not doing a new build would also be helpful, if that's not hard to add. That could easily be something to put in later too.

epixa · 2016-10-28T14:19:33Z

We definitely should add windows support, and let's remove the "experimental" notice. We either do this in a way that we're happy with or we don't do it at all. If any just wants to experiment with stuff, then feel free to build the automation into a local node script and use it for awhile before doing a PR.

w33ble · 2016-11-16T23:26:58Z

@thomasneirynck what's the status of this? Still something you're planning to take a crack at?

thomasneirynck · 2016-11-16T23:35:36Z

@w33ble I haven't looked at this in a few weeks. This is probably not something I'll look at in the next few weeks either. I still like the idea though, but it needs TLC.

thomasneirynck · 2017-02-27T19:37:10Z

Closing. Little need to continue this, the gradle-tasks in ES make it fairly straightforward to run ES/X-pack from source.

`@elastic/eui`: `106.0.0` ⏩ `106.1.0` `@elastic/eslint-plugin-eui`: `2.1.0` ⏩ `2.2.1` [Questions? Please see our Kibana upgrade FAQ.](https://github.com/elastic/eui/blob/main/wiki/eui-team-processes/upgrading-kibana.md#faq-for-kibana-teams) New props and defaults have been added to `EuiFlyout`, including `includeSelectorInFocusTrap`, plus new global CSS variables for push mode offset control. Additionally, we introduced a small spacing fix for invalid `EuiDualRange` append component and added support for custom styles on `EuiTreeView` nodes via the css key. :warning: **React 16 is now deprecated:** EUI no longer tests against it or fixes issues specific to it. ## Changes - I removed forced click in: `x-pack/test/security_solution_cypress/cypress/tasks/rule_details.ts` - I removed `{enter}` in: `x-pack/test/security_solution_cypress/cypress/tasks/common/rule_actions.ts` (see #228517) ## Package updates ### `@elastic/eui` #### [`v106.1.0`](https://github.com/elastic/eui/releases/v106.1.0) - Added `--euiPushFlyoutOffsetInlineStart` and `--euiPushFlyoutOffsetInlineEnd` global CSS variables set by the `EuiFlyout` in `push` mode. ([#8872](elastic/eui#8872)) - Reduced the `min-width` for inputs in `EuiRange` and `EuiDualRange` ([#8866](elastic/eui#8866)) - Added `includeSelectorInFocusTrap` prop for `EuiFlyout` ([#8849](elastic/eui#8849)) - Added component defaults for `EuiFlyout` that include `includeSelectorInFocusTrap` and `includeFixedHeadersInFocusTrap` ([#8849](elastic/eui#8849)) **Bug fixes** - Fixed flaky manual return focus behavior on `EuiFlyout` by relying on `FocusTrap` returning focus instead ([#8878](elastic/eui#8878)) - Fixed `EuiFlyoutChild` closing its parent `EuiFlyout` on `Escape` keypress ([#8878](elastic/eui#8878)) - Fixed support for `css` key in items object passed to `EuiTreeView` ([#8864](elastic/eui#8864)) **Deprecations** - Deprecated support for React 16. Please update to a modern version of React or stay on EUI version 106.0.0 if you can't switch right now. ([#8868](elastic/eui#8868)) ### `@elastic/eslint-plugin-eui` #### [`v2.2.1`](https://www.npmjs.com/package/@elastic/eslint-plugin-eui/v/2.2.1) **Bug fixes** - Removed `no-css_color` entry which mapped to a duplicate `no-css-color` rule ([#8888](elastic/eui#8888)) #### [`v2.2.0`](https://www.npmjs.com/package/@elastic/eslint-plugin-eui/v/2.2.0) - Added new `prefer-eui-icon-tip` rule. ([#8877](elastic/eui#8877))

`@elastic/eui`: `106.0.0` ⏩ `106.1.0` `@elastic/eslint-plugin-eui`: `2.1.0` ⏩ `2.2.1` [Questions? Please see our Kibana upgrade FAQ.](https://github.com/elastic/eui/blob/main/wiki/eui-team-processes/upgrading-kibana.md#faq-for-kibana-teams) New props and defaults have been added to `EuiFlyout`, including `includeSelectorInFocusTrap`, plus new global CSS variables for push mode offset control. Additionally, we introduced a small spacing fix for invalid `EuiDualRange` append component and added support for custom styles on `EuiTreeView` nodes via the css key. :warning: **React 16 is now deprecated:** EUI no longer tests against it or fixes issues specific to it. ## Changes - I removed forced click in: `x-pack/test/security_solution_cypress/cypress/tasks/rule_details.ts` - I removed `{enter}` in: `x-pack/test/security_solution_cypress/cypress/tasks/common/rule_actions.ts` (see elastic#228517) ## Package updates ### `@elastic/eui` #### [`v106.1.0`](https://github.com/elastic/eui/releases/v106.1.0) - Added `--euiPushFlyoutOffsetInlineStart` and `--euiPushFlyoutOffsetInlineEnd` global CSS variables set by the `EuiFlyout` in `push` mode. ([elastic#8872](elastic/eui#8872)) - Reduced the `min-width` for inputs in `EuiRange` and `EuiDualRange` ([elastic#8866](elastic/eui#8866)) - Added `includeSelectorInFocusTrap` prop for `EuiFlyout` ([elastic#8849](elastic/eui#8849)) - Added component defaults for `EuiFlyout` that include `includeSelectorInFocusTrap` and `includeFixedHeadersInFocusTrap` ([elastic#8849](elastic/eui#8849)) **Bug fixes** - Fixed flaky manual return focus behavior on `EuiFlyout` by relying on `FocusTrap` returning focus instead ([elastic#8878](elastic/eui#8878)) - Fixed `EuiFlyoutChild` closing its parent `EuiFlyout` on `Escape` keypress ([elastic#8878](elastic/eui#8878)) - Fixed support for `css` key in items object passed to `EuiTreeView` ([elastic#8864](elastic/eui#8864)) **Deprecations** - Deprecated support for React 16. Please update to a modern version of React or stay on EUI version 106.0.0 if you can't switch right now. ([elastic#8868](elastic/eui#8868)) ### `@elastic/eslint-plugin-eui` #### [`v2.2.1`](https://www.npmjs.com/package/@elastic/eslint-plugin-eui/v/2.2.1) **Bug fixes** - Removed `no-css_color` entry which mapped to a duplicate `no-css-color` rule ([elastic#8888](elastic/eui#8888)) #### [`v2.2.0`](https://www.npmjs.com/package/@elastic/eslint-plugin-eui/v/2.2.0) - Added new `prefer-eui-icon-tip` rule. ([elastic#8877](elastic/eui#8877))

add grunt-task to build ES from source

0d435aa

thomasneirynck added dev review labels Oct 27, 2016

remove crufty dependency

4a7c60e

thomasneirynck added v5.1.0 v6.0.0 and removed dev labels Oct 27, 2016

epixa reviewed Oct 27, 2016

View reviewed changes

tasks/es-start.js

*

* @param grunt

*/

module.exports = function (grunt) {

Copy link

Contributor

epixa Oct 27, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

export default

w33ble reviewed Oct 27, 2016

View reviewed changes

tsullivan reviewed Oct 27, 2016

View reviewed changes

w33ble assigned w33ble and tsullivan Oct 27, 2016

tsullivan reviewed Oct 27, 2016

View reviewed changes

thomasneirynck added needs updates and removed review v5.1.0 v6.0.0 labels Nov 16, 2016

epixa added the Team:Operations Kibana-Operations Team label Nov 17, 2016

tsullivan removed their assignment Dec 21, 2016

w33ble assigned thomasneirynck and unassigned w33ble Dec 26, 2016

thomasneirynck removed their assignment Jan 4, 2017

thomasneirynck closed this Feb 27, 2017

Add grunt-task to build ES from source #8866

Add grunt-task to build ES from source #8866

Uh oh!

Conversation

thomasneirynck commented Oct 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

epixa Oct 27, 2016

Choose a reason for hiding this comment

Uh oh!

w33ble left a comment

Choose a reason for hiding this comment

Uh oh!

w33ble Oct 27, 2016

Choose a reason for hiding this comment

Uh oh!

w33ble Oct 27, 2016

Choose a reason for hiding this comment

Uh oh!

spalger Oct 27, 2016

Choose a reason for hiding this comment

Uh oh!

tsullivan Oct 27, 2016

Choose a reason for hiding this comment

Uh oh!

w33ble commented Oct 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

w33ble commented Oct 27, 2016

Uh oh!

tsullivan commented Oct 27, 2016

Uh oh!

tsullivan Oct 27, 2016

Choose a reason for hiding this comment

Uh oh!

thomasneirynck commented Oct 28, 2016

Uh oh!

w33ble commented Oct 28, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

epixa commented Oct 28, 2016

Uh oh!

w33ble commented Nov 16, 2016

Uh oh!

thomasneirynck commented Nov 16, 2016

Uh oh!

thomasneirynck commented Feb 27, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

thomasneirynck commented Oct 27, 2016 •

edited

Loading

w33ble commented Oct 27, 2016 •

edited

Loading

w33ble commented Oct 28, 2016 •

edited

Loading