Move kibana-keystore from data/ to config/ by jbudz · Pull Request #57856 · elastic/kibana

jbudz · 2020-02-18T15:31:48Z

This is a breaking change to move the location of kibana-keystore.
Keystores in other stack products live in the config directory, so this
updates our current path to be consistent.

Closes #25746

This doesn't respect kibana.yml currently, and I would like to push that for the next iteration. The config service currently depends on the server to be running, while the keystore does not start the server. A portable config service is ideal, but it's fairly coupled. The next steps would include

build time kibana.yml with separate paths for deb/rpm and archives
find a way to fetch config values without a running server

This is a breaking change to move the location of kibana-keystore. Keystores in other stack products live in the config directory, so this updates our current path to be consistent. Closes elastic#25746

elasticmachine · 2020-02-18T15:31:51Z

Pinging @elastic/kibana-operations (Team:Operations)

tylersmalley · 2020-02-18T16:02:44Z

Mind adding this to the breaking changes markdown as part of this PR?

joshdover · 2020-02-18T17:20:05Z

find a way to fetch config values without a running server

This is something @spalger has ran into recently as well. I think it makes sense to extract our config logic into a standalone package. Is there any issue for this?

Is there any reason this can't be done in a BWC way on the 7.x branch as well? It'd be nice to fallback to the old location and log a warning.

tylersmalley · 2020-03-10T16:42:50Z

Does this also resolve #31171?

jbudz · 2020-03-19T15:26:20Z

@elasticmachine merge upstream

…kibana-keystore

jbudz · 2020-03-19T15:36:19Z

Added breaking changes docs. A new PR targeting 7.x will remove the docs changes and add a logged deprecation. This should be okay for standalone review, but I'll wait to merge until both are approved

jbudz · 2020-03-19T20:15:47Z

@elasticmachine merge upstream

pgayvallet · 2020-03-23T08:38:27Z

src/core/server/path/index.ts

+
+/**
+ * Get the path where the config directories are stored
+ * @internal
+ */
+export const getConfigDirectory = () => findFile(CONFIG_DIRECTORIES);


Misleading comment. This resolves to the actual config directory.

Outside of the PR, but the getConfigPath description is also wrong (Get the path where the config files are stored), as it resolves the path to the config file itself.

…kibana-keystore

pgayvallet

LGTM for platform changes

tylersmalley · 2020-04-16T20:38:43Z

@elasticmachine merge upstream

elasticmachine · 2020-04-16T20:38:45Z

merge conflict between base and head

tylersmalley · 2020-04-16T20:40:03Z

@jbudz where are we at with this?

ghost · 2020-05-05T21:24:42Z

Do I understand correctly that key-store file will be created in the same directory as kibana.yml?
I am the owner of https://github.com/elastic/elastic-stack-installers, currently working on Kibana Windows MSI package. I found that currently bundled kibana-keystore.bat doesn't respond to --config and --path.data args. This presents a problem for MSI based install, since on Windows I need to move all user-serviceable data to a different location away from main install path.

ref: elastic/elastic-stack-installers#60

jbudz · 2020-05-05T21:34:40Z

Yep you got it. The file location won't be configurable, but it will respect the configuration directories path.

ghost · 2020-05-05T21:37:32Z

The file location won't be configurable

🤔 "Won't" as in "never", or ... ? This will present a problem wrt UX dealing with keystore CLI on Windows.

tylersmalley · 2020-06-12T16:19:53Z

@jbudz is this ready for another pass?

jbudz · 2020-06-23T17:17:37Z

@elasticmachine merge upstream

jbudz · 2020-06-23T17:17:56Z

Yep it is.

jbudz · 2020-06-29T14:57:00Z

@elasticmachine merge upstream

tylersmalley

Couple requests, but in testing I am getting an error when running scripts/kibana_keystore

 $ node scripts/kibana_keystore.js --help
/home/tyler/elastic/kibana/src/legacy/server/keystore/keystore.js:78
      throw e;
      ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be one of type string, Buffer, or URL. Received type undefined
    at Object.openSync (fs.js:435:3)
    at readFileSync (fs.js:343:35)
    at Keystore.load (/home/tyler/elastic/kibana/src/legacy/server/keystore/keystore.js:87:24)
    at new Keystore (/home/tyler/elastic/kibana/src/legacy/server/keystore/keystore.js:34:10)
    at Object.<anonymous> (/home/tyler/elastic/kibana/src/cli_keystore/cli_keystore.js:57:18)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Module._compile (/home/tyler/elastic/kibana/node_modules/pirates/lib/index.js:99:24)
    at Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Object.newLoader [as .js] (/home/tyler/elastic/kibana/node_modules/pirates/lib/index.js:104:7)
    at Module.load (internal/modules/cjs/loader.js:653:32)

docs/migration/migrate_8_0.asciidoc

tylersmalley · 2020-06-30T23:52:54Z

src/cli_keystore/cli_keystore.js

  .version(pkg.version)
  .description('A tool for managing settings stored in the Kibana keystore');

+function getKeystore() {


This function is pretty important - can we add test coverage there?

skmizuho · 2020-07-02T02:06:17Z

Quick q on this, please. When it comes to upgrading from one's current version to version 8.0, will we need to copy the keystore from data to config directory to get it to work? How will the migration work?
Thank you

tylersmalley · 2020-07-02T04:16:17Z

@skmizuho, in this PR we are first checking for the presence of the keystore in the data directory. Be sure that when we make changes, we will include this information in the breaking changes for 8.0 and plan to include a notice for it in the upgrade assistant.

skmizuho · 2020-07-02T05:33:23Z

Thank you - understood. Big supported of this change by the way! It makes the configuration much simpler.
Cheers

…kibana-keystore

jbudz · 2020-07-07T18:57:00Z

Hmm - can you give another try? I did have a commit with that error at one point. Also pushed changes up for the other changes

tylersmalley · 2020-07-09T23:42:43Z

@elasticmachine merge upstream

jbudz · 2020-07-13T14:03:03Z

@elasticmachine merge upstream

kibanamachine · 2020-07-13T15:48:35Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 74ab52a

Build metrics

✅ unchanged

History

💛 Build #60541 was flaky bb6bc84
💚 Build #59480 succeeded 5dfce32
💔 Build #57176 failed 30abff0
💚 Build #55609 succeeded 6f7621e
💚 Build #53634 succeeded beee61f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Move kibana-keystore from data/ to config/ This is a breaking change to move the location of kibana-keystore. Keystores in other stack products live in the config directory, so this updates our current path to be consistent. Closes #25746 * add breaking changes * update comment * wip * fix docs * read from both keystore locations, write priority to non-deprecated * note data directory fallback * add tests for get_keystore Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

jbudz · 2020-07-13T15:59:29Z

7.9/7.x: c9b6f8a

gchaps · 2020-08-03T22:00:03Z

@jbudz This PR has the release_note:breaking label and therefore should be included in the Breaking Changes doc. Please add a description of the change to this PR that I can use in the doc. For an example, take a look at the 7.8 Breaking Changes doc.

With elastic#57856 the default keystore location was moved from the data folder to the config folder to align with other applications in the stack. This updates the secure settings docs to reference the correct location. Closes elastic#111915

Move kibana-keystore from data/ to config/

9443a0a

This is a breaking change to move the location of kibana-keystore. Keystores in other stack products live in the config directory, so this updates our current path to be consistent. Closes elastic#25746

jbudz added release_note:breaking Team:Operations Kibana-Operations Team v8.0.0 labels Feb 18, 2020

jbudz requested a review from a team as a code owner February 18, 2020 15:31

Merge branch 'master' into mv/kibana-keystore

f7c5772

elasticmachine and others added 3 commits March 19, 2020 11:26

Merge branch 'master' into mv/kibana-keystore

4255fc3

Merge branch 'mv/kibana-keystore' of github.com:jbudz/kibana into mv/…

f327c59

…kibana-keystore

add breaking changes

909eaa5

Merge branch 'master' into mv/kibana-keystore

625b9c3

pgayvallet reviewed Mar 23, 2020

View reviewed changes

jbudz added 3 commits March 30, 2020 10:15

Merge branch 'master' into mv/kibana-keystore

fc13d8f

update comment

f6c5439

Merge branch 'mv/kibana-keystore' of github.com:jbudz/kibana into mv/…

2991ba2

…kibana-keystore

pgayvallet approved these changes Mar 30, 2020

View reviewed changes

Merge branch 'master' into mv/kibana-keystore

cd18ed8

mistic mentioned this pull request May 5, 2020

Kibana paths configuration elastic/elastic-stack-installers#63

Open

Merge branch 'master' into mv/kibana-keystore

6f7621e

Merge branch 'master' into mv/kibana-keystore

30abff0

tylersmalley requested changes Jun 30, 2020

View reviewed changes

jbudz added 5 commits July 7, 2020 10:17

Merge branch 'master' into mv/kibana-keystore

7edcc9d

note data directory fallback

e6fb8e1

add tests for get_keystore

3c36f55

Merge branch 'mv/kibana-keystore' of github.com:jbudz/kibana into mv/…

ad92279

…kibana-keystore

Merge branch 'master' into mv/kibana-keystore

5dfce32

Merge branch 'master' into mv/kibana-keystore

bb6bc84

tylersmalley approved these changes Jul 10, 2020

View reviewed changes

Merge branch 'master' into mv/kibana-keystore

74ab52a

jbudz added the v7.9.0 label Jul 13, 2020

jbudz merged commit 94ef03d into elastic:master Jul 13, 2020

jbudz added the backported label Jul 13, 2020

jbudz mentioned this pull request Sep 13, 2021

[docs] Update keystore location #111994

Merged

Conversation

jbudz commented Feb 18, 2020 • edited by tylersmalley Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Feb 18, 2020

Uh oh!

tylersmalley commented Feb 18, 2020

Uh oh!

joshdover commented Feb 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylersmalley commented Mar 10, 2020

Uh oh!

jbudz commented Mar 19, 2020

Uh oh!

jbudz commented Mar 19, 2020

Uh oh!

jbudz commented Mar 19, 2020

Uh oh!

pgayvallet Mar 23, 2020

Choose a reason for hiding this comment

Uh oh!

pgayvallet left a comment

Choose a reason for hiding this comment

Uh oh!

tylersmalley commented Apr 16, 2020

Uh oh!

elasticmachine commented Apr 16, 2020

Uh oh!

tylersmalley commented Apr 16, 2020

Uh oh!

ghost commented May 5, 2020

Uh oh!

jbudz commented May 5, 2020

Uh oh!

ghost commented May 5, 2020 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylersmalley commented Jun 12, 2020

Uh oh!

jbudz commented Jun 23, 2020

Uh oh!

jbudz commented Jun 23, 2020

Uh oh!

jbudz commented Jun 29, 2020

Uh oh!

tylersmalley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tylersmalley Jun 30, 2020

Choose a reason for hiding this comment

Uh oh!

jbudz Jul 7, 2020

Choose a reason for hiding this comment

Uh oh!

skmizuho commented Jul 2, 2020

Uh oh!

tylersmalley commented Jul 2, 2020

Uh oh!

skmizuho commented Jul 2, 2020

Uh oh!

jbudz commented Jul 7, 2020

Uh oh!

tylersmalley commented Jul 9, 2020

Uh oh!

jbudz commented Jul 13, 2020

Uh oh!

kibanamachine commented Jul 13, 2020

💚 Build Succeeded

Build metrics

History

Uh oh!

jbudz commented Jul 13, 2020

Uh oh!

gchaps commented Aug 3, 2020

Uh oh!

Reviewers

Assignees

Labels

jbudz commented Feb 18, 2020 •

edited by tylersmalley

Loading

joshdover commented Feb 18, 2020 •

edited

Loading

ghost commented May 5, 2020 •

edited by ghost

Loading