Enable packages & CLIs to read core configuration without starting the core server

[joshdover]

There are several use cases that have come up which need to be able to read the kibana.yml configuration before the core server is started or without starting the server at all:

• APM instrumentation
  • To instrument dev CLIs, we need to be able to read config to initialize the APM agent without startig the server
  • To initialize the APM agent in the server process itself, we need to be able to read config before the server is started
  • We currently are working around this with environment variables but it has these drawbacks:
    • Violates the "prefer one way to do things" principle
    • Cloud does not have an easy way to configure env vars on a per-instance basis, limiting our ability to test APM instrumentation on Cloud before rolling it out more widely.
• @kbn/optimizer
  • Right now the CLI only allows configuration via CLI flags, rather than via the kibana.yml file
  • Causes different behavior when running via CLI and running via yarn start (which does read from yml file and then passes into the CLI)

Proposed Solution

• Extract src/core/config into a new @kbn/config package
• Move some key configuration schemas into the @kbn/config package
  • Any configs that need to be read before the server starts will need to be present in this schema(s). For instance, APM agent configuration, optimizer configuration, path.* configuration.
  • Service-specific configurations should not need to move (eg. Elasticsearch, HTTP, SavedObjects).

---

We may need to validate that importing this future @kbn/config package before initializing the APM agent will actually work correctly since this package itself also imports other code (Joi, for example). We may not have instrumentation on those packages, so we must make sure the impact radius of this is small enough to be acceptable.

Another caveat is that config deprecations defined by Core or plugins will not be executed when using this bare @kbn/config package. We must ensure that any config deprecations that touch the config keys we move into this package are moved into @kbn/config as well.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[pgayvallet]

Is creating a new package really necessary? Would importing a standalone / autonomous config service from core instead be acceptable ? Or is there the specific requirement to not load core code when accessing the config (and if that is the case, wouldn't importing the service directly from its file, such as `src/core/server/config/standalone_service', be no different than loading it from a package)?

Moving the service to a package would mean decouple all the services from their config schema and associated types, as the 'core' config schemas would need to also be moved, which is something I would really like to avoid to be honest.

The standalone service could also have the advantage to be preconfigured with core's deprecations. We won't be able to do it for deprecations registered by plugins, but this still seems like a significant upside.

[joshdover]

Is creating a new package really necessary? Would importing a standalone / autonomous config service from core instead be acceptable ? Or is there the specific requirement to not load core code when accessing the config (and if that is the case, wouldn't importing the service directly from its file, such as `src/core/server/config/standalone_service', be no different than loading it from a package)?

Moving the service to a package would mean decouple all the services from their config schema and associated types, as the 'core' config schemas would need to also be moved, which is something I would really like to avoid to be honest.

If we can make that work without introducing any circular dependencies, I'm all for it. Seems like a better first option to me 👍

[pgayvallet]

@joshdover Should we close this? The APM need has need addressed with #77855 and when we last talked about it with @spalger, the need for the optimizer was not urgent.

[joshdover]

Fine by me, if anyone disagrees, please re-open