From 04ab2d2fc9b5476d42026ff87efcce919eaef8c6 Mon Sep 17 00:00:00 2001 From: Rudolf Meijering Date: Wed, 16 Jul 2025 23:28:41 +0200 Subject: [PATCH] [docs] ES tutorial on index initialization (#224628) ## Summary We're seeing a lot of plugins try to use their indices before being properly initialized leading to search_phase_execution_exceptions since shards aren't immediately available. This guide aims to give guidance on how to create indices and be sure they're ready for indexing/searching in a distributed environment like Kibana. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Jean-Louis Leysens (cherry picked from commit e175389b8cb5558907f132ecb0f5b86519b7b808) --- dev_docs/nav-kibana-dev.docnav.json | 5 +- dev_docs/tutorials/elasticsearch.mdx | 156 +++++++++++++++++++++++++++ 2 files changed, 160 insertions(+), 1 deletion(-) create mode 100644 dev_docs/tutorials/elasticsearch.mdx diff --git a/dev_docs/nav-kibana-dev.docnav.json b/dev_docs/nav-kibana-dev.docnav.json index a6460064f902e..eb9e07f4e74a7 100644 --- a/dev_docs/nav-kibana-dev.docnav.json +++ b/dev_docs/nav-kibana-dev.docnav.json @@ -164,6 +164,9 @@ { "id": "kibDevTutorialSavedObject" }, + { + "id": "kibDevTutorialElasticsearchClient" + }, { "id": "kibDevTutorialVersioningHTTPAPIs", "label": "Versioning HTTP APIs" @@ -659,4 +662,4 @@ ] } ] -} +} \ No newline at end of file diff --git a/dev_docs/tutorials/elasticsearch.mdx b/dev_docs/tutorials/elasticsearch.mdx new file mode 100644 index 0000000000000..df93fe0c0fb5a --- /dev/null +++ b/dev_docs/tutorials/elasticsearch.mdx @@ -0,0 +1,156 @@ +--- +id: kibDevTutorialElasticsearchClient +slug: /kibana-dev-docs/tutorials/elasticsearch-client +title: Access the Elasticsearch Client in a Plugin +description: Learn how to use the Elasticsearch client in your Kibana plugin. +date: 2025-06-19 +tags: ['kibana','onboarding', 'dev', 'elasticsearch', 'tutorials'] +--- + +This guide covers how to obtain the Elasticsearch client in both your plugin's start lifecycle and route handler context. It explains the difference between asCurrentUser and asInternalUser, and demonstrates best practices for robust concurrent index initialization. + +## Getting the Elasticsearch Client + +### From the Plugin Start Contract + +In your plugin's `start` method, the `core` parameter provides access to the Elasticsearch client: + +```ts +import type { CoreStart, Plugin } from '@kbn/core/server'; + +export class MyPlugin implements Plugin { + public start(core: CoreStart) { + const esClient = core.elasticsearch.client.asInternalUser; + // You can now use esClient to call Elasticsearch APIs + } +} +``` + +- Use `asScoped(request)` to perform actions as the request-authenticated user. +- **Note**: In HTTP route handlers you receive a client already scoped to the user `elasticsearch.client.asCurrentUser` instead of needing to call `asScoped(request)`. +- Use `asInternalUser` for system-level operations that should bypass user permissions (see security considerations). + +### From a Route Handler Context + +When defining a server route, you can access the Elasticsearch client from the route handler's `context` parameter: + +```ts +router.get( + { path: '/api/my_plugin/search', validate: false }, + async (context, request, response) => { + const esClient = (await context.core).elasticsearch.client.asCurrentUser; + // Use esClient to call Elasticsearch APIs + const result = await esClient.search({ + index: 'my-index', + body: { query: { match_all: {} } } + }); + return response.ok({ body: result }); + } +); +``` + +- `context.core.elasticsearch.client.asCurrentUser` executes requests as the user making the HTTP request. +- Always prefer `asCurrentUser` in route handlers to respect user permissions. + +## Example: Safely Initializing, Indexing, and Searching an Index + +When working in distributed environments (such as multiple Kibana instances), it is common for several instances to attempt to create the same index at the same time. To ensure the index is ready for ingesting and searching, all instances should: + +1. Attempt to create the index. +2. If the index already exists (or creation succeeds), wait until the index status is **green** before proceeding. + +```ts +import type { CoreStart, Plugin } from '@kbn/core/server'; +import type { ElasticsearchClient } from '@kbn/core-elasticsearch-server'; + +async function ensureIndexReady(esClient: ElasticsearchClient, indexName: string, mappings: object) { + try { + // Attempt to create the index even if it may already exist + // Use a long timeout and set requestTimeout slightly longer to avoid socket closure + await esClient.indices.create( + { + index: indexName, + mappings, + timeout: '300s', + // Allow for 0 or 1 replicas for higher availability on multi-node clusters, but continue to work on single-node (dev) clusters + auto_expand_replicas: '0-1', + }, + { requestTimeout: 310_000 } + ); + } catch (error) { + if (error?.body?.error?.type !== 'resource_already_exists_exception') { + throw error; + } + // Index already exists - this is expected in multi-instance environments + // Do NOT log this as an error since it's an expected scenario that we + // gracefully handle + } + + // Wait for the index to be ready (green status) + // Use a long timeout and set requestTimeout slightly longer to avoid socket closure + // Note: On serverless, the health API is only available to the internal user + await esClient.cluster.health( + { + index: indexName, + wait_for_status: 'green', + timeout: '300s', + }, + { requestTimeout: 310_000 } + ); +} + +export class MyPlugin implements Plugin { + public start(core: CoreStart) { + const esClient = core.elasticsearch.client.asInternalUser; + const indexName = 'my-index'; + const mappings = { + properties: { + title: { type: 'text' }, + description: { type: 'text' }, + }, + }; + + ensureIndexReady(esClient, indexName, mappings) + .then(() => { + // Index is ready for ingesting and searching + }) + .catch((err) => { + this.logger.error(`Failed to initialize index ${indexName} in start: ${err?.message || err}`); + }); + } +} +``` + +### Why Wait for Green Status? + +- The create index call can return successfully if the timeout was reached after the operation was accepted but the primary shards are not yet available (in this case the response will include `acknowledged: true, shard_acknowledged: false`). +- Multiple Kibana instances may race to create the same index. If a given instance receives the "index already exists" exception it does not provide any guarantee that the primary shards are available yet. +- After creation (or if the index already exists), always wait for the index to reach **green** status before indexing or searching. This ensures all primary shards are available, preventing failed operations. +- NOTE: On stateful **yellow** status would be sufficient, but serverless requires waiting for green status. + +### Logging Best Practices + +When implementing index initialization: + +- **Do NOT log `resource_already_exists_exception` as an error** - this is completely expected when multiple Kibana instances start simultaneously +- **DO log genuine initialization failures** - like an index still not being available after the timeout + +```ts +// Good: Only log actual errors +} catch (error) { + if (error?.body?.error?.type !== 'resource_already_exists_exception') { + this.logger.error(`Failed to create index ${indexName}: ${error?.message || error}`); + throw error; + } + // Don't log - this is expected behavior +} +``` + +--- + +## Security Considerations + +- Use `asScoped` or `asCurrentUser` for user-initiated actions to enforce security +- Use `asInternalUser` only for trusted, internal operations that must not be restricted by user permissions. +- Never use `asInternalUser` in route handlers that respond to user requests. +