Re-implement Elasticsearch keystore encoding in Go for hot-reloadable secure settings#8987
Closed
pebrc wants to merge 5 commits intoelastic:mainfrom
Closed
Re-implement Elasticsearch keystore encoding in Go for hot-reloadable secure settings#8987pebrc wants to merge 5 commits intoelastic:mainfrom
pebrc wants to merge 5 commits intoelastic:mainfrom
Conversation
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
🔍 Preview links for changed docs |
Collaborator
Author
|
buildkite test this -f t=TestReloadableKeystore,p=gke,s=9.3.0-SNAPSHOT |
This commit introduces a Go-based implementation of the Elasticsearch keystore format, eliminating the need for Kubernetes Jobs to manage keystores. The new approach enables hot-reloading of secure settings without pod restarts on Elasticsearch 9.3+. Key features: - Pure Go implementation of Elasticsearch keystore format (V7) - Lucene codec-compatible file structure with CRC32 checksums - AES-GCM encryption with PBKDF2-HMAC-SHA512 key derivation - Automatic hot-reload via ES reload_secure_settings API - Settings hash optimization to avoid unnecessary keystore regeneration - Status-based caching to skip redundant reload API calls - Symlink-based Secret mounting for automatic updates The keystore is automatically created on ES 9.3+ clusters (even without user secure settings) to ensure the hot-reload infrastructure is always in place. Users can opt-out via the eck.k8s.elastic.co/disable-reloadable-keystore annotation. Closes: elastic/cloud-on-k8s#XXXX
Adds comprehensive e2e tests for the Go-based reloadable keystore feature: - TestReloadableKeystore: Tests the full lifecycle including: - Cluster startup without secure settings (keystore pre-created) - Adding secure settings (hot-reload, no pod restart) - Updating secure settings (hot-reload, no pod restart) - Verifies keystore annotations (settings hash, digest) - TestReloadableKeystoreDisabled: Tests the opt-out mechanism via the eck.k8s.elastic.co/disable-reloadable-keystore annotation, verifying fallback to init container-based keystore creation.
Collaborator
Author
|
Closing this in favour of a future implementation based on the file-settings infrastructure in Elasticsearch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #8922
This PR implements the Elasticsearch keystore file format in Go within the ECK operator, enabling hot-reloadable secure settings for Elasticsearch 9.3+ without requiring pod restarts or the complexity of a Kubernetes job-based approach.
Motivation
The approach to reload the Elasticsearch keystore without restarts as discussed in #8958 uses a fairly complex Kubernetes job mechanism. This PR takes a different approach: re-implementing the keystore encoding in Go so the operator can directly generate keystore files that Elasticsearch can read.
Changes
Core Keystore Implementation (
pkg/controller/elasticsearch/keystore/)version.go: Version selection logic mapping ES versions to keystore format versions (v4-v7) with appropriate cryptographic parameterscodec.go: Lucene codec header/footer implementation with CRC32 checksumscrypto.go: PBKDF2-HMAC-SHA512 key derivation and AES-256-GCM encryption with AADentries.go: Entry serialization using Java's modified UTF-8 format (DataOutputStream compatible)keystore.go: Main API for creating keystore filesreconciler.go: ES-specific keystore reconciliation with settings hash cachingIntegration
eck.k8s.elastic.co/disable-reloadable-keystoreto fallback to init container approach_nodes/reload_secure_settingsAPI and verifies convergence via keystore digestKey Technical Details
The Go implementation matches Elasticsearch's
KeyStoreWrapper.java:0x3FD76C17DataOutputStreambinary format (big endian, 2-byte UTF string length prefix)Testing
Known Limitations / Future Work
Related