diff --git a/docs/features/opensearch/engine-api.md b/docs/features/opensearch/engine-api.md index 8e53a7c00..2a678d0f4 100644 --- a/docs/features/opensearch/engine-api.md +++ b/docs/features/opensearch/engine-api.md @@ -54,6 +54,8 @@ graph TB | `Engine.DeleteResult` | Result of a delete operation, includes `found` flag | | `Engine.NoOpResult` | Result of a no-op operation for replication | | `Engine.Operation` | Base class for operations (Index, Delete, NoOp) | +| `Engine.prepareIndex()` | Instance method to prepare index operations (v3.4.0+) | +| `Engine.prepareDelete()` | Instance method to prepare delete operations (v3.4.0+) | ### Engine.Result Methods @@ -129,13 +131,19 @@ public enum Type { | Version | PR | Description | |---------|-----|-------------| +| v3.4.0 | [#19551](https://github.com/opensearch-project/OpenSearch/pull/19551) | Move prepareIndex and prepareDelete methods to Engine class | +| v3.4.0 | [#19950](https://github.com/opensearch-project/OpenSearch/pull/19950) | Make NoOpResult constructors public | | v3.3.0 | [#19275](https://github.com/opensearch-project/OpenSearch/pull/19275) | Make all methods in Engine.Result public | ## References +- [Issue #19550](https://github.com/opensearch-project/OpenSearch/issues/19550): Feature request for custom document parsing +- [Issue #19949](https://github.com/opensearch-project/OpenSearch/issues/19949): Feature request for public NoOpResult constructors - [Issue #19276](https://github.com/opensearch-project/OpenSearch/issues/19276): Feature request for public Engine.Result methods +- [Introduction to OpenSearch Plugins](https://opensearch.org/blog/plugins-intro/): Blog post explaining Engine Plugin interface - [Engine.java](https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/engine/Engine.java): Source code ## Change History +- **v3.4.0** (2025-12-09): Added `prepareIndex()` and `prepareDelete()` instance methods to Engine class; made `NoOpResult` constructors public; deprecated static methods on IndexShard - **v3.3.0** (2025-09-12): Made `setTranslogLocation()`, `setTook()`, and `freeze()` methods public to enable custom Engine implementations diff --git a/docs/releases/v3.4.0/features/opensearch/engine-refactoring.md b/docs/releases/v3.4.0/features/opensearch/engine-refactoring.md new file mode 100644 index 000000000..e222f6428 --- /dev/null +++ b/docs/releases/v3.4.0/features/opensearch/engine-refactoring.md @@ -0,0 +1,141 @@ +# Engine Refactoring + +## Summary + +OpenSearch v3.4.0 introduces engine refactoring changes that improve extensibility for custom engine implementations. The changes move document preparation methods (`prepareIndex`, `prepareDelete`) from `IndexShard` to the `Engine` class and make `NoOpResult` constructors public, enabling plugins to implement custom storage engines that can bypass the default mapping framework. + +## Details + +### What's New in v3.4.0 + +This release includes three key refactoring changes to the Engine layer: + +1. **Moved `prepareIndex` and `prepareDelete` to Engine class** - These methods are now instance methods on the `Engine` class instead of static methods on `IndexShard`, allowing custom engines to override document preparation logic. + +2. **Made `Engine#loadHistoryUUID()` protected** - Enables subclasses to access history UUID loading functionality. + +3. **Made `Origin#isFromTranslog()` public** - Allows custom engines to check if an operation originated from translog replay. + +4. **Made `NoOpResult` constructors public** - Custom engine implementations can now create `NoOpResult` instances when overriding the `noOp()` method. + +### Technical Changes + +#### Architecture Changes + +```mermaid +graph TB + subgraph "Before v3.4.0" + IS1["IndexShard"] + E1["Engine"] + IS1 -->|"prepareIndex(static)"| E1 + IS1 -->|"prepareDelete(static)"| E1 + end + + subgraph "After v3.4.0" + IS2["IndexShard"] + E2["Engine"] + CE["Custom Engine"] + IS2 -->|"engine.prepareIndex()"| E2 + IS2 -->|"engine.prepareDelete()"| E2 + CE -->|"extends"| E2 + CE -->|"override prepareIndex()"| CE + end +``` + +#### New Methods in Engine Class + +| Method | Description | Visibility | +|--------|-------------|------------| +| `prepareIndex()` | Prepares an index operation by parsing source document | `public` | +| `prepareDelete()` | Prepares a delete operation | `public` | +| `loadHistoryUUID()` | Loads history UUID from commit data | `protected` | + +#### Visibility Changes + +| Class/Method | Before | After | +|--------------|--------|-------| +| `NoOpResult(long term, long seqNo)` | package-private | `public` | +| `NoOpResult(long term, long seqNo, Exception failure)` | package-private | `public` | +| `Origin#isFromTranslog()` | package-private | `public` | +| `Engine#loadHistoryUUID()` | private | `protected` | + +### Usage Example + +Custom engine implementations can now override document preparation: + +```java +public class CustomEngine extends Engine { + + @Override + public Engine.Index prepareIndex( + DocumentMapperForType docMapper, + SourceToParse source, + long seqNo, + long primaryTerm, + long version, + VersionType versionType, + Engine.Operation.Origin origin, + long autoGeneratedIdTimestamp, + boolean isRetry, + long ifSeqNo, + long ifPrimaryTerm + ) { + // Custom document parsing logic + // Can bypass default mapping framework for specialized use cases + long startTime = System.nanoTime(); + ParsedDocument doc = customParse(source); + Term uid = new Term(IdFieldMapper.NAME, Uid.encodeId(doc.id())); + return new Engine.Index(uid, doc, seqNo, primaryTerm, version, + versionType, origin, startTime, autoGeneratedIdTimestamp, + isRetry, ifSeqNo, ifPrimaryTerm); + } + + @Override + public NoOpResult noOp(NoOp noOp) { + // Can now create NoOpResult directly + return new NoOpResult(noOp.primaryTerm(), noOp.seqNo()); + } +} +``` + +### Migration Notes + +The static methods on `IndexShard` are now deprecated: + +```java +// Deprecated (v3.4.0) +@Deprecated(since = "3.4.0", forRemoval = true) +public static Engine.Index IndexShard.prepareIndex(...) + +@Deprecated(since = "3.4.0", forRemoval = true) +public static Engine.Delete IndexShard.prepareDelete(...) + +// New approach +engine.prepareIndex(...) +engine.prepareDelete(...) +``` + +Existing code using the static methods will continue to work but should migrate to the instance methods on Engine. + +## Limitations + +- Custom engines must properly handle all parameters passed to `prepareIndex()` and `prepareDelete()` +- Overriding these methods requires understanding of the document lifecycle and translog integration +- The deprecated static methods will be removed in a future release + +## Related PRs + +| PR | Description | +|----|-------------| +| [#19551](https://github.com/opensearch-project/OpenSearch/pull/19551) | Refactor to move prepareIndex and prepareDelete methods to Engine class | +| [#19950](https://github.com/opensearch-project/OpenSearch/pull/19950) | Update NoOpResult constructors in the Engine to be public | + +## References + +- [Issue #19550](https://github.com/opensearch-project/OpenSearch/issues/19550): Feature request for custom document parsing +- [Issue #19949](https://github.com/opensearch-project/OpenSearch/issues/19949): Feature request for public NoOpResult constructors +- [Introduction to OpenSearch Plugins](https://opensearch.org/blog/plugins-intro/): Blog post explaining Engine Plugin interface + +## Related Feature Report + +- [Full feature documentation](../../../../features/opensearch/engine-api.md) diff --git a/docs/releases/v3.4.0/index.md b/docs/releases/v3.4.0/index.md index 2bfeaa25e..cc4a31d52 100644 --- a/docs/releases/v3.4.0/index.md +++ b/docs/releases/v3.4.0/index.md @@ -7,6 +7,7 @@ - [Build Tool Upgrades](features/opensearch/build-tool-upgrades.md) - Gradle 9.1 and bundled JDK 25 updates - [Concurrent Segment Search](features/opensearch/concurrent-segment-search.md) - Performance optimization by omitting MaxScoreCollector when sorting by score - [Dependency Updates (OpenSearch Core)](features/opensearch/dependency-updates-opensearch-core.md) - 32 dependency updates including Netty 4.2.4 for HTTP/3 readiness +- [Engine Refactoring](features/opensearch/engine-refactoring.md) - Move prepareIndex/prepareDelete to Engine class and make NoOpResult constructors public - [JDK 25 Support](features/opensearch/jdk-25-support.md) - Painless scripting compatibility fix for JDK 25 ClassValue behavioral change - [Lucene Integration](features/opensearch/lucene-integration.md) - Remove MultiCollectorWrapper and use Lucene's native MultiCollector API - [Lucene Upgrade](features/opensearch/lucene-upgrade.md) - Bump Apache Lucene from 10.3.1 to 10.3.2 with MaxScoreBulkScorer bug fix