Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/features/opensearch/engine-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ graph TB
| `Engine.DeleteResult` | Result of a delete operation, includes `found` flag |
| `Engine.NoOpResult` | Result of a no-op operation for replication |
| `Engine.Operation` | Base class for operations (Index, Delete, NoOp) |
| `Engine.prepareIndex()` | Instance method to prepare index operations (v3.4.0+) |
| `Engine.prepareDelete()` | Instance method to prepare delete operations (v3.4.0+) |

### Engine.Result Methods

Expand Down Expand Up @@ -129,13 +131,19 @@ public enum Type {

| Version | PR | Description |
|---------|-----|-------------|
| v3.4.0 | [#19551](https://github.com/opensearch-project/OpenSearch/pull/19551) | Move prepareIndex and prepareDelete methods to Engine class |
| v3.4.0 | [#19950](https://github.com/opensearch-project/OpenSearch/pull/19950) | Make NoOpResult constructors public |
| v3.3.0 | [#19275](https://github.com/opensearch-project/OpenSearch/pull/19275) | Make all methods in Engine.Result public |

## References

- [Issue #19550](https://github.com/opensearch-project/OpenSearch/issues/19550): Feature request for custom document parsing
- [Issue #19949](https://github.com/opensearch-project/OpenSearch/issues/19949): Feature request for public NoOpResult constructors
- [Issue #19276](https://github.com/opensearch-project/OpenSearch/issues/19276): Feature request for public Engine.Result methods
- [Introduction to OpenSearch Plugins](https://opensearch.org/blog/plugins-intro/): Blog post explaining Engine Plugin interface
- [Engine.java](https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/engine/Engine.java): Source code

## Change History

- **v3.4.0** (2025-12-09): Added `prepareIndex()` and `prepareDelete()` instance methods to Engine class; made `NoOpResult` constructors public; deprecated static methods on IndexShard
- **v3.3.0** (2025-09-12): Made `setTranslogLocation()`, `setTook()`, and `freeze()` methods public to enable custom Engine implementations
141 changes: 141 additions & 0 deletions docs/releases/v3.4.0/features/opensearch/engine-refactoring.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Engine Refactoring

## Summary

OpenSearch v3.4.0 introduces engine refactoring changes that improve extensibility for custom engine implementations. The changes move document preparation methods (`prepareIndex`, `prepareDelete`) from `IndexShard` to the `Engine` class and make `NoOpResult` constructors public, enabling plugins to implement custom storage engines that can bypass the default mapping framework.

## Details

### What's New in v3.4.0

This release includes three key refactoring changes to the Engine layer:

1. **Moved `prepareIndex` and `prepareDelete` to Engine class** - These methods are now instance methods on the `Engine` class instead of static methods on `IndexShard`, allowing custom engines to override document preparation logic.

2. **Made `Engine#loadHistoryUUID()` protected** - Enables subclasses to access history UUID loading functionality.

3. **Made `Origin#isFromTranslog()` public** - Allows custom engines to check if an operation originated from translog replay.

4. **Made `NoOpResult` constructors public** - Custom engine implementations can now create `NoOpResult` instances when overriding the `noOp()` method.

### Technical Changes

#### Architecture Changes

```mermaid
graph TB
subgraph "Before v3.4.0"
IS1["IndexShard"]
E1["Engine"]
IS1 -->|"prepareIndex(static)"| E1
IS1 -->|"prepareDelete(static)"| E1
end

subgraph "After v3.4.0"
IS2["IndexShard"]
E2["Engine"]
CE["Custom Engine"]
IS2 -->|"engine.prepareIndex()"| E2
IS2 -->|"engine.prepareDelete()"| E2
CE -->|"extends"| E2
CE -->|"override prepareIndex()"| CE
end
```

#### New Methods in Engine Class

| Method | Description | Visibility |
|--------|-------------|------------|
| `prepareIndex()` | Prepares an index operation by parsing source document | `public` |
| `prepareDelete()` | Prepares a delete operation | `public` |
| `loadHistoryUUID()` | Loads history UUID from commit data | `protected` |

#### Visibility Changes

| Class/Method | Before | After |
|--------------|--------|-------|
| `NoOpResult(long term, long seqNo)` | package-private | `public` |
| `NoOpResult(long term, long seqNo, Exception failure)` | package-private | `public` |
| `Origin#isFromTranslog()` | package-private | `public` |
| `Engine#loadHistoryUUID()` | private | `protected` |

### Usage Example

Custom engine implementations can now override document preparation:

```java
public class CustomEngine extends Engine {

@Override
public Engine.Index prepareIndex(
DocumentMapperForType docMapper,
SourceToParse source,
long seqNo,
long primaryTerm,
long version,
VersionType versionType,
Engine.Operation.Origin origin,
long autoGeneratedIdTimestamp,
boolean isRetry,
long ifSeqNo,
long ifPrimaryTerm
) {
// Custom document parsing logic
// Can bypass default mapping framework for specialized use cases
long startTime = System.nanoTime();
ParsedDocument doc = customParse(source);
Term uid = new Term(IdFieldMapper.NAME, Uid.encodeId(doc.id()));
return new Engine.Index(uid, doc, seqNo, primaryTerm, version,
versionType, origin, startTime, autoGeneratedIdTimestamp,
isRetry, ifSeqNo, ifPrimaryTerm);
}

@Override
public NoOpResult noOp(NoOp noOp) {
// Can now create NoOpResult directly
return new NoOpResult(noOp.primaryTerm(), noOp.seqNo());
}
}
```

### Migration Notes

The static methods on `IndexShard` are now deprecated:

```java
// Deprecated (v3.4.0)
@Deprecated(since = "3.4.0", forRemoval = true)
public static Engine.Index IndexShard.prepareIndex(...)

@Deprecated(since = "3.4.0", forRemoval = true)
public static Engine.Delete IndexShard.prepareDelete(...)

// New approach
engine.prepareIndex(...)
engine.prepareDelete(...)
```

Existing code using the static methods will continue to work but should migrate to the instance methods on Engine.

## Limitations

- Custom engines must properly handle all parameters passed to `prepareIndex()` and `prepareDelete()`
- Overriding these methods requires understanding of the document lifecycle and translog integration
- The deprecated static methods will be removed in a future release

## Related PRs

| PR | Description |
|----|-------------|
| [#19551](https://github.com/opensearch-project/OpenSearch/pull/19551) | Refactor to move prepareIndex and prepareDelete methods to Engine class |
| [#19950](https://github.com/opensearch-project/OpenSearch/pull/19950) | Update NoOpResult constructors in the Engine to be public |

## References

- [Issue #19550](https://github.com/opensearch-project/OpenSearch/issues/19550): Feature request for custom document parsing
- [Issue #19949](https://github.com/opensearch-project/OpenSearch/issues/19949): Feature request for public NoOpResult constructors
- [Introduction to OpenSearch Plugins](https://opensearch.org/blog/plugins-intro/): Blog post explaining Engine Plugin interface

## Related Feature Report

- [Full feature documentation](../../../../features/opensearch/engine-api.md)
1 change: 1 addition & 0 deletions docs/releases/v3.4.0/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
- [Build Tool Upgrades](features/opensearch/build-tool-upgrades.md) - Gradle 9.1 and bundled JDK 25 updates
- [Concurrent Segment Search](features/opensearch/concurrent-segment-search.md) - Performance optimization by omitting MaxScoreCollector when sorting by score
- [Dependency Updates (OpenSearch Core)](features/opensearch/dependency-updates-opensearch-core.md) - 32 dependency updates including Netty 4.2.4 for HTTP/3 readiness
- [Engine Refactoring](features/opensearch/engine-refactoring.md) - Move prepareIndex/prepareDelete to Engine class and make NoOpResult constructors public
- [JDK 25 Support](features/opensearch/jdk-25-support.md) - Painless scripting compatibility fix for JDK 25 ClassValue behavioral change
- [Lucene Integration](features/opensearch/lucene-integration.md) - Remove MultiCollectorWrapper and use Lucene's native MultiCollector API
- [Lucene Upgrade](features/opensearch/lucene-upgrade.md) - Bump Apache Lucene from 10.3.1 to 10.3.2 with MaxScoreBulkScorer bug fix
Expand Down