OpenSearch's Unreliable Dependency - A Call for Enhanced API Compatibility

[peternied]

This issue aims to shed light on a frustrating experience that is negatively impacting contributors, including myself. The intention is not to shame or punish but rather to address the problem constructively and foster a conversation for a potential solution.

Describe the bug

OpenSearch's main does not have any requirements for public API compatibility. This often causes changes on snapshot builds that break compilation of plugins or integration tests of plugins to fail. This issues are difficult to debug and consumer considerable time on its plugins.

Breaking changes are backported in previous major version numbers, these layers make it hard to know where new issues are coming from.

To Reproduce

Maintain a plugin that depends on OpenSearch.

Expected behavior

There are several options:

Keep the Status Quo:

• OpenSearch core continues to grow with added functionalities.
• Plugins contribute more functionality into the core codebase to reduce the burden of maintenance.

Strictly Follow SemVer:

• OpenSearch adheres to Semantic Versioning, ensuring no public API changes without a major version increment.
• Plugin teams can have operational build pipelines and better control over migration.
• OpenSearch core contributions are more strictly monitored, limiting the introduction of non-essential breaking changes.

Increase Verification of Pull Requests:

• OpenSearch pull requests undergo verification against plugins' GitHub Actions checks before merging.
• Plugin teams integrate test suites that are run before pull requests are merged, enabling them to be considered as a blocking consumer.

Additional context

• Issue impact security plugin Investigate CI Failures security#3064
• Core refactoring initiative- driving this kind of issue [Discuss] Core Foundation Code Refactoring #5910

[davidlago]

• Linking related issue Add workflow to check compatibility #8486

[peternied]

Recent change that impacted all plugins:

• [Refactor] CircuitBreaker foundation classes to core library #9006

[peternied]

@nknize I know you are at work with refactoring components in core [1], the breaking changes being pushed out to plugin teams are painful. Is there a quantification of how far along that work is / for how much time plugin teams will have broken builds due to these changes?

• [1] [Discuss] Core Foundation Code Refactoring #5910

[nknize]

Hey @peternied. I missed this message before posting my comment on the PR to detect compatibility impact.

Is there a quantification of how far along that work is / for how much time plugin teams will have broken builds due to these changes?

I'd like to target Phase 0 of #8110 to be complete no later than 2.11.

Of course that could slip, however, depending on the downstream surface area impacted once module-info.java is added to the core modules. That effort will likely bias towards minimal exports, which means downstreams will be temporarily cut off from some classes they currently extend. We'll take each breakage on a case by case basis and bias towards exporting, but quickly redesigning what needs to be redesigned to properly enforce semver.

In the meantime any mechanisms we can create to surface downstream impact early will help us assist downstreams in quickly opening PRs to change namespaces. Perhaps someone could put together a slackbot to post to a #build channel?

[peternied]

@nknize Its good to hear we have a near-term target for those changes.

Building off of this - do we know if there is a list of planned breaking changes we need before we think we can target a semver safe API?

[peternied]

@nknize Do you know who would be able to help assemble that list of breaking changes that should be expected?

[peternied]

Five days ago downstream pull requests [1] were needed to handle the task foundaction refactor [2] This change was backported today, and presumably will require another set of changes to be backported.

• [1] Follow up pull requests to handle task refactor
  • Fix - Breaking Change Main OpenSearch project - action move to core ml-commons#1200
  • [Build Break] React to changes in ActionListener and ActionResponse from #9082 security#3153
  • Fixed compilation errors after refactoring in core neural-search#249
  • Fixed compilation errors after changes in core foundation classes ml-commons#1202
  • React to changes in ActionListener and ActionFuture from #9082 job-scheduler#463
  • Breaking change OpenSearch main project - Action movement sql#1958
  • Fixed compilation errors after refactoring in core foundation classes k-NN#1038
  • Fix from upstream core.action changes observability#1590
  • [Backport 2.x] React to changes in ActionListener and ActionFuture from #9082 job-scheduler#467
  • [Refactor] rebase to support latest core snapshot security-analytics#510
  • opensearch-project/reporting@ce1c03f
  • make common-utils core compatible after core #9082 common-utils#507
  • Fixing Action imports and defaults AnomalyResult grade to 0 if negative anomaly-detection#976
  • Address core refactor changes performance-analyzer#522
  • Fix compilation issues in main asynchronous-search#292
• [2] [Refactor] Task foundation classes to core library - pt 1 #9082

[peternied]

Capturing this proposal on how we could change the consumption model within OpenSearch Project

How does apache foundation handle cases like this?

It's varies by project. I can only speak for Lucene and Solr (and Elasticsearch, and even OpenSearch). Solr (and Elasticsearch, and OpenSearch core) absorbs upstream changes "on demand" the same way we do it here with Lucene. This is what I have been suggesting for a while.

Instead of upstream core OpenSearch repo force feeding per-merge SNAPSHOT artifacts to all downstreams, the downstreams should manually (or on some automated, e.g., weekly, cron period) trigger a snapshot build and update their ${opensearch_version} in build.gradle to their own created sha (e.g., 2.10.0-SNAPSHOT-4373c3b). This would keep downstream builds stable. The unpleasant side affect would depend on how often those snapshots are updated, but that would be under the control of the downstream repo community. It's not perfect (breaking changes never are) but least then the downstream controls their own pain. This happens a lot on Solr, OpenSearch, Elasticsearch when large changes are made to lucene (e.g., JPMS, gradle change from ant, Endianness changes)

From #9367 (comment)

[nknize]

Follow up comment:

I wonder if we could invent a "toggle" mechanism to allow downstreams to switch between the two approaches at will? Maybe there's a GHA to enable/disable "force fed" snapshots? This could prove useful during large scale refactors to clot the bleeding? Downstreams cut over to the manual snapshot consumption model, and switch back when the change storm has passed?