Implement error_trace parameter for HTTP Bulk requests

[fdesu]

Description

This PR adds support for showing detailed stack traces when Bulk HTTP Apis, such as _bulk, _msearch, etc. are called.

The error_trace was already implemented from the beginning and it works for Bulk APIs when an internal error (e.g. an NPE) occurs. Detailed stack traces are not included in the response (when some items of the batch have failed) the request itself is treated as success, which it should. Thus the serialization of response doesn't use the error_trace request parameter which, for the proper serialization of stack traces should be converted to rest.exception.stacktrace.skip parameter. So, rest.exception.stacktrace.skip uses fallback which is true (i.e. skip printing stack traces).

This change properly transforms error_trace to rest.exception.stacktrace.skip in all cases, so all REST requests serialize stack traces consistently as per REST request parameters.

Related Issues

Resolves #19945

Check List

• Functionality includes testing.
  - [ ] API changes companion pull request created, if applicable.
  - [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 2cd3982: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 75.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.27%. Comparing base (a2e1fa2) to head (819b07e).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...opensearch/rest/action/RestToXContentListener.java	60.00%	1 Missing and 1 partial ⚠️
.../main/java/org/opensearch/rest/RestController.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #19985   +/-   ##
=========================================
  Coverage     73.27%   73.27%           
+ Complexity    71553    71540   -13     
=========================================
  Files          5789     5789           
  Lines        327144   327153    +9     
  Branches      47156    47157    +1     
=========================================
+ Hits         239715   239731   +16     
+ Misses        68235    68222   -13     
- Partials      19194    19200    +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[owaiskazi19]

Overall LG. Can we also add a rest based test https://github.com/opensearch-project/OpenSearch/tree/main/rest-api-spec/src/main/resources/rest-api-spec/test/bulk?

[fdesu]

@owaiskazi19 thanks for the quick review! I've smoothen rough edges and added more rest tests (also to _mget, _msearch, _mtermvectors). Would you please take another look?

[github-actions]

❕ Gradle check result for 1731418: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❌ Gradle check result for 6855608: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 91995ff: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[fdesu]

last commit failed with:

click

> Task :client:rest-high-level:spotlessJava
> Task :server:spotlessJava
build complete, generating: /var/jenkins/workspace/gradle-check/search/build/67466.tar.bz2

[Incubating] Problems report is available at: file:///var/jenkins/workspace/gradle-check/search/build/reports/problems/problems-report.html

FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':libs:opensearch-plugin-classloader:spotlessJava'.
> java.io.IOException: Failed to load eclipse jdt formatter: java.lang.RuntimeException: java.io.EOFException

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to generate a Build Scan (powered by Develocity).
> Get more help at https://help.gradle.org/.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':libs:opensearch-nio:spotlessJava'.
> Cannot fingerprint input property 'stepsInternalEquality': value 'com.diffplug.spotless.ConfigurationCacheHackList@2154da26' cannot be serialized.

merged latest main just in case

[github-actions]

❌ Gradle check result for f27f08b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[fdesu]

The EhCacheDiskCacheTests#testComputeIfAbsentConcurrently has failed

> Task :plugins:cache-ehcache:test

REPRODUCE WITH: ./gradlew ':plugins:cache-ehcache:test' --tests 'org.opensearch.cache.store.disk.EhCacheDiskCacheTests.testComputeIfAbsentConcurrently' -Dtests.seed=61C1CA88B6AE6E7C -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=zh-Hans-SG -Dtests.timezone=Africa/Monrovia -Druntime.java=25

EhCacheDiskCacheTests > testComputeIfAbsentConcurrently FAILED
    java.lang.AssertionError: expected:<1> but was:<2>
        at __randomizedtesting.SeedInfo.seed([61C1CA88B6AE6E7C:E07ABDBC48E12612]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.cache.store.disk.EhCacheDiskCacheTests.testComputeIfAbsentConcurrently(EhCacheDiskCacheTests.java:422)

but the #16159 is closed. Perhaps it needs to be reopened... 🤔

[owaiskazi19]

Task :server:spotlessJava

Did you run ./gradlew spotlessApply?

[fdesu]

Task :server:spotlessJava

Did you run ./gradlew spotlessApply?

@owaiskazi19 I’m pretty sure I did 😅
Anyway, spotless’s classloader failed to load eclipse jdt formatter. If violations were in formatting or so, it usually prints the diff for what went wrong.

[github-actions]

✅ Gradle check result for 819b07e: SUCCESS

[owaiskazi19]

LGTM. Thanks @fdesu for adding more tests for other apis as well. @andrross another look?