[Backport 2.x] Add X-Opaque-Id to search request metadata for query insights (#13374)

[ansjcy]

Manual backport PR for #13374

(cherry picked from commit 9d3cf43)

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❕ Gradle check result for dcb30c3: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 54.54545% with 5 lines in your changes missing coverage. Please review.

Project coverage is 71.39%. Comparing base (0dd892c) to head (dcb30c3).
Report is 315 commits behind head on 2.x.

Files	Patch %	Lines
...action/search/SearchRequestOperationsListener.java	0.00%	3 Missing ⚠️
.../insights/core/listener/QueryInsightsListener.java	80.00%	0 Missing and 1 partial ⚠️
...opensearch/action/search/SearchRequestContext.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #14078      +/-   ##
============================================
+ Coverage     71.28%   71.39%   +0.11%     
- Complexity    60145    61497    +1352     
============================================
  Files          4957     5045      +88     
  Lines        282799   289021    +6222     
  Branches      41409    42211     +802     
============================================
+ Hits         201591   206359    +4768     
- Misses        64189    65287    +1098     
- Partials      17019    17375     +356     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

I think there might be some thread context related failures on main so let's get to the bottom of that first: https://build.ci.opensearch.org/job/gradle-check/40215/testReport/junit/org.opensearch.recovery/RelocationIT/testIndexSearchAndRelocateConcurrently__p0___cluster_indices_replication_strategy___SEGMENT___/

[jed326]

This shouldn't be causing any of the test failures as all the changes are in the query plugins insight which isn't being enabled in the tests. The culprit is probably this one: #13172

[ansjcy]

Fixed the test failure in #14084