Errors when handle concurrent requests [BUG]

[mengweieric]

Describe the bug
When different requests hitting the same endpoint _plugins/_ppl concurrently, sometimes all the requests get the same responses.

To Reproduce
Steps to reproduce the behavior:
If you curl endpoint with two different requests concurrently, sometimes you are getting correct responses, but sometimes you are getting the same response.

`
curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_plugins/_ppl -d "{ 'query': 'source=opensearch_dashboards_sample_data_flights' }" | jq ".size" & curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_plugins/_ppl -d '{ "query": "source=opensearch_dashboards_sample_data_flights | stats count() by span(timestamp, "1h")" }' | jq ".size"
[1] 25313 25314
1008
1008
[1] + done curl -s -H 'Content-Type: application/json' -XPOST -d | jq ".size"

curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_plugins/_ppl -d "{ 'query': 'source=opensearch_dashboards_sample_data_flights' }" | jq ".size" & curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_plugins/_ppl -d '{ "query": "source=opensearch_dashboards_sample_data_flights | stats count() by span(timestamp, "1h")" }' | jq ".size"
[1] 25334 25335
1008
200
[1] + done curl -s -H 'Content-Type: application/json' -XPOST -d | jq ".size"
`

Expected behavior
Always getting the correct responses for different requests under concurrent context.

Plugins
SQL

Host/Environment (please complete the following information):

• OS: MacOS Catalina
• Version 10.15.7

[chloe-zh]

I reproduced this issue in my local cluster, and located the issue within the plugin:

https://github.com/opensearch-project/sql/blob/main/plugin/src/main/java/org/opensearch/sql/plugin/rest/RestPPLQueryAction.java#L163

  @Override
  protected RestChannelConsumer prepareRequest(RestRequest request, NodeClient nodeClient) {
    ...
    return channel -> pplService.execute(pplRequest, createListener(channel));
  }

Debugged the ppl request around, and found the requests intaken by the channel are of different request instances but sometimes they have the same content.
Looks like the pplService does not handle the incoming request very well probably they are they are always consuming the same node client.

[dai-chen]

@mengweieric @chloe-zh Is span function merged or still in personal branch? I tried similar commands but cannot reproduce the issue locally. Could you share which OpenSearch version and branch was used? Thanks!

[chloe-zh]

@dai-chen Hi Chen, the span is in personal branch, but it seems not related to the span functionality, because I didn't use the branch but also reproduced the issue. Here is the command I used with curl in cli:

curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_plugins/_ppl -d "{ 'query': 'source=opensearch_dashboards_sample_data_flights' }" | jq ".size" & curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_plugins/_ppl -d '{ "query": "source=opensearch_dashboards_sample_data_flights | stats count()" }' | jq ".size"

[chloe-zh]

@dai-chen the branch is here: https://github.com/chloe-zh/opensearch-sql/tree/observability/poc/ppl-stats

[dai-chen]

@dai-chen Hi Chen, the span is in personal branch, but it seems not related to the span functionality, because I didn't use the branch but also reproduced the issue. Here is the command I used with curl in cli:

curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_plugins/_ppl -d "{ 'query': 'source=opensearch_dashboards_sample_data_flights' }" | jq ".size" & curl -s -H 'Content-Type: application/json' -XPOST localhost:9200/_plugins/_ppl -d '{ "query": "source=opensearch_dashboards_sample_data_flights | stats count()" }' | jq ".size"

Okay, I tried same command on different index but cannot reproduce the issue. Will double check the code. Thanks!

[dai-chen]

The root cause seems here: https://github.com/opensearch-project/sql/blob/main/plugin/src/main/java/org/opensearch/sql/plugin/rest/RestPPLQueryAction.java#L100

For some reason, pplRequest is field member instead of a local variable in method. Remove it and the concurrent issue seems gone. Will prepare PR once confirmed.