@jainankitk Do you know someone who would be interested in looking this further?

Let me look into it.

The diff you are mentioning in the issue is old one. This is fixed in all of the branches as part of this PR: 3b4e11d#diff-4e901163f39dfc072ae4d7f93a43f0958bdfdf2bb82f044b2ec9f3ae13fa66df

The issue was when the header is not set in thread context then it would return null for queryGroupId because the threadContext itself is not null

@kaushalmahi12 The reproduction steps shared by @doug-numetric in this issue work for me. I'm able to generate that log statement on a min distribution install of OpenSearch 2.18.

I think the mechanism to set the queryGroupId for scroll action is missing. To mitigate the logging from QueryGroupTask class, we can do the following

curl -XPUT "localhost:9200/_cluster/settings" -H "Content-Type: Application/json" -d '{"persistent" : { "logger.org.opensearch.wlm.QueryGroupTask": "ERROR" }}'

@doug-numetric I suppose this is only happening for scroll APIs?

@kaushalmahi12 Yes, as far as I can find, only the scroll APIs are affected.

I'm also seeing this error popup on a fresh cluster running the re-index API.

this problem also appears with simple requests against the /search API when you just send enough of them at the same time. tested using a stock 2.18.0 release (i disabled the security plugin for simplicity).

1. create a test index:

curl --location --request PUT 'http://localhost:9200/testindex' \
--header 'Content-Type: application/json' \
--data-raw '{
    "mappings": {
        "properties": {
            "testfield": {
                "type": "text"
            }
        }
    }
}'

2. add test data (probably not even needed):

curl --location --request POST 'http://localhost:9200/testindex/_doc' \
--header 'Content-Type: application/json' \
--data-raw '{
    "testfield": "foo"
}'

3. run a lot of GET requests against the API; i used oha for this, letting it run for 2 minutes with the default 50 parallel requests:

oha -z 2m http://localhost:9200/testindex/_search

i got the message 14x in the OS log during this time.

FYI @kaushalmahi12. There's a couple more cases here beyond scroll. @rursprung's case suggests there might be a race condition during search triggering this case.

Let me see what is causing this. Thanks rursprung bringing this to our attention.