[BUG] Indexing a document through the bulk api bypasses the 512 byte _id size limit

[trend-adam-kinnell]

Describe the bug
There is currently a hard limit on the size of a document _id field of 512 bytes when indexing a document. In this case we get an error like below.

{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: id [...] is too long, must be no longer than 512 bytes but was: ...;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: id [...] is too long, must be no longer than 512 bytes but was:...;"},"status":400}

However, we are able to use the _bulk api to create a new document that bypasses this limit. The document is created successfully and no error is returned.

Of particular interest is that reindexing will fail for any documents with an _id longer than 512 bytes. As a workaround we are able to exclude these documents with a query.

To Reproduce
Steps to reproduce the behavior:

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" --name opensearch-node -d opensearchproject/opensearch:latest
curl -X PUT "https://localhost:9200/myindex" -ku admin:admin

# This fails as the _id field is longer than 512 bytes
curl -X PUT "https://localhost:9200/myindex/_create/aaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbc" -ku admin:admin -H 'Content-Type: application/json' -d'{}'

# This succeeds, despite the length of the _id
curl -XPOST "https://localhost:9200/myindex/_bulk" -ku admin:admin -H 'Content-Type: application/json' -d '
{ "update" : {"_id" : "aaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbbaaaaaaaaaaabbbbbbbbbbbc", "_index" : "myindex", "retry_on_conflict" : 3} }
{ "upsert": {}, "doc": {} }
'

Expected behavior
The same limitations on _id length should be applied no matter what method is used to index a document.

Plugins
Default

Screenshots
N/A

Host/Environment (please complete the following information):

• OS: Official Docker image https://hub.docker.com/r/opensearchproject/opensearch

"version" : {
    "distribution" : "opensearch",
    "number" : "2.6.0",
    "build_type" : "tar",
    "build_hash" : "7203a5af21a8a009aece1474446b437a3c674db6",
    "build_date" : "2023-02-24T18:57:04.388618985Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },

Additional context
We originally found and reproduced this issue on an ES 7.10 cluster, but we have confirmed it is still present in the latest version of OpenSearch (2.6.0 at time of writing).

It is likely that a fix to this issue would be considered a breaking change.

[trend-adam-kinnell]

As a workaround to allow us to successfully reindex, we added a query to exclude any affected records so the request looks like:

POST _reindex?slices=auto&refresh&pretty&wait_for_completion=false
{
  "source": {
    "index": "<SOURCE>",
    "query": {
      "bool": {
        "filter": {
          "script": {
            "script": {
              "source": "doc['_id'].value.length() <= 512",
              "lang": "painless"
            }
          }
        }
      }
    }
  },
  "dest": {
    "index": "<DESTINATION>",
    "version_type": "external"
  }
}

We also used this query to count the number of records present in an index that were over the size limit and would fail to be reindexed:

GET /myindex/_count
{
  "query": {
    "bool": {
      "filter": {
        "script": {
          "script": {
            "source": "doc['_id'].value.length() > 512",
            "lang": "painless"
          }
        }
      }
    }
  }
}

[dblock]

@AdamKinnell-TrendMicro Thanks for reporting this, great catch. Turning your sample queries into a failing unit test would be a great next step if you want to work on this. I think you're right, if we do fix this it will have to go on main only (3.0).

[andrross]

It's probably worth looking at how the single document index API and _bulk API had different validations. Specifically, are there any other limits not being consistently enforced? Also can the code be refactored so that future bugs like this cannot happen?

/cc @mch2

[ankitkala]

I took a look at the validation logic, we have a validation in place for create/index (IndexRequest here). So any request to index and bulk (with create) have a single source of validation.

But this logic is not enforced for bulk updates(UpdateRequest here). So if a user tries to update a doc which doesn't exist, he can use a longer id and it would succeed.

I've created a PR to have this logic in UpdateRequest.validate as well: #8039

PS: Ideally we 'd want the same validation for delete as well but decided against it as that can lead to failures in existing doc deletion(given that there already might be some docs exceeding the limit).