Skip to content
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/reference/how-to/size-your-shards.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -544,3 +544,48 @@ PUT _cluster/settings
}
}
----

[discrete]
==== Number of documents in the index shard cannot exceed [2147483519]
Comment thread
stefnestor marked this conversation as resolved.
Outdated


Elasticsearch shards reflect Lucene's underlying https://github.com/apache/lucene/issues/5176[index
`MAX_DOC` hard limit] of 2,147,483,519 (`(2^31)-129`) docs. This figure calulates
Comment thread
stefnestor marked this conversation as resolved.
Outdated
from the sum of `docs.count` plus `docs.deleted` as reported by <<indices-stats,Index Stats>>
Comment thread
stefnestor marked this conversation as resolved.
Outdated
per shard. Attempting to surpass this limit, e.g. via ingestion's <<index-modules-translog,translog>>,
will cause error
Comment thread
stefnestor marked this conversation as resolved.
Outdated

[source, txt]
Comment thread
leemthompo marked this conversation as resolved.
Outdated
----
Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the shard cannot exceed [2147483519]]
----

or this error's sister stack trace will appear

[source, txt]
----
Caused by: java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519
Comment thread
stefnestor marked this conversation as resolved.
Outdated
----

TIP: This calculation works off all docs within the shard, including nested docs.
Comment thread
stefnestor marked this conversation as resolved.
Outdated
Therefore, it's results are not guaranteed to line up to the <<search-count,Count API>>
Comment thread
stefnestor marked this conversation as resolved.
Outdated
which masks nested docs.
Comment thread
stefnestor marked this conversation as resolved.
Outdated

Encountering this error usually suggests an issue with sharding strategy being
out of compliance with the above recommendations on this page, specifically at least
<<shard-size-recommendation,aiming for shards of up to 200M documents>>.
Comment thread
stefnestor marked this conversation as resolved.
Outdated

Frequently, the `docs.deleted` for the index will be elevated that
<<indices-forcemerge,expunging deleted docs>> can work as a short-term stop gap to regain
ability to admin the index, e.g. for `my-index-000001` this command would appear
Comment thread
stefnestor marked this conversation as resolved.
Outdated

[source,console]
----
POST my-index-000001/_forcemerge?only_expunge_deletes=true
----
Comment thread
leemthompo marked this conversation as resolved.

This will kick off an asynchronous task which can be monitored via the <<tasks,Task Management API>>.
Comment thread
stefnestor marked this conversation as resolved.
Outdated
Comment thread
stefnestor marked this conversation as resolved.
Outdated

Longer-term, this error can usually be resolved by <<docs-delete-by-query,removing unneeded docs>>
from the index or by <<indices-split-index,Splitting>> or <<docs-reindex,Reindexing>> the index with
corrected shard strategy settings.
Comment thread
stefnestor marked this conversation as resolved.
Outdated