Skip to content

Expand How to tune for disk usage #25562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 21, 2017
Merged

Expand How to tune for disk usage #25562

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eac143b
Expand How to tune for disk usage
cwurm Jul 5, 2017
07280bf
Update shard size recommendation
cwurm Jul 7, 2017
9448918
Clarify shard recovery and segments
cwurm Aug 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs/reference/how-to/disk-usage.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,20 +135,44 @@ PUT index
--------------------------------------------------
// CONSOLE

[float]
=== Watch your shard size

Larger shards are going to be more efficient at storing data. To increase the size of your shards, you can decrease the number of primary shards in an index by <<indices-create-index,creating indices>> with less primary shards, creating less indices (e.g. by leveraging the <<indices-rollover-index,Rollover API>>), or modifying an existing index using the <<indices-shrink-index,Shrink API>>.

Keep in mind that large shard sizes come with drawbacks, such as long full recovery times.

[float]
=== Disable `_all`

The <<mapping-all-field,`_all`>> field indexes the value of all fields of a
document and can use significant space. If you never need to search against all
fields at the same time, it can be disabled.

[float]
=== Disable `_source`

The <<mapping-source-field,`_source`>> field stores the original JSON body of the document. If you don’t need access to it you can disable it. However, APIs that needs access to `_source` such as update and reindex won’t work.

[float]
=== Use `best_compression`

The `_source` and stored fields can easily take a non negligible amount of disk
space. They can be compressed more aggressively by using the `best_compression`
<<index-codec,codec>>.

[float]
=== Force Merge

Indices in Elasticsearch are stored in one or more shards. Each shard is a Lucene index and made up of one or more segments - the actual files on disk. Larger segments are more efficient for storing data.

The <<indices-forcemerge,`_forcemerge` API>> can be used to reduce the number of segments per shard. In many cases, the number of segments can be reduced to one per shard by setting `max_num_segments=1`.

[float]
=== Shrink Index

The <<indices-shrink-index,Shrink API>> allows you to reduce the number of shards in an index. Together with the Force Merge API above, this can significantly reduce the number of shards and segments of an index.

[float]
=== Use the smallest numeric type that is sufficient

Expand Down