Skip to content

Prevent Elasticsearch OutOfMemory issues from filling up disks #351

@GUI

Description

@GUI

Earlier today our ElasticSearch servers hit an OutOfMemory issue. By default, Elasticsearch is configured to do a heap dump whenever it hits an out of memory issue. When the out of memory error happened earlier today, the default disk partition used for writing the heap dump happened to be smaller than the heap dump file, which resulted in that disk partition completely filling up. Other processes on the server weren't affected by this partition filling up, but this still isn't great to have a fill disk partition.

Since we're probably not too interested in inspecting the heap dumps in these out of memory situations, I've disabled the heap dumps from occurring in the future, which should prevent the disk partitions from filling up: NREL/api-umbrella@920621e

The real underlying issue is why Elasticsearch hit an hard OutOfMemory error to begin with, but it looks like there was an odd culmination of separate admins from different agencies all performing analytics queries over long durations in the admin at the same time. If this happens again, we can try doing some more elasticsearch tuning, but the better answer is probably the fabled analytics task (#235... sigh.. which, unfortunately I haven't had time to wrap up, but I'll try to revisit that soon to see if we can finally, finally, finally push that over the finish line).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions