Bdtjtk

Scenario:

• We use Elasticsearch & logstash to do application logging for a moderately high traffic system


• This system generates ~200gb of logs every single day


• We use 4 instances sharded; and want to retain roughly last 3 days worth of logs


• So, we implemented a "cleanup" system, running daily, which removes all data older than 3 days

So far so good. However, a few days ago, some subsystem generated a persistent spike of data logs, resulting in filling up all available disk space within a few hours, which turned the cluster red. This also meant, that the cleanup system wasn't able to connect to ES, as the entire cluster was down -on account of disk being full. This is extremely problematic, as it limits our visibility into what's going on -and blocks our ability to see what caused this in the first place.

Doing root cause analysis here, a few questions pop out:

• How can we look at the system in eg Kibana when the cluster status is red?


• How can we tell ES to throw away (oldest-first) logs if there is no more space, rather than going status=red?


• In what ways can we make sure this does not happen ever again?

Date based index patterns are tricky with spiky loads. There are two things to combine this for a smooth setup without needing manual intervention:

1. Switch to rollover indices. You can then define that you want to create a new index once your existing one has reached X GB. Then you don't care about the log volume per day any more, but you can simply keep as many indices around as you have disk space (and leave some buffer / fine tune the watermarks).


2. To automate the rollover, removal of indices, and optionally setting of an alias, we have Elastic Curator:
   
   
   
   
   • Example for rollover
   
   
   • 
     Example for delete index, but you want to combine this with the count filtertype
     
   
   
   
   

PS: There will be another solution soon, called Index Lifecycle Management. It's built into Elasticsearch directly and can be configured through Kibana, but it's only around the corner at the moment.

How can we look at the system in eg Kibana when the cluster status is red?

Kibana can't connect to ES if it's already down. Best to poll Cluster health API to get cluster's current state.

How can we tell ES to throw away (oldest-first) logs if there is no more space, rather than going status=red?

This option is not inbuilt within Elasticsearch. Best way is to monitor disk space using Watcher or some other tool and have your monitoring send out an alert + trigger a job that cleansup old logs if the disk usage goes below a specified threshold.

In what ways can we make sure this does not happen ever again?

Monitor the disk space of your cluster nodes.