ELK oops

Moral of the story:  if you want to retain 7 days of logs aggregated across all of your APIs... make sure u have enough disk space for all the Elasticsearch indices.

Notice the 'free disk space' bottoms out and then once the system is hosed there is disk space reclaimed by the daily 'cleanup script'.  One index gets purged a day until they are all gone (and no new ones were created cuz the system was hosed due to the crash).