Trick: recovering from "no space left on device" issues with MySQL

Just read Ronald Bradford's post on an unnecessary 3am (emergency) call. I sympathize! Running out of disk space makes for some weird MySQL behaviour, and in fact whenever I encounter weird behaviour I verify disk space.

But here's a trick I've been using for years to avoid such cases and to be able to recover quickly. It helped me on such events as running out of disk space during ALTER TABLEs or avoiding purging of binary logs when slave is known to be under maintenance.

Ronald suggested it -- just put a dummy file in your @@datadir! I like putting a 1GB dummy file: I typically copy+paste a 1GB binary log file and call it "placeholder.tmp". Then I forget all about it. My disk space should not run out -- if it does it's a cause for emergency. I have monitoring, but sometimes I'm hoping to make an operation on 97%-99% utilization.

If I do run out of disk space: well, MySQL won't let me connect; won't complete an important statement; not sync transaction to disk -- bad situation. Not a problem in our case: we can magically recover 1GB worth of data from the @@datadir, buying us enough time (maybe just minutes) to gracefully complete so necessary operations; connect, KILL, shutdown, abort etc.