Friday, July 3, 2015

Downtime Fri 03 Jul 2015, ~12:30-13:30 UTC

After much puzzling around MySQL, it seems apache2 was hammering the living my-goodness out of the MySQL instance on apache1. There is no obvious reason for this to have occurred ... but I disabled Apache on apache2 on the assumption that flying on one engine beats crashing on two. Investigation proceeds, further hiccups may occur.

Update: The answer: MySQL on apache1 saw apache2 as coming from its internal IP, so connections from it hung when it tried a reverse DNS lookup on this IP. Simple fix: skip-name-resolve in my.cnf.