red alert!
We were recently getting e-mails from our database-guard script:
From: root@mx.byte.nl (Cron Daemon) To: tech@byte.nl Subject: Cron /root/bin/sqlslayer.pl Alert! Defcon red for database8.c1 !! (228/750 conns) Saving processlist & user stats to /tmp/sqlslayer/20090825152504.database8.c1.processlist /tmp/sqlslayer/20090825152504.database8.c1.userstats
Due to a bug in the version of MySQL we are using one database user was able to block all other connections from executing queries.When this happened, the number of open connections would grow rapidly. That’s when our sql-slayer script comes into action and kills all inactive or long lasting connections to spare the server resources (ususal number of connections is 20-50, here it was 250 and growing).
It’s quite annoying for people who get the ‘MySQL server has gone away’ message without apparent reason. On the other hand, not killing the queries caused the server to jam totally within minutes.
The reason was bug: https://bugs.launchpad.net/ourdelta/+bug/352840.
We have just patched the servers and we hope that the cause of the alarms (and broken queries) is now solved.


Written by wojtek in:
Written by melanie in:
Written by allard in: 
Written by Gertjan Oude Lohuis in: 
Written by gruus in:
Written by Cipriano Groenendal in: 