So I had some server unpleasantness that I noticed after I posted my 2014 Goals post. The site just froze and logins were barely possible.
I immediately SSH’d into the prgmr.com server, but was greeted by an extremely slow console. I actually had to access the VPS Xen configuration menu and do a hard shutdown of the instance. When I rebooted and accessed the server through the out-of-band console, I was getting out of memory errors galore.
The culprit? A combination of MySQL, ClamAV and Amavisd.
After disabling ClamAV and Amavisd, I was able to stabilize the server, but the 1GB RAM VPS was still dangerously low on memory with 10-12 MySQL threads that I couldn’t clear.
So I took the unprecedented move to rebuild the server. Two days later I am running fine again. This time, I was a little bit careful of the installed services–with an emphasis on pruning unneeded services.
Hopefully I won’t ever have to do it again.