Happy Mongrels
Posted by Jesse Andrews on Oct 19, 2007
Scaling userscripts.org has been a fun journey. Thanks to monit, it has become a much easier journey.
You may notice on the 16th, memory usage decreased dramatically. Userscripts.org had a problem, the code that generates those nice colorized views of the scripts leaks memory. Massive amounts of memory. I'm not sure if it is the fault of coderay, hpricot, something I'm doing or ... but trying to colorize 20KB userscripts makes mongrels (application servers for rails) grow by hundreds of megabytes.
For a long time I would log in periodically and restart everything so that the mongrels didn't grow to consume more memory than we have. I messed up a few times in the last couple months and the server would start swapping and the site go down.
Not any more. Thanks to monit, I've got some simple rules that will keep the dogs at bay.
I have 10 rules like this in my /etc/monit/monitrc:
check process uso_mongrel_4000 with pidfile /home/deploy/app/shared/log/mongrel.4000.pid
group uso_mongrel
start program = "/usr/bin/mongrel_rails cluster::start -C /home/deploy/app/current/config/mongrel_cluster.yml --clean --only 4000"
stop program = "/usr/bin/mongrel_rails cluster::stop -C /home/deploy/app/current/config/mongrel_cluster.yml --only 4000"
if failed port 4000 protocol http
with timeout 10 seconds
for 2 cycles
then restart
if totalmem > 100 Mb then restart
if cpu is greater than 60% for 2 cycles then alert
if cpu > 80% for 3 cycles then restart
if loadavg(5min) greater than 10 for 8 cycles then restart
if 3 restarts within 5 cycles then timeout
The result is that every few hours, someone renders a page that causes mongrels to leak massive amounts of memory, but monit restarts them within a minute. Right now we are running 10 mongrels, so even if one is down, site performance isn't effected.
Deploying a new copy of the code is still easy:
- Check new code to my local git repository
- Push my changes to my git server
- cap deploy
To keep capistrano and monit from fighting I needed to switch from palmtree using mongrel_cluster to restart to using monit. I updated my deploy.rb file removing palmtree and adding:
set :monit_group, 'uso_mongrels'
namespace :deploy do
desc "Restart the Mongrel processes on the app server by calling restart_mongrel_cluster."
task :restart, :roles => :app do
sudo "/usr/sbin/monit restart all -g #{monit_group}"
end
end
Now those are some happy dogs!
You could comment on this post if you were logged in.
login to vote
Memory leak is a hard problem to fix. Well done!
login to vote
Yes monit rocks, used it a couple of years ago to do pretty much the same as you did.
Had a website built on Zope that had made it's purpose in life to kill the Sun Netra that it ran on and nagging me in the middle of the night with sms telling me to fix it.
Well... monit gave me my sleep back :-)
login to vote
very good idea