Dealing with High Traffic Sites - my notes distilled

This post was inspired by recent crashes of sites like radiojamaica.com (they seem to have found their feet again though), the Trinidad Guardian and Express during the Trinidad and Tobago elections.

Disclaimer: this is based on research and "on the ground" experience. I'm sure there are persons who have significantly more experience than me.

If these strategies seem overwhelming, then grab an expert to implement them for you. Also expect to have to spend a bit.

The following are server administration strategies for dealing with high traffic sites. This hardly has anything to do with what CMS you use, the principles are the same, this has to do with proper deployment for high traffic sites. If you have to deal with significant traffic here are some of the things you'll want to do:

Big Picture

1. Start monitoring immediately (assuming you aren't measuring already), get as much data as you can on memory usage, load etc...

2. Throw hardware at it (more memory etc...)

Strategies

Here are the important strategies:

• load balancing
• monitoring (monitor memory usage, monitor spikes etc...)
• sql sharding (breaking up your database) (see: http://www.slideshare.net/RockeTier/how-sharding-turned-mysql-into-the-internet-defacto-database-standard-moshe-kaplan-rocketier)
• go asynchronous (do like a flickr engineer, queue whatever can be queued)
• serve as much as you can statically, pictures, files, videos
• caching - I recommend Varnish (I put this last, because it shouldn't be used to gloss over underlying problems. Pun intended)

One of the best presentations I've seen on this matter was done by Elizabeth Leddy at the 2009 Plone Conference.

update (June 28, 2010): Check for misconfigurations. The worst mistake I made recently was pointing my cachesetup at the wrong port for the my proxy cache, so of course none of the sites were actually benefitting from the caching.

Configuring Munin.zope - message to future me

When working with munin.zope your buildout should look like this:

parts =
...
munin

eggs =
...
munin.zope

[munin]
recipe = zc.recipe.egg
eggs = munin.zope
scripts = munin=munin
arguments = http_address='${ports:client1}',ip_address='${hosts:client1}', user='${users:credentials}'

The arguments line was not very clear to me. Here's how it works.

'http_address' is really the port number (e.g. 8080) that your zope runs on.

'ip_address' can be an ip address or domain name (e.g. localhost, or 1.2.33.44)

'user' is really the user credentials in the form 'username:password'.

So future me, hope that helps to save you a few days of knocking your head on the wall.

Sending mail from EC2 Servers

Sending mail from EC2 servers, is for the most part the same as sending email from any other dedicated server. Setup Postfix and then use the "sendmail" provided by postfix. In your web applications just use "localhost" as your smtp server. This works, except that Amazon EC2 IPs are now blacklisted by Spamhaus, so corporations that fight spam based on Spamhaus info will block all your messages.

That's exactly what happened to us. I'm not the only one having email sending issues with EC2, there are several articles about the Spamhaus, EC2 issue.

Solution
Use an SMTP server external to EC2 for handling mail sending. I first tried to use gmail, but gmail alters the sender address, so that all messages appear to come from gmail, not quite what I needed.

1. Add a relayhost entry to postfix (for me this was in /etc/postfix/main.cf)
   relayhost = mail.otherserver.com
2. On the host that will be acting as a relay (in this example mail.otherserver.com also in the /etc/postfix/main.cf file), I added the reserved ip address for my EC2 instance to the mynetworks list
   mynetworks = 127.0.0.1/8 11.20.30.40 #where this represents the ip of my ec2 server

That's it.
After running /etc/init.d/postfix restart on both servers, no more blacklisting issues.

The problem with this is that I end up needing to have a server outside of the cloud, which brings a single point of failure and works against the scalability of cloud based solutions.

Hmmm, wonder if I could run my smtp server in Google App Engine, somehow.

Growing an XFS partition on Amazon EBS

Amazon EBS is Amazon's Persistent Storage service. I think of it as the ability to have a "thumbdrive in the clouds". I can then "attach" that drive to an Amazon EC2 instance (a machine in the clouds).

Today I needed to add space to a filesystem. The filesystem was 10G XFS partition and I wanted to make it 30G. (don't try this if you're not using XFS, some of these notes are XFS specific).

With the help of Elastic Fox this is pretty easy. Also make sure that you have the xfs_growfs application installed on your server. On Ubuntu/Debian 'apt-get install xfsprogs'.

update: You can do all of this with Amazon Web Console now, Elastic Fox is not the recommended choice.

1. Under ElasticFox > Volumes and Snapshots, I selected the Volume I wanted to work with and chose "create a new snapshot from this volume"
2. Under ElasticFox > Volumes and Snapshots, I selected the newly created snapshot and chose "create new volume from this snapshot" (making sure to specify a size of 30G).
3. Then I detached the old volume and attached the new volume (in this case to /dev/sdp)
4. Finally, I logged in to the server and ran the xfs_growfs command 'xfs_growfs /dev/sdp', my filesystem "magically" grew to be 30G.

I've been working, on and off with EC2 for a few months now. We now have clients hosted on EC2 based infrastructure. Amazon's Elastic Compute Cloud (EC2) is a utility that makes it possible to run virtual machines (Linux,Windows and others) on remarkably powerful infrastructure provided by Amazon.

update:
Recently I tried attaching my new volume to a different device '/dev/sdg' in this case. When I tried to mount the new device I got a 'wrong fs' error. The problem is that I was not able to mount the new device while another device with the identical UUID existed. The solution was to run the following command which generates a new UUID for my new device:

xfs_admin -U generate /dev/sdg

Adventures with Supervisord

Supervisord has a companion program "supervisorctl". Turns out that supervisord does not reread its configuration file unless you forcefully restart it. My initial thought was that it might be enough to change the configuration and then simply run a command like "supervisorctl restart instance1". Unfortunately my new settings were not detected.

(wasted nuff time...)

In the end I had to kill supervisord then restart it.

Reflecting on this, it does make sense, the role of supervisorctl is controlling and interfacing with supervisord, not to look behind its back for new settings.

NGINX entity too large, no more

Sometimes a simple sentence can make a world of difference. It did that for my nginx configuration this morning. Since setting nginx in front of Plone, I've had an "413 entity too large" error for files bigger than 1 MB.

My configuration uses a special conf/proxy.conf file which looks like this::

# proxy.conf
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 30m;
client_body_buffer_size 2m;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 2m;
proxy_buffers 4 1m;
proxy_busy_buffers_size 2m;
proxy_temp_file_write_size 2m;

The wrong way
Previously my 'server' section in my nginx.conf used to look like this:

server {
listen 1.2.3.4;
server_name example.com www.example.com;
location / {
rewrite ^/(.*)$ /VirtualHostBase/http/www.example.com:80/example/plone/VirtualHostRoot/$1;
}
location /VirtualHostBase/ {
proxy_pass http://sharedhosting;
include conf/proxy.conf; #this is apparently a bad place to put this
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}

}


The right way
So I read the following comment at the slicehost forums:

"Try moving the directive from inside the location block to just inside the server block."


So now I've moved it to "just inside the server block" so my 'server' section in my nginx.conf now looks like this:

server {
listen 1.2.3.4;
server_name example.com www.example.com;
include conf/proxy.conf; #now it works, phew!
location / {
rewrite ^/(.*)$ /VirtualHostBase/http/www.example.com:80/example/plone/VirtualHostRoot/$1;
}
location /VirtualHostBase/ {
proxy_pass http://sharedhosting;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}

}