I have been noticing something interesting lately on some of the sites that I run that cater mainly for mobile devices. The servers have been reporting zero load but no-one can connect to the site and the server will not serve any pages.
Luckily I read an article recently describing this problem and it is a very easy problem to fix.
The problem is spoon-feeding requests
The problem I have been seeing is web server spoon-feeding. Normally your web server will be a high spec machine with lots of bandwidth but requests come from a home computer or an iPhone on limited broadband with lower bandwidth so the bottleneck is usually not your server.
Imagine that a normal web page is 50kb and someone on an iPhone makes a request to view your page, your web server will respond very quickly. However the iPhone on 3G network can only read the data in small packets a bit at a time, so your web server ends up spoon-feeding the 50kb a bit at a time. If your Apache server can handle a maximum of 80 clients at a time and you have 80 slow connections (e.g. mobile devices) your server will not accept any more connections and will refuse new requests.
The light, heavy approach
The way to solve this is to have a light web server as a load balancer (such as nginx) in front of your heavy Apache server, ideally on a separate server but you can do this even on the same one. Where as Apache will max out at 80 connections a light server should be able to accept a few thousand connections. So now your light web server can take care of the spoon-feeding and frees up Apache to handle new requests.
Another great reason to use a load balancer is that it makes scaling much easier if you ever need to scale.
I have now implemented this very simple, elegant architecture and it works really, really well. If you have any questions setting this up let me know and I will try and help you out where I can.