Balancing server load on traffic-rich sites

Webmasters are unlikely to go deeply into server settings and technical intricacies - this is what administrators are supposed to do. However, you have to understand the logic, at least to be able to explain how you want your system set up and for what purposes.

Let us take a server hosting a CJ site as an example. While your CJ is not over 100K daily, you have few things to worry about. Any P4 CPU will handle it even set up by default. If the server load keeps growing, your CPU will have a hard time, and there might be not enough RAM. Your site will work slower, the server can freeze. You lose traffic - and money.

There is a simple method which increases server productivity without software upgrade. All you need to do is separate the loads generated by dynamic content (scripts) and static content (pictures).

In a typical configuration you have an Apache daemon^[?] which processes the content as goes to your visitors. It also processes queries to PHP and CGI scripts. The more PHP functions (php, geo, modrewrite etc.) are used in the HTTP-daemon, the more RAM it occupies, and CPU time, too. To display static content (pics, movies, archives etc.) rich Apache functionality is excessive, with all its features and modules. It is enough to have a fast and light-weighted daemon with limited functionality - nginx, for instance.

To implement this you need 1 free server IP. You have to tell Apache to ignore IP so that the daemons don't conflict. Then you install nginx to this IP and provide it with a path to the directory with sites - telling your administrator to do this is recommended.

You as the webmaster have just to change the URL to the thumbs in your rotator so that they are loaded through nginx, not Apache. Have a look at 6fetish9.com as an example. In Streamrotator (streamscripts.com) it looks like this:

You change this in subtemplates:
<img src=http://6fetish9.com/streamrotator#THUMB# ...>
to
<img src=http://194.187.99.117/6fetish9.com/streamrotator#THUMB# ...>

On the picture you see "thumb properties"- Location points not to the domain itself, but to the IP of the static server which contains the graphics.

After you follow these simple steps you'll experience a noticeable decrease in CPU and RAM load. If you had an overloaded server when you moved the daemons, you'll also see an increase in productivity.

If your server is over 500K daily, you should seriously think about moving the thumbnail traffic to another server. In this case these will be your steps:

• Buy a server to host graphics, nothing too serious, just a P4 for up to 100Mbit of static traffic - if you use nginx.
• Set up the content synchronization:
  • use rsync^[?] to copy not all the content but only the content which changed. 5Gb of CJ content takes about 2-20 minutes to synchronize depending on the number of content updates on the original server (we synchronize our content once per hour using cron).
  • set up nginx on the static server;
  • give nginx the root directory where the synchronized content goes.
• A webmaster just has to update the URLs to the thumbs.

If you had a similar static scheme nginx on that server where Apache is, to move the thumb traffic to the new static server you just have to replace the IP from which the pics load in the rotator database.

To illustrate the advantages of this approach we made a test with CJs that really exist and operate.
Our experiments were made on a 1-CPU Athlon server with 1Gb of RAM. There were 14 minor CJs on the server with 160K of daily traffic total.
Before the test the server worked in this way, processing only the scripts and loading graphics from a separate static server by webazilla.com.

Here are the Mrtg and top parameters before the test:

Note:
top - a standard *nix tool for basic system resource monitoring, pay attention to load average on the pic.
load average - this shows CPU load, current and average for 5/15 minutes. Normally on loaded servers this is not more than 3. If the number reaches 10 it means the server is overloaded and is likely to crash.

I started the test on 14:50 and switched the thumb traffic to the same Apache which processed the scripts.

On this screenshot of mrtg it is clearly seen that the connection load jumped from 2 to 10 Mbit. It is also obvious that since the test was run and until the server crashed the traffic decreased - it means that the overloaded server could not process everything and lost visitors.

As the test was in progress I checked whether the domains are alive every 5-10 minutes to stop the test if the server dies completely.
10 minutes since the start

25 minutes since the start

35 minutes since the start

In 15:35 (45 from the start) the server almost stopped responding to requests, only the top data sometimes refreshed. The sites did not load, shell commands did not execute.

15:44 the server is up due to the fast support service.

Immediately after rebooting the server I changed the settings so that the thumbs load from the same server but this time via nginx. The traffic amount was almost restored and started growing - this is typical for a CJ after a minor crash.

Here's the top screenshot made 1 hour after the server was up.

Comparing this screenshot with the first one you'll see that the load when using a local static server is still heavier that using a remote server. However, this load is not critical for CJ sites.

In 17:10 I switched the static traffic back to the remote server, and the server kept running in the default mode.

This is a graph of server load, the server which hosted the pics. The decrease means no traffic which happened when we switched servers.
This graph clearly shows that after the tests were finished the connection load was restored to its original value.

In conclusion I will give screenshots of top and mrtg of my static server from which 6 servers hotlink the thumbs.

There are no working CJs on this server, just content (the thumbs), hotlinked through nginx.

What conclusions can we draw from this test?
By simple switching of the thumb traffic to a local static daemon you easily turn a server overloaded till it crashes into a stable system with a load not exceeding normal.

Apart from optimizing server usage, these methods save our money. We recommend more expensive and reliable servers to host scripts. For thumb traffic your server may be not that expensive and not that fast. You could easily host 100Mbit of static content on a 1-CPU P4 with 1Gb of RAM.

Mitch
Professional administrator mitch @ aka-root.com