Web Toolkit Benchmarks-CDS

The Hardware

• Dell Inspiron 8500 laptop w/ 512M RAM
• 2Ghz Mobile Intel Pentium 4
• bcm4400 100Mb/s Ethernet adapter
• RedHat Linux 9 (kernel 2.4.21)

• Dual 800MHz PIII Dell Precision 620
• RedHat Linux 9 (kernel 2.4.20-8smp)
• httperf 0.8
• 3com 100Mbs NIC

• C3500XL Cisco switch 100Mb/s full duplex forced

The method

httperf was employed to deliver page requests for a number of fixed request rates ranging from 10 requests/second to 700 requests/second. No session information was maintained between requests (no cookies sent to server). Each request rate was sustained for 150 seconds so that the web server's reply rate could stabilize. During the request period, the reply rate was sampled at 5 second intervals (by httperf) and the average result was graphed using gnuplot as a function of attempted request rate. Each measurement was performed, when possible, using each of the following server software configurations

• apache httpd-2.0.48 for static page and apache+mod_perl (2.0) for dynamic content
• jakarta Tomcat version 5.0.16 (native HTTP server), JVM 1.4.2
• Jigsaw version 2.2.3, JVM 1.4.2
• VisualWorks 7.2 web toolkit HTTPServer (WaveHTTPRequestBroker)
• Squeak (3.6 image 3.6g-2 VM) + KomHttpServer 5
• STX + my Comanche 5 port

1k static page

In this case a 1024 byte html file was requested from each server. Note that the responses varried in length since each server presented a different MIME header in the reply. Header lengths varried from about 500 bytes to 800 bytes. I am in the process of rewriting this benchmark so that a different file is used with each server so that each response is 1k byte. The current results are useful, however, in assessing how well the server stands up to a high request rate.

Comments:

• The VisualWorks WebToolkit server no longer dies at high connection rates
• The WebToolkit server causes the file descriptor problem described in this note
• In my current configuration I could not saturate any of the servers except VisualWorks so their behavior when reaching their maximum request servicing rate could not be determined for the static file case.

Small Active document (JSP, SSP etc)

In progress...

Moderate Active document (JSP, SSP etc)

In progress...

Small Servlet

In progress...

Moderate Servlet

In progress...

Note: Since httperf was configured with a 5 second timeout, a server which does not reset connections (such as the VisualWorks server) will cause a backlog of open sockets and eventually cause the client to run out of outgoing request sockets causing the actual request rate to drop below the attempted request rate. I have not found a convenient way to represent this on these graphs so I've made annotations where appropriate.

VW7.0 benchmarks -- out of date

(A note that these benchmarks are out of date, in that they apply only to VW 7.0. They did point out two issues - one was memory policy issues, where millions of one-shot requests were generating sessions that got promoted into old space before they expired, and were not getting cleaned up until the server really started to run out of memory (by which time it was quite possibly swapping and getting terrible performance). The other was that many unix systems, notably including Linux, have a hard limit of 1024 simultaneous open socket connections for a process. If that limit is exceeded, you get an error that masquerades as an "out of file handles" error, and the error handling was not dealing properly with that exception during a socket accept. Both of these problems were dealt with for the 7.1 release, and with that the VW server scales up quite consistently with the other technologies shown. David is going to re-run the benchmarks against 7.1, but it will probably take him a while to get to it. Many thanks to David for pointing out these issues and helping us get them addressed. -- Alan Knight. Aug 11, 2003)

Hardware info

Server:

• 800MHz PIII Sony Viao PCG-FX250 laptop
• RedHat Linux 7.3 + all patches (incl. updated kernel)
• Integrated 100Mbs NIC

Client:

• Dual 800MHz PIII Dell Precision 620
• RedHat Linux 8.0 + all patches (incl. updated kernel)
• httperf 0.8
• 3com 100Mbs NIC

Benchmarks

• Small servlet test
• Active document technologies (JSP, SSP etc)
• Static file test

Small servlet

A simple servlet producing 93 bytes of data (resulting from directing printing a String into the response) was produced for each of the the following configurations:

• Apache 2.0.43 + mod_perl
• httpd.conf (supplied with apache and produced with vanilla configure)
• VisualWorks 7 (with 7.0a VM) + Web Toolkit + TinyHttpServer
• Memory Upper Bound = 128Mb (Server has 256Mb of RAM with > 128Mb free)
• Growth Regime Upper Bound = 32Mb
• Session Timeout: Normal=100s, Stressed=10s
• Jigsaw (java-based) web server
• with JDK1.4.1
• Precompiled Jigsaw classes
• JVM run without any command line arguments (I will experiment with the various GC and performance options later)
• maxClients=30, maxThreads=85, maxFree=15 (after some playing these seemed like optimal settings for my configuration)
• tomcat-4.1.12 (native HTTP server)

Note that the VisualWorks server becomes hopelessly bogged down when it's maximum connection rate is exceeded (effectively serving only a very small number of connections per second). The other servers in this test seem to meet some fixed reply rate once they become saturated with requests. I expect that the few outlying data points for tomcat at high connection rates might be related to problems with my configuration.

Note that calling the Apache+mod_perl case a "servlet" is a bit misleading since in all other cases, once the servlet class is loaded it is not loaded again. However, I do not know how to operate mod_perl in this fashion. As far as I know, mod_perl loads and compiles my perl script every time I access the page. This is more like an "active document technology" so I list the test results there as well.

Active document technologies (JSP, SSP etc)

Same configuration as above, this time using a JSP file. The file actually contained no Java or Smalltalk code (it was a static file). The apache+mod_perl configuration was exactly the one used above (the 93 byte servlet).

Here we can see that both tomcat and VisualWorks tank at relatively low connection rates. Both eventually threw "out of memory" exceptions and stop serving content (as should be clear from the graph).

Static file test

This test is designed to see how well the server can sustain high connection rates. I have found this result useful in characterizing how a server will perform when serving static content to a client. Note that "static content" often implies pages with lots of images causing large bursts of requests. Some servers fail under such circumstances (dropping requests or timing out). I have also benchmarked session oriented bursts

In this test I stress the server by making stateless (no session) connections at rates from 50 to 1000 per second. The failure point is determined by connection rate at which requests begin to be dropped.

Here is my shell script for running httperf:

#!/bin/sh
for i in `seq 50 10 1000`; do
   echo ${i}...
   httperf --hog --server 10.11.11.14 --port 80 --uri /index.html --rate ${i} --timeout=5 --num-con=`expr $i \* 150` | tee test_${i}
   sleep 5
done

My file index.html is 1024 bytes long. Some notes about this script

• I run 150 seconds, no matter what the connection rate. This is so that httperf can get a good estimate of the reply rate (according the httperf docs you should run for at 150 seconds).
• The sleep is presumably to allow the server to return to an inactive state (if said server differentiates between stress levels). The hope is that during 5s of inactivity, VW would do a compacting GC if needed. I also played with removing this sleep and the effect was minimal (most prominant only when the server was begining to become overwhelmed by the hits -- presumably due to TCP connections which were queued from the last test still being processed by the server).
• I allow 5 seconds for timeout. This is too long at high connection rates and I will look at various other possibilities once I get a more stable VW server (discussed more below)
• For the tests against apache it is useful to increment the connection rate by larger values (I used 50) in the `seq...` line.

VisualWorks TinyHttpServer server

I've noticed several problems with the Web Toolkit package:

1. It overrides two methods in the Wave package which previously used ensure: but changes them to valueOnUnwind: breaking the original semantics. I recommend considering changing these methods so that they use ensure:.
2. DelegatingIPServer's "fork a process for every request" policy results in problems when it can't keep up with the current request rate. The number of open file descriptors grows which can cause an unhandled exception (the accept: call in the server loop is not protected so getting a new file handle associated with the new socket can cause this exception). I recommend limiting the number of active processes and either blocking before accept: when there are too many active or accepting then closing the new socket immediately (this is what Jigsaw and apache do). I also recommend placing an exception handler around the accept:. Trusting a high ulimit is not a solution...
3. Under moderate load (near but below the maximum supported without missing connections), the VisualWorks image grows until a "memory emergency" occurs. At this point the server no longer functions. I don't yet understand the memory management policy but this doesn't seem reasonable under any conditions since the server could manage its memory by restricting the connections it handles and agressively garbage collecting. Jigsaw, on the other hand, maintained a stable memory profile no matter how great a load was presented to it.

Software configurations

Web server configurations (the numbers are used in the table below)

1. Apache 2.0.43
1. httpd.conf (supplied with apache and produced with vanilla configure)
2. Apache 2.0.43
1. highperformance.conf (supplied with apache and produced with vanilla configure)
3. VisualWorks 7 (with 7.0a VM)
1. TinyHttpServer
2. Web Toolkit
3. ProcessEnvironment isDevelopingOverride: false (very important)
4. VisualWorks 7 (with 7.0a VM)
1. TinyHttpServer
2. Wave FileResponder modified to support simplistic caching
3. ProcessEnvironment isDevelopingOverride: false (very important)
5. Squeak 3.2-4 Comanche (kom-5.0)
1. LTD server (max connections 10)
2. StaticFile module modified to support caching
6. ST/X comanche (my port of comanche to ST/X)
7. Jigsaw (java-based) web server
1. with JDK1.4.1
2. Precompiled Jigsaw classes
3. JVM run without any command line arguments (I will experiment with the various GC and performance options later)
4. maxClients=30, maxThreads=85, maxFree=15 (after some playing these seemed like optimal settings for my configuration)
8. VisualWorks 7 (with 7.0a VM)
1. LTDHttpServer -- limit number of simultaneous connections to 10
2. Web Toolkit
3. ProcessEnvironment isDevelopingOverride: false (very important)
9. VisualWorks 7 (with 7.0a VM)
1. With "Webtoolkit sendfile hack" by Cees de Groot in Cincom public store repository
2. LTDHttpServer -- limit number of simultaneous connections to 1000
10. Apache 1.3.27 + mod_fastcgi_2.2.12 + VisualWorks
1. Note: mod_fastcgi is not yet available for Apache2 (except in beta)
11. tomcat-4.1.12 (native HTTP server)
12. Apache 2.0.44 + proxy to VisualWorks
1. Proxying from apache to WebToolkit

Results

Config	Max conn/s (*)	Response time in ms	Behavior on failure (+)
1	>900 (a)	0.7-1.7	stable-reset (b)
2	>900 (a)	0.5-1.7	stable-reset (b)
3	120	4-8 (c)	crash (d)
4	290	2-6 (c)	crash (d)
5	260	2.9-4.2 (c)	stable-timeout
6	160	2.1-3.2	stable-timeout
7	460 (e)	1.5-5.0	stable-reset
8	130	5-9	stable-timeout
9	160	4.8-50	stable-timeout
10	70	11-15	crash (f)
11	380	1.2-2.9	stable-timeout

Notes:

(*) Max conn/s is the highest rate at which the server performed error free (all connections resulted in response within the timeout period).

(+) The failure behavior represents what the server does when I far exceed its ability to service all connections. If the server continues to function, it usually either times-out on requests it can't handle or it resets them (usually causing a browser to re-send the request).

(a) I could not produce a server failure with my configuration. There were a few failures due to problems on the client but they were not reproducable. Otherwise the server worked reliably up to the highest connection rate I could produce (as listed).

(b) The failure behavior of apache was determined by running on a considerably slower server platform, a PI-100Mhz. It became possible to exceed apache's connection rate threshold.

(c) Response time grew much larger near and above max connection rate

(d) The various error handling setting in the WaveServer GUI seem to have no effect on the handling of this exception. It always raises an unhandled exception dialog.

(e) This was the least reproducable of all of the results in this table. In some cases it had some failed connections at 390-400 connections per second but then worked above that (up to 460 connections per second). In most cases there were no more than 10-20 connection failures under 460 connections per second.

(f) At the reported connection rate, apache would begin complaining of zero-length headers in the response. It would also complain of timeouts (30seconds) and segfaults in the VW fast cgi gateway. After the test at around 100 cps, the VisualWorks image stops responding to user input (ctrl-Y won't revive it but Shift-Ctrl-Y will present the emergency evaluator)

Conclusions and comments

From the results that I have so far I have made a few conjectures. Please chime in if you think I've reached the wrong conclusion(s):

• The TinyHttpServer fails (throws an exception effectively rendering the server useless) at high connection rates due to OS limitations on the max number of allowed open file handles. Apache seems to avoid said restriction by using multiple OS processes and monitoring the number of open handles and refusing or delaying connections, if necessary, to avoid having too many open.
• Without caching, the Webtoolkit exacerbates the open file descriptor problem by opening the requested file for each request.
• In my opinion, since a reset results in most browsers re-trying a request, servers which reset connections which they can't possibly handle have a better chance of seeming responsive even under high stress. Servers which timeout when they cannot handle a connection will usually seem sluggish.

Session oriented benchmarks

Work in progress Please be patient as I work on this over my x-mas and new year's break.

This benchmark differs from the one above in the following ways:

1. Sessions are created at a specified rate
2. A scripted sequence of requests are made for each session
1. The client requests a page
2. As soon as the page is received several requests for "images" are made concurrently.
3. After a "think time" this process proceeds to the next page/set of images
4. when the end of the script is reached the session ends
3. If the server configuration supports keep-alive, that feature is used by the client (httperf)
4. The client keeps a single cookie for each session (and all versions of my app use this single cookie as the session key)

The application

This test was performed on configurations including mod_perl, Java Servlets/JSPs, Smalltalk Servlets/SSPs, and PHP. Essentially the same application was written for each of these languages/APIs.

Technical problems

Given the differences between the technologies used in this benchmark, several issues arrose:

• Controlling session expiration uniformly in each language/server was tough. I'd like to know when to expire a session but who ever "logs out" of a web app? Normally sessions expire after some timeout. At high rates of session creation, though, this will result in large memory demands on the server. Given that the lifetime of my benchmark (a few minutes) is shorter than any practical session timeout we cannot count on this mechanism to avoid a memory problem. The next possibility is to consider a session deactivation/reactivation mechanism (or use one supplied by the package) whereby the "least recently used" sessions, for example, are stored in on disk when memory usage becomes too high. For the purposes of this benchmark I report values for two cases:
• No one ever logs out (what happens to the server as its memory consumption rises?)
• Everyone logs out at the end of their script (ie the last page accessed is a logout page) and their session data is dropped at that point
• While Smalltalk Servlet/SSP and Java Servlet/JSP resulted in nearly identical apps, perl and PHP looked somewhat different. The Servlet/server page combination used a simple strategy: server pages were for display purposes only, and servlets were used to complete actions and then redirect to server pages for subsequent displays. This is a pretty common scheme although not as popular as the "ball of mud" server page approach. Since PHP is a document scripting technology, I moved servlet's over to php documents which completed their action and then did a redirect. For mod_perl I simply had some perl scripts generate page content while others completed actions and then redirected.
• I used an in-memory data source for these benchmarks. I'm working on using a PostgreSQL datasource for each app.

Software configurations

Web server configurations (the numbers are used in the table below)

1. Apache 2.0.39 + mod_perl