CincomSmalltalkWiki: Web Toolkit Benchmarks

We have done some load testing on the web toolkit which might be of interest.

Disclaimer: We do not claim that these are truly representative benchmarks. They are extremely simple, run on low-end hardware, do not test SMP or clustered performance, and are not truly representative of real applications. They give a first indication that the Web Toolkit can perform competitively, but if you want to know how it will perform on your application, try it yourself on something representative of your situation. If you have other benchmarks, we would be interested to post them.

Here are the details of the test environment:

Client = 800MhZ Toshiba laptop running Win2K
Server=No-name Celeron 466 running Red Hat 6.2

both with 256MB RAM
Network = 802.11b encrypted wireless network talking to a 10Mb hub

Two test scenarios (done using the very nice shareware tool WebStress 4.4) (a) 100 simultaneous users, 5 runs (b) 500 simultaneous users, 5 runs

Web Toolkit (using absolute IP address, hitting /servlet/AreYouAlive)
Apache, (using absolute IP address, hitting AreYouAlive.html (contents = 'success'))

Test (a) run 4 times

VW: 70-94 thousand hits/hour, average user wait time of 1.11 to 1.25 seconds.
Apache: 72 to 101 thousand hits/hour, average user wait time of 0.85 to 1.36 seconds

Test (b) run 3 times

VW: 136 to 159 thousand hits/hour, average user wait time of 3.9 to 4.1 seconds
Apache: 65 to 102 thousand hits/hour, average user wait time of 5.5 to 6.0 seconds.

It would seem that VW can scale pretty well, and in this test we were only running one application server - Apache runs 8 instances by default. This test isn't trying to compare VW to Apache as a webserver, merely trying to show that VW as an application server works well.

Back to Benchmarks

What were the threadpool settings of Apache? Note that the scaleablity of Apache is capped/managed by these settings.--Reinout Heeck

Apache in the default configuration as installed in Red Hat 6.2. If someone can tell me where to find the settings I can list them. Alan Knight.

Another benchmark, by Cees de Groot. Notes: This is comparing serving static content, either as HTML or as an SSP with no dynamic content. Apache was running on a different machine than VW.

Let me produce some figures. Client in both cases is a dual 800/PIII connected through 100Mbit (switched) to the servers. The same document was used, document size was 247 bytes in both cases.

1. Apache running on an IBM pizzabox (933/PIII, local SCSI drive)

Concurrency Level: 100 Time taken for tests: 9.629 seconds Complete requests: 5000 Failed requests: 0 Total transferred: 2878752 bytes HTML transferred: 1240928 bytes Requests per second: 519.26 Transfer rate: 298.97 kb/s received

---

Concurrency Level: 500 Time taken for tests: 9.119 seconds Complete requests: 5000 Failed requests: 0 Total transferred: 2877033 bytes HTML transferred: 1240187 bytes Requests per second: 548.31 Transfer rate: 315.50 kb/s received

2. VisualWorks running on an IBM server (Dual 733/XeonIII, RAID-5), file 'test.html', debugging switched off

Concurrency Level: 100 Time taken for tests: 17.103 seconds Complete requests: 5000 Failed requests: 65 (Connect: 0, Length: 65, Exceptions: 0) Total transferred: 1870365 bytes HTML transferred: 1218945 bytes Requests per second: 292.35 Transfer rate: 109.36 kb/s received

Concurrency Level: 500 Time taken for tests: 17.062 seconds Complete requests: 5000 Failed requests: 56 (Connect: 0, Length: 56, Exceptions: 0) Total transferred: 1873776 bytes HTML transferred: 1221168 bytes Requests per second: 293.05 Transfer rate: 109.82 kb/s received

3. VisualWorks again, but file is now called "test.ssp":

Concurrency Level: 100 Time taken for tests: 21.584 seconds Complete requests: 5000 Failed requests: 54 (Connect: 0, Length: 54, Exceptions: 0) Total transferred: 3103858 bytes HTML transferred: 1562936 bytes Requests per second: 231.65 Transfer rate: 143.80 kb/s received

---

Concurrency Level: 500 Time taken for tests: 20.872 seconds Complete requests: 5000 Failed requests: 449 (Connect: 0, Length: 449, Exceptions: 0) Total transferred: 2855985 bytes HTML transferred: 1438116 bytes Requests per second: 239.56 Transfer rate: 136.83 kb/s received

=== My conclusion: over the range of these tests, both seem to scale equally well, although VW seems to have some trouble at high loads (I'd need to see what these failed requests are). With a performance at 50% of Apache, I think VW does extremely well as an application server (and quite fine as a general webserver).

OBTW: these were direct connections. With the CGI adapter in between, performance goes way down (waaaay down to about 25% of the direct figures, in my tests)

=== Cees, unless you test Apache and VisualWorks on the same hardware and OS platform (as Alan did) you are not doing a valid comparison. Please take your measurements again using the same platform for both. Eliot Miranda

I've been running a series of tests using httperf version 0.8. I'm putting them on a separate page Web Toolkit Benchmarks-CDS as I get time. David Shaffer