More Seaside Testing
I ran some more load testing this morning, using a 5 minute load test rather than the simple one minute test. The test was pretty simple:
- Start the appropriate Seaside image
- Run the load test, which goes to the counter app, increments to 4, and then decrements back to 0
- That test runs over and over, for 20 (simulated) users, over 5 minutes
This morning, I ran the tests over my LAN, using a Windows box on the network, with the images all running on the MacBook Pro. The summarized results:
| Platform | Pages | Avg Sessions per Second | Avg Pages per Second | Starting Pages per Second | Ending Pages per Second |
| Seaside 2.7 on VW 7.5 | 2418 | 0.86 | 8.1 | 24.7 | 4.9 |
| Seaside 2.8 on Squeak | 10,819 | 2.1 | 36.1 | 117 | 5.8 |
| Seaside 2.8 on Squeak (some tuning) | 20,745 | 7.6 | 69.2 | 70.3 | 69.6 |
| Seaside 2.8 on VW 7.6 | 25,696 | 9.5 | 85.7 | 80.3 | 86 |
| Seaside 2.8 on VW 7.6 (some tuning) | 34,188 | 12.6 | 114 | 112 | 115 |
| Ruby on Rails 1.8.6 | 2225 | 7.4 | 7.4 | 6.9 | 7.4 |
The Squeak image I used was from the "One click experience" - VW 7.5 is available for free download (non commercial), and the VW 7.6 image is available to vw-dev members. Notice how the 7.6 image stayed very stable as it ran - the image ramped up, and then stayed stable. All three images were responsive as I ran the tests, and all there were in good shape at the end of the tests.
Updates:
The "some tuning" in VW amounted to executing this, saving, and restarting the image
ObjectMemory sizesAtStartup: #(10.0 10.0 5.0 10.0 5.0 5.0 5.0)
Caveat: Similar improvements are likely possible in other environments with appropriate tuning
No sooner did I post that, than John McIntosh sent me these suggestions for Squeak:
Smalltalk setGCBiasToGrowGCLimit: 64*1024*1024. Smalltalk setGCBiasToGrow: 1. SmalltalkImage current vmParameterAt: 5 put: 8000.
As you can see in the table, they made a difference. The Squeak image's UI was unresponsive at the end of the test, though
The Ruby comparison needs some explanation - I installed RoR 1.8.6 (via download and compile) on my MacBook Pro this morning. I then created the "Hello World" example here, with the greeting coming via the View. I then ran a load test for 5 minutes, where the test was simply to load the page 9 times in a row (as the counter app was hit 9 times, between load and increment/decrement). So ultimately, RoR was doing a lot less - and doing it a lot slower :)
Technorati Tags: smalltalk, cincom smalltalk, web2.0, scaling, ruby, ruby on rails
Comments
More data
[Esteban] November 19, 2007 11:19:07.441
Hi James:
Impressive performance. Can you upload somewhere the raw reports generated by WAPT ?
Regards,
Bad Drop in PPS
[Patrick Logan] November 19, 2007 17:01:59.818
Woah. That Squeak drop in PPS from 117 to 5.8 is, well, not good. There should be some low-hanging stuff for *someone* to look at.
Feeling Better
[Patrick Logan] November 19, 2007 17:56:06.581
The tuned, sustained squeak numbers look better. But I wonder what the curve looked like for that original run starting at 117 PPS, how long that lasted above the 70 PPS avg on the tuned run.