Travis has posted on his SUnitToo tools - he bugged me about it today in the IRC channel when I mentioned that I was adding some tests to my Blog code while I refactored. So, I went ahead and loaded his stuff - it's in the public Store, but (and this stuff is being changed all the time) I had to load the penultimate version:

When prompted, load this version of the pre-req:

So what does that give you? Well, take a look at the browser view with two new tests that haven't been run yet:

The empty beaker indicates that no tests have run, and the tests have "?" next to them. See the changes when I run the tests?

I've chopped the bar at the bottom of the browser, but you get more indications now. So - let me break one of the tests:

That's pretty nice. There are other goodies as well - if you hit "debug", you'll get right to the failing assertion (where the old SUnit lands inside the SUnit code itself). All in all, if you use SUnit in VW or OS8, you should move to SUnitToo. Ohh - one more thing. To do that switch, check the "View" menu and toggle which testing tools to use. You can move back and forth at will, and the new stuff is API compatible with the old.

Technorati Tags: sunit, smalltalk

I ran across this page in my aggregator before lunch, and flagged it for followup. That time has come - it must be testing day on this blog. Here's what jumped at me:

I give up on libxml for the time being, and think instead of Chris Petrilli’s comment that ruby (and python) performance is “not quite in the league of Smalltalk (or Lisp, likely), which have extremely mature VMs with on-the-fly compilation and optimization”. Is Smalltalk then much faster than python or ruby, or comparable with C, for the task of parsing moderately large XML files?

No. Time to load and parse my iTunes library file, an 11mb Apple plist, on a 1 GHz G4 Powerbook with VisualWorks Non-Commercial 7.3.1: about three minutes.

That didn't seem right - I use the XML code in VW extensively, so I'm pretty familiar with it. I grabbed my iTunes file (only 2.7 MB) and parsed that - took 5.5 seconds. Well, the two caveats are, that's a smaller file, and my hardware isn't his hardware. With that in mind, I went ahead and created a large XML file. I grabbed the default feed file for BottomFeeder, and saved it as an XML feed list instead of as a binary dump - like this:

file := Tools.XMLConfigFileSupport.XMLConfigFile 
                     filename: 'g:\vw74\image\feeds.xml'.
file saveObject: RSSFeedManager default subscribedFeedsFolder.
file saveConfiguration

That just dumps the 80 sample feeds into a (pretty verbose) XML format - I ended up with a 13 MB file. That seemed large enough, so I tried the parse on that:

content := 'feeds.xml' asFilename contentsOfEntireFile.
parser := XMLParser new.
parser validate: false.
Time millisecondsToRun: [parser parse: content readStream]

That last line times the execution - it ran in 17.9 seconds. Not a couple of seconds, but not 3 minutes, either. There was some GC going on during that, so I'm sure that things could be improved by simply configuring VW with a larger bite of old space up front - in dealing with large amounts of data, a fair bit of time is going to be chewed up either in allocating more memory, or GC'ng if we hit the current limits (as per the memory policy in place).

For this kind of parse to take 3 minutes, either the hardware would have to be very slow, or memory limits would have to be set badly for dealing with larger files. I'm not entirely sure what was going on.

Update: I ran the same code on my Mac Mini - it has a 1.3 Ghz G4 processor, and a paltry 256MB of RAM. The 2.7 MB file parsed in 12.8 seconds, the 13 MB file in 44.7 seconds. Not speedy, but not the 3 minutes reported by Alan Little either - and the Mini is no high end Mac.

I had a look at the Great Computer Language Shootout site this morning, since there's some VW Smalltalk code (and results) posted there. The comparison I was drawn to happened to be with Mono based C# (based on a comment here). There are some issues with the comparison, however:

• Have a look at this test - scroll down, and look at the execution. The source code is filed-in, and then the test is executed. That slows things down. Update - apparently, the code is not filed in first on the site test.
• Have a look at the C# version - the code is compiled, and then executed.

So I did the same thing I did with Troy's post over the weekend - I downloaded the code and did some local shaking out. To get it loaded, I had to create a namespace called ComputerLanguageShootout first, and also create a class named Benchmarks. Once that was done, I could file-in the code.

Then, I tried this:

"File in code, then execute"
Time millisecondsToRun: [Smalltalk.ComputerLanguageShootout.Benchmarks 
                                                  nbody: 1000000].

That ran in 11.818 ms. I repeated the process a few times to make sure that those numbers weren't outliers, and they weren't. The original post here had a bunch of comments about filing in first, but I was wrong about that.

So, on to the profiler. Running that, it turns out that the bulk of the time is setting in the Body>>and:velocityAfter: method. Looking at that, we see the following code:

and: aBody velocityAfter: dt
   | dx dy dz distance mag |
   dx := x - aBody x.
   dy := y - aBody y.
   dz := z - aBody z.

   distance := ((dx*dx) + (dy*dy) + (dz*dz)) sqrt.
   mag := dt / (distance * distance * distance).

   self decreaseVelocity: dx y: dy z: dz m: aBody mass * mag.
   aBody increaseVelocity: dx y: dy z: dz m: mass * mag

It's no big surprise that this test (or any other test that is heavy on arithmetic) is going to be faster in C# (or C, or Java) than in Smalltalk. Why? Smalltalk isn't that fast on math, floating point math in particular. So if you intend to code up some low level mathematical model, the calculations should be in a different language (and our customers tend to do that - model in Smalltalk, low level math in C or C++). What this really points out is that micro-benchmarks aren't that useful. Most applications push data around, typically involving a database - and math operations don't dominate those kinds of applications. In which case, something that lets you code faster is going to help. As usual, pick the best tool for the job at hand.

On the Smalltalk IRC channel, it was pointed out that a lot of the other language tests vary just as widely. Ultimately, the shootout site simply isn't doing great cross language tests.

Troy posted a comparison of some C# logging code and a simple translation to Smalltalk - and then noticed that the Smalltalk code was a lot slower (nearly 5x). Well, this calls for the profiler. I got the same results for all three of his scenarios in Smalltalk, so I'll compare the code for the class test and for the instance test. First, I loaded Troy's code. Second, I went to the Parcel Manager and loaded the Advanced Tools (there's a legacy name for you) - which brings in the profiler. Profiling the code:

 
 
 

TimeProfiler profile: [TimeLoggers timeClassLogger: 200000].

Leads to this result:

You can click through to see the larger view. The upshot is, the original logging code sends #today to class Date multiple times within the logging code (instead of once). Additionally, it turns out that #expandMacrosWith: - while convenient - is fairly expensive. You can see all that in the expanded view. In any case, here's the original code:

 
 
 

writeToLog: aString 
	(self fileStream)
	       nextPutAll: ('%<!--         
<1p>/<2p>/<3p>' 
	                               expandMacrosWith: Date today
dayOfMonth
	                               with: Date today monthIndex
	                               with: Date today year)
 

To change the performance, I did two things. First, I changed the code above to just print the string that was sent in, and then I changed the calling code to send in the string with the Date formatted. I could have encapsulated that in the new code for the above, but in any case, here's the new caller:

 

       1 to: iterations
	       do: 
	               [:i | 
	               si 
	                       writeToLog: ('Testing write to log via
basic logger - line number ', i printString,  
	                                       ' ', todayString)].

A better variation would have the todayString done in the logger - it doesn't force work on the user of the code. In any case, this variation yields logging code that is still slower than the C# code, but not absurdly so. I also posted that - 1.1 is the faster (but less well coded) version, with 1.2 being the better way. That code has the caller simply send in a string, to which the date will be appended, as follows:

 

   | todayString |
	todayString := Date today printFormat: #(2 1 3 $/ 1 1 ).
	(self fileStream)
	       nextPutAll: '<--  ';
	       cr;
	       nextPutAll: aString, ' ';
	       nextPutAll: todayString;
	       cr;
	       nextPutAll: '-->';
	       cr;
	       flush

On my system, that runs a a bit faster than the better encapsulated version, but not much.

The upshot of all this? Two things:

• Direct translations of code rarely lead to good results. Each language system has strengths and weaknesses, and direct translations don't hit the strengths
• Never make guesses about performance - run the profiler.

Again, all the code for this is in the public Store.

MemoRanda gives a nice example of why unit tests are worthwhile.

Martin Fowler brings up a point about testing that can't be hammered enough: isolation, isolation, isolation:

If tests sometimes pass and sometimes fail on a run without any code changes, or tests pass when run in some suites but fail when run in others; nine times out of eight the reason is that there is some shared data between tests that isn't being properly reinitialized. When that happens just running a test can be the difference between other tests passing and failing. The result is an intermittent failure - always the worst because you can't reliably reproduce it.

More than once, I've needed a pounding with the cluestick over this sort of thing...

Cyrus explains some error reporting code in the current beta of VisualStudio - basically, VS is either going to throw a "recoverable Watson" at you when an exception occurs, or queue them up for later reporting. The idea is to have more information sent back to the team in Redmond. I'd be curious to know how this works out, for a very simple reason - in my experience, people - users and developers both - never read the text of the error messages that come up. They tend to hit <return> and move along. It looks like the default for this is going to be "send the information to MS". It will be interesting to see how that works for them.