Smalltalk and my misinterpretations of life

It's Friday now I just finished playing Mirrors Edge on my PS3. The game was challenging, difficult, frustrating and fun. Any game that gets you wriggling about on the couch trying to move your character when you're jumping from one roof top to another is a good game in my book. My heart raced with excitement many times during the game. Well done Dice. That said - I assume it'll be episodic? the ending was a classic hollywood "leave a sequel open" ending.

I don't think I'll feel comfortable playing a game again unless I can run, jump, duck and climb like I could in Mirrors Edge.. it's an expectation changer.. why can't my super awesome character know Parkour in every game?

Interestingly enough, while playing Mirrors Edge, I wanted more of Fallout 3 - I wanted to traverse the city by myself, without a story driving me in a specific direction. The Mirrors Edge experience was much like Half-Life 2.. you're in a mad rush to escape the gunfire and you feel like you've made the decision for which way you're going to go by yourself.. exceptional level design if you can pull that off. But at the same time, there was this big beautiful city and I just wanted to -explore-.

Fallout 3 was a fun game too. It had a flaw though - there are lots and lots and LOTS of side quests, but unless you go looking for them, you'll never know. The main plot only takes you to about a 5th of the map. The plot choices you can make become more limited as you progress through the game (Heck, I wanted to join the Enclave! viva la pure humanity!) but no matter what you do, you have only three real outcomes in the game. One applauds you, the other two vilify you. I don't think the game should have done that - instead, it should simply have state what happened and the result of those actions.. it should not have judged you.

The ending for Fallout 3 was a disappointment for me, mostly because the last two acts were unbelievably simple compared to the rest of the game. I was armed to the teeth ready for a huge final showdown. Don't get me wrong - LIberal Prime bellowing "Anchorage will be liberated" and "Democracy is non-negotiable" is just /brilliant/... but still, I felt cheated by the ending and judging.

That said, Fallout 3 had one of the best in-game atmospheres I've experienced to date. All the paraphernalia, posters, dust ball storms, radio stations, .. just everything was put together so well. It sucked you in to a post apocalyptic future and made you live it.. right down to your choice to kill off the slavers or join them. Very fun. If you're currently playing Fallout 3 - make sure you find the Republic of Dave. Hilarious.

One more final thing before I finish this blog post - how is the OpenGL 3.0 branch in Cincom Smalltalk going? Well, here's an example of 500,000 particles with physics running in the GPU. You can find the code for this as Lesson #19 in OpenGL-Lessons in the public store. You'll need a DirectX 9 compatible card to run it.

So an Aussie moves to the USA and gets himself a funky iPhone. AT&T must be the provider unless you do some hackery. New to the country, no credit rating - they'll sell me a phone but only if I put down a $500 deposit. Okay, I say, and I get myself a very cool iPhone. I'm happy.

Yesterday I decided I'd like to call Australia. I can do this from Skype but I thought I'd call from my iPhone instead. No sir, you must first enable international dialing... so I rang up support and they put me through to the international dialing enabling person.. who told me I can't do it until I've had my service with them for 90 days. Ahem, I haven't even been in the *country* for 90 days yet.

They can work around this, they say, but alas, they cannot do it for me because I had no credit rating and had to put down a deposit... uhm... so you're telling me someone who can fork out $500 as a deposit and wants to waste money calling long distance isn't someone you're going to provide access to? are you guys mad?

Jerks... well, I can wait until next month and my 90 days will be up, but I'm still annoyed at them, so I'm writing this blog post! let this be a lesson to you idiots, don't piss off your customers with stupid policies! especially if they ring up with an Aussie accent asking to have access to call Australia.

I really really like John Williams' work.. a lot. So this tickled me pink: Star Wars - an a capella tribute to John Williams.

Sometimes I feel productive, sometimes I don't. I didn't realize, however, just how much more productive I can sometimes be.

Recently, my Diabetes doctor here in the USA suggested we try using Lantus instead of Levemir. The Levemir wasn't working so well, as shown by a continuous blood glucose monitor over a period of four days.

One and a half weeks later, I'm still trialing the Lantus, but I'm starting to see some effects. First off - the last two days I've awoken with a normal blood glucose range! .. this is great!

But more importantly, these last two days I've been healthier than I have been in YEARS.. all thanks to this new doctor I've got. One interesting thing that happened without my consent.. time slower down.

I mean, who wants that to happen? It's lunch time now, it's been lunch time for at least a few hours by my normal standards. Time is ticking by very very very slowly. I'm getting a lot done. Yesterday I worked on WebVelocity, OpenGL, Multicore, SharedMemory and some other misc stuff. Today is only half over and I've done a lot more work than I normally get through already too.

So much so, I'm starting to realize that "getting up in the morning" means I have lots of time to shower, eat breakfast, etc instead of jumping straight in to work. I have -time-... I haven't had time since I was a kid. Now it's lunch time.. the clock is slowly ticking away and I've realized I have plenty of time to go out and buy myself something to eat and come home.. normally I'd just keep working because I don't have that much time.

Hopefully my brain won't burn up or something.

Anyway.. I thought people should know that if you feel like you don't have time to do the basics as well as doing your work.. may be there's something medically wrong with you that can be fixed?

Many moons ago I made a Spellchecker package for VisualWorks with my friend David Price. I used it in TypeLess my experimental irc client and in Bottom Line, the blog posting tool for BottomFeeder and recently it's found its way in to the RBCodeHighlighter so that you can spell check your code, comments and strings. It's also made it in to Martin Kobetic's pet project "Stampy" as well as our internal bug tracking software called Mars.

Spelling checking is really really useful... so it's no surprise to me that people wanted to use it in many things. The implementation we did originally was very efficient, speed-wise. It could spell check in less than a microsecond and likewise, find spell suggestions in less than a microsecond. The algorithms we applied were (and are) state of the art... however, we did nothing about the memory consumption.

The free English wordlist that comes with debian is ~900k big. The Spellchecker implementation we had made used 6.8mb. Woops.. we never actually looked at it, because it never mattered to us. But now that we're pondering putting it in to VisualWorks proper, the size does tend to stick out like a sore thumb.

Martin goaded and dared me to shrink its size. In half an hour I'd dropped it down to 1.95mb, which is a pretty good improvement. While munching on lunch, I realized I could shrink it even further and I got it own to first 1.33mb and then 1.17mb. I was pretty pleased at this point and going further would mean dumping objects and programming it like a C programmer.

In the last few years, especially with the OpenGL programming I've been doing, I've become more and more comfortable with using the powerful object system we have to do low level programming more easily than if I were in C itself. Having stopped at 1.17mb, I pontificated with Martin that in theory, it would be possible to shrink the dictionary down to 376kb at the most extreme and 640kb at its simplest.

I was reluctant to throw away the objects and try a new approach.. because it worked. Make it work, make it right, make it fast - it'd already been done, all three categories had been satisfied. More than that - a fourth category had been satisfied - make it efficient. So going any further would mean there was a fifth category could be satisfied - make it anal.

Martin suggested that I was simply trying to convince myself that it couldn't be done.. challenge accepted. I made a new package called Spellchecker2, which replaced Spellchecker and SpellcheckerEnglishDictionary. It stores its network of nodes in a DwordArray and the final outcome was this: Same speed for spell checking, word suggesting, but the memory foot print is a mere 484.5kb.

As well as that, I used the #once technique so that the DwordArray that is created for the vocabulary is shared between the CompiledMethod and the Vocabulary object you create. That means that the true foot print of the English dictionary running in the image is actually 484.5kb and nothing more.

The API changed between Spellchecker and Spellchecker2. For one thing - the dictionary is now stored in utf8, so if you want to do any spell checking, you have to convert your strings in to utf8 first. We could add some convenience methods for this.

You also have to transform the vocabulary object in to a mutable vocabulary before you can add words. This turns the raw byte data in to objects that you can manipulate and play with. When you're done, they collapse back in to the byte data and are saved on the vocabulary object. That means if you start adding your own words, you don't pollute the original.

vocabulary := Spellchecker2.English vocabulary.
vocabulary hasWord: ('testing' asByteArrayEncoding: #utf8).
vocabulary suggestWords: ('tseting' asByteArrayEncoding: #utf8).
vocabulary wordsStrings.
vocabulary mutableWhile: [:node | node addWord: ('smalltalk' asByteArrayEncoding: #utf8)]

So if you're interested, take a look and have a play. The software is free to use for any purposes. I'm hoping this version will be the version that makes it in to VisualWorks itself.

(Note: You need VisualWorks 7.7)

For those who read the right wiki, they would have known that it's possible to get the address of an object in Smalltalk. (Though, if it's in new space or old space, it may move.. perm space and fixed space are safe though).

The technique was complicated and messy. It involved having a new class with one instance variable. you would fill an instance of it with the object you want the address of. Then a method would create an UninterpretedBytes instance, become the instance with the bytes, set the slot and then return the slot.. this would give you back the pointer to the object.

If this sounds complex, that'd be because it is. What's going on here? Well basically the method that's executing keeps on running after the become has run. The object 'self' has become an uninterpreted bytes, but the method thinks it still has a slot that it can fill. The method fills the slot but in reality just writes in to the uninterpreted bytes.

At the Smalltalk Superpowers, a new way of finding the address of an object was revealed and improved upon. It is a stunning technique. Basically, what if we could avoid creating a new class and calling become: ?

When you call super in a method, the receiver that we're doing a super search from is stored as a literal in the CompiledMethod. That means we can change it - just by inspecting it and setting a new object in the slot. Now, when we run the method, the super call will actually search from a different class hierarchy but still not realize that it's about to splat memory in dangerous ways.

All that we need is a class that has one slot - eg: Model, which has an instance variable called dependents. We need one of its subclasses, which I chose ValueModel. John Brant demonstrated this technique using ApplicationModel. Now we implement a method like #addressOf: anObject

addressOf: anObject
super myDependents: anObject

Now, we inspect the method and change its first literal to ValueModel. We need to implement this on some kind of data array - such as ByteArray or UninterpretedBytes. John Brant's version used ByteArray but I immediately realized there was no reason we couldn't just use LargePositiveInteger and cut straight to the chase.

LargePositiveInteger>>addressOfObject: anObject
super myDependents: anObject

LargePositiveInteger classs>addressOfObject: anObject
^((self basicNew: 4) addressOfObject: anObject) compressed

(LargePositiveInteger compiledMethodAt: #addressOfObject:) literalAt: 1 put: ValueModel

So what happens here? We have 4 bytes which represent an integer of some arbitrary length (we need to do this because SmallInteger is not the full 32 bits, but a large positive integer of 4 bytes is). We trick the VM in to filling the first slot, 'dependents' by calling the method from a different class hierarchy, filling in the first slot with the pointer - which fills our 4 bytes.

Finally, we compress the LargePositiveInteger so that, if it can, it'll become a SmallInteger again (but if not, we still have a valid number to work with). We don't strictly need to do the compress step, if for some reason you feel that speed is important.

The cool thing about this technique is that it's just two methods and a method instance hack and we're done. The code is not intention revealing, sure, but hey, it works without having to change the VM.

I'm really impressed with this new technique.

The day was broken up in to two, we lately bound, dynamically linked our small short 15 minute snippets together to talk about our interesting hacks, for good for evil and had a real fun time.

After each snippet, the group would give it thumbs up or thumbs down. Thumbs down meaning it's evil and thumbs up meaning it's good. We generally judged by whether we would want to use this technique for an actual production system.. it's a sign of success if you get all thumbs down in my opinion :)

In the first half we saw some good hacks with prototyping, Martin kicked it off by showing how Gemstone changes the class of a session to optimize between normal execution and debugging execution.

I showed off BeforeYouLeave (all thumbs down!), a way to reduce terminal nodes in a tree hack using thisContext and many others. Some of them very humorous. Another example was the ability to return any value from a current stack frame in the Squeak debugger - we have that in VW too. This superpower is so useful it got full thumbs up.

After all thumbs down on BeforeYouLeave, I got all thumbs up on Jobs which I use for OpenGL, to control animations. The prototyping stuff shown was interesting, it paralleled my BareBehavior, IamMyOwnClass and Whatever, so I decided to skip those and talked to Adrian Kuhn about those techniques I used for him to apply to his own Prototypes work.

The second half of the day we put up our ideas on the wall and voted on them in terms of interest, since we weren't sure how much time we'd have to get through everything. Everybody is involved, engaged and having a good time. Lots of people have contributions and we've even got some stories emailed in to us and read out by Travis.

Now we're hearing about Duck R_ping which is basically ducktyping by any means necessary. It's for evil. When we have an evil talk, we dim the lights. When we have a "for good" we turn the lights back on. It adds to the atmosphere. In this case, he fakes an ActiveRecord object without having to read from the database, which can be slow.

Now it's John's turn to talk about how to get the address of an object. First he shows the "remove the first character from a string" by turning it in to a large positive integer, bit shifting it, then turning it back. That's a cute evil trick that one.

John now shows us the traditional way to get the address of an object. It's a complicated technique using another fake. You fill a slot in your object with the object you want the address for, then you can use a become: trick. You can use the same trick to find the bit pattern of numbers like 1.. which is 7 because of the two tag bits.

The new technique involves changing a literal in a method to change who your class is.. which is used to lookup super calls, which John realized meant you could reuse an existing class to fill in an instance variable. He now have a one method implementation for getting the address of an object.

With a different technique, he can catch the VM copying methods in to the code cache too, very fascinating. This way you can find out that a method is no longer in the code cache - you can also count how many times it has been put in to the code cache and thrown out again, neat if you want to see if your code is optimized or not.

Next, Travis is showing off super super, a neat tricky way to call the super method of the superclasses superclass (or further up the hierarchy). Travis destroyed his image by calling super beyond the class hierarchy, woops. So evil, it destroyed itself.

Now Travis has moved on to CachedBlockClosure, which we use for Assets, so it's a superpower, but something we actually use in the product. This technique requires changing the class of the object. We've seen this a few times today, but this time we see an interim Array object so that the shape can be changed become it is turned from BlockClosure in to CachedBlockClosure.

Next, on to Ward Cunningham's SymbolValue hack, Travis is presenting. Apparently, in Ruby, they got it wrong - the rails folks apparently did it badly.. one guys opinion (apparently it's slow, but built in to ruby 1.9, so the speed penalty is gone). ... But we love it in Smalltalk (some of us anyway).

To finish off SymbolValue, I talked about HigherOrderMessaging. Now, on to Monkey Patching, we can no longer say "No animals were verbed in this talk". Monkey Patching is basically changing methods on a class, or adding to the class.. something we Smalltalkers do all the time without batting an eyelid.. but Rubyists seem to emphasis the technique more.

A good example here is to patch Array to change [], to record who is calling array index, run tests, get the profile information. The Ruby profiler is a beautiful 55 lines of code, but it is incredibly slow... so this technique is more convenient.

Next I just talked about WriteBarriers, Martin wrote the previous write-barrier like thing, so he still needs to review it. We're thinking of putting it in the base image, but we need Martin to pontificate on it first.

Now we're on to MessageRecorder for Squeak, which can record -all- message sends, which is trickier than it sounds. It's like a full execution trace, but it also means you can replay the message sends later. It's a bit like the way Glorp does block query specifications. When you replay, you replay it on a particular target. Pretty neat really. Currently, it can only be replayed once. I suggested that he could make a parse tree instead and then he could compile too.

Now Randy Coulman is talking about Fitness in Smalltalk. James Robertson has a video of this from Smalltalk Solutions. But basically, you write your tests in wiki, reflecting off the executable environment. It's neat but i'm not sure it's for me personally.

To finish off, I presented the X86 Assembler I wrote. It got mostly up thumbs, which is fun.

Martin McClure and Travis Griggs ran the session and it was highly enjoyable, thanks guys!

Off to Nashville, this will be fun - I've never been there before. OOPSLA is definitely an event that is fun for me, talking programming languages at its deepest level - Lisps 50th birthday on Monday as well (or roughly there abouts).

There'll be Smalltalk Superpowers on Monday too, a Seaside BOF on Tuesday (tentative) and Gemstone is putting on a tutorial on Wednesday about Seaside where I'll get a brief chance to show off WebVelocity. I'll also show off WebVelocity, in more detail, at the Seaside BOF.

I'm also hoping to document some of the events, as co-conspirator for James Robertson's Smalltalk cheerleader (ra ra, but don't imagine him with pompoms). It'll also be good to catch up with friends and colleagues at the event.

My only concern is that TiVo might wipe some shows I want to watch if the hard disk gets full... just kidding, there's plenty of space!