Objology

For the last two releases, most of the "custom" widgets we've added to the system are done in a way different than classical VisualWorks widgets. Examples are the new comparison tool, the store feedback widgets, the bundle structure tool, and the prerequisite tool. These have served as a testground to try out some different approaches than classically found in VisualWorks. One the questions that comes up from some colleagues is "so, if you're doing widgets a new way, what is that way?"

The rest of this post is about one aspect of that "different" way. As I've wandered far and wide through other widget frameworks, I've learned that a) "MVC" is very popular and b) for every different framework there is a new and unique interpretation of "controller." One of the "different" ways I've been building widgets is without controllers. There are two aspects to deal with in this "controllerless" new world. One is how we deal with mouse interaction, the other keyboard interaction. The aspect of interest here today is keyboard interaction.

The original VisualWorks widget framework based on MVC, sometimes moniker'ed "wrapper," didn't have a keyboard focus concept. What ever widget was below the mouse and was willing, was where your keyboard input went. Round about version 2.0 of VisualWorks (then an add-on product to ObectWorks 4.1), the idea of keyboard focus got added. Which is a long winded way of saying, I didn't design this, I just learned how to work it. To have a controllerless keyboard savvy widget, you should add the following methods to your VisualPart subclass. The example code is what the AbstractComparisonRollupView does with them, as examples.

• view - The object responsible for managing keyboard events (KeyboardProcessor) assumes it's got an object in the role of controller and it can send view to it. Well the whole point is that your view is playing the role of both the controller and view.

  view
  	^self

• controller - Same sort of story as above, just from the other direction.

  controller
  	^self

• desiresFocus - This shoud return a boolean. The KeyboardProcessor sends this method to a keyboard consuming object when it's trying to determine if you'd like to be tabbed into. You can simply return true, or do something more complex, such as filter based on whether the receiver is "enabled."

  desiresFocus
  	^true

• requestFocusIn - This shoud return a boolean. Before the KeyboardProcessor will activate your widget as the currently focused widget, it must answer true to this message. This is used, in the current framework for example, to do user entry validation.

  requestFocusIn
  	^true

• requestFocusOut - This shoud return a boolean. Before the KeyboardProcessor will deactivate your widget (making it no longer the currently focused widget) it must return true to this (if it returns false, your widget retains focus). This can be used, for example to complete user edits, or validation.

  requestFocusOut
  	^true

• activate - This is sent to your widget, after you've negotiated the requestFocusIn/requestFocusOut/desiresFocus dance. It's where you make things happen as a result of focusing your widget. If your widget draws differently when it's "focused", you may want to invalidate here.

  activate
  	self invalidate

• deactivate - This is sent to your widget, after you've negotiated the requestFocusIn/requestFocusOut/desiresFocus dance. It's where you make things happen as a result of defocusing your widget. If your widget draws differently when it's "focused", you may want to invalidate here.

  deactivate
  	self invalidate

• hasControl - This is my least favorite of the API, because it looks like a testing method. You can couple it with activate if you like, since they are similar. But it's not sent at the same time as activate it is. It is sent when the window containing your widget gets focus, and your widget is the focused widget in the window. So it's very good for toggling indication of focus that only shows up whether the window is focused or not.

  hasControl
  	self header invalidate

The last thing you have to do, is make the KeyboardProcessor aware that you are interested in keyboard. This is usually done with a line something like

	theWindow keyboardProcessor addKeyboardReceiver: myView

This last requirement in particular I consider the most onerous, and we hope to make this more "snappable" real soon. You shouldn't have to worry about registering your widgets with the window (and subsequently unregistering them when cleanup is required).

One of the things I've never liked about the VisualWorks differencing tools, is that I have to choose between source and text differencing. Usually I want source. Source compares the source code tokens. But sometimes, it shows nothing. Why? Because the only change is a word in the method comment. Unfortunately, it only shows me the whole comment is changed, not the actual localized changes. Changes only in whitespace are equally distracting. The current solution embodied by DiffList, and a UI tool called Differator, doesn't really get where I want to.

What I really want one is ONE difference mode, that shows all of this at the same time. Pursuing this goal, lend me on an interesting journey. The results are in the mar10.2 7.8 build that will come out tomorrow.

To begin with, we have to separate two distinct parts. One is comparing texts. The other is what to do with the differences we find. Let's leave the second behind for a minute. If we solve the first problem generally enough, we'll have a myriad of choices open to us for the second.

A good starting place, is to go to the classic Unix diff command line utility. After digging around for a while with this, we'll discover two things: 1) It's not really about text at all 2) the algorithm found at the heart of the diff program is a nicely tuned piece of work.

The basic diff algorithm starts out on the basis, that for any two sequences (whether they're sequences of characters, words, textlines, bytecodes, or even DNA markers), there is a sequence that is a "longest common sequence". There may be multiple resolutions to this, but none of them is longer than the other. You can find lots of interesting stuff about this around the net. A simple example is the sequences h-u-m-a-n and c-h-i-m-p-a-n-z-e-e. The longest common sequence (LCS) between the two is h-m-a-n. Once we have that, we can determine the steps it takes to turn the first sequence into the second. This is sometimes called the shortest edit script. In this case, we end up with the following edit script: Insert c, copy h, delete u, copy m, insert p, copy a and n, insert z and e and e. This gives a directional evolution from one to the other. Want to go the other direction though? Just transpose inserts with deletes and keep matches in place.

With this kind of information, we can envision a number of different ways of highlighting differences between texts, or DNA sequences, etc. We can do them in a unified view. We can do them in side by side views. We can even discern replacements from true insertions or deletions, because it's a paired delete/insert sequence.

The mar10.2 build adds a new differences: API to SequenceableCollection. You can take any two sequences and determine how to evolve one from the other. You can feed it DNA symbols, strings, numbers, whatever you can think of to put in a SequenceableCollection. The result is an array of SubsequenceDifference subclasses, one for each of the different kind of edits.

At the heart of the algorithm is a modified/ported version of a generalized version of the Meyer's Diff algorithm. This too, you can find lots of stuff on the net about. You'll even find a couple of visualizations. What the Myers adaptation of the LCS exploits is that usually, the algorithm is applied in cases where differences are few compared to matches. It also integrates the process of finding the LCS with the process of determining steps, it ends up being a linear space algorithm. To say it's a hairy nasty work of indirection and head scratching, is an understatement. It makes sorting algorithms look like kindergarten reading.

I ported it 3 times. Each implementation I found had a test suite, so I gathered those into one big test suite and added a couple edge cases. The first port was of a C# implementation. This seemed easy, but I ended up with out of bounds errors (the algorithm relies heavily on 0 based array addressing). So I tried a Ruby version. It was ironically harder to port (this is the language that's supposed to be a lot like Smalltalk). And it too had issues. Then I ported a generalized C version (generalized in the sense it was parameter-izable), and it worked! All the tests passed. There's irony for you.

How much faster is it? Given the degenerate case of determining how to get from Object comment to Object comment reverse, the VisualWorks DiffList which is a hybrid LCS searcher and produces only "unmatching range" information, takes 12.5 seconds to run. The differences: method does it in under half a second. Implementing a naive recursive LCS method, has to be killed after grinding for 3 hours.

At least two good things have come out of this. One is that there is now a general purpose method to quickly and efficiently produce complete shortest edit scripts between any two sequences. Application programmers don't have to be tied to the IDE's method of highlighting differences between two strings, they can do whatever they want. Application Programmers can use it in contexts beyond source comparison (e.g. comparing load lists, or determining undo differences, or DNA sequencing, or schedule comparison, etc).

Secondly, we've separated what to diff with how to diff completely. This means that I can have a method on RBScanner which scans source tokens. But it can go further. Since said scanner knows where comments are, we can produce strings for the comments. In fact we can go further, we can break the comments up into words easily. We can even note when the token is a string literal token and break it up into words as well. And we can insert elements for white space. So that when we difference Smalltalk methods, we get everything broken down to the granularity we'd like. Which was what I wanted in the first place.

A couple of months ago, I was daunted by the prospect of having to do yet another Look and Feel set of subclasses for the VisualWorks widget set. The technique of subclassing is hopelessly over-leveraged in what is there.

The debate about whether to do native widgets or not, and in what form factor has gone round and round for years now. It will continue to do so. Somewhere in there, I pondered "what would happen if I could draw standard widgets faithfully (or pretty darn close), but still emulate them? Is there a half way point there?"

I found that for OSX and Windows, there actually was to some degree. I discovered that all of the graphic resources that OSX uses are actually in a database and can be pried out and reused. You have to make the right decisions about how to use them, but if you use them correctly, you're doing the same thing OSX is. On Windows the story was even better. uxTheme.dll is the theming engine that Windows uses to evolve it's look and feel, as well as customize widgets for other platforms (such as mobile devices) -- much thanks to Michael Lucas-Smith for pointing me towards this.

So I began experimenting with that, and some other ideas that we've already been hinting at, in a package that I dubbed Skinny. It's so dubbed, because I'm interesting in getting to a point, where changing the skin of a VW app (whether the IDE itself, or something else), is relatively easy.

Here's a screencast that shows me playing with version 93 which is in the Open Repository.

Here's a list of questions/answers one might ask about such an experiment:

Q: Which versions of VisualWorks do I have to have for this?

A: You need a pretty recent 7.8 dev build. There are small things from this project, that we've been rolling into VisualWorks proper.

Q: Do you plan on integrating this in the product someday?

A: I don't know yet. It could be that we end up doing yet another subclass tree and just harvest the uxTheme.dll drawing know how. Or maybe more of a hybrid. I don't know yet.

Q: Does the Windows look and feel work on other platforms?

A: No. It doesn't.

Q: What other skins have you played with?

A: There's an OSX one. It uses Cairo to render the various graphics excavated out the OSX graphics database. Ironically, it actually works better at the moment on either Windows of X11, since Cairo doesn't interleave efficiently with Quartz drawing. It works there, but it's pretty jerky. Great benchmark for really improving Cairo on OSX.

Q: Which widgets have you built for?

A: Push Buttons, Radio Buttons, Check Boxes, Scroll Bars, Tab Controls, Splitters (the splitters don't build thru UIBuilders tho)

Q: What if I'm more interested in not using UIBuilder technology and just simply snapping widgets together?

A: Look at the various class side example methods on Skinny objects.

Q: What known issues are there already?

A: Haven't made any of them keyboard navigable (waiting to integrate some changes to the core to simplify KeyboardProcessor first). The Scrollbars don't get their thumbs sized initially right, they require a click to do so. Lots of others, I'm sure.

Q: What IDE tools seem to work?

A: In the screencast, I show workspace, inspector, and settings tool. There's actually one page in settings tool that has issues still (a store one with a dataset). The browser kind of works. Some of the widgets don't pick up the skinny skin. The FileBrowser tanked last time I tried it. I have used the debugger on it successfully before, but currently have it turned off.

Q: Why are you blogging about it?

A: Always a good idea to give people a peek at what kind of things are going on. I'd also be interested in people trying to use it with their own apps. I'm curious how long it would stay alive and if there's an obvious "implement this next" vector.

"An idea, like a ghost, must be spoken to a little before it will explain itself." - Charles Dickens

Sometimes, we do things in our programming practices that just seem right. But we may struggle to figure out how to illustrate why it feels right to colleagues and peers. We have a vague idea, and some of the words involved, but a succinct explanation evades us. This post, is an attempt to shed a little light on some of the principles I follow in regards to "unit testing." Some of this fell into place, just days ago, though I've been doing this a certain way for 12 years now.

Back in the 50's, there was a sort of revolution that took place in the manufacturing world. Quality was the issue. Leading role was played by E. W. Demming. Some of the plot themes were things like process control, quality control, statistical process control, shewart charts, kanban cards, etc. It was a big hit in Japan initially, and would eventually make its way over to the States and other places. As a Mechanical Engineering student, I had this stuff pounded into my head through long years of college.

One of the things that struck me about these ideas (ideas that would eventually be caught up in and respun as "lean or agile" manufacturing practices) was a subtle shift in quality testing. Many factories convinced themselves that they had quality because they did testing. Classic manufacturing testing philosophy, said that you assembled the gadget, and at the end of the factory, you ran the assembled device thru a screen of tests. If it passed these tests you shipped it. In this era, the bolt vendors could be pretty lax in their tolerances; assembly personal at the plant would take up the slack, pairing bigger bolts with bigger washers and smaller bolts with smaller washes. As long as the machine at the end performed it's function nominally, it was good enough. Of course, when the device came under stress, this variety led to failures and subpar performance.

Demming and crew, felt that this was pointless to catch defective products that late in the cycle. It was wasteful. So they pushed testing and quality control back into the process. They advocated tolerances on the component parts, rather than the unit as a whole. The basic idea is that if you tighten up the quality on the parts that compose your device, your device ends up the better. In fact, if you had a controlled process, and high quality inputs, then you could confidently ship higher quality products while actually reducing end testing. In years to come, those that made the change to this mentality survived, those that did not, mostly failed.

It's instructive to me, that while both methods advocated and adhered to "testing", the Demming approach and application of testing was the one that produced better results.

Since the "splashdown" of XP in 1998 and the following furor of Unit Testing in Software, I've witnessed similar parallels in the application of testing software. Everyone claims they do it. When Ken Treis and I came home from OOPSLA 98, we determined we'd try this. We put together a version of a Wiki server for VisualWorks, and we used workspace scripts to make test that we got the right http responses when we sent different kinds of queries against our server. And having done that, we went right on writing software the way we always had. It didn't really matter how we architected the code as long as it passed these high level tests. Of course, we kept discovering all kinds of edge cases. Though I was well versed in the Demming school of thought about testing the components, rather than the end product, I missed the potential corollary.

It wasn't until Camp Smalltalk 2000, when Ken had the chance to work with Ron Jeffries, that we got a chance to see it done right. Ken saw, and quickly shared with me, what a real unit test looked like. The light bulb went off for us. This would change the way we developed our algorithms at a component level. I've been a happy unit tester since. I am firmly entrenched in the belief that unit testing is to software what component testing is to manufacturing. Usually, it takes the form of having a particular test class for each meaningful class in your system. There are of course adaptations, but more often than not, I find that the adaptations are rationalizations that allow people to test in the "old school" style while telling themselves they're doing it differently and making a real difference.

In manufacturing, the point is to reliable reproduce and accepted design. In software, we don't do that. The ability to reproduce our software designs, is not in question. The creative part of what we do, is to create algorithms, of various sizes and shapes, of varying complexities. When we write unit tests for the components (or objects) of our algorithms, we're embracing the same set of principles that drove the quality revolution. Try it out. Don't write so many tests for the overall features of your system. Try writing tests for your components. One to one.

Adrian Kuhn has been blogging a series of Smalltalk Superpowers. What a wonderful blog topic. And a great job Adrian is doing too. He's a worthy Smalltalk Superhero.

I'm having an adventure in superpowering right now. I need, for reasons as of yet undisclosed (superheros live secret lives, ya know), to have my own subclass of UILookPolicy. UILookPolicy participates with UIBuilder objects to compile (build) user interfaces from WindowSpec objects. Much of its behavior is in UILookPolicy, but it is in fact an abstract class. It has a handful of platform specific subclasses.

The problem, is that I don't want a whole new subclass. I don't want to re-implement all of UILookPolicy. What I want is a generic subclass of whatever the default subclass is, whether it's a MacOSXLookPolicy or a WinXPLookPolicy.

But the subclass card has been played already. To get that kind of structure with normal everyday living, I'll have to make a subclass of each subclass. Or...

I can step into a booth and don my Lightweight Classes costume. Here's what it looks like

UILookPolicy>>myOwnPolicy

	| lightweightClass |
	lightweightClass := MyLookPolicy copy.
	lightweightClass superclass: self class.
	^lightweightClass new

With this costume on, I can send the message myOwnPolicy to any subclass instance of UILookPolicy (e.g. a MotiffLookPolicy instance) and end up with a new instance which implements the behavior of MyLookPolicy as if it were a subclass of that class. When in reality, it's a subclass of UILookPolicy, and by default, a sibling to the likes of WinXPLookPolicy and friends.

This allows me to derive from any standard LookPolicy object, one that overrides methods like button:into:, but otherwise, inherits all of the behavior of the original instance.

Chris Thorgrimsson sent me a link to his new blog where he's talking about some of the things they're doing with Cairo and VisualWorks. The demos and stuff I show are OK, but it's always way cooler to see it in real life stuff. Thanks for sharing Chris. Chris actually prompted this post a while back.

It's the hill climbing factor. It doesn't matter if you've just climbed the smallest mole hill next to Mount Rainier, if you've worked hard to get to the top of that hill, and you've struggled to get there, you experience the biggest high having achieved a local maximum when you get to the top.

I've often theorized that this is why the masses never have taken to the higher level langauges such as Smalltalk, Lisp, ML. Building and enumerating a doubly linked list in C is a challenge! Doing it in Smalltalk. Meh. You run out of easy hills to feel good about climbing.

Tonite, I'm feeling that kind of feeling. After banging my head, googling through obscure references, I've got to the point where I can do a one-click doit to download all of the Cairo, Zlib, Libpng sources and compile Zlib and Libpng, using Microsoft's freely available Express tools.

I'm simply amazed that Microsoft took over the world, so to speak, with such an operating system/environment, where automating anything is so difficult. The flip side, is that getting an automated download and configure/compile gives me a huge rush.

On the way, I'm having continued fun with Herl. I've further fleshed out some of it's OS interface parts for Windows now. Keeping in the tradition of "making it up as I go."

This last week, I've been trying to get CairoGraphics libraries built on the breadth of platforms we support VisualWorks on. Doing so makes it easy for customers to use CairoGraphics with VisualWorks in their own projects, and if we manage to get ALL the platforms, it makes it so we can begin to improve our own IDE with its capabilities. It's unknown how many, if any, we'll get included for preview for the 7.7 release, but something is better than nothing. Windows and MacOSX seems a good probability at this point. X11 is proving more difficult.

Having built CairoGraphics libraries a couple of times, over the last year or so, with long pauses in between, I often find the documentation is not quite up to date, that I end up having to ask questions of their very helpful mailing list or irc channel. I'm not GNU build guru, so they're probably naive. Which led me to want to script the download and building of these libraries. So that it was reproducible. And precisely documented.

Bash is the script environment I'm most comfortable with. But I'm no wizard. Especially when it comes to doing higher level algorithmic things. I have to ask for help for those, and I end up missing Smalltalk anyway.

So I thought I'd take the rudimentary ScriptingSupport package that we include as preview in VisualWorks, and try that. At this point, it's still pretty young. It implements stuff to make command line execution easier, and adds the & binary sugar, which is a handy way of concatenating single objects onto collections (as opposed to , which concatenates multiple elements).

And then I just decided to have fun. I followed the following design principles, which seem to be at the heart of the design of most scripting languages.

1. Typing sucks. It's no fun. So add lots of sugar to make things terse and conserve keyboard srokes.
2. Use lots of infix characters.
3. Make it up as you go. As you encounter new situations, add more stuff, don't try to come up with any sort of simple unifying theme. Just throw more stuff on.
4. Typing sucks. The other kind. Support few basic types, and abuse them. Just mash them together. You should be deluded into thinking you have don't have to worry about types, except now and then again in surprising ways. Don't make me convert from one type to another.

The only non-traditional requirement I added, was that I don't want to build a new parser or anything. So it has to strictly speaking, still be Smalltalk.

It's still a work in progress; I'm not out to start a movement. I expect derision and flames. Its all a lark to me right now. I've named my little Domain Specific Scripting Language "Herl". Ken Treis actually suggested the name; thanks Ken. It's been replicated up to the Open Repository and has a package comment that looks like any scripting language reference manual.

For those who aren't that interested (all of you likely), a snapshot of the "script" I wrote with it, is probably enough.

rootDir := '$(HOME)/BuildCairo'.
rootDir rm.
rootDir mkdir.
rootDir cd.

"Download everything"
page := #(curl   -s 'http://www.cairographics.org/releases/' ) exec.
cairo := (((page -@ $>)  detect: [:each | 'LATEST-cairo-.*' ?= each]) -@ $<) _1.
(#(curl -s) & ('http://www.cairographics.org/releases/' , cairo) , #(-o 'cairo.tgz')) exec.
pixman := (((page -@ $>)  detect: [:each | 'LATEST-pixman-.*' ?= each]) -@ $<) _1.
(#(curl -s) & ('http://www.cairographics.org/releases/' , pixman) , #(-o 'pixman.tgz')) exec.
page := #(curl  -s 'http://pkgconfig.freedesktop.org/releases/' ) exec.
pkgconfig := ((((page -@ $>)  select: [:each | each =? 'pkg-config-.*' ]) sortBy: [:each | (each -@ $.) _2 asNumber]) last -@ $<) _1.
(#(curl -s) & ('http://pkgconfig.freedesktop.org/releases/' , pkgconfig) , #(-o 'pkgconfig.tgz')) exec.
#(curl -s 'ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-1.2.40.tar.gz' -o 'libpng.tgz') exec.

"Untar and rename all 4"
#(tar -xzf 'pkgconfig.tgz') exec.
'pkgconfig.tgz' rm.
(rootDir ls: 'pkg.*') _1  mv: 'pkgconfig'.
#(tar -xzf 'libpng.tgz') exec.	
'libpng.tgz' rm.
(rootDir ls: 'libpng.*') _1 mv:  'libpng'.
#(tar -xzf 'pixman.tgz') exec.	
'pixman.tgz' rm.
(rootDir ls: 'pixman.*') _1 mv: 'pixman'.
#(tar -xzf 'cairo.tgz') exec.	
'cairo.tgz' rm.
(rootDir ls: 'cairo.*') _1 mv: 'cairo'.

"Build, install pkg-config"
(rootDir / 'pkgconfig' ) cd.
(rootDir / 'pkgconfig' / 'configure' & ('--prefix=' , (rootDir / '') forExec)) exec.
#(make) exec.
#(make install) exec.

"Environment variables for our pkg-config"
'PKG_CONFIG' setenv: rootDir / 'bin' / 'pkg-config'.
'PKG_CONFIG_PATH' setenv: rootDir / 'lib' / 'pkgconfig'.

"Setup environment variables for fat binary compilation"
'MACOSX_DEPLOYMENT_TARGET' setenv: '10.4'.
'LDFLAGS' setenv: '-arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk'.
'CFLAGS' setenv: '-Os -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk'.

"Build, install libpng"
(rootDir / 'libpng') cd.
(rootDir / 'libpng' / 'configure' & ('--prefix=' , (rootDir / '') forExec) & '--disable-dependency-tracking') exec.
#(make) exec.
#(make install) exec.

"Build, install pixman"
(rootDir / 'pixman') cd.
(rootDir / 'pixman' / 'configure' & ('--prefix=' , (rootDir / '') forExec) & '--disable-dependency-tracking' ) exec.
#(make) exec.
#(make install) exec.

"Build, install cairo"
(rootDir / 'cairo') cd.
(rootDir / 'cairo' / 'configure' & ('--prefix=' , (rootDir / '') forExec) & '--disable-xlib' & '--disable-dependency-tracking') exec.
#(make) exec.
#(make install) exec.

"Move dylibs to platform staging directories"
(rootDir / 'Frameworks') mkdir.
rootDir / 'lib' / 'libpng12.0.dylib' cp: rootDir / 'Frameworks'.
rootDir / 'lib' / 'libpixman-1.0.dylib' cp: rootDir / 'Frameworks'.
rootDir / 'lib' / 'libcairo.2.dylib' cp: rootDir / 'Frameworks'.

"Make them all relocatable"
(rootDir / 'Frameworks') cd.
#(install_name_tool -id '@executable_path/../Frameworks/libpng12.0.dylib' 'libpng12.0.dylib') exec.
#(install_name_tool -id '@executable_path/../Frameworks/libpixman-1.0.dylib' 'libpixman-1.0.dylib') exec.
#(install_name_tool -id '@executable_path/../Frameworks/libcairo.2.dylib' 'libcairo.2.dylib') exec.
(#(install_name_tool -change) & (rootDir / 'lib' / 'libpixman-1.0.dylib') , #('@executable_path/../Frameworks/libpixman-1.0.dylib' 'libcairo.2.dylib')) exec.
(#(install_name_tool -change) & (rootDir / 'lib' / 'libpng12.0.dylib') , #('@executable_path/../Frameworks/libpng12.0.dylib' 'libcairo.2.dylib')) exec.

"Clear build variables"
'MACOSX_DEPLOYMENT_TARGET' unsetenv.
'LDFLAGS' unsetenv.
'CFLAGS' unsetenv.

Feel like Herl-ing now?