I commented on a "Java dynamic code" post earlier, and got a few comments - so I pushed up a screencast showing how to generate new code (methods and classes) at runtime in Smalltalk. It's way, way simpler than the Java example, and I show that in the screencast. Watch it here; enjoy.

Technorati Tags: java, smalltalk, dynamic languages

I've alluded to the patching capabilities of BottomFeeder before, but it seems that I've never actually gone into much detail. I had an email on that this morning, so I figured I should post on it.

The basic capabilities are built into the product - assuming you've left the compiler in your runtime, loading new code in via file-in or parcel loading is easily possible. What I added for my application was this:

• An HTTP based interface, allowing the application to query a server for updates
• an XML based manifest system whereby the client can check what's already loaded versus what's on the server
• An ability to have the updates loaded after download, without the normal development-time dialogs

If you have access to the Public Store Repository, you can check out the package PatchFileDelivery (which should have no dependencies on BottomFeeder - I may have to weed a couple out). The name is something of a misnomer, due to the evolution of the package. When I started, I was delivering small patch parcels, but I moved away from that and on to delivery of new versions of already loaded parcels - it made version control in the runtime a lot simpler. The basic steps look like this:

1. Execute an HTTP Query to the server for the XML manifest
2. Client compares Manifest to what's loaded
3. Client offers any available updates to the user
4. Selected updates are downloaded via HTTP
5. If requested (and if possible), updates are loaded immediately

So let's take a brief look at those steps. The manifest is a collection of ComponentDefinition objects. That class looks like this:

Smalltalk.Patch defineClass: #ComponentDefinition
	superclass: #{Core.Object}
	indexedType: #none
	private: false
	instanceVariableNames: 'parcelName parcelFilename version oldVersion releaseDate vwVersion isPlugin descriptiveName description fileSize allowDynamicLoad '
	classInstanceVariableNames: ''
	imports: ''
	category: 'PatchFileDelivery'

The important pieces of that are the version, parcelName, and allowDynamicLoad. The last one of those is a flag to the client - if it's false, the component in question requires a restart. That comes up when I have an update that does major changes to the UI, for instance. There are ways of dealing with that, but I figured a restart was simpler, and haven't really received complaints. The version is just that - a value derived from Store, the source file repository I use. The name and version are compared to what's loaded to come up with the list for the user.

Once that's figured out, the user's selections are downloaded via HTTP. The only difference between the base HTTP capabilities of VW and what I do is the progress dialog - and that's code that was submitted by Bob, one of our engineers. The change? Additional code in a subclass of the relevant HTTP class to raise notifications of status. Once the code is downloaded, it's either simply saved, or saved and loaded. If it's the latter, the following code gets used to load the new version of the parcel:

actuallyLoadParcelFrom: parcelFile

	[[Parcel loadParcelFrom: parcelFile] on: Parcel parcelAlreadyLoadedSignal, CodeStorageError
		do: [:ex | ex resume: true]] 
			on: DuplicateBindingsError
			do: [:ex | ex resume]

The first handler catches the "already loaded" signal - which normally raises a dialog. I didn't want that, so I catch it and just have the system resume with true. Which illustrates one of the cool things about Smalltalk exception handling, btw - the ability to rewind back and have the system move along with the correct answer.

The second handler catches transient errors that arise during the load of the new parcel - in some situations, the new version of code can look like a duplicate code binding. That gets resolved as the load continues, so I just catch it and continue. Which again demonstrates the coolness of Smalltalk exception handling.

Once that's done, we have the new code loaded and are ready to run again. What about the next startup? Well, BottomFeeder looks in a known directory for new versions of code, and that's where the upgrade manager saves them. So when the application is started up, it does step (5) from above. And that's it - pretty simple.

I get asked about Smalltalk exception handling from time to time, and - specifically - I get asked how it differs from what's done in Java. The primary difference is in how the context stack is handled. In Java, when you get to the handler code, the stack has been unwound and is just gone. Not so in Smalltalk - as a matter of fact, the stack is an argument held in the exception. Here's a simple example - I'll use the NetResources library to try and retrieve a non-existant url, and toss a breakpoint into the handler code so you can see what's going on. Here's the relevant code:

	response := [client executeRequestDo: 
					[:connection | client getNotifyingResponse: connection]]
		on: (self class httpExceptions, Error) 
		do: [:ex | self behaviourForException: ex. nil].

We've wrapped the actual HTTP request in a handler that sends all exceptions to a method called #behaviorForException:. That's done so that we can respond appropriately based on the kind of exception that crops up. Here's that method:

behaviourForException: ex
	ex class = Security.SSLBadCertificate
		ifTrue: [Security.X509.X509Registry default addCertificate: ex parameter parameter].
	(self class possibleTimeoutExceptions includes: ex class) 
		ifTrue: [self updateCacheResponseCodeOnly.
				^self class triggerTimeoutEvent: url].
	(ex isResumable and: [self class exceptionsWeShouldResume includes: ex class] ) 
		ifTrue: [ex resume]
		ifFalse: [self reportTheErrorType: ex]

See that part at the bottom that checks for resumable (and worth resuming) exceptions? If we get there, what will happen is simple - the system will return to the point where the exception got thrown, and simply proceed as if it hadn't happened. You might ask yourself, why would we want to do that? As it happens, there's code in the calling method to handle things like HTTP redirects and Authorization requests - so those exceptions are simply resumed. So anyway, a brief demonstration - here's a screen shot of an attempt to fetch a non-existant URL:

Now, here's an inspector on the exception:

And drilling down, the context itself:

All of which shows how it can be useful to have access to the full stack in a handler. resuming isn't the only thing you can do, either - you can have the exception return to the (original) caller with some default value, you can resend the exception (or another) - it's pretty wide open.

Alan Lovejoy pointed me at a page that compares Java and Ruby, and suggested that I provide Smalltalk examples. The Smalltalk is going to be a lot like the Ruby code - both are dynamic languages with what people now call "duck typing".

The page sets up a simple example with a pair of classes - Word and Definition:

You can go to the linked site to see the Java and Ruby classes - here are the class definitions in Smalltalk:

 

Smalltalk.Words defineClass: #Word
	superclass: #{Core.Object}
	indexedType: #none
	private: false
	instanceVariableNames: 'spelling partOfSpeech definitions synonyms
'
	classInstanceVariableNames: ''
	imports: ''
	category: 'Words'

Smalltalk.Words defineClass: #Definition
	superclass: #{Core.Object}
	indexedType: #none
	private: false
	instanceVariableNames: 'word definition exampleSentence '
	classInstanceVariableNames: ''
	imports: ''
	category: 'Words'

In Ruby, it looks like you can define getters/setters in the class definition. In Cincom Smalltalk, the environment does the same thing for you, offering to define getters/setters (and a default initializer) as part of creating the class. So in step one, we get a small bit further than Ruby, due to the tools we have. As to the methods, let's start with the initiazers - I had to modify the default initializer so that it matched the code in the example page - first Definition, then Word:

 


initialize
	"Initialize a newly created instance. This method must answer the
receiver."

   word := nil.
	definition := nil.
	exampleSentence := Set new.

initialize
	"Initialize a newly created instance. This method must answer the
receiver."

   spelling := nil.
	partOfSpeech := nil.
	definitions := Set new.
	synonyms := Set new.

So on to the meat - the tools mostly created what I needed, and I just had to modify the defaults. The class definitions have a lot of things we don't need to worry about right now - I created them via the class creation wizard, seen below:

That wizard removes a lot of the typing drudgery of class creation - mind you, old hands (like me) can still do it the "old fashioned way" in the browser. Let's define the #addDefinition: and #addSynonym: methods now - they auto-check for duplicates by being Set objects:

 

addDefinition: definitionString
	self definitions add: definitionString.

addSynonym: synonymString
	self synonyms add: synonymString.

Now creating a new instance is easy - we create a class side convenience method for creating new words::

 

newWord: wordString partOfSpeech: partOfSpeechString
	^self new
	       spelling: wordString;
	       partOfSpeech: partOfSpeechString.

Some explanation - the #new method creates the new object and the #spelling: setter is immediately sent to that. The semi-colon is a cascade, which allows us to send the #partOfSpeech: setter to the same object as the first message went to. Note the name of the method: #newWord:partOfSpeec:. It's self describing. Instead of #new, we can create our own class creation protocol that is self explanatory. Now to use it:

 

"Create a Word"
word := Word newWord: 'ebullient' partOfSpeech: 'adjective'.
word addDefinition: 'Overflowing with Enthusiasm'.
word addDefinition: 'Boiling up or over'.

Notice how clear that is - and how much clearer it is than the Java example on that page (IMHO, it's clearer than Ruby, simply because of the keyword message). Keyword messages make it possible to make your code very intention revealing - which is a good thing. This is part of why Smalltalkers tend to send you to the source instead of to some doc - because the source (if done decently) is documentation. Yes, you can write bad Smalltalk, and I've done plenty of that myself :)

If you look at the explanation of collection protocol on that page, everything he says about Ruby's collections applies to Smalltalk. We can add things into the collections without fear of them blowing up. In fact, if we make sure that the things we add are polymorphically (i.e., message signature) identical to each other, it doesn't matter what they are at all. I take great advantage of that in BottomFeeder.

Moving past the example, the linked page goes into Ruby iteration. We have the same power in Smalltalk - we can iterate over any kind of collection like this:

someCollection do: [:each | each doSomethingHere]. 

The [ ] syntax is for a closure - the first place you'll find them in Smalltalk is in the logical operations (ifTrue/ifFalse, etc). They provide a lot more power though, and the fact that they are used to define the common operations (doWhile, et. al.) is telling - those operations are not reserved word syntax in Smalltalk - they are messages. Which means that you can define your own versions. I've seen plenty of people add #do: to class Object so that any object can be treated like a collection, for instance.

The entire discussion of duck typing applies to Smalltalk as well. Rather than formal interfaces, Smalltalkers worry about polymorphic equivalence. You can implement a matching protocol between any kind of objects that make sense, and then use them together. A common example used in intro classes is a FinancialInstrument - without regard to inheritance, just define a financial protocol amongst all the objects, and then use it - for instance, a #currentValue method might determine what an instrument is worth right now - and be very different for various objects that share little else. Even so, they can all be placed in a collection, and current worth calculated like this:

instruments inject: 0 into: [:subTotal :next | subTotal + next currentValue].

The comment for #inject:into:, since it may not be clear to non-Smalltalkers:

Accumulate a running value associated with evaluating the argument, binaryBlock, with the current value and the receiver as block arguments. The initial value is the value of the argument, thisValue.

That pretty much covers what the IBM page went into. The main difference you're going to see in Smalltalk is that it's not file-oriented. You work in an image, and you create code in a browser. You can save packages out into loadable modules though (we call them parcels here at Cincom) - and this server is fired up exactly that way. I have a basic image which starts, grabs a set of parcels, loads them, configures itself as a set of post-load actions, and then goes. Interested? Grab the Cincom Smalltalk non-commercial here.

Sometimes, being too aggressive about handling exceptions can be worse than not being aggressive enough. Over the last couple of months, Michael has been tweaking the NetResources library (it's in the public Store) to deal with some locking issues in the caching code. Last month, there was a change made that caused a few problems - it started trying to handle exceptions that it shouldn't. Here's the relevant code snippet:

	response := [client executeRequestDo: [:connection | client getNotifyingResponse: connection]]
		on: (self class httpExceptions, Error) do: [:ex | self behaviourForException: ex. nil].
	response ifNil: [^self].

The #executeRequestDo: message send executes the HTTP request (or not; it actually implements conditional-get). When you make an HTTP request, any number of bad things can happen - you can get timeouts, other network errors, redirects - if you don't get a response object, you'll get an exception. Some of those exceptions (like redirect) can (and usually should) be resumed - while others (server error) most certainly should not be. Still others - timeouts - can be resumed or restarted, but the decision is an application level issue.

Note the exceptions being caught - #httpExceptions is a list of http and network level exceptions, and Error is a catch-all. The logic is in #behaviorForException:. The mistake was in the old version of this method, and in the old version of the handling code. In the previous version of the above, the code was catching Error (i.e., pretty much everything, undifferentiated). Here's the handling code:

behaviourForException: ex
	ex class = Security.SSLBadCertificate
		ifTrue: [Security.X509Registry default addCertificate: ex parameter parameter].
	(self class possibleTimeoutExceptions includes: ex class) ifTrue: [^self class triggerTimeoutEvent: url].
	(ex isResumable and: [self class exceptionsWeShouldResume includes: ex class] ) 
		ifTrue: [ex resume]
		ifFalse: [self reportTheErrorType: ex]

What that does is check the sort of exception we got, and then handle it based on that information. The old code just checked whether the exception could be resumed and then did so; that led to situations where BottomFeeder would report low level socket errors - the code being resumed was in no state to be resumed. The relevant check is not only whether the exception could be resumed, but whether it should be resumed. In this case, that check is a simple check against this list:

exceptionsWeShouldResume
	"which ones should we actually resume?"

	^Array
		with: Security.SSLBadCertificate
		with: Net.HttpRedirectionError.

Note that the handler stuffs the certificate, which allows us to resume. In the original caller (excerpted at the top) there's logic for dealing with a redirect, so that gets dealt with. Other errors are presumed to be transient, and simply reported. For the purposes of an application like BottomFeeder, where we'll try to read a feed every hour (or whatever the interval has been set to), there's no reason not to ignore most errors. The only additional handling - which is done at the feed level - is to check the response to differentiate between a 404 (presumed to be transient) and a 410 (permanently gone). In the latter case, the app automatically disables the feed in question.

The bottom line is this - you shouldn't mindlessly resume exceptions. It's as sloppy as swallowing MNU errors, and gets you into states that are really, really hard to diagnose.

Update: Fixed the code snippet

One of the slower operations on the blog server right now is a text search - I don't cache all blog entries in memory, and I have the same (OS) process that gets the request fetch all the items in order to index and do the search. That's fairly expensive, and it makes the image unresponsive if it happens at the same priority as everything else.

The solution in VW terms is to have that expensive search take place in another process. The problem? Well, there's the browser on the far end that made the request, and can't deal with having the server fork the request and hand back nothing. Enter class Promise. From the comment:

A Promise represents a value that is being computed by a concurrently executing process. An attempt to read the value of a Promise will wait until the process has finished computing it. If the process terminates with an exception, an attempt to read the value of the Promise will raise the same exception.

In short, what that means is that you can push an expensive calculation off to another process, and have the one that is waiting for the result just wait for it. In server terms, said process goes into a non-runnable state, which lets other things happen.

At this point, understanding the VW process model is useful. Processes at the same priority level do not time slice. So, as requests come in from browsers, the server forks off processes at the same priority - and they all compete for the CPU. In general, if they complete quickly or end up waiting on I/O, it all works out ok. The latter is what I needed in this case - I needed to have the process wait until results were ready. Here's the old code:

	allResults := self 
		actuallySearchFor: searchText 
		inTitle: searchInTitle 
		inText: searchInText.
	^allResults asSortedCollection: [:a :b | a timestamp > b timestamp].

Inside that method, the actual search happens, which can be slow. So, I modified it like so:

	promise := [self 
		actuallySearchFor: searchText 
		inTitle: searchInTitle 
		inText: searchInText] promiseAt: Processor userBackgroundPriority.
	allResults := promise value.
	^allResults asSortedCollection: [:a :b | a timestamp > b timestamp].

What that does is set up a process that runs at the given priority, with a Semaphore signalling when results are ready. As a consequence, the value allResults isn't produced until that process finishes - and the process waiting for that result goes into a non-runnable state until that happens. Which means that other processes at that priority level don't get starved for CPU as that one munches through the backlog of posts searching.

Realistically, I need to add a cache here, and I'm in the process of doing that. In the meantime, this pushes an expensive task to the background, and optimizes server responses for everyone else.

I should do a screencast on this, but I don't have time for one today. This is the other side of the client demo I showed off a week or so ago. Let's say you want to expose some Smalltalk service as a web service - how does that work? Well, let's posit a class like this one, with a simple API. These are all reduced size images; you can click through any of them for a full size image:

So what we have is a simple class, with a method that answers a Timestamp. What we want to do is expose that as a service. So, we grab the wizard from the launcher after loading the WSDL Wizard support, and select the appropriate option:

Then, we select the class we want to expose, and select the method(s) (via the list of protocols) that we want to make available:

Next, we have to define the service interface - Smalltalk is dynamically typed, but not everything in the world is - so we have to define the type information that this API will expose. The wizard supports defining complex data types, but this example posits a simple one - Timestamp:

Next, we tell the wizard what to call the class that will support the actual server API, and what package to shove it into:

Next, we tell it to generate a client for us:

With that done, the wizard hands us some code suitable for use as a simple test case, in a workspace:

Last, we want it to generate some WSDL, so that other tools (or even another Smalltalk image) can talk to this service:

Once we've done all that, you should have a client and server suitable for testing - as seen below:

And that's pretty much it - it works the same way for more complex cases. All in all, pretty easy to work with.

With the new tab support in BottomFeeder, I thought I should explain one of the customizations - the tab menus. As it happens, the tab widget in VisualWorks doesn't support menus out of the box. So how did I add them? Well, first off, I noticed that Typeless has tab menus (Typeless also ships as a BottomFeeder plugin, which is how I use it). So, I loaded TL from the Public Store and started having a look around. It turns out that Michael extended the widget in the most natural fashion, by creating a new controller subclass. Normally, the tab bar is managed by a class called TabControlBarController. So, first thing - he created this class:

Smalltalk.UI defineClass: #TabControlBarControllerWithMenu
	superclass: #{UI.TabControlBarController}
	indexedType: #none
	private: false
	instanceVariableNames: 'menuHolder performer owner mouseDownIndex '
	classInstanceVariableNames: ''
	imports: ''
	category: 'UIBasics-Controllers'

Then you have to modify the view class, so that it uses this controller (or, at the time you create the view, replace it. This code just replaces the default controller) - here's TabControlBarView>>defaultControllerClass:

defaultControllerClass
	^TabControlBarControllerWithMenu

That makes sure that we always get menu support when we create a tab widget - assuming that the new controller class is implemented properly. Here's what's going on there:

• First, add some new instance variables to handle menus and the state required for them: menuHolder performer owner mouseDownIndex
• implement #yellowButtonPressedEvent: so as to know when to pop the menu
• implement all the other supporting methods

There's a small side story in the name of #yellowButtonPressedEvent: - back in the day, at Xerox PARC, the mouse actually had colored buttons - red, yellow, and blue. Those names still exist in the code for VW (and, I think, for Squeak). Anyway - the method:

yellowButtonPressedEvent: event 
	"we about to bring up a menu, unlock the event
	queue so that we can process expose events."
	| index |
	view numberOfElements = 0 ifTrue: [^self]. 
	index := self findElementFor: (self sensor cursorPointFor: event).
	(self owner respondsTo: #adjustTabbarMenuFor:) ifTrue:
		[self owner adjustTabbarMenuFor: index].

	self sensor windowSensor queueLocked: false.
	self processMenuAt: event globalPoint centered: true.
	^nil

The reference to #adjustTabbarMenuFor: is something I missed out of the gate. I had to implement that in my class, in order to ensure that the menus were set up correctly - it looks like this:

adjustTabbarMenuFor: anIndex 
	| tabModel sub tabbed menu |
	anIndex = 0 ifTrue: [^self].
	tabModel := self browserTabs at: anIndex.
	sub := self widgetAt: #feedID.
	tabbed := self getComponentFromSubcanvas: sub withID: #TwoflowerHTML1.
	menu := self class tabMenu.
	(tabbed widget tabBar component controller)
		menuHolder: (ValueHolder with: menu);
		performer: tabModel;
		owner: self.
	self checkTabMenuEnablement: menu

All of which does the following - make sure that a new tab has a menu, and that it's attached to that menu. As well, make sure that menu items are in the right enablement state. There's a few other setup methods, but you can see all of that by browsing the package RSSExtensions in the Public Store. The next important thing was setting the tabs up properly in my UI class. In my #postOpenWith: method (which fires after the UI opens, I added the following:

self presetTabs.
tab := self changedTab.
self setupTabMenu: tab

presetTabs
	| browserTab |
	browserTabs := OrderedCollection new.
	browserTab := RSSZoomItem new.
	browserTabs add: browserTab.
	self tabs list add: browserTab displayString.
	self tabs selectionIndex: 1.
	self setupEventHandlingForTab: browserTab

changedTab
	"changed to a new tab; adjust"

	| index  browserTab sub tabbed |
	index := self tabs selectionIndex max: 1.
	browserTab := self browserTabs at: index.
	sub := self widgetAt: #feedID.
	tabbed := self getComponentFromSubcanvas: sub withID: #TwoflowerHTML1.
	tabbed widget client: self htmlModel spec: #windowSpec builder: self builder.
	(browserTab feed isNil or: [browserTab feed isFake])
		ifFalse: [self focusOnItem: browserTab item].
	self tabs selectionIndex = 0
		ifTrue: [self tabs selectionIndex: 1].
	self setCurrentTabDetails.
	self setupHTMLPane.
	self setBrowserEvents.
	self restoreHistoryFrom: browserTab.
	^browserTab

setupTabMenu: aZoomItem
	| tabbed sub menu |
	sub := self widgetAt: #feedID.
	tabbed := self getComponentFromSubcanvas: sub withID: #TwoflowerHTML1.
	menu := self class tabMenu.
	(tabbed widget tabBar component controller)
		menuHolder: (ValueHolder with: menu);
		performer: aZoomItem;
		owner: self.
	self checkTabMenuEnablement: menu

Those three methods are the heart of the support. The first one sets the tabs up - with the proper domain model (in this case, an object that holds the selected feed, the selected item, and the history), and sets the current tab index. The second method, #changedTab, fires whenever a tab is selected. It ensures that the correct stuff gets displayed, and that state stays correct. The last one, #setupTabMenu: makes sure that menus are set up for selected tabs.

There's some infrastructure there - event handling, and some application specific stuff - but that's the gist of it. If you need help doing something like this, just send me an email.