How to do an Http Query

One of the things that JIS asked for was examples - examples of how to do various things in Smalltalk. Well, we've been talking about this at Cincom, and some of us are going to start posting examples from parts of the product that we are familiar with. I've done a fair bit of work in HTTP for BottomFeeder, so I figured I'd go through the basics on that. In general, when I post examples, I'll be using the examples category

How to do an HTTP Query in VisualWorks

If all you need is a quick result (when dinking in a workspace, for instance), you can just do the following:

HttpClient new get: 'http://www.yahoo.com'

Now, that will work - and it will even do the right thing in the face of a redirect (either 301 or 302). However, it's not terribly robust - it won't handle any of a variety of of hiccups you might get from a web request (server errors, timeouts, etc). Sure, you can wrap that message send in an exception handler (handling an ExceptionSet of issues) - but you still won't get all the things you want. You will likely want to be able to do things like:

• Specify the content-type
• Specify what kinds of responses you'll handle (mod-gzip, etc)
• Specify the User-Agent and/or the Referer string
• etc

For that, you need to dig into the ancillary classes surrounding the basic HttpClient (and HttpsClient) class and construct a custom request. Here's how you would create a basic HttpRequest object:

setUpRequestFor: url
	request := HttpRequest new.
	(request getFieldAt: 'User-Agent') value: 'My Agent String Here'.
	request contentType: 'text/html'.
	request accept: '*/*'.
	request method: 'GET' url: url.
	

Now, that's hardly an exhaustive list of data that you can attach to a request. You can specify things like cookies, and conditional-get information - but you get the idea. The (request getFieldAt: 'blah') value: 'value' pattern can be used to specify arbitrary headers.

So we have a request object. What do we do with it? We execute the request and we get a response. Here's how you execute the request:

httpExceptions
	| set |
	set := ExceptionSet new.
	set add: HttpRedirectionError. 
	set add: HttpObjectNotFound.
	set add: Smalltalk.OSErrorHolder notReadySignal.
	set add: OS.OSErrorHolder notReadySignal.
	set add: OS.OSErrorHolder peerFaultSignal.
	set add: OS.OSErrorHolder inaccessibleSignal.
	set add: OS.OsTransferFaultError.
	set add: HttpServerError.
	set add: HttpTimeout.
	set add: HttpStatusLineError. 
	set add: SSLBadCertificate.
	^set

executeRequest: request
	| response |
	exceptions := self httpExceptions.
	client := HttpClient new.
	client keepAlive: true.
	^[client executeRequest: request]
			on: exceptions
			do: [:ex | nil].

Now, even that's not the full range of what you might need to do. In BottomFeeder, I trap a few specific exceptions for handling. With that out of the way, what do I do with the response? For things like redirects, I have to check the response:

httpGetFor: someUrl
	request := self setUpRequestFor: someUrl.
	response := self executeRequest: request.
	(response notNil and: [response isMoved])
		ifTrue: [self handleMovedObjectFrom: response].
		ifFalse: [response notNil and: [response inUnauthorized]
					ifTrue: [self handleAuthorizationRequestFor: response]]

Now, bear in mind that handleMovedObject: and handleAuthorizationRequestFor: are not part of the standard API. As it happens, the standard Http libraries handle basic authorization and redirects - but not necessarily in a way that is sufficient for applications (i.e., BottomFeeder wanted to silently re-label feed urls, and it wanted to handle Digest Auth - and the standard libraries do not do Digest Auth yet). Beyond those two, this is all boilerplate VW code. So let's put the "high level" API we've just defined in one piece:

responseOrNil := self httpGetFor: someUrl

Now we have a response (or nil). We can send the contents message to that and get the results. Then you can do whatever is appropriate - parse it, store it, display it, whatever. The basics of this are in two packages in the public store - Http-Access, which is used in BottomFeeder now, and a much cleaner implementation in NetResources, which is what Bf will be moving to shortly. I'll go through how to construct a POST next time I give an example.