PackageDescription: BlockPipeline


Block Pipeline

Last published: June 6, 2007 by 'mkobetic'

Defines 2 Classes
Extends 3 Classes

This allows to link a sequence of 2 argument blocks and execute them in parallel. The blocks have an :in and :out argument and are hooked up so that the :out of one block is linked with the :in of the next block. The hook up is done using a kind of a shared read/write stream. For the first prototype we'll use a SharedQueue, but extended to provide an indication of pipe closure to know when to terminate the associated process. The extension is called FiniteSharedQueue and you can #close it and test if it is #atEnd.

The package is used like this

( [ :in :out | out nextPut: $a asInteger + in next - 1 ] |
[ :in :out | out nextPut: in next asCharacter ] |
[ :in :out | out nextPut: in next asUppercase ]
) out: Transcript;
in: (1 to: 26) readStream

It will require adding dummy TextCollector>>close to make it work, but it's cute this way :-) .

LIMITATIONS

Two issues come to mind immediately.

1) If you want to get serious with this, it would probably be good to replace the shared queue with some sort of shared, fixed size, circular buffer to prevent one process from generating a huge intermediate queue without letting the next one consume it. That should help with the flow through the pipeline. Ideally the buffer would also be fully polymorphic with streams, so that you can use the full arsenal, including #next:into:startingAt:, nextAvailable: and such.

2) There is the off chance that the processes can get themselves suspended without being held strongly and get GCed while running. I have a version that holds onto the processes in a 'workers' collection in the BlockPipeline to prevent that. But I didn't want to complicate the picture with that. The workers collection may be useful for other uses too, providing a nice handle on the processes.