I was at ECOOP last week, though I didn't stay til the end. My favorite part was the talk by Joe Armstrong on Erlang, and talking with him afterwards. I met Joe last year and have been following Erlang every since.

Erlang is going to be a very important language. It could be the next Java. Its main problem is that there is no big company behind it. Instead, it is being pushed as an open source project. Its main advantage is that it is perfectly suited for the multi-core, web services future. In fact, it is the ONLY mature, rock-solid language that is suitable for writing highly scalable systems to run on multicore machines.

Erlang started out over 20 years ago as a kind of concurrent Prolog. Joe Armstrong invented it, and he has been the main person pushing it. He was at Erickson, the Swedish telecom company, and the first big Erlang project was a electronic switching system that was built by a few hundred people, who wrote a few million lines of code. The emphasis was always on reliability, not raw speed, and the system has an incredible reliability history. Joe claims they have achieved "nine 9's of reliability".

What does that mean, "nine 9's of reliability"? It means 1 second of downtime in 1 billion seconds, or 1 minute of downtime in 1 billion minutes. Now, a billion seconds is roughly 30 years. A billion minutes is roughly 2000 years. This system has been in production for ten years or more, but less than 15 (I think). They have sold hundreds of them, perhaps thousands, but I think hundreds. Two hundred systems at ten years apiece give 2000 years of operations, so they can say they have "nine 9's of reliability" if they have had less than 1 minute of downtime total for all the systems they have installed.

"Five 9's" is about five minutes of downtime in a year, and is pretty good. People who are fanatical about reliability want six 9s, or even seven 9s. It is unheard of to have nine 9s. But a system built in Erlang has done it.

That is not what is going to make Erlang big, though. Not enough people care about reliability. Another thing that is not going to make Erlang big is that "sequential Erlang" is a functional programming language. Or that "concurrent Erlang' is an object-oriented language. The thing that is going to make Erlang big is that it is the only mature language with a rock-solid implementation and good set of libraries that lets you write software that can scale seamlessly from a single processor system to a hundred processor system. In a few years, all our desktop systems and laptops will be multiprocessors, and the only way to make our applications run faster on them is to make them use multiple processors.

When you build a system in Erlang, you write a set of processes that communicate only by passing messages between them. There is no shared state; the only way to communicate with a process is to send messages to it. (Or, to have it write to the file system, which is cheating. Or call C code, which is also cheating. There are actually lots of ways to cheat, but we'll ignore them.) Unlike Java or Smalltalk, where you only write threads/processes when you want concurrency, Erlang programmers use processes for modularity, reliability, and reuse. Then they get concurrency for free. Go build your application on a single processor. In theory, you could write it all as a single process, but no decent Erlang programmer would do that. They are more likely to write it as thousands of processes. It doesn't hurt performance on a single processor and makes it likely that it will make good use of a multiprocessor. Then, put it on a ten processor system and make your system run ten times faster (probably eight or nine times faster, but that is still pretty good).

Of course, just because you wrote your system as thousands of processes doesn't mean that it is scalable. You could have bottlenecks, as with any system. Processes could spend all their time waiting on other processes. But messaging is asynchronous in Erlang, and it is common to send a message to a process and to forget about it, expecting the result to be forwarded to another process. There are a number of design patterns in Erlang that tend to make systems scalable.

Erlang comes with a bunch of libraries. A lot of them are for building or using various kinds of internet services. Erlang has web servers and databases. The Erlang community has been positioning it as the language for choice for building reliable web servers and web services. But the package I find most interesting is the OTP, or the "open telecom platform". Not surprisingly if you look at the names of Erlang packages, it has nothing to do with telecom. It is a framework/platform for building systems that can run decades without being turned off, while updating your software every day and even replacing your hardware periodically. This is needed by telecom applications, but is also useful for on-line banking, on-line stores, and any web service that you want others to build upon.

Joe Armstrong as just finished a book on Erlang that is being published by the Pragmatic Programmers. Joe has a article about his book, too.

It is a very good book, and must reading for anybody interested in Erlang. The thing that bugs me about the book (and about his talks) is that he make more fuss than he should about the functional language aspect of Erlang and not enough about the OO aspect. In fact, he denies that it is OO.

Processes in Erlang are objects. At the beginning of my course on OO design, I explain three views of OO programming. The Scandinavian view is that an OO system is one whose creators realize that programming is modeling. The mystical view is that an OO system is one that is built out of objects that communicate by sending messages to each other, and computation is the messages flying from object to object. The software engineering view is that an OO system is one that supports data abstraction, polymorphism by late-binding of function calls, and inheritance.

Erlang is a perfect example of the actor model, which is an example of the mystical view. Processes certainly support data abstraction and polymorphism. An Erlang process is a function that reads from the incoming message queue, pattern matches to find a particular message, and then responds to it. A function structured in this particular way is similar to a class in Smalltalk. Moreover, given several kinds of processes that have a common protocol and that share some things in common, it is easy to factor out the commonality into a function that they can both call. This is similar to class inheritance. So, you could even say that Erlang supports inheritance, though it does it very differently than in Java or Smalltalk. I imagine that many Erlang programmers think about programming as modeling. So, Erlang fits all the characteristics of an OO system, even though sequential Erlang is a functional language, not an OO language.

One way that Erlang differs from OO languages is its emphasis on failure. Any message can fail. Processes don't raise an exception, they fail. Systems are structured as worker processes at the bottom that are likely to fail, with manager processes above them that restart the failed processes. Because programmers expect processes to fail, they

Joe makes too much of functional programming because he says that lack of mutable state implies no locks. However, it is really lack of SHARED state that implies no locks. You could write processes in Basic, perl, or C. I'm sure that lots of people will look at Erlang and say "we can add that to our language". In my opinion, it is the concurrent programming aspects of Erlang that make it special, along with its mature implementation and powerful library designed for concurrency and reliability.

I do not believe that other languages can catch up with Erlang anytime soon. It will be easy for them to add language features to be like Erlang. It will take a long time for them to build such a high-quality VM and the mature libraries for concurrency and reliability. So, Erlang is poised for success. If you want to build a multicore application in the next few years, you should look at Erlang.

I got a notice from the library that a book was overdue. Before I return it, I will write a brief review.

The book is "Building by the Book: Pattern-Book Architecture in New Jersey" by Robert Guter and Janet Foster. The book describes the effect of "pattern books" on New Jersey by showing pictures of hundreds of New Jersey houses and comparing them with the patterns that inspired them.

In the 1700s, there were two kinds of architectural books. One was for very rich landowners, and these expensive and elaborate books described how to build mansions and palaces. The other was for builders. These were much smaller, designed to fit in a carpenter's pocket, and showed how to build things that the customer imagined.

In the 1800s, there arose a new kind of architectural literature called the pattern-book. These were designed for middle income people, and described small cottages just as often as bigger homes. They were mass produced and so were readily available. They had a huge impact on houses in the US.

I quote from page 5 of the book:

"Because of its position between the great metropolitan centers of New York and Philadelphia, New Jersey was the suburban conduit that carried much of this pattern-book intelligence and saw it produce built examples. In the 1870s and 1880s a veritable colony of pattern-book authors lived in Bergen County, where several towns became laboratories for their work. Both George Woodward (author of Country Homes 1865) and Elisha Hussey (who wrote Home Building in 1876) maintained New York City offices but settled their families in Rutherford, a burgeoning railroad suburb, where houses from their designs were built. At about the same time, Daniel T. Atwood expanded his New York City architectural practice to New Jersey, where he became a developer as well when he bought a large far tract for subdivision in Tenafly Many of the homes he built there appeared in Atwood's Country and Suburban Houses (1871) and Atwood's Modern American Homesteads (1876),"

Apparently these pattern books died out with the growth of architecture as a profession.

Recently there has been a discussion on the squeak-dev mailing list about when an RDBMS is appropriate. The discussion started with a comment by JJ, who said that they were very appropriate for the right problem. Howard Sterns naturally asked what problems were the right problems. This set off a flurry of messages, only a few of which I am noting.

Many people on the list liked OODBMSs, but Todd Blanchard (in his blog) argued against them. James Foster said that Gemstone did not have these problems. Nobody noted that Gemstone demoed at the first OOPSLA in 1986 and is a respectable sized company, while the OODBMSs that people criticize are either fairly new or with very small companies or both. Making a DBMS fast, reliable, etc. is hard. When you reuse a software component you are making your system depend on it and if your system is mission-critical then you do not want it to depend on software that is not reliable or is not well supported. One of the main advantages of RDBMSs is that they are old, boring technology that is produced by companies that have been selling them for a long time and that knows how to support them. And make money off them!

Gemstone is a great alternative if you are using Smalltalk. It is expensive, but not compared to large Oracle systems. I know they have some bridges so that non-Smalltalk programs can use it, but its whole model is so Smalltalk-oriented that I can't see why any company would use it unless the most critical parts of their software were written in Smalltalk. And as far as I know, none of the other OODBMs are as good as it is.

Goran pointed out that systems based on RDBMSs tend to grow unpredictably and become big balls of mud. But most systems tend to grow unpredictably and become big balls of mud! The usual argument for RDBMSs is that by making a good data model, you can have all sorts of applications reuse the data, and that the data will live much longer than the applications. As Goran pointed out, this would be nice if it happened, but in practice, the data is often highly coupled to the application.

If you don't have good architects, big systems will end up as big balls of mud. A lot of companies live with it, but there are certainly big payoffs if you can avoid it. The main problem is that there aren't enough good architects to go around. One of the advantage of a RDBMS is that it is fairly easy to understand so below average programmers can still get systems running. Below average programmers will not build the best architectures, though. So, the fact that RDBMs lead to big balls of mud is actually a sign of an advantage. You can use a RDBMS when you have good architects and when you don't. You'll end up with completely different systems, but that is because of the quality of your architects, not because of the technology you use.

I posted a response, too. I said that my favorite persistence pattern was the one of Prevayler, but that an experienced architect should know about XML, OODBMSs, and RDBMSs, because each had its place. In fact, there are at least two ways of using an RDBMS from an OO language. One is to use a object-relational mapping system, but another is to use SQL directly and to have the domain model in tables rather than the objects. I prefer a slight generalization for Smalltalk; have a library of Query objects that let you model the queries as objects rather than as strings. This works well when your applications are basically analyzing existing RDBMs and not changing them.

One of the advantages of thinking of RDBMSs as a pattern is that it makes you stop asking "is it a good idea" and start asking "what are its advantages and disadvantages" and "when should we use it instead of its alternatives". For example, if speed is your number one criterion then you probably shouldn't use a RDBMS. That is why RDBMSs are big in enterprise applications and not in supercomputer applications. An enterprise app must be fast enough, but if you can process all your transactions every day and can give responses to people in a second or less, your app is probably fast enough. Enterprise developers often complain about performance, but compared to real-time developers or supercomputer programmers, performance is not very important to them, and the fact that RDBMSs are so popular among them is proof of that. On those occasions where performance IS important, get a new database.

One of the advantages of an RDBMS is that it tolerates bad data pretty well. It is usually not hard to look at your data and see what is wrong. XML is like this, too. OO databases tend not be like this, though Gemstone is an exception because you can use Smalltalk tools to inspect and modify the data.

I started this a week ago and have more to say. I will post it and hope I have time to add to it soon.

I'll be in NYC Dec 11-15. I might have an evening free. If you are interested in meeting me when I am there, send me some e-mail.

The last few weeks I've been looking at Pavel Krivanek's project to break Squeak into smaller modules. Unlike most people, who start at the top and try to carve out pieces, he is starting at the bottom. He has developed several images. KernalImage is a Squeak image without any GUI. It has collections, numbers, classes, the compiler, but little else. It has a text-based UI that can evaluate Smalltalk expressions, and it can load files from disk. Pavel has a fileIn for Monticello, so then you can load any other package, as long as it doesn't have a GUI.

His other image is called MinimalMorphic. He has a Monticello package for it, so you can load it from KernalImage. MinimalMorphic isn't particularly minimal, but it has had some stuff (like eToys) removed and I think he wants to remove more. It is a basic Morphic environment, showing that you can separate MVC from Morphic.

KernalImage is 2.8 megabytes, which is a lot smaller than the 15 megabytes of the 3.9 release.

The really cool part is that all this is generated from a model of Squeak. He has a "bootstrapping" system that has a model of Squeak in Squeak. You can start with the final 3.9 image and fileIn a model of the KernelImage. His system will figure out what needs to be deleted from the image and create a fileIn that deletes everything that shouldn't be there. Then you start with a fresh 3.9 image, fileIn his deletion script, and you get the KernelImage. You can then use Monticello to load MinimalMorphic.

I found it hard to do much with KernalImage except to load MinimalMorphic. Pavel says you can load Seaside and that there is a Seaside app that will let you browse code in the image. But the way I have looked at the classes in KernalImage is to load the bootstrap environment.

Pavel keeps his sources at http://www.comtalk.net/Squeak/98

To get an image with the model of the KernalImage in it, unzip bootstrap-7067.zip

Then, starting with a 3.9 release of Squeak, file in

- mirrors exporting fix

- ChangesOrganizer

- wait2ms

- Bootstrap

- Bootstrap-Tools

- Mirrors

Open a "mirror browser" with MirrowBrowser open, and you can browse the KernalImage classes.

The whole system is undergoing change. But I think it represents the future of Squeak.

One of the important events of OOPSLA was the long-awaited first paper on Intentional Software. This paper was part of Onward!, which is a track that focuses on high-risk but potentially high-impact ideas. Charles Simonyi has been working on this for years, but his only previous presentation was a talk at USENIX in 1997. Most of what we know about the project comes from what other people have said about it, because lots of people have seen demos and the work has not really been done in secret, though something it seems that way. The paper doesn't reveal anything surprising, but gives a general overview of the topic. It is important because it is a stake in the ground.

I just watched a video of Martin Fowler giving a talk at JAOO on Domain Specific Languages. If you find the Intentional Software paper hard to understand then you should watch this video. In fact, you should probably watch it anyway! As usual, Martin does a wonderful job of taking ideas that everybody else makes complex and making them simple. He focuses on the heart of the issue. He comes up with simple examples that illustrate the point perfectly. If you like his talk, read his papers.

My only complaint about Martin's talk is that he didn't mention Smalltalk and that he said that the Lisp people have been making DSLs for 20 years. I learned about them about 20 years ago, and the Lisp people had been doing it for 20 years before that, so I think they have been doing it for 40 years. Neverthless, DSLs are only starting to become a hot topic. OOPSLA has had a Domain Specific Modeling workshop for four or five years, and I'm hoping that next year it will grow into a symposium. These are ideas that need to become ubiquitous.

Martin's talk reminded me that XML is often used for DSLs, and that this is closely tied to Intentional Software. I've been thinking about better refactoring support in Eclipse for all the XML files that go along with modern Java apps, and I think this insight will help.

I was planning to post about OOPSLA before I went, but I got so busy getting ready that I never did. So, I'll give some of the highlights from my point of view. For those who don't know, OOPSLA is the leading conference on object-oriented technology, the place where new ideas tend to come out. This year, the main new thing that I saw was Ultra-Large Scale. There was a keynote, a panel, and a workshop about this idea. An ultra-large scale system is to most computer systems as a city is to a building. There is no one group in charge. There is constant change, constant failure, and no requirements. Nevertheless, it all works. A ULS system is too big and important to turn off, so it keeps changing as it runs. The internet is the best example, though lots of companies' internal systems approach ULS because they grew out of many systems that were built or acquired over many decades. There is an SEI tech report on ULS.

One of their catch-phrases is "Scale changes everything." This is true if you ask about designing a ULS system, but not if you are talking about designing a component of a ULS system. Designing a ULS is more like making zoning laws than making UML diagrams. The focus is entirely on architecture. But making a component of a ULS system is about dealing with change, with security, and with reliability in the face of a hostile world. In other words, it is fairly similar to what we are doing now.

Martin Rinard had a keynote on failure-oblivious computing. His main point is that software should detect certain errors, prevent others, and just keep on going no matter what. It should never crash. He gave three examples of problems that could be handled this way; buffer overruns, memory leaks, and infinite loops. Buffer overruns are a problem with C but not Java or Smalltalk or any other language that checks array accesses. Memory leaks are less of a problem with a language with automatic memory management, but not entirely. Infinite loops are always a problem. He had good results, but I think you would get better results in a language like Erlang where you ensured that the overall system never crashed by making it out of components that frequently crashed. A buffer overrun, use of too much memory or too much time would all cause a process to be killed. A higher-level manager process would restart. I think this would be a better way of building web apps, which was the kind of application that Rinard was looking at.

I thought it was interesting that Rinard's topic was so closely related to ULS. My guess is that the OOPSLA organizers designed things that way.

There were lots of good things from OOPSLA, and I'll post about more later.

Scott Rosenberg is discussing the Mythical Man-Month, which is one of my favorite books.

One of the comments that someone else makes about the book is:

Developers who've worked on large projects (but taken a think-first approach) are familiar with the phenomenon in which there's initially little externally-visible progress, but then a critical mass is achieved and things appear to move with incredible speed. The later rapid production of results is crucially dependent on the original thought processes, in which the developers seemed to be wasting time exploring alternatives, or just sitting and thinking.

This comment is closely related to Domain Driven Design, which I am reading for the fourth or fifth time. Eric Evans calls the process of taking in lots of facts and making a good domain model "knowledge crunching". It always involves a lot of flailing around. Once you get a good model then you can make a lot of time, but it can take awhile. Eric does not advocate a "think-first" approach, though. He wants people to get experience, to experiment. He advocates starting with a naive model and then refactoring it into a more powerful, more supple one. But he still expects to see the phenomena of slow progress until a critical mass appears.