There is a new call to action called Software Engineering Method and Theory with a large number of signatories.  Wednesdays are when Darko Marinov and I and a few of our students usually meet for lunch and talk about software engineering issues.  Yesterday we decided to talk about SEMAT and we had about a dozen people come, much more than usual.

First, I think it is great that people are acknowledging there is a problem and want to do something about it.  The state of the practice in software development is pretty dismal.  Some groups do a great job, but most do not.  As I tell the students in my software engineering course, if you manage requirements, make sure the developers talk to each other, release working code regularly, have some sort of a systematic testing process, use build and version control tools, and periodically stop and see how you are doing and how you can improve, you will be better than 90% of the groups out there.  Of course, I could be exaggerating.  Maybe it is only better than 75%.

The call to actions states:

Software engineering is gravely hampered today by immature practices. Specific problems include:

• The prevalence of fads more typical of fashion industry than of an engineering discipline.
• The lack of a sound, widely accepted theoretical basis.
• The huge number of methods and method variants, with differences little understood and artificially magnified.
• The lack of credible experimental evaluation and validation.
• The split between industry practice and academic research.

I support completely the second, fourth and fifth points.  I think the first and third are symptoms, not causes, and fairly unimportant symptoms.

One of the things that makes me worry is that when academics say "theory" they mean math, but that is not necessarily what people in industry mean.  I was very happy to see that Alistair Cockburn was a signatory, because in my opinion he is one of the best theory-developers for software engineering.  Even though he has a PhD, he definitely doesn't act like an academic.  He is a consultant and firmly in the industry camp.  His theories have no math in them, but are instead more like theories in sociology or anthropology.  That is because he focuses on people more than product. 

Part of the problem is defining software engineering.  Do they mean all software development, or only software development practiced by people with software engineering degrees? Or with software engineering in their titles?  What about the typical bank or insurance company, which has software development groups whose managers have little software development background, and where the average developer does not even have a CS degree, must less formal training in software engineering?  I think the main problem with software development is that most groups do not do the things that we have known for 30 years to be important.  Most people who manage software groups do not really understand software development, so they are incapable of doing a good job.

I think that the theory that all software developers need is the kind that Alistair teaches.  In addition to the theory, we need a set of standard base-line practices.  We need training programs to ensure that the theory and practices are known.  I think we should take a broad view of software engineering.  Anybody who is developing software for somebody else should have a certain set of minimal standards.  Certainly what NASA and Microsoft does counts as software engineering, but they are doing a better than average job and are not the ones that most need s widely accepted theory.  The biggest need is with the bottom half of development groups.

A nice description of refactorings in Eclipse.  I'm putting it here so I can find it later when I want to give it to someone.

http://www.kickstarter.com/projects/nickd/cadence-and-slang-is-a-book-about-interaction-design

This is an interesting reference to Christopher Alexander, an interesting idea for fund-raising, and what could be an interesting book.  I pledged $40 for the book and I hope to see a copy!

-Ralph

Nick Chen, one of my grad students, wrote a very interesting blog about parallelizing matrix multiplication.  His main point is that you can get better performance from low-level optimizations than you can from parallelization, and that a lot of people are parallelizing the naive, unoptimized version which means that their results are meaningless.  In general, MM is a lousy example for parallelization, since people should not be writing MM routines but should be using one of the many libraries that has it already optimized.

There are slides of Barbara Liskov's talk about work leading to her Turing Award.  They give the complete references to the papers that she said had a big impact on her work.

A closely related talk was the Onward! essay by William Cook, On Understanding Data Abstraction.  His basic point is that abstract data types and objects are different ways to achieve data abstraction.   An ADT has a set of functions that are able to access a data type.  The details of the data type are hidden from the programmer, but the functions are well known and the compiler knows the representation of the data type.   An object is a function that returns a set of functions that can access the data.  The details of the functions are hidden from the programmer, and the compiler has no idea about the representation of the data.  OO programs are much more extensible because you can replace one object with another at run-time, i.e. each time through a loop a call to an object's method can result in a different function being invoked.  However, you can't do this with an ADT, because the functions of an ADT require a particular representation.

William's point is that this is more important that identity or inheritance.  You can add identity and inheritance to ADTs and you do not get nearly as big a change as you get when you add "message sending"/"virtual functions"/"polymorphism by late-binding of function calls".

My class has been reading through Joe Armstrong's PhD thesis, and today we are discussing chapter 6, "Building an Applclication".

One of the things that I learned from this thesis is that an average Erlang programmer working on a project built using OTP does NOT write code that sends and receives messages to other processes.  Instead, they reuse "behaviors", which are components written by expert programmers that hide all the parallel programming.  There are a fairly small number of behaviors, and a typical Erlang programmer will write a purely sequential "module" and then select a behavior and parameterize it with the module.  Thus, if the behaviors are written correctly, some errors are impossible because the "modules" are restricted.

The chapter only describes five behaviors, but seems to indicate that there are more. Still, I think it is a small number.

The relationship between a module and a behavior is similar to that of a class and its abstract superclass.  Each behavior requires that the module that it runs providea a particular set of operations, just like an abstract class defines abstract methods that its subclasses must implement.

The hard part of using this architecture is breaking a system down into behaviors that are in the library.  The thesis acknowledges this, but says that with a little practice, people figure it out.  Not very helpful, but probably true.

Today we discussed three pattens from the pattern language that I'm working on with people from Berkeley.  I think all three were written by Berkeley grad students.  However, they are based on patterns from the book "Patterns for Parallel Programming".  They all from the parallel programming strategies section, Task Parallelism, Recursive Splitting, and Discrete Event.  I started off writing this before class, and am finishing it afterwards.

I thought they were all badly written.  They are too abstract.  There are not enough examples.  There are a lot of unnecessary words.  It was hard to see the pattern.

Patterns should start with a good example.  None of these did. 

I thought Recursive Splitting was the best of the group.  I knew the pattern beforehand, of course, but I knew all of them.  It suffered from using a programming language that nobody understood for the first example, but it had lots of examples.  The problem was well-stated, and it covered the important topics.  From our class discussion, it was obvious that a lot of the points were not as clear as they should be, but that is typical. 

Patterns should start with simple and obvious examples, and move on to more complex ones.  The first example of a recursive data structure in this paper was "array".  Only to a mathematician!  Most programmers do not think of array as recursive, and the paper should have started with an obviously recursive data structure like trees.  Even graphs are not naturally recursive.

Part of the problem with Task Parallelism is that the pattern in the book isn't very clear, either.  This is a broad and fuzzy pattern.  When most programmers say "Task Parallelism", they mean something different from this pattern.  I'm starting to think it is just a bad pattern, and should be replaced.  I'm sure that the grad students who worked on this tried valiantly to pull it off, but it just doesn't work.  I liked the example of Monte Carlo Method, but the students didn't resonate with it.  Several said they wanted code.  But I know that even if they understood the example, they would not have understood the pattern.

I think the pattern should be replaced with "Embarrasingly Parallel".  This is a well-known term with a well-defined meaning.  The pattern could talk about how even the easy cases of parallelism are not easy.  You have to choose grain size and figure out how to start processes.  Usually the rsults have to be combined in some way.  Currently, Task Parallelism is more than just embarassingly parallel, it slides off into systems with enough dependencies between tasks to put them in a different category, but that is why the pattern is so fuzzy.  Make it crisp!  Narrow the pattern!  Patterns should not be all things to all people, they need to be specific and identifiable.

Discrete Event is schizophrenic.  When I was reading the first part, I thought "actors / asynchronous message passing" but then when I read the example, they said "discrete event simulation".  Discrete event simulation is not the most obvious application for an actor system.  Usually discrete event simulation has a global clock, and replacing it with a distributed clock is highly non-trivial.  The other people in the discussion thought that the pattern WAS about message passing, so I have decided that the problem with this pattern is that they chose a bad example.

There is hope for these patterns.  Everyone in my class this semester is going to write a pattern (or a dozen patterns, in some cases) and the first drafts of many of them will be just as bad.  It is important to keep at it, and to not stop until the pattern is done.  These aren't done yet.

Thursday was the last day of OOPSLA.  I spent the entire day in Onward!  I had said at the beginning of the week that I would spend most of my time in Onward!, but in fact I only went to one keynote in the track before Thursday. There are just too many things to do at OOPSLA!

Also, the more observant among you might realize that I'm posting this on Saturday.  It is obvious I am not a natural twitterer.  I have so much to say about Thursday that it will take me days to post it all.

One of the Onward! essays was "An Exploration of Program as language" by Elisa Baniassad and Clayton Myers.  Elisa gave a memorable talk on it.  The key idea is that programs are languages.  In fact, they are more like natural languages than programming languages are, because programming languages are formal but the language of programs is not.  The talk gave examples of poetry and jokes in programs.

Afterwards, Dave Ungar said "This is all obvious to anyone who has programmed a long time.  What is new?"

What is new is that people have not studied programs as languages.  It might be an obvious idea, but it has not really been explored.  When researchers study programming, they think of it as theorem proving or theory building, but not as constructing a new language.  They have not taken ideas from linguistics and applied them to programs.  The talk didn't give much insight into how linguistics shines a light on programming, except by using the analogy that learning a large program is a lot like moving to Hong Kong and learning Chinese.

I liked that analogy, because I tell students every year that learning a large system is like moving to New York.  At first you just remember a few landmarks and carry a map.  If you are like me, you might sketch a simplified version of the map and try to memorize it.  You start to learn neighborhoods, and how to get from one place to another.  Eventually you are comfortable with going to a new place.  After half your life, you might even tall people you have "mastered" New York, but of course nobody knows everything about New York.  it is just too big.

I've been reading a book called "The Power of Bable" by John McWhorter, which is a book on how languages develop by a top-notch linguist.  I bet he quotes from a hundred languages in this book.  I can't believe one person could know so many languages!  It is very well written.  Literate, modern, insightful. 

"After all these years, the true roots of my fascination with language remain the same as on that day on a Philadelphia sidewalk when I lost my innocence: that anything I write or say in this language can be said in about six thousand other ways, with completely different words and with grammars so different that they can almost strain the credulity of the outsider."  Page 3.

The book is about how languages develop, grow, and change.  One of the main points is that all languages change all the time.  Change is normal for languages.  He talks about pidgeons, which are languages that nobody speaks natively but that two or more groups use to speak to each other, and creoles, which often start as pidgeons but that people now speak natively.  He gives rules for how language borrow from each other, and how they change by simplifying or by differentiating.  It sounds a lot like refactoring!

One of his main points is that there is no different between dialect and language.  To put it very strongly, the differences between British English and American English differ only in degree, not in kind, to the differences between American English and Mandarin.  I know how unbelievable this is.  I will not try to convince you.  I will only say that if you are the least bit intrigued, you should get the book, and you will probably be convinced.

When I first heard about the idea that programs were languages, I said to myself "that is certainly true for big programs, but it isn't true for small programs".  But that is just like saying that Mandarin and English are certainly different languages, but British and American are not.  McWhorter would agree that it makes sense to say that every program is its own language, though some program languages are very similar to others.

There are so many thoughts swirling through my head that if I keep chasing them, I will not finish this today.  So, I will close with the chapter titles of "The Power of Bable".  I love these titles!  They show that John McWharter is creative and thoughtful even about chapter titles.

The First Language Morphs into Six Thousand New Ones

The Six Thousand Languages Develop into Clusters of Sublanguages

The Thousands of Dialects Mix with One Another

Some Languages are Crushed to Powder but Rise Again

The Thousands of Dialects of Thousands of Languages All Developed Far Beyond the Call of Duty

Some Languages Get Genetically Altered and Frozen

Most of the World's Languages Went Extinct

Next semester, I am going to run a seminar studying this book and trying to apply it to programming.  If you want to participate in some way, please send me e-mail.  This applies especially to people not students at Illinois, since they will get other announcements of it.