I've moved my blog to a new server, mlucassmith.tumblr.com. It's not actually a full copy of this blog, so this blog stays - but it is my new blog, where I'll be posting new content. Please subscribe.

Recently I have been working on some C++ classes which make heavy use of templates. In particular there are two classes Vector, T, N and Matrix<T, M, N=M> where you specify the type of the elements and the number of elements for both. Examples of how you would use these are:

Vector<double, 3> myVector(1, 2, 3);
Matrix<float, 3> myIdentityMatrix;

As you can imagine, typing out that full declaration everywhere gets a little tiresome after a while. What I really wanted was a way to alias Vector3 to mean Vector<double, 3> yet at the same time be able to easily do Vector3<float>.

The most logical thing you might try and do is to template the typedef, eg:

template <typename T=double> typedef Vector<T, 3> Vector3;

However, you'll quickly discover that this is an unimplemented feature of C++0x. So what can you do instead? The simplest thing that came to mind at first was to make a subclass:

template <typename T=double> class Vector2 : public Vector<T, 2> {
Vector2 (void) : Vector () {}
Vector2 (T x, T y) : Vector (x, y) {}
};

The obvious problem here is that you have to re-implement the constructors and destructors each time you do this and if new constructors are added, they have to be added to the subclasses as well.

A thread on stack overflow had a different solution to the problem. While C++ doesn't let you template typedefs, it does let you template structs, which can contain typedefs:

template <typename T=double> struct Vector2 {
typedef Vector<T, 2> type;
};

This approach is a true alias, however it requires you to declare your variables like this:

Vector2<>::type myVector;

That's almost as much typing as you would be doing before - albeit without having to redeclare the types and dimensions every time. As I discussed this problem with a friend, he suggested that I template a namespace and then import the types with using, but it turns out that the following is invalid:

using LinearMath::Vectors<>;

You cannot have a template-id in a using-declaration. It introduces ambiguities in to the namespace. But this got me thinking about importing stuff in to your namespace and I realized I could actually do it using subclassing. Since all my code was generally in classes already, it was a simple matter of creating a class that I could inherit from.

template <typename T=double> class Vectors {
public:
typedef Vector Vector2;
};

class MyClass : Vectors<> {
Vector2 myVector;
};

What's particularly neat about this is that all the code in my class gets to uniformly use the same vector types and matrix types where appropriate - and they are true aliases, so two compatible types blend together without issue.

This goes to demonstrate the already well known fact that C++ is a very unpolished language with lots of rough edges. Templates exist, but they are not uniform across the language and there are many holes you can easily fall in to. Without namespace templates, typedef templates, using templates.. I was luckily able to find a solution using multiple inheritance.

Recently I was asked to make a demo application that makes a standard WebVelocity application using scaffolding with active record look and work on the iPhone. There are three tricks to making a web page look like a native iPhone app: 1) colours, 2) orientation detection, 3) animations. In this demo I show off #1 and #2 and leave #3 for any adventurous jQuery users out there. The demo is in the Examples repository of WebVelocity under the name 'IPhone' and here is a video of it in action:

http://dl.dropbox.com/u/20291/Cincom%20Smalltalk/WebVelocity-20100804-iPhone.mov (5.2mb)

As you read this (assuming you're not reading this months from now) the final product media is being cut and burnt and tested. The software is done and as proof of that, you can actually start using it right now on Amazon Web Services.

So now that we've reached this milestone for the product, I wanted to talk about what it has become. It started off as VisualWorks with Seaside and Glorp and a few web tools to help you build applications more efficiently. But for 1.1, we thought bigger — what if we focused on making it super easy to develop and deploy your applications in the cloud itself?

We revamped all the tools, removed the old GUI layer and have the whole environment running in a web browser using HTML5, Javascript, Smalltalk, jQuery, Sweat, Blood and Tears. Now that we're in the web browser fully, you can use tabs to divide your work up and you can collaborate with other developers by sharing the web address around.

We wanted to make it super easy to get up and running this time around. With version 1.0 you had to run your own server or pave your own path to run it on top of Amazon EC2. This time we come packing with a complete solution running on top of EC2 and Amazon RDS if you so desire.

I've never worked on a product from scratch before that has so many buzzwords attached to it that it actually does. You can literally have your development environment up and running in the cloud, developing application collaboratively, deploying them to multiple instances in the cloud, load balancing it and sharing a cloud based database system in fifteen minutes flat. In fact, the most awkward part of the whole experience is the lack of tools from Amazon for Amazon RDS.

After watching the videos below it becomes difficult to think of WebVelocity as VisualWorks with a web skin over the top of it. It has become its own creation, a mythos of Smalltalks prowess all to its own. But unlike other variations of Smalltalk, we've done our best to reduce barrier to entry. Any one who can afford a few cents to fire up an Amazon EC2 instance can be using Smalltalk in minutes. We have examples, documentation and state of the art technology under the hood.

We started off chasing the tail of Ruby on Rails but at this point it's futile to compare ourselves to them too. Ruby has never believed in tools — we do, in a big way. Can you imagine writing code in a web browser, running it in a second tab to see how it works, discovering a bug and debugging with edit and continue in your main tab and then having the second tab instantly finish loading as if nothing had ever gone wrong? ... now can you imagine that you don't know how to fix the bug, so you just send the bug over to a friend working on his laptop at a coffee shop and having him fix it in moments?

Today we can get a true glimpse of what the future of cloud computing will be like for everyone. For we who are in the know we can start enjoying its virtues today. Please enjoy the videos produced by my colleague Jerry below:

• WebVelocity on Amazon Web Services part 1
• WebVelocity on Amazon Web Services part 2

Expect to see the website updated soon with the release and if you're a customer already, expect to get the commercial CD soon.

Six years ago I wrote a blog post about development utilities that I used on a day to day basis in VisualWorks. I stumbled across it in a Google search today and I thought it'd be interesting to contrast what I was using back then with their status today.

• UIPainter -- Almost nothing has changed with this part of the product since 2004. For the most part, it does its job fairly well, but I personally think it's starting to show a bit of ware and tear with age.
• ProgrammingExtensions -- This is as useful as it was back in 2004, specifically for the extension that lets you middle click on a widget and inspect it.
• Method-History -- this extension no longer works. This differed from the two built-in tools in that it monitored the system for changes and recorded them so you could find what changed quickly. We have two tools in the system, the first scans the changes file for the method in question and the other looks up store versions of the method. Squeak makes the changes file scan fast by indexing it - we do not. Still, all in all, I don't miss this tool - but I might consider re-introducing it as an extension to Searchlight one of these days.
• AutoComplete -- I haven't had an auto completer loaded in to my image in -years-. This is quite bizarre because I use them all the time when I'm working in other tools like Xcode of VirtualStudio. The state of the art here hasn't changed, but I suspect one of these days we may hook it up to the Searchlight database for faster suggestion searching.
• ExtraEmphasis -- This has slowly plodded along and will one day help push the state of text attributes forward in VisualWorks. For now, it's still in use by many people - but it's not in the base.
• MouseWheelX11 -- The bug was fixed, this package is obsolete.
• FileProgress -- This parcel is still around and it might be useful when you're copying files around. I haven't even thought of this thing since 2004 apparently.
• RBSUnitExtensions -- I actually use SUnitToo(ls) now because most of my tests are written on top of SUnitToo. This package is still around and works, but I prefer SUnitToo(ls) integration in to the code browser better.
• RBRegexExtensions -- I use this on a day to day basis. It's just one of those useful swiss army knives. I want to integrate this kind of capability in to WebVelocity.
• RBStoreExtensions -- I haven't used this in a long time. I use some internal tools we have here at Cincom which integrate in to our issue management system instead.
• SmaCC Runtime -- I haven't needed to load this in a long time. I prefer using PEGs now in Xtreams where I can. The PEG technology is behind the syntax highlighters in WebVelocity 1.1
• RBBytecodeTool -- Still the same as it was before. It's neat for learning how bytecodes are applied under the hood in VisualWorks but not really useful day to day.
• ThreePaneSelectorsBrowser -- I've had reports that this might be broken in the latest version of VisualWorks. I don't use it any more, I use Searchlight-Tools instead. It doesn't do quite the same job (allowing you to quickly see related methods to a search) but it does other things much better.
• RBCodeHighlighting -- This guy has been upgraded over the years and has many extensions to it - such as a spell checker!
• HyperRBCodeHighlighting -- This one fell off the wagon a while ago and hasn't had much love compared to the RBCodeHighlighting package.
• Arbor Hypertext -- Avoid if at all possible I'd say. This is oooolllddd tech. Heck we integrated URL navigation in to the comment panes a year or so back as part of the RBCodeHighlighting package and the base image.
• All Advanced Tools -- Same as it was before, the profiling tools are the most important part of this package set.
• ImageCompression -- Well it's there and I use it, but it's one of those tools that lives on the side of my packaging process. Set and forget.
• ThrottledCursor -- The bug was fixed, this package is obsolete.
• NewPrerequisiteEngine -- Integrated!, this package is obsolete.
• DMTrippyEnhancements -- I'm not sure if this is compatible any more, but it might be. It added an XML pane when you're inspecting XML dom nodes, which was neat and useful for me at the time.
• ExtraIcons -- Deprecated, new icons are integrated the whole environment.
• StackOverflow -- Still useful from time to time, especially when you're running code in a sandbox.
• NextVersionPundlePublishing -- I haven't looked at this in years, I don't even remember what problem it was trying to solve exactly.
• File Repository -- Defunct, file versioning is supported by Bundles, however that support is pretty awful. Personally, I'd just use svn, git or mercurial now.
• LearningMenus -- It did its job when I made it, data discovered with it was used to enhance the product as a whole over several years. We may resurrect this when designing new UIs from time to time.
• SpsMergeToolMods -- Defunct, there are a whole new set of tools for merging and changes in the product now.
• AddInstVarAtCompileTime -- Merged a long time ago.
• ProportionalTabs -- Perhaps this might be useful to people still, but I haven't seen the issue it was addressing in a while now. It may have been the kinds of UIs I was making back then had a lot of tabs, but that doesn't bother me much these days.
• RBDiffs -- Integrated.
• RB_Tabs -- Not integrated, but really I think this type of functionality should be in the product.
• BuffersAreBack -- I guess this is still relevant, if you want 'old style' buffers in the RB - the RB already has buffers though and they work just fine.

Recently I was working on a relational database and I was juggling the eternal act of normalization with efficiency. There are many books about database normalization written by people who love writing books about database normalization. I've read a couple myself and it all seemed reasonable back in university.

But it's the 21st century and it dawned on me that normalization is the futile act of trying to manually compress data. If you break up your data records in to its related component parts to -avoid duplicate data- then you're literally trying to re-invent compression ..badly.

So let's say we let compression deal with our data storage. Smart hashing would mean you'd easily be able to store just as much data in "document" format. But how do you identify when two records are actually the same? I suspect this is the job of hashing and indexing. If two people have the same (and more interestingly - similar) address, statistical machine analysis will find this fact without a programmer having to define the concept of address as its own table.

I've also seen a few impressive search engines that work by doing exactly this - compressing the data and using hashes to look it up rapidly. That combined with similar indexing and automatic data normalization through reduction seems interesting to me. It could find patterns that you'd never think of normalizing normally but ultimately make your program more efficient, based on how you're using the data.

You can also use the same technique to apply indexing automatically. It is compression after all, so finding the kinds of data you're looking for is the job of statistical machine learning. Is there anyone out there doing this right now? I'd be interested in trying to use this kind of database over a classical relational database.

What a mess! The next big step to take for 3D graphics in VisualWorks is to get some serious models in to the system, with bones, animations, facial structures and animations, UVW mapping, high polygon-count -> low triangle count with bump mapping, etc. It turns out this is a huge black hole of technology.

Premise 1: Making a 3d modeler is a lot of work, not because of the rendering and interactive capabilities - both of which you'd need for any serious 3d program/game - but because of the transformative properties of such an application. You also have to consider the kind of modeler you want to make. The simplest is polygonal modeling, but this is incredible difficult to use to make organic looking shapes. Next is NURBS which is used by the big guns, Maya and 3ds Max. Splines and patches are the poor mans version of NURBS. Next, Primitives which is a set of geometrics such as balls, cyilnders, cubes that can be used and parameterised as the building blocks of a model. Finally, there is the newer techniques of sculpt modeling where you use things like subdivision surfaces or voxels to build shapes from blobs.

Premise 2: Importing from a proprietary format or open format is less work than building a whole modeler, so let's just do that kthx bai. Well, interesting in theory but it turns out in practice there are a few stumbling blocks here. First off, most of the formats you can use do not include any bone structures or animation. If they do include animation often it is a single key-frame timeline without any markers to indicate start/stop points of an animation, let alone evolution between animation modes.

The 'serious' format of the business right now is still .fbx which is Autodesk's format supported by Maya and 3ds and it is the main import format in to Microsoft's XNA. This is the most 'standard' of the formats because they provide a free SDK written in C++ to import their .fbx files in to memory for you to then render. The format is proprietary and as we well know, integrating with a C++ library from VisualWorks is a real pita. Despite this, fbx includes animation data, bone structures, just about everything you might ever need to render something properly. Yet, Microsoft built pipeline tools to convert the .fbx format in to their own format - the SDK contains a HUGE swath of code to render a scene, manipulate cameras, etc, all of which is redundant and out dated when you build a custom rendering pipeline such as deferred shading and SSAO.

Premise Each format of course is dictated by the kind of modeling technique used - as described above, are you going to get lists of triangles, NURBS, patches or primitives? What do you do with all that once you've got it. One of the advantages of the .fbx SDK is that it comes with an API to tesselate the model in to triangles - I expect this is what Microsoft does in their pipeline. But even here we're making too many assumptions about the end result dataset. What if we want more triangles? What if want to bump map?Too many what ifs.

3: It's what the big guys do, so it's probably what we should do. Cry Engine 3, one of the more famous 3d engines around has ways of importing content just like Microsoft XNA. But, it becomes apparent very quickly that all but the most simplest of data comes from the 3d modeler in to Cry Engine. Terrain is their game, they build roads, trees, oceans, mountains. They also do facial animations, body animations, bone structuring, sound tagging.. all kinds of stuff that cannot be done inside of 3ds Max or Maya. So essentially, even when you're using the biggest baddest 3d engine out there, you're molding the clay, then moving over to their engine to do the heavy lifting.

Conclusion: Everything is one giant mess. Even if you go with 3ds Max or Maya, or a free variant such as Wings 3d or a cheap variant such as Milkshape 3D, you're still only going to get as much data out of it as you can invest in to it - which is to say, usually not very much. If we go back to Premise #1 we posit that it's hard to make a 3d modeler. It's probably not as hard as everything else you have to do after you've made your model. Cry Engine 3 lets you build the world while you're in it; sound familiar? very smalltalk-ish isn't it. So why not have the whole tool chain from go-to-woe all in the one engine? Then, rendering in Maya is never going to look different to how it looks in the final rendering engine with sun-moon cycles, retinal brightness adjustment, material and shader integration, mood lighting, sound integration (yes, sounds do affect the way you see things), etc.

What this has led me to conclude is that the main reason to support 3ds Max or Maya is to import talent - any one who is familiar with these tools can build you 3d models very easily. Unfortunately for you, the models will often be boneless, animation-less, facial animation-less, jointless, muscleless and be stacked with all kinds of custom shader options, unique primitives you don't support, etc.

What's the alternative? Make your own 3d modeler as part of your 3d engine. The downfall here is that you have to train up the talent to use your software - however, your entire content pipeline can be fully integrated between the software and the modelers as well as the world/terrain editor. There is another hidden pitfall too - your assets are less portable if they're only stored in your own proprietary format. I'd bet money that Duke Nukem Forever hit this problem when they were porting between engines.

So my next step now is to take the dive, start making a 3d modeler / world / terrain editor all in one (since they are very similar in nature) which will further allow me to build a geometry rendering / deferred shading renderer with SSAO for high quality rendering, as well as day-night cycle. Two other engines that aren't "big guns" that I take inspiration from are Quel Solaar and Leadworks. If nothing else, it'll educate me more and I should have fun trying.

Now that we're very close to releasing a beta, I thought it was time to do a screencast to give everybody a sneak peek at what's been since WebVelocity 1.0. The answer is: a lot. A whole new editor, a whole new look refresh, a whole new slew of tools, a whole new debugger, a whole new version of Seaside underneath, support for SQLite3 active record, Amazon EC2 deployment capabilities, now runs in Firefox, Chrome and Safari.. and a whole lot more.

If you would like to try out WebVelocity 1.1 beta, email Arden Thomas or James Robertson and they'll get you started. You can download the movie directly from here: http://dl.dropbox.com/u/20291/Cincom%20Smalltalk/WebVelocity-20100603-WV11BetaPreview.mov.