I've noticed that BottomFeeder could, with enough feeds, start taking up large amounts of RAM. Running in a development environment showed that I wasn't generating any leaks, so it was either MemoryPolicy, ObjectMemory settings, or both. I've been twiddling with the MemoryPolicy settings for a couple of weeks now (with some good help from Terry Raymond) - but that wasn't solving the whole problem.

Then it occurred to me what was probably happening - premature tenuring. VisualWorks allocates new objects in Eden (and uses the two survivor spaces to manage new objects). When survivor space fills up, and new objects aren't ready to die yet - they survivors have to go somewhere to make room for the new stuff. That somewhere is Old Space. So what if you allocate a lot of new objects quickly, and they keep filling up the survivor spaces? You'll get rapid tenuring of objects that would otherwise die. These end up in Old space, where they won't die until you your system runs against a memory limit (either the one imposed by MemoryPolicy or a hard system limit). At that point, you'll see the compaction and GC icons a lot if the allocation pattern continues, because those new and survivor spaces still aren't big enough - and growth is capped. I've talked about this issue before; this is kind of a continuation of that earlier post.

So what do you do? You go look at ObjectMemory class - #spacesAtStartup:. I had to create a new base for this because changing the settings here require an image save, and BottomFeeder (and most end user Smalltalk apps) never save the image at runtime. What the settings allow you to do is scale the various space sizes up. Now, why is this a problem in an application like BottomFeeder?

Consider what BottomFeeder does - every N minutes, it wakes up and trawls through your subscriptions, looking for new content. That means do an HTTP query (grab text). Parse the results into an XML document (a largish new object). Then create item objects from the xml, and check against what we have (more new objects). I am currently subscribed to 140 channels, which makes for a lot of new objects - fairly quickly - during an update scan. So with eden and the survivor spaces set to their default sizes, lots of these objects end up in old space, which is not where they should be - none of them are really "permanent" from an application perspective.

I ended up sending this message as part of the new build:

ObjectMemory sizesAtStartup: #(10.0 10.0 3.0 1.0 1.0 2.0 1.0).

The important numbers are the first two - they ask the system to make eden and the survivor spaces 10x bigger than they normally would be - and with that increased size, there's less chance of premature tenuring. This is an important thing to keep in mind for all kinds of applications - what looks like an application level memory leak may well have a system level root. Now, it's always a good idea to look at what's going on before you start mindlessly twiddling these numbers - if you have a dependency issue, where lots of objects are ending up in the DependentsFields shared variable, for instance, you have an entirely different problem. So keep this technique in mind, but look at the behavior of your application before applying it.