Line Ups, Reported by Reinout Heeck
Last week, there was an interesting blitz of comments in one of Rich Demers blogs. Reinout Heeck had a lot of interesting things to point out; one that intrigued me, was his company's adoption and use of "...lineups as implemented by Cees de Groot..." I had seen these a while back, but not kept up to date. I emailed Reinout and asked him to write more about these things. He not having a blog right now, I told him I'd host his comments for him and he kindly obliged. Thank you Reinout for the very informative report. I'll be taking a closer look at Lineups real soon now. My only remaining question is... how is "Heeck" pronounced?
And now for the very good report...
In our shop we have found Store's bundles to be troublesome for a variety of reasons. Here are the three important ones:- Bundles require that their content be defined only once, IOW package overrides coexisting with an earlier definition in one bundle is not supported by Store. This means that bundles cannot be used to version the state of (library code)+(patches)+(our code).
- Bundles turn into a versioning bottleneck when the team is larger than a couple of people (specifically in a "everybody owns the code" process).
- The tree presentation of Pundles in the tools lures people to use them as classifiers rather than for code layer management. The Pollock project for instance uses the Pundle hierarchy to classify agents vs artists instead of modeling the code layering.
Editors Comment. I couldn't agree more with these. Well and succinctly put.
We pretty much abandoned Bundles and started to use the prerequisite chain of Packages to manage our code configurations. In various discussions this has been referred to as "lineups" but in the public Store repository the code is called TIO-Store Extensions courtesy Cees de Groot.
When you load this (tiny) package you get an extra item Publish Configuration in the pundle hierarchy menu. Assuming you have a package 'MyPackage' selected this will recursively walk through the prerequisites of 'MyPackage' and record which versions are loaded in the image. If any of those prerequisites are unpublished it will open a publish dialog on them and abort, else it will continue by updating a package named 'MyPackage Configuration'. This configuration package only contains a prerequisite list specifying all recorded prerequisites with their version.
Early in use we found it hard to discern which packages play the role of configuration generator so we introduced a naming convention for these by having their name (confusingly) end in 'Config'. We leave these packages empty, they only specify top level prerequisites without version. We also souped up the RB to recognize both these conventions and show specialized icons.
Not much later it turned out that what we wanted to version was what our staging and build bots were doing, so in practice humans don't use the new menu item much. Instead the bots turn out to be doing the publishing of configurations. Humans are forced to keep the dependencies straight (which isn't a bad thing.
Moving to configurations has cracked the three problems I mentioned above:
- A configuration can "contain" multiple package overrides
- The versioning bottleneck is gone since there is no hierarchy of bundles that needs merging into.
- We did a "step back" in the browser in that the package list is now so overpopulated that we cannot navigate it with mouse gestures. The usage pattern has turned into that of a searchable keyword database (by use of the "find package" dialog). This shift of usage has forced package naming sanity onto us, again not a bad thing.
Since configurations are just Packages they can be used as prerequisites too, IOW we got configuration composition (with a well-understood load ordering) for free as well.
If you step back you may notice that there is almost zero code involved. Most of the shift revolves around conventions that form a framework with very simple semantics once bundles are no longer involved.
less Stores more!
Comments
Prereqs
[Avi Bryant] June 22, 2004 21:08:09.324
Neat. That's precisely the solution I came up with for Monticello as well. But if you'll forgive my ignorance of Store: how do prereqs between packages work? Does a package have another package as a prereq, or a version another version, or a version another package, or... ? So, for example: package A depends on B. I save version 1 of A, when I have version 1 of B loaded. Later I save a version 2 of B. If someone else tries to load version 1 of A, which B do they get?
In Monticello dependencies are currently very strict (version->version, so you would get B version 1 in the above case). This is good in some cases but really annoying in others, so I'm curious how "lineups" deal with it.
How Prereqs Work in Store
[Travis Griggs] June 23, 2004 0:34:59.104
Avi,
Store has two prerequisite streams: "development" and "deployment." This is stupid, but it is what it is; I blogged about this a week or so ago.
The first, the "deployment" one--is used with parcels--models each prereq as a two element entity. The first element is the name of the prerequisite, the second is the version string. The version string can be empty, which means any version can be used. I'd love to see a simple sort of algebra ala Debian style prereqs (e.g. >= 1.2, etc.).
The second, the "development" one--is used with buckages/pundles--extends said model by adding a 3rd entity, a symbol which specifies, #package, #parcel, #bundle, or #any. This is used to help resolve ambiguities in component names (e.g. you have a bundle named "BuYaz" and a parcel named "BuYaz", both in your current search path.
"Heeck" pronounced like Hake as cake
[Terry] June 23, 2004 6:52:50.645
Re: Line Ups, Reported by Reinout Heeck
[ Reinout Heeck] June 23, 2004 10:02:47.141
Comment on Line Ups, Reported by Reinout Heeck by Reinout Heeck
Today I found this little nugget on our wiki as well:
To Publish all Work
In the package browser, select the top-level configuration package and do Package->Publish LineUp. Store publish dialogs will open for all prerequisite dirty pundles. Normally, you don't take the second step here of publishing the LineUp itself, you just use the menu entry as a shortcut to publish a whole bunch of pundles in one go.
BTW, terry's comment above is legible on the web view of the blog, not in the rss feed :-/
Re: Line Ups, Reported by Reinout Heeck
[Cees de Groot] June 25, 2004 3:16:13.841
Comment on Line Ups, Reported by Reinout Heeck by Cees de Groot
Note that the current lineup stuff is a tad old - Monday I'll polish it and re-publish it.
Another thing I wrote yesterday is a global merge thingy. When loaded, you can ask the merge tool to merge in all branches that have a certain pattern in their version naming scheme. The idea if that if you start work on item123, when publishing you make sure that 'item123' appears in the version name. E.g. when publishing version 103, you publish as 103.item123.1, etcetera. Now, when you are ready to merge your stuff into the main line of development, you load that, say 'merge item123', and you'll get a MergeTool containing all the deltas across packages. Something, incidentally, CVS has had only for the last 15 years or so... Another menu item will open a package version browser.
OBTW - whatever I publish regarding this stuff will be usable only by shops who, like Soops, abandonded bundles. IMNSHO, the concept is so broken (and so nicely subsumed by the Config/Lineup business Reinout explained and the global branching/merging described here) that I decided not to care about supporting it.
Re: Line Ups, Reported by Reinout Heeck
[Cees de Groot] June 25, 2004 3:28:56.340
Comment on Line Ups, Reported by Reinout Heeck by Cees de Groot
Something Reinout didn't report (or not clearly enough) is the Config/Lineup dichotomy we created - it relates to Travis' explanation of prereq handling in Store (btw: we have at Soops dev prereqs == deployment prereqs in order not to get confused, we even have enhanced the environment to warn when this is not the case).
Basically, a Config is a configuration, a Lineup is a snapshot of that configuration. The human creates 'My Stuff Config' (with prerequisites 'Foo' and 'Bar' - version-less so it'll take the most recent version when you load it or whatever you have specified as your prerequisite policy), and then when a particular version of that configuration needs to be kept around for eternity (like when you send stuff off-site), you do 'publish lineup' and the system creates and/or updates 'My Stuff Config Lineup', which will have 'Foo 1.2' and 'Bar 1.3' as prerequisites; it is then published so you will have 'My Stuff Config Lineup 1.0' which is what you can write on the CD label.
For main-line development, you load 'My Stuff Config' (in Soops case, we have a single top-level Config which pulls in other Configs which loads maybe yet more Configs and real packages); when you have to patch, you simply load 'My Stuff Config Lineup 1.0' and you will get exactly the state of the image the customer has (module some base bundles you probably don't want to have in your prerequisite list).
Re: Line Ups, Reported by Reinout Heeck
[ Reinout Heeck] June 25, 2004 11:20:09.688
Comment on Line Ups, Reported by Reinout Heeck by Reinout Heeck
Reinout didn't report (or not clearly enough)
This is due to differences between the current version we use and the published version I referred to above. To avoid confusion I advise to wait for Cees' to-be-released version and interpret his words as being most authoritative.
Aha!
[Avi Bryant] June 25, 2004 17:29:18.293
Cees, thanks for the clarification on configs. So Monticello is doing the right thing by storing exact prereqs when it takes a snapshot. What's missing is the idea of loading a config rather than a specific version/lineup. In Monticello's case, a config probably ought to point to a specific repository: you don't really want to say "load the most recent version you know about anywhere", but rather "load the most recent version from http://cdegroot.com/tio/stable". Which may in turn include configs that point to other repositories for other packages...
Re: Line Ups, Reported by Reinout Heeck
[cdegroot] June 28, 2004 6:59:14.070
Comment on Line Ups, Reported by Reinout Heeck by cdegroot
Ok, I've published a package that I think should work well outside Soops. See http://swiki.cdegroot.com/lineups for details
[Eric] July 1, 2004 20:32:46.239
I posted a quandry about Lineups on my blog here.
Any answers would be appreciated.