Inspired by Jon Udell's post, I grabbed the mean temperature data he mentioned. That's when the fun began :)

The problem with the data is the same problem you get with any dataset - it's never completely clean, and you always have to do something to sponge it off. So it was here. There were a number of small issues I ran across as I created some Smalltalk scripts to make sense of the data:

• The usage of -9999 to indicate bad/no data for a given month. That wasn't really a data problem, so much as an undocumented convention
• It took me a bit to find a weather station near me that had a large data set - I eventually settled on the DCA (near Washington National Airport) one
• The data was normally separated by whitespace (within a record), and CR between. However, some of the uses of -9999 didn't have surrounding whitespace.
• Some data seemed to be duplicated (duplicated years for the same station), but with slightly different values for some months. What did that mean? Not a clue :)

None of those were insurmountable, but they did make a "quick" look at the data harder. First came the scrubbing, then the "quick" look.

The thing is, that's not really a problem for someone with some software skills, but it will throw anyone without them. Even an Excel import would have foundered on the data that didn't have whitespace, for instance. So the sad thing is, it's even harder to deal with this stuff than Jon let on :/

Oh, and what did I discover? My wife's memory was right: summer's were slightly hotter back in the 80's (at least around here. YMMV)

Update: One man's broken data is another man's misunderstood format. Turns out that the records are fixed width, not white space delimited. Shows what I know :)

Technorati Tags: public data