A view from the hill

Debian just published an update to their stable distribution etch dubbed etch-and-a-half. The big news here for everyone not accustomed with Debians release cycle anyway is that it's the first time that it's not only a point release that fixes security issues but it's an update to the stable release that gasp adds support for new hardware, too. It even brings important fixes for some applications.

First of all, that's a tremendously good step in the right direction. In the past I've been bitten more than once by the long release cycle of Debian, outlasting all care taking of hardware compatibility. It's the primary reason that the only machine under my direct control running Debian is my rusty private workstation at the moment, all other machines needed newer drivers and hence are installed with Ubuntu. So, if Debian finally realizes that it has to change something to support newer hardware that's more than welcome.

However, it's not really etch-and-a-half. It's not much more than a new kernel which I would compile regularly from kernel.org myself anyway. Admittedly, there are also two new xservers (for nv and intel), but that's still not very helpful for people with newer ati hardware, for instance. But I'm a developer and much more than any hardware hassle I'm much more bitten by really outdated development tools. I've written only one blog entry about the issue, but it's a topic that most developers could write books about -- or maybe not, most of them probably just compile newer versions as needed and are done with the issue. This looks like the easy way out but it doesn't really solve the issue: if you've got to deploy your software you'll need to ensure that what you need is there. But using newer versions of base libraries forces you to deploy them yourself, which also means that you got a problem whenever a security issue will be found in those libraries. On the contrary, with system libraries you don't need any manual deployment of libraries you're only using and you can hope to participate of all system updates. So, in my eyes that's clearly a big advantage for using system libraries. But with Debian (and etch-and-a-half doesn't change this) you're stuck with older libraries which a) might lack long wanted functionality and b) are incompatible with versions from other vendors, which is a pain if you need to deploy on different platforms. Now, granted that the latter problem is not only tied to the age of libraries, but it would be a lot smaller if Debians general release cycle wouldn't be that long. I know of software vendors who have thus decided to develop for Ubuntu but not for Debian, in order to minimize portability issues due to library/functionality mismatches between platforms. To me, that looks like quite high a price to pay.

James Hague blogs about how writing a spellchecker used to be a major feat of software engineering, the main point being that memory was a main problem that's no longer an issue today. While that is certainly true, the two issues (memory, spellchecking problem) are not as much related as James claims.

Without making it explicit, he makes the assumption that for spell checking to work you have to use a dictionary consisting of entire (correctly spelled) words. That would have been a very unwise way to go in the 80ties and it still is nowadays. /usr/share/dict contains roughly 240.000 words and you find that impressive? I don't. The real number of words of a language is much larger, which is easier to see in languages like German where you can easily build compound words. Hence, linguists typically try to capture as much information as they can in word building rules.

This will leave you with three things: a dictionary of words which are not built according to the generally applicable rules (for instance necessary for foreign words prominent in your language of choice), and a set of stems, basic building blocks of words. Quite to the contrary of James' claim that 3-5 lines of Perl are everything you need, the thing to keep in mind here is that this is the only way to solve the "spellchecking problem": any full-form approach to spellchecking is doomed to fail because a language's vocabulary does not consist of a closed set of words. How else could publishers of dictionaries go on selling new versions of their dictionaries every year? Another point is that in some languages (e.g. German) some aspects of spelling might be dependent on context (grammatical and semantical context). I.e. in order to be able to decide whether it's correct to write "Fahren" you need to have a basic understanding of the grammatical composition of the sentence it appears in: it might be a noun derived from "fahren" (to drive) or it might be situated at the beginning of a sentence (and sentence boundary detection is not always trivial either). Now, if your spellchecker just checks a dumb list of words it can't decide whether "Fahren" is correct or wrong.

With regard to the implementation issue, using a non-full form approach also helps: The nice thing is that you don't have as much redundancy and you can spellcheck a lot of words you've never heard of just by following the rules. Unfortunately, the number of stems is still huge if you want to do achieve a good rate, so of course clever tricks to deal with the list of stems is still required. There are of course still different routes you might take to implement spellchecking: what ispell or aspell are doing, for example, is mostly using a full-form approach using only minor modifications to recognize some simple word production rules. This is not what you will find in state-of-the-art spell checkers.