Programming

Today, I was bitten by this problem: Illegal :ASCII character starting at byte position 16. I had some data I wanted to exchange with my Postgres DB but somehow, my Emacs consistently lost his connection to SBCL when I hit data with special chars (German umlauts encoded in ISO Latin 1). Investigating further, it became clear that SBCL had the problem without Slime, too. I was rather surprised then to see the solution in the thread linked to above: It was indeed a slime setting that fixed the problem. On a second look, I don't encounter the exact same problem, as I have trouble with encoded data in the database, not with the initial connect. But the solution suggested by Nikodemus Siivola helps me as well. There have been at last two other threads on the mailing list with regard to Unicode and Postgres, the last one is from April this year. Somebody from the slime camp could probably explain what that slime-setting does to SBCL that it affects SBCL connection to Postgres via CL-SQL. Perhaps I'll go and have a look. Ah, the joys of open source.

Update:I was under a wrong impression. I previously had set SB-IMPL::*default-external-format* to :LATIN-1, a suggestion from the same thread. The slime setting only handles the communication issue with Emacs, but setting the variable on the sbcl side fixes the real problem.

Which Programming Language are You?

Moving back in time is easy when the tools you're using are a versioning control system like CVS or RCS [1], (X)Emacs, it's versioning control interface VC, a Common Lisp system such as SBCL and Slime, the superior lisp interaction mode for Emacs. Yesterday, I was debugging a part of my current hobby project which involves a web interface made with Uncommon Web (UCW). One bit of that web interface that I hadn't paid particular attention to in the last round of modifications had stopped working. But the source that dealt with that particular bit seemed quite innocent to me.

I then looked through the log of modifications I did to the file (which is under version control via RCS) by hitting "C-x v l" (equivalent to "M-x vc-print-log"). Turned out that I did some modifications between versions 1.5 and 1.6 to the part under consideration. Hitting "C-x v ~" (vc-version-other-window) and entering numbers 1.5 (and afterwards 1.6) re-generated the old versions, XEmacs directly visiting these corresponding buffers. This feature alone is worth noting: When extracting the older versions, the current version won't be overwrittten, the older versions will be put into the filesystem with the version number added as a suffix, e.g. foo.lisp.~1.5~.

Now, to see what was happening in between those two versions, I first looked at the differences between the two files. "M-x ediff" and selecting the two buffers, it was easy to spot the part of the code I changed. Next, have a look at the effects caused by the changes: For that I did a "C-c C-k" (slime-compile-and-load-file) on the old versions respectively, so I could then see the old behaviour in real life -- i.e., I could directly use the web app in that older version in my Firefox without any further ado. Of course, this also has to do with the fact that there weren't many interactions between the part of the app in question and other parts of the system, in particular I needn't to check out older versions of other files for the compilation to succeed. Actually, loading that old version certainly broke other functionality of the app, but that wasn't the issue: I only wanted to figure out how the old version worked wrt. to that particular part that wasn't working in my current state, hence a fully working old version wasn't what I was looking for.

Now, it did require a little more close attention to the changes I made to discover the mistake. But I think it's amazing in how few steps it's possible to regenerate an old behaviour. All the while, my current state of affairs remained largely untouched, I didn't need to create any copies or move files around or do a larger recompilation. I think that's a highly effective way of discovering the introduction of a bug.

Update: Kent Pitman, who's giving cll another of his (lately very rare) visits, pointed to one of his older papers (Accelerating Hindsight), which I'm surely have seen before but totally forgotten, which talks about how good Common Lisp is for rapid prototyping. What I've shown above is actually just an example of two points in this paper: dynamic redefinition and editor integration. Go figure.

Footnotes:
[1] Yeah, I know, using rcs or cvs and not bazaar, mercurial or git makes me look like an oldtimer. That said, I do use subversion in my day job, too. I stick with rcs and cvs for smaller (read: one person involved) projects mainly due to one reason: There is no support for the newer kids on the block in the VC shipping with XEmacs. Yes, I know that the VC shipping with FSF Emacs has support for subversion (at least), but unfortunately it's not compatible.

Reading Bill Clementson's blog pointed me to Edi Weitz new STARTER-PACK Lisp package, which tries to overcome one of Common Lisp perceived entry level problems for newcomers, when compared to e.g. Python. Python is known to come with "batteries included" which actually means that you'll have lots of libraries for everyday problems installed out of the box. For Common Lisp, not much has been accomplished in this area, as most people still seem to be debating on comp.lang.lisp about standarization. Edi's starter pack instead does the right thing: It provides a simple selection of common packages, some of which could be called to have achieved de-facto standard level (e.g. his own CL-PPCRE package). Unfortunately, it's for Windows and Lispworks only, which Bill ported to Mac/LW.
Being a Debian user since many years (should be roughly ten years now), I think that it would be fairly simple to achieve something similar (but not at all comparable) to STARTER-PACK. There are already a lot of Common Lisp packages in Debian, what's missing, however, is a CL-TASK package. Such a package could do what all other task-* packages do in Debian: Have dependencies on the most common packages. This would provide at least a starting point for the newbies with CL in Debian. I think, I'm going to discuss this with Peter van Eynde via mail and also on cll.

It has been claimed that Common Lisp would provide better error handling than other languages, where "better" equates equally to "provide superior error handling facilities and to "clear error messages". As a mere application programmer, I strongly believe in that this should carry over to libraries, but unfortunately lately I get the impression that this tradition isn't given enough respect by some of the more younger kids on the block.

As my pet peeve example of late, consider UCW: it's a prime example of how to assemble a lot of functionality by relying on other libraries. Unfortunately, standing on giants shoulders can be become quite a task if the giants start shaking. To me, UCW gives really horrible error messages. Most of the time, you'll get no idea where to look for the error.

As a simple example, when UCW encounters an error while parsing some tal file, you just might get the error message "NIL is not a character", which won't give you any clue what caused the problem. Sometimes, you won't get an error message at all, but simply get no result at all. As an example, I took one tal file from some test project, modified it a little and tested it on my new application. I didn't get any output from the variables referenced. After a lot of time spend debugging the lisp-side of the code, I finally saw the light: I forgot to change the lisp package referred to in the tal-file header, hence UCW tried (unsuccessfully) to access variables that didn't even exist. A warning (in UCWs log) would have been very helpful.

Another problem might be related to the code walking UCW does: sometimes you may get an error about unbound variables while trying to render some component. With UCW this problem occurs more often (and I think this might be due to the code walking), as sometimes a (with-slots (foo) component &body) is not sufficient to avoid the accessor, i.e. (foo component). Well, that kind of error isn't uncommon: it's easy to mistype a variable name or to forget (or -- as in UCWs case -- to not knowing that you should) use the syntax for a variable when instead you should have used a CLOS accessor. So usually, you'll take a quick glance at the error message or the backtrace and then fix the error in the erroneous piece of code. Not so with UCW: usually it's near to impossible to find the damned piece of code because the backtrace won't show you even the name of the component. The name of the variable typically will not be of much help either, if you happen to move state information through several components, (re-)using the same slot names all over the place.

So, to sum up, what I believe is that UCW falls short on is coping with the errors its building blocks may generate. The result is that while the resulting code for some application may be terse and even pretty [1], it may take a lot of time to get it right because of the amount of time wasted debugging unclear error messages. Please, library designers: support your local errors.

Footnotes:
[1] I don't think that the code for an UCW application is pretty, but that's another story.

Finally, I wanted to give the not-so-new-by-now SBCL 1.0 a whirl. Google led me to Peter van Eynde's Dapper packages, which you can install if you add the last of the following lines to your /etc/apt/sources.list:

root@elendil:~# grep cl-dapper /etc/apt/sources.list
deb http://people.debian.org/~pvaneynd/cl-dapper-packages ./

Unfortunately, on Dapper this breaks CL-SQL or more exactly the version 3.53 it is shipping with. SBCL, or more exactly PCL barfs

When attempting to set the slot's value to (OR NULL T) (SETF of SLOT-VALUE),
the slot TYPE is missing from the object
#<CLSQL-SYS::VIEW-CLASS-DIRECT-SLOT-DEFINITION CLSQL-SYS::VIEW-DATABASE>.
[Condition of type SIMPLE-ERROR]

I haven't seen any complaint about this on the CL-SQL mailing list, so I guess I'm the only one using such an old CL-SQL version with the shiny new SBCL. So, currently I have a broken CMUCL on Debian Sarge and a broken CL-SQL on Ubuntu Dapper. Sigh.

Update:Manually switching to CLSQL solved the problem. Just in case you have relied on Debian or Ubuntu to supply you with the right packages, I found replacing them to be dead easy. I just 'apt-get remove'd cl-sql, but left cl-uffi and cl-md5 in place. Then, I untar'ed the source archive of CLSQL to /usr/local/src/clsql/. One link from there to /usr/share/common-lisp/source and a 'cd /usr/share/common-lisp/systems/ ; for i in ../source/*.asd; ln -s $i . ; done' later, SBCL 1.0 loaded CLSQL 3.8 like a charm.

A recent apt-get broke CMUCL again:

Reinstalling for cmucl
Recompiling Common Lisp Controller for cmucl
Installing Common Lisp Controller in CMU CL ...
Error in batch processing:

Error in function UNIX::SIGSEGV-HANDLER: Segmentation Violation at #x10044F88.
FAILED

Hooray.

SBCL, also known as Steel Bank Common Lisp, has reached version 1.0! Congratulations! (via Bill Clementson) Now I just have to download and test it.

Just by coincidence, I came across Bookzilla today, a German online book shop that supports the FSF Europe. It was my first visit to the site at all. Imagine how surprised I was to see this book (also available online from here) on the start paage. Does that mean that all Bookzilla visitors are potentially enlarging the CL community soon?

William Bland points to two new libraries available from cliki.net, both targetting at fuzzy string matching: Levenshtein (the Cliki link currently isn't working) is probably a library implementing a search via the Levenshtein distance, which is a measure of how many edit operations one has to perform on a string to convert it to the other one. Soundex algorithm indexes words by their sound when pronounced in English, for example to account for differences in spelling. At least this library was part of the CMU AI repository before, but not ASDF-install'able.

I had several times the need for these algorithms in the past and only time constraints kept me from enrolling my own versions. Nice to see that now libraries from the CMU archive show up on Cliki, the CMU archive is a nice archive of many AI related libraries (although today one wonders why string matching should be considered as artifical intelligence).

Update: The Cliki link is working again.

Troels Henriksen writes in his blog about library use in the Common Lisp community. It's really funny that he cites UCW as a good example of library usage, whereas I have encountered several problems exactly because of UCWs library dependencies. Another example would be CL-SQL .

Now, in Troels comment section, some unknown poster suggested looking at the integrated XEmacs packaging system. I'm not convinced that this solves the real problem, though, as it doesn't deal with version issues at all besides providing a feature to update to a newer version. But incompatibilites between versions is the major obstacle identified (correctly) by Troels. Revisit UCW: UCW includes (or at least it did when I last checked some months ago) several forked-off versions of some libraries it depends on.

I'm also not convinced that fixing or enhancing the package tools is the right way to go. I think that the major issue is that many CL libraries are far from the stableness that e.g. CL-PPCRE provides. Otherwise, e.g. UCW wouldn't need to provide a box set of it's own versions of libraries, and believe me, I'm very grateful for that box set. I think that the only real solution is working on the libraries.

I also believe there is another reason for many programmers not to rely on external libraries: The fear of bloat. CL traditionally has a very bad reputation with regard to execution speed and memory usage. Now, right after loading UCW in a fresh SBCL, I see the following in top:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11861 schauer 15 0 894m 79m 15m S 0.0 7.8 0:13.22 sbcl

This is a lot of memory which might be a reason why many programmers might feel it's easier to start developing their own application-specific solution.

Another reason might be that Lisp programmers often try to accomplish a solution in terms of the problem domain and this might sometimes conflict with the approaches taken in the libraries. To some degree, this is the Not-Invented-Here issue in cloak (eek, I don't like it this way, I'll go my way instead), but it also touches a lot on issues of style which can't be argued about.

It's only when you start breaking those scripts into smaller, reusable pieces that you start worrying about file dependencies. Rubys 'require' is still several generations away from CLs require and defsystems.

What I like about Ruby is that it's close to Perl in that it's nearly as concise as Perl and that many idioms from Perl are nearly the same in Ruby.

What I dislike about Ruby is that it's close to Perl in that it's nearly as concise as Perl and that many idioms from Perl are nearly the same in Ruby.

I.e., while that eases getting started a lot, it also implies that you will stumble over a lot of nitty gritty details, e.g. forgetting that you'll need to announce a parameter to a block AKA an anonymous subroutine. It also lures one into keeping old habits, e.g. just using a hash to store data instead of creating a new object/class that uses hashes internally or using Ruby as another imperative programming language.

I'm finally fed up with Perl. And I'm more the occasional user than a professional. Probably I should attribute it to my cons-cell based stomach that always nearly made me throw up when I commited an attrocity such like accessing an array as a hash value. As there's more to consider than personal taste when choosing programming languages, I'm afraid, CL won't be the next tool. And no, I won't create a DSL to generate Perl code from Lisp, although the idea has some appeal to it. Instead I will take a closer look at one of the more lisp-like script languages. After some googling around, Ruby seems more attractive to me than Python, in particular as functional programming seems to be closer tied to the community.

But starting with it on my current project already showed some of it problems. To name the most prominent one: lack of good documentation. I miss (documentation 'foo :function) and perldoc -f foo alike, and no, ri foo is nowhere near perldoc. Another problem is the Emacs integration: coming from Slime, ruby-mode and especially the interactive submode for irb reminds me of vacuum cleaners. Perhaps I should take this as a hint to take another look at Eclipse?

UCW is a beast to set up. Don't even think about getting it to run on a naive Debian Sarge installation with the shipped lisp libraries. Fortunately, they do provide a boxset in the meantime, which eases the pain a considerable bit.

But even with the new tutorial,that also ships with the boxset, I had to dig quite a bit around how I make my new test application known to ucw. /etc/ucw/applications.d? No. path-to-ucw-boxset/ucw_dev/etc/conf.lisp? No. path-to-ucw-boxset/ucw_dev/etc/start.lisp? No. Finally, adding to path-to-ucw-boxset/start.lisp did the trick, as should have been obvious to me as I fire up UCW by sbcl --load path-to-ucw-boxset/start.lisp. Oh well. And I haven't even started to move the UCW configuration to use mod_lisp.

And of course, the boxset messes around with your ASDF settings, which it has to, otherwise it would run with the different versions of it's more common libraries. You'll not only need to add your private apps system file to where the boxset expects it (or mess around with the setting of the default-registry in (one of the) conf file) but you'll also need to additionally add (links) for all additional CL libraries (say CL-SQL) your app depends on. That's not funny but a royal PITA as it means you'll do very stupid Unix sysadmin tasks instead of fun web programming with UCW.