A view from the hill

In case you're interested in continuation-based web development with Common Lisp, it might be of interest to learn that Drew Campsie has voluntered to take over UCW development. Marco Baringer, initiator and maintainer of UncommonWeb (aka UCW) seemed somwhat relieved that finally somebody with more time on his hand steps up to take over.

That Marco hadn't had the time to keep pushing UCW is quite obvious from the mailing list archive. Lately Attila Lendvai seemed to me (an innocent external observer) to had pushed UCW foreward, however, as he Attila announced, he wants to go into a different direction than Drew.

I've been using ucw-dev during the last two years and the current situation really called for a change, in my opinion. UCWs main problem, as I observe it, is lacking documentation and clearly stated goals which way UCW will go. The automatically extracted documentation doesn't contain information how the bits fit together and all tutorials are a) incomplete, b) outdated as far as they are targetting ucw-dev, whereas development has mainly happened in ucw-ajax. However, it was not all clear that ucw-ajax really would become the main line of UCW.

I like UCWs component architecture a lot and will eagerly follow what is going to happen. With two prominent developers such as Marco and Attila more or less leaving the project, it will be interesting to see with how much activity development will happen from now on.

This is just a minor rambling inspired by a recent thread on cll about the ups and downs of using Rails. What I find really overwhelming is the lack of proofs that other (typically Lisp-based) frameworks really have so much benefit over the established more traditional frameworks (e.g. Rails, Zope, plain PHP, ...). All I can see is starter documents, tutorials etc. that show how to program yet another blog or reddit clone in some particular framework. I miss fair comparisons and detailed discussions why and where exactly that one particular way of doing things has benefits. And especially with regard to those reddit clone prototypes: AFAICT, reddit was far beyond implementing some half-baked prototype (in Lisp) when they decided to switch (to Python). Yes, for demo purposes reinventing the wheel with some unround edges might be acceptable, but in reality only rolling wheels will be sold and this is where the challenge is.

I believe that while it's nice that frameworks help with implementing your 500 line application, they really need to show their value with large applications. Where is a discussion of a large on-line shopping system in, say, UCW and CL-SQL, with secure shopping over SSL, login handling, input validation, distribution over multiple servers etc.? Now, I'm not suggesting that UCW, Weblocks etc. can't be used to build such websites, but it would be nice to see a much more in-depth explanation/discussion/comparison of how to do it. That would also help increase the credibility of Lisp or at least, of lispers discussing (other) web frameworks.

Gary King discusses some issues of XSLT. He concludes:

I still feel as if there is less here than meets the eye; that it ought to be possible to express these graph-transformations more elegantly and with less fuss and bother.

To me, the only interesting part of XSLT isn't XSLT itself, it's the embedded XPath where XSLT's power comes from. XPath is a pair of scissors for cutting out the nodes of an XML document instance you're interested in and is about as powerful (and hence horrible to get right) as regular expressions (although it's something completely different, of course). XSLT is just the framework which provides a relatively boring set of instructions and control-flow operations. And it's "clean functional" design only gets in the way every so often. Sure, coming from a Prolog and Lisp background, I know all the tricks to work around the hassle, but quite often it would be way easier to just set that damned variable *again*. If I want to do clean functional programming, I know where I can find a language without that XML syntax, thanks. Give me the unclean intermix of programming styles offered by Common Lisp anyday, please.

Being a Lisp guy just like Gary, I would be happy to use a language with parenthesis, as I have to do XML transformations quite often and the intermix of elements from different namespaces (XSLT namespace and target namespace) is mostly a pain to read and write. Using xslide, an Emacs mode for XSLT, eases the pain a considerable bit, but it still isn't really fun. And sometimes I need to hack XSLT in a proprietary system which only provides editor capabilities that know nothing about XML (think notepad-style editing capabilities) and then XSLT suddenly has the next most horrible syntax imaginable. However, the corporate world has its own rules and to be fair, the major benefit of XSLT is that you have one standard syntax for XML transformations rather than having xyz numbers of incompatible library APIs.

Update: Also in response to Gary, David Lichteblau pointed out his approach to (macro-)pre-processing XSLT stylesheets to overcome some of Gary's problems.

Moving back in time is easy when the tools you're using are a versioning control system like CVS or RCS [1], (X)Emacs, it's versioning control interface VC, a Common Lisp system such as SBCL and Slime, the superior lisp interaction mode for Emacs. Yesterday, I was debugging a part of my current hobby project which involves a web interface made with Uncommon Web (UCW). One bit of that web interface that I hadn't paid particular attention to in the last round of modifications had stopped working. But the source that dealt with that particular bit seemed quite innocent to me.

I then looked through the log of modifications I did to the file (which is under version control via RCS) by hitting "C-x v l" (equivalent to "M-x vc-print-log"). Turned out that I did some modifications between versions 1.5 and 1.6 to the part under consideration. Hitting "C-x v ~" (vc-version-other-window) and entering numbers 1.5 (and afterwards 1.6) re-generated the old versions, XEmacs directly visiting these corresponding buffers. This feature alone is worth noting: When extracting the older versions, the current version won't be overwrittten, the older versions will be put into the filesystem with the version number added as a suffix, e.g. foo.lisp.~1.5~.

Now, to see what was happening in between those two versions, I first looked at the differences between the two files. "M-x ediff" and selecting the two buffers, it was easy to spot the part of the code I changed. Next, have a look at the effects caused by the changes: For that I did a "C-c C-k" (slime-compile-and-load-file) on the old versions respectively, so I could then see the old behaviour in real life -- i.e., I could directly use the web app in that older version in my Firefox without any further ado. Of course, this also has to do with the fact that there weren't many interactions between the part of the app in question and other parts of the system, in particular I needn't to check out older versions of other files for the compilation to succeed. Actually, loading that old version certainly broke other functionality of the app, but that wasn't the issue: I only wanted to figure out how the old version worked wrt. to that particular part that wasn't working in my current state, hence a fully working old version wasn't what I was looking for.

Now, it did require a little more close attention to the changes I made to discover the mistake. But I think it's amazing in how few steps it's possible to regenerate an old behaviour. All the while, my current state of affairs remained largely untouched, I didn't need to create any copies or move files around or do a larger recompilation. I think that's a highly effective way of discovering the introduction of a bug.

Update: Kent Pitman, who's giving cll another of his (lately very rare) visits, pointed to one of his older papers (Accelerating Hindsight), which I'm surely have seen before but totally forgotten, which talks about how good Common Lisp is for rapid prototyping. What I've shown above is actually just an example of two points in this paper: dynamic redefinition and editor integration. Go figure.

Footnotes:
[1] Yeah, I know, using rcs or cvs and not bazaar, mercurial or git makes me look like an oldtimer. That said, I do use subversion in my day job, too. I stick with rcs and cvs for smaller (read: one person involved) projects mainly due to one reason: There is no support for the newer kids on the block in the VC shipping with XEmacs. Yes, I know that the VC shipping with FSF Emacs has support for subversion (at least), but unfortunately it's not compatible.

It has been claimed that Common Lisp would provide better error handling than other languages, where "better" equates equally to "provide superior error handling facilities and to "clear error messages". As a mere application programmer, I strongly believe in that this should carry over to libraries, but unfortunately lately I get the impression that this tradition isn't given enough respect by some of the more younger kids on the block.

As my pet peeve example of late, consider UCW: it's a prime example of how to assemble a lot of functionality by relying on other libraries. Unfortunately, standing on giants shoulders can be become quite a task if the giants start shaking. To me, UCW gives really horrible error messages. Most of the time, you'll get no idea where to look for the error.

As a simple example, when UCW encounters an error while parsing some tal file, you just might get the error message "NIL is not a character", which won't give you any clue what caused the problem. Sometimes, you won't get an error message at all, but simply get no result at all. As an example, I took one tal file from some test project, modified it a little and tested it on my new application. I didn't get any output from the variables referenced. After a lot of time spend debugging the lisp-side of the code, I finally saw the light: I forgot to change the lisp package referred to in the tal-file header, hence UCW tried (unsuccessfully) to access variables that didn't even exist. A warning (in UCWs log) would have been very helpful.

Another problem might be related to the code walking UCW does: sometimes you may get an error about unbound variables while trying to render some component. With UCW this problem occurs more often (and I think this might be due to the code walking), as sometimes a (with-slots (foo) component &body) is not sufficient to avoid the accessor, i.e. (foo component). Well, that kind of error isn't uncommon: it's easy to mistype a variable name or to forget (or -- as in UCWs case -- to not knowing that you should) use the syntax for a variable when instead you should have used a CLOS accessor. So usually, you'll take a quick glance at the error message or the backtrace and then fix the error in the erroneous piece of code. Not so with UCW: usually it's near to impossible to find the damned piece of code because the backtrace won't show you even the name of the component. The name of the variable typically will not be of much help either, if you happen to move state information through several components, (re-)using the same slot names all over the place.

So, to sum up, what I believe is that UCW falls short on is coping with the errors its building blocks may generate. The result is that while the resulting code for some application may be terse and even pretty [1], it may take a lot of time to get it right because of the amount of time wasted debugging unclear error messages. Please, library designers: support your local errors.

Footnotes:
[1] I don't think that the code for an UCW application is pretty, but that's another story.

UCW is a beast to set up. Don't even think about getting it to run on a naive Debian Sarge installation with the shipped lisp libraries. Fortunately, they do provide a boxset in the meantime, which eases the pain a considerable bit.

But even with the new tutorial,that also ships with the boxset, I had to dig quite a bit around how I make my new test application known to ucw. /etc/ucw/applications.d? No. path-to-ucw-boxset/ucw_dev/etc/conf.lisp? No. path-to-ucw-boxset/ucw_dev/etc/start.lisp? No. Finally, adding to path-to-ucw-boxset/start.lisp did the trick, as should have been obvious to me as I fire up UCW by sbcl --load path-to-ucw-boxset/start.lisp. Oh well. And I haven't even started to move the UCW configuration to use mod_lisp.

And of course, the boxset messes around with your ASDF settings, which it has to, otherwise it would run with the different versions of it's more common libraries. You'll not only need to add your private apps system file to where the boxset expects it (or mess around with the setting of the default-registry in (one of the) conf file) but you'll also need to additionally add (links) for all additional CL libraries (say CL-SQL) your app depends on. That's not funny but a royal PITA as it means you'll do very stupid Unix sysadmin tasks instead of fun web programming with UCW.

Getting started with CL-SQL was actually a pretty easy experience. I stumbled over one problem though, and it has a name, even. The packages shipping with Sarge are outdated. So outdated, that I couldn't get my files compiled that would otherwise work fine from the REPL or even load successfully as source files. And, hey, most of the time they would even compile with ASDF, unless I was using load-system from Slime.

Actually, that wasn't the first time I had trouble with Common Lisp packages on Debian. Getting UCW to work actually required a far newer version of sbcl than that old 0.8.12 shipping with Sarge. I built one myself, however, thanks to the work of Rene van Bevern (why isn't he on Planet Lisp?), a recent version (0.9.11) is available from backports.org, so that shouldn't be a problem anymore. New versions of CL-SQL are not available, though, so I copied the newer version of the files shipping with Breezy, which finally fixed the problem.