A view from the hill

While I'm usually a happy VC user (the SCM mode of Emacs) as the next guy, I recently haven't always been so happy. In particular when I'm working on a larger set of changes in multiple files (using Subversion or Mercurial), the commit-per-file approach of VC just doesn't cut it, because usually you want to commit all changes you have to make for a fix or feature to happen in a single transaction. That's simply not possible with VC, so I usually fall back to the command line. That I can then use Emacs for the commit message is only a slight quantum of solace.
An integration of SCM features into dired (which I happen not to be a big fan of, btw.) looks to me like the obvious way to go. I'm hence very happy to learn that this is more or less happening (or has already happened) with the past-Emacs 22.1 version, according to this blog entry on manipulating changeset-based VCS from within Emacs. It's unlikely that this will be up to match Tortoise or TortoiseHg, but still ...

Steve Yegge blogs about XEmacs needs to die and I would like to add my own little comment on the issue. First, perhaps some background: I'm a long-term XEmacs user, since about 1995, I think. I started hacking Elisp around 1996, implementing a major mode for the Otter theorem prover and various other stuff. Since 1997 or so, I had been somewhat active on the xemacs-beta mailing list, reporting build successes/failures, participating in discussions etc. and even writing an article on the then-new XEmacs port to Windows in the german iX magazine. So, perhaps, I'm a bit biased towards XEmacs. That I've been using XEmacs instead of Emacs had a lot to do with two factors: at my university, the local Emacs guru used XEmacs and provided a very complete configuration to start with. To start hacking Prolog source didn't require any configuration on my side at all. When I tried to re-establish that configuration on my own linux box, using Gnu Emacs was totally out of the question, which is basically the second reason: XEmacs came with a lot of batteries included, whereas Gnu Emacs only came with a directly dying low battery at best. And third, the XEmacs interface experience was a lot friendlier than with the naked Gnu Emacs.

Now, at the time I was actively following XEmacs development, XEmacs had a whole lot of man power behind it, with lively discussions and an exciting movement of adding new features. At the same time, RMS discussed switching Gnu Emacs to Guile (a scheme dialect), which me, as a Common Lisp user, scared me off even more. Around 2002 or so, my spare time for XEmacs dropped to zero, so I unsubscribed the development mailing list and was just a happy user. Until two seemingly unrelated things happened: first of all, Unicode was no longer to be ignored and it was obvious that the Mule (multiple languages for emacs or some such) for XEmacs wasn't up to the task, especially not on Windows. That was quite a problem for me back then with the then current stable XEmacs 21.4. Now, six years later, XEmacs 21.5 is still not there (aka released as the new stable version) and it's unicode support still sucks (not so much as it did, say, two years ago, but it still fails a lot of tests from Markus Kuhn unicode test suite). The second development was that the Emacs team, which I had only barely been aware of previously, somehow gained a enourmous momentum and catched up a lot of ground, where XEmacs had been the leader by far. In particular, as of Emacs 22, nobody in its sane mind could argue against the fact that the current Emacs handles Unicode etc. far better than XEmacs (this is especially true for XEmacs 21.4 on Windows, but even the current beta XEmacs 21.5-b28 on Unix isn't where it should be). This seems to be directly related to the fact that a lot of the key developers of the end of the last century no longer actively participate in XEmacs development. I think only few people from the XEmacs crowd would argue against the impression that XEmacs development has more or less cringed to a halt and the distribution of development power between Emacs and XEmacs development is nowadays reversed to the situation from the 90s.

That being said, I still stick with XEmacs. Last time I tried using Gnu Emacs (that's the Emacs 22 from Ubuntu Hardy), I still found that I'm far more accustomed with the XEmacs intrinsics on how to perform specific actions than with how it works in Emacs. And I never got around to clean up my mess of configuration files to properly support the various flavours plus adding the extension libraries that still don't ship with Emacs (Slime, for instance). Finally, with more and more users switching to Emacs, there has to be someone reporting problems,

Now, back to Steves blog post: I think he has some valid points in it, but for the most part I disagree. First of all, while I agree that Eclipse and similar IDEs are still not at the greatest enemy to Emacs (regardless of flavour). I'm currently again using Eclipse at work and it's a pain finding out how to properly associate an unknown file extension with some specific editor (mode). And it's a memory hog that results in unbelievable handling (on my dual core 2GB equipped desktop) that is even worse than my first XEmacs experience on my 486/33 with 8MB ram in 1995. Eight megs and constantly swapping? Hah! I can't believe that even today I have to hear that Emacsen would be too complicated, given that for any task/configurations I spend endless time searching/clicking through the gazillions of settings in Eclipse.

However, I also don't believe that a marriage between Emacs and some web-browser is the way to go. There are much more pressing issues to move Emacs into the next century than switching from Elisp to <whatever>. If you're interested in extending your editor, you have to learn about the way how to do that. But that is totally independent from the question which editor you're going to use in the first place.
What is a much more pressing need in my opinion is finally implementing multi-tasking for the (X)Emacs core. Still being locked in a single-threaded application model is sooo 1990s. It's not becoming more and more ridicuosly, it's utterly unbelievable.

In one other point, though, Steve is right on track: the divergence between APIs is becoming a bigger problem with every passing day. I think it's very unfortunate that a) Emacs developers don't let them inspire by the existing APIs in XEmacs and vice versa and b) that there are too few XEmacs developers that merge current Emacs enhancement into the XEmacs codebase (the other way 'round is always problematic due to license issues).

So, before I would consider pursuing a major architectural change like XUL or some such for Emacs, I would first work on the much lower-hanging fruit of narrowing the gap between the two flavours. It's a pity I myself only have the spare time to write such blog posts instead of actively working on code towards that goal.

Every now and then, I'm still blown away by the elegant power that the repl (read-eval-print-loop) of Common Lisp provides for everyday tasks. A recent example: I needed to fix a broken column definition in one of my Postgres databases. I'm not a DB pro, so I will just drop the column. But of course, we want to retain the old data, so here we go (using CL-SQL):

SOMEDB> (connect +db-spec+)
#
SOMEDB> (defvar *olddata* (select [id] [data] :from [sometable]))
*OLDDATA*
SOMEDB> (execute-command "ALTER TABLE sometable DROP COLUMN data;")
NIL
SOMEDB> (execute-command "ALTER TABLE sometable ADD COLUMN data varchar;")
NIL
SOMEDB> (loop for (id value) in *olddata*
                  do (update-records [sometable] 
                                 :attributes '(data) 
                                 :values (list value) 
                                 :where [= [id] id]))
NIL
SOMEDB> (disconnect)

That's it. First we open a connection, store away the olddata (which will be returned as a list of tuples) in a global variable, modify the table and finally restore the data. Now, what I find really nice about this is that I can operate on the data as if I had an intersection of psql, shell and a real programming language, which is pretty much the point of this blog post. I think that Ruby probably provides a similar environment with irb. And, hey, today I learned that XEmacs comes with an interface to postgres, so I might have been able to do it from within my favourite editor, too ...

Dear Lazyweb, I'm looking for a way to automate unit testing with the help of ASDF. I'm using XLUNIT at the moment, but this isn't really relevant. What I want to achieve is that on every compilation of some source file, it's corresponding test file will be loaded (and hence the tests it contains run). However, what I seek to avoid is simply adding the test files to the component definition; the test files should be kept separately. From what I gather from the ASDF documentation, this should be possible using :perform forms, but the very same docs leave me wondering how. What I found (looking at Jörg Höhles asdf file for iterate) is how to load and run a complete test-system, but this is not what I want to do. Ideally, I would like to have a single perform instruction looking something like this:

(defmethod asdf:perform ((op test-op) c)
  (when (asdf:glorified-find-component "my-component" c)
        (let ((component-test (find-component-test c)))
            (when component-test
                   (asdf:operate 'asdf:load-op 'component-test)))))

Here #'glorified-find-component needs to recursively follow the components parent and #'find-component-test should return a component c-test (with c expanded, obviously). Now, I guess I'm just clumsily reinventing the wheel and hence I'm wondering if somebody has already solved "the problem".

While I'm at it (it referring to testing), Stefil looks like a very interesting test environment in case you're developing your programs with Emacs and Slime. From there, I found a link to Phil Gregory's test framework comparison which I found much more enlightening than the ALUs list of test frameworks.

For a review, I needed to get the track list of a given CD. As the track list wasn't available via CDDB, I went to some large online store and found the tracklist. I need to convert it to XML, though. The original data I fetched looks like so:

1. Fox In A Box
2. Loaded Heart
3. All Grown Up
4. Pleasure Unit
...

whereas I need:

<li id="1">Fox In A Box</li>
<li id="2">Loaded Heart</li>
<li id="3">All Grown Up</li>
...

After cutting the original data to my Emacs, writing out a simple file and using Perl for that simple transformation seemed just gross. In the past, I've been an Emacs hacker. But no more, or so it seems, since it took me nearly half an hour just to come up with this simple function:

(defun tracklist-to-li (point mark)
  "Generate a string with <li>-elements containing tracks.
Assumes that one every line of region, a track position 
and the track name is given."
  (interactive "r")
  (save-excursion 
    (goto-char point)
    (let ((current-pos (point))
	  (result ""))
      (while (re-search-forward "^\\([0123456789]+?\\)\.[ \t]+\\(.*\\)$" 
				mark t)
	(setq result
	      (concat result
		      "<li id=\""
		      (match-string 1)
		      "\">"
		      (match-string 2)
		      "</li>\n"))
	(setq current-pos (point)))
      (message result))))

What took the most time was that I've had forgotten to escape the grouping parenthesis in the regular expression and that it took me a little while to accept that there is really no \d or equivalent character class in Emacs regexps. Which probably means that I've been doing too much in Perl, sed and the like. OTOH, it just may hint at the horror of regular expressions handling in Emacs. What I also dislike is that whenever you want some result in Emacs and see it, too, you have to invoke an interactive operation like message. Of course, there is IELM, but this doesn't really help you for interactive functions operating on regions.

And five minutes later, I realize I need to convert some string like "The (International) Noise Conspiracy|The
Hi-Fives|Elastica" into a similar list structure. With a simple cut & paste and roughly 30 seconds later, I have

[bauhaus->~]perl -e '$a="The (International) Noise Conspiracy|The Hi-Fives|Elastica"; @a=split("\|",$a); 
foreach $b  (sort @a) { print "<li>$b</li>\n"; }'

Hmm. Perhaps I've come quite a long way on the dark side already ...

On the other hand, in Ruby, this is just as simple (I'm using irb, the interactive ruby shell here):

irb(main):008:0> a="The (International) Noise Conspiracy|The Hi-Fives|Elastica"
=> "The (International) Noise Conspiracy|The Hi-Fives|Elastica"
irb(main):009:0> a.split("|").each {|string|
irb(main):010:1* print "<li>"
irb(main):011:1> print string
irb(main):012:1> print "</li>\n"
irb(main):013:1> }
<li>The (International) Noise Conspiracy</li>
<li>The Hi-Fives</li>
<li>Elastica</li>
=> ["The (International) Noise Conspiracy", "The Hi-Fives", "Elastica"]

The difference here is the implicit array Ruby generates, which of course in Perl you could hide in the array position of the foreach loop. Note the annyoing misfeature of irb to always show the prompt even when your still continuing your current input line.

In Common Lisp we can do it just as short:

CL-USER> (let* ((a "The (International) Noise Conspiracy|The Hi-Fives|Elastica")
		          (splits (ppcre:split "\\|" a)))
	           (loop
      	              for string in splits
	              do 
	                  (format t "<li>~S</li>~%" string)))
<li>"The (International) Noise Conspiracy"</li>
<li>"The Hi-Fives"</li>
<li>"Elastica"</li>
NIL

The same thing here: The result of the split could have been easily embedded in the loop.

The lesson, of course, is that in the end this example only serves to show that things that are easy to achieve in a high-level are indeed easy to achieve. Or to put it otherwise that the use of regular expressions is no more a discriminating feature between programming languages.

Gary King discusses some issues of XSLT. He concludes:

I still feel as if there is less here than meets the eye; that it ought to be possible to express these graph-transformations more elegantly and with less fuss and bother.

To me, the only interesting part of XSLT isn't XSLT itself, it's the embedded XPath where XSLT's power comes from. XPath is a pair of scissors for cutting out the nodes of an XML document instance you're interested in and is about as powerful (and hence horrible to get right) as regular expressions (although it's something completely different, of course). XSLT is just the framework which provides a relatively boring set of instructions and control-flow operations. And it's "clean functional" design only gets in the way every so often. Sure, coming from a Prolog and Lisp background, I know all the tricks to work around the hassle, but quite often it would be way easier to just set that damned variable *again*. If I want to do clean functional programming, I know where I can find a language without that XML syntax, thanks. Give me the unclean intermix of programming styles offered by Common Lisp anyday, please.

Being a Lisp guy just like Gary, I would be happy to use a language with parenthesis, as I have to do XML transformations quite often and the intermix of elements from different namespaces (XSLT namespace and target namespace) is mostly a pain to read and write. Using xslide, an Emacs mode for XSLT, eases the pain a considerable bit, but it still isn't really fun. And sometimes I need to hack XSLT in a proprietary system which only provides editor capabilities that know nothing about XML (think notepad-style editing capabilities) and then XSLT suddenly has the next most horrible syntax imaginable. However, the corporate world has its own rules and to be fair, the major benefit of XSLT is that you have one standard syntax for XML transformations rather than having xyz numbers of incompatible library APIs.

Update: Also in response to Gary, David Lichteblau pointed out his approach to (macro-)pre-processing XSLT stylesheets to overcome some of Gary's problems.

Today, I was bitten by this problem: Illegal :ASCII character starting at byte position 16. I had some data I wanted to exchange with my Postgres DB but somehow, my Emacs consistently lost his connection to SBCL when I hit data with special chars (German umlauts encoded in ISO Latin 1). Investigating further, it became clear that SBCL had the problem without Slime, too. I was rather surprised then to see the solution in the thread linked to above: It was indeed a slime setting that fixed the problem. On a second look, I don't encounter the exact same problem, as I have trouble with encoded data in the database, not with the initial connect. But the solution suggested by Nikodemus Siivola helps me as well. There have been at last two other threads on the mailing list with regard to Unicode and Postgres, the last one is from April this year. Somebody from the slime camp could probably explain what that slime-setting does to SBCL that it affects SBCL connection to Postgres via CL-SQL. Perhaps I'll go and have a look. Ah, the joys of open source.

Update:I was under a wrong impression. I previously had set SB-IMPL::*default-external-format* to :LATIN-1, a suggestion from the same thread. The slime setting only handles the communication issue with Emacs, but setting the variable on the sbcl side fixes the real problem.

I'm using XEmacs whereever I can, but for preaching to the still unconvinced, the guided tour to Emacs as of the FSF will be just as useful. Of course, XEmacs ships with a much larger portion of modes and utilities, but unfortunately, development has been quite slowly in the last years and Emacs 22 as of today has come a long way to overcome a lot of the obstacles of older versions.

In an older blog post, I noted that there is no editor integration for the newer distributed versioning systems with Emacs. Turns out, I was quite wrong. First of all, there is an entire page about handling version control in Emacs at the Emacs Wiki. I haven't looked at all of them, but stumbling over DVC, I must confine that there is work under way to have a decent interface in Emacs to those new kids on the version control block. Looks as if DVC is even build or shipping with XEmacs. No more excuses around here, move along ...

Moving back in time is easy when the tools you're using are a versioning control system like CVS or RCS [1], (X)Emacs, it's versioning control interface VC, a Common Lisp system such as SBCL and Slime, the superior lisp interaction mode for Emacs. Yesterday, I was debugging a part of my current hobby project which involves a web interface made with Uncommon Web (UCW). One bit of that web interface that I hadn't paid particular attention to in the last round of modifications had stopped working. But the source that dealt with that particular bit seemed quite innocent to me.

I then looked through the log of modifications I did to the file (which is under version control via RCS) by hitting "C-x v l" (equivalent to "M-x vc-print-log"). Turned out that I did some modifications between versions 1.5 and 1.6 to the part under consideration. Hitting "C-x v ~" (vc-version-other-window) and entering numbers 1.5 (and afterwards 1.6) re-generated the old versions, XEmacs directly visiting these corresponding buffers. This feature alone is worth noting: When extracting the older versions, the current version won't be overwrittten, the older versions will be put into the filesystem with the version number added as a suffix, e.g. foo.lisp.~1.5~.

Now, to see what was happening in between those two versions, I first looked at the differences between the two files. "M-x ediff" and selecting the two buffers, it was easy to spot the part of the code I changed. Next, have a look at the effects caused by the changes: For that I did a "C-c C-k" (slime-compile-and-load-file) on the old versions respectively, so I could then see the old behaviour in real life -- i.e., I could directly use the web app in that older version in my Firefox without any further ado. Of course, this also has to do with the fact that there weren't many interactions between the part of the app in question and other parts of the system, in particular I needn't to check out older versions of other files for the compilation to succeed. Actually, loading that old version certainly broke other functionality of the app, but that wasn't the issue: I only wanted to figure out how the old version worked wrt. to that particular part that wasn't working in my current state, hence a fully working old version wasn't what I was looking for.

Now, it did require a little more close attention to the changes I made to discover the mistake. But I think it's amazing in how few steps it's possible to regenerate an old behaviour. All the while, my current state of affairs remained largely untouched, I didn't need to create any copies or move files around or do a larger recompilation. I think that's a highly effective way of discovering the introduction of a bug.

Update: Kent Pitman, who's giving cll another of his (lately very rare) visits, pointed to one of his older papers (Accelerating Hindsight), which I'm surely have seen before but totally forgotten, which talks about how good Common Lisp is for rapid prototyping. What I've shown above is actually just an example of two points in this paper: dynamic redefinition and editor integration. Go figure.

Footnotes:
[1] Yeah, I know, using rcs or cvs and not bazaar, mercurial or git makes me look like an oldtimer. That said, I do use subversion in my day job, too. I stick with rcs and cvs for smaller (read: one person involved) projects mainly due to one reason: There is no support for the newer kids on the block in the VC shipping with XEmacs. Yes, I know that the VC shipping with FSF Emacs has support for subversion (at least), but unfortunately it's not compatible.

In BaWü gibt es seit einiger Zeit die glorreiche Idee, dass die Schüler, die im sog. Rheingraben in der Nähe (30km) zu Frankreich wohnen, ab dem nächstem Schuljahr an den Gymnasien in den fünften Klassen mit Französisch statt mit Englisch beginnen. Während die Idee auf den ersten Blick ja ganz vernünftig klingt (es ist durchaus hilfreich, wenn man die Sprache seines Nachbarn versteht), bringt sie auf den zweiten Blick eine Menge Probleme. Eine Webseite der Jungen Union (!) www.region-fuer-englisch.de listet einige davon auf (Danke, Kris). Am heftigsten sind m.E. nach folgende zwei Punkte: 1) Der Wechsel von Realschule auf das Gymnasium wird damit fast unmöglich gemacht. 2) Bei einem Umzug aus dem Rheingraben in eine andere Region hat man augenblicklich eine miese Note in Englisch. Und das, obwohl gleichzeitig überall mehr Durchlässigkeit zwischen den Schulsystemen und auch Flexibilität bzgl. der eigenen Lebenssituation gefordert wird. Liebe Politiker, wenn ihr schon das Verständnis zwischen den Ländern fördern wollt, solltet Ihr dafür nicht das Verständnis zwischen den Schülern im eigenen Land vergessen.

Troels Henriksen writes in his blog about library use in the Common Lisp community. It's really funny that he cites UCW as a good example of library usage, whereas I have encountered several problems exactly because of UCWs library dependencies. Another example would be CL-SQL .

Now, in Troels comment section, some unknown poster suggested looking at the integrated XEmacs packaging system. I'm not convinced that this solves the real problem, though, as it doesn't deal with version issues at all besides providing a feature to update to a newer version. But incompatibilites between versions is the major obstacle identified (correctly) by Troels. Revisit UCW: UCW includes (or at least it did when I last checked some months ago) several forked-off versions of some libraries it depends on.

I'm also not convinced that fixing or enhancing the package tools is the right way to go. I think that the major issue is that many CL libraries are far from the stableness that e.g. CL-PPCRE provides. Otherwise, e.g. UCW wouldn't need to provide a box set of it's own versions of libraries, and believe me, I'm very grateful for that box set. I think that the only real solution is working on the libraries.

I also believe there is another reason for many programmers not to rely on external libraries: The fear of bloat. CL traditionally has a very bad reputation with regard to execution speed and memory usage. Now, right after loading UCW in a fresh SBCL, I see the following in top:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11861 schauer 15 0 894m 79m 15m S 0.0 7.8 0:13.22 sbcl

This is a lot of memory which might be a reason why many programmers might feel it's easier to start developing their own application-specific solution.

Another reason might be that Lisp programmers often try to accomplish a solution in terms of the problem domain and this might sometimes conflict with the approaches taken in the libraries. To some degree, this is the Not-Invented-Here issue in cloak (eek, I don't like it this way, I'll go my way instead), but it also touches a lot on issues of style which can't be argued about.

I'm finally fed up with Perl. And I'm more the occasional user than a professional. Probably I should attribute it to my cons-cell based stomach that always nearly made me throw up when I commited an attrocity such like accessing an array as a hash value. As there's more to consider than personal taste when choosing programming languages, I'm afraid, CL won't be the next tool. And no, I won't create a DSL to generate Perl code from Lisp, although the idea has some appeal to it. Instead I will take a closer look at one of the more lisp-like script languages. After some googling around, Ruby seems more attractive to me than Python, in particular as functional programming seems to be closer tied to the community.

But starting with it on my current project already showed some of it problems. To name the most prominent one: lack of good documentation. I miss (documentation 'foo :function) and perldoc -f foo alike, and no, ri foo is nowhere near perldoc. Another problem is the Emacs integration: coming from Slime, ruby-mode and especially the interactive submode for irb reminds me of vacuum cleaners. Perhaps I should take this as a hint to take another look at Eclipse?