A view from the hill

Auf den ersten Blick ist die Entscheidung des Bundesverfassungsgerichts zum Thema Online-Durchsuchungen ein Erfolg für die Gegner. Wie die Süddeutsche etwa im Artikel Kampf um Troja kommentiert, kann man das Urteil und insbesondere das neue Grundrecht als Mahnung an den Gesetzgeber sehen, etwas vorsichtiger beim Datenabgriff vorzugehen. Auf den zweiten Blick kann man an den Reaktionen (etwa bei den Tagesthemen) schön sehen, dass die Legislative versuchen wird, sich die wenigen Krümel herauszupicken, die etwas Interpretationsfreiraum bieten und diese Krümel dann so aufzublasen, dass so gut wie alles darein passt.

Man schaue sich etwa an, was nach einem Bericht der Stuttgarter Zeitung für das neue Polizeigesetz für BW geplant ist bzw. wie BWs Innenminister das aktuelle Urteil des BVG interpretiert. Nimmt man dann noch hinzu, was etwa die Richter zur Handhabung des Urteils meinen, dann sieht man sehr schön, dass mit dem Urteil die Probleme eben nicht vom Tisch sind. Ganz im Gegenteil ist zu erwarten, dass wir Gesetze bekommen, die -- im Rahmen der vom BVG vorgegebenen Grenzen -- so schwammig wie möglich bleiben, die Exekutive und Jurisdiktive mangels Kapazitäten eh den Vorgaben nicht im gewünschten/erforderlichen Maßen nachgehen kann und das neue Grundrecht schlussendlich als Papiertiger darstehen könnte.

Sicher, erfreulich bleibt, dass es jetzt dieses Grundrecht gibt und das man überhaupt die Chance hat, sich darauf zu berufen. Das ändert aber eben wenig daran, dass wir momentan einen Widerspruch zwischen Legislative und Jurisdiktive haben. Wir werden sehen, wie lange sich die Politiker diese unbequemen Wächter der Grundrechte noch ansehen werden, aber andererseits kann man sich ja auch darauf verlassen, dass die "normative Kraft des Faktischen" (in diesem Fall die Unterkapazitäten) mehr Spielraum lassen.

This post is mainly a reference post about a particular topic whose solution wasn't immediately obvious to me from the docs to CL-SQL. Using CL-SQL with (enable-sql-reader-syntax), I had written a routine that looks basically likes this:

(defun data-by-some-criteria (criteria &key (dbspec +db-spec+)  (dbtype +db-type+))
  (with-database (db dbspec :database-type dbtype :if-exists :old)
    (let (dbresult)
      (if criteria
	  (setq dbresult
		(select 'some-model 'other-model
			:where [and [= [some.criteria] criteria]
			            [= [some.foreign_id] [other.id]]]
			:order-by '([other.name] [some.foreign_id] 
				          [year] [some.name])
			:database db))
          (setq dbresult
		(select 'some-model 'other-model
			:where [and [null [some.criteria]]
			            [= [some.foreign_id] [other.id]]]
			:order-by '([other.name] [some.foreign_id] 
				          [year] [some.name])
			:database db))
      (when dbresult
	(loop for (some other) in dbresult
	      collect some)))))

This is ugly because the only difference between those two select statements is the check for the criteria, but I had no idea how to combine the two select statements into one, because it's not possible to embed lisp code (apart from symbols) into an sql-expression (i.e. the type of arguments for :where or :order etc.).

With the next requirement things would become far worse: The order-by statement needs to get more flexible so that it is possible to sort results by year first. Given the approach shown above this would result in at least four select statements, which is horrible. So, naturally I wanted a single select statement with programmatically obtained :where and :order-by sql expressions.

Step 1: It occured to me that it should be possible to have the arguments in a variable and simply refer to the variable. E.g., using a more simple example:

(let (where-arg)
  (if (exact-comp-needed)
      (setq where-arg '[= [column] someval])
      (setq where-arg '[like [column] someval]))
  (select 'model :where where-arg))

So I could now have my two different where-args and two different order-args and use a single select statement. Main problem solved.

Step 2: But for the :where arg in my original problem, only a small fraction of the sql-expression differs. So how do I avoid hard coding the entire value of where-arg? How can I combine some variable part of an sql-expression with some fixed parts? I.e, ultimately I want something like:

(let (comp-op where-arg)
  (if (exact-comp-needed)
      (setq comp-op '=)
      (setq comp-op 'like))
  (setq where-arg '[ <put comp-op here> [column1] someval])
  (select 'model :where where-arg))

But with CL-SQL modifying the reader, there seems to be no way to make <put comp-op here> work. I didn't knew how to get the usual variable evaluation into the sql-expression, or how to escape from CL-SQL's sql-reader-syntax to normal lisp evaluation.

Somewhere in the back of my head where was that itch that CL-SQL might offer some low-level access to sql expressions. And indeed it does. There are two useful functions, sql-expression and sql-operation. sql-operation "returns an SQL expression constructed from the supplied SQL operator or function operator and its arguments args" (from the cl-sql docs), and we can supply the operator and its arguments from lisp -- which is exactly what I want.

Now, the nice thing is that it's easy to mix partly handcrafted sql expressions with CL-SQL special sql syntax constructs that will be automatically handled by the reader (if you enable it only via enable-sql-reader-syntax, of course). I.e., for <put comp-op here> we can use sql-operation, but the rest stays essentially the same:

(let (where-arg)
  (if (exact-comp-needed)
      (setq where-arg (sql-operation '= [column1] someval))
      (setq where-arg (sql-operation 'like [column1] someval)))
  (select 'model 'other-model  :where where-arg))

Now, coming back to my original problem, based on this approach I can split out the common part of the :where and :order arguments and combine those with the varying parts as needed and hand them down to a single select statement. Problem solved.

It's interesting to see that more and more environments for interactive development are popping up. If you're wondering what I'm refering to, it's what has traditionally been labelled as an interpreter: you get some kind of prompt like in a command shell and can directly enter and run program constructs. In Ruby, irb is the tool to use, whereas most, if not all, Common Lisp systems provide you a REPL whenever you start-up the system. REPL is an abbreviation for read-eval-print-loop and this pretty much sums up how interactive development works: you enter some code into the system, and as soon as you hit enter (i.e. finish up the particular line of code), the system will read and evaluate it ("interpret" it, although CL systems may also compile it automatically), presenting you with the results.

To me, this seems like more people come to understand that the classic edit-compile-debug cycle of traditional compiled languages like C isn't particular well-suited to a bottum-up programming style, as advocated by the agile programming hype. Where is the beef? I've been told that the reason why the asian folk don't use forks and knives when dining stems from the opinion that when dining one merely wants to eat, while all handcrafting (like slicing the meat) belongs to the a-priori cooking. The edit-compile-debug cycle tends to produce code more akin to the western steak style, while interactive development tends to produce smaller code parts, all well prepared (read: tested) from the beginning, so you don't end up with bloody interiors as easily. That's because of two reasons, I think: In an interactive environment testing a piece of code is cheap and easy. Hack up a small routine and enter it. Next step: call the routine with some sample data. No need to compile. In case there's an error, you'll be thrown into the debugger right away. With the traditional edit-compile-debug cycle, you're gonna edit the code, save it, call the compiler, edit another piece of code testing it, compile that, too, run that and only then you'll see if it succeeds. This might scare developers from running such tests to often and in result to produce larger routines/code blocks. Second reason: in the interactive environment you'll typically have somewhat less comfort then in a full-blown editor. So I'm arguing that the slight discomfort will help you in keeping things small because then you'll not gonna need far reaching edit support.

There is, however, a price to pay: For starters, because your interactive system provides you with a single environment during development, it's easy to mess it up. That's different with the traditional edit-compile-debug cycle where every time you compile you run the code in a clean environment. This means, old code refering to refactored functions will result in (usually: compilation) errors whereas in interactive development you may miss some spot in need for modification because the old refactored code is still around.
Second, when fiddling with some functions, it may happen that one hacks up several variants, maybe with only slightly differing names or argument lists, so it may happen you forget some crucial code when finally saving your carefully crafted solution (where saving here typically means copy it over to some editor, not saving an image as you may do in Smalltalk or Lisp). Another thing is that one has to be careful not to confuse bottum-up interactive development with hacking without thinking or design-less programming. This threat is all to common because with an interactive environment it's just so easy to get started. Unfortunately, without thinking you'll soon end up with problems one and two and even worse, it's likely that after a little while nobody (including you, the author) will be able to maintain the resulting code. But if you're aware of the potential pitfalls, interactive development is a nice way to do explorative programming which avoids a lot of time consuming labour you have to do with the traditional edit-compile-debug cycle.

As a side-note, environments offering incremental compilation like Eclipse aim at a similar goal, but only go half the way as they only take care of the edit-compile part. Only when one combines that with a test-driven development style you arrive at a similar point, because now the test code, when automatically run, will take care of immediately running the code in question (the eval/print part as far as testing the code is concerned).

To finally come to the reason for this blog post, I was pleased to learn about the Perl Console, which is not the same as the Perl debugger. I'm eager to try it out, although I'm sure that interactive development with a repl and the executable brain dump that Perl code usually is, is a highly dangerous mix.

Update: There is also a series of articles about implementing a REPL in Perl which may be of interest.

This is just a minor rambling inspired by a recent thread on cll about the ups and downs of using Rails. What I find really overwhelming is the lack of proofs that other (typically Lisp-based) frameworks really have so much benefit over the established more traditional frameworks (e.g. Rails, Zope, plain PHP, ...). All I can see is starter documents, tutorials etc. that show how to program yet another blog or reddit clone in some particular framework. I miss fair comparisons and detailed discussions why and where exactly that one particular way of doing things has benefits. And especially with regard to those reddit clone prototypes: AFAICT, reddit was far beyond implementing some half-baked prototype (in Lisp) when they decided to switch (to Python). Yes, for demo purposes reinventing the wheel with some unround edges might be acceptable, but in reality only rolling wheels will be sold and this is where the challenge is.

I believe that while it's nice that frameworks help with implementing your 500 line application, they really need to show their value with large applications. Where is a discussion of a large on-line shopping system in, say, UCW and CL-SQL, with secure shopping over SSL, login handling, input validation, distribution over multiple servers etc.? Now, I'm not suggesting that UCW, Weblocks etc. can't be used to build such websites, but it would be nice to see a much more in-depth explanation/discussion/comparison of how to do it. That would also help increase the credibility of Lisp or at least, of lispers discussing (other) web frameworks.

While the doors at plattentests.de are still open for voting on the best music of 2007, I'm going to be away for the rest of the year, so I'm going to post my favourites now.

Best records of 2007:

In case you're interested in object-oriented programming in Lisp and happen to speak German, you'll be happy to learn that Pascal Costanza of ContextL and AspectL fame (apart of his postings to cll), has given a talk about CLOS, the Common Lisp Object System at the Hasso-Plattner Institut, Potsdam. And while we're at it, Pascal also has a very interesting blog post about the origins of the advice facility.

German:
Falls Dich, lieber Leser, interessiert, wie man objekt-orientiert in Lisp programmiert, solltest Du Dich darüber freuen, dass Pascal Costanza, bekannt durch ContextL und AspectL (abgesehen von seinen Postings in cll), einen Vortrag in Deutsch über CLOS, dem Common Lisp Object System, am Hasso Plattner Institut in Potsdam gehalten hat. Und weil wir gerade dabei sind, Pascal hat außerdem einen sehr interessanten Blogeintrag über die Geschichte von 'advice'.

Just as with their first albums, "Towards the bright light" and "The back room", it's hard to ignore the impulse to compare Interpols new record "Our love to admire" to the new album "An end has a start" by The Editors. While I did reviews of the first two albums for plattentests.de (here and here), I haven't reviewed the new ones there, so I'm gonna take a step at an informed comparison here, generalizing a little and also taking into account the slew of other bands that people tend to ignore.

Zugschlus bloggt in einer Serie über Text-UIs and GUIs. Da ich kein Cookie-Freund bin, hier eine Nachfrage/Gegenargument. Ich blicke seine grundsätzliches Thema gar nicht. Die Frage des UI ist sehr auf die Frage der Konfiguration der Applikation konzentriert, dabei dürfte die doch idR. eher den unbedeutenden Teil des Nutzerinterfaces ausmachen. Ich bezweifle etwa sehr die Grundannahme des verlinkten Eintrags: Warum sollte eine Applikation mit GUI keine Textkonfigurationsdatei haben? Klassisches Beispiel dafür dürften etwa diverse Tcl/Tk-Applikationen sein, oder etwa FVWM.

Auch dieses Jahr findet bei plattentests.de der Jahrespoll der besten Alben, Songs, etc. statt.
Und wie jedes Jahr gibt es auch wieder fette Pakete mit CDs zu gewinnen. Alles weitere findet ihr dann dort.

Every now and then, I'm still blown away by the elegant power that the repl (read-eval-print-loop) of Common Lisp provides for everyday tasks. A recent example: I needed to fix a broken column definition in one of my Postgres databases. I'm not a DB pro, so I will just drop the column. But of course, we want to retain the old data, so here we go (using CL-SQL):

SOMEDB> (connect +db-spec+)
#
SOMEDB> (defvar *olddata* (select [id] [data] :from [sometable]))
*OLDDATA*
SOMEDB> (execute-command "ALTER TABLE sometable DROP COLUMN data;")
NIL
SOMEDB> (execute-command "ALTER TABLE sometable ADD COLUMN data varchar;")
NIL
SOMEDB> (loop for (id value) in *olddata*
                  do (update-records [sometable] 
                                 :attributes '(data) 
                                 :values (list value) 
                                 :where [= [id] id]))
NIL
SOMEDB> (disconnect)

That's it. First we open a connection, store away the olddata (which will be returned as a list of tuples) in a global variable, modify the table and finally restore the data. Now, what I find really nice about this is that I can operate on the data as if I had an intersection of psql, shell and a real programming language, which is pretty much the point of this blog post. I think that Ruby probably provides a similar environment with irb. And, hey, today I learned that XEmacs comes with an interface to postgres, so I might have been able to do it from within my favourite editor, too ...

Liebe Leser, zuerst die offensichtliche Nachricht: Es ist Mitte November. In other news, 2+2=4. Aber nicht so offensichtlich ist, dass der Schneebericht des Liftverbunds Feldberg 50cm Schneelage verkündet sowie fünf geöffnete Lifte am kommenden Wochenende. Schön, dass die Brettlemärkte in unserer Umgebung schon waren. Weniger schön, dass ich meine neuen Ski erst noch einstellen lassen muss. Aber Schlittenfahren dürfte auch genug Spaß machen.

Die taz versucht sich an der Erklärung Wie Indie-Rock den Soul verlor. Die taz und/oder auch der zitierte Frere-Jones scheinen jedoch eher jeglichen Plan verloren zu haben.

Zum einen ist es absurd, so zu tun, als wäre in den 70ern die Rockbewegung voll von schwarzer Musik gewesen. Fakt ist, dass natürlich Bands wie The Clash im Reggae-Umfeld umtriebig waren. Aber das war bei weitem nicht bei allen so. Wo ist etwa das Reggae-Stück der Sex Pistols oder von Joy Division? Dass es zeitgleich zur Explosion der Punk- und Independent-Szene auch noch ein Revival (und somit nicht eine Initialzündung) der Beschäftigung von Weißen mit schwarzer Musik gab, ist sicher eher Zufall als Absicht. Erstens gab es in der Disco-Musik der 70er viel schwarze Musik, Soul, der den Namen auch verdiente. Zweitens war gerade Reggae populär. Drittens wurde nochmal (Revival!) die Musik, die die jamaikanischen Einwanderer in die armen Viertel Englands mitbrachten, populär: Ska, mit dem typischem Offbeat, wie ihn zu der Zeit etwa die Specials, Selector oder Madness spielten. Da die Skinheads und die Punks in den gleichen Ecken hausten, ist es nicht so verwunderlich, dass es da Vermischungen gab. Aber was gab es denn jenseits vom Offbeat-Griff bei Police und Clash? Man sollte sich vielleicht mal daran erinnern, dass nach dem Ur-Punk gleich der coole Synthie-Pop bzw. New Wave gefeiert wurde.

Zum anderen ist es ebenso absurd, zu behaupten, dass im heutigen Indierock keine schwarzen Wurzeln mehr erkennbar wären. Nehmen wir ein sehr plakatives Beispiel: Bloc Party. Sehr erfolgreich mit einem farbigem Sänger. Jetzt kann man sich natürlich streiten, ob der Sound von Bloc Party "Soul" enthält, aber "Rhythmusgefühl" und "genreübergreifendes Zitieren" findet sich da ganz sicher. Aber nehmen wir noch ein anderes Beispiel daher: Die Blues Explosion, besser bekannt auch als John Spencer Blues Explosion. Sie zelebrieren ausschließlich den modernen Rückgriff des Indierocks auf alte "schwarze" Musik. Dann hätten wir noch so Bands wie "At the Drive-In", die eine sehr moderne "schwarze" Variante des modernen Post-Hardcore feierten. Was genau ist denn eigentlich der angeblich verlorene Soul? Gilt etwa ein Jimmy Hendrix auch als "schwarze Musik"? Dann sind "Motorpsycho" oder "The Mars Volta" Beleg dafür, wie sinnlos die Behauptung ist.

Moderne Musik wäre ohne den Einfluss schwarzer Musik nicht denkbar. Das gilt für Indiemusik genauso wie für das alltägliche Radiogedudel.

Some time ago, I was looking at splitting some text with Elisp, Perl, Ruby and Common Lisp. Yesterday, when I again had to do quite the same thing, it occurred to me that the Common Lisp solution was unnecessary complex/long. I'm not a Perl guru, but I believe the following is probably hard to beat even with Perl:

CL-USER> (format t "~{<li>~A</li>~%~}" (cl-ppcre:split "\\|" "Kim Wilde|Transvision Vamp|Ideal|Siouxsie and the Banshees|Nena|Iggy Pop"))

<li>Kim Wilde</li>
<li>Transvision Vamp</li>
<li>Ideal</li>
<li>Siouxsie and the Banshees</li>
<li>Nena</li>
<li>Iggy Pop</li>
NIL

For the uninitiated, it's not the cl-ppcre library which is interesting here but the built-in iteration facilities of format. See the Hyperspec on the control-flow features of format for details. Now, I usually tend to avoid the mini languages that come with Common Lisp like the one of format or loop when writing real programs, but when using Lisp as a glorified shell they come in very handy.

Dear Lazyweb, I'm looking for a way to automate unit testing with the help of ASDF. I'm using XLUNIT at the moment, but this isn't really relevant. What I want to achieve is that on every compilation of some source file, it's corresponding test file will be loaded (and hence the tests it contains run). However, what I seek to avoid is simply adding the test files to the component definition; the test files should be kept separately. From what I gather from the ASDF documentation, this should be possible using :perform forms, but the very same docs leave me wondering how. What I found (looking at Jörg Höhles asdf file for iterate) is how to load and run a complete test-system, but this is not what I want to do. Ideally, I would like to have a single perform instruction looking something like this:

(defmethod asdf:perform ((op test-op) c)
  (when (asdf:glorified-find-component "my-component" c)
        (let ((component-test (find-component-test c)))
            (when component-test
                   (asdf:operate 'asdf:load-op 'component-test)))))

Here #'glorified-find-component needs to recursively follow the components parent and #'find-component-test should return a component c-test (with c expanded, obviously). Now, I guess I'm just clumsily reinventing the wheel and hence I'm wondering if somebody has already solved "the problem".

While I'm at it (it referring to testing), Stefil looks like a very interesting test environment in case you're developing your programs with Emacs and Slime. From there, I found a link to Phil Gregory's test framework comparison which I found much more enlightening than the ALUs list of test frameworks.

Morgen abend wird in Freiburg eine der dezentralen Kundgebungen gegen Vorratsdatenspeicherung stattfinden. Die Demo unter dem Namen "Freiheit statt Angst" findet am 6.11. ab 17.00 Uhr am Münsterplatz statt. Ich werde leider nicht da sein können, weil ich mal wieder in Mannheim weile, aber hoffentlich finden sich auch so genügend Leute.