January 23rd, 2009
A folding language
In the descriptions of Ioke, I’ve lately favored using the concept of a “folding language”. I think that description feels very right, and I’ll explain a bit more about that here. I didn’t come up with the term though – Steve Yegge did, in a passing remark in his post “Wizard School”. It’s a funny read, actually. When I read it, the remark about folding languages stayed with me, and for me, Ioke is exactly that.
Let me first reproduce the paragraph from Yegge’s post that mentions this concept:
Of course, blasting out hundreds of thousands of lines a year is missing the point. If a Wizard is given complete technical control over a project (which is usually the smartest thing to do, but companies are rarely very smart), the Wizard will typically write in one of the super-succinct “folding languages” they’ve developed on campus, usually a Lisp or Haskell derivative.
They call them Folding Languages because they write code that writes code that writes code… Wizards swear by it, and there’s no question that they can produce amazingly compact, fast, clean-looking code. But 90% of the devs out there claim they can’t read it, and whine a lot about it to their bosses. Given that most companies can only afford a few Staff Wizards, the Wizards are usually forced to capitulate and use Java or C++. They’re equally comfortable in any language you throw at them, though, and if you force them to use a verbose language, well, you get what you ask for. It still amazes me that companies are bragging about how many lines of code their Wizards have produced. Potential startups take note: your competitors are usually idiots.
Especially in the evolution from Ioke 0 to Ioke S – when I added syntactic macros – the ways I can write code manipulating code is really nice. I am used to it from Lisp, of course, but to me Ioke have this in spades. It feels like a Lisp turned inside out.
Ioke is designed to be expressive, and being able to fold code is an important part of this. Earlier today, Sam Aaron talked about folding languages on IRC, and he said this (I have added some punctuation, that doesn’t change the meaning of the quote):
When I read it I first thought that all languages support folding to some extent, but some languages have limitations – such as the limitations of paper, i.e. with paper you can only fold it so many times before it’s impossible to add another fold, whereas languages like Ioke seem to support infinite folding. And some languages are just so brittle, that folding is pretty impossible.
That pretty much says it all. =) If you want a good example of how Ioke supports code folding, go to the git repository and take a look at the evolution of the Enumerable mixin from Ioke 0 to Ioke S. Let me take as an example the implemention of any?. (I will provide the real source, except for the documentation text). In Ioke any? can take zero, one or two arguments. If you give zero arguments, the method will return true if there are any true values in the collection. If you give one argument, that argument is a message chain that will be applied to every element in the collection. If that message chain returns true for any element, then the any?-method returns true. Finally, the two argument version takes one variable name, and one piece of code that uses that variable name. The code will be executed once for every element with the current element bound to the name. Some usage examples:
[nil, false, "foo"] any? ; => true [1, 2, 3, 4] any?(odd?) ; => true [1, 2, 3, 4] any?(x, x*x > 10) ; => true
As you can see, these versions all make sense. And what is more, this pattern of taking zero, one or two arguments is pretty common in the Enumerable implementations. Most of the implementations are actually variations on this theme.
So, the implementation of any? in Ioke 0 looked like this:
Mixins Enumerable any? = macro( len = call arguments length if(len == 0, self each(n, if(cell(:n), return(true))), if(len == 1, theCode = call arguments first self each(n, if(theCode evaluateOn(call ground, cell(:n)), return(true))), lexicalCode = LexicalBlock createFrom(call arguments, call ground) self each(n, if(lexicalCode call(cell(:n)), return(true))))) false)
Pretty interesting code, right? Well, there are several problems with it. First, the actual logic of what it does is spread out in three different places, one for each variation in arguments. The second problem is that this code doesn’t check the argument arity. If you give another combination of arguments, it will fail noisily. But the most important problem is that this structure is the same for most of the Enumerable methods. Take a look at two more, none? and find.
Mixins Enumerable none? = macro( len = call arguments length if(len == 0, self each(n, if(cell(:n), return(false))), if(len == 1, theCode = call arguments first self each(n, if(theCode evaluateOn(call ground, cell(:n)), return(false))), lexicalCode = LexicalBlock createFrom(call arguments, call ground) self each(n, if(lexicalCode call(cell(:n)), return(false))))) true) Mixins Enumerable find = macro( len = call arguments length if(len == 0, self each(n, if(cell(:n), return(it))), if(len == 1, theCode = call arguments first self each(n, if(theCode evaluateOn(call ground, cell(:n)), return(cell(:n)))), lexicalCode = LexicalBlock createFrom(call arguments, call ground) self each(n, if(lexicalCode call(cell(:n)), return(cell(:n)))))) nil)
Can you see the difference and describe what they do? I can definitely say that for me, maintaining these methods got problematic quickly. And as you see, the structure is exactly the same, but since the insides of it are quite different, it would be hard to do something about this code in Java, except refactor out small methods for different things. But the essential duplication would still be there.
In Ioke S, the implementation of any? looks like this:
Mixins Enumerable any? = enumerableDefaultMethod( ., if(cell(:x), return(true)), false)
enumerableDefaultMethod takes three arguments of code. The first argument is what should be done before (nothing in this case, hence the dot). The second argument is the code that should be executed on each iteration. And the final argument is the code that should be used to return something after the iteration. The n cell is the same as in the long implementations – the current value in the collection. The cell x is the value after applying the code to it. With this understanding in place, you can see that any? will go through each element, check if the transformed value is true, and in that case return true. If none is true, it returns false. Notice that enumerableDefaultMethod is not an exposed method. It’s not part of any API – it’s just a syntactic macro used to make it really easy to implement these Enumerable-methods.
It also gives some other things. For example, it will automatically add arity-checking, so that this method will now return a meaningful condition if given the wrong kind of arguments.
And lets go to the Ioke S implementations of none? and find:
Mixins Enumerable none? = enumerableDefaultMethod( ., if(cell(:x), return(false)), true) Mixins Enumerable find = enumerableDefaultMethod( ., if(cell(:x), return(cell(:n))), nil)
Here you clearly see the difference between any? and none?. And it is also clear what find does too, namely if any transformed value is true, return the original value for that transformation. Otherwise return nil.
Now, enumerableDefaultMethod is actually implemented using dsyntax. And dsyntax is a syntax that returns a new syntax, that will finally execute and expand the implementations. And that is what I mean with code folding. You can take coded abstractions and fold the code until the only thing you need to write is what makes this different from the defaults. The above code could probably have been even easier to read by assuming that an omitted first argument means there will be no initialization. So as you can see, this process never really stops. But it allows you to code on a higher level. And when you find the right abstractions, what you end up writing is much closer to the business domain. And that is really what most developers want, right?
