SpocP and Lisp.


For a few years I’ve been involved with an open source project called SpocP, through work. SpocP is a framework and protocol for handling mostly any authorization tasks any application can have. The base distribution is a server written in C, which when started with rules definitions answers TCP (or socket) requests with YES or NO, and attaching a blob if the rule says so. If you want to, you can write all your authorization information directly in SpocP rules. This is usually not that interesting since the common case is that you have a data source somewhere that you’d like to use as basis for authorization. Maybe a LDAP server with role information, maybe an RDBMS, or something entirely different.

The interesting thing about SpocP is the format used for Rules and Queries. (Rules are what SpocP uses to provide an answer, a Query is a specific authorization request translated into the SpocP Query language. In human terms a rule might be “All persons who have OU=itc in our LDAP server have access to all pages of the web application named ‘itdoc'”, and a typical query that might answer YES for this rule is “Can the person with uid ‘olagus’ access the page ‘showLinks.do’ for application ‘itdoc’?”). The format for queries and rules in SpocP is based on S-expressions (which is a fancy word for more-or-less-Lisp syntax). The reason for using S-expressions is the interesting fact that S-expressions have the mathematical property that one S-expression can be compared to another, and judge less-than-or-equal or not. This is all that’s needed to implement generic authorization facilities, and also some other fascinating possibilities. As an example, take a rule file looking like this:

(spocp (action view) (resource (app “itdoc”) (page “admin.do”)) (subject (uid)))
(spocp (action view) (resource (app “itdoc”) (page)) (subject (uid)))
(spocp (action admin) (resource (app “kimkat”)) (subject (uid) (role)))

and a query like this:

(spocp (action view) (resource (app “itdoc” “http://itdoc.it.ki.se”) (page “showLinks.do”)) (subject (uid “olagus”) (role “user”) (loggedIn 200505260115)))

When receiving this request, SpocP will walk through all rules until it finds anyone where the RULE is <= QUERY. If it does find one matching it will return "YES" otherwise "NO". In this case rule number 2 would make the answer "YES". You've now seen the first part of SpocP matching. But how do we involve external data sources? With something called boundary conditions. The idea is that for each rule matching, check the attached boundary conditions, and if these return positive answers too, the whole expression is deemed true. A typical boundary condition may look something like this:

urn:spocp:ldapset:ldap.ki.se;ou=people,o=ki.se;{uid & ${uid}}/ou & “itc”

which more or less is the same as the LDAP query (&(uid=$UID)(ou=itc)).

This boundary conditions checks if the second element of the list with tag “uid” exists in our LDAP server, and if that entry also has an ou with value “itc”. Boundary conditions can be chained and linked with and, or and not, and you can also predefine boundary conditions so you don’t have to define the same one more than once.

I recommend using SpocP if you have a need to decouple your authorization from the application. As soon as you have defined what kind of queries a typical application may do, you can implement this on a central server, and when your business rules change, you can change the SpocP definitions without having to change the application. There is not much extra complexity involved and you buy yourself some very nice flexibility. There are SpocP client libraries available for most of the standard languages.

The fun thing with SpocP is that the fundamental operation of checking less-than-or-equal between two S-expressions have always been defined mathematically, and in some highly optimized C-code. When fooling around with this, I found a simple executable definition of the SpocP core in Common Lisp. The whole thing look like this:

(defun starform-p (list)
(eql (first list) ‘*))

(defun starform-match (query rule)
(let ((form (second rule))
(data (cddr rule)))
(case form
(any (member query data))
(prefix (string-equal (car data) (string query) :end2 (length (car data))))
(suffix (string-equal (car data) (string query) :start2 (- (length (string query)) (length (car data)))))
(range (member query data)) ;;not implemented
(set (member query data))
(t t))))

(defun matches-p (query rules))
(some #'(lambda (rule) (match-p query rule)) rules))

(defun match-p (query rule)
(if (and (atom query) (atom rule))
(eql query rule)
(if (starform-p rule)
(starform-match query rule)
(every #’match-p query rule))))

The only thing missing here is one of the so called star-forms, range. Otherwise this is a complete definition of the SpocP core evaluation of rules. To use it, do something like this:

(matches-p ‘(spocp (action view) (resource “foo”)) ‘((spocp (action admin) (resource) (subject (uid) (role “admin”)))(spocp (action view) (resource “foo”))))



Craig Larman at KI.


A few weeks ago, Craig Larman visited KI and had a one day presentation about agile development to some 30 people from different professions at KI. I was there and some parts were very interesting.

Of course, most of it is not new for the developers, but the we got some good firepower for convincing management about some more realistic practices. KI haven’t been very good at embracing the new and hot in development methodology the latest years, but hopefully this is about to change.

So, what about the presentation? The first part, ’til lunch, were mostly statistics and “management-convincing” arguments why agile is good practice. The only really interesting part were Larman’s dissecting of Royce’s original paper on the waterfall method. It’s really totally obvious to anyone that actually reads the paper instead of just looking at the graphs, that Royce really is against the waterfall method except for very small projects when all requirements are known upfront and cannot be changed.

The afternoon session contained much more of interest, at least for me. He talked some about good tools to practice good Agile, naming continuous integration, test driven development, good version control, self organizing teams and good dissemination of information through wikis as key points. IT Center – that is, me and me 4 colleagues – at KI are pretty good at this. The only weak point is our use of Wiki. I have some plans for this, since I totally agree with Craig about this. Wikis are unique in the way they combine simplicity with expressability and information processing.

After this I finally got something really worthwhile out of the presentation; Craig presented the first good argument for pair programming that I’ve ever heard. Regarding Pair Programming I’m probably like most programmers: very skeptical. I have tried it a few times and I don’t like it. But anyway, Craig’s premise was a situation where you had programmers working on several projects simultaneously, and also having maintenance work on applications they had the responsibility for.

In this case it’s very easy that people get unproductive, due to insistent interruptions from other projects or immediate maintenance work. Usually this most of the maintenance work is also person dependent. (This situation describes exactly my groups current work environment.) When dealing with issues like this it makes sense to create a rotating schedule where programmers work on one project at a time, or completely on maintenance. When something that is dependent on someone in a project shows up, one of the maintenance programmers pair program with this person to fix the problem, which ensures that the next time this problem appears, there will be at least two persons capable of handling it.

All in all, it was an interesting session and I felt I got out of there knowing more than I knew before. The most important thing is still having someone to have as reference when talking with management about adopting more agile methods.

Maybe I’m imagining this, but it feels like since Craig was here, we’ve been allowed much more slack in using technology that’s not “industry standard”, but more agile to solve tasks suited for problems suited for it. For example using Ruby on Rails to create internal test and development tools for bigger projects seems to be totally accepted now.



Announcing RbYAML


This is an announcement of the availability of RbYAML, a pure Ruby YAML parser, based on the Python project PyYAML3000. The last two weeks I’ve converted this to Ruby, with more or less the same functionality. The glaring hole is Unicode, which isn’t there right now.

The project will be updated regularly, and will also soon move to RubyForge.

Right now the homepage is here: http://rbyaml.ologix.com

In a few days I’ll post a longer blog with some information about the Python-to-Ruby translation, which had some interesting surprises, at least for me.



JRuby and YAML.


Due to my long time away from such matters, the proposed YAML engine for Java that I intended to create have not got so far. Right it seems as if YAML is a major bottleneck in JRuby, so I started looking around. Loo and behold, at http://jyaml.sourceforge.net/ there seems to exists something that’s almost exactly what we need. So I’ve impromptu decided to discontinue my JAML project and instead tackle the task of making this engine work together with JRuby.

RubyGems is really close right now. Each step takes us closer and closer, and right now it feels as if we have 99.99% of everything needed. If just the SourceForge CVS-system could start working we could actually really get this of the ground now.



JavaOne is go


Alright, San Francisco: here we come! Our development group is going to JavaOne together this year. Actually, this is the first time there has been a development group to talk about, since I was it’s only member until this summer. And there seems to be much fun going on. I’m particularly looking forward to the sessions with a focus on other languages. Sometimes I find it strange to have lots of Java Certifications and still loathe the language.

Things I’m not going to miss: The JRuby session, Best practices with Generics and other Tiger features, What’s new and exciting with AspectJ, Rapid web application development with Grails, Portlet Best Practices, Trails in depth, Server-side scripting, Scripting in Mustang, Effective Java reloaded, Mustang and Dolphin. And these are just a few. Wow, it’s going to be a busy week, and trying to meet up with lots of friends and contacts in SF is going to make a complete experience of this.

By the way, it’s interesting to see that there are many sessions on Spring, Dependency Injection and other stuff that make Java lightweight. One title especially caught my eye: “DI: Configuration Files Must Die”. My take on the configuration file-issue is to have it as executable as possible. A project I’m working with on and off in purview of Ologix will use Common Lisp and judicious use of macros to create the configuration information. The nifty thing about this is that I haven’t yet decided what the main language will be. It could be Lisp, it could be Ruby or it could be Java. But whatever I choose it will still be fairly easy for me to handle the configuration issue. In Lisp I can just read-eval it, in Ruby it’s very easy to parse and in Java I could use Jatha and expand the macros to calls into the Java structure. Not very efficient, maybe, but configuration is not performance. Flexibility is king.

Hope to see you in San Francisco.



Current status


Due to massive loads of work I have been unable to write here for a few months. That will probably change as of right now, since the current projects are finally soon closing down and the workload returns to more or less normal.

Oh, and if you for some reason haven’t read Steve Yegge’s stuff yet, go do it here and on his blogspot here. This guy know what he’s talking about.



Ruby off the Rails, Ruby for Java developers.


A very interesting article from IBM Developerworks: Ruby off the Rails.

On a similar note, I propose that Jakarta Struts should be renamed to Java in Jail.



The power of YAML and XML.


Me and my colleagues have recently spent some time debating the readability and usability of YAML versus XML. As expected, the opinions vary wildly. Because of this, I’ve spent quite some time thinking about these issues. I really believe that the question is one about power, that readability, usability and succinctness are what defines the power of a data format. This discussion is restricted to formats which are text based, human and computer readable, and are mostly used for configuration and data interchange. XML is of course used also for RPC and build scripts, among others, but these uses have been agreed to be a nonoptimal, for many reasons.

So, I will use a simple example in three different encodings. The example is the common configuration of a data source for the use of a library. This is intentionally a simple problem, but it tends to show the main differences between encodings.
For this setup – without using binary or compressed formats – the simplest encoding is probably a line based, character separated value-file like this (where hash is a comment line):
(This example wraps in stupid way. Just imagine it’s only two lines.)


#name, type, database, server, uid, pwd, param1, param2, param3, param4, param5
development, postgres, my_dev, db.dontexist.com, mydev, secretpwd, encoding=LATIN1,,,,

The problem with this approach is apparent; there is no easy way of knowing which field means what. You need a comment just for the sake of the human reader to provide this information. If you for some reason write the password and username in the wrong order, the dataformat will not catch this. Another very real trouble is that as soon as you have to specify information specific to one instance of a database you have to use a new language inside the field. (In this case the small language is only defined as name=value, but it’s still another concept to get used to, which is not part of the regular data format). In regard to this, it’s not possible to define lists or maps without defining extra syntax for it.
The positive point with this encoding is that it’s probably as short as it can be, without being binary or compressed. It’s very easy to parse, but it’s not generic at all.

Example number 2 is a data source definition from JBoss:


<datasources>
<local-tx-datasource>
<jndi-name>Development</jndi-name>
<connection-url>
jdbc:oracle:thin:@db.dontexist.com:1521:orcl
</connection-url>
<driver-class>
oracle.jdbc.OracleDriver
</driver-class>
<user-name>jbossdev</user-name>
<password>secretpwd</password>
</local-tx-datasource>
</datasources>


It’s longer, of course, but it’s still a very simple XML-document. No namespaces, no entities, no DTD’s, no attributes. It’s just very basic XML. And sure, it’s more readable than the custom format. And it is generic, in the sense that you can define different formats and have a validating parser read it for you. But is it easy to read? I don’t think so, there’s to much noise. Sure, the element end tags make it easy seeing what ends when, but that’s not a big deal anyway, since most XML documents actually use indentation to make it easy enough for a human being to read. Compare the previous document with this:


<datasources><local-tx-datasource>
<jndi-name>Development</jndi-name>
<connection-url>jdbc:oracle:thin:@db.dontexist.com:1521:orcl
</connection-url><driver-class>oracle.jdbc.OracleDriver
</driver-class><user-name>jbossdev</user-name>
<password>secretpwd</password></local-tx-datasource>
</datasources>

which is the exact same document, but well. I wouldn’t edit it without a good editor to check for validity. Especially if it’s on a production machine.
As I’ve said, it’s good for reading, if it’s your first time with the data format, but after a while it gets quite annoying trying to sort out the relevant information from all the end tags. Another problem is how to add simple non standard parameters. For example, if I wanted to add an Oracle-specific parameter, I would in this example have to change the JDBC connection string. This is not the fault of XML, of course. The other route would have been to add some element like this
<param name=”encoding”>LATIN1</param>
to the XML schema. But this denies the use of end tags to see where attribute mappings end. If we have long blobs of information stored in these params, and if there is more than one we can’t see from the ending which one we’re at. So that point is lost here.
Another thing which sometimes is a problem is the lack of simple mappings and lists as datatypes in XML. Sure, you can define them in your own schema, or remap CDATA-sections to your own format, but this is not part of the spec, and will never be readable by a standard XML-parser without help from you.
You will never be able to do something like this in DOM: ((List)node.getNodeValue()).iterator() since nodeValue is defined to be a String and there is no intrinsic sequence in XML values.

Lastly, YAML. This example is taken from Rails:


development:
adapter: postgresql
database: xyz_dev
host: db.dontexist.com
username: railsdev
password: secretpwd
encoding: LATIN1

Notice that indentation and newlines matter in YAML. Actually, it’s this context sensitivity that
makes it extremely readable for human beings, but still being parseable by machines. One thing that is immediately noticeable is that the “encoding”-parameter – which is PostgreSQL specific – looks exactly the same as the other parameters. No no-YAML construct is used to represent this. The next interesting point is that even though you’ve never seen YAML before, you should be able to reason out that something that’s called “development” have a few properties attached to it, with the values seen.
In reality, what happens in a standard YAML-parser reading this, is that a map will be created with one entry with key “development” and as value have one map instance with the keys and values specified. YAML have support for three datatypes; mapping, sequence and scalar. These types can be specialized to a specific implementation, which means that any object can be serialized as YAML without very much work. In the Ruby implementation every object gets a new method added, called to_yaml, which returns the YAML representation of that instance. There is a static method called YAML::load which returns the correct object for the serialized YAML stream sent in.

All three formats have much more advanced features, of course. You can do whatever you want with the specific format. XML have schema, validation, XPath and much other. YAML have tags, aliases and some more things. But when looking at all this from the perspective of what you can easily represent inside the language, without making new meta language constructs, and still retain readability and easy understandability, well. My vote is on YAML. I have seen wrong end tags in XML more times than I can ever count, and I still find it impractical to extract relevant information from convoluted documents. Have you ever tried reading a slightly complex WSDL-definition? Change it by hand? It isn’t fun.

Of course, S-expressions is a good alternative, which have many of the nice points of YAML, while not having the dependency on whitespace for structure. But the whitespace is fun. I look at the lists and text files I’ve scattered all over my computer, filled with notes for myself, and I realize that most of it could be readable by YAML without a change. That is more or less how I write lists and mappings for myself, without ever intending it to be read by computers. I sure don’t write notes for myself in XML.

I would never program in YAML, or make a turing complete language out of it, but for pure data representation that should be easily readable and writable by humans, while still being as succinct as possible, it’s probably the best thing right now.
But would there ever be a need for using the data to drive programs, S-expressions is the way to go.

A last note, if you believe YAML to complex, try out JSON. It’s actually a proper subset of YAML and should be read and writable by all YAML-compliant processors. The same thing goes for standard Java properties-files.



YAML, Java and JRuby.


There is currently no good support for YAML in Java. YAML is a very nice data format for all people tired of XML (and who isn’t?). It’s easy on the eyes, easy to understand and easy to write. It’s also rapidly becoming the exchange format of choice for many dynamic languages like Ruby, Python and Perl. If you’re more interested in YAML, some information can be found at http://www.yaml.org.

Of course there should be a Java implementation for YAML. There exists some old half hearted tries, but nothing that actually works all right. And since JRuby do not have YAML support yet, a Java implementation would help that situation too.

So I’ve started a new open source project – soon to be hosted at Sourceforge – called JAML. It’s very straight forward right now, based on PyYAML right now, since Syck is write-only code, and most other systems were based on automatic lexers and parsers. As soon as the project is public at Sourceforge, I will post a note here. In the meantime, if you’re interested in helping out, or looking at the code, send me a mail at ola AT ologix.com.



Unorthodox Hibernate.


Today I did something slightly unorthodox with Hibernate. Or at least I believe it is. And I couldn’t find a better way of doing it. You be the judge.

First, some background. Most of the system I’m working on is fairly normal with JavaBeans and whatnot. But we do have twenty-odd of something called types. These types are supposed to be our own implementation of the type-safe enum pattern, with the added complexity that the values of these enums is saved in the database, each value have one or many localized strings attached to it. So for, so good. The old solution was based on some sql and some reflective calling of static getInstance-methods. This was clearly not possible with Hibernate. Now, let me be clear about this. What was really the point was getting Hibernate to return the same instance for the same key every time.

After some fiddling about with this, it turned out that the best solution was to go grope around the innards of Hibernate and implement my own EntityPersister for this task. (By the way, if you should go down this route – and are using Hibernate 3 – be aware that the API-docs are not totally up to date, regarding demands on constructor signature and such.)

According to all documentation I could find, implementing your own EntityPersister was not something that should ever be needed by a user of Hibernate. Clearly this is not so, or I’m missing something obvious. So, my question is simple, how should I have managed this with existing Hibernate tools? And no, modifying the model objects are not an option. To much dependencies.