I just checked in a few updates to my openssl branch for JRuby. Boy is it tricky getting everything right. It seems like every DER format Java crypto emits differs from the OpenSSL DER output. And it’s really incompatible. As an example I have been forced to reimplement the DER dumping for X509 certificates myself, and that’s not the only place.

But the work is actually going forward; as fast as I can make it when I’m only doing this in my spare time and my regular work takes lots of time right now. I can’t say for sure when it will be finished or usable, but I know for a fact that most of the MRI tests run now. What’s missing is PKCS#7, X509 CRL’s and X509 cert-stores, plus the regular SSL socket support. Not much, compared to what actually works.

But that leads to me to two issues. We have recently agreed that OpenSSL support will require BouncyCastle and Java 5. There is really no other way to get this working. 1.4.2 is fine for basic Digest support and some of the RSA/DSA support, but Java is sorely lacking in the ASN.1 and X509 department. Nothing whatsoever. Which is why we need BouncyCastle, which is fairly complete. I have only been forced to reimplement one or two central classes. Quite good. But SSL support is another story. As you may know, 1.4.2 has SSLSocket and SSLServerSocket. The problem is this: they aren’t any good. As a first, they are blocking, and there isn’t any support in 1.4.2 for NIO SSL sockets. Whoopsie. Which explains the requirement on Java 5. Tiger adds the SSLEngine class which can be used to implement NIO SSL, with the caveat that it heightens complexity. I have only taken a cursory look at this yet. Right now I want the other stuff working first, since there are so many dependencies on them.

But it’s really going forward. Now, if I only had this as my day job, this would be finished in a few days… Alas, that’s not the way it is. Expect further updates in a week or two.

No Comments | By Ola Bini | In: Uncategorized | tags: bouncycastle, java, jruby, openssl, ruby, x509. | #

I just found out something very disturbing and annoying. Java Property-files (and ResourceBundles in extension) will only load files in ISO-8859-1 format. Of course, you can use Unicode escape codes in the files, but that is not very convenient. Java is, over all, really good at internationalization, but this is one sore spot. And it doesn’t seem to go away in Java 6 either. Isn’t it highly ironic that ResourceBundles can’t use regular UTF-8-files?

4 Comments | By Ola Bini | In: Uncategorized | tags: java, properties, resourcebundles, utf-8. | #

This weekend I started work on OpenSSL in JRuby. It’s a pretty big undertaking, and it’s actually worse than I suspected from the beginning, so I’m going to tell a bit about my endeavours and have far I’ve gotten right now. First of all, if someone is interested in the work, I have set up a branch for this, since it will take time and be very big. The branch is called svn://svn.codehaus.org/jruby/branches/openssl.

The approach
I have investigated several variants of implementing OpenSSL in JRuby. One that seemed easy was to find a JNI-library that wraps OpenSSL in Java. But it seems there are no full scale versions implemented anywhere. I also checked around other script-on-JVM-languages to see how they had solved it, but I didn’t find anything close to OpenSSL functionality. Which left me with the approach I’ve decided to try out: implement as OpenSSL compatible Java code as possible with JCE and JSSE. I’m convinced this is doable, but right now it feels quite hard.

The progress
So, how far have I gotten these last days? Pretty far, but not far enough. I’m basing the work on MRI’s test suite for OpenSSL. From those I have test_digest.rb and test_cipher.rb passing all tests. This doesn’t sound like much, but especially test_cipher was a pain to get running.

The plan from hereon is to get the utils.rb-file to load, which means implementing OpenSSL::PKey::RSA and OpenSSL::PKey::DSA, getting the basics of OpenSSL::X509 in place and also find a way to fix OpenSSL::ASN1. Oh well, I’ve got loads of time for this. Or not. =)

The problem
The real problem when implementing this, is the fact that Ruby’s OpenSSL support is… Well, how shall I put it? Thin, you might say. It’s basically a a wrapper around the C-library, which means that the disconnect when implementing this functionality with JCE is quite large. Just translating OpenSSL cipher names to the JCE equivalent is a challenge. But the big problem with the ciphers was initiating the key and IV (initialization vector). I have tried all the PBE solutions available, including the versions in BouncyCastle ending with “-OPENSSL”. No luck.

The problem is that Ruby uses the function called EVP_BytesToKey, which, according to the documentation implements PKCS#5 1.5 with a few tricks up its sleeve. Not very nice. In the end I had to implement my own version of this to generate keys. And since I had to look like mad for this information, I will here give you the implementation to this function in Java. Just use the return value to initialize your own SecretKey-implementation and instantiate an IvParameterSpec and you should be set to go: (note, I release this into the public domain. And note, this is just quick, ported code to show the concept.)

    public byte[][] EVP_BytesToKey(int key_len, int iv_len, MessageDigest md, byte[] salt, byte[] data, int count) {
       byte[][] both = new byte[2][];
       byte[] key = new byte[key_len];
       int key_ix = 0;
       byte[]  iv = new byte[iv_len];
       int iv_ix = 0;
       both[0] = key;
       both[1] = iv;
       byte[] md_buf = null;
       int nkey = key_len;
       int niv = iv_len;
       int i = 0;
       if(data == null) {
           return both;
       }
       int addmd = 0;
       for(;;) {
           md.reset();
           if(addmd++ > 0) {
               md.update(md_buf);
           }
           md.update(data);
           if(null != salt) {
               md.update(salt,0,8);
           }
           md_buf = md.digest();
           for(i=1;i<count;i++) {
               md.reset();
               md.update(md_buf);
               md_buf = md.digest();
           }
           i=0;
           if(nkey > 0) {
               for(;;) {
                   if(nkey == 0) break;
                   if(i == md_buf.length) break;
                   key[key_ix++] = md_buf[i];
                   nkey--;
                   i++;
               }
           }
           if(niv > 0 && i != md_buf.length) {
               for(;;) {
                   if(niv == 0) break;
                   if(i == md_buf.length) break;
                   iv[iv_ix++] = md_buf[i];
                   niv--;
                   i++;
               }
           }
           if(nkey == 0 && niv == 0) {
               break;
           }
       }
       for(i=0;i<md_buf.length;i++) {
           md_buf[i] = 0;
       }
       return both;
   }

No Comments | By Ola Bini | In: Uncategorized | tags: evp_bytestokey, java, jce, jruby, openssl. | #

The last few weeks have seen some discussion regarding refactoring tools for dynamic languages. The basic questions are if it is possible, and if so, how effective it would be. More information about the debate in question can be found here, here and here. I’m not really going into the fray here; I just wanted to provide my hypothesis on something tangential to the issue.

The interesting point is something Cedric said in his blog:

And without this, who wants an IDE that performs a correct refactoring “most of the time”?

The underlying assumption here is that there can actually exist a refactoring tool that works all the time. So, my question is this: “Can a Refactoring tool be 100% effective, (where effectiveness is defined as completely fulfilling the refactoring preconditions, postconditions and invariants and without introducing dangers or errors in the code).”

My hypothesis (and there will be no rigorous proof of my position) is based on one axiom:

Automated refactoring is a subset of the Church-Turing halting problem.

I have no direct proof for this position, but it seems intuitive to me, that for a refactoring to always be completely correct, you would need to know things about the program that isn’t always entirely possible to predict only from code. In this case, test runs would be necessary, and in those cases the halting problem enters. Now, for a strongly, statically typed language you would have to go to some effort to actually produce a program that couldn’t be safely refactored, but the possibility is still there.

One commenter on one of the blog entries above said that a precondition for 100% refactoring of Java would be that you didn’t use reflection and other meta-tricks. But the problem is, to avoid the halting problem you would have to remove enough features of Java to make it into a Turing-incomplete language. And by then it wouldn’t be usable for general purpose programming.

I see no way out of this dilemma, unless my axiom is wrong. But if it is correct, there can never ever be a 100% effective refactoring tool.

8 Comments | By Ola Bini | In: Uncategorized | tags: church, halting problem, java, refactoring, ruby, turing. | #

I’m very pleased to announce that JvYAML 0.2 was released a few minutes ago. The new release contains all the things I’ve talked about earlier and a few extra things I felt would fit good. The important parts of this release are:

• The Dumper – JvYAML is now a complete YAML processor, not just a loader.
• Loading and dumping JavaBeans – This feature is necessary for most serious usage of YAML. It allows people to read configuration files right into their bean objects.
• Loading and dumping specific implementations of mappings and sequences. Very nice if you happen to need your mapping to be a TreeMap instead of a HashMap.
• Configuration options to allow 1.0-compatibility with regard to the ! versus !! tag prefixes.
• The simplified interface have been substantially improved, adding several utility methods.
• Lots and lots of bug fixes.

So, as you can see, this release is really something. I am planning on spending a few nights this week integrating it with JRuby too. And soon after that we will be able to have YAML completely in Java-land. That is great news for performance. It also makes it easier to just have one YAML implementation to fix bugs in, instead of two.

A howto? Oh, you want a guide to the new features? Hmm. Well, OK, but it really isn’t much to show. How to dump and object and get the YAML string back:

 YAML.dump(obj);

or dump directly to a file:

 YAML.dump(obj,new FileWriter("/path/to/file.yaml"));

or dump with version 1.0 instead of 1.1:

 YAML.dump(obj, YAML.options().version("1.0"));

dumping a JavaBean:

 String beanString = YAML.dump(bean);

and loading it back again:

 YAML.load(beanString);

That’s more or less it. Nothing fancy. Of course, all the different parts underneath is still there, and you can provide your own implementation of YAMLFactory to add your own specific hacks. If you want to dump your object in a special way, you can implement the YAMLNodeCreator interface, and your own object will be in charge of creating the information that should be used to represent your object.

No Comments | By Ola Bini | In: Uncategorized | tags: java, jvyaml, yaml. | #