Charles, Tom and Nick to EngineYard – and the future of JRuby

Most people have already heard the news that Charles, Tom and Nick are going to Engine Yard to work on JRuby. I’ve been asked for my opinion by a few people, and I’ve also seen some common reactions that I would like to comment on. Of course I only speak for myself, not for Charles, Tom or Nick, and definitely not for neither Sun, Oracle or Engine Yard.

Lets get the congratulations in order first. This is great news for Charles, Tom and Nick, and I definitely wish them well with at their new work. I totally understand their move and would have done the same thing if I had been in the same situation.

This is also good news for the JRuby project. The main concern from Charles and company has been to ensure that the JRuby project doesn’t suffer – that has been the overriding concern in this decision. Of course, having Nick be able to work on JRuby proper will also be great. Another full time resource.

Now for some of the comments and worries. Tim Anderson writes in his blog about it: The problem with some of the conclusions in this blog, especially that Oracle should have done a better job at reassuring Charles & co about the future of JRuby, goes totally against what is even possible for a company in this situation to do. I’ve heard this comment from several different places, so let me make this very plain. It would have been grossly illegal for any representative from Oracle to give ANY indication to Charles, Tom or Nick about what their intention for JRuby was. It will continue to be this way until the buyout is done. For all we know, Charles, Tom and Nick might have asked any Oracle person they could find what would happen, but they wouldn’t have been able to get an answer they could rely on. That’s how these things work.

Seeing as that insecurity would be around for quite some time, and since this merger is pretty big, it was a reasonable doubt from the JRuby guys perspective that Oracle wouldn’t give any indication for quite some time. During that time the JRuby development would be in jeopardy. So they made a decision to ensure the safety of the project. (When I mean safety of the project, I of course mean continued full time resources for working on it). From this perspective they didn’t really have any choice. This is no indication whatsoever of anything else. It is no indication of Oracle’s future Java strategy, it is no indication of what will happen with languages on JVM in the future. It is just a rational decision based on what can be known right now.

Many from the Ruby and JRuby community has expressed concerns that Engine Yard is primarily a Rails company, and that Rails bugs will take priority over Java integration or other pieces of the JRuby story. This is simply not true. Read any interview with Charles or any of the official announcements. The JRuby focus from Engine Yard will definitely not have overriding Rails concerns.

Another worry I’ve heard is that Engine Yard now “owns” core developers for MRI, Rubinius and JRuby, and as such can use this power to control the future of Ruby. <insert evil laugh here>.

Yes. Engine Yard does have lots of power over the future of Ruby right now. Is that a bad thing? All the above projects are proper open source projects, and nothing EY can do will stop that. EY is a next generation company. They understand open source and they swear by it. Just look at how much internal infrastructure they have opened up and released for general consumption. There can be no doubt that EY believes in open source.

If you’re really worried though… This is your chance to influence things. Submit patches to MRI, Rubinius or JRuby. Contribute enough and you will become a core developer, and you will have as much power as Engine Yard or any of the other core developers. (Remember that only 3 of the 8ish JRuby core developers work for Engine Yard). Once again – if you’re worried, do something about it. Don’t spread FUD.

Personally, I think the future of Ruby is looking bright.

Can bytecodes perform well?

I really need to write an answer to a comment that appeared on my post about Rubinius by Kragen Sitaker. It has two points in it, and I really want to address one of them at some length. So here goes. The comment read like this:

“It is byte code based. This means it’s easier to handle performance.”

If by “handle” you mean “not have”.

“garbage collection/object memory can be switched out to use another algorithm”

In your dreams. I suspect you’ll find out that the architecture isn’t as clean as you think it is when you try to plug in some other existing garbage collector or object memory.

It sounds like a good project, but I don’t think its advantages are going to be so enormous as you think they are.

Woha. That’s hard to argue with. Isn’t it? Or maybe not. Let’s being with the second point. Is Rubinius architecture clean and decoupled enough to allow swapping of GC and object memory? My answer is a resounding yes. But of course you can go and look for yourself. Of course, it’s possible to make a system where you can decouple the GC. Sun’s standard JVM does this, for example. But the interesting point isn’t whether Rubinius’ hooks for this is clean enough, but if they are cleaner than MRI’s. If you have ever tried to switch out the GC in MRI, you know that Rubinius beats MRI hands down in this regard. If not, you can ask Laurent Sansonetti at Apple, who have actually done most of the work to switch out the GC in MRI if that was a fun experience.

Let’s see, what’s next? Oh yeah. Bytecodes are always slow. No argument here. C++ will always beat a bytecode based engine. And C will almost always beat C++. And assembler (in the right hands) will usually beat C. But wait… Isn’t Java bytecode based? And doesn’t Java 6 perform on the same level as C in many cases, and in some cases performing better than C? Well, yes, it does… And wasn’t Smalltalk always based on bytecodes? Most Smalltalk engines performed very well. Why is MRI switching to bytecodes for 1.9? And why has Python always been bytecode based? And why is the CLR bytecode based? Why was even Pascal using bytecodes back in the day? (OK, that is cheating… Pascal used bytecodes for portability, not performance, but it still worked well in that aspect too). Erlang is bytecode based.

Basically, there are some cases where a static language will benefit from straight compilation down to hardware machine codes. OCaML is a typical example of such a language. Due to the extremely stringent type requirements of the language, the emitted code is usually faster than C. But that is the exception, and only works for bondage-tightly typed languages. When talking dynamic languages, bytecodes is the real path to good performance. Granted, a naive implementation of a bytecode engine will not perform well. But that is true for a compiler too. The difference is that the part interpreting bytecodes is usually a quite small part of the whole runtime system, and it can be switched out for better performance, piecemal or all together.

There are other reasons. For example, statically typed languages like Java and the CLR family of languages use bytecodes because it gives the runtime system the opportunity to dynamically change the machine code running based on statistics and criteria found out during runtime. This means that your application will actually have better performance in the parts where it counts, and the parts that are not heavily used will not be optimized. (That’s what HotSpot does, for example). This is not possible in a clean compilation to machine code. Java would never have had the performance it has now if it weren’t for the bytecodes.

So please, stop spreading this myth. It is NOT true and it has NEVER been true.

Rubinius is important

I predict that parts of this blog posts will make certain people uncomfortable, annoyed and possibly foamy mouthed. If you feel that you’re of this disposition, please don’t read any further.

As I’m working on JRuby, I obviously think that JRuby is the best solution for many problems I perceive in the MRI implementation currently. I have been quite careful to never say anything along the lines that JRuby is better than anything else, though. I will continue with that stance. However, I won’t restrict myself in the same way regarding Rubinius.

In fact, I’m getting more and more convinced that for the people that don’t need the things Java infrastructure can give you, Rubinius is the most important project around, in Ruby-land. More than that, Rubinius is MRI done right. If nothing substantial changes in the current timeline and plans for Ruby 1.9.1, I predict that Rubinius will be the CRuby implementation of choice within 6 months. Rubinius is an implementation done the way MRI should have been. Of course, Matz have always focused on the language, not the implementation. I’m very happy about that, since it means that we have an outstanding language.

But still. Rubinius will win over MRI and YARV. I’ve had this thought for a while, and I’m finally more or less convinced that it’s true. Of course, there are a few preconditions. The first and most important one is that Rubinius delivers 1.0 as planned, by the end of the year and that it doesn’t have abysmal performance. Or if YARV would happen to be totally finished and perfectly usable in the same time frame, things might take a different turn.

Why is Rubinius so good, compared to the existing C implementations? There are a number of good reasons for this:

  • It is byte code based. This means it’s easier to handle performance.
  • It has a pluggable, very clean architecture, meaning that for example garbage collection/object memory can be switched out to use another algorithm.
  • It is designed to be thread safe (though this is not really true yet), and Multi-VM capable.
  • It works with existing MRI extensions.
  • Most of the code is written in Ruby.
  • It gives you access to all the innards, directly from your Ruby code (stuff like MethodContexts/BlockContexts, etc).
  • The project uses Valgrind to ensure that the C code written is bullet proof.

Anyway. I put my money on Rubinius. Of course, that doesn’t mean I don’t think JRuby have a place to fill in the eco system. In fact, the real interesting question is what will happen when both Rubinius and JRuby have become more mature. I’d personally love to see more cooperation and sharing between the projects. Not a merging, since the goals are too separate, but it would be wonderful if JRuby could use the same Ruby code for all the primitive operations as Rubinius does.

Right now we have a simple Rubinius engine in JRuby, that can interpret and run some simpler byte codes from Rubinius.

JRuby and Rubinius are both extremely important. Right now I believe JRuby is more important, since it opens up a totally different market for Ruby, and gives the benefits of Java to Ruby. Rubinius has another place to fill.

Of course, being who I am, I have also looked into what would be required to port Rubinius to Java, using the same approach directly instead of going through JRuby. If you decide to use Java’s Garbage Collector, Java Threads, and reuse the JRuby parser you would end up with about 40 files of C code to port. Most of these are extremely easy, and none is really that hard. And what you would end up with is something that would run the same things Rubinius does, but with the possibility of invoking Java code at the same time. (Of course, I hope that Evan reserves a block of about 8-16 bytecodes that can be implementation dependent – these Jubinius would use to interop with Java code).

RailsConf Europe recap

I am finally back from a week of travel. Funny, it feels like much more than a week – but I guess that’s because there were some interesting mishaps with some of the flights.

Last Sunday I traveled to Berlin to attend RailsConf Europe. I arrived kind of late and was really tired and out of it during the Monday. I didn’t find most of the tutorials going on that enlightening – though it was fun to see David, Aslak and Dan present on RSpec together. I liked the mind map format they used instead of regular slides. Due to general tiredness and a really bad migraine I went to bed early. But before that I managed to see Dave Thomas introductionary keynote. It was interesting and extremely well presented; the theme was art and how it can help you as a programmer to use this metaphor to understand the things we do more closely. I didn’t find anything really new in the presentation and I’ve heard several people say that they would prefer if Dave had spent some time talking specifically about Rails instead. I tend to agree.

The second day of the conference was really good. DHH delivered a keynote that basically said that there is nothing really new happening with Rails. After that it was time for sessions. I guess none of the first presentations made any real impact on me, since I don’t remember what they were. After lunch I saw Nic Williams talk on meta programming; the talk was really fun. I didn’t learn anything new, but I had a fun time while doing it.

Charles and Tom did their JRuby presentation. And I also saw Evan Phoenix Rubinius talk. It was very interesting – if I could stand programming in such a low level language as C, I would probably spend more time helping them than I do now. I will write more about Rubinius – probably tomorrow – I haven’t really said everything I need to say here.

After the sessions there were more keynotes. Roy Fielding talked about REST. I gotta admit I didn’t really hear much of it though, since I was spending time fixing an annoying bug. After that Craig McClanahan talked for a few minutes about Rails. It was very enlightening to know how well regarded Rails is within Sun. Some people seems to have a different view on all this though, seeing conspiracy and dirty dealings in the way Sun is working Rails. I’ll need to cover that in a separate blog post too.

ThoughtWorks threw a party after the days session. That was very fun and well attended.

All the JRuby team attending (Charles, Nick, Tom and me) gathered together on the evening and did a JRuby Q&A BOF. Lots of people there, and a very free form of presentation made this one of the highlights for me. I had great fun and I hope the audience did too.

Finally the Wednesday… Cyndi Mithell from ThoughtWorks did a very nice keynote about why Ruby and Rails may be ready to cross the chasm and get a strong hold in the enterprise.

Koz and Marcel did a very good Rails Best Practices session. Down to earth, simple, totally useful advice on things to avoid in a Rails application and what to do instead.

And then it was time for my presentation about JRuby in ThoughtWorks. I think it went very well, but it became a little bit too corporate for my taste. I’ll need to make sure that doesn’t happen the next time. Maybe some more code in the presentation? =)

Most of the rest of the day was spent networking, hanging out in the exhibit hall and stuff like that. And then RailsConf was over.

My flight to Sweden from Berlin that evening didn’t really happen as I had expected it too. Instead I had to spend the night on a hotel in Frankfurt and take an early flight from there to Gothenburg.

Overall I had a very good time at RailsConf this year. It’s a worlds difference from RailsConf last year in London, which I felt was a real waste. This year the energy was high, much interesting things going on and lots of nice and smart people. Not as good as RailsConf in Portland earlier this year, but still very well worth attending. It seems that RailsConf has found a good balance in sessions. The only thing I can wish for would be more interesting choices for the tutorial day.

A JRuby Rubinius machine

When I get bored with JRuby, I tend to go looking either at other languages or other language implementations. This happened a few days ago, and the result is what I will here document. Begin by creating a file called fib.rb:

def fib(n)
if n < 2
fib(n - 2) + fib(n - 1)

p fib(15)

The next part requires that you have a recent version of Rubinius installed:

rbx compile fib.rb

This will generate fib.rbc. Next, take a recent JRuby version and run:

jruby -R fib.rbc

And presto, you should see 610 printed quite soon. This is JRuby executing Rubinius bytecode. I was quite happy about how it was to get this far with the functionality. Of course, JRuby doesn’t support most bytecodes yet, only those needed to execute this small example, and similar things. We are also using JRuby’s internals for this, which means that Rubinius MethodContext and such are not available.

Another interesting note is that running the iterative Fib algorithm like this with -J-server is actually 30% faster than MRI.

This approach is fun, and I have some other similar ideas I really want to look at. The best part about it though, is that I got the chance to look at the internals of Rubinius. I hope to have more time for it eventually. Another thing I really want to do some day is implement a Jubinius, which should be a full port of the Rubinius runtime, possibly excluding Subtend. I think it could be very nice to have the Smalltalk core of Rubinius working together with Java. Of course, I don’t have any time for that, so we’ll see what happens in a year or two. =) Maybe someone else does it.

Rubinius and Lisp

Yesterday on the #rubinius-channel, me, Evan and MenTaLguY had the beginnings of a discussion about a Lisp dialect for the Rubinius VM. Since this sound exactly like the kind of thing I’ve been thinking about for some time, I got excited. But anyway, Pat Eyler covered the issue much better and did a small interview with us that can be read here. Ruby is wonderful. Rubinius is sweet. With Lisp added to the mix, it will be better.

I’ll probably follow this up with some more information and ideas, as soon it’s fleshed out.