Ruby Design meeting


Yesterday marked the first of the Ruby design meetings, where most of the Ruby implementers got together on IRC and started hashing out the solutions to several current concerns, and worked on getting more cooperation in the design of Ruby features.

This is slated to become a weekly meeting, and it’s a huge deal. This will make the lives of all Ruby implementations much easier, and the meeting yesterday actually accomplished some very nice things.

There were representatives from JRuby, Rubinius and macruby present, and of course also Matz, Koichi, Nobu and Tanaka from the Ruby core team.

Some of the highlights was a decision to start working on a common API for MultiVM, initial acceptance to add the RubySpecs (coming from Rubinius originally) into the 1.8 and 1.9 build process, meaning that regression testing will be much better from now on, and also having most implementations using the same specs for compliance testing. In reality, this takes us one step closer to a real executable specification that everyone agrees on and has official blessing from Matz.

In conjunction with this, a decision to set up continuous integration for 1.8 and 1.9 was made. The exact practicalities is still to be decided, but the decision to get it done is also very important.

All in all, these are excellent news, and I’m feeling extremely hopeful about more cooperation between the Ruby implementors.

If you’re interested in exactly what happened, you can find the agenda, action items and log here: http://ruby-design.pbwiki.com/Design20080421.



JtestR 0.2 released!


And so, JtestR 0.2 has finally been released. The highlights include support for Expectations and TestNG, RSpec stories and lots of other goodies.

Here is the release announcement:

JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.2 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
– Integrates with Ant and Maven
– Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
– Customizes Mocha so that mocking of any Java class is possible
– Background testing server for quick startup of tests
– Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

New and fixed in this release:
JTESTR-10 It should be possible to run TestNG tests
JTESTR-12 Buildr support
JTESTR-13 CC.rb should be able to run JtestR tests
JTESTR-17 Tests should be groupable and runnable per groups
JTESTR-21 Support RSpec stories
JTESTR-28 JtestR should include expectations
JTESTR-30 code coverage support
JTESTR-31 Autoloading of Java constants
JTESTR-32 Can’t load IA 32-bit .so on a IA 32-bit platform
JTESTR-33 JtestR should use latest version of JRuby
JTESTR-34 Errors when project is in path with spaces on Windows XP
JTESTR-37 Can’t expect a specific Java exception correctly
JTESTR-38 Problem with mocking Java classes
JTESTR-39 RSpec story runner seems to require rubygems
JTESTR-40 Package missing or.jruby.exceptions.RaiseException
JTESTR-43 It should be possible to get the generated mock class without instantiation
JTESTR-44 New output files start with a whitespace
JTESTR-45 RSpec raise_error and Test/Unit assert_raise and assert_nothing_raised handled JRuby NativeException stuff correctly.

Team:
Ola Bini – ola.bini@gmail.com
Anda Abramovici – anda.abramovici@gmail.com



Connecting languages (or polyglot programming example 1)


Today I spent some time connecting two languages that are finding themselves popular for solving wildly different kinds of problems. I decided I wanted to see how easy it was and if it was a workable solution if you would want to take advantage of the strengths of both languages. The result is really up to you. My 15 minutes experiment is what I’ll discuss here.

If you’d like, you can see this as a practical example of the sticky part where two languages meet, in language-oriented programming.

The languages under consideration is Ruby and Erlang. The prerequisite reading is this eminent article by my colleague Dennis Byrne: Integrating Java and Erlang.

The only important part is in fact the mathserver.erl code, which you can see here:

-module(mathserver).
-export([start/0, add/2]).

start() ->
Pid = spawn(fun() -> loop() end),
register(mathserver, Pid).

loop() ->
receive
{From, {add, First, Second}} ->
From ! {mathserver, First + Second},
loop()
end.

add(First, Second) ->
mathserver ! {self(), {add, First, Second}},
receive
{mathserver, Reply} -> Reply
end.

Follow Dennis’ instructions to compile this code and start the server in an Erlang console, and then leave it there.

Now, to use this service is really easy from Erlang. You can really just use the mathserver:add/2 operation directly or remotely. But doing it from another language, in this case Ruby is a little bit more complicated. I will make use of JRuby to solve the problem.

So, the client file for using this code will look like this:

require 'erlang'

Erlang::client("clientnode", "cookie") do |client_node|
server_node = Erlang::OtpPeer.new("servernode@127.0.0.1")
connection = client_node.connect(server_node)

connection.sendRPC("mathserver", "add", Erlang::list(Erlang::num(42), Erlang::num(1)))

sum = connection.receiveRPC

p sum.int_value
end

OK, I confess. There is no erlang.rb yet, so I made one. It includes some very small things that make the interfacing with erlang a bit easier. But it’s actually still quite straight forward what’s going on. We’re creating a named node with a specific cookie, connecting to the server node, and then using sendRPC and receiveRPC to do the actual operation. The missing code for the erlang.rb file should look something like this (I did the minimal amount here):

require 'java'
require '/opt/local/lib/erlang/lib/jinterface/priv/OtpErlang.jar'

module Erlang
import com.ericsson.otp.erlang.OtpSelf
import com.ericsson.otp.erlang.OtpPeer
import com.ericsson.otp.erlang.OtpErlangLong
import com.ericsson.otp.erlang.OtpErlangObject
import com.ericsson.otp.erlang.OtpErlangList
import com.ericsson.otp.erlang.OtpErlangTuple

class << self
def tuple(*args)
OtpErlangTuple.new(args.to_java(OtpErlangObject))
end

def list(*args)
OtpErlangList.new(args.to_java(OtpErlangObject))
end

def client(name, cookie)
yield OtpSelf.new(name, cookie)
end

def num(value)
OtpErlangLong.new(value)
end

def server(name, cookie)
server = OtpSelf.new(name, cookie)
server.publish_port

while true
yield server, server.accept
end
end
end
end

As you can see, this is regular simple code to interface with a Java library. Note that you need to find where JInterface is located in your Erlang installation and point to that (and if you’re on MacOS X, the JInterface that comes with ports doesn’t work. Download and build a new one instead).

There are many things I could have done to make the api MUCH easier to use. For example, I might add some methods to OtpErlangPid, so you could do something like:

pid << [:call, :mathserver, :add, [1, 2]]

where the left arrows sends a message after transforming the arguments.

In fact, it would be exceedingly simple to make the JInterface API downright nice to use, getting the goodies of Erlang while retaining the Ruby language. And oh yeah, this could work on MRI too. There is an equivalent C library for interacting with Erlang, and there could either be a native extension for doing this, or you could just wire it up with DL.

If you read the erlang.rb code carefully, you might have noticed that there are several methods not in use currently. Say, why are they there?

Well, it just so happens that we don’t actually have to use any Erlang code in this example at all. We could just use the Erlang runtime system as a large messaging bus (with fault tolerance and error handling and all that jazz of course). Which means we can create a server too:

require 'erlang'

Erlang::server("servernode", "cookie") do |server, connection|
terms = connection.receive
arguments = terms.element_at(1).element_at(3)
first = arguments.element_at(0)
second = arguments.element_at(1)

sum = first.long_value + second.long_value
connection.send(connection.peer.node, Erlang::tuple(server.pid, Erlang::num(sum)))
end

The way I created the server method, it will accept connections and invoke the block for every time it accepts a connection. This connection is yielded to the block together with the actual node object representing the server. The reason the terms are a little bit convoluted is because the sendRPC call actually adds some things that we can just ignore in this case. But if we wanted, we could check the first atoms and do different operations based on these.

You can run the above code in server, and use the exact same math code if you want. For ease of testing, switch the name to servernode2 in both server and client, and then run them. You have just sent Erlang messages from Ruby to Ruby, passing along Java on the way.

Getting different languages working together doesn’t need to be hard at all. In fact, it can be downright easy to switch to another language for a few operations that doesn’t suit the current language that well. Try it out. You might be surprised.



Pragmatic Static Typing


I have been involved in several discussions about programming languages lately and people have assumed that since I spend lots of time in the Ruby world and with dynamic languages in general I don’t like static typing. Many people in the dynamic language communities definitely expresses opinions that sound like they dislike static typing quite a lot.

I’m trying to not sound defensive here, but I feel the need to clarify my position on the whole discussion. Partly because I think that people are being extremely dogmatic and short-sighted by having an attitude like that.

Most of my time I spend coding in Java and in Ruby. My personal preference are to languages such as Common Lisp and Io, but there is no real chance to use them in my day-to-day work. Ruby neatly fits the purpose of a dynamic language that is close to Lisp for my taste. And I’m involved in JRuby because I believe that there is great worth in the Java platform, but also that many Java programmers would benefit from staying less in the Java language.

I have done my time with Haskell, ML, Scala and several other quite statically typed languages. In general, the fact that I don’t speak that much about those languages is that I have less exposure to them in my day-to-day life.

But this is the thing. I don’t dislike static typing. Absolutely not. It’s extremely useful. In many circumstances it gives me things that really can’t have in a dynamic language.

Interesting thought: Smalltalk is generally called a dynamic language with very late binding. There are no static type tags and no type inference happening. The only type checking that happens will happen at runtime. In this regard, Smalltalk is exactly like Ruby. The main difference is that when you’re working with Smalltalk, it is _always_ runtime. Because of the image based system, the type checking actually happens when you do the programming. There is no real difference between coding time and runtime. Interestingly, this means that Smalltalk tools and environments have most of the same features as a static programming language, while still being dynamic. So a runtime based image system in a dynamic, late bound programming language will actually give you many of the benefits of static typing at compile time.

So the main take away is that I really believe that static typing is extremely important. It’s very, very useful, but not in all circumstances. The fact that we reach for a statically typed programming language by default is something we really need to work with though, because it’s not always the right choice. I’m going to say this in even stronger words. In most cases a statically typed language is a premature optimization that gets extremely much in the way of productivity. That doesn’t mean you shouldn’t choose a statically typed language when it’s the best solution for the problem. But this should be a conscious choice and not by fiat, just because Java is one of the dominant languages right now. And if you need a statically typed language, make sure that you choose one that doesn’t revel in unnecessary type tags. (Java, C#, C++, I’m looking at you.) My current choice for a static language is Scala – it strikes a good balance in most cases.

A statically typed language with type inference will give you some of the same benefits as a good dynamic language, but definitely not all of them. In particular, you get different benefits and a larger degree of flexibility from a dynamic language that can’t be achieved in a static language. Neal Ford and others have been talking about the distinction between dynamic and static typing as being incorrect. The real question is between essence and ceremony. Java is a ceremonious language because it needs you to do several dances to the rain gods to declare even the simplest form of method. In an essential language you will say what you need to say, but nothing else. This is one of the reasons dynamic languages and type inferenced static languages sometimes look quite alike – it’s the absence of ceremony that people react to. That doesn’t mean any essential language can be replaced by another. And with regads to ceremony – don’t use a ceremonious language at all. Please. There is no reason and there are many alternatives that are better.

My three level architecture of layers should probably be updated to say that the stable layer should be an essential, statically typed language. The soft/dynamic layer should almost always be a strongly typed dynamic essential language, and the DSL layers stays the same.

Basically, I believe that you should be extremely pragmatic with regards to both static and dynamic typing. They are tools that solve different problems. But out industry today have a tendency to be very dogmatic about these issues, and that’s the real danger I think. I’m happy to see language-oriented programming and polyglot programming get more traction, because they improve a programmers pragmatic sensibilities.



JvYAMLb finally released as separate project


So I’ve finally made the time to extract JvYAMLb from JRuby. That means that JvYAML is mildly deprecated. Of course, since JvYAMLb only uses ByteLists it might not be the best solution for everyone.

If you’re interested in downloading the 0.1 release, you can do it at the Google Code download site http://code.google.com/p/jvyamlb/downloads/list.



Presentations this and next week


Wow. JRuby 1.1 is out! Go get! (And incidentally, wouldn’t JRuby be the perfect way for Google to support Ruby in Google Apps Engine?)

I’ll be doing a talk tonight at .NETAkademien about Ruby and Rails. It’s mostly an introduction.
You can find more info here.

On Thursday I’ll speak about JRuby at Developer Summit in Stockholm. I’ll be in the Dynamic Languages track and you can read the abstract and more information here.

And finally, next Monday I’ll talk about JRuby at the Stockholm Ruby User Group. You can find out more information about that event here.

Overall, it feels like the conference season is starting up again. I’ll be presenting at JavaOne in a month too.

I’ve been extremely quiet here lately. It’s been a bit slow in JRuby land, actually, and my focus has been elsewhere. Hopefully I’ll be able to start posting some really cool stuff soon – I have something up my sleeve that should be possible to talk about soon.



Meta-level thinking


I have been trying to figure out what characterizes some of the best programmers I know, or know about. It’s a frustrating endeavor of course, since most of these people are very different from each other, and have different experiences and ways to think and learn about stuff. Not to mention the fact that programmers tend to be highly individualistic.

But I think that I’m finally zeroing in on something that is general enough but also specific enough to categorize most of these people, and the general mind needed to be a really good programmer. In this blog post I’ll call that “meta-level thinking”, and I’ll explain more about what I mean as we go along.

When people try to become better programmers there are generally a few different kinds of advices you can get. The ones that seem prevalent right now is (among others):

  • Learn – and understand – Lisp
  • More generally, learn a new language that is sufficiently different from your current knowledge
  • Work with domain specific languages
  • Understand metaprogramming
  • Read up on the available data structures (Knuth anyone)?
  • Implement a compiler and runtime system for a language

Of course, these are only a few examples, and the categories are a bit fuzzy. These items are based on most of my experiences so they tend to be a bit language heavy. But I believe that what you practice by choosing any of these routes is an ability to abstract your thinking. In short, I believe that being able to abstract and understand what goes on in a programming language is one way to become more proficient in that language, but not only that – by changing your thinking to see this part of your environment you generally end up programming differently in all languages.

This is why Lisp has a tendency to change the way people write Java code. Lisp teaches you metaprogramming and DSL’s but they don’t really do it in a way that need words. DSLs and metaprogramming are just part of Lisp. It’s so much a part of the structure that you don’t see it. But when you turn to Java you’ll need to start working with these abstractions on a different level. Ruby programmers embrace metaprogramming and this changes the way these Ruby programmers think about things.

I’m really happy to see metaprogramming and DSLs getting more and more focus, because I really believe that understanding them is a really good way to make programmers better. Of course, you can get the same effect by writing a copmiler and runtime system – as Steve Yegge proposes. But I disagree that you really need that experience. There are other ways you can get the same way of looking at the world.

I call this meta-level thinking. I think it’s mostly a learned ability, but also that there is an aptitude component. Some of the best programmers I’ve met just have a mind that fits very well. It’s interesting to note that this kind of meta-level thinking is not an ability that is only applicable to programming. In fact, that’s probably just a matter of genetic luck that the same ability works very well for programming as for many other things. I think that there is a connection between certain abilities and a capacity for meta-level thinking too. Like music – it’s interesting to see how many programmers that have artistic leanings, and specifically towards music. It would be interesting to see some statistics about this.

In conclusion, this is my somewhat fuzzy view about what is one of the most important abilities that contributes to create brilliant programmers. If you have any interesting ideas about other ways you can reliably practice this ability, please improve on my list.



The contract of IO#read


It’s interesting. After Charlie made an immense effort and rewrote our IO system, basically from scratch, I have started to find bugs. But these are generally not bugs in the IO code, but bugs in Ruby libraries that depend on the way MRI usually works. One of the more annoying ones are IO#read(n), where n is the length you want to read.

This method is not guaranteed to return a string of length n, even if we haven’t hit EOF yet. You can NEVER be sure that what you get back is the length you requested. Ever. If you have code that doesn’t check the length of the returned string from read, you are almost guaranteed to have a bug just waiting to happen.

Of course, it might work perfectly on your machine and every other machine you test it on. The reason for this is that read(n) will usually return n bytes, but that depends on the socket implementation or file reading implementation of the operating system, it depends on the size of the cache in the network interface, it depends on network latency, and many other things. Please, just make sure to check the return values length before going ahead and using it.

Case in point: net/ldap has this exact problem. Look in net/ber.rb and you will see. There are also two – possibly three – bugs (couple of years old too) that reports different failures because of this.

One thing that makes this problem hard to find is the fact that if you insert debug statement, you will affect the timing in such a way that the code might actually work with debug statement but not without them.

Oh, did I mention that StringIO#read works the same way? It has exactly the same guarantees as IO#read.



After QCon London 2008


This week has mostly been taken up with QCon London. I spent most of Monday, Wednesday, Thursday and Friday here, and I thought that I’d take the opportunity to write up some of my impressions and thoughts about the sessions I attended.

First, in general the conference definitely didn’t disappoint me. It held at least as high standard as I had expected from earlier QCon and JAOO conferences. Solid speakers, a wide range of exciting topics and lots of interesting people made for a grade A conference.

I started out on the Monday with listening to my colleagues Neal Ford, Rebecca J Parsons and Martin Fowler give a tutorial on domain specific languages. I’ve seen bits and parts of this tutorial before, but seeing as the three speakers are working on evolving it to a full and coherent “pedagogical framework” for teaching DSLs, the current presentation had changed quite a bit since the last time. I really liked it and I recommend it to anyone interested in getting a firm grasp about what DSLs are. Having Rebecca talk about external DSLs in the context of parsers and grammars makes total sense, and the only real thing I would say was a problem was the time allotted to it. Her part of the subject was large enough that 75 minutes felt a bit rushed. Of course, I don’t see how Martins or Neals parts could be compressed much more either, so maybe the subject actually is too large for a one day tutorial? Anyway, great stuff.

For several reasons I decided to spend the Tuesday working from the office instead of attending tutorials again.

During the Wednesday I mostly spent my time in the exhibition hall, talking to people and doing general networking. For some reason the tracks I was least interested in had all been scheduled on the same day, so I was lazy and worked on other stuff in the ThoughtWorks booth.

The evening keynote on Wednesday by Martin Fowler and Jim Webber was hilarious, and also managed to get a quite important message across. I had a good time.

Thursday started the session attending for me, beginning with Markus Völters presentation of XText in the DSL track. Highly informative and something that I’ll keep in mind if I see something that would be benefit from it. The approach is definitely not for all problem domains, of course.

After that. Venkat Subramaniam gave a talk about how to blend dynamic languages with Java. This talk was useful in explaining why you’d want to do something like this, and why it’s such a powerful technique. It also served to set up my talk – which was next in that track – about evolving the Java platform. My talk went well, but I had the timing for it really messed up, so I ran out of material 10 minutes earlier than I expected. Neal Gafter was in the audience and helped out with some corrections during the talk. =)

Finally I headed back to the DSL track and saw Avi Bryant talk about DSLs in Smalltalk and then Magnus Christerson and Henk Kolk talk about the Intentional Workbench. Lots of neat stuff in both of these presentations.

Then there was the speakers dinner… Lots of interesting discussions with lovely people. =)

And then, more quickly than I had expected, the final day of QCon arrived. Me as a Ruby person and programming language nerd had quite a good selection of tracks. I ended up seeing Ted’s presentation on F#, which made me feel: wow! Microsoft took ML and did exactly what they’ve done to all languages on the CLR – added support for .NET objects in the mix. The talk ended with a quite strange discussion about whether F# actually helps with concurrent programming or not, and why a functional programming language has primitives that allow you to have mutable state.

After that I did my talk in the Ruby track, talking about more advanced things regard JRuby. It ended up being great fun, and I spent lots of time in the talk answering questions and showing how seamlessly things work with JRuby. I ended up eating up 10 minutes of everyone’s lunch time, but I had a great time and I thing most in the audience had too.

Feeling happy and finished with my contributions, I ended up in the Erlang talk by Joe Armstrong. It gave a quite good overview of why Erlang was created and how it solves some of the problems in that particular problem domain. There is no doubt that Armstrong is an entertaining talker, but his buffoon image gets a little tiring and repetitive after a while. Some of the things that interested me in the talk was missing too. He started out saying that Erlang solves a particular problem, but then expanded that into something that sounded like “Erlang should be used for everything, everywhere”. I tried to ask a question related to that, but the answer didn’t really go in the direction I was interested in.

I stayed in the languages track and saw the introduction to Scala, which is always fun, except that I’d already learned most of the things showcased. The most interesting about the presentation was the audience interest and questions.

Finally I realized that my contributions were not over at all, since I’d agreed to be part of the closing Ruby panel. This ended up revolving quite a lot around the question whether Ruby and Windows ever will be a good match, if this is important, and if we really want to push Ruby into all kinds of environments.

The closing panel were OK, but nothing special. It ended the day on a good note, but at that time I was tired enough to fall asleep in my chair. For some reason this always happens that last day of conferencing.

Anyway. I had a great time and I look forward to being back the next time. I can definitely recommend QCon as one of the best conferences around in this industry.



I’m at QCon


I’m attending QCon in London this whole week. I’ll give a talk on Thursday called “Evolving the Java platform”, talking about different things that might be a part of the next generation Java platform. On Friday I give the talk “JRuby: Power on the JVM”, which is a more advanced JRuby talk than the regular introduction talks I usually give.

Hope to see you around!