JRuby 1.1.4 Released


It’s late and I don’t have time to write something witty about this. This is the release announcement:

JRuby 1.1.4 is the fourth point release of JRuby 1.1.  The fixes in this
release are primarily obvious compatibility problems and performance
enhancements.  Our goal is to put out point releases more frequently for
the next several months (about 3-4 weeks a release).  We want a more
rapid release cycle to better address issues brought up by users of JRuby.

Highlights:

– Massive refactoring of Java integration layer
– 2-20x speed up of most features (calls, construction, arrays)
– Many long-standing Ruby/Java interaction bugs fixed
– Existing features made more consistent, reliable
– Closures can be passed as interface to static methods, constructors
– Java exceptions can be raised/rescued directly from Ruby
– Massive memory efficiency improvements (a lot less GC)
– Beginning of Ruby 1.9 support (enabled with –1.9 flag)
– Native complex/rational
– Additional efficiency, performance work in the interpreter
– Memory leak under –manage repaired
– FFI subsystem for calling C libraries
– syslog module from Rubinius is working and included
– win32 API support started
– Thread pooling improved (at least one production user now)
– Array concurrent-access improvements
– 72 issues resolved since JRuby 1.1.3

Issues fixed:
JRUBY-231        Provide attr_reader, attr_writer, and attr_accessor for JavaBean style getters & setters
JRUBY-1183     New closure conversion should prefer methods with convertable args over those without
JRUBY-1300     Masquerading of native Java exceptions
JRUBY-1326     Error invoking overloaded Java constructor
JRUBY-1562     Declaration of certain method name (setJavaObject(Xxx x)) will throw an exception using BSF
JRUBY-1615     Raising java exceptions from ruby causes TypeError
JRUBY-1707     Unable to raise Java exceptions of derived types
JRUBY-1735     Java Integration wraps to much
JRUBY-1839     closure conversion fails for blocks
JRUBY-1964     Determine what test/specs are needed to be written in order to refactor java integration post 1.1
JRUBY-1976     Working with JavaMethods doesn’t work.
JRUBY-2136     $VERBOSE = true; require ‘tmpdir’ gives non-fatal Java exception
JRUBY-2192     YAML parser does not appear to deserialize object types.
JRUBY-2204     Syslog module is not available for JRuby
JRUBY-2236     NPE in isDuckTypeConvertible
JRUBY-2287     Storing ruby objects in java classes instances
JRUBY-2377     Wrong line numbers for ArgumentError for Java calls
JRUBY-2429     Cannot Catch Core Java Exceptions From JRuby Internals in Ruby Code
JRUBY-2439     Trying to subclass a Java class from a signed .jar will crash on you.
JRUBY-2449     Implement closure convention for static java methods
JRUBY-2561     JavaField.set_value(foo, nil) breaks
JRUBY-2673     Java exceptions do not return the wrapped exception when getStackTrace is called
JRUBY-2680     When JIT Compiler compiles the append_features in the ruby\site_ruby\1.8\builtin\javasupport\proxy\interface.rb the compiled code slows down by a factor of 10
JRUBY-2741     OSGify jruby.jar in the release jruby distribution
JRUBY-2749     Make RaiseException show the exception message and the Ruby stack trace
JRUBY-2803     Bad performance calling Java classes
JRUBY-2823     Can’t reference Java’s constants that start with a lower case character
JRUBY-2828     Rational#% differs from MRI when argument is negative
JRUBY-2843     Issues with BasicSocket#close_read
JRUBY-2847     A non-existant jar + dir on the load path causes require to error
JRUBY-2850     In some cases, reopened Java objects cannot find methods on Ruby objects subclassed from Java
JRUBY-2854     AST offset error for StrNode and DStrNode
JRUBY-2857     Coercion error with public member variables
JRUBY-2863     Nested Interfaces can’t find the correct method when Java calls Ruby
JRUBY-2865     Can’t extend a class in default package
JRUBY-2867     Wrong overloaded Java method called when both int and float signatures exist
JRUBY-2869     IO.select fails to block with nil timeout
JRUBY-2870     [REGRESSION] Converting a Ruby array to a Java array (of Object references) broken
JRUBY-2872     JSpinner cannot accept Fixnum for it’s value
JRUBY-2873     FFI needs a way to specify call convention
JRUBY-2874     TCPSocket#new and TCPServer#new crash JRuby when the specified port is out of range (negative or bigger than 65k)
JRUBY-2879     net/ftp library is broken if mathn is also loaded
JRUBY-2880     Regression: 17 new RubySpec failures and 3 unit tests falirues caused by r7327 (Array changes)
JRUBY-2881     JAVA_HOME with () breaks JRuby on windows
JRUBY-2882     Incorrect subclass for constructor arg throws internal JRuby error
JRUBY-2886     Extending a final Java class should be rescuable as a normal Ruby exception type
JRUBY-2890     UDPSocket.recvfrom should block until something is available
JRUBY-2891     UDPSocket.bind throws a Java Error when already bound on Java 5
JRUBY-2892     JRuby releases use random copies of joni svn HEAD
JRUBY-2893     mspec runs need to pass properties through -T argument for compilation, etc
JRUBY-2894        When spec runs fail, Ant is not terminating with a failure message
JRUBY-2899     Using JavaEmbedUtils.rubyToJava causes problems when passing RubyObject-derived parameters back in to Ruby code
JRUBY-2903     Allow implementing Java interfaces with underscored method names
JRUBY-2905     NoMethodError does not give a useful message when thrown in BSF
JRUBY-2906     IOError message is garbled when java.io.IOException message is multi-byte character.
JRUBY-2907     method_missing invocation paths end up boxing arguments twice, among other inefficiencies
JRUBY-2910     Object#send is not specific-arity
JRUBY-2915     Exception construction performance is poor
JRUBY-2918     jruby 1.1.3 + activescaffold 1.1.1 generating RJS error
JRUBY-2919     Time.-(Time) does not include microseconds and is off by 10
JRUBY-2923     Eliminate (unknown) from trace elements
JRUBY-2924     JMX support added leaks memory like crazy
JRUBY-2927     Calling interface method on specific instance from Java doesn’t work.
JRUBY-2928     Same issue with hashCode and toString for Interfaces.
JRUBY-2929     Java Integration with regards to arrays of classes are broken
JRUBY-2931     Templater error causes merb-gen (0.9.4 and trunk) to fail on JRuby 1.1.3
JRUBY-2932     Move static soft reference timezone cache to be runtime-specific cache to remove complexity of dealing with soft references
JRUBY-2938     Calling JavaUtil.convertJavaToUsableRubyObject throws AssertionError
JRUBY-2943     Memory leak in closure coercion
JRUBY-2944     Java caller gets null when calling a method on a Ruby object implementing an interface method declared to return Object[], and the Ruby object returns an array of Ruby objects subclass of Hash converted with #to_java
JRUBY-2946     New invokers attempt to access argument list of non-overloaded methods with incorrect arity
JRUBY-2947     Multidimensional array conversion broke in recent Java integration refactoring

Not too bad, is it? Now go find out more at http://www.jruby.org, and download at http://dist.codehaus.org/jruby/



Stu’s Java.next series


If you haven’t already seen this, let me totally recommend Stuart Halloway’s series on the languages he call Java.next. The series look at several different aspects of these languages (Groovy, Scala, Clojure, JRuby), contrast them with each other and Java. Highly recommended if you are in any way interested in the languages that will soon replace Java for much application development.

The three published parts are:

  1. Java.next: Common Ground
  2. Java.next #2: Java Interop
  3. Java.next #3: Dispatch


JtestR 0.3.1 Released


JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.3.1 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
– Integrates with Ant, Maven and JUnit
– Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
– Customizes Mocha so that mocking of any Java class is possible
– Background testing server for quick startup of tests
– Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

New in the 0.3.1 release is upgrade of JRuby to revision r7479 which includes several new Java Integration features, upgrading of ActiveSupport to 2.1.0, fixing a severe memory leak in the background server and some minor usability features.

New and fixed in this release:
JTESTR-50 Difference in functionality when stubbing a method on a Java class vs a Ruby class using mocha
JTESTR-51 Mocking of classes lacking default constructors results in a NameError
JTESTR-53 Push the JtestR JRuby builds to maven repos
JTESTR-56 Upgrade ActiveSupport
JTESTR-57 Make it possible to use local versions of libraries.
JTESTR-59 No output when no tests found.
JTESTR-60 OutOfMemoryError
JTESTR-61 Documentation improvments – ant test-server
JTESTR-62 Having the jtestr.jar in the base directory doesn’t work
JTESTR-63 Update JRuby version



Where is the Net::SSH bug


Yesterday I spent several hours trying to find the problem with our implementation of OpenSSL Cipher, that caused the Net::SSH gem to fail miserable during negotiation and password verification. After various false leads I finally found the reason for the strange behavior. But I really can’t decide if it’s a bug, and if it’s a bug where the bug is. Is it in Ruby’s interface to OpenSSL, or is it in Net::SSH?

No matter what cipher suite you use for SSH, you generally end up using a block cipher, mostly something like CBC. That means an IV (initialization vector) is needed, together with a key. The relevant parts of OpenSSL used is the EVP_CipherInit, EVP_CipherUpdate and EVP_CipherFinal family of methods. Nothing really strange there. The Ruby interface matches these methods quite closely; every time you set a key, or an IV, or some other parameter, the CipherInit method is called with the relevant data. When CipherUpdate is called, the actual enciphering or deciphering starts happening, and CipherFinal takes care of the final block.

At the point EVP_CipherFinal is called, nothing more should be done using the specific Cipher context. Specifically, no more Update operations should be used. The man page has this to say about the Final-methods:

After this function is called the encryption operation is finished and no further calls to EVP_EncryptUpdate() should be made.

Now, what I found was that same documentation is not part of the Ruby interface. And Net::SSH is actually reusing the same Cipher object after final has been called on it. Specifically, it continues the conversation, calling update a few times and then final. The general flow for a specific Cipher object in Net::SSH is basically init->update->update->final->update->update->final.

So what is so bad about this then? Well, the question is really this: what IV will the operations after the first final call be using? The assumption I made is that obviously it will use the original IV set on the object. Something else would seem absurd. But indeed, the IV used is actually the last IV-length bytes of encrypted data returned. Is this an obvious or intended effect at some level? Probably not, since the OpenSSL documentation says you shouldn’t do it. The reason it works that way is because the temporary buffer used in the Cipher context isn’t cleared out at the end of the call to final.

In contrast, the Java Cipher object will call reset() as part of the call to doFinal(). Where reset() will actually reset the internal buffers to use the original IV. So the solution is simple for encryption. Just save away 8 or 16 bytes of the last generated crypto text and set that manually as the IV after the call to doFinal. And what about decryption? Well, here the IV needs to be the last crypto text sent in for deciphering, not the result of the last operation.

So Net::SSH seems to work fine with JRuby now. I’m about to release a new version of JRuby-OpenSSL including these and many other things.

But the question remains. Is it a bug? If it is, is it in the Ruby OpenSSL integration, or in the Net::SSH usages of Ciphers? If it’s in the Net::SSH code, why does it actually work correctly when communicating with an SSH server? Or is this behavior of using the last crypto text as IV something documented in the SSH spec?

Enlightenment would be welcome.



JtestR, RubyGems, and external code


One question I’ve gotten a few times now that people are starting to use JtestR, is how to make it work with external libraries. This is actually two different questions, masquerading as one. The first one regard the libraries that are already included with JtestR, such as JRuby, RSpec or ActiveSupport. There is an open bug in JIRA for this, called JTESTR-57, but the reason I’ve been a bit hesitant to add this functionality until now, is because JtestR actually does some pretty hairy things in places. Especially the JRuby integration does ClassLoader magic that can potentially be quite version dependent. The RSpec and Mocha integration is the same. I don’t actually modify these libraries, but the code using them is a bit brittle at the moment. I’ve worked on fixing this by providing patches to the framework maintainers to include the hook functionality I need. This has worked with great success for both Expectations and RSpec.

That said, I will provide something that allows you to use local versions of these libraries, at your own risk. It will probably be part of 0.4, and if you’re interested JTESTR-57 is the one to follow.

The second problem is a bit more complicated. You will have seen this problem if you try to do “require ‘rubygems'”. JtestR does not include RubyGems. There are both tecnnical and non-technical reasons for this. Simply, the technical problem is that RubyGems is coded in such a way that it doesn’t interact well with loading things from JAR-packaged files. That means I can’t distribute the full JtestR in one JAR-file if I wanted RubyGems, and that’s just unacceptable. I need to be able to bundle everything in a way that makes it easy to use.

The non-technical reason is a bit more subtle. If RubyGems can be used in your tests, it encourages locally installed gems. It’s a bit less pain to do it that way initially, but remember that as soon as you check the tests in to version control (you are using version control, right?) it will break in unexpected ways if other persons using the code doesn’t have the same gems installed, with the same versions.

Luckily, it’s quite simple to work provide functionality to JtestR, even if no gems are used. The first step is to create a directory that contains all the third party code. I will call it test_lib and place it in the root of the project. After you have done that you must first unpack your gems:

mkdir test_lib
cd test_lib
jruby -S gem unpack activerecord

When you have the gems you want unpacked in this directory, you can add something like this to your jtestr_config.rb:

Dir["test_lib/*/lib"].each do |dir|
  $LOAD_PATH << dir
end

And finally you can load the libraries you need:

require 'active_record'


JtestR 0.3 Released


JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.3 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
– Integrates with Ant, Maven and JUnit
– Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
– Customizes Mocha so that mocking of any Java class is possible
– Background testing server for quick startup of tests
– Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

The 0.3 release has focused on stabilizing Maven support, and adding new capabilities for JUnit integration.

New and fixed in this release:
JTESTR-47 Maven with subprojects should work intuitively
JTESTR-42 Maven dependencies should be automatically picked up by the test run
JTESTR-41 Driver jtestr from junit
JTESTR-37 Can’t expect a specific Java exception correctly
JTESTR-36 IDE integration, possibility to run single tests
JTESTR-35 Support XML output of test reports

Team:
Ola Bini – ola.bini@gmail.com
Anda Abramovici – anda.abramovici@gmail.com



JRuby RailsConf hackfest next Thursday


LinkedIn, Joyent and Sun Microsystems is sponsoring a JRuby hackfest in conjunction with RailsConf. It will happen next Thursday from 6:30 PM in Portland, there will be some food and beer and so on. Oh, Charles, Nick, Tom and me will be there – bring your laptops and any and all questions/patches/bugs/ideas with regards to JRuby.

Read more in Charles blog, here: http://headius.blogspot.com/2008/05/jruby-pre-railsconf-hackfest-on.html

Remember to RSVP to Charles if you’re coming. Space is limited so RSVP as soon as possible.



JavaOne 2008, the other half


So, the Thursday got a late start. For some strange reason I didn’t feel motivated to go see the Intel General Session, so I showed up for Nick’s session about JRuby on Rails deployment instead. Nick did a good job of outlining both the problem and the solution, and I have to say that this presentation was a good end cap for the JRuby week at JavaOne.

I had planned to go to some sessions after that, but I ended up hacking on JRuby instead. An interesting parser bug reared its head.

So the next session I went to was the Filthy Rich Clients one. Quite entertaining, although my interest in Swing is not what it used to be.

The final session of the day was the BOF about writing great technology books. This proved highly enjoyable since joining Josh Bloch and Brian Goetz as panelists were Burt Bates and Kathy Sierra. They did a wonderful job talking about how to write books that captures the readers attention and how to correctly use the brains weaknesses against it. I am tempted to say that this was the best session of the whole JavaOne. Brilliant.

On the Friday I was up early and sat in on Goslings Toy Story. Always funny, and some cool things there. For a geek like me, the CERN stuff and jMars was especially cool.

I managed to see quite a lot of sessions during the rest of the day. More Effective Java was useful as always, the Maxine Virtual Machine looks really cool, Neal Ford did an excellent job of comparing JRuby and Groovy, highlighting both the differences and similarities between the two. Finally the Jython session talked about some of the implementation challenges we in the JRuby team have wrestled with too, implementing a highly dynamic language on top of the JVM.

All in all, this JavaOne definitely stood out as the first non-Java-language JavaOne for me. And I didn’t even attend a single one of the gazillions of JavaFX presentations. It was a good year in general, and specifically for dynamic languages.

Oh, and my book is the 5th bestseller in the conference bookstore. Yay!



JtestR 0.2 released!


And so, JtestR 0.2 has finally been released. The highlights include support for Expectations and TestNG, RSpec stories and lots of other goodies.

Here is the release announcement:

JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.2 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
– Integrates with Ant and Maven
– Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
– Customizes Mocha so that mocking of any Java class is possible
– Background testing server for quick startup of tests
– Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

New and fixed in this release:
JTESTR-10 It should be possible to run TestNG tests
JTESTR-12 Buildr support
JTESTR-13 CC.rb should be able to run JtestR tests
JTESTR-17 Tests should be groupable and runnable per groups
JTESTR-21 Support RSpec stories
JTESTR-28 JtestR should include expectations
JTESTR-30 code coverage support
JTESTR-31 Autoloading of Java constants
JTESTR-32 Can’t load IA 32-bit .so on a IA 32-bit platform
JTESTR-33 JtestR should use latest version of JRuby
JTESTR-34 Errors when project is in path with spaces on Windows XP
JTESTR-37 Can’t expect a specific Java exception correctly
JTESTR-38 Problem with mocking Java classes
JTESTR-39 RSpec story runner seems to require rubygems
JTESTR-40 Package missing or.jruby.exceptions.RaiseException
JTESTR-43 It should be possible to get the generated mock class without instantiation
JTESTR-44 New output files start with a whitespace
JTESTR-45 RSpec raise_error and Test/Unit assert_raise and assert_nothing_raised handled JRuby NativeException stuff correctly.

Team:
Ola Bini – ola.bini@gmail.com
Anda Abramovici – anda.abramovici@gmail.com



Connecting languages (or polyglot programming example 1)


Today I spent some time connecting two languages that are finding themselves popular for solving wildly different kinds of problems. I decided I wanted to see how easy it was and if it was a workable solution if you would want to take advantage of the strengths of both languages. The result is really up to you. My 15 minutes experiment is what I’ll discuss here.

If you’d like, you can see this as a practical example of the sticky part where two languages meet, in language-oriented programming.

The languages under consideration is Ruby and Erlang. The prerequisite reading is this eminent article by my colleague Dennis Byrne: Integrating Java and Erlang.

The only important part is in fact the mathserver.erl code, which you can see here:

-module(mathserver).
-export([start/0, add/2]).

start() ->
Pid = spawn(fun() -> loop() end),
register(mathserver, Pid).

loop() ->
receive
{From, {add, First, Second}} ->
From ! {mathserver, First + Second},
loop()
end.

add(First, Second) ->
mathserver ! {self(), {add, First, Second}},
receive
{mathserver, Reply} -> Reply
end.

Follow Dennis’ instructions to compile this code and start the server in an Erlang console, and then leave it there.

Now, to use this service is really easy from Erlang. You can really just use the mathserver:add/2 operation directly or remotely. But doing it from another language, in this case Ruby is a little bit more complicated. I will make use of JRuby to solve the problem.

So, the client file for using this code will look like this:

require 'erlang'

Erlang::client("clientnode", "cookie") do |client_node|
server_node = Erlang::OtpPeer.new("servernode@127.0.0.1")
connection = client_node.connect(server_node)

connection.sendRPC("mathserver", "add", Erlang::list(Erlang::num(42), Erlang::num(1)))

sum = connection.receiveRPC

p sum.int_value
end

OK, I confess. There is no erlang.rb yet, so I made one. It includes some very small things that make the interfacing with erlang a bit easier. But it’s actually still quite straight forward what’s going on. We’re creating a named node with a specific cookie, connecting to the server node, and then using sendRPC and receiveRPC to do the actual operation. The missing code for the erlang.rb file should look something like this (I did the minimal amount here):

require 'java'
require '/opt/local/lib/erlang/lib/jinterface/priv/OtpErlang.jar'

module Erlang
import com.ericsson.otp.erlang.OtpSelf
import com.ericsson.otp.erlang.OtpPeer
import com.ericsson.otp.erlang.OtpErlangLong
import com.ericsson.otp.erlang.OtpErlangObject
import com.ericsson.otp.erlang.OtpErlangList
import com.ericsson.otp.erlang.OtpErlangTuple

class << self
def tuple(*args)
OtpErlangTuple.new(args.to_java(OtpErlangObject))
end

def list(*args)
OtpErlangList.new(args.to_java(OtpErlangObject))
end

def client(name, cookie)
yield OtpSelf.new(name, cookie)
end

def num(value)
OtpErlangLong.new(value)
end

def server(name, cookie)
server = OtpSelf.new(name, cookie)
server.publish_port

while true
yield server, server.accept
end
end
end
end

As you can see, this is regular simple code to interface with a Java library. Note that you need to find where JInterface is located in your Erlang installation and point to that (and if you’re on MacOS X, the JInterface that comes with ports doesn’t work. Download and build a new one instead).

There are many things I could have done to make the api MUCH easier to use. For example, I might add some methods to OtpErlangPid, so you could do something like:

pid << [:call, :mathserver, :add, [1, 2]]

where the left arrows sends a message after transforming the arguments.

In fact, it would be exceedingly simple to make the JInterface API downright nice to use, getting the goodies of Erlang while retaining the Ruby language. And oh yeah, this could work on MRI too. There is an equivalent C library for interacting with Erlang, and there could either be a native extension for doing this, or you could just wire it up with DL.

If you read the erlang.rb code carefully, you might have noticed that there are several methods not in use currently. Say, why are they there?

Well, it just so happens that we don’t actually have to use any Erlang code in this example at all. We could just use the Erlang runtime system as a large messaging bus (with fault tolerance and error handling and all that jazz of course). Which means we can create a server too:

require 'erlang'

Erlang::server("servernode", "cookie") do |server, connection|
terms = connection.receive
arguments = terms.element_at(1).element_at(3)
first = arguments.element_at(0)
second = arguments.element_at(1)

sum = first.long_value + second.long_value
connection.send(connection.peer.node, Erlang::tuple(server.pid, Erlang::num(sum)))
end

The way I created the server method, it will accept connections and invoke the block for every time it accepts a connection. This connection is yielded to the block together with the actual node object representing the server. The reason the terms are a little bit convoluted is because the sendRPC call actually adds some things that we can just ignore in this case. But if we wanted, we could check the first atoms and do different operations based on these.

You can run the above code in server, and use the exact same math code if you want. For ease of testing, switch the name to servernode2 in both server and client, and then run them. You have just sent Erlang messages from Ruby to Ruby, passing along Java on the way.

Getting different languages working together doesn’t need to be hard at all. In fact, it can be downright easy to switch to another language for a few operations that doesn’t suit the current language that well. Try it out. You might be surprised.