JRuby on Rails on Google App Engine


This is the third post in a series detailing information about the newly announced Google App Engine support for Java. In this post I thought I’d go through the steps you need to take to get a JRuby on Rails application working on GAE/J, and also what kind of characteristics you should expect from your application.

You need a fairly new copy of JRuby. Most of the changes needed to JRuby was added to JRuby trunk right after the JRuby 1.2 release, so check out and build something after that. The newest Rails version works fine too.

Once you have the basic Rails app set up, there are few things you need to do. First of them is to install Warble and pluginize it, and finally generate the Warble configuration file. You do that by doing “jruby -S gem install warble”, “jruby -S warble pluginize” and then “jruby -S warble config”. The last two should be done in the root of the Rails application.

You should freeze the Rails gems too. Once you have done that, you need to go through all the files there and remove anything that isn’t necessary. As it turns out, GAE/J has a hard limit on a 1000 files, and a typical Rails application will end up with much more files then that. You can remove all of ActiveRecord, all the test directories and so on.

Since you’re on GAE/J, you won’t need ActiveRecord, so you should not load it in config/environment.rb. The next step is to modify your warble.rb file. These are the things you need to do:

First, make sure that the needed GAE/J files are included, by doing:

config.includes = FileList[“appengine-web.xml”, “datastore-indexes.xml”]

You should also set the parameters for how many runtimes will be started:

config.webxml.jruby.min.runtimes = 1
config.webxml.jruby.max.runtimes = 1
config.webxml.jruby.init.serial = true

The last option is available in trunk version of JRuby-rack. If you don’t have min=1 and max=1 then you need this option set, because otherwise JRuby-rack will actually start several threads to initialize the runtimes.

Finally, to be able to use newer versions of the libraries, you need to set what Java libraries are used to the empty array:

config.java_libs = []

You will add all of the jar-files later, in the lib directory.

The last configuration option that I added is something to allow Rails to use DataStore as a session store. You can see how this is done in YARBL.

I have set several options in my appengine-web.xml file. The most important ones are to turn off JMX and to set os.arch to empty:

      <property name="jruby.management.enabled" value="false" />
      <property name="os.arch" value="" />

This is all pretty self explanatory.

One thing that I still haven’t gotten to work correctly is “protect_from_forgery”, so you need to comment this out in app/controllers/application.rb.

You need to put several jar-files in the lib-directory, and you actually need to split the jruby-complete jar, since it is too large for GAE/J in itself. The first jar-file is the appengine-api.jar file. You also need a late build of jruby-rack, and finally you need the different slices of the jruby-complete jar. I use a script like this to create several different jar-files:

#!/bin/sh

rm -rf jruby-core.jar
rm -rf ruby-stdlib.jar
rm -rf tmp_unpack
mkdir tmp_unpack
cd tmp_unpack
jar xf ../jruby-complete.jar
cd ..
mkdir jruby-core
mv tmp_unpack/org jruby-core/
mv tmp_unpack/com jruby-core/
mv tmp_unpack/jline jruby-core/
mv tmp_unpack/jay jruby-core/
mv tmp_unpack/jruby jruby-core/
cd jruby-core
jar cf ../jruby-core.jar .
cd ../tmp_unpack
jar cf ../ruby-stdlib.jar .
cd ..
rm -rf jruby-core
rm -rf tmp_unpack
rm -rf jruby-complete.jar

This creates two jar-files, jruby-core.jar and ruby-stdlib.jar.

These things should more or less put everything in order for you to be able to deploy your application to App Engine.

YARBL

As part of my evaluation of the infrastructure, I created a small application called YARBL. It allows you to have blogs, and post posts in them. No support for comments or anything fancy at all really. But it can be expanded into something real. I use both BeeU and Bumble in YARBL. BeeU allow me to make sure that only logged in users that are administrators can actually post things or change the blog. This support was extremely easy to add through the Google UserService.

You can see a (hopefully) running version at http://yarubyblog.appspot.com. You can find the source code in my GitHub repository: http://github.com/olabini/yarbl.

Bumble

Bumble is a very small wrapper around DataStore, that allow you to create data models backed by Google’s DataStore. It was developed to back YARBL, so it really only supports the things needed for that application.

This is what the data model for YARBL looks like. This should give you a feeling for how you define models with Bumble. One thing to remember is that the DataStore actually allows any properties/attributes on entitites, so it fits a language like Ruby very well.

class Person
  include Bumble

  ds :given_name, :sur_name, :email
  has_many :blogs, Blog, :owner_id
end

class Blog
  include Bumble

  ds :name, :owner_id, :created_at
  belongs_to :owner, Person
  has_many :posts, :Post, :blog_id, :iorder => :created_at
end

class Post
  include Bumble

  ds :title, :content, :created_at, :blog_id
  belongs_to :blog, Blog
end

To actually use the model for something, you can do things like these:

Blog.all

Post.all({}, :limit => 15, :iorder => :created_at)

blog = Blog.get(params[:id])
posts = blog.posts

Blog.create :name => name, :owner => @person, :created_at => Time.now

Post.all.each do |p|
  p.delete!
end

Here are most of the supported methods. The implementation is incredibly small and you really can’t go wrong with it. Of course, it is not tuned at all, so it does lots of fetches it could avoid. I’m happily accepting patches! The code can be found at http://github.com/olabini/bumble.

BeeU

When working with Google’s user service, you can use BeeU – a very small framework for helping with some things. You basically get a few different helper methods. There are three different filter methods that can be used. These are assign_user, assign_admin_status and verify_admin_user. The first two will create instance variables called @user and @admin respectively. The @user variable will contain the UserService User object, and @admin will be either true or false if the user is logged in and is an administrator or not. The last one will check that the current user is an administrator. If not logged in, it will redirect to a login page, and if logged in but not administrator, it will respond with a Not Authorized. These three methods should all be used as before filters.

There is a high level method called require_admin that you can use to point out what methods should be protected with admin access. This is really all you need.

Finally, there are two methods that generate a login-URL and a logout-URL, both of these will redirect back to where you were when the URL’s were generated.

BeeU can be found in my GitHub repository: http://github.com/olabini/beeu.

Summary

Overall, JRuby on Rails works very well on the App Engine, except for some smaller details. The major ones are the startup cost and testing. As it happens, you can’t actually get GAE/J to precreate things. Instead you’ll have to let the first release take the hit of this. Now, GAE/J does a let of preverifying of bytecodes and so on, so startup is a bit more heavy than on other JDKs. One runtime takes about 20 seconds wall time to startup, so the first hit takes some time. The good news is that this used to be worse. The last few weeks, the infrastructure has gotten a lot faster, and I’m confident this will continue to improve. It is still a problematic thing though, since you can’t precreate runtimes, which means that some request will end up taking quite a bit longer than expected.

It’s interesting to note that performance is actually pretty good once it gets running. I’ve seen between 120ms to 500ms for a request, depending on how much calls to DataStore is involved on the page – these times are not bad, considering what the infrastructure needs to do. It also seems mostly limited to the data access. If I’d had time to integrate memcaching, I could probably improve these times substantially.

The one remaining stickler for me is still testing. It’s not at all obvious how to do it, and as I noted in my earlier post there are some ways around it – but they don’t really fit in the way most Rails applications are built. In fact, I have done mostly manual testing on this application, since the cost of automating it seemed to be costly.

In all, Google App Engine with JRuby on Rails, is a really compelling combination of technology. I’m looking forward to the first ThoughtWorks project with these pieces.


55 Comments, Comment or Ping

  1. Have you tried JAR-ing your gems to avoid hitting the 1000 file limit?

    More details here:
    http://blog.nicksieger.com/articles/2009/01/10/jruby-1-1-6-gems-in-a-jar

    April 8th, 2009

  2. Would Bumble be nice as DataMapper source? This way you could easily switch between datasources and so on. Just a thought…

    April 8th, 2009

  3. Iain:

    Well, the good thing is that Bumble is small enough to throw away entirely. I know wycats was thinking about whipping up a real DM adapter for GAE though.

    April 8th, 2009

  4. Michael:

    No, I avoided trying that since I was a bit scared to introduce another shaky piece. I wrote much of the low-level support for that in JRuby, and I know it can be tricky. Especially since Rails likes to do lots of file-based tricks. That said, it should probably work fine.

    April 8th, 2009

  5. Damian

    “jruby -S gem install warble”

    Should that be

    “jruby -S gem install warbler” ?

    April 8th, 2009

  6. Pygy

    Couldn’t the full java SQLiteJDBC driver work with App Engine?
    Not for a big app, obviously, though (-;

    April 8th, 2009

  7. Sam

    Hi,
    nice write up on how to get a rails app deployed.
    I wrote a little tutorial bringing up a sinatra app and levarage some of your scripts.

    You can find it here. http://blog.bigcurl.de/

    Thanks
    Sam

    April 8th, 2009

  8. Pygy

    Replying to my own post: actually, it won’t be possible, since filesystem writes are forbidden.

    I thought I had found a free Camping hosting service…

    April 8th, 2009

  9. Thanks very much for this. It gave me a jump on getting Ramaze running on GAE as well, which, thanks to help from manveru and #jruby, works quite well.

    I plan on writing that up in the next day or so.

    April 9th, 2009

  10. Mahadev Azoba

    Man. You’re brainy!

    Luckily, for the rest of us, we have Azure :-)

    April 9th, 2009

  11. Daniel

    What will happen if I user gems written in C, such as hpricot?

    I recomend your article, in the part you talk about freezing the gems, advising to take care with this.

    April 9th, 2009

  12. Thank you for your great post. I’ve translated this post into Japanese.
    http://d.hatena.ne.jp/technohippy/20090408#1239196300
    Thanks again.

    April 10th, 2009

  13. Charlie Melbye

    This is pretty cool, I’m glad that Ruby is finally able to run on Google App Engine.

    Just so you know, your demo at http://yarubyblog.appspot.com/ is currently down with 500 Internal Server Errors for me.

    April 12th, 2009

  14. Great work Ola. Just an FYI. I think the first command you have in one of the beginning paragraphs to install warbler,

    “jruby -S gem install warble”

    should instead be

    “jruby -S gem install warbler”

    Excited to start my first JRuby app on GAE!

    April 17th, 2009

  15. Is it possible to load the App Engine (Datastore, Mail APIs, etc) environment in Jirb?

    April 22nd, 2009

  16. Regarding my previous comment, Ivan Schneider came up with a solution to loading the GAE environment manually: http://dev.massivebraingames.com/past/2009/4/17/unit_tests_on_google_app/

    April 25th, 2009

  17. I had a lot of trouble getting gem install warble to work. Maybe you meant gem install warbler?

    May 7th, 2009

  18. Jon

    Thanks for the great article. One thing I noticed is that I can’t seem to get my rails app to use map.route in routes.rb. From looking around and reading some docs I think this has something to do with Rack. I have very little familiarity with java web apps so sorry for the noob question. Maybe you can just point me in the right direction?

    May 17th, 2009

  19. Peter Moran

    Hi Ola. I’ve modified BeeU to work with my GAE Merb application. Would this be of any interest?

    July 9th, 2009

  20. Hi,

    I have JRuby + Sinatra on GAE up and running without problems.

    My question is, it is possible to use some kind of “autoreload” like shotgun? It’s too bad restarting dev_appserver.rb everytime I want to see my changes.

    Thanks in advance.

    Regards
    Francisco

    September 26th, 2009

  21. I am new to GAE sorry if it is silly question,
    is it possible to deploy directly my existing application in GAE, Or we need to start from scratch
    thanks in advance.

    January 18th, 2010

  22. Brilliant post, so many things I don’t know! Thanks for compiling and sharing.

    April 14th, 2010

  23. I avoided trying that since I was a bit scared to introduce another shaky piece. I wrote much of the low-level support for that in JRuby, and I know it can be tricky. Especially since Rails likes to do lots of file-based tricks. That said, it should probably work fine. Yes I totatly agree with this.

    April 14th, 2010

  1. A long time ago… - September 6, 2009

Reply to “JRuby on Rails on Google App Engine”