Tomorrow (Tuesday) evening, at 6pm, I will give a presentation about Ioke, at the ThoughtWorks office in Bangalore. This is an open event. More information and registration here: http://www.thoughtworker.com/events/geeknight-ola-bini.
One of those interesting facts about evolution is that most of the species that ever existed are now extinct. There are good reasons for this. The average life time for a mammalian species is a few million years. There are currently about 5400 species of mammals alive right now, and the mammalian class has existed for about 300 million years. If the current number is representative, there has been about 810 000 different species of mammals during the history of that class. So less than 1% are alive today. And this is only talking about mammals.
Many programming languages have died out, but there are still several old programming languages making a living. Cobol is still the backbone of much infrastructure in the world – it is even evolving still (Cobol 2002 include support for object orientation). Making another analogy with evolution, Cobol would be the sharks of programming languages. They have been around for a long, long time – at least 400 million years in the case of sharks, and 50 for Cobol – and are still recognizable. Fortran is another language that has been used for a long time, and has evolved substantially.
But we also have newer languages. From the view of business and technology, Java and other C-based languages are way ahead. In the dynamic realm, we have Perl, Python and Ruby. All of them over 14 years old. And then we have the really new languages, like Scala and Clojure, who are rapidly gaining use.
Maybe it is a bad thing that these languages that we use day to day have been so long lived. I’m not saying they should die out totally – I know that will never happen. But it might have been better if we had spent energy on fixing the Java language, instead of creating more and more tools to fix deficiencies in the language. The good part about Java is the JVM, and that could have been at least as good now even if we had evolved the language more.
It’s obvious that humanity isn’t evolving like other animals. Natural selection is still happening, but the results are getting skewed by better medicine, social infrastructure and many other inventions. These are tools that make it better for us – with the side effect that natural evolution is changing course. Something similar seems to have happened with Java – we have created so many tools to fix Javas problems, that there isn’t enough pressure to fix the language. I tend to believe that this is a good strategy for humanity, but a bad strategy for language development.
I’m happy that more and more new languages are gaining play on the JVM, on LLVM, the CLR and Parrot. That’s great. But on the other hand I’m seeing threads on Ruby-talk asking whether Ruby can stay ahead. Why would we want that? Wouldn’t it be better to take the best pieces of Ruby, and build on that? In the keynote at the last RubyConf, Dave Thomas proposed that people fork the language. I think that would be a great thing if it started to happen more.
I guess I’m sounding a bit like “yeah, all the current effort is good, but it could be better!”. And it could be. I think the way most people look at languages are all wrong.
Michael Feathers wrote a post on Artima called Stunting a Framework. Maybe we should look at language development in that way? Is this another way to approach DSLs, or can we get more useful, smaller languages, by using such an approach? I think it would be very interesting, no matter what.
Languages should die and be replaced. There should be less cost involved in developing a language. Languages should work on common infrastructure that makes this easy – and we’re finally hitting that mark with machines such as the JVM, LLVM, CLR and Parrot.
So – you should go out and fork your favorite language. Or your least favorite language. Fix the things that you don’t like with it. Modify it for a specific domain. Remove some of the cruft. Create a new small language. It doesn’t have to be big. It doesn’t have to do everything – one of the really cool things about functional languages is that they are often have a very small core. Use a similar approach. Hack around. See what happens.
No existing language will die in a long time. But we need new ideas. New fresh ways of looking at things. And new languages that incorporate what other languages do right. That’s how we will finally get rid of the things we don’t like in our current languages.
You should be ready to abandon your favorite language for something better.
ThoughtWorks will give two free seminars at Berns, Tuesday April 21, 2009.
The morning seminar will be given by Marcus Ahnve, talking about the business value of lean and agile, and how it can serve as a competitive advantage – especially in difficult times.
The afternoon seminar will be given by Zach Exley, who will give a behind-the-scenes look at the technological side of the Barack Obama campaign.
ThoughtWorks founder and chairman Roy Singham will kick off both seminars by giving his views on why ha believes Scandinavia has a great future in the area of software development.
The events are open for anyone – and you can attend either one or both. For registration and more info: http://connect.thoughtworks.com/stockholmEvent/.
As mentioned in a post a few minute ago here, Google has released App Engine support for Java. This is obviously very cool – and I’ve spent a few weeks testing several things using it. It should come as no surprise that my main goal with this investigation has been to see how dynamic languages fit in with the Java story.
The good news are these: JRuby works very well on the infrastructure. I will spend some more time in another post detailing what you have to do to get a JRuby on Rails application working on Google App Engine. In this post I’ll talk a bit about the different kind of restrictions a language implementation will run into, and what needs fixing.
Several other people has been testing languages such as Groovy, Scala, Clojure and Jython. My own experiments have been focused on JRuby and Ioke. At the moment, Ioke still doesn’t run on GAE/J, but the issue is something I hope will be fixed soon.
When looking at GAE/J, it’s important to keep in mind the security restrictions that Google has been forced to implement, to make the Java implementation totally safe for them. This includes restrictions of many kinds, and some of them might come as a bit of a surprise in some cases. One of the larger things you will notice is that some classes aren’t available – and you will get a ClassNotFoundException if you try to use them from your application. Personally, I believe that using a SecurityException when trying to load these might have been better, but this fact remains: many classes you expect will not be there.
Among the classes that are there (and the important parts of the JDK are there) there are many that will give you different kinds of security related problems too. JRuby trunk has been fixed for all these issues, so it should work without modification.
GAE/J restricts quite a lot of what you can do with the file system. One of the things that surprised me was that calling methods like java.io.File#canRead on a restricted file might throw a SecurityException. Basically this means that all file access in an implementation need to wrap these calls in try-catch blocks.
In JRuby, I solved this by an approach that Ryan Brown gave us – creating a subclass of java.io.File that wraps all these method and return something reasonable. canRead should for example just return false if it gets a SecurityException.
It’s very hard to secure a thread scheduler – there are ways of screwing up things that are basically impossible to guard against. That means GAE/J does not support threads at all. You can’t create new ones, you can’t create new ThreadGroups or change most settings on these threads.
This is something that is less problematic for some languages, and more problematic for others. I know that Lift (in Scala) for example had some trouble, since it relied very heavily on actors, implemented using thread pools.
Java’s reflection capabilities are very powerful, and most of the reflection methods throw several kinds of exceptions. On GAE/J you will have to guard most reflective accesses against SecurityException too. One of the things that many dynamic languages do is to use “setAccessible” on all methods. This will fail on some methods that Google thinks you shouldn’t have access to. Several of the methods on Object are among these.
In some cases, the bytecode verifier is a bit stricter than for other JDKs. It’s important to try out many corners of the application and see that it works correctly. Of course, if your language generates code at runtime, this is even more important. The good news is that I haven’t seen any problems at all with JRuby. The bad news is that the parser for Ioke doesn’t even load (and that is static Java code). This seems like a small problem in the verifier where a stack height of 0 causes it to fail, so hopefully it will be shortly fixed.
One of the early problems for Clojure was some intricacies in the way GAE/J handles class loaders. One of these is that doing ClassLoader.getSystemClassLoader() caused a SecurityException.
It is not immediately obvious to me how you can test applications written for GAE/J in another language. The intuition is that you would be able to use the local development server to run tests, but many of the things above don’t work exactly the same in the local dev server, and some things are problematic locally but won’t cause any trouble on the server. One thing I’ve noticed is that JRuby doesn’t load correctly, because the dev server doesn’t actually load things from jar-files in such a way that JRuby can load several property files from the same jar-file. This issue doesn’t actually exist on the real servers.
You can use unit tests to test parts of your application, but you need to make sure to stub out all the calls to the Google APIs. This is actually kinda hard in Java, since one of the negative aspects about the GAE/J APIs is that they are built around singleton factories. It is very hard to inject new functionality there.
With JRuby, you can of course override these methods and unit test without running them. The main problem with this kind of unit testing is that it won’t give you any real security on the server – since you might still run into several kinds of security exceptions.
I ended up implementing a very small unit testing framework for sanity checking. This allows you to trigger a test run by going to a specific URL. Of course, this approach sucks.
At the end of the day, it seems the best kind of testing you can do is functional testing using something like Selenium or WebDriver. Or Twist. GAE/J allow you to have different versions of an application, so one way you could utilize this automatically is to allow your automatic test run deploy to a version called “test”, and then you can use a specific url to get the latest version. Say your app is deployed on “testgae”, you can allow your CI to test against “test.latest.testgae.appspot.com”, while the production environment is still running on “testgae.appspot.com”. It’s still not perfect, but it gives you some flexibility and a possibility to run continuous integration on the correct infrastructure.
This is the third post in a series detailing information about the newly announced Google App Engine support for Java. In this post I thought I’d go through the steps you need to take to get a JRuby on Rails application working on GAE/J, and also what kind of characteristics you should expect from your application.
You need a fairly new copy of JRuby. Most of the changes needed to JRuby was added to JRuby trunk right after the JRuby 1.2 release, so check out and build something after that. The newest Rails version works fine too.
Once you have the basic Rails app set up, there are few things you need to do. First of them is to install Warble and pluginize it, and finally generate the Warble configuration file. You do that by doing “jruby -S gem install warble”, “jruby -S warble pluginize” and then “jruby -S warble config”. The last two should be done in the root of the Rails application.
You should freeze the Rails gems too. Once you have done that, you need to go through all the files there and remove anything that isn’t necessary. As it turns out, GAE/J has a hard limit on a 1000 files, and a typical Rails application will end up with much more files then that. You can remove all of ActiveRecord, all the test directories and so on.
Since you’re on GAE/J, you won’t need ActiveRecord, so you should not load it in config/environment.rb. The next step is to modify your warble.rb file. These are the things you need to do:
First, make sure that the needed GAE/J files are included, by doing:
config.includes = FileList[“appengine-web.xml”, “datastore-indexes.xml”]
You should also set the parameters for how many runtimes will be started:
config.webxml.jruby.min.runtimes = 1
config.webxml.jruby.max.runtimes = 1
config.webxml.jruby.init.serial = true
The last option is available in trunk version of JRuby-rack. If you don’t have min=1 and max=1 then you need this option set, because otherwise JRuby-rack will actually start several threads to initialize the runtimes.
Finally, to be able to use newer versions of the libraries, you need to set what Java libraries are used to the empty array:
config.java_libs = []
You will add all of the jar-files later, in the lib directory.
The last configuration option that I added is something to allow Rails to use DataStore as a session store. You can see how this is done in YARBL.
I have set several options in my appengine-web.xml file. The most important ones are to turn off JMX and to set os.arch to empty:
<property name="jruby.management.enabled" value="false" />
<property name="os.arch" value="" />
This is all pretty self explanatory.
One thing that I still haven’t gotten to work correctly is “protect_from_forgery”, so you need to comment this out in app/controllers/application.rb.
You need to put several jar-files in the lib-directory, and you actually need to split the jruby-complete jar, since it is too large for GAE/J in itself. The first jar-file is the appengine-api.jar file. You also need a late build of jruby-rack, and finally you need the different slices of the jruby-complete jar. I use a script like this to create several different jar-files:
#!/bin/sh rm -rf jruby-core.jar rm -rf ruby-stdlib.jar rm -rf tmp_unpack mkdir tmp_unpack cd tmp_unpack jar xf ../jruby-complete.jar cd .. mkdir jruby-core mv tmp_unpack/org jruby-core/ mv tmp_unpack/com jruby-core/ mv tmp_unpack/jline jruby-core/ mv tmp_unpack/jay jruby-core/ mv tmp_unpack/jruby jruby-core/ cd jruby-core jar cf ../jruby-core.jar . cd ../tmp_unpack jar cf ../ruby-stdlib.jar . cd .. rm -rf jruby-core rm -rf tmp_unpack rm -rf jruby-complete.jar
This creates two jar-files, jruby-core.jar and ruby-stdlib.jar.
These things should more or less put everything in order for you to be able to deploy your application to App Engine.
As part of my evaluation of the infrastructure, I created a small application called YARBL. It allows you to have blogs, and post posts in them. No support for comments or anything fancy at all really. But it can be expanded into something real. I use both BeeU and Bumble in YARBL. BeeU allow me to make sure that only logged in users that are administrators can actually post things or change the blog. This support was extremely easy to add through the Google UserService.
You can see a (hopefully) running version at http://yarubyblog.appspot.com. You can find the source code in my GitHub repository: http://github.com/olabini/yarbl.
Bumble is a very small wrapper around DataStore, that allow you to create data models backed by Google’s DataStore. It was developed to back YARBL, so it really only supports the things needed for that application.
This is what the data model for YARBL looks like. This should give you a feeling for how you define models with Bumble. One thing to remember is that the DataStore actually allows any properties/attributes on entitites, so it fits a language like Ruby very well.
class Person include Bumble ds :given_name, :sur_name, :email has_many :blogs, Blog, :owner_id end class Blog include Bumble ds :name, :owner_id, :created_at belongs_to :owner, Person has_many :posts, :Post, :blog_id, :iorder => :created_at end class Post include Bumble ds :title, :content, :created_at, :blog_id belongs_to :blog, Blog end
To actually use the model for something, you can do things like these:
Blog.all Post.all({}, :limit => 15, :iorder => :created_at) blog = Blog.get(params[:id]) posts = blog.posts Blog.create :name => name, :owner => @person, :created_at => Time.now Post.all.each do |p| p.delete! end
Here are most of the supported methods. The implementation is incredibly small and you really can’t go wrong with it. Of course, it is not tuned at all, so it does lots of fetches it could avoid. I’m happily accepting patches! The code can be found at http://github.com/olabini/bumble.
When working with Google’s user service, you can use BeeU – a very small framework for helping with some things. You basically get a few different helper methods. There are three different filter methods that can be used. These are assign_user, assign_admin_status and verify_admin_user. The first two will create instance variables called @user and @admin respectively. The @user variable will contain the UserService User object, and @admin will be either true or false if the user is logged in and is an administrator or not. The last one will check that the current user is an administrator. If not logged in, it will redirect to a login page, and if logged in but not administrator, it will respond with a Not Authorized. These three methods should all be used as before filters.
There is a high level method called require_admin that you can use to point out what methods should be protected with admin access. This is really all you need.
Finally, there are two methods that generate a login-URL and a logout-URL, both of these will redirect back to where you were when the URL’s were generated.
BeeU can be found in my GitHub repository: http://github.com/olabini/beeu.
Overall, JRuby on Rails works very well on the App Engine, except for some smaller details. The major ones are the startup cost and testing. As it happens, you can’t actually get GAE/J to precreate things. Instead you’ll have to let the first release take the hit of this. Now, GAE/J does a let of preverifying of bytecodes and so on, so startup is a bit more heavy than on other JDKs. One runtime takes about 20 seconds wall time to startup, so the first hit takes some time. The good news is that this used to be worse. The last few weeks, the infrastructure has gotten a lot faster, and I’m confident this will continue to improve. It is still a problematic thing though, since you can’t precreate runtimes, which means that some request will end up taking quite a bit longer than expected.
It’s interesting to note that performance is actually pretty good once it gets running. I’ve seen between 120ms to 500ms for a request, depending on how much calls to DataStore is involved on the page – these times are not bad, considering what the infrastructure needs to do. It also seems mostly limited to the data access. If I’d had time to integrate memcaching, I could probably improve these times substantially.
The one remaining stickler for me is still testing. It’s not at all obvious how to do it, and as I noted in my earlier post there are some ways around it – but they don’t really fit in the way most Rails applications are built. In fact, I have done mostly manual testing on this application, since the cost of automating it seemed to be costly.
In all, Google App Engine with JRuby on Rails, is a really compelling combination of technology. I’m looking forward to the first ThoughtWorks project with these pieces.
About a year ago, Google released their first beta version of App Engine – it allowed deployment and hosting of web applications. These applications were restricted to the Python language. About 5 minutes ago, Google announced that they have released a Java version of App Engine.
I have been involved in this for a few weeks – since ThoughtWorks is a Google Enterprise Partner – and it’s been a very interesting time. This post and a few others will take a closer look at what I’ve been experimenting with.
First of all, GAE/J is not based on Dalvik, as far as I can tell. It is a full Java implementation, so you compile your applications locally, using any standard JDK and then upload them. Google recommends Java 6 for this, but Java 5 works too.
The actual interface to GAE/J uses the standard Java Servlet API, so if you have something that works with it, chances are you won’t have to do many changes to your application.
Google also gives access to several different APIs, including the User service, Memcache service, Mail service, URL fetching service, Image service and DataStore service. These all give access to different pieces of the Google machinery. For me, the most interesting parts were the User service, that makes it possible to use the regular Google authentication infrastructure, and the DataStore service that makes it a snap to use Googles data storage infrastructure. For regular Java applications, you can use either JDO or JPA to interact with the DataStore, but Google also gives access to the low level APIs too.
As part of the GAE/J release, you get access to a local development server. It tries to mimic the full environment as closely as possibly. For the specific type of application Google expects most people to write, it works very well – but if you go outside of this beaten path, many things get a bit shaky. I ended up not using it very much.
So, GAE/J is a very cool platform to target cloud applications to. Obviously Python is still a valid choice too, but the combination of apps built in Python and applications running on GAE/J seems like a very powerful choice.
ThoughtWorks has recently been spending much time in this area and we have gotten some good experience with it. We look forward to be able to work with applications for Google App Engine, written in Java, or any of the other languages supported. (If you follow todays blog posts, you will see that I’m not the only ThoughtWorker who has explored alternative languages on this platform).
My esteemed colleagues have also written up their experiences with the Java pieces of Google App Engine. You can read it here: http://paulhammant.com/blog/google-app-engine-for-java-with-rich-ruby-clients.html, http://elhumidor.blogspot.com/ and http://blog.sriramnarayan.com/.
A few days back I said something on Twitter that caused some discussion. I thought I’d spend some time explaining a bit more what I meant here. The originating tweet came from Debasish Ghosh, who wrote this:
“greenspunning typechecking into ruby code” .. isn’t that what u expect when u implement a big project in a dynamically typed language ?
My answer was:
@debasishg really don’t agree about that. if you handle a dynamically typed project correctly, you will not end up greenspunning types.
Lets take this from the beginning. The whole point of duck typing as a philosophy when writing Ruby code is that you shouldn’t care about the actual class of an object you get as argument. You should expect the operations you need, to actually be there, and assume they are. That means anyone can send in any kind of object as long as it fulfills the expected protocol.
One of the problems in these kind of discussions is that people conflate classes with types in dynamic languages. In well written Ruby code you will usually end up with a type for every argument – that type is a number of constraints and protocols that will wary depending on what you do with the objects. But the point is that it generally will make things more complicated to equate classes with these types, and you will design classes without any real purpose. Since you don’t have static checking, you don’t need to have overarching classes that act as type specifiers. The types will instead be implied in the contract of a method.
So far so good. Should you keep this in mind when designing a method? In most cases, no. I tend to believe that you will end up conflating classes and types. That’s what I’ve seen on several projects at least. The first warning sign is generally kind_of? checks. Of course, you can do things this way, but you will restrict quite a lot of the power of dynamic typing by doing this. One of the key benefits of a dynamically typed language is the added flexibility. If you end up greenspunning a type system, you have just negated a large part of the benefit of your language.
The types in a well defined system will be implicitly defined by what the method actually does – and specified by the tests for that method. But if you try to design explicit types, you will end up writing a static type system into your tests, which is not the best way to develop in these languages.
After a slightly longer development cycle then the last time, I am finally happy to announce that Ioke E has been released.
Ioke is a language that is designed to be as expressive as possible. It is a dynamic language targeted at the Java Virtual Machine. It’s been designed from scratch to be a highly flexible general purpose language. It is a prototype-based programming language that is inspired by Io, Smalltalk, Lisp and Ruby.
Homepage: http://ioke.org
Download: http://ioke.org/download.html
Programming guide: http://ioke.org/wiki/index.php/Guide
Wiki: http://ioke.org/wiki
Ioke E is the third release of Ioke. It includes many new features from Ioke S, among them full support for Java integration (including implementing interfaces and extending classes), many new methods in the core libraries, IOpt, TextScanner, BlankSlate, better support for regexp interpolation and escapes in regexps, support for Unicode analysis and much more.
Ioke E also includes a large amount of bug fixes.
Features:
The many things added in Ioke E could not have been done without the support of several contributors. I would like to call out and thank:
T W <twellman@gmail.com>
Sam Aaron <samaaron@gmail.com>
Carlos Villela <cv@lixo.org>
Brian Guthrie <btguthrie@gmail.com>
Martin Elwin <elvvin@gmail.com>
Felipe Rodrigues de Almeida <felipero@gmail.com>
Wes Oldenbeuving <narnach@gmail.com>
Martin Dobmeier <martin.dobmeier@googlemail.com>
Victor Hugo Borja <vic.borja@gmail.com>
Bragi Ragnarson <bragi@ragnarson.com>
Petrik de Heus
Since all the rage in blog posts at the moment is laziness, I thought I’d take a look at how something can be lazy in Ioke. There are several ways of doing this, but most of them are really easy. I haven’t decided exactly how the implementation in core should look like, but it would probably be something like the following.
Basically, for laziness in a dynamic language you want something that lazily evaluates itself when requested the first time, and after that always return that value. Something like this:
foo = Origin mimic foo bar = lazy("laziness has run!" println. 42) "helo" println foo bar println foo bar println
Here we create a lazy values that returns 42, and prints something to prove what is happening. The lazy-method contains the magic. Any code could be submitted here. In this case the code can lexically close over variables outside, if wanted.
How would we implement “lazy” then? Something like this would suffice:
DefaultBehavior lazy = dmacro( [code] laziness = fnx( result = code evaluateOn(call ground) cell(:laziness) become!(result) result ) )
This code creates a lexical block and returns it. That block is a closure around the argument code. When called, it will evaluate that code, and then change itself to become that result value. Most of the magic here is really in the “become!” operation. And that is how simple it is. Since Ioke tries to make things representation independent, this implementation of lazy works in basically all cases.
On Tuesday, the Scandinavian Developer Conference will happen in Gothenburg. It’s turning out to be a really nice conference with lots of cool presentations. ThoughtWorks is hosting a track on Emerging Technology, where we’ll have lots of things that I have a particular interest in.
In my track there will be a presentation about Erlang, by Erik Stenman. I have a presentation about Groovy, and one about JRuby. I have a talk by John Davies about cloud computing, and one about Scala. And I also have a session about Smalltalk, and finally Marcus Ahnve will give a talk called “The simplest thing that could possibly work”. All in all it will be a great day. Hope to see you there.
|
I am a software developer working for Centro de Autonomía Digital, mainly with privacy, security and cryptography issues. I have worked on many different things in my life, including programming language design and implementation, security, privacy and cryptography, artificial intelligence and more. |
|
Quito |