A few months ago I was interviewed for a master thesis on Polyglot Programming by Hans-Christian Fjeldberg. This is now finished and available online. You can download it here: http://theuntitledblog.com/wp-content/uploads/2008/08/polyglot_programming-a_business_perspective.pdf
If you haven’t already seen this, let me totally recommend Stuart Halloway’s series on the languages he call Java.next. The series look at several different aspects of these languages (Groovy, Scala, Clojure, JRuby), contrast them with each other and Java. Highly recommended if you are in any way interested in the languages that will soon replace Java for much application development.
The three published parts are:
This is a continuation of my previous posts describing layers of code written in different programming languages. I have thought about the things involved for a while, and had several discussions with people about it. There were some parts that I didn’t describe as well as I thought in my posts, and I will try to do better in this one.
The core of these ideas are based on polyglot programming, the thinking that you should use several different languages in a project, based on which languages are better suited for different parts of it. Another term for this concept is Language-oriented programming. So how do you organize a polyglot system? The most natural way for me is to divide it into layers. In most cases you will find that different categories of languages will be better suited to different layers of the application.
In my original post I identified three layers that can be used to organize polyglot systems. These layers are the stable layer, the dynamic layer, and the domain layer. There are several reasons for organizing them this way, and I’ll take a harder look at each of the layers further down. But first let me note that these layers are usually depicted in the form of a pyramid, with the stable layer being that base. That is definitely not how I think about it. In fact, I see it as an inverted pyramid, where the stable layer is the tip of the pyramid, providing the base. The Dynamic layer is the middle part. The domain layer should be the largest part and will very often include more than one dynamic language. So in my mind I represent the different domain languages as smaller pyramids standing upside down, covering the base area. Now, the dynamic layer can also be divided into smaller parts like this, based on language or functionality. This is a bounded fractal representation, which is the reason for the title of this blog post.
This diagram shows how I think about it:
Of course, the smaller pyramids can be all the same language and system, or several different ones. It all depends on the application or system you are building. So you can for example use a combination of Ruby, Java and external or internal DSLs:
Or you could use Clojure, Scala and JavaScript:
Or any other combination you can imagine. As long as the combination is what’s best suited for the problem.
Let’s take a look at the definitions of the different layers. There have been some discussion about the names I’ve chosen for them, so let me describe a little more what the responsibility of each part is, and why it’s in that part of the system.
The Domain Layer
This layer is the simplest. This is where all the actual domain rules are defined. In general that means one or more domain specific languages. It doesn’t really matter if they are internal or external. This model see them as the same layer. This part of the system is what needs to be malleable enough that it should be possible to change rules in production, allow domain experts to do things with it, or just plain a very complicated configuration. The languages used in this layer are mostly external DSLs, but can also include extremely DSL-friendly languages like Ruby, Python or Groovy.
The Dynamic Layer
Neal Ford argues that this layer isn’t so uch about dynamic, as it is about essence. That was never my intention. The problem is that even if you take a language like Scala, which is usually classified as an essential language, Scala requires compilation. To me, compilation is ceremony, which means that it’s one extra thing you don’t want to care about when writing most of your application code. That’s why this layer needs to be dynamic. This is where languages like Ruby, Groovy, Python, JavaScript, Clojure and others live.
The Stable Layer
I view the stable layer as the core set of axioms, the hard kernel or the thin foundation that you can build the rest of your system in. There is definitely advantages to having this layer be written in an expressive language, but performance and static type checking is most interesting here. There is always a tradeof in giving up static typing, and the point of having this layer is to make that tradeof smaller. The dynamic layer runs on top of the stable layer, utilizing resources and services provided.
Another important feature of this layer is that this is where all interfaces are defined. By interfaces I mean external API’s. They need to be hard for other clients to be able to trust them. But the implementations for them lives in the dynamic layer, not in the stable. By doing it this way you can take advantage of static type information for your API’s while still retaining full flexbility in implementation of them. Languages in the stable layer can be Java, Scala or F#. It should be fairly small compared to the rest of the application, and just provide the base necessary services needed for everything to function.
The most common objection I hear from people about this strategy is the same as for the general polyglot programming idea: if we have a proliferation of languages in a system, it will be harder to find skilled programmers who can work with it.
This objection is true to a degree, but there are several ways around it. First, I have to say that I don’t believe this is such a big problem as many others think. Programmers nowadays depend on their tool chains quite heavily, all of them including many advanced features that takes lots of time to learn. But most programmers doesn’t even view their languages as tools. In my mind, the programming language is the most important tool. And once we start using better languages for systems, many of the things we need other tools for will disappear or become less of a problem.
I tend to believe that programming languages are quite easy to learn as soon as you understand the fundamental building blocks of programming languages. And if you don’t have a fair understanding of these building blocks, I would say that you probably aren’t using your current language as well as you should either. I see this as part of being responsible programmers.
I also believe quite strongly that if we used better languages for our code, many code bases would be smaller, easier to understand, easier to maintain and cost less – which means you could afford to find a more skilled programmer to do the work for you. This would mean that both parties win – the programmer gets more interesting work and better code, while the client gets more worth for his money in less time.
OK, so this isn’t necessarily anything new, but I had to go with the running joke of the two blog posts this post is more or less a follow up to. If you haven’t already read them, go read Yegge’s Dynamic Languages Strikes Back, and Beust’s Return Of The Statically Typed Languages.
So let’s see. Distilled, Steve thinks that static languages have reached the ceiling for what’s possible to do, and that dynamic languages offer more flexibility and power without actually sacrificing performance and maintainability. He backs this up with several research papers that point to very interesting runtime performance improvement techniques that really can help dynamic languages perform exceptionally well.
On the other hand Cedric believes that Scala is bad because of implicits and pattern matching, that it’s common sense to not allow people to use the languages they like, that tools for dynamic languages will never be as good as the ones for static ones, that Java generics isn’t really a problem, that dynamic language performance will improve but that this doesn’t matter, that static languages really hasn’t failed at all and that Java is still the best language of choice, and will continue to be for a long time.
Now, these two bloggers obviously have different opinions, and it’s really hard to actually see which parts are facts and which are opinions. So let me try to sort out some facts first:
Dynamic language have been around for a long time. As long as statically typed languages in fact. Lisp was the first one.
There have been extremely efficient dynamic language implementations. Some of the Common Lisp implementations are on par with C performance, and Strongtalk also achieved incredible numbers. As several commenters have noted, Strongtalks performance did not come from the optional type tags.
All dynamic languages in large use today are not even on the same map with regards to performance. There are several approaches to fixing these, but we can’t know how well they will work out in practice.
Java’s type system is not very strong, and not very static, as these definitions go. From a type theoretic stand point Java does not offer neither static type safety nor any complete guarantees.
There is a good reason for these holes in Java. In particular, Java was created to give lots of hints to the compiler so the compiler can catch errors where the programmer is insoncistent. This is one of the reasons that you very often find yourself writing the same type name twice, including the type name arguments (generics). If the programmer makes a mistake at one side, the compiler will be able to catch this error very easily. It is a redundancy in the syntax that makes Java programs very verbose, but helps against certain kinds of mistakes.
Really strong type systems like those Haskell and OCaML use provide extremely strong compile time guarantees. This means that if the compiler accepts your program, you will never see any runtime errors from the type system. This allows these compilers to generate very efficient code, because they know more about the state of the application at most points in time, compared to the compiler for Java, which knows some things, but not nearly as much as Haskell or OCaML.
The downside of really strong type systems is that they disallow some extremely common expressions – these are things you intuitively can imagine, but it can’t be expressed within the constraints of such a type system. One solution to these problems is to add higher kinds, but these have a tendency to create more complexity and also suffer from some of the same problems.
So, we have three categories of languages here. The strongly statically checked ones, like Haskell. The weakly statically checked ones, like Java. And the dynamically checked ones, like Ruby. The way I look at these, they are good at very different things. They don’t even compete in the same leagues. And comparing them is not really a valid point of reasoning. The one thing that I am totally sure if is that we need better tools. And the most important tool in my book is the language. It’s interesting, many Java programmers talk so much about tools, but they never seem to think about their language as a tool. For me, the language is what shapes my thinking, and thus it’s definitely much more important than which editor I’m using.
I think Cedric have a point in that dynamic language tool support will never be as good as those for statically typed languages – at least not when you’re defining “good” to be the things that current Java tools are good at. Steve thinks that the tools will be just as good, but different. I’m not sure. To a degree I know that no tool can ever be completely safe and complete, as long as the language include things like external configuration, reflection and so on. There is no way to include all dynamic aspects of Java, but using the common mainstream parts of the language will give you most of these. As always this is a tradeoff. You might get better IDE support for Java right now, but you will be able to express things in Ruby that you just can’t express in Java because the abstractions will become too large.
This is the point where I’m going to do a copout. These discussions are good, to the degree that we are working on improving our languages (our tools). But there is a fuzzy line in these discussions, where you end up comparing apples and oranges. These languages are all useful, for different things. A good programmer uses his common sense to provide the best value possible. That includes choosing the best language for the job. If Ruby allows you to provide functionality 5 times faster than the equivalent functionality with Java, you need to think about whether this is acceptable or not. On the one hand, Java has IDEs that make maintainability easier, but with the Ruby codebase you will end up maintaining a fifth of the size of the Java code base. Is that trade off acceptable? In some cases yes, in some cases no.
In many cases the best solution is a hybrid one. There is a reason that Google allows more than one language (C++, Java, Python and JavaScript). This is because the languages are good at different things. They have different characteristics, and you can get a synergistic effect by combining them. A polyglot system can be greater than the sum of it’s parts.
I guess that’s the message of this post. Compare languages, understand your most important tools. Have several different tools for different tasks, and understand the failings of your current tools. Reason about these failings in comparison to the tasks they should do well, instead of just comparing languages to languages.
Be good polyglot programmers. The world will not have a new big language again, and you need to rewire your head to work in this environment.
Today I spent some time connecting two languages that are finding themselves popular for solving wildly different kinds of problems. I decided I wanted to see how easy it was and if it was a workable solution if you would want to take advantage of the strengths of both languages. The result is really up to you. My 15 minutes experiment is what I’ll discuss here.
If you’d like, you can see this as a practical example of the sticky part where two languages meet, in language-oriented programming.
The languages under consideration is Ruby and Erlang. The prerequisite reading is this eminent article by my colleague Dennis Byrne: Integrating Java and Erlang.
The only important part is in fact the mathserver.erl code, which you can see here:
-module(mathserver).
-export([start/0, add/2]).
start() ->
Pid = spawn(fun() -> loop() end),
register(mathserver, Pid).
loop() ->
receive
{From, {add, First, Second}} ->
From ! {mathserver, First + Second},
loop()
end.
add(First, Second) ->
mathserver ! {self(), {add, First, Second}},
receive
{mathserver, Reply} -> Reply
end.
Follow Dennis’ instructions to compile this code and start the server in an Erlang console, and then leave it there.
Now, to use this service is really easy from Erlang. You can really just use the mathserver:add/2 operation directly or remotely. But doing it from another language, in this case Ruby is a little bit more complicated. I will make use of JRuby to solve the problem.
So, the client file for using this code will look like this:
require 'erlang'
Erlang::client("clientnode", "cookie") do |client_node|
server_node = Erlang::OtpPeer.new("servernode@127.0.0.1")
connection = client_node.connect(server_node)
connection.sendRPC("mathserver", "add", Erlang::list(Erlang::num(42), Erlang::num(1)))
sum = connection.receiveRPC
p sum.int_value
end
OK, I confess. There is no erlang.rb yet, so I made one. It includes some very small things that make the interfacing with erlang a bit easier. But it’s actually still quite straight forward what’s going on. We’re creating a named node with a specific cookie, connecting to the server node, and then using sendRPC and receiveRPC to do the actual operation. The missing code for the erlang.rb file should look something like this (I did the minimal amount here):
require 'java'
require '/opt/local/lib/erlang/lib/jinterface/priv/OtpErlang.jar'
module Erlang
import com.ericsson.otp.erlang.OtpSelf
import com.ericsson.otp.erlang.OtpPeer
import com.ericsson.otp.erlang.OtpErlangLong
import com.ericsson.otp.erlang.OtpErlangObject
import com.ericsson.otp.erlang.OtpErlangList
import com.ericsson.otp.erlang.OtpErlangTuple
class << self
def tuple(*args)
OtpErlangTuple.new(args.to_java(OtpErlangObject))
end
def list(*args)
OtpErlangList.new(args.to_java(OtpErlangObject))
end
def client(name, cookie)
yield OtpSelf.new(name, cookie)
end
def num(value)
OtpErlangLong.new(value)
end
def server(name, cookie)
server = OtpSelf.new(name, cookie)
server.publish_port
while true
yield server, server.accept
end
end
end
end
As you can see, this is regular simple code to interface with a Java library. Note that you need to find where JInterface is located in your Erlang installation and point to that (and if you’re on MacOS X, the JInterface that comes with ports doesn’t work. Download and build a new one instead).
There are many things I could have done to make the api MUCH easier to use. For example, I might add some methods to OtpErlangPid, so you could do something like:
pid << [:call, :mathserver, :add, [1, 2]]
where the left arrows sends a message after transforming the arguments.
In fact, it would be exceedingly simple to make the JInterface API downright nice to use, getting the goodies of Erlang while retaining the Ruby language. And oh yeah, this could work on MRI too. There is an equivalent C library for interacting with Erlang, and there could either be a native extension for doing this, or you could just wire it up with DL.
If you read the erlang.rb code carefully, you might have noticed that there are several methods not in use currently. Say, why are they there?
Well, it just so happens that we don’t actually have to use any Erlang code in this example at all. We could just use the Erlang runtime system as a large messaging bus (with fault tolerance and error handling and all that jazz of course). Which means we can create a server too:
require 'erlang'
Erlang::server("servernode", "cookie") do |server, connection|
terms = connection.receive
arguments = terms.element_at(1).element_at(3)
first = arguments.element_at(0)
second = arguments.element_at(1)
sum = first.long_value + second.long_value
connection.send(connection.peer.node, Erlang::tuple(server.pid, Erlang::num(sum)))
end
The way I created the server method, it will accept connections and invoke the block for every time it accepts a connection. This connection is yielded to the block together with the actual node object representing the server. The reason the terms are a little bit convoluted is because the sendRPC call actually adds some things that we can just ignore in this case. But if we wanted, we could check the first atoms and do different operations based on these.
You can run the above code in server, and use the exact same math code if you want. For ease of testing, switch the name to servernode2 in both server and client, and then run them. You have just sent Erlang messages from Ruby to Ruby, passing along Java on the way.
Getting different languages working together doesn’t need to be hard at all. In fact, it can be downright easy to switch to another language for a few operations that doesn’t suit the current language that well. Try it out. You might be surprised.
It’s interesting. One of the common reactions I’ve heard to my recent writings about polyglot programming is something I really don’t understand. Actually, I’ve heard the same objection to other persons writing about polyglot.
The objection is that just because I propose polyglot programming – using several different programming languages for different purposes in the same system – I can use whichever language
and as such should not try to find better languages or say that certain languages are bad.
But that’s really a confounding of the issue. Just because I can use any language in the dynamic layer doesn’t mean I should. In fact, just because polyglot programming as a strategy means you will use more than one language, it is even more important to be careful and use the best languages available for the task. Which is why I’m working to improve JRuby, why I’m evaluating Scala as a replacement for Java, why I’m working on a language based on Io. It’s all about using the best languages. I may be a polyglot, but I’m definitely not a panglot or omniglot.
|
I am a software developer working for Centro de AutonomÃa Digital, mainly with privacy, security and cryptography issues. I have worked on many different things in my life, including programming language design and implementation, security, privacy and cryptography, artificial intelligence and more. |
|
Quito |