This idea came to me as a tweet, but was way too long for a tweet. As it turns out, it’s also something I’ve been saying a lot lately, since it’s the answer to one of the very common arguments against dynamic languages.
The argument usually goes like this: “Dynamically typed languages are fine for smaller programs and simple web applications, but if you’re building something big, something that will be several millions of lines of code, you really need all the tools you can only get from a statically typed language”. Where the statically typed language mentioned is typically Java. The argument is still common enough without any specific language name mentioned though.
Interestingly, there are several problems and fallacies in this seemingly simple argument. I’m not going to tackle all of them here, but just focus on the beginning. Namely the idea that you are building something big. First of all, how do you know that? Have you done anything like it before? And how do you know it would be big in a different language? How do you know you can’t decouple the application in such a way you won’t have to build anything big? There are so many assumptions that can be questioned here. But at the end of the day, my glib answer that summarizes this usually goes something like this:
“Optimizing for your software project becoming big is the same as optimizing a car to hit a rock wall – you are optimizing for failure”
I firmly believe that becoming big is really failure. Once you have a big enough project you have lost. It might still work, but the cost will be extreme, and maintaining it will be a large burden too.
This is the reason I like agile. It emphasizes small, working pieces all the time. If you work with code this way, you can’t really become big. Instead, your project will be forced to be modularized and divided into smaller, more logical components that are highly cohesive and decoupled from each other.
40 Comments, Comment or Ping
Very well put!
October 13th, 2009
Good point. I tend to see a codebase of hundreds of thousands of line as a failure.
October 13th, 2009
And you’ll need tools to manage all those modularized components in some sensible way. The idea might be correct, but it’s not ‘the answer’ to that argument. It just pushes it back a little.
October 13th, 2009
Wise words .. I’m remind of:
http://en.wikipedia.org/wiki/Unix_philosophy
“Write programs that do one thing and do it well. Write programs to work together.”
October 13th, 2009
There are valid use cases for software that requires many thousands of lines of code. You seem to be saying that if you choose a particular language or methodology, you have lost simply because the project is large. Or because of the choice of language? I’m not sure what you are saying.
How would you replace a green-screen ERP system? With a big project or a small one?
I’m really happy that my car has been designed, tested and includes safety features so that if I hit a rock wall I will probably survive, even with partial offset. Good optimization. What’s wrong with that?
October 13th, 2009
Granted, few if any of us reading this will ever have this particular opportunity, but like all generalities, this one fails too. Consider the software running the LHC. Or any rockets/satellites. Or hospital equipment. Or a government tax system. Some of these systems are just … big. While Ola does give some qualifiers, “Have you ever done this before?”, etc., sometimes the answer is “yes, I have”. And, while you may LIKE small pieces that do one thing well, does not make it a rule.
That said, I agree with the post, but there are of course many many counter examples.
October 13th, 2009
Dom:
The words “optimizing for” are operative in my sentence. Adding safety features to a car is not the same thing as optimizing to hit a rock wall. You can do it – it’s hideously expensive and it isn’t really worth it for almost all cars.
October 13th, 2009
Dom:
Also, yes, there are valid use cases for software with a large number of lines of code. But you never know what those cases will be ahead of time. I’m saying that if you choose language/methodology for the assumption that it will be large, you have started out on in the wrong place.
October 13th, 2009
OK sure. It’s just that your title is “Plan to write big software and you have already lost”. For those of us who do plan to write big software, based upon our use cases, and then do so, it is a little affronting to be told that we are lost, started in the wrong place, and we should have done a small project instead.
I guess that I disagree that a big project is automatically a failure.
October 13th, 2009
When the spec is measured in hundreds of pages and the number of integration test cases are above 1000 you can’t satisfy them with small app. A system of small apps, or a system of modules with narrow interfaces are not so much different. In the end you can do bad design in any language.
October 13th, 2009
I agree absolutely
October 13th, 2009
I agree with the comments, some projects are big anyway and need tons of libraries that does a lot of things. In that case, A big chunk or plenty of small ones is either way a pain to keep consistent and to maintain.
I both case a good architecture will be needed to save the project.
In what i agree with you, is that you don’t need the overhead of a system designed to handle huge project for a small one.
JEEE enterprise server to handle 5000 records and 10 screens is masochism indeed.
Where is the border between those 2 extremes is probably a mix of culture, common sense, history , budget and politics
October 13th, 2009
OK, I like your notion that becoming big is failure. But how do you look at websites like Amazon.com? That’s BIG with capital letters. How did they lose? They operate successfully due to marginal costs on all fronts including their website.
I would like to hear your ideas on projects like Amazon.com.
October 13th, 2009
This a concise yet necessary shared observation on software design. The problem invariably becomes feature creep or inheriting someone elses working code.
Not every problem can be optimally fast, and also optimally sized for simplicity. As an alg coder of quite a few years of course I prefer and love simple easy to read, remember, and modify code. But then someone comes along and says, we need this fairly quick code, parallelized, ported to these other platforms, and faster than real time. Dynamically size and share memory as needed to scale to any number of available nodes, and have it driven by a GUI, but make those buttons overlay an elliptical earth model…..
As you can imagine, writing simple code is a desirable thing for programmers. Unfortunately we don’t dictate the terms and requirements, real physical systems do.
October 13th, 2009
I just point them to Squeak. Squeak has a rediculous list of features and capabilities, and lots of source code. It was written in Smalltalk.
October 13th, 2009
isn’t JRuby a big software project? :)
October 13th, 2009
Yet cars are optimized to hit a brick wall, cause practice tought us that it is bound to eventually happen.
October 13th, 2009
Amazon uses lots of smaller sevices to power their business.
October 13th, 2009
People pay money to write “big” software because it’s what they need; because it’s difficult, it pays fairly well.
I work on Delphi, the compiler. The whole thing, IDE, debugger, compiler, etc., adds up to a fairly substantial size. It can’t really be made smaller, because that would cost functionality which our customers pay us for.
The platitude you express in this post isn’t really grounded in any reality I’m familiar with.
October 13th, 2009
Zomg:
That’s not true. “Optimized for” means it’s what the thing in question does best. Cars are optimized to be transport people around in a reasonably safe manner with a reasonably low energy expenditure using the current infrastructure. Cars sometimes can handle hitting a brick wall, but it’s not what they do best.
October 13th, 2009
I don’t buy this argument.
Suppose you have a library used by a number of in-house projects. You find some code (an API method) in that library that you suspect isn’t being used by anything, and wonder if you can remove it. After all, no one likes big code bases when small ones will do. And no one wants to maintain code that isn’t used.
Don’t you find static typing makes it easier to verify whether anything is using that code? Isn’t it easier to use “Find References” on a method, as compared to deleting it, and running the test suite of every app that uses the library? Heck, assuming the client apps even have comprehensive tests…
October 13th, 2009
Well put Ola. The most difficult art that we must master as software craftsmen is to solve a “big” problem with a small solution.
Keeping it simple is key to making a successful product. Throwing a whole lot of money and “bodies” at a problem will likely just make the solution bigger not better. Not better since you will likely choose mainstream technologies rather than the best technologies – “we could not find 200 erlang programmers so we opted for java…”
October 13th, 2009
“Divide each difficulty into as many parts as is feasible and necessary to resolve it.”
Renee Descartes.
October 14th, 2009
I’ve read well but I don’t want to give you a high score for the article. Well, I don’t know exactly how long you’ve been coding in software area however planning & analyzing are slightly different compared to cording as you may probably know. From my perspective, you need to work on brainstorming & designing rather than programming. I’m pretty sure that it will make your opinion changed and give you more ideas.
October 14th, 2009
I’m gonna blow the quote… but it’s something like “make things as simple as possible, but no simpler”. The last part of that is the key. I think the opposite of what you propose is also true, and if you plan ahead to be small then you’ve already lost.
Often it’s obvious if the end result will be small or large. Are you writing a single service to manage logins? Even if it ends up being huge for it’s problem domain it will still be comparatively small to a full blown international accounting package.
It’s better to design minimally and let nature take its course. I would imagine the program language decided on would be determined more by team experience and problem domain than expected code base size.
October 14th, 2009
I distinguish big projects from fat ones. Big projects will ever exist, because sometimes the needs are huge, complex (but I get your point regarding splitting the big stuff into smaller more maintenable pieces). Fat projects are monolithic ones, undecoupled, untested and hated by their own developers and instigators.
And sometimes the most difficult decision to take is to kill these kind of projects and rebuild, because usually killing something big and fat is perceived as wasting the money invested in it ; while in truth, all that money just served you to skill up, learn and know what you won’t ever be wanting anymore.
October 14th, 2009
Define Big
October 14th, 2009
Nice article, I agree. My typical answer to the “several million lines of code” argument goes like this:
“Several million lines of code? Don’t use [static language], and you can get that down to a single million or less.”
October 14th, 2009
It’s a shame you’ve conflated two issues here – that of the static/dynamic debate and the notion that big projects are bad. I mean, I totally agree with the sentiment of your post, I just think that it’s easy for people to get confused about which issue they’re arguing about ;-)
I think that the notion of a big project being a bad project is totally right, although as Rob says, perhaps the definition of big needs expanding upon. I would say that a project should be no larger than is understandable by a group of people whose cardinality is less than the number of fingers on your hand. Failing that, it needs to be broken into subprojects and so on recursively until the initial condition is met.
This means I’m making a distinction between product and project, whereby a product is the sum of all the constituent projects. Daniel mentioned Amazon, and if I’m not mistaken, they see their product as a large set of individual projects. So do Google. Mustang hit the nail on the head with his Descartes quote. Plato also said that it’s important to be careful about the correct decomposition of the product into projects:
“First the taking in of scattered particulars under one Idea, so that everyone understands what is being talked about…Second, the separation of the Idea into parts, by dividing at the joints, as nature directs, not breaking any limb in half as a bad carver might.”
Finally, Martin Ellis seemed to miss the point by focussing directly on the static/dynamic argument. He makes a good point about how tooling can help remove unused code. However, I don’t believe it’s a case of removing unused code, rather it’s more of a point of not writing it in the first place. And sure, tooling might help you find out if a certain routine is actually referenced elsewhere in the source, but it certainly doesn’t help you know if it’s actually needed, or whether it could be folded into a more communicative structure.
In conclusion, I think it’s a shame that we live in a number-driven society where we always seem to play for the quantative win.
October 14th, 2009
“Optimizing for your software project becoming big is the same as optimizing a car to hit a rock wall”
Yes, but cars WILL and DO crash into brickwalls. In the real world where humans aren’t perfect, and factors are non-deterministic. So designing for that in mind is a huge safety consideration.
October 15th, 2009
In my opinion building a system is like raising a child: You have to grow it. Start with an adult, would be a frankensteinesque approach.
So, even if I would build a worldwide big huge whatever system, I would try to find meaningful features to be released after weeks, not months or years. Building a large system without early feedback cycles is a large risk (in theory and real life): It might be late, expensive, it might not work, or not be what was wanted.
Inspired by daily Dueck 73 http://www.omnisophie.com/day_73.html
(german only)
October 15th, 2009
This was already covered to a degree in a Blog Rant by Steve Yegge:
http://steve.yegge.googlepages.com/is-weak-typing-strong-enough
He says ultimately it comes down to the problem domain. Deep problems have deep solutions. I honestly don’t think you’ve addressed all the points he raises.
Also – I’d argue that no one really sets out to build a large system – some are just accidentally successful and keep growing.
I’d also content that there is a management imperative to keep languages at the ‘lowest common denominator’ – so that managers have a negotiation advantage when it comes to hiring.
It’s lovely to live in a land where you just start new projects all the time. But not all projects have that freedom. A lot of projects exist because what went before them was successful and so a slight variation on that is being paid for.
Also +1 for argument between planning and designing vs programming.
October 21st, 2009
Isn’t it the case that any well written application is actually composed of numerous smaller ones? An application comprising millions of lines of code will actually be thousands of modules.
The issues then are not the language but the architecture.
I’m not sure why people here are saying that big s bad and no one aims for a big solution. Have a look at the applications running the corporate world. Huge things – loo at an insurance underwriting system or claims handling system, or other lines of business systems supporting numerous products and thousands of users,.
Actually this is a rubbish article totally out of touch with the real world of business and government.
November 10th, 2009
The problem with Squeak Smalltalk is the classes and methods are global, so it’s easy to break the interdependences. Unix shows the way for system to grow without getting too complicated in the meantime, I believe.
December 8th, 2009
how can we determine a project s small, medium or large..is there any limitation for no. of loc and fp???
January 10th, 2010
Reply to “Plan to write big software – and you have already lost”