Wednesday, 8 October 2008

Mono 2.0

The Mono Project was intended to bring the benefits of Microsoft's excellent .NET framework and, in particular, the Common Language Run-time (CLR) to other operating systems such as Linux and Mac OS X. The home page even boasts that Mono is "positioned to become the leading choice for development of Linux applications".

If Mono provided a solid foundation then it could well become the platform of choice on systems like Linux but the 1.x implementations have been plagued by reliability and performance problems.

Firstly, the design and implementation of a performant concurrent garbage collector is the single most difficult of the platform's core features to implement but is of absolutely critical importance. The implementation of a real GC for Mono was originally postponed and the Boehm GC was used instead, being described as an "interim measure" over 5 years ago. The Boehm GC is fundamentally flawed in this context, leaving programs to leak memory until they are killed by the OS.

Secondly, Mono's code generator generates native code that is slower than almost all other compiled languages.

Five years have passed since then and Mono 2.0 has just been released but, contrary to previous announcements, this new version is still built upon Boehm's flawed garbage collector and our SciMark2 benchmark implementation in F# shows that Mono 2.0 is still 3× slower than .NET for numerical algorithms.

So Mono appears to be no closer to its goal of being the leading choice for development of Linux applications. This begs the question, how might Mono development be improved in the future? Failing to implement a working GC for Mono forced the Mono developers to spend a great deal of time and effort fixing bugs and addressing performance issues related specifically to the Boehm GC that, in the grand scheme of things, are worthless because that GC was only ever a stop-gap measure. Mono originally reinvented the code generator because mature reusable alternatives like LLVM were not yet available. However, the most difficult aspect of implementing a code generator for Mono is maintaining harmony with the GC but Mono still lacks a working GC so there is no harmony to be maintained. Consequently, it seems logical that the Mono developers would be wise to adopt the LLVM project to handle their code generation (because it already provides much better performance) and then continue trying to build a real GC for mono in harmony with LLVM-generated code. In fact, other projects such as PyPy are progessing so much more rapidly than Mono that they may be the first to provide a complete backend.

Hopefully the Mono project will provide the solid foundation that it intends to in the future but, in the mean time, we shall restrict ourselves to the use of robust garbage collectors.


Casper Bang said...

I seem to remember Miguel writing about how they plan to make use of the OpenJDK project to improve certain aspects of Mono. The GC would be a primary candidate one would imagine.

Having said that, I use Mono applications every single day and have never actually experienced any leaking that caused problems with my applications or OS.

Clinton Begin said...

In a world where Ruby and Python are fast enough for most, even PyGTK+ apps... I don't think this level of performance should be the #1 consideration for all applications. In fact, i think it's pretty rare that an app fails due to this kind of performance flaw. We all know it will get better. Remember when Java was slow? I'm a Java and Ruby developer, and I have to say that Mono is really, really impressive

Rich client development with Mono on Linux is especially compelling, considering how badly Java's Swing has sucked for 10 years -- mono is already ahead there.

If you're writing some sort of scientific simulation, low level graphics, calculations, or a device driver, sure... use C.

Otherwise, performance is no reason to shy away from Mono, even now.

Bob said...

How about shifting to Vala? Might be easier to swallow.

Flying Frog Consultancy Ltd. said...

Casper, from what I've heard it will be almost impossible to reuse the GC from the OpenJDK project because it is so closely tied to their code emitter. Does this mean the Mono project are giving up on the GC they have been developing in-house for several years?

Clinton, I appreciate that performance isn't always important but Mono's use of Boehm undermines its reliability, particularly for long-running apps. That would be a real problem for us.

Thanks for the reference to Vala, Bob. That project looks very interesting indeed. If they really can create a decent language with a real-time GC from scratch then I will certainly give it a go.

Alan said...

So you've found a specific case where mono is 3x slower. I can show you a case where mono is 10x slower than .NET. I can also show you a case where mono is 10x faster than .NET.

You talk about 'leaks' and link to a five year old post to the mailing list. That is fairly irrelevant. Do you really think that after 5 years there have been no bug fixes applied to mono and/or its GC?

You link to a benchmark which is inaccessible, so why bother. If you're going to link to a benchmark or its results, they must be easily accessible to the public. So for all intents and purposes, your claim of 3x slower is useless. It may very well be true - i'm not disputing whether you're right or not, but it's still useless because I can't replicate your results.

If you want to complain about monos performance, or monos GC, then post a reproducible testcase so everyone can see the problem. Also, linking to articles which are 5.25 years old and trying to make them seem relevant weakens your argument unless you can show that the exact same issue is still prevalent in the current mono release.

Flying Frog Consultancy Ltd. said...

Alan, SciMark2 is a freely available benchmark suite composed of five different tests totalling hundreds of lines of code: Fast Fourier Transform, Successive Over Relaxation, Monte Carlo, Sparse Matrix Multiply and LU decomposition. Mono 2.0 is 3.12×, 1.95×, 6.05×, 2.16× and 1.65× slower than .NET on each of them, respectively, on this 2.2GHz Athlon 64 X2 in 32-bit. You can run the code here to reproduce the results just as I and the Mono developers have already done.

Mono's current GC is leaky by design (the Boehm GC is "conservative") so the "bug fixes" you refer to can only ever hope to push the bugs around. The problem will not be fixed until Mono is given an accurate GC. The Mono developers know this and that is why Boehm's GC was only ever supposed to have been an "interim measure" and they have been striving to implement a GC of their own for years. But they continue to fail and I see that they have pushed back the deadline again recently.

Alan said...

I'm sure you've seen the thread on the mono-list. For those that haven't, those performance numbers are outdated since mono 2.2. The fastest test on mono is 0.25x slower than MS.NET and the slowest is 2x slower. Though that test was run with MS.NET on windows and mono on linux, so they are not quite directly comparable. There are platform differences which change performance. You'd need mono 2.2 on windows to get good results.

Monos new GC is also conservative. A conservative GC is not leaky by design. It's performance characteristics and collection characteristics differ from a precise GC, but that doesn't make one a better choice than another in all scenarios.

Alan said...

A good link:

Flying Frog Consultancy Ltd. said...

Alan, I contributed to the thread on the Mono mailing list so I am, of course, aware of it. The unverified benchmark results indicating that the unreleased Mono 2.2 on Linux is 1.25-2.0× slower than .NET on Windows are certainly compelling but the quantification "0.25× slower than .NET" is wrong and the results for the JVM are anomalous so I am waiting to verify these results myself before announcing anything.

Your assertion about conservative GCs is wrong. Boehm gives a counter example in the section "An Embarrassing Failure Scenario" of this article where he explains why his conservative GC leaks memory on common stream (lazy list) and queue implementations. In this aticle he explains why programs will leak an unbounded amount of memory with a conservative GC in general and describes how this can be worked around by rewriting the software.

This is why conservative GCs are completely unsuitable for a project like Mono. If Mono's new GC is still not accurate then I think it is a real shame and a complete write-off. Professionals are unlikely to touch a platform with such a shaky foundation.

Alan said...

Your mind, your decision. There are plenty of professionals dying to use mono. There are plenty of professionals who wouldn't touch it. You just happen to be part of the latter.

Though it is a nice appendum to this post to note that MS.NET also suffers from the same embarrassing failure as described in this thread:

Paolo Giarrusso said...

@Alan, in the end it was proved that with optimizations on, the example doesn't leak in .NET ( Fixing the bug for debug mode would probably hinder source debugging I guess: the leaked memory is dead by liveness analysis, but seems still in scope, and so the user of the debugger might want to look at it. I don't see an obvious fix, there might be a smarter one but I don't think so.