Friday, 30 January 2009

Mono 2.2 vs OCaml vs .NET vs LLVM vs JDK

Mono 2.2 was recently shipped and a new JIT code generator was the main selling point promoted by the Mono project because it provides the first significant performance improvement Mono has seen for many years. We used the F# version of the SciMark2 benchmark to benchmark this new version of Mono against Mono 2.0 and some competing VMs using their native SciMark2 implementations:

The composite results show that the new code generator in Mono is indeed substantially better at producing code that is 40% faster on average than the previous code generator from Mono 2.0 according to these results. However, Mono 2.2 is still a long way behind its competitors. Specifically, both LLVM and Sun's JVM are over 2.2× faster than the latest version of Mono.

SciMark2 is composed of five different tests: Fast Fourier transform, successive over-relaxation, Monte Carlo, sparse matrix-vector multiply and LU matrix decomposition. We can get more detailed comparison of the VMs by looking at the individual scores on each of these five tests:

We can see immediately that the Monte Carlo test from the SciMark2 benchmark had anomalous results. Specifically, Mono 2.0 was 12× slower than LLVM and Mono 2.2 is now only 3× slower than LLVM. The Monte Carlo test is an int intensive random number generator.

These performance improvements in Mono are compelling but we are still deterred by some fundamental design flaws that undermine Mono's reliability.


9 comments:

Andrew Bent said...

"...but we are still deterred by some fundamental design flaws that undermine Mono's reliability."

I'd be interested in you elaborating as to what these are.

SeveredCross said...

I'd be curious to see about the implementation of the test--I know some of the Java tests in the Debian Language Shootout do better than the equivalent Mono tests because the Java programs use native libraries and optimizations that the C# programs don't. Does SciMark suffer from something similar?

Flying Frog Consultancy Ltd. said...

@Andrew: Leaks. Mono has a broken tail call implementation that leaks stack space until it dies on non-self tail calls which are ubiquitous in functional programming languages like F# (e.g. in the monadic parser combinator library FParsec and in F#'s own asynchronous workflows). Mono relies upon Boehm's conservative GC for automatic reclamation but this is not a proper GC and, in particular, is known to leak on lazy lists (e.g. streams) and queues.

@SeveredCross: SciMark2 is a good benchmark that does not suffer from such problems provided you translate the code verbatim as we did and do not pull in custom libraries (e.g. for the FTT and LU decomposition). The original Java code is freely available here and there are translations to many languages available elsewhere. Our F# translation is here.

m said...

Mono uses Boehm in precise GC mode. The claims about the leaks on lazy lists is bogus.

Your tests were also performed on Windows, a platform that Mono has not been tuned for.

These tests show how Mono compares from Windows to Linux:

http://www.csharp-architect.com/images/MonobenchmarksIncLinux.gif

Other than the GC-bound binary tree test, the performance improvements are very significant.

Flying Frog Consultancy Ltd. said...

@M

Your statement about Mono using Boehm in precise mode is contrary to Mono docs which state that some scans are conservative and the rest are only "mostly precise".

Your statement about lazy lists leaking being "bogus" is contrary to Boehm's own description in the section "An Embarrassing Failure Scenario" of this article. Moreover, I just constructed a trivial concrete example using queues and verified that Mono leaks when .NET does not.

Your claim that our tests were performed under Windows is wrong. We used Linux for all tests except F# on .NET.

Finally, the performance of a broken VM is of little interest.

m said...

You would not mind posting the code that produces such a leak, would you?

Because I was unable to reproduce the scenario that you describe (the article you linked to).

Unless the answer is "become a subscriber to our journal to get the code" which seems to be a recurring topic around here.

Flying Frog Consultancy Ltd. said...

@m

The code for all of the tests I have referred to is freely available. The memory leak was described in detail here along with the code. That code is based upon the queue implementation from here. You just have to push and pop elements from a non-empty queue and Mono leaks. Everything your element refers to is leaked.

I cannot be sure exactly when this problem will arise in more general terms (which makes it even worse) but I suspect Mono can leak all cyclic data structures. In the context of functional programming, that is likely to mean closures as well as common containers.

Alan said...

Just cross posting my response from the other blogpost as it's more relevant here:

http://flyingfrogblog.blogspot.com/2009/01/mono-22-still-leaks-memory.html?showComment=1233788880000#c6767092037514148675

ciplogic said...

The two main limitations of Mono in Scimark benchmark were limited raw performance for Mono JIT and a slow garbage collector. Java 7 adds too some extra features by default (mostly: Escape Analysis by default, G1 garbage collector) and .Net 4.0 brings a new GC (probably a not improved code generator). LLVM also improves it's code generator which may mean that after two years the scores may change a little.
May you run the benchmarks at start of 2011 with updated runtimes? May improve the benchmark results and using Mono with SGen and LLVM should improve both code quality and score results.