Saturday, 31 July 2010

Mathematica 7 review: buggy but fun!

At only £195+VAT, the Mathematica 7 Home Edition is just too tempting as an executive toy but it still seems to be far too buggy to be taken seriously. After just a few hours of playing around, a variety of bugs have become apparent. Every Mathematica user fears the dreaded error box that marks the loss of all unsaved data:

Fortunately, a really serious bug in the FFT routines of Mathematica 7.0.0 was fixed for the 7.0.1 release. This was a showstopper for customers of our time-frequency analysis add-on. The severity and ubiquity of this bug really highlights just how little quality assurance goes into Wolfram's software which, in turn, goes to show how a unimportant correctness is in the creation of commercially-successful software products, even if they are used in aerospace engineering!

The first bug is in the new support for parallelism in Mathematica. Although it is only supposed to handle 4 cores, it produces pages of errors when run on a machine with more cores such as our 8 core system:

Half of the spawned kernels die and the errors are about miscommunications. Looks like it was not tested on anything beyond a quadcore.

Another bug appears when trying to load the example matrices as described in Mathematica's own documentation:

And there seems to be a nasty bug in the MinCut function that is supposed to find partitions that break the fewest edges because this function causes an access violation (aka segmentation fault) every time we try to use it:

A question on Stack Overflow highlighted another bug, this time in Mathematica's regular expression engine which crashes on non-trivial inputs, again losing all unsaved data:

So, although we do not hesitate to recommend Mathematica as an executive toy or even fun educational software for children, we still cannot recommend Mathematica for serious use due to the pervasiveness of serious bugs in this software. On the other hand, Mathematica's commercial success really demonstrates just how unimportant quality is in the software world.

As an aside, it is interesting to note that Mathematica is one of the languages with a bleak future in the face of multicore computing due to fundamental design issues. Specifically, the Mathematica system is largely built around the Mathematica language which is a term rewrite language evaluated by repeated substitution using symbolic replacement rules. The rules are, by definition, read and written via one giant global rule table and, consequently, direct concurrent use from multiple threads would introduce massive contention for the global table. The only viable solution is to duplicate the entire system separately on each core and communicate via message passing. This foregos the benefits of shared memory offered in a highly-efficient hardware-accelerated form on all multicore computers and, consequently, parallelism in Mathematica is orders of magnitude slower than in other languages. For example, even ignoring the distribution of definitions required to use parallelism in Mathematica, its ParallelTable construct is over 250× slower than the equivalent Array.Parallel.init in F#.

Still, Mathematica's accompanying literature alone justifies the price of just £195!