Tuesday, 6 November 2018

Benchmarking in the web age

The TechEmpower website contains some fascinating benchmarks of servers. The results on this benchmark of multiple requests to servers provide some insight into the performance characteristics of .NET on a modern problem. Specifically, the C# on ASP.NET Core solutions range from 2.5-80× slower than fastest solution which is written in Rust. In fact, C# is beaten by the following programming languages in order:

  1. Rust
  2. Java
  3. Kotlin
  4. Go
  5. C
  6. Perl
  7. Clojure
  8. PHP
  9. C++

Furthermore, .NET Core is Microsoft's new improved and faster version of .NET aimed specifically at these kinds of tasks. So why is it beaten by all those languages? I suspect that a large part of this is the change in workload from the kind of number crunching .NET was designed for to a modern string-heavy workload and I suspect .NET's GC isn't as optimised for this as the JVM is. As we have found, .NET has really poor support for JSON compared to other languages and frameworks, with support fragmented across many non-standard libraries. Amusingly, OCaml has standardised on Yojson which we have found to be much faster and more reliable despite the fact that it is a hobby project.

Another interesting observation is that C and C++ are doing surprisingly badly. The fact that Perl and PHP are beating C++ is doubtless because they are calling out to C code but that seems less likely for Rust, Java, Kotlin and Go. One interesting hypothesis I read on the interwebs is that C++ developers tend to copy strings a lot in order to avoid memory management bugs whereas Rust protects its developers from such bugs so they can be more efficient without having to worry. If true, that would be quite amazing because the user-land code behind these benchmarks isn't that big and, yet, C++ has already failed to scale and has developers reaching for one-size-fits-all-badly solutions.

Whatever the case, one thing is clear: the time is ripe for new languages, VMs and frameworks.

No comments: