Saturday, 7 August 2010

"Scala is foremost an industrial language"

In a recent interview about Scala and Clojure, Martin Odersky of Scala gave some interesting answers including the following:

Q: Rich Hickey is as well-read in the academic papers as anyone, but it’s Scala that has gained the perception as an “academic language”. Why do you think that has happened?

A: I think it’s mostly people who want to put Scala down making that comment. They take my EPFL affiliation and the papers we publish as an excuse to attach that label to the language. What’s funny is that often senior industrial Scala programmers get also accused as being academic. All this is rather ironical because Scala is foremost an industrial language with many well known companies using it. By contrast it’s much less taught at universities than Haskell or Lisp, let alone Java!

This raises the obvious question: in what sense is Scala "foremost an industrial language"?

As we understand it, Scala is developed by an academic team led by professor Odersky at an academic institution with academic funding for the sole purpose of academic research and this has culminated in a new academic paper every 7½ months on average over the past decade. That is not true of any industrial programming languages. Indeed, from our experiences with Scala this is reflected as a growth in esoteric language features when basic IDE support is neglected. Some people are trying to use Scala in industry, but that is true of many academic languages. Some funding has come from industry, including a surprising grant from Microsoft to update the .NET port of Scala, but that is presumably a tiny fraction of the total funding that has been spent on Scala. So this seems like rather an odd claim.

37 comments:

Jonathan Shore said...

I have to agree. The scala team is out of touch with real application developers. The scala team has put very little focus on IDE *and* performance problems.

For instance for comprehensions are *very* slow. I and many others have had a request in for over a year and nothing has been done about it (in spite of there being an obvious solution).

The most unfortunate aspect of the language is its burgoening complexity. Writing code is so much more difficult when one has to deal with explicit type annotation in a complex type system. Ocaml / F# are liberating in this way ...

Flying Frog Consultancy Ltd. said...

@Jonathan: That's very interesting, thank you.

Now, what confuses me is why on Earth these academics have started pretending to be industrialists. My guess is that this is the fault of their academic funding bodies who, I presume, have started asking them to promote real-world use of their research.

Indeed, Haskell is allegedly about to get a grant from Microsoft for real-world promotion in the context of parallelism (where I think Haskell is still firmly in the lab).

Charlie Martin said...

Possibly because, come right down to it, better intellectual tools beat programming environments over time?

gruenewa said...

Performance: Scala is one of the fastest programming languages at the alioth benchmarks game. It is only outperformed by C/C++.

Tool Support: Scala is supported in several major IDEs such as Eclipse, Netbeans, IntelliJ IDEA.

Scala as an industrial language: For me it seems that many language design decisions where made with industry in mind. A good example is the support for XML literals. This makes scala well suited choice for XML heavy enterprise applications like web-portals, SOAs and the like.
I don't know exactly what it is, Scala enables me to develop software rapidly. If I want to call a webservice, I just copy the payload as XML literal into the program and use ajava.net.URLConnection to submit it ... task accomplished within 2 minutes.
At the same time Scala enables me to express algorithms and program structure in a very clean way: Often the pseudocode for an algorithm turns out to be valid Scala code.

Flying Frog Consultancy Ltd. said...

@Gruenewa: "Scala is one of the fastest programming languages at the alioth benchmarks game. It is only outperformed by C/C++". On the 64-bit quadcore page, Scala comes halfway down after ATS, C, C++, Java, Haskell, Ada, Java and Go. Scala is 6-7× slower than other languages on the Fasta and k-nucleotide benchmarks. And, of course, the benchmarks game doesn't even include .NET.

"Tool Support: Scala is supported in several major IDEs such as Eclipse, Netbeans, IntelliJ IDEA". In practice, we found Scala's IDE support to be seriously flakey, often crashing. Jonathan Shore has apparently found the same thing.

"Scala enables me to develop software rapidly". Absolutely, Scala is a major advance over Java. I was just surprised to see Martin Odersky claiming that Scala was foremost an industrial language.

Cay Horstmann said...

Ummm, if you want to know in which context "Scala is foremost an industrial language", maybe it would be helpful to read the context of the interview?

He wasn't comparing Scala with C# or Cobol. The interview context is a comparison between Clojure and Scala.

In that context, Odersky's quote makes sense to me. Scala has experienced a decent amount of industrial adoption, and the Scala team is listening to their industrial users to a larger degree than what you would find in most academic communities.

Geir said...

Tool Support: Scala is supported in several major IDEs such as Eclipse, Netbeans, IntelliJ IDEA.

My next project is going to attempt to use Scala, and I have been following Scala the last year or so.

Eclipse does not, for any meaningful and productive value of "support", support Scala. The Scala plugin was unstable and woefully lacking in features when I went on holiday, which would be in mid-july. I have ended up following nightly builds in an attempt to rid myself of most bugs.

Scala is a language that is still under development. Every point release in the past has broken backwards binary compatibility, causing problems with the toolchain support we commonly use in Java. I have been told Odersky has said that is about to stop, though.

Scala has many of the same issues you can find some other languages. There are several ways to accomplish the same behaviour, fracturing your codebase style-wise. If you have a developer who is eager to show his grasp of the esoteric parts of the language, the code will be unreadable. In the past, integration with java libraries has been harder than it needed to be, although 2.8 looks much better. And Scala is a magnet for would-be library writers. Everyone seems to want to reimplement stuff like MVC, ORM, test and actor libraries by themselves.

And here I am, still thinking that Scala has potential. We are still going to see what it can do during the autumn.

Flying Frog Consultancy Ltd. said...

@Cay: You talk about Clojure as if it were an academic language when, in fact, it was created by a consultant in industry and is funded entirely by industry. Just look at the number of individuals and companies who have put their money where their mouths are by contributing to it financially.

@Geir: You experience basically reflects my own, except I experienced the same unreliability and was told that it was about to be fixed over 3 years ago! Granted Scala is still in development but far more effort has been put into it and, yet, these non-academic features have been neglected for years.

Tezka said...

Hi,
I heard Clojure has a port of .NET now. Have you had a chance to try it out? I am wondering if it's as stable as the Java version. On a relevant note, are you aware of any effort to port F# to the JVM?

Matt R said...

Scala is a language created in academia, but now being adopted by industry. When Odersky says in 2010, Scala is "foremost an industrial language", I would hope that he means that the primary purpose of Scala is now as a language for solving practical problems, rather than as a vehicle for language research.

Whether or not Scala will get widespread adoption, I don't know, and yes, better IDE support is a prerequisite, but it's certainly doesn't hurt that the creator of Scala views the language in this light. Your comments about "pretending" for the sake of "academic funding bodies" seem unnecessarily cynical.

steve said...

Sorry Jon,

is this the way you interact with other academic professionals? (Assuming you didn't just put money on the table to get the "title")

This is deeply saddening. Do you really think you need this kind of publicity?

Taking quotes out of context, suggesting someone is lying without even evaluating his or your own claims properly?

I hope this kind of attitude is not the one of the whole F#/OCaml community.

Concerning tool support: It's better than most dynamic languages and the esoteric ones you prefer: OCaml and F#.

Although tool support is not on par with Java's, there are many people working on it, with different approaches as you can see if you would compare Netbeans, Eclipse and IntelliJ.

Somehow claiming that a language designer should concentrate on IDE is absurd.

Why don't you just let the IDE developers payed to work on better IDE support concentrate on building better IDE support and let language designers design languages?

Everybody should do what he's best at.

Maybe you should think about what you're good at _except_ provoking unnecessary flame wars?

Flying Frog Consultancy Ltd. said...

@Tezka: "I heard Clojure has a port of .NET now. Have you had a chance to try it out?" Not yet but I should... :-)

Matt R: "I would hope that he means that the primary purpose of Scala is now as a language for solving practical problems, rather than as a vehicle for language research". I think there is a big leap of faith from Odersky hoping it will be useful for industry and it being foremost industrial.

"Your comments about pretending for the sake of academic funding bodies seem unnecessarily cynical". As long as Odersky's team is funded primarily to do research, Scala will not be foremost industrial and these non-academic deficiencies will surely remain unaddressed.

@Steve: "Why don't you just let the IDE developers payed to work on better IDE support concentrate on building better IDE support and let language designers design languages?" Where are Scala's paid IDE developers? They don't exist because Scala does not have any significant industrial funding.

Matt R said...

@FFC: "Where are Scala's paid IDE developers? They don't exist because Scala does not have any significant industrial funding."

Scala support in Eclipse was, in fact, funded by EDF trading at one point. However...

"As long as Odersky's team is funded primarily to do research, Scala will not be foremost industrial and these non-academic deficiencies will surely remain unaddressed."

Why must they remain unaddressed? I think Steve is correct -- it is not necessary or even desirable for language designers to write IDEs (or build tools, or web frameworks, etc...) My observation is that there are plenty of smart people building such things outside of EPFL -- and primarily practitioners, not researchers.

Flying Frog Consultancy Ltd. said...

@Matt R: "My observation is that there are plenty of smart people building such things outside of EPFL". Interesting. Who?

Jonathan Shore said...

Academic or Industrial, Scala seems to have positioned itself as the complex C++ to C successor to Java. Like C++, Scala has added significant syntactic and type complexity, while maintaining some compatibility with java semantics and syntax.

Some unfortunate problems for Scala:

- unlike Clojure or F#, Scala does not interoperate well with java.
- type system complexity is exposed to the programmer
- complex syntax patterns (avoidable in many cases, yet exist and present in libraries)
- very little in the way of type inference

I am not currently an active user of F# (just experimenting with it), but it seems to have gotten the balance right

I like aspects of Scala and yes, it generally is very fast (though its comprehensions have issues). From my experience with both F# and Scala, F# gets out of the way and lets one focus on the algorithm. With Scala constantly dealing with having to write out and determine the appropriate type annotation. The more functional code one writes in scala, the more complex some of these types get ...

So when I read "academic", I interpret as "out of touch". Where I am concerned, Scala is out of touch with what I want.

steve said...

@Jonathan:

Could you explain in what ways Clojure and F# interoperates well with Java?

- Which problems do you currently see?

- It is often claimed that Clojure has a superior integration with Java, but I still don't see on what these claims are based on. In most examples I have seen the integration is quite awkward.

- To my knowledge F# doesn't even run on the JVM, how should the integration work? (I hope you don't have Java->IKVM->CIL<-F# in mind...)

- Considering type inference: Feel free to describe an algorithm which supports full Hindley-Milner type inference in combination with a type system supporting overloading, overriding, sub classing, multiple inheritance and type classes.

@Jon: In my opinion it is very unfortunate that you resorted to writing snarky comments and picking other persons' comments apart as you see fit.
Could we agree on a more friendly and professional level of commnuication?

Considering its limited feature set F# has sometimes a more terse syntax than Scala and is therefore in some fields nicer to use.
But you forget that Scala is a general purpose language (which OCmal/F# hardly are) and has extensive libraries and frameworks compared to F#.

Flying Frog Consultancy Ltd. said...

@steve: "Feel free to describe an algorithm which supports full Hindley-Milner type inference in combination with a type system supporting overloading, overriding, sub classing, multiple inheritance and type classes". In other words, you want a type system that tries to support every academic language feature under the sun instead of choosing a pragmatic trade-off. That approach to language design is precisely what makes Scala academic and F# or Clojure industrial.

steve said...

@Jon: I begin to wonder on which planet you spent your last 30 years on. :-)

Calling overloading, overriding, sub classing, multiple inheritance and type classes "[...] academic language feature [...]" is well - surprising. These things are considered basic capabilities since decades and are used in many languages today.

Apart from that, I was sincerely interested in Jonathan's suggestion when I asked him. Sorry if you took that as an personal attack against you.

The reasons why Scala does not support every type-inference feature under the sun (funny how phrases can be twisted, isn't it?) is well understood and accepted. It was a pragmatic trade-off.

One last thing: You complained about Scala's IDE support. So far I couldn't even find a _sign_ of IDE support for F#. MonoDevelop doesn't even mention it apart from here: http://monodevelop.com/Developers/Tasks/Language_Bindings/F%23
So what is your definition of "IDE support"? Maybe we have different ideas.

Additionally I find it very irritating how you are trying to abuse Clojure in your proxy war against Scala.

Last time I looked Clojure and Scala developer had a really friendly realtionship and used each others code to improve their language/libraries/algorithms.

Maybe I missed something which happened in the mean time?

Jon, I'm tired of behavior and have decided to end the discussion with you at this point.

Please improve your social skills.

Jonathan Shore said...

@steve I meant with Clojure and F# that they integrate directly with their respective JVMs. Having used clojure, F#, and Scala, I would say F# has a the best integration, followed by clojure and then Scala.

Understandably the Scala datastructures are not java datastructures. However, it is difficult to use Scala objects from another JVM language (such as Java). It is (or was) difficult to use reflection in the context of Scala.

The nice aspect of F# when used to implement classes is that these classes are 1st class in the .NET VM and can be directly used by other languages.

I recognize that Scala is impeded by Java's poor type-system and that Clojure does not even have classes in the traditional sense.

Jonathan Shore said...

oops, I meant respective VMs not JVMs

Flying Frog Consultancy Ltd. said...

@steve: "These things are considered basic capabilities since decades and are used in many languages today". Type classes are a basic feature found in many languages today?

"So far I couldn't even find a _sign_ of IDE support for F#". Visual Studio 2010?

@Jonathan: I agree. Moreover, I think that is a direct result of F# having more industrial backing than Clojure which has far more industrial backing than Scala. Perhaps Martin Odersky's startup company will close that gap but claiming that Scala is already "foremost an industrial language" is just a triumph of hope over reality.

Matt R said...

@FFC: "Who?" Off the top of my head, the larger Scala frameworks and tools communities include: Lift, Akka, sbt, the IDE plugins for Eclipse/Netbeans/IntelliJ -- all outside of academia.

"you want a type system that tries to support every academic language feature under the sun instead of choosing a pragmatic trade-off. That approach to language design is precisely what makes Scala academic and F# or Clojure industrial."

Your argument comes across as a bit tendentious. I'd put it that Scala has merely chosen a different trade-off that not necessarily less pragmatic.

Flying Frog Consultancy Ltd. said...

@Matt R: "Your argument comes across as a bit tendentious. I'd put it that Scala has merely chosen a different trade-off that not necessarily less pragmatic". Many of Scala's core language features are completely untested in the real world. That is not true of F#, which adopted only the most tried and tested features from predecessors like OCaml. Maybe Scala's design decisions will turn out to be pragmatic but, for now, they are squarely academic whereas F# and Clojure are squarely industrial.

Matt R said...

> Maybe Scala's design decisions will
> turn out to be pragmatic but, for now,
> they are squarely academic whereas F#
> and Clojure are squarely industrial.

I don't see how you draw this distinction. The majority of Scala's language features are pilfered from OO and FP. You'd be hard pressed to argue that the OO features are "squarely aacdemic". Further, if Scala's FP features are "academic", then this would apply equally to F# and Clojure.

Flying Frog Consultancy Ltd. said...

@Matt R: "I don't see how you draw this distinction". All of the core features in F# had already been tried and tested in industry by OCaml users including Microsoft themselves. In contrast, Scala includes many features (e.g. type classes, mixins, higher kinds and even its basic syntax) that have only ever been seen before in the most esoteric academic languages.

"this would apply equally to F# and Clojure". Neither F# nor Clojure have any of these esoteric features. These languages are both based upon tried and tested cores.

Jonathan Shore said...

I'd say both languages have academic origins. That is not very important, IMO. More objective might be to measure them on a number of axis:

- performance (F# A-, Scala A-)
- language interop (F# A+, Scala B-)
- IDE support (F# A-, Scala C)
- amount of syntax (F# A+, Scala B-)
- type inference (F# A, Scala C+)
- platform interop (F# C, Scala A+)

Ok, no explanation for the above in this short comment, but something like that might be a more productive comparison.

As for whether Scala can do vs whether F# can, it does define the boundaries of what their limitations are, however, I think the above criteria are more important for general applications.

Probably the biggest handicaps for Scala are a broken generics system on the JVM and lack of value types. Perhaps with the above 2, Scala might have taken a somewhat diff dir. I get the impression that Scala was designed around the JVM and with a goal to make its syntax approachable for java devs.

Jonathan Shore said...

above comment was chopped: One para should have read:

As for whether Scala can do [esoteric algorithm] vs whether F# can, it does define the boundaries of what their limitations are, however, I think the above criteria are more important for general applications.

Matt R said...

@FFC:

> All of the core features in F# had
> already been tried and tested in
> industry by OCaml

Compared to Java/C#/C++ et al, the industrial usage of Ocaml, F#, Scala and Clojure combined is negligible. I don't really think you can take industry adoption as evidence for any of these languages being "tried and tested".

> type classes, mixins, higher kinds...
> have only ever been seen before in
> the most esoteric academic languages.

We could dispute this (mixins "only seen before in the most esoteric academic languages"? Come on!), but I'll stand by my assertion that the *majority* of Scala's language features are derived from OO and FP. Sure, there are some novel features, especially relative to mainstream languages, but I'd wager I could find some in F# and Clojure too. This is a good thing! After all, why would you go to a new language if it didn't have different features beyond those used in mainstream languages?

What do you consider the "tried and tested core" of Clojure? No Lisp has seen significant use in industry.

Flying Frog Consultancy Ltd. said...

@Jonathan: I agree except for performance due to the JVM's issues with generics and value types that you mentioned.

@Matt R: "Compared to Java/C#/C++ et al, the industrial usage of Ocaml, F#, Scala and Clojure combined is negligible". Just as the industrial use of type classes is negligible compared to the use of pattern matching.

"I don't really think you can take industry adoption as evidence for any of these languages being tried and tested". That does not follow. By the same logic, an irrelevant comparison of the popularity of C++ and milk shows that C++ is not tried and tested.

"I'll stand by my assertion that the *majority* of Scala's language features are derived from OO and FP". I fail to see the relevance of your assertion. This never had anything to do with OO vs FP.

"This is a good thing!". Only for an academic language.

"why would you go to a new language if it didn't have different features beyond those used in mainstream languages". Because we prefer to gamble on languages with features that have been tried and tested by tens of thousands of industrial users rather than on features that are fresh out of academia.

"No Lisp has seen significant use in industry". Mathematica is a Lisp derivative with millions of users that sustains a multinational company with hundreds of employees. Naughty Dog used Common Lisp in several major computer games. Microsoft used Lisp to develop the CLR. Lisps have seen significant use in industry.

Matt R said...

@FFC:

> I fail to see the relevance of your
> assertion. This never had anything
> to do with OO vs FP.

You are arguing that Scala's features are not "tried and tested". I'm countering that the majority of Scala's language features are neither experimental nor novel because they are well-known features pilfered from OO and FP languages. If you want to make an "argument by pedigree" for a language, then Scala's influences are no less industrial than those of F# or Clojure.

Flying Frog Consultancy Ltd. said...

@Matt R: "I'm countering that the majority of Scala's language features are neither experimental nor novel because they are well-known features pilfered from OO and FP languages". Higher kinds and implicit defs are obvious counter examples.

"Scala's influences are no less industrial than those of F# or Clojure". That is obviously not true.

Matt R said...

> Higher kinds and implicit defs are obvious counter examples.

My statement was about the majority of Scala's features, so a couple of counter-examples don't refute the general assertion. But even these features are not exactly crazily experimental. Higher kinds are, of course, not at all novel to Scala. Implicits are used in practice to achieve the same effect as extension methods or type classes -- again, features existing in other languages and well-understood.

> "Scala's influences are no less
> industrial than those of F# or
> Clojure". That is obviously not
> true.

I'll say it obviously *is* true! But I suspect we'll have to agree to disagree on that.

Flying Frog Consultancy Ltd. said...

@Matt R: "Higher kinds are, of course, not at all novel to Scala". Sure, they are found in at least one other extremely academic language that has never seen any serious use in industry.

"Implicits are used in practice to achieve the same effect as extension methods or type classes -- again, features existing in other languages and well-understood". How can they possibly be well understood when no more than a few dozen people in the world have ever tried to use them for real work? The only testament to their viability is an obscure academic paper by none other than Martin Odersky himself...

Matt R said...

> Sure, they are found in at
> least one other extremely
> academic language that has
> never seen any serious use in
> industry.

Well, you say that, but again, none of Haskell, Ocaml, F#, Clojure or Scala has seen serious use in industry -- not as I'd define "serious", anyway. But we've started to go in circles.

Do please feel free to have the last word, but I'll just reiterate my personal impression of Scala's "cultural context", as it were: Scala exists to integrate the historically industrial (OO) with the historically academic (FP). The language originates in academia, and people are certainly still working on PLT research with the language, but the centre of gravity of the community seems to be devolving from EPFL into the commerical sphere. I find the statement that you quoted from Martin Odersky to be more an indication of Scala's intended future velocity than its historical position.

Flying Frog Consultancy Ltd. said...

@Matt R: "none of Haskell, Ocaml, F#, Clojure or Scala has seen serious use in industry -- not as I'd define serious, anyway". Our OCaml business boomed due to the serious use of OCaml in industry. Microsoft Windows depends upon it. Intel's CPUs depend upon it. There's no way these were not serious industrial applications of OCaml.

"I find the statement that you quoted from Martin Odersky to be more an indication of Scala's intended future velocity than its historical position". Yes, I imagine that is what he meant.

jay paul said...

Really nice post, you got great blog and Thank you for sharing This excellently written content. Waiting for next one.

HP - 15.6" Laptop - 4GB Memory - 500GB Hard Drive - Black (2000-2d20nr)

HP - Pavilion TouchSmart 15.6" Touch-Screen Laptop - 4GB Memory - 750GB Hard Drive - Black

Knurd said...

In the grand scheme of things (and given the ideal intersection of time and resources):

Academic > Industrial
Theoretical > Conventional

(albeit one might argue that all theories are based upon conventions anyway, but it underscores my point, which is quite simply scope of utility)

It just so happens that software, patterns, requirements, markets, and derivatives of Candy Crush evolve too quickly for theory to payoff and rend the industrial approach expensive by comparison. I imagine once machines start writing code for us then theoretical academic languages are the only tenable option. Until then, carbon-based units rule the roost, forget things, get scared of "old code", and just want to go home early and play some Guild Wars.