Sunday, March 20, 2011

The Truthiness Is Out There

For the past 5 years or so, I (like many others) have argued that Javascript is the assembly language of the internet platform. Over this period, some of the obstacles that limit the applicability of said platform have been slowly pushed aside. Things like client side storage, or decent performance.

However, Javascript remains a seriously limited language for platform implementation. Here are some of the problems.

Concurrency primitives. There aren’t any. Now I really should be thankful for that, as the last thing I want is another shared-state concurrency threading model a la Java. And yet, ingrate that I am, I remain dissatisfied. Yes, I can write my own scheduler to provide pseudo-concurrency, but there are no primitives that let me find out how much true concurrency is available and to let me use it. Nor is there any efficient way for me to preempt an activity.

This lack of appropriate primitives for platform construction is a recurring theme. Take serialization for example. If I need to write a serializer that can incrementally store and retrieve arbitrary objects (say, because I want to implement orthogonal persistence) , I hit difficulties with things like closures. A closure in Javascript is a black box. This makes sense most of the time - but not for the system designer. One wants mechanisms that permit manipulation of the structure of all program elements - closures, prototypes, what have you.

Of course, the challenge is to do this while preserving security. Not everyone should be able to do this - but a program should be able to do it on its own objects, for example.

Another, somewhat related, problem area is stack manipulation. I want to implement an efficient debugger with fix-and-continue debugging for example. Or resumable exceptions.

Weak pointers are another problem. For example, Newspeak mixins need to track all their invocations, so that when a mixin definition is modified, all classes derived from can be updated. You’d like to use a weak collection of these mixin invocations for that purpose.

I’ve never been happy with the approach that says that the only true encapsulation mechanism in the language is closures. I find that very low level. I want objects that can hide their internals directly (private members) - and of course, I want a mechanism to get around that in some ways, so I can program the system in itself (and write things like serialization).

I miss doesNotUnderstand:, which I can emulate by going through certain hoops. There is work going on to alleviate this with proxys but I don’t see them doing what I really want. I can however, use them to implement a mechanism that does.

All of this may be too much to ask of a language where false can sometimes be interpreted as true, and where equality is non-transitive. But it isn’t too much to ask for the backbone of internet programming.

Tangent: the occasional truthiness of false is a case study in the pitfalls of language design. It stems from the interaction of two bad decisions. First, we have the implicit coercion of any type to a boolean - a nasty C legacy. Then we have primitive types, which leads to (non-transparent) autoboxing. Since any object is truthy, and autoboxing false creates an object, you can end up with an automatic, hidden conversion that interprets false as true.

I know that there is a lot of ongoing work to resolve this on the ECMAScript standards committee, whose members seem well aware of many of these issues. The timeline for addressing these problems is however, rather depressing. Between the time it takes to revise a standard, and the time it takes for it to be implemented and widely adopted (so you can actually rely on it) we may see these things fixed by the early 2020s (I kid you not).

Will that make Javascript a language a human should program in? I doubt it, but that shouldn’t be the goal. The goal should be to provide a foundation that will help in building more attractive languages on top of Javascript and the browser.

In this vein, work continues on Newspeak for the browser. We have a pretty solid Newspeak-to-Javascript compiler, though we still need to improve performance and add key platform functionality. At some point, I hope we can release this.

The vision for the Ministry of Truthiness goes beyond just a compiler of course - we want Hopscotch as well of course. Calling the DOM API from Newspeak is possible of course, but not really attractive. We also want the IDE in the browser as well. At least as much of it as possible - debugging might require using a browser extension or something due to the difficulties cited above.

Doing all this on top of Javascript has proven tedious and frustrating, and I hope things improve more quickly in the future; but we will get there in time.

19 comments:

  1. I fear you have been deceived by misleading terminology. JavaScript's equality predicate is a proper equivalence relationship, including transitivity -- it is simply not called "equality". The bĂȘte noire which is called "equality" and notated "==" is not only not equality at all, it can and should be utterly excluded from practical use of the language.

    Similarly with objects. A JavaScript native object is an a-list, and that's all it is. If you want actual objects, you will have to build them yourself out of closures, and the opaqueness of closures makes no difference if you construct the functions yourself, as you can make them self-revealing.

    ReplyDelete
  2. Hi Gilad,
    I believe you've made an error in your post. JavaScript does have concurrency in the form of callbacks. What it doesn't have is parallelism. Node.js exemplifies this. Highly concurrent yet single-threaded (the JavaScript-part of it, at least).

    As for compiling your own language to JS, I wish I was more optimistic about that. These things tend to be more like code-generators than proper compilers and give terrible feedback. Getting a JS-error-Message referring to some bit of generated code is just annoying.
    I don't have any better ideas, though, and I hope you'll prove me wrong. For now, I'll just try to get comfy with the good parts of JavaScript.

    If Google's native client takes off, that might make a nice compilation target... C is tried and true for that purpose.

    ReplyDelete
  3. Actually, there are tools out there which allow you to compile your Java code to Javascript code already. GWT is one of them. GWT integrates lately with additional widget sets, so you are not limited just to what Google provides.

    OTOH, Javascript is a much more flexible language than Java or C++. THere's a google tech talk on youtube, where the speaker calls Javascript Scheme with C-like syntax, and goes on to explain that the only reason it's got the C-like syntax is marketing, but the designer's intention was a Scheme-like language right from the start. While there are definitely several stupidities in the language, using it, once you truly understand it, can make you highly productive.

    The same video clip mentioned above states that right now Javascript is the language in which most code is written (with no comment whatsoever about the overall quality of that code). The proportion of Javascript code will probably increase even more, as web apps which move logic to the client start to replace desktop apps for business processes. I don't think most deveopers would jump on a technology they don't consider adequate for what they need.

    I don't know the tools/language/environment you are working on, which should compile to Javascript eventually. But the mere idea looks scary to me (which is why I also don't use GWT). It's more ore less like a C compiler which only knows to compile a release version of the code. In case you need to debug the resulting binary, you're in for quite some headache. Instead, I think approaches like those of ext, qooxdoo or smartclient are a lot easier to implement, and provide quite a usable solution for creating a very powerful development platform.

    ReplyDelete
  4. What do you mean by 'the assmbly language of the internat'? When assembly language was introduced it led to an order of magnitude improvement in programmer productivity over its predeccessor of straght machine code.

    It has no real limitations because you can do anything with it, but you lack the pre-built functions of higher level language.

    So if you compare assembler to languages such as Java or the script languages such as Ruby and Python Assembler is harder to write well and productivity much lower. But if you take Assembler in the context in which it evolved it was a brilliant language.

    ReplyDelete
  5. You wrote: "I (like many others) have argued that Javascript is the assembly language of the internet platform."

    It's certainly true that many have made this argument, however it also seems true that to date no-one has managed to change the Internet by revolution. It's always evolved. While there have been many attempts like GWT, CoffeeScript, Pyjamas, etc none has really taken off on an Internet scale, so I would argue from empirical evidence that Javascript is not the assembly language of the Internet, it is the programming language of the Internet.

    New languages may get to inform future revisions of Javascript, but even that is a long way from becoming reality.

    (Disclamer: I work for, but don't speak for, Mozilla)

    ReplyDelete
  6. Speaking of having the IDE in the browser, have you seen WebVelocity (VisualWorks Framework to build Seaside applications)?

    You write your process flow in Smalltalk and it gets translated at runtime to Javascript and HTML and CSS. Isnt that the direction you want things to move?

    ReplyDelete
  7. Claus,

    WebVelocity is a step in the right direction. So is Seaside. But I ultimately expect to run the live IDE in the browser, and specify the GUI at a high level with Hopscotch.

    ReplyDelete
  8. Hi,
    I may be pessimist, but I don't see JS language adopting those missing features (like concurrency primitives) before years.
    This being said, there may be another way to get those features: while JS language is evolving slowly, and not towards those features AFAIS, there is another target: the JS VM. So far, all (?) browsers have adopted a JS VM to deal with embedded JS code; then, like Java, there is room to define JS VM's *standard* bytecodes: following that perspective, developers would be authorized to link with HTML pages either JS source code (as today), or JS compiled code (bytecodes). And JS inventors might be happy too not exposing low-level, or advanced, features, like concurrency primitives, to web developers.

    ReplyDelete
  9. I agree that it may be a long time before JS supports what I want. Perhaps 2020 or so before the standards are adopted and widely deployed.

    However I see no movement toward a standard JS VM.

    A lot can happen between now and 2020 of course to mak ethis entire discussion moot.

    ReplyDelete
  10. I suspect JavaScript is too flawed to make a good web-asm.

    More promising is the direction taken by Google's Native Client - I can envisage a Newspeak implementation sitting well on that. Unfortunately, that loses all pretence of portability.

    This really emphasises the shortcomings of the Web architecture generally. It seems that currently the best option is to adopt JavaScript, for all its flaws, for client development. It is possible to use a subset of JS carefully, as argued by Doug Crockford in JavaScript: The Good Parts.

    ReplyDelete
  11. Regarding the IDE running in the web browser there is now Amber (former JTalk) - an environment close to usual Smalltalks like Pharo and others.

    See http://amber-lang.net/

    ReplyDelete
  12. Amber is a long way from a full fledged Smalltalk IDE, but I'm glad to see it is making progress. Since last I checked, Amber added support for doesNotUnderstand: which is very important. It also added support for thisContext - which tells me it is likely to be very slow.
    Many other features are non-standard.

    But the main thing is that we are seeing progress in the general direction. It's just sooo slow.

    ReplyDelete
  13. As far as I understand Newspeak basically describes Dart. Are there any other articles that describe how "Newspeak" should be implemented? :)

    ReplyDelete
  14. Alexander,

    I cannot say anything about Dart. In particular, I will not feed any of the ongoing speculation - because that's all it is, speculation. If all you are interested in is Dart, please be patient. GOTO is just 3 weeks away.

    If you are truly interested in Newspeak - well, go to newspeaklanguage.org, where you can download the system, and find links to audio and video presentations, forums, papers, a tutorial and a spec.

    ReplyDelete
  15. Well, I've kinda expected to read this :( However I really hope that Dart will realize some of the assertions made here http://markmail.org/message/uro3jtoitlmq6x7t

    ReplyDelete
  16. @Gilad Bracha

    "I agree that it may be a long time before JS supports what I want. Perhaps 2020 or so before the standards are adopted and widely deployed.

    However I see no movement toward a standard JS VM.

    A lot can happen between now and 2020 of course to mak ethis entire discussion moot."

    AFAIK, ActionScript is a superset of JavaScript, and it was executed inside its own VM, after being translated into bytecode.

    While JavaScript is today the main language of web dev, can we imagine to have both on the client side, the JS language for mainstream web dev, and a standard VM (with its standards bytecodes) for defining alternative languages for web dev ?

    Different needs put pressure on the shoulders of JS (due to it's current limits, and it's slow evolution). But as people wants other languages on the client-side *now*, and in other to offer equal treatment for all these alternatives, I don't see a better solution than introducing a standard (byte-code) VM that would be another *standard *way to support scripting inside web pages, in parallel with JavaScript. Such a VM would be a way to avoid language war as each party would be able to concentrate on its own language, after accepting such a standard VM as platform. On the contrary, today, without such a platform base, we have unproductive language wars.

    ReplyDelete
  17. @Gilad Bracha

    Let's add something (to my previous comment) to be clear enough.

    There would be still language wars, in both cases, with, or without, a (standard) bytecode VM within browsers.
    But, such a (standard) bytecode VM is mainly THE solution I see, to put some efforts in common, and to let it accepted within all browsers: as such a VM is mostly language-neutral, very few language/browser implementors/developers might consider it as a no-go.

    The other solution I see is waiting for (slow) JavaScript solution, and until then, trying to overcome/bypass current situation's drawbacks with various tools, like emscripten, Dart VM, other JavaScript++, etc. Well, we have had already Google Gears before HTML 5 rise...

    ReplyDelete
  18. I don't know if a standard VM helps. It might help certain languages, whose model fits the VM. But there are always others whose model does not. The key is having the right primitives. Certainly it might be easier to do that with a fresh start, but VM experts will tell you that its best to tailor the machine to a given language, and there is no such thing as a universal VM.

    Then there is the small matter of agreeing on the standard. It will take some years as well.

    So, 1 year and half after the initial discussion, I still don't see it happening. However, the trend for new languages for the web is of course more and more evident.

    ReplyDelete
  19. "So, 1 year and half after the initial discussion, I still don't see it happening. However, the trend for new languages for the web is of course more and more evident."

    Sure, standard VM for JS does not take off.

    But at the opposite, from my window, it just looks like every company, or community, is making +/- its own ActionScript clone (that is, JS + class-based OOP + optional typing or type inference...). So, we have different JavaScript++ :
    - ActionScript
    - Dart
    - TypeScript
    - Opa
    - Haxe
    - etc.
    versus JavaScript (as JS is still evolving).

    With all these initiatives, I am wondering how much it's going to affect mainstream dev, or to say things a bit differently, how much the multiplicity of such initiatives is going to affect going mainstream.

    But, of course, among all these initiatives, the Google's one has better chances to succeed ;-) due to Google nature.

    NB: as a side-effect, a successful JavaScript++ may kill PHP as:
    - JavaScript++ : JS + class-based OOP + optional typing or type inference
    - PHP : class-based OOP + no typing at all (~ some sort of untyped Java, with closures)
    Both (JS++ and PHP) stand somewhat between JS and Java. So, I am just innerly wondering how much space there is between JS and Java for multiple languages, and for multiple successful languages too.

    ReplyDelete