A DOMain of Shadows

One of the advantages of an internal DSL  over an external one is that you can leverage the full power of a general purpose programming language. If you create an external DSL, you may need to reinvent a slew of mechanisms that a good general purpose language would have provided you: things like modularity, inheritance, control flow and procedural abstraction.

In practice, it is unlikely that the designer of the DSL has the resources or the expertise to reinvent and reimplement all these, so the DSL is likely to be somewhat lobotomized. It may lack the facilities above entirely, or it may have very restricted versions of some of them. These restricted versions are mere shadows of the real thing; you could say that the DSL designer has created a shadow world.

I discussed this phenomenon as part of a talk I gave at Onward in 2013. This post focuses on a small part of that talk.

Here are three examples that might not always be thought of as DSLs at all, but definitely introduce a shadow world.

Shadow World 1: The module system of Standard ML.

ML modules contain type definitions. To avoid the undecidable horrors of a type of types, ML is stratified.  There is the strata of values, which is essentially a sugared lambda calculus. Then there is the stratum of modules and types. Modules are called structures, and are just records of  values and types. They are really shadow records, because at this level, by design, you can no longer perform general purpose computation. Of course, being a statically typed language, one wants to describe the types of structures. ML defines signatures for this purpose. These are shadow record types. You cannot use them to describe the types of ordinary variables.

It turns out one still wants to abstract over structures, much as one would over ordinary values. This is necessary when one wants to define parameterized modules.  However, you can’t do that with ordinary functions. ML addresses this by introducing functors, which are shadow functions. Functors can take and return structures, typed as signatures. However, functors cannot take or return functors, nor can they be recursive, directly or indirectly (otherwise we’d back to the potentially non-terminating compiler the designers of ML were trying so hard to avoid in the first place).

This means that modules can never be mutually recursive, which is unfortunate since this turns out to be a primary requirement for modularity. It isn’t a coincidence that we use circuits for electrical systems and communication systems, to name two prominent examples.  

It also means that we can’t use the power of higher order functions to structure our modules. Given that the whole language is predicated on higher order functions as the main structuring device, this is oddly ironic.

There is a lot of published research on overcoming these limitations. There are papers about supporting restricted forms of mutual recursion among ML modules.  There are papers about allowing higher-order functors. There are papers about combining them. These papers are extremely ingenious and the people who wrote them are absolutely brilliant. But these papers are also mind-bogglingly complex.  

I believe it would be much better to simply treat modules as ordinary values. Then, either forego types as module elements entirely (as in Newspeak)  or live with the potential of an infinite loop in the compiler. As a practical matter, you can set a time or depth limit in the compiler rather than insist on decidability.  I see this as a pretty clear cut case for first class values rather than shadow worlds.

Shadow World 2: Polymer

Polymer is an emerging web standard that aims to bring a modicum of solace to those poor mistreated souls known as web programmers. In particular, it aims to allow them to use component based UIs in a standardized way.

In the Polymer world, one can follow a clean MVC style separation for views from controllers. The views are defined in HTML, while the controllers are defined in an actual programming language - typically Javascript, but one can also use Dart and there will no doubt be others. All this represents a big step forward for HTML, but it remains deeply unsatisfactory from a programming language viewpoint.

The thing is, you can’t really write arbitrary views in HTML. For example, maybe your view has to decide whether to show a UI element based on program logic or state. Hence you need a conditional construct. You may have heard of these: things like if statements or the ?: operator. So we have to add shadow conditionals.

<template if="{{usingForm}}">

is how you’d express  

if (usingForm) someComponent;

In a world where programmers cry havoc over having to type a semicolon, it’s interesting how people accept this. However, it isn’t the verbose, noisy syntax that is the main issue.

The conditional construct doesn’t come with an else of elsif clause, nor is their a switch or case. So if you have a series of alternatives such as

if (cond1) {ui1}

else if (cond2) {ui2}

        else {ui3}

You have to write

<template if = "{{cond1}}">

<ui1>

</template>

<template if = "{{cond2 && !cond1}}">

<ui2>

</template>

<template if = "{{cond3 && !cond2 && !cond3}"}>

<ui3>

</template>

A UI might have to display a varying number of elements, depending on the size of a data structure in the underlying program. Maybe it needs to repeat the display of a row in a database N times, depending on the amount of data. We use loops for this in real programming. So we now need shadow loops.

<template repeat = "{{task in current}}">

There’s also a for loop

<template repeat= "{{ foo, i in foos }}">

Of course one needs to access the underlying data from the controller or model, and so we need a way to reference variables. So we have shadow variables like

{{usingForm}} 

and shadow property access.

{{current.length}}

Given that we are building components, we need to use components built by others, and the conventional solution to this is imports. And so we add shadow imports.

<link rel = "import” href = "...">

UI components are a classic use case for inheritance, and polymer components support  can be derived from each other, starting with the predefined elements of the DOM, via shadow inheritance.  It is only a matter of time before someone realizes they would like to reuse properties from other components in different hierarchies via shadow mixins.

By now we’ve defined a whole shadow language, represented as a series of ad hoc constructions embedded in string-valued attributes of HTML.  A key strength of HTML is supposed to be ease-of-use for non-programmers (this is often described by the meaningless phrase declarative). Once you have added all this machinery, you’ve lost that alleged ease of use - but you don’t have a real programming language either. 

Shadow World 3: Imports

Imports themselves are a kind of shadow language even in a real programming language. Of course imports have other flaws, as I’ve discussed  here and here, but that is not my focus today. Whenever you have imports, you find demands for conditional imports, for an aliasing mechanism (import-as) for a form of iteration (wildcards).  All these mechanisms already exist in the underlying language and yet they are typically unavailable because imports are second-class constructs.

Beyond Criticism

It is very easy to criticize other people’s work. To quote Mark Twain:

I believe that the trade of critic, in literature, music, and the drama, is the most degraded of all trades, and that it has no real value 

So I had better offer some constructive alternative to these shadow languages. With respect to modularity, Newspeak is my answer. With respect to UI, something along the lines of the Hopscotch UI framework is how I’d like to tackle the problem. In that area, we still have significant work to do on data binding, which is one of the greatest strengths of polymer. In any case, I plan to devote a separate post to show how one can build an internal DSL for UI inside a clean programming language. 

The point of this post is to highlight the inherent cost of going the shadow route. Shadow worlds come in to being in various ways. One way is when we introduce second class constructs because we are reluctant to face up to the price of making something a real value. This is the case in the module and import scenarios above. Another way is when one defines an external DSL (as in the HTML/Polymer example). In all these cases, one will always find that the shadows are lacking. 

Let’s try and do better.

Taking it to Th’emacs

Emacs is my preferred text editor. I don’t use old-fashioned text editors as much as I used to, because I often need more specialized tools.  I use IDEs for various programming languages and other things when producing high quality documents. And yet, I often wish I could subsume these with a tool that had the basic goodness of emacs.

What makes emacs interesting all these decades after its inception is not what keyboard shortcuts it supports or what its basic editing functionality is. Rather, what matters are some of its underlying design principles.

• Emacs has built-in scripting language. 
• That language (elisp), while it has many flaws, is a flexible, dynamic, and rather general purpose language. 
• All of the editor’s functionality is exposed via APIs written in the scripting language.

As a consequence, you can control everything emacs does programmatically. This makes emacs extensible in a way that is far deeper and more powerful than a plug-in architecture. A plug-in architecture must anticipate every desired extension to the system. Since it never can, it always disappoints in the end. Along the way it grows ever more complex and bloated in its futile attempt to foresee every possible need. With a language, you can code any extension you need.

If the scripting language is truly dynamic, and allows you to not only extend but also modify the running system, the possibilities are truly unlimited.

The points above are not limited to editors. They are fundamentals of system design.

There was a time when even Microsoft recognized this, making apps that could be programmed via VisualBasic. Sadly, they concocted a security nightmare, because malware can also control your application. Which is why being able to secure your application’s scripting language is critical as well. 

Last June, I spoke at QCon NY, and demonstrated a number of interesting web based systems that had some these properties (as well as a couple that did not, but were interesting for other reasons):

• Leisure is a presentation manager that incorporates a lazy functional programming language. Modulo some person-years of engineering, it is to PowerPoint what emacs is to NotePad. 
• The Lively Kernel could be thought of as a GUI builder scripted via Javascript; this an insult to Lively of course, because it is much more than that.  In Lively, the GUI is the GUI builder; every GUI you make is extensible and modifiable in the same way.
• Lastly I showed Minibrowser, a prototype web based IDE for Newspeak. Like all IDEs in the Smalltalk tradition, it can be extended and modified from within itself.

We really need an Emacs for the modern age.  An editor, surely, but one that lets you edit rich text, images, audio and video. In fact, you should be able to embed arbitrary widgets. And of course it needs to be scriptable I just explained. So you might evaluate code that creates a UI element and inserts into the editor.  

Now you can make the editor modify its own GUI. In fact, the editor can be extended into a general purpose GUI builder just like Lively. And every such GUI can modify itself if you wish; sometimes you may wish to modify it so it can no longer modify itself, and then you have a frozen application. Your editor has become an IDE. In fact, it is a live literate programming environment. 

If the editor’s scripting language interoperates well with the surrounding environment, it can be used to control the computer and everything the computer itself controls. You can check in to the environment and hardly ever leave. You can lead your cyber life in it.: email, social media, live chats, streaming audio and video can all be incorporated. Moreover they can all be controlled and customized by you, the lucky user. 

To a degree, Lively is such an editor. It’s biggest drawbacks are a lack of polish due to lack of engineering resources and that its scripting language is Javascript.

Now, imagine that the editor was polished and robust. Even more importantly the code you created in this environment was modular and secure and written in an elegant and principled language. Imagine you could deploy the same code either on the web, or natively on both desktop and mobile. Imagine that the applications built with the language support online and offline use out of the box, automatically synchronizing data and code between clients and servers. Imagine that they have built-in support for collaboration, either syncing in real time or merging offline as required. 

Of course, it is the vision of such a language and platform that has always motivated the Newspeak project.  I have discussed many of these points before. In particular, I’ve talked about the weaknesses of traditional IDEs (see for example this post and this one), and the need for a platform that supports synchronization over the net (here and again here) for a long time. Yet the message bears repeating. 

Newspeak on Dart

We've wanted to run Newspeak to run in a web browser for a long time - actually, since before the Newspeak project started.  The need for better programming languages for the internet platform has been evident for a while.  In a better world, the Newspeak project would be focused on that from day 1.  

However, this world is suboptimal.  It was only in 2010 that I spent some time sketching out the first version of NS2JS, the Newspeak-to-Javascript compiler. It was a toy, but in 2011, Vassili Bykov managed to bring it to the verge of self hosting. Performance, however, was abysmal.  Having since joined the Dart project and learned more and more about the challenges Javascript poses as a compilation target (or anything else) I guess this really should not have been a surprise.

We were able to reuse a part of that effort this summer when Ryan Macnak came to Google in Mountain View as an intern with the Dart team. His mission: NS2Dart, a Newspeak to Dart compiler. A detailed report is available at

https://docs.google.com/document/d/1pU_nautpK49pJzwZkJM1NhACcad0Hjq0rjIomLSWQ1Y/edit

The results are promising, though there is still a long way to go. Despite considerable losses due to impedance mismatches, on the Dart VM, NS2JS comes within a factor of 2 of the Squeak based Newspeak implementation (NS2Squeak). Given that the current version of NS2Squeak is twice the speed it used to be (thanks to Cog)  it looks like performance is tolerable already. And since Dart will get much faster (the numbers in the tech report are already out of date), the future looks bright.

Now, when someone says the future looks bright, you should be getting nervous. So, just sign the dotted line, and then I'll get to the caveats ...

The current version relies on access to Dart VM's embedding API, which means you cannot run it in the browser. Hopefully, over time enough functionality will get added in the way of mixins and mirror builders that the same results can be accomplished in pure Dart. That is not yet certain, but let's retain the uncharacteristically optimistic tone of this post, and assume that does happen.

Since most browsers are likely to run Dart via translation to Javascript for some time, we need to look at the NS-Dart-JS pipeline: compiling Newspeak via NS2Dart, and feeding the result to dart2js (The Dart to Javascript compiler) to obtain Javascript code that runs in any browser.

Early on, NS2Dart could run on top of dart2js (by cheating on things like dynamic mixin application) and results were much better than with NS2JS. This is to be expected: dart2js does the heavy lifting for us.  The dart2js compiler already has 4-5 times as much code as NS2JS and is being developed by a very skilled team who understand JS performance inside and out.  Furthermore, it has taken a while to get to this point and performance work is by no means done yet.  So I don't feel too bad about our efforts on NS2JS.

Nevertheless, there is a lot of uncertainty over how well Newspeak can run on top of current Javascript implementations. A key problem area are non-local returns (NLRs). These are essential for user defined control constructs. They are not supported in Dart precisely because it is not clear how efficiently they can be implemented on top of Javascript.

Mirror support on NS2Dart is very partial - the mapping to Dart's mirrors is pretty clean so far, but Dart mirrors are a work in progress, and so far only cover introspection. 

Then there remains the small matter of the UI. I'd love to see the Hopscotch GUI on the browser, but that is a ton of work as well.  We'll see what UI solutions shape up in that space - we can always just call out to whatever UI Dart exposes (Ryan also implemented a Dart Alien API). So altogether, it might be a tad premature to declare victory, but we are making progress.

The dream remains to get a fully functioning system, Hopscotch, IDE and all that works well in the browser. However, I can imagine that many applications could get by with a lot less.  

The details are in Ryan's report and the code is in the Newspeak repo, where you can also get the latest Newspeak images with which to view it (I recommend you do that rather than use the old release; a new release will be out very soon). Thanks are due  to Vassili (whose work on NS2JS made this possible)  and to my colleagues on the Dart team, who graciously supported this effort in various ways, and most of all to Ryan.

Altogether, getting a top quality programming experience on the web requires a major effort.   Dart is beginning to make this possible.

The Truthiness Is Out There

For the past 5 years or so, I (like many others) have argued that Javascript is the assembly language of the internet platform. Over this period, some of the obstacles that limit the applicability of said platform have been slowly pushed aside. Things like client side storage, or decent performance.

However, Javascript remains a seriously limited language for platform implementation. Here are some of the problems.

Concurrency primitives. There aren’t any. Now I really should be thankful for that, as the last thing I want is another shared-state concurrency threading model a la Java. And yet, ingrate that I am, I remain dissatisfied. Yes, I can write my own scheduler to provide pseudo-concurrency, but there are no primitives that let me find out how much true concurrency is available and to let me use it. Nor is there any efficient way for me to preempt an activity.

This lack of appropriate primitives for platform construction is a recurring theme. Take serialization for example. If I need to write a serializer that can incrementally store and retrieve arbitrary objects (say, because I want to implement orthogonal persistence) , I hit difficulties with things like closures. A closure in Javascript is a black box. This makes sense most of the time - but not for the system designer. One wants mechanisms that permit manipulation of the structure of all program elements - closures, prototypes, what have you.

Of course, the challenge is to do this while preserving security. Not everyone should be able to do this - but a program should be able to do it on its own objects, for example.

Another, somewhat related, problem area is stack manipulation. I want to implement an efficient debugger with fix-and-continue debugging for example. Or resumable exceptions.

Weak pointers are another problem. For example, Newspeak mixins need to track all their invocations, so that when a mixin definition is modified, all classes derived from can be updated. You’d like to use a weak collection of these mixin invocations for that purpose.

I’ve never been happy with the approach that says that the only true encapsulation mechanism in the language is closures. I find that very low level. I want objects that can hide their internals directly (private members) - and of course, I want a mechanism to get around that in some ways, so I can program the system in itself (and write things like serialization).

I miss doesNotUnderstand:, which I can emulate by going through certain hoops. There is work going on to alleviate this with proxys but I don’t see them doing what I really want. I can however, use them to implement a mechanism that does.

All of this may be too much to ask of a language where false can sometimes be interpreted as true, and where equality is non-transitive. But it isn’t too much to ask for the backbone of internet programming.

Tangent: the occasional truthiness of false is a case study in the pitfalls of language design. It stems from the interaction of two bad decisions. First, we have the implicit coercion of any type to a boolean - a nasty C legacy. Then we have primitive types, which leads to (non-transparent) autoboxing. Since any object is truthy, and autoboxing false creates an object, you can end up with an automatic, hidden conversion that interprets false as true.

I know that there is a lot of ongoing work to resolve this on the ECMAScript standards committee, whose members seem well aware of many of these issues. The timeline for addressing these problems is however, rather depressing. Between the time it takes to revise a standard, and the time it takes for it to be implemented and widely adopted (so you can actually rely on it) we may see these things fixed by the early 2020s (I kid you not).

Will that make Javascript a language a human should program in? I doubt it, but that shouldn’t be the goal. The goal should be to provide a foundation that will help in building more attractive languages on top of Javascript and the browser.

In this vein, work continues on Newspeak for the browser. We have a pretty solid Newspeak-to-Javascript compiler, though we still need to improve performance and add key platform functionality. At some point, I hope we can release this.

The vision for the Ministry of Truthiness goes beyond just a compiler of course - we want Hopscotch as well of course. Calling the DOM API from Newspeak is possible of course, but not really attractive. We also want the IDE in the browser as well. At least as much of it as possible - debugging might require using a browser extension or something due to the difficulties cited above.

Doing all this on top of Javascript has proven tedious and frustrating, and I hope things improve more quickly in the future; but we will get there in time.

The Brave New World of Full Service Computing

I’ve been trying to explain this for some six years with little success, but I’ll try again. It may be easier this time.

For the past generation, we have been living in a world of self-service computing, more commonly knows as personal computing. The person using/owning a personal/self service computer is responsible for managing that computer themselves. All the digital chores - installing and updating software, preventing malware infection, backing up storage etc. are the user’s responsibility. Most users hate that.

Worse, most users really can’t handle these chores at all. They run ancient versions of applications because they are deathly afraid to tamper with anything that somehow works. Their computers are warrens of digital disease, crawling with viruses. Their data is rarely backed up properly.

The evolution of network technology enables a new model: full service computing. Full service computing means that you abdicate a level of fine control over the computer to a service that handles your digital housekeeping. I discussed some implications of this in a post back in January.

You still choose applications that you want - but you must get them from the service; the service installs these applications for you; it keeps them up to date as long as you use the service; and it it ensures that they are safe to use. The service also backs up your data automagically, allowing you to easily search through old versions as needed. In the extreme case, the service actually provides you with the computers you use. That helps it ensure seamless software/hardware compatibility.

We can increasingly see vignettes of this brave new world. Web apps running in the cloud are one step in this direction; web apps storing data locally are a step further. A complete platform designed to hide the raw machine from the user is even closer to this vision. Such platforms started out in the world of mobile phones, and are clawing their way upwards from there: Android and iPhone/iPad. Windows 7 mobile is quite similar to the iPhone model (odd, isn’t it?). And ChromeOS is not dissimilar.

Tangent: How does Google keep Android and ChromeOS aligned? It doesn’t - I believe they are making it up as they go along. Eventually they should converge.

We still haven't seen an integration of the app store model with backup, but it can't be that far off. An option for MobileMe(insert Apple trademark here) subscribers to have an image of their iPad backed up on Apple's servers and browsable via a Time Machine (and again, Apple trademark) style interface is easy to imagine. This can then extend to data you want to share collaboratively, while still retaining a local copy for offline use.

In time, most users will gravitate away from the self service computers they use today to software service platforms. This goes hand in hand with simpler UIs such as the iPad’s. One of the technical features in this evolution is the disappearance of the file system in favor of a searchable database of objects, as I mentioned in a previous post.

Tangent: The move from personal computing to software services accessed via simpler devices is what is truly significant about the iPad. Those who worry about details such as whether it has a camera or Flash are completely missing the point.

All this is already happening, which makes it easier to explain the idea of objects as software services (see the video) now than in 2004/2005. Consider the trend of programming languages toward a pure object model: all data are objects.

Tangent: This is orthogonal to functional programming. Objects can be immutable, as in, e.g., Avarice.

So, we now have programs dealing with objects in primary storage running on platforms whose secondary storage consists of objects as well. Initially, programmers find that they must explicitly manage the transition of data from primary to secondary storage.

The programmer’s task is simplified if that transition can be automated. This is an old idea known as orthogonal persistence. The Achilles’ heel of orthogonal persistence was the divergence of in-memory data representation and persistent data representation. This argued for manual management of the transition between primary and secondary storage.

However, circumstances are changing. The difference between transient (in-memory) and persistent (secondary storage) representations was driven by several factors, none of which apply in the full service computing model.

The first factor was that programs changed their internal representation while persistent data lived forever. Orthogonal persistence broke down when old data became incompatible with program data structures.

In our brave new world, programs co-evolve with their data so this is not an issue. The service manages both the application and secondary storage, updating them in lock step. Thus, we have data that is not only orthogonally persistent, but orthogonally synchronized.

The second factor was that persistent data formats were a medium of exchange between different programs with different internal data representations. This relied on the file system, may it rest in peace. The new applications are in fact compelled to keep their persistent data in silos tied to the application, and communicate with other applications in constrained ways.

This sets the stage for a model where programs are written in a purely object oriented language that supports orthogonal synchronization of both programs and data. This model eases the programmer’s task, and sits perfectly in the world of full service computing.

I have been discussing language support for this vision of software services publicly since my talk at DLS 2005, and privately long before. Newspeak’s modularity and reflection features dovetail perfectly with that.

Newspeak allows programs to be represented as objects; these objects are compact and self contained, thanks to the modularity of Newspeak code. Newspeak programs can be updated on the fly, atomically. Together, these two features provide a foundation for synchronizing programs. And the Newspeak VM supports shallow object immutability, which helps us synchronize data. On this basis, we have implemented experimental support for orthogonal synchronization, though it is very immature. I hope that in the future, we can provide an open software service based on this idea.

Of course, the radical edge the objects as software services vision is the complete abolishment of versions for libraries as well as applications. That is, I admit, untested, controversial and futuristic. However, with respect to applications, it is already starting to happen.

Closing the Frontier?

How is a programming language interpreter like pornography? They’re both banned at the iPhone App store. The restrictions on interpreters disturb me (this blog does not deal with the other topic).

What if I want to write an iPhone app in my favorite programming language? Or my second favorite? The freedom to program in the language of my choice is a big concern of mine. Even though Squeak Smalltalk runs on the iPhone, and Newspeak runs on top of Squeak, I’m out of luck. Packaging the entire runtime with each app is not attractive.

Tangent: In principle, one should be able to write a compiler that spits out Objective-C. After all, Objective-C incorporates Smalltalk style message sends, down to the keyword syntax. In practice, Objective-C on the iPhone doesn’t support true garbage collection, so it just seems too painful. This may change soon - maybe as soon as Wednesday., with iPhone OS 4 and/or the iTablet (pick your favorite rumor).

The iPhone is a prime example of a trend where our computing platforms become more restricted. As we move toward software as a service rather than an artifact, the computer is no longer as personal; it is very much under control of the service provider. In this case Apple, in other cases Amazon or Google or Microsoft. I’d be surprised if the rumored iTablet won’t work on the same model: rather than an open version of MacOS, a semi-closed world with an app store.

I completely understand why service providers place these restrictions. What if my app is a storefront for apps? Whither the app store’s revenue? And then there are security concerns.

On the iPhone, there are administrative restrictions. Other platforms are more open in that respect - but you tend to run into technical problems. Take Android - you can distribute a language runtime, but it is the Javanese VM that’s in charge.

No one will prevent you from implementing your own system on a JVM or on .Net. However, the VM, by its architecture, will make things difficult. If it does not support the right primitives, you find that your language is a second or third class citizen.

Examples: the typed VM instruction sets are a problem for dynamically typed languages. On .Net, the DLR provides some support, but you’ll never get to peak performance using it as it stands today. On the JVM, invokedynamic will ship some sunny day, and that will be better. Even then, if you want real dynamism, you’ll find it very hard. How do you change the class of an object? How do you change the shape of a class? Or its superclass? The solutions are convoluted and/or suboptimal.

On the desktop, you can choose to bypass these platforms, but the action is increasingly moving elsewhere. Looking ahead, the dominant platforms won’t be traditional operating systems like Windows, MacOS and Linux; far fewer people will use them directly than today. Instead, we’ll have the web (as in, e.g., ChromeOS), and semi-enclosed service platforms: The moral equivalent of iTablet OS (regardless of what Apple does this Wednesday). The “real OS” may still be there underneath, but virtually no one will care.

Each of these service platforms will combine a client side, running on phones, tablets, e-books, laptops (e.g., iPhone, Android, Kindle, ChromeOS) and desktops; and a server side (e.g., Azure, Google, E3, iTunes/MobileMe) supporting storage/backup, software update and distribution and services we haven't invented yet.

The most open of these virtual platforms is of course, the web browser. Mercifully, it is relatively free of administrative restrictions and its machine language is much more flexible than MSIL or JVM byte codes (JVML). Which is not to say that you don’t have to go through some very convoluted and costly hoops. Think of weak pointers/finalization; or stack manipulation (debuggers, tail recursion) etc.

There are two threads here:

1. The narrow technical one, which is about providing abstractions that are good enough to support alternate ways of doing things. One wants flexible designs, especially in the languages that are supposed to be at the heart of general purpose open platforms. Of course, this is much easier said than done.
2. The broad technological/social/commercial one: the trend toward software services and the cloud is pulling us away from the personal computer and the individual control it entails. Of course, one of the great attractions here is the idea of a service that relieves much of the responsibility for digital housekeeping. The vast majority of people don’t want all that control with the trouble that goes with it. Balancing this with the freedom to innovate at the platform and language level, and the freedom to choose among languages, is hard.
   

Long term, the platforms that have more flexibility will benefit more from new ideas, which gives me a basis for unnatural optimism :-)

The Language Designer’s Dilemma

Last week I was at lang.net 09 and DSL Dev Con. It was great fun, as always. The best talk was by Erik Meijer. Use the link to see it - but without live video of Erik in action, you can't really understand what it was like. Such is performance art.

I also gave a talk, and there were many others, but that is not the point of this post. Rather, this post was prompted by one specific, and excellent, talk - Lars Bak’s presentation on V8. Lars clearly enjoyed his visit to the lion’s den; more importantly, perhaps Microsoft will finally shake off their apparent paralysis and produce a competitive Javascript engine.

That, in turn, will make it possible to distribute serious applications targeting the web browser, with the knowledge that all major browsers have performant Javascript engines that can carry the load.

It’s all part of the evolution of the web browser into an OS. This was what Netscape foresaw back in 1995. And it is a perfect example of innovator’s dilemma.

Innovator’s dilemma applies very directly to programming languages, and Todd Proebsting already discussed this back in 2002.

To recap, the basic idea is that established players in a market support sustaining innovation but not disruptive innovation. Examples of sustaining innovation would be the difference between Windows 2000 and Windows XP, or between Java 1.0 thru Java 6.
Sustaining innovations are gradual improvements and refinements - some improvement in performance, or some small new feature etc.

Disruptive innovation tends to appear “off the radar” of the established players. It is often inferior to the established product in many ways. It tends to start in some small niche where it happens to work better than the established technology. The majors ignore it, as it clearly can’t handle the “real” problems they are focused on. Over time, the new technology grows more competent and eats its way upward, consuming the previous market leader’s lunch.

We’ve seen this happen many times before. Remember Sun workstations? PCs came in and took over that market. Sun retreated to the server business, and PC’s chase it up towards the high end of that market, until there’s nowhere left to run.

In the programming language space, take Ruby as an example. Ruby is slow, until quite recently lacked IDE support etc. It’s easy for the Javanese to dismiss it. Over time, such technology can evolve to be much faster, have better tooling etc. And so it may grow into Java’s main market.

Don’t believe me? Java in 1995 was just a language for writing applets. It’s performance was poorer than Smalltalk’s, and nowhere near that of C++. There were no IDEs. Who's laughing now?

Javascript is an even clearer case in point. It was just a scripting language for web browsers. Incredibly slow implementations, restricted to interacting with the just the browser. No debuggers, no tools, no software engineering support to speak of.

Then V8 came along and showed people that it can be much faster. Lars’ has doubled performance since the release in September, and expects to double it again within a year, and again 2-3 years after that.

Javascript security and modularity are evolving as well. By the time that’s done, I suspect it will far outstrip clunky packaging mechanisms like OSGi. This doesn’t mean you have to write everything in Javascript - I don’t believe in a monolingual world.

Tangent: Yes, I did spend ten years at Sun. A lot of it was spent arguing that Java was not the final step in the evolution of programming languages. I guess I’m just not very persuasive.

The web browser itself is one of the most disruptive technologies in computing. It is poised to become the OS. Javascript performance is an essential ingredient but there are others. SVG provides a graphics model. HTML 5 brings with it persistent storage - a file system, as it were - only better. You may recall that Vista was supposed to have a database like file system. Vista didn’t deliver, but web browsers will.

This trend seems to make the traditional OS irrelevant. Who cares what’s managing the disk drive? It’s just some commoditized component, like a power supply. We can already see how netbooks are prospering. This isn’t good news for Windows.

Of course, you might ask why Microsoft would go along with this? Is IE’s Javascript so inadequate on purpose? Maybe it is part of the master plan? Well, I don’t believe in conspiracy theories.

It does look as if Microsoft is facing a lose-lose proposition. Making IE competitive supports the movement toward a web-browser-as-OS world. Conversely, if they let IE languish, as they have, they can expect its market share to continue to drop.

However, you cannot stop the trend toward the Web OS by making your tools inferior. You can only compete by innovating, not by standing still.

I wouldn’t count Redmond out just yet. They have huge assets - a terrific research lab, armies of smart people in product land as well, market dominance, and vast amounts of money. They also have huge liabilities of course - and those add up to innovator’s dilemma.

In any case, for language implementors, it’s clear that one needs to be able to compile to the internet platform and Javascript is its assembly language. Web programming will evolve into general purpose programming, using the persistent storage provided by the browser to function off line as well as online.

As many readers of this blog will recognize, this sets the stage for the brave new world of objects as software services. I hope to bring Newspeak to the point where it is an ideal language for this coming world. It’s a bit difficult with the very limited resources at our disposal, but we are making progress.

The entire progression from conventional computing to the world of the Web OS is taking longer than one expects, but is happening. The time will come when we hit the tipping point, and things will change very rapidly.

java'scrypt

Everyone is talking about cloud computing these days; I should add some vapor to the mist.

I began thinking seriously about the topic a bit after April fools day 2004, when gmail was released. I started using gmail shortly therefater. After about 30 minutes, I was convinced that web clients had a long way to go.

It is true that gmail showed the world that you could do far more with javascript in a browser than I or most other people had realized.

Javascript and Ajax have come a long way since then. For an impressive demonstration of what’s possible, see the Lively Kernel, a realization of John Maloney’s Morphic GUI ideas of Self and Squeak, in Javascript. It only works in Safari 3.0 and some advanced Firefox builds, but that situation will improve in time.

Javascript’s performance remains an issue; this will also improve dramatically in the foreseeable future. However, with all the talk about computing in the cloud, I think some caveats are in order.

Many real applications will need to be used offline, and so will need to store state on the client. They are NOT going to be pure server side web applications. Google has finally acknowledged as much with the introduction of Google Gears. Microsoft’s LiveMesh naturally emphasizes this even more.

I say naturally, because Microsoft has an interest in the personal computer, while Google has an interest in taking us to an updated version of X terminals and 1960’s time sharing.
Of course, there are other technologies in this space, like Adobe Air. Macromedia (now part of Adobe) saw this coming earlier than most, going back to at least 2003 with its Central project.

With all these pieces in place, we may finally have the underlying elements necessary to build good applications delivered through the web browser.

Why has this taken so long? In theory, Java was supposed to deliver all this over ten years ago. Java started as a client technology, and applets were supposed to do exactly what advanced Ajax applications do today, only better.

Of course, the problem was that Java clients behaved very poorly; rather than fix the client technology, Sun captured the lucrative enterprise business. To understand why Sun behaved that way, why it had to behave that way, read The Innovator’s Dilemma.
Maybe I’ll expand on that theme in another post.

The result of Java’s failure on the client was that the entire vision of rich network-enabled client applications was put on hold for a decade. It is now once again coming to the fore, but there are other languages in that space now, Javascript chief among them.

I’m not advocating writing clients in Javascript directly. Javascript is the assembly language of the internet platform (and the browser is the OS). It’s flexible enough to support almost anything on top of it, and poorly structured enough that humans shouldn’t be writing sizable applications in it.

I am not in favor of the attempts to make Javascript 2 be another Java either. Javascript’s importance stems from the existing version’s position in the browser, and from its flexibility. It should be kept simple. Javascript's value increases the more uniformly it is implemented across browsers. A new, complex language will take a long time to reach that point.

People should program in a variety of languages that suit them, and have these compiled to Javascript and HTML (GWT is an example of that, though not one I really like). One shouldn't have to deal with the hodgepodge of Javascript and HTML (broadly defined to include XML, CSS etc.) that constitute web programming today.

I don’t expect native clients to go away any time soon though. They will provide a better experience than the browser in many ways for quite a long time. The advantage of writing in a clean, high level language and platform is that it can be compiled to run either on the “WebOS” or on a native OS.

What one wants to see in such a platform is the ability to take advantage of the network in a deep way - for software delivery and update and for data synchronization. Which brings me back to my theme of objects as software services (see the video), which I’ve been preaching publicly since 2005 ( see my OOPSLA 2005 presentation).

One should be able to write an application once, and deploy it both to the web browser and natively to any major platform. In all those scenarios, one should be able to update the software and data on the fly, run it on- and offl-ine and and monitor it over the network continuously. And all this needs to be done in a way that preserves user’s security. This last bit is still a big challenge.

The technology to do this is becoming available, even though everything takes much longer than one expects. The vision has its root in research in the 1980s, continued in the 1990s with Java (and others, now largely forgotten, like General Magic's Telescript) and will be probably be realized in the 2010s, per Alan Kay’s 30 year rule.

SOBs

A lurid headline is a proven way of grabbing attention. SOBs are Serviced Objects, objects that can be downloaded to your machine from a server, and thereafter serviced by it. Any modifications you make to the SOB get saved on the server, and any updates to the SOB by the service provider get delivered to you automatically. The former provides the SOB (and you) with backup, audit trail, and sharing across multiple users or devices. The latter provides for bug fixes and feature updates to the SOB on an ongoing basis.

All of this sounds suspiciously like a web app, except the fact that the SOB gets downloaded - which means it is available off line, can use all the features of your machine and in general doesn’t suffer the limitations of web apps. Another important distinction is that the core underlying technology is hotswapping, so the applications don’t need to be restarted when updated.

I’ve given a number of talks about this over the past few years; the latest was at Google earlier this month. Since that talk is available via Google Video, I thought I’d bring up the topic here.

The term I usually use for this idea is Objects as Software Services. It is more descriptive than SOB, but could be confused with generic stuff like SOAs and conventional web apps. Another descriptive name would be Live Network-Serviced Applications (LNSA). The “live” distinguishes them from most existing NSAs, which typically need to be taken down before being updated.

This leads to us to NSOOP, Network-Serviced Object Oriented Programming, and its support via NSOOPLs (NSOOP Languages). Though I just made these terms up, that is a big part of what the talk is about: language and platform features that enable LNSAs.

If this interests you, please see the talk. There are also earlier slides, and a brief write up.