One of the advantages of an internal DSL over an external one is that you can leverage the full power of a general purpose programming language. If you create an external DSL, you may need to reinvent a slew of mechanisms that a good general purpose language would have provided you: things like modularity, inheritance, control flow and procedural abstraction.
In practice, it is unlikely that the designer of the DSL has the resources or the expertise to reinvent and reimplement all these, so the DSL is likely to be somewhat lobotomized. It may lack the facilities above entirely, or it may have very restricted versions of some of them. These restricted versions are mere shadows of the real thing; you could say that the DSL designer has created a shadow world.
I discussed this phenomenon as part of a talk I gave at Onward in 2013. This post focuses on a small part of that talk.
Here are three examples that might not always be thought of as DSLs at all, but definitely introduce a shadow world.
Shadow World 1: The module system of Standard ML.
ML modules contain type definitions. To avoid the undecidable horrors of a type of types, ML is stratified. There is the strata of values, which is essentially a sugared lambda calculus. Then there is the stratum of modules and types. Modules are called structures, and are just records of values and types. They are really shadow records, because at this level, by design, you can no longer perform general purpose computation. Of course, being a statically typed language, one wants to describe the types of structures. ML defines signatures for this purpose. These are shadow record types. You cannot use them to describe the types of ordinary variables.
It turns out one still wants to abstract over structures, much as one would over ordinary values. This is necessary when one wants to define parameterized modules. However, you can’t do that with ordinary functions. ML addresses this by introducing functors, which are shadow functions. Functors can take and return structures, typed as signatures. However, functors cannot take or return functors, nor can they be recursive, directly or indirectly (otherwise we’d back to the potentially non-terminating compiler the designers of ML were trying so hard to avoid in the first place).
This means that modules can never be mutually recursive, which is unfortunate since this turns out to be a primary requirement for modularity. It isn’t a coincidence that we use circuits for electrical systems and communication systems, to name two prominent examples.
It also means that we can’t use the power of higher order functions to structure our modules. Given that the whole language is predicated on higher order functions as the main structuring device, this is oddly ironic.
There is a lot of published research on overcoming these limitations. There are papers about supporting restricted forms of mutual recursion among ML modules. There are papers about allowing higher-order functors. There are papers about combining them. These papers are extremely ingenious and the people who wrote them are absolutely brilliant. But these papers are also mind-bogglingly complex.
I believe it would be much better to simply treat modules as ordinary values. Then, either forego types as module elements entirely (as in Newspeak) or live with the potential of an infinite loop in the compiler. As a practical matter, you can set a time or depth limit in the compiler rather than insist on decidability. I see this as a pretty clear cut case for first class values rather than shadow worlds.
Shadow World 2: Polymer
Polymer is an emerging web standard that aims to bring a modicum of solace to those poor mistreated souls known as web programmers. In particular, it aims to allow them to use component based UIs in a standardized way.
In the Polymer world, one can follow a clean MVC style separation for views from controllers. The views are defined in HTML, while the controllers are defined in an actual programming language - typically Javascript, but one can also use Dart and there will no doubt be others. All this represents a big step forward for HTML, but it remains deeply unsatisfactory from a programming language viewpoint.
The thing is, you can’t really write arbitrary views in HTML. For example, maybe your view has to decide whether to show a UI element based on program logic or state. Hence you need a conditional construct. You may have heard of these: things like if statements or the ?: operator. So we have to add shadow conditionals.
<template if="{{usingForm}}">
is how you’d express
if (usingForm) someComponent;
In a world where programmers cry havoc over having to type a semicolon, it’s interesting how people accept this. However, it isn’t the verbose, noisy syntax that is the main issue.
The conditional construct doesn’t come with an else of elsif clause, nor is their a switch or case. So if you have a series of alternatives such as
if (cond1) {ui1}
else if (cond2) {ui2}
else {ui3}
You have to write
<template if = "{{cond1}}">
<ui1>
</template>
<template if = "{{cond2 && !cond1}}">
<ui2>
</template>
<template if = "{{cond3 && !cond2 && !cond3}"}>
<ui3>
</template>
A UI might have to display a varying number of elements, depending on the size of a data structure in the underlying program. Maybe it needs to repeat the display of a row in a database N times, depending on the amount of data. We use loops for this in real programming. So we now need shadow loops.
<template repeat = "{{task in current}}">
There’s also a for loop
<template repeat= "{{ foo, i in foos }}">
Of course one needs to access the underlying data from the controller or model, and so we need a way to reference variables. So we have shadow variables like
{{usingForm}}
and shadow property access.
{{current.length}}
Given that we are building components, we need to use components built by others, and the conventional solution to this is imports. And so we add shadow imports.
<link rel = "import” href = "...">
UI components are a classic use case for inheritance, and polymer components support can be derived from each other, starting with the predefined elements of the DOM, via shadow inheritance. It is only a matter of time before someone realizes they would like to reuse properties from other components in different hierarchies via shadow mixins.
By now we’ve defined a whole shadow language, represented as a series of ad hoc constructions embedded in string-valued attributes of HTML. A key strength of HTML is supposed to be ease-of-use for non-programmers (this is often described by the meaningless phrase declarative). Once you have added all this machinery, you’ve lost that alleged ease of use - but you don’t have a real programming language either.
Shadow World 3: Imports
Imports themselves are a kind of shadow language even in a real programming language. Of course imports have other flaws, as I’ve discussed here and here, but that is not my focus today. Whenever you have imports, you find demands for conditional imports, for an aliasing mechanism (import-as) for a form of iteration (wildcards). All these mechanisms already exist in the underlying language and yet they are typically unavailable because imports are second-class constructs.
Beyond Criticism
It is very easy to criticize other people’s work. To quote Mark Twain:
I believe that the trade of critic, in literature, music, and the drama, is the most degraded of all trades, and that it has no real value
So I had better offer some constructive alternative to these shadow languages. With respect to modularity, Newspeak is my answer. With respect to UI, something along the lines of the Hopscotch UI framework is how I’d like to tackle the problem. In that area, we still have significant work to do on data binding, which is one of the greatest strengths of polymer. In any case, I plan to devote a separate post to show how one can build an internal DSL for UI inside a clean programming language.
The point of this post is to highlight the inherent cost of going the shadow route. Shadow worlds come in to being in various ways. One way is when we introduce second class constructs because we are reluctant to face up to the price of making something a real value. This is the case in the module and import scenarios above. Another way is when one defines an external DSL (as in the HTML/Polymer example). In all these cases, one will always find that the shadows are lacking.
Let’s try and do better.