Saturday, July 11, 2009

A Ban on Imports (continued)

In my previous post, I characterized imports as evil, and promised to expand upon non-evil (yay verily, even good) alternatives. First, to recap:

Imports are used for linking modules together. Unfortunately, they are embedded within the modules they link instead of being external to them. This embedding makes the modules containing the imports dependent on the specific linkage configuration the imports represent.

Workarounds like dependency injection are just that: workarounds (e.g., see this post). They are complex, cumbersome, heavyweight. OSGi even more so. Above all, they are unnecessary - provided the language has adequate modularity constructs.

So, which languages have sufficient modularity support? I know of only two such languages: Newspeak and PLT Scheme. ML has a very elaborate module system, but ultimately it does not meet my requirements.

Modules and their definitions (these are two distinct things) should be first class and support mutual recursion. This isn’t the case in ML, though some dialects do support mutual recursion.

Tangent: The difficulty in ML, incidentally, is rooted in the type system. It is very hard to typecheck the kind of abstractions we are talking about. Worse, if you want to make your type declarations modular, your modules end up having types as members. This can lead you into deep water with types of types (making your type system undecidable). To avoid that trap, ML opts to stratify the system, so that modules (that contain types) are not values (that have types).

Not surprisingly then, progress on these issues comes from the dynamically typed world. Over a decade ago, the Schemers introduced Units.

The biggest difference between Newspeak modularity constructs and units is probably the treatment of inheritance. In Newspeak our module definitions are exactly top level classes, which reduces the number of concepts while allowing module definitions to benefit from inheritance.

There are strong arguments against inheritance of module definitions. For example, you cannot reliably add members to a module definition, because they might conflict with identically named members in the heirs of that definition. Specifying a superclass (or super module definition) looks like a hardwired dependency as well.

On the other hand, being able to reuse module definitions via inheritance is very attractive. Especially if you can mix them in freely.

Ultimately, we decided that the benefits of unifying classes and module definitions outweighed the costs.

Take the argument above regarding extending module definitions with new members. Newspeak was designed with an eye toward a completely networked world, where software is a service, not an artifact. In such a world, you can find all your heirs - just as if you were working on your own private application in your IDE. So if you need to add a member to a module definition, you should be able check who is mixing it in and what names they have added.

Tangent: This may still sound radical today, but this world is moving into place as we speak:
V8 gives the web browser the performance needed to be a platform for serious client software.
HTML 5, Gears etc. provide such software with persistent storage on the client
Chrome OS makes it obvious (as if it wasn’t clear enough before) that this in turn commoditizes the OS, and that the missing pieces will keep coming.

Likewise, in the absence of a global namespace, top level classes do not inherit from any specific superclass (and nested classes don’t either because all names are late bound) . Overall, the downside of allowing inheritance on module definitions doesn't apply in Newspeak.

The upside compared to conventional constructs is huge. It means you can easily take entire libraries, create multiple instances of them (each with its own configuration), mix them into new definitions, write polymorphic code that can work simultaneously with different instances or even different implementations of the API etc. You can store the libraries and their instances in variables, pass them as parameters, return them from computations, hold them in data structures, serialize them to disk or over the wire - all with the same mechanisms you use for ordinary classes and objects.

This economy of mechanism is important. It means you don’t have to learn a variety of specialized and complex tools to build modular systems. The same basic tools you use to implement basic CS101 examples will serve across the board. This will carry through to other areas like tooling: an object inspector can be used to inspect a “package”, for example. Altogether, your system can be much smaller - which makes it easier to learn, faster to load, likelier to fit on small devices etc. Simplicity is an advantage in itself.

As I explained in the first half of this series, the only need for a global namespace is for configuration: linking the pieces of an application together. There are several ways you can deal with the configuration/linkage issue. It’s a tooling issue. We use the IDE, as I described in an older post. So using the running example from part 1, we can write:

class SoundSystem usingPlatform: platform andPlayer: player = {

|

(* dependencies on platform might include things like the following: *)

List = platform Collections List. (* You can see how this replaces an import *)

mp3Player = player usingPlatform: platform withDock: self.

|
}{ ... }


class IPhone usingPlatform: platform withDock: dock = {


|

(* dependencies on platform elided *)

myDock = dock.

|

}{ ... }


class Zune usingPlatform: platform withDock: dock = {


|


(* dependencies on platform elided *)

theirDock = dock.

|

}{ ... }

and then create instances in the IDE (which provides us with a namespace where SoundSystem, iPhone(tm) and Zune(tm) are all bound to the classes defined above):

sys1:: SoundSystem usingPlatform: Platform new andPlayer: IPhone.

sys2:: SoundSystem usingPlatform: Platform new andPlayer: Zune.

tm: Did you know? iPhone is trademark of Apple; Zune is a trademark of Microsoft.

Variations on the above are possible; hopefully, you get the idea. If not - well, don’t worry, I probably won’t explain it again.

The absence of a global namespace has additional advantages of course: there’s no static state, and it’s good for security (but that is for another day).

17 comments:

  1. Tangent: I'm not convinced that your "types of types" scenario is actually undecidable. The *inference* may be undecidable, but the "types of types" you describe sound a lot like a very bizarre kind system. Unless the kind system itself is undecidable, the interaction between it and the type system should not be. Do you have a reference which explores this possibility?

    ReplyDelete
  2. Have you tried building (or at least stubbing) an entire application that way? abstracting out _every_ module dependency looks quite painful.

    ReplyDelete
  3. Alice ML doesn't have the stratification you described. Modules that are used in a dynamic way are checked at link time.

    http://www.ps.uni-sb.de/alice/manual/modules.html

    ReplyDelete
  4. Daniel:

    You mean there are non-bizarre kind systems?
    AFIK, ML modules are modeled via dependent types. The ML literature on this goes back decades. I won't swear it couldn't be done in a non-stratified way. Anyway, these days people are less concerned if a type system isn't decidable, as long as it can function in practice.

    ReplyDelete
  5. Daniel:

    You mean there are non-bizarre kind systems?
    AFIK, ML modules are modeled via dependent types. The ML literature on this goes back decades. I won't swear it couldn't be done in a non-stratified way. Anyway, these days people are less concerned if a type system isn't decidable, as long as it can function in practice.

    ReplyDelete
  6. Salzan:

    We've written a lot of code in Newspeak: at this point, we have the entire IDE and GUI stack in Newspeak for example. In the public release, you'll find large parts of that, though not all: debugger, parser combinators, regular expression package, serializer/deserializer, unit testing framework, compiler, collections etc.


    Jules: I'll take a look at your link, though I'm skeptical.

    ReplyDelete
  7. Jules:

    Looks interesting. I note that the type system is undecidable AND that some things are checked dynamically. Not that I mind, but it isn't ML anymore :-)

    ReplyDelete
  8. It always strikes me as a little odd that undecidable type systems are so widely deployed and even relied upon. It's almost like some weird, deviant religion that compells its followers to merely accept "on faith" that every type system works as expected.

    With that said, it's true that many (if not most) type systems in real-world languages are undecidable. Scala is a particularly annoying offender in this department. It's type system is actually more powerful than the simply typed lambda calculus. If you enable recursive types, its type system actually becomes Turing Complete, making "undecidable" a dishearteningly vast understatement.

    Speaking of Scala, its kind system isn't too bad. At least, it isn't as bad as the types as types module idea. Kinds are certainly a bizzarrely abstract realm of type theory.

    ReplyDelete
  9. Gilad, how do you feel about Scalas module system? Seems like it has a lot of the properties that you request (at least superficially).

    ReplyDelete
  10. Fredrik:

    First, to be clear: I'm not requesting anything. I've implemented the system I want in Newspeak, and we've built a fair amount of stuff with it.

    Regarding Scala. Scaaa is way better than anything else on the Java platform. If I had to develop on the JVM, I would use it (at least until I got Newspeak working well).

    Scala still has a global namespace and imports and packages. These are significantly superior to those in Java, but fundamentally suffer the same problems. Presumably you are not referring to all that stuff, but to the use of objects as modules.

    The latter is indeed very much in the same spirit as what I suggest. Of course, it is a discipline that you have to enforce yourself, since the language makes it all to easy to fall into the bad habits of traditional packages, FQNs, imports etc. There s little to prevent you from referencing a fully qualified name inadvertently, for example.

    This is why I prefer a system that is much simpler and more uniform. Scala is a great language to be sure - but it still makes compromises that I would rather not see.

    ReplyDelete
  11. Gilad, thank you for the reply. Very inspiring. I will try to avoid the import statement when developing in Scala!

    Sorry for the "request part" of my first comment: a lack of precision on my part, english is not my native language.. I'm fully aware of newspeak! And very much looking forward to it :)

    ReplyDelete
  12. Your encoding looks very similar to the one I gave last time for Scala ...

    ReplyDelete
  13. Miles,

    It's nice that Scala lets one produce a good modular structure if one really tries. What's unfortunate, as I noted in an earlier comment, is that the language has a host of standard mechanisms that work against you - like packages, imports etc.

    If you carefully restrict use of packages and imports to to top level configuration code only, you may be able to emulate what Newspeak's module system does for you by default.

    ReplyDelete
  14. Old Borland's Delphi / Object Pascal had/has a Unit system that seems to match your requirements too.

    ReplyDelete
  15. "Newspeak was designed with an eye toward a completely networked world, where software is a service, not an artifact. In such a world, you can find all your heirs - just as if you were working on your own private application in your IDE. So if you need to add a member to a module definition, you should be able check who is mixing it in and what names they have added."

    This seems completely disjoint to me. Yes, we are moving into a world where software is a service, and where everything is networked. But that does not mean that you can know all your heirs! Quite the opposite. Private use of a service is a common use case; consider how many internal tools companies make. Even relatively small companies (say 3000 people) depend on internal tools which are built from external services. But they don't want others to know this; their tool is part of what gives them an edge. Even if companies are willing to publicize fact that they are mixing in a service, they may still not want others to see what names they have added.

    In other words, I do not believe that software as a service allows you to argue that we can now treat modules as a closed system. They are just as open as before.

    ReplyDelete
  16. Ciera,

    I'm well aware of that objection, and many others. There are legal issues, and privacy issues galore.
    I've discussed them in some of my talks on the subject - say, the YouTube video at
    http://video.google.com/videoplay?docid=-162051834912297779

    To support true services, there would need to be contractual relationships between customers and suppliers - just as with anything else (like contractors). I know it is a radical proposition, and it will evolve gradually, over an extended time.

    ReplyDelete
  17. Dependency injection is not Heavy Weight, it is just that way in Java ... because Java is Java.

    If you want, there is an incredible and very minimal explanation about what DI is all about, with examples in Ruby.

    http://onestepback.org/index.cgi/Tech/Ruby/DependencyInjectionInRuby.rdoc

    ReplyDelete