Emacs is my preferred text editor. I don’t use old-fashioned text editors as much as I used to, because I often need more specialized tools. I use IDEs for various programming languages and other things when producing high quality documents. And yet, I often wish I could subsume these with a tool that had the basic goodness of emacs.
What makes emacs interesting all these decades after its inception is not what keyboard shortcuts it supports or what its basic editing functionality is. Rather, what matters are some of its underlying design principles.
- Emacs has built-in scripting language.
- That language (elisp), while it has many flaws, is a flexible, dynamic, and rather general purpose language.
- All of the editor’s functionality is exposed via APIs written in the scripting language.
As a consequence, you can control everything emacs does programmatically. This makes emacs extensible in a way that is far deeper and more powerful than a plug-in architecture. A plug-in architecture must anticipate every desired extension to the system. Since it never can, it always disappoints in the end. Along the way it grows ever more complex and bloated in its futile attempt to foresee every possible need. With a language, you can code any extension you need.
If the scripting language is truly dynamic, and allows you to not only extend but also modify the running system, the possibilities are truly unlimited.
The points above are not limited to editors. They are fundamentals of system design.
There was a time when even Microsoft recognized this, making apps that could be programmed via VisualBasic. Sadly, they concocted a security nightmare, because malware can also control your application. Which is why being able to secure your application’s scripting language is critical as well.
Last June, I spoke at QCon NY, and demonstrated a number of interesting web based systems that had some these properties (as well as a couple that did not, but were interesting for other reasons):
- Leisure is a presentation manager that incorporates a lazy functional programming language. Modulo some person-years of engineering, it is to PowerPoint what emacs is to NotePad.
- The Lively Kernel could be thought of as a GUI builder scripted via Javascript; this an insult to Lively of course, because it is much more than that. In Lively, the GUI is the GUI builder; every GUI you make is extensible and modifiable in the same way.
- Lastly I showed Minibrowser, a prototype web based IDE for Newspeak. Like all IDEs in the Smalltalk tradition, it can be extended and modified from within itself.
We really need an Emacs for the modern age. An editor, surely, but one that lets you edit rich text, images, audio and video. In fact, you should be able to embed arbitrary widgets. And of course it needs to be scriptable I just explained. So you might evaluate code that creates a UI element and inserts into the editor.
Now you can make the editor modify its own GUI. In fact, the editor can be extended into a general purpose GUI builder just like Lively. And every such GUI can modify itself if you wish; sometimes you may wish to modify it so it can no longer modify itself, and then you have a frozen application. Your editor has become an IDE. In fact, it is a live literate programming environment.
If the editor’s scripting language interoperates well with the surrounding environment, it can be used to control the computer and everything the computer itself controls. You can check in to the environment and hardly ever leave. You can lead your cyber life in it.: email, social media, live chats, streaming audio and video can all be incorporated. Moreover they can all be controlled and customized by you, the lucky user.
To a degree, Lively is such an editor. It’s biggest drawbacks are a lack of polish due to lack of engineering resources and that its scripting language is Javascript.
Now, imagine that the editor was polished and robust. Even more importantly the code you created in this environment was modular and secure and written in an elegant and principled language. Imagine you could deploy the same code either on the web, or natively on both desktop and mobile. Imagine that the applications built with the language support online and offline use out of the box, automatically synchronizing data and code between clients and servers. Imagine that they have built-in support for collaboration, either syncing in real time or merging offline as required.
Of course, it is the vision of such a language and platform that has always motivated the Newspeak project. I have discussed many of these points before. In particular, I’ve talked about the weaknesses of traditional IDEs (see for example this post and this one), and the need for a platform that supports synchronization over the net (here and again here) for a long time. Yet the message bears repeating.
34 comments:
Perhaps except for security concerns, lighttable matches your description.
The difference between editors providing an extension "API" and scriptable editors (which ideally are written in the scripting language itself) is the "API granularity": In the latter, the API is not restricted to a few accessible objects and functions; instead the entire editor's functionality and implementation language is exposed. It's very powerful indeed, and it's also a great way to shoot into one's foot. The true test for an editor/editing system is whether the author is willing to use it w/o reservation for _all_ her critical day to day work. Otherwise, it's just a toy.
An an aside, it's possible to write truly extensible editors even in less dynamic languages. Here's one example, I'm sure there are others: In 1990, Clemens Szyperski wrote the extensible "Write" editor for the Oberon system ( ftp://ftp.inf.ethz.ch/doc/tech-reports/1xx/151.pdf ): The basic data model of the Write editor was a sequence of abstract "characters", they would provide functionality to access their sizes and how to draw, serialize, and deserialize themselves, etc., but the editor didn't have to know more about them. As a result it was possible to extend the editor with all kinds of character "objects", including pictures, graphs, and animated objects (see the .pdf report). The editor had a clever serialization format that made all the extensions look like "comments" when read by a plain-text editor or compiler. As a result it was possible to write source code in Write that contained pictures for documentation. And since extensions were written in the same language as the editor (Oberon or Oberon-2), it was easy to change the editor in more powerful ways as well.
The editor was not a "toy": Several people (incl. me) used it daily as the main editing tool while at the ETH CS department. It was tied to the Oberon system and language and thus unfortunately didn't find a larger user base (Smalltalk and Newspeak IDEs have the same problem).
Emacs may be great, but compared to what's possible it was an anachronism even in 1990.
Hi Robert,
Yes, of course much more is possible. I use emacs as an example because it is more widely used than anything else of that flavor.
One reason I emphasize the use of dynamic language is because I want to go beyond just offering an API for all the editor functionality; I want to be able to modify any part of it on the fly. That's the difference between emacs and Smalltalk.
You could add such support to any language, even a statically typed one. In practice, almost all exemplars are dynamically typed, because it is so much easier to do it that way.
Also, there may be some confusion around the term "dynamic language". I've never really like it. It confuses dynamic typing with dynamic program modification. Arguably, support for self modification makes the language dynamic by definition - even if it is statically typed.
A couple more thoughts.
a. The "shoot yourself in the foot" argument is overstated. In reality it happens quite rarely. If one has good recovery mechanisms it's well worth the trade off.
b. Thanks for reminding me about the Oberon work. You might also remember the flyweight glyphs in Strongtalk :-).
Maybe chromedeveditor can fill that gap :)
That Oberon editor (or a descendant) was commercially available for both Mac and Windows from Oberon Microsystems as Oberon/F, later Blackbox Component Builder, which is currently available in open source for Windows. (I found the Mac (68K) version one of the most enjoyable and productive environments I've used)
MaysonicWrites: Cool. Almost makes it worth running Windows (but not quite :-)).
I remember the first Oberon Microsystems demo I saw. The major thing lacking was a debugger. The demo was given to the Strongtalk team, and when we asked about the debugger, we were told "Just look at the code". Guilty parties unnamed to protect the innocent :-).
Sadly, neither of those two great PL enterprises prospered.
Racket and DrRacket, of course... It gives you the dynamic thing (unrelated to typing, since it can dynamically load code that is written in Typed Racket, of course), and the language provides the kind of protection that you want, in several ways.
The thing is that this protection is used in DrRacket to a point where you *cannot* modify the editor while its running, and instead you get a conventional-looking REPL. But it is most definitely possible to do that: when you run some GUI code in it, there is no subprocess -- the code is running inside the same Racket process that drives DrRacket -- only DrRacket goes out of its way to avoid any changes to itself. I always wished that someone would pick up the ball and make some DrRacket plugin that breaks that barrier -- it's totally doable and probably not even too difficult. (And would make a good point for people who see it and just assume that it can't.)
One topic I'm missing from the discussion is how to make programming tools that are suitable to work in the context of distributed programs. Todays applications are rarely single program artifacts running locally. Whether you create Web applications or not, it is very likely that building, deploying, and maintaining non-trivial software will involve distributed software artifacts. Possibly written in different languages and with different technology stacks. A development tool for this century should support inspection, evaluation, and debugging for these kind of applications. Emacs offers with SLIME, tramp and similar systems more than a lot of other development tools do. But even that is pretty restrictive and a much better support is conceivable and important. So apart from rich text, media elements and arbitrary widgets I think that a modern development tool needs to readily support networked and distributed applications.
Robert,
This is why the vision has to incorporate a model for synchronization, as I mention in the post. There are several possible models for that - TeaTime, Operational Transforms, or an extension of Distributed Source Control to data. And yes, realistically we need a multilingual back end for this.
As a followup to what Eli said about Racket and DrRacket, there's also the Slideshow package for Racket (http://docs.racket-lang.org/slideshow/index.html), for creating scriptable presentations, much like Leisure that you cite in your post.
I would also like to propose a distinction between scriptable tools (such as editors) and scriptable documents (such as live presentations). The difference is that a tool is personal, whereas documents are frequently shared and co-authored by several people. This puts more stringent demands on the security and longevity of languages used in documents.
To make a point that will probably sound —deceptively, I suspect— narrow in focus, I've really wondered about the use of dynamic scope in emacs lisp. My suspicion is that fexprs are superior to macros in, essentially, all situations where macros are used; but fexprs were abandoned by mainstream lisp at the same time as dynamic scope, and I shake my head over that because in my experience, the one thing that can really sabotage fexprs so they become a disaster is combining then with dynamic scope. If lisp hadn't used dynamic scope for its first two decades of life, I suspect fexprs might never have been rejected. And, it seems to me fexprs ought to be particularly useful in a markup context (which seems subtly related to programmable text editing); but, having never personally done much emacs hacking, I really wonder what part dynamic scope plays in the emacs programming context.
(It's almost eerie to see the comment about ability to extend in a way that restricts later extensions --- since that sort of flexibility to decide what will be flexible later is rather at the heart of my approach to abstractive power.)
Konrad,
The distinction you make is valid, but I think we are moving away from it. Almost every application one produces ought to support collaboration/synchronization.
That is why I've argued for the past decade that support for synchronization belongs in the core platform.
While the message about editors and IDEs is starting to register, the message about sync as a platform feature isn't nearly as far. I can see that in the response to this post as well.
John,
I have not paid any attention to fexprs, I confess. A naive question: if it is easy enough to delay evaluation of an expression, can't we just delay evaluation of the arguments? For example,in the Smalltalk family, [e] is a closure that evaluates to e when invoked. And you typically have the means to reflect upon it. Seems to avoid the difficulties. But hey, this is getting pretty far from the topic of the post
Great post! The overlap between Emacs, lisp machines and Smalltalk is huge and receives very little discussion.
> I want to go beyond just offering an API for all the editor functionality; I want to be able to modify any part of it on the fly. That's the difference between emacs and Smalltalk.
I don't follow this. What does Smalltalk lack in this regard? Everything in Smalltalk can be modified, other than the VM (Emacs likewise).
Whilst Emacs and Smalltalk (e.g. Pharo) are philosophically similar, Emacs has grown in a way other hackable environments haven't (yet). I would attribute Emacs' success to:
* Being multipurpose. I can write C or Fortran or Haskell in Emacs. As a result I can keep hacking on and living in my Emacs environment.
* Not inward looking. Smalltalk has its own DVCS (Monticello), its own collaboration websites (smalltalkhub), etc. It's not able to exploit the larger technology ecosystem.
* A packaging solution. These days it's really easy to get your elisp into other people's hands, and to collaborate on improving packages. It produces a virtuous cycle of improvement and ecosystem growth.
Anyway, thank you for some pointers to some very interesting projects!
I think some of the problems related to adoption of such systems have to do with a closed world view: a single language, single separate platform that doesn't interoperate with anything else, no open standards so impossible to create a market based on the idea. That said, I'm not sure what would be the killer approach that would excite the masses. I set out to do something similar to what is described in this blog and to what Newspeak is doing, but for the JVM. I thought, the JVM is not well-suited for a live system like Self or Smalltalk, but maybe I can get close enough, and maybe I can make it so that people use their favorite programming paradigm. The result is a barely maintained system that only I and few colleagues have been using. It's a bit of a mish-mash technically, but I thought it being JVM and all, practicality would breathe motivation and momentum to push the ideas. Didn't quite work. Why doesn't everybody see that live, self-modifying programming environments are the way to go? Why don't people agree to suffer through the initial stages of such a technology without it being all shiny and hip. Not sure, haven't been going to industry conferences lately to see what makes people tick...
Well, fexprs seem at least in danger of digressing. And yet, somehow I sense all of this converges. (I have a feeling I should be asking, at this, point, why we still programming using text, beyond mere tradition.) I'll see if I can answer your "naive" question anyway.
> I have not paid any attention to fexprs, I confess.
Oh, that's very common. :-) I wouldn't have myself, except that in about 1998 or so, while constructing Scheme interpreters, I hit on something that appealed to my sense of elegance, that I later realized was a variety of fexpr.
> A naive question: if it is easy enough to delay evaluation of an expression, can't we just delay evaluation of the arguments?
In the Elder Days when fexprs coexisted with dynamic scope, as best I can figure, fexprs didn't do much more remarkable than delay/prevent argument evaluation. However, when fexprs synergize with lexical scope, they're a powerful tool for controlled manipulation of scope. In Kernel, I use fexprs to define lambda, apply, let and its variants, and various devices for safe import/export of symbols between environments.
Boris,
I agree that part of it has to do with the isolation of these systems, at least historically. But a good FFI is all you really need yo solve that problem. Ultimately, it's a cultural problem. After all, Java is just as isolated (and its FFI is actually much worse than Newspeak's).
I never pursued such systems in the JVM, because it is so poorly suited to running them. It's just seemed too painful.
Vilfredo,
I wasn't saying that Smalltalk was more limited. In some systems, like Squeak, even the VM is modifiable (though not live). On the contrary, I'm not sure if you can change the basics in emacs as freely as in Smalltalk.
Your point about Smalltalk being inward-looking is well taken. In Newspeak we made sure we had true source code we could put under version control. We integrate with mercurial and git (we used to svn, but that has fallen by the wayside). We did a real FFI and used WIndows native GUI. Now we a re working on a web version. But ultimately, there is a cultural gap.
John,
Thanks for the answer. It sounds like you can use it as alternative primitive in an elegant way.
I've been doing my own editor for the language I'm designing, and find starting with a blank canvas (metaphorically and literally) to be quite liberating. However, real estate management is my biggest problem: sure I could plop an image (or widget!) down in the editor anywhere I wanted to, but it won't necessarily look good nor will it be very usable.
Light Table also trying to improve this situation (they are inspired by code bubbles), but I think they also hit a bunch of design constraints with respect to what you can usably do with a 2D space, so it keeps getting more conventional as its design moves along.
Also, let's not forget Lively's predecessor: Morphic! I re-read their '95 UIST paper every few months to re-grock its awesomeness.
I think it's more about market then culture. When Java came out, we talked about it as a cute toy, and the marketing onslaught by Sun, a rich and powerful company at the time, was both annoying and adorable in a way. Neither I nor any of the more experience programmers and mentors that I had in the mid-90s believed Java would become mainstream. Yet, it did. Was is better than, say, Delphi? I don't think so. But Sun managed to create a whole market around the platform. That's how any other language/platform succeeds. Java got into people's heads through the browser (an established universally needed tool), but that's not where it made inroads really. The main use was and still is server-side development. So it's not important how you get in, as long as you get in somehow. That's why I thought one has no choice but to play along with some mainstream platform, if the goal is widespread adoption that is, and not only academic appreciation.
I agree that built-in collaboration support for authoring platforms will be essential. But still there is a need for a clear separation between the platform and the document. Not for technical reasons, but for reasons of ownership.
When I write a book, I want to have full control over it. I want it to remain readable beyond the lifetime of the software that was used to write it. If my documents depend crucially on embedded scripts, then I want those scripts to be in a language that's well-defined and independent of any software vendor who might go bancrupt.
The problem with computing is that it evolves at a much faster rate than anything else. Few intellectual artefacts are as short-lived as software. Most books remain of interest for decades.
"Most books remain of interest for decades" is patently false. Most books are actually forgotten after a few years (especially with all the crap being published today), it is just...well...we've forgotten them. With that view, writing for today is probably a better idea than writing for tomorrow. If something turns out to be seminal, then it will be remembered through extra effort (like old Greek texts).
I said interest, which is not the same as value or popularity. Those are three almost orthogonal axes.
Books recognized as valuable are indeed kept alive by copying, translating, etc., but this process is also much slower than the change in computing technology. If publishing technology makes a book unreadable after ten years, then many potentially valuable contributions will not be recognized as such before disappearing.
10 years is quite all right. I don't believe we've lost the technology to read and interpret anything that has been written since the dawn of computing. It is probably sufficient to keep building Rosetta stones. I still think some amount of ephemerality is not going to hurt us. Not every thought is meant for the ages (though the way we are going, it will be anways saved for datamining by future generations).
We may not have lost any fundamental technology, but much digital information has rotten to the point of making recovery economically unfeasible.
An example from my own experience: Apple's latest Keynote (version 6) can't open presentations made with version 4. You must open and save with version 5 to convert, but version 5 is no longer available. Fortunately I still had a Mac with version 5 available when I realized the incompatibility. It's gone since then, so if I missed one of my old presentations (much less than ten years old), I can consider it lost.
Yet, say you had a PDF rendering or paper print outs of your slide deck. Would your situation be any better?
Slightly. I keep my old presentations mainly for re-using slides from them in future presentations, usually with minor modifications. A PDF is no good for editing, so I'd have to re-create the slides manually, which however is still less work than re-inventing them.
Sorry to go off topic, but I know you've blogged about immutability before, and didn't know if you were aware of this. I suspect you may be the kind of person who would know of prior art to prevent this patent:
http://patents.stackexchange.com/questions/10128/prior-art-for-us-patent-application-20140196008-immutable-object-types
Sean: The raw data for the 1960 U.S. census is literally rotting, as it exists only on punched cards in a non-climate-controlled storage facility. By comparison, the 1860 census data is handwritten in books, and is in good shape and quite accessible.
John Cowan: Those bits have long been digitized and you can find the microdata for both censuses online. And I'm sure you'll agree map reducing over a warehouse of paper records is equally hard whether they are punch cards, or handwritten.
Getting back to Gilad's point, I'm sure the most powerful/useful editor these days is not a type writer.
A minor note, it seems that Leisure's document format is 100% emacs org mode format http://orgmode.org/. An opportunity for us org mode users to experiment with even more lively-ness in our documents.
Yes it is, very deliberately as I understand it.
Post a Comment