What do I mean by this? In an IDE written in a dynamic language (such as Smalltalk or Self or Lisp), the IDE code itself can be modified on the fly. This is a double edged sword, as I’ll explain shortly. I’ll start with the good edge.
Suppose you want to modify the IDE for some reason. It lacks a feature you need, or something is buggy. If it’s a proprietary system, you can file a bug with the vendor, and hope that they pay attention, so that in the next release, in a year or two, you’ll get the fix.
If the system is open source, you can just go ahead and implement the fix/feature. Now if that system is written in a mainstream, conventional language, you just need to recompile the compilation unit with the fix, and rebuild your system. Of course, in order to develop the fix, you had to load a project with the source for your IDE and probably go through several edit-compile-debug cycles. When it all worked, you replace your existing installation with your new build. I have the strange feeling that all this setting up the IDE as a project within itself, rebuilding and reinstalling could take quite a lot of time and effort.
In an IDE like Smalltalk (or Newspeak, or Self), the IDE source is always available from within itself. In such a system, if you change its source code, the change takes effect immediately. For example, the Newspeak class browser supports a pop up menu for methods, that lets you find references to the method and to the messages sent within it. In the screen shot below, the speech bubble on the right denotes this menu
However, the initial version did not support the same functionality for slots.
How do we fix this? Well, for starters, we need to find where the relevant source code is. How do you do this? The old fashioned way is to go on a trek through the source code trying to figure it out. What subdirectory should we start with? Which file should we open first? What should we grep for?
Fortunately, Newspeak makes it easy, as most IDE components have an Inspect Presenter menu item, that will open an object inspector on the actual presenter in question; a presenter on a presenter.
For example, if we want to find out how the references menu for methods works, we can look at a method like the one above. On the far right is a drop down menu:
This opens an object presenter, like this:
Toward the top, is a line marked class; it contains a hyperlink to the class of the object being inspected - the class ExpandableMethodPresenter. Now we know where the code that presents methods resides! If we click on the link, the Hopscotch browser will show us the class.
Having found the code that manages the presentation of methods, and the implementation of the references menu, we next want to find the code that presents slots, so we can modify it. We do the same thing again - invoke the Inspect Presenter menu item, but this time on a slot.
Once we’ve found the code and made the change, we can test it right away. Just hit refresh in the browser, and you’ll see the new reference bubble next to slots.
All of this is part of what Dan Ingalls, and Smalltalkers in general, mean when they say the system is alive. It’s the key property you seek in every system - its ability to respond to changes and mutate while running, which gives you the kind of interactivity you get in the real world with real objects. Live code in the IDE is the opposite of dead code - stuff written in files, which needs to be explicitly loaded into a runtime to do anything.
This all is very cool, but it also has a downside. Suppose you have a bug in your code, causing it to crash. You may be able to fix it in the debugger - but maybe the root of the problem is in another class. You need to browse it - but now, every time you try and browse a class, it crashes. You’ve shot yourself in the foot.
This is where you almost wish you were looking at dead code, by browsing files in an editor (or a conventional IDE). Fortunately, we know how to deal with this problem as well.
In this case, we want to manipulate the modified IDE from another, unmodified one, as a dynamic program. That way we still enjoy immediate feedback for our changes, while jkeeping our feet safe. Once it works, we want to be able to easily suck in the changes into the unmodifed version - without restarting, rebuilding, or going back to the big bang.
We haven’t implemented this yet - but much of it’s been done before, in the mid-90s, by Allen Wirfs-Brock in the FireWall project at ParcPlace. You do it using mirror based reflection.
One of the many things mirrors facilitate, is managing reflection in a distributed setting. Java’s JDI is an example of a mirror API was designed to do just that - though it has serious limitations because it has to work on a variety of JVMs. If you design the mirror API correctly, and build your IDE on it, the IDE can work almost identically on programs within the same process, in another process, or across the internet.
This ability to manage reflective changes to the IDE via a separate process is a luxury, which probably explains why it generally isn’t implemented, The reality is that shooting yourself in the foot is infrequent, and you can always recover (if only by saving your changes to a file before testing them). The benefits of liveness far outweigh the risks.
Nevertheless, in time, I expect to address this in the Newspeak IDE. In the meantime, it’s alive, even if support for death is lacking. Of course, the broader lesson is that IDEs especially, should be implemented using dynamic languages.
The above echoes my previous post from June, but I hope that the concrete examples are helpful. The liveness makes it easy to implement features like Inspect Presenter, which let you identify where to change the IDE, but even more critically, liveness allows you actually make the change and get instant feedback.
In closing, I want to emphasize that this is not an unprincipled approach. All of the above is possible because the IDE can reflect on itself. Self reference is at the heart of computing. It’s all about recursion. I find it strange that many theoretically oriented computer scientists, who can wax poetic about the Y combinator, eschew languages and systems that apply the same principles of self-application to themselves. I’m sure this is will change though; the good ideas win in the end.
15 comments:
People like self-reference, but I think they are wary of systems that don't have good spatial and temporal separation between the actual application of the self-reference and the self itself.
That is, it is nice to have e.g. a compiler implemented in the language it compiles, and all is well and good: improving the language will improve the compiler, and improving the compiler is improving the language (in so far as the compiler is a true definition of the language).
But there is in this case always an extra level of indirection, in the form of the source. This gives people a sense of control. The decision of choosing to make the experimental target the main running executable is deliberate and conscious.
However, when you get into directly self-referencing, things start getting uncomfortably unstable - not necessarily in practice in running systems, but in people's minds. People get the feeling that they may be sawing off the branch while sitting on the wrong side of the cut. With the compiler case above, they are sitting in an entirely different tree.
It's just sandboxing of changes. It makes people feel safer.
However, there are other more prosaic issues, such as source control, distributed development, merging multiple branches, comparing diffs visually, grepping through source - with all the semantic loss that implies - that we have well-developed technology for. Most of the languages that have enabled self-modification in the way you describe have been image-based languages, but I don't think higher-level (than text) source control and storage mechanisms have had convincing solutions yet.
There's a nice discussion of this by the Self people in a 1997 CACM article. Unfortunately they're not freely available online anywhere that I know of.
Emacs
Yeah buddy EMacs totally rock.
Jiff
www.internet-anonymity.net.tc
Hi Gilad,
Interesting stuff.
Eclipse addresses these problems with 1) plug-ins (which I haven't seen mentioned about Newspeak) and 2) the plug-in framework.
Whenever you are working on a plug-in or on Eclipse itself, you can just launch your project as a second instance of Eclipse, which gives you the best of both worlds: a sandbox that can't crash your main IDE if you make a mistake and the performance that statically typed languages usually give you over dynamically typed ones.
I would love to have my cake and eat it too, but I have a strong suspicion that you are going to have a very tough times getting decent performances for your IDE if you keep going down that path...
Hope I'll be proven wrong.
--
Cedric
Barry: Thanks for the thoughtful comments.
Sandboxing changes is what I alluded to when commenting on the FireWall project.
wrt source control:
Indeed, the mainstream has invested huge efforts in issues like source control. and there is much to be learned there. But the mainstream never rises above the lowest common denominator of text.
Newspeak, unlike Smalltalk, has a complete syntax which you can put in a file, so it interoperates with the mainstream version control systems. Not that we don't want better solutions.
To all those who said: Emacs!
Yes, a fine example, and my favorite text editor. But also a very dated example. One needs an Emacs for the 21st century. Hopscotch is the foundation for such a beast.
Hi Cedric,
Thanks for reading.
A problem with plug ins is that they only let you customize what the plugin architect foresaw. And they don't really address the dynamicity you get from a true metacircular system. You cannot fake that.
As for performance, you're conflating several different issues. The peak performance of the language, and the performance of the IDE.
Self and Strongtalk long ago showed that dynamic language performance can come pretty close to mainstream. Not all the way, but good enough that most applications will gladly pay the price in exchange for the benefits of high level language.
V8 is going to bring this understanding into the mainstream.
As for IDEs - even running on a Smalltalk interpreter, Squeak is very responsive IDE. Mainstream IDEs take far too long to start up, and don't have a very snappy feel. Strongtalk, for example, is far more responsive than any mainstream IDE.
The point of this post was partly that regardless of the target language, the IDE itself is an application where dynamism is very valuable, and so worth writing in a dynamic language.
Thanks for the response, Gilad.
Agree with the fact that plug-ins only cover what the author foresaw, but after a little while, this becomes a non-issue as the IDE evolves. I think Eclipse has reached a modularity point where there is very little that it can't do (a bit like emacs).
I wasn't actually conflating the performance issues, I was just being unclear :-)
I was specifically referring to the performance of an IDE written in a dynamic language, especially one that allows all the features that you described in this article. Considering how long it took for IDEA and Eclipse to reach decent performances in normal usage (I'm talking about things like text scrolling, auto completion, highlighting, refactoring, etc..., not the start-up time, which represents a minuscule fragment of the time spent in an IDE), I still think that such an IDE is going to be very clearly CPU-bound.
Again, just theoretical skepticism, which can be easily cleared by making an early version of your IDE available to everyone :-)
Cedric,
Regarding IDE performance: it's been a non-issue in Smalltalk systems for many years. Partly, because the IDE is architected differently: perhaps scrolling through a file with 50K lines might be a problem (though in Strongtalk it wouldn't) - but you don't do that. You typically edit individual methods (and even these are kept short as a matter of good style).
In Newspeak, every keystroke requires parsing the entire text by a fairly naively implemented parser combinator based colorizer, running on the Squeak interpreter - and yet it's usually fine, because the text is short.
We definitely do need to improve performance - and running on V8 seems like an attractive possibility.
A Newspeak prototype will be available in not that long. In the meantime, you can grab a free download of Squeak, or better yet, VisualWorks or Dolphin Smalltalk and see if responsiveness is an issue.
The whole dynamic IDE experience that Smalltalk and friends provide needs to be experienced and digested for a while before addiction takes hold :-). Try it and see !
A bit strange, Gilad, that you spent so little time explaining what exactly one would want to adapt about the IDE. I might agree that its painful to source/tweak/recompile an IDE in a static language, but I've never been inclined to do that with my Vim or Eclipse (to name some regulars) anyway. For example, I don't think I ever encountered a critical bug in them.
Of course, there are always small adaptations (such as key bindings, style issues, etc.), but IDEs are usually extremely customizable and cover these things already. (Or it's not a big issue to live without them.) I don't just mean by checkboxes and some preference panels, but also by an integrated scripting language or pluggable external commands. (Yes, you can only extend what the IDE creators foresaw, but it goes a long way...)
Next, I could imagine that you want to support some new language, toolset or paradigm in your favorite IDE, or that you want to add a substantial new feature (say, for example, version control). In that case, developing an IDE plugin or fork is unavoidable, but since all of these cases are a considerable endeavors anyway, I don't think the start-up cost of a static language is a big deal. Maybe the opposite, because a static language and type system might help you navigate, understand and consistently adapt a large code base that you are not familiar with.
Finally, in an atypical scenario, I could imagine projects where you create one or more DSLs for a specific project and consequently develop large parts of the code base in those DSLs (similar to how Knuth created WEB to develop TeX). In that case, you quickly need to support these DSLs in your IDE. I agree that this case might not be supported by the mainstream, but I can still think of better solutions than a full metacircular system, for example, pluggable compiler frontends.
You can use AOP to do this in even non-dynamic languages like Java.
I think the Smalltalk style should be called "full immersion programming" and my sense (as a non smalltalker) is that the "sensible" approach (eg as barry describes -written in itself, but not running in itself) is a very different (and poorer) animal.
I started using Python a while ago -and was disappointed by how the IDE's turned their backs on liveness. They seem to have static language IDE envy.
I wonder if Dan Ingalls has spent any time programming in(!) second life? (for that matter, i wonder what he thinks about the collision of names between lively kernel and google lively)
I work on a large simulation for the Department of Defense. We are lucky to get two releases out a year because of red tape, and even emergency patches can take a month. So one of our former jWarriors came up with an idea he called Method Source Overlays. Simply put, we allow the users to make changes to methods and store those changes in their scenarios just as they would any changes to the data concerning ships and planes. For instance, they could change the implementation of altitude in BallisticMissile to always return 15 if they so choose. When the scenario initializes, we spin through these overlays, compile them, and away we go with new code.
This idea proved so popular that our largest group of users has written close to 300 overlays in their current version. We extended the idea to also allow them to write overlays that change the way the guis work. Having so many overlays presents a challenge when debugging (is it the code or is it an overlay?), and makes integration more difficult, but that’s why they pay me the Big Buck.
All of this is possible only because we deal with a world of live objects than can treat code as data. I’ve always like programming, but I love Smalltalk, and I look forward to seeing NewSpeak.
Post a Comment