Monday, November 3, 2014

After EclipseCon is Before EclipseCon

Now that the EclipseCon Europe 2014 is over, it's time to focus on the next big community event: EclipseCon North America 2015 - especially since the deadline for the call for paper is already approaching. Better get your session proposal ready soon, if you want to share something new, cool, interesting or enlightening with your peers. If you are really fast, you may even reach the deadline for the early bird picks. Chances won't get better.
In other words: San Francisco. March 2015. EclipseCon. Submit now!

In case you don't know it yet: EclipseCon North America will again feature theme days that focus on special topics, one of those will be dedicated to Xtext. If you want to share insights about your application of domain-specific languages, how you solved challenges in your language implementation or how you use the framework in general, I can only encourage you to submit a talk for the Xtext track.
As every year, I expect EclipseCon to be a great community event with deep technical content. Like nowhere else, you can get in touch with the committers of the various Eclipse projects, discuss solutions and have a great time. So even if you don't plan to submit a proposal, make sure to save the date: March 9 - 12 in sunny California, EclipseCon NA!
Still not convinced? Check out the impressions from past EclipseCons and see what you are going to miss!

Monday, October 6, 2014

Musing about Eclipse UX Metaphors: The Blocking Build


For the upcoming version of Xtext we are revising the approach to building. It appears to be promising to rethink the overall lifecycle of the Xtext builder to aim at:
  1. Better user experience by introducing a non-blocking build infrastructure
  2. Improved performance due to improved parallelization
  3. Incremental builds also on the command line

The Problem

The Xtext framework implements an Eclipse builder and is thereby immediately affected by the builder's user experience metaphor (even bad experience is still experience). Whenever a file is saved or a project is explicitly built, the user is essentially locked out from doing work in the editor.

Go Home, Eclipse! You're drunk!
That's not because the editor isn't generally usable during the build. But it turns out, that it becomes quite a habit to Eclipse users to save early and save often. As soon as you wrote some code and you save the file that you're working on, the builder kicks in and tries to validate the new file state. Since you are continuing to edit, it's quite likely that you hit save again and are confronted with that modal dialog with greetings from the 90s. Of course you don't see this message all the time when you save a file since the incremental build is usually quite fast, but when you see it, it is definitely not what you expected.

Some Background

Generally speaking, the Eclipse builder is responsible to produce artifacts from the source files in a project. There may be different builders configured for the very same project and the term artifact does not only describe compilation results in the form of new files, but also validation markers. While a builder is running for a project, it holds an acquired lock not only for that project including its contained files and folders but for the entire workspace. This ensures that there are no intermitted events that remove or modify any state on disk (details have been discussed here). And this is where the trouble starts from the users perspective.
On the one hand, the locking prevents from unexpected modifications within Eclipse, on the other hand it gets in the way of users since they can no longer work without interruption. The thing was apparently designed to ensure consistency within the workspace between sources and compilation result. But if you look into the dirty corners, the paid price is way too high. The blocking mechanism introduces only the impression of safety but can never guarantee it. Literally every external process may still perform I/O operations on the very same files and the build would go bananas since the state known to Eclipse is no longer in sync with the actual state on disk. But that's probably another can of worms that is not subject of this post. Instead let's focus on ways to improve the situation which may lead to a more responsive UI.

Action Items

For Xtext, we are currently analyzing how we can change the way we build files and projects. Rather than getting in the way of the user, we are thinking about performing the build in the background without unnecessary blocking. The main goal with that regards is to move the complete build out of the coarse grained project lock and break it into manageable, smaller pieces. E.g. as soon as the files are loaded, they don't need to be locked anymore. In the validation phase only the markers are written but not the entire files. For incremental builds, only a small subset of files needs to be considered in the first place.
This breakdown of locking is desirable on various level. First and foremost, the user experience would be improved a lot since Xtext would present fewer blocking dialogs to the user. Another positive effect is that the build and its lifecycle would be essentially decoupled from the Eclipse builder and its related UI components. By factoring out the build cycle, Xtext can support incremental compilation on the command line, too.
In times of many-cores, it also becomes more and more interesting to parallelize the build to go full throttle with todays CPUs. The leverage the potential there, the build process itself has to be analyzed carefully. The Xtext build inherently runs in multiple passes that are currently strictly sequential, especially in the context of Eclipse projects. These steps are performed for each individual project during a build.
  1. First of all, the resources in a project have to be indexed to collect all the reachable names and derive an index structure that can be seen as a global symbol table.
  2. After all symbols and names are known, the cross references are resolved and their resolution state is cached as reference descriptions. Currently also the validation is performed on that stage but that can be seen as step 2.5
  3. The last step is the code generation. All resources are processed to create derived artifacts from them.
There are already means in Xtext to perform some steps in parallel. E.g. the loading of files into memory for stage (1) can be done in parallel rather than sequentially since Xtext 2.0. In the future, we want to improve on that and allow a lot more parallelization. Given that the build would be decoupled from the Eclipse builders lifecycle, we could index all the resources in the workspace at the same time. In phase (1), there is no need for one project to wait for another. Multiple projects would be processed in parallel rather than sequentially. Also the reference resolution can be done in parallel - at least if the projects do not depend on each other transitively. For the code generation, there is already support for parallelization since Xtext 2.7, but there's still room for improvements, e.g. we could not only generate resources within a single project in parallel but also run the full build concurrently for multiple projects.
But there's even more that we are discussing right now about the way Xtext projects are build within Eclipse. We are looking into means to preserve the index state if a project is closed by the user, for example. Instead of rebuilding the entire project, the builder state would be available immediately after the project is reopened, similar as with plain Java projects. Also the general handling of archives and resources in these archives is under review. For bigger projects, it may pay off to have precomputed information available that is packaged together with the resources in the archive.
In the end, the overall goal is to improve the perceived performance and the responsiveness of the IDE. Never ever should a user action be blocked by some task the IDE is performing in the background. The build should also be decoupled from the Eclipse infrastructure. With that regards, the contracts for each build step have to be sharpened and of course correctness should not be traded for concurrency. Exciting times!

Friday, October 3, 2014

Testing multiple Xtext DSLs

Recently there was a question in the Eclipse forum about testing multiple Xtext languages. The described scenario involves two language and one should have cross references to the other. Since this usecase caused some headaches in the past, Michael Vorburger provided a patch (Thanks for that!) that adds information about that particular topic to the official Xtext documentation. The updated docs are available since the recent 2.7 release. To provide some additional guidance, I hacked a small example that can be used as a blueprint if you want to get a jump start on that issue. This example also documents briefly how Xtext's unit test facilities can be used for effective language testing.

Key to testing the infrastructure of a DSL is the setup of a proper test environment. The Guice services have to be registered, EMF has to be initialized and obviously everything has to be cleaned up again after a test has been executed. For that purpose, Xtext provides a dedicated Junit4 test runner that uses an injector provider to do the initialization. The nice thing about that approach is that you can directly inject the necessary services into your unit test class. Exaclty as you are used to in your production code, too.

When it comes to testing multiple DSLs, basically the same infrastructure can be used, though you have to make sure, that all involved languages are properly initialized. For that purpose, a custom injector provider has to be derived from the one, that was already generated for your language. The to-be-written subclass needs to takes care of all the prerequisites and register the dependent languages. This mainly involves delegation to their injector providers.

Now that the setup is ready, we can test cross references between multiple DSLs. It is important to know that these references are only properly resolved if all models are available in the same resource set. That's why we need to use an explicit resource set in the tests. Besides that, it's the programming model that you know from Xtext and EMF in general.

A complete examplary test is available on Github.