My Eclipse Summit Europe 2010

This years Eclipse Summit in Ludwigsburg is around the corner and I'm eagerly looking forward to it: The program promises to be interesting and a conference is always a great opportunity to meet nice people - especially if they owe me a beer :-)

Sven and I will give a live demo on the development of an Xtext based CSS editor that leverages the expressivness of Xbase. You'll see how easy it is to create behavioral languages with Xbase. It was really a pleasure to prepare the demo and I think we have something awesome in the pipeline. Make sure that you attend our session on Thursday, 11:30 am.

However, the conference starts on Tuesday so first things first: The first item on my personal conference schedule is the tutorial on Android development powered by Eclipse Modeling. Jan, Holger and Dennis will help you to create a language with Xtext to model your first Android app. A team from Obeo will assist you in the second part of the tutorial with Acceleo to actually generate the application from your model. Tuesday afternoon is symposia time. I'll attend the modeling symposium which promises to hold some interesting lightning demos and discussions. There, I will show how to integrate models with existing Java code by means of the JVM Types Ecore model.

On Wednesday, Sven will give a talk about dependency injection and I'd advise every developer who does not yet use and breath DI to attend this session as it's about a really a great programming model. After you've been injinfected you'll not want to miss it - especially if you use Google's DI framework Guice. Furthermore I'm looking forward to the sessions on GIT, B3 or the Code Recommenders project. I think they cover exciting technologies and projects.

Besides my own session on Thursday I can recommend Heiko's and Peter's talk about mobile development with Eclipse Modeling. I've heard this talk (or a slightly other version of it) before and it's entertaining and technically interesting.

Altogether, the odds are good that it's going to be a great conference. So in case you are one of the lucky guys who registered in time for this years ESE: See you there!

Fun with Xtext: Language Mixins

There are rumors that Xtext does only have rudimentary support for language libraries or language reuse and does not allow to create 'advanced stuff'. And it is actually true, that Xtext has only first class support for extending only one other language. But with some creativity and customizing, it is easily possible to create really nice editors for intermixed languages. I've produced a small screencast that demonstrates how an editor for template proposals (read: a template language) may look like when it's aware of your target syntax.



Besides the smart editor for template proposals there is another interesting feature in this screencast: As template proposals are usually edited on a preference page, I implemented a small prototype that shows how a form field may leverage the features of a full blown Xtext editor.

Both prototypic implementations are only based on Xtext 1.0 APIs and not yet publicly available. But I'ld assume that at least parts of them will be moved into the Xtext Indigo stream. Stay tuned!

How to Reduce Memory Consumption of the Xtext Builder?

As the Xtext builder loads every affected resource into a single resource set to update its cached state and validate the changes, it tends to require a huge amount of memory for a clean build of a really large project. If there are thousands of files involved it may happen that the machine simply runs out of available physical memory. As a side-effect performance degraded significantly due to paging and the vm crashed with an OutOfMemoryError.

During the past weeks we worked hard to fix this for Xtext SR1 which is due end of August. And the good news is - we could tackle the problem at its root. Finally we were able to build huge projects with a reasonable heap size that previously refused to build with 8Gb++.

The solution was a (more or less) simple divide and conquer algorithm: We split the job into two well defined phases and clustered each phase. This enables the builder to release memory early and thereby drastically reduce the peak amount of required heap.

To enable the clustering Xtext builder in your IDE, you have to override some central functionality that drives the builder for every language in your running IDE. Simply add an extension to your ui plugin that overrides the common configuration.
<br /> <extension point=<br /> "o.e.x.ui.shared.overridingGuiceModule"><br /> <module class=<br /> "..builder.clustering.ClusteringModule"/><br /></extension><br />
The second step is to override a binding in the ui module of your language to be communicate with the clustered builder. This implies that each Xtext based language that is installed in the IDE, has to expect the clustering builder instead of the default implementation. Otherwise you'll be confronted with an exception due to incompatible settings. The bindind in your ui module replaces the component that is used by the scope provider in the builder context:
<br /> @Override<br />public void<br /> configureIResourceDescriptionsBuilderScope(<br /> Binder binder) {<br /> binder.bind(IResourceDescriptions.class)<br /> .annotatedWith(<br /> Names.named(AbstractGlobalScopeProvider<br /> .NAMED_BUILDER_SCOPE))<br /> .to(CurrentDescriptions.ResourceSetAware<br /> .class);<br />}<br />
Besides this global change to reduce the memory consumption, you may want to optimize some language specific implementations. It is always a good idea to review the number of exported objects. There is usually no need to export any object that has a name. E.g. a local variably is only reachable from inside the resource, it can simply be skipped when the exported objects are computed. Another good candidate is the fragment provider. The default fragments for resources are quite generic and somehow verbose. It will not hurt if you can come up with a shorter, resource specific representation. Both customization points will usually not only make your language consume less memory but even have potential to improve the performance as well because you can use language specific information in your implementation.

How to Deploy Xtext to a Headless Plain-Java Environment?

Yesterday, Sven and I were at the Eclipse Democamp in Hannover which was organized by brox IT-Solutions and Bredex - thanks for the nice event. I met interesting people and we discussed various topics. One particular question about Xtext was raised several times from different sides: How do I use my language in an environment without Eclipse, e.g. on a build server? As this was already possible from the first minute even with the former oAW version of Xtext and with Xtext 0.7.0, too, I was kind of suprised. In fact, any execution of the generator workflow uses Xtext in a standalone environment - without OSGi, Equinox or any dependency on Eclipse UI. The problem seems to be another one: How can I determine all the required libraries to deploy my Xtext-driven generator?

It turns out that this task is suprisingly easy with Eclipse. We'll use a minor trick to simplify the necessary steps. First of all, we have to create a new Run Configuration for a Java Application. Actually this one will never be executed, but we need to refer to this one in the next step. Just create a new configuration for your generator-project and use the org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher as the main class:

Now that we have a run configuration for the generator project, let's export the project as a runnable jar. Choose Export... -> Runnable JAR file from the context menu of the generator project. The wizard will ask for a launch configuration and that's where we use the one that we created a second ago:

Select the options of your choice, point to a target file and hit the Finish-button. Eclipse will collect required libraries and add the necessary entries to the MANIFEST.MF in the runnable JAR file. As I already said, it's suprisingly simple.

Last but not least, I recommend to create a batch file or shell script that starts a virtual machine, points to the exported archive and refers to the module name of the generator workflow, e.g.:
java -jar exported.jar workflow.MyDslGenerator
It's probably a good idea to wrap the invocation of the Mwe2Launcher in a custom main class to encapsulate the name of the generator workflow and pass additional arguments to the workflow.

New In Xtext: Case Insensitive Languages

A quick glimpse at the New and Noteworthy features of Xtext 1.0 reveals that the framework allows to develop languages that are case insensitive.

To mark upper and lower cases meaningless for keywords or cross references, you just have to replace two default fragments in the MWE2 workflow, add an option and hit the Run button:
<br /> ..<br />fragment = AntlrGeneratorFragment {<br /> options = {<br /> // ignore case keywords<br /> ignoreCase = true<br /> }<br />}<br />..<br />fragment =<br /> scoping.ImportNamespacesScopingFragment {<br /> // ignore case cross references<br /> ignoreCase = true<br />}<br />..<br />fragment =<br /> ContentAssistParserGeneratorFragment {<br /> options = {<br /> // ignore case keywords<br /> ignoreCase = true<br /> }<br />}<br />..<br />
The o.e.x.g.parser.antlr.ex.rt.AntlrGeneratorFragment and o.e.x.g.parser.antlr.ex.ca.ContentAssistParserGeneratorFragment replace the two fragments, that are configured by default: o.e.x.g.parser.antlr.XtextAntlrGeneratorFragment and o.e.x.g.parser.antlr.XtextAntlrUiGeneratorFragment.

Xtext provides an API that allows to define custom, local scopes that are ignore case as well. Have a look at the utility class o.e.x.resource.ignorecase.IgnoreCaseScopes to learn more about it. The configured generator fragments will take care of the rest, e.g. the default bindings for the global scopes, the parser and the serializer.

As you can see it is possible to define case insensitivity independently for keywords and cross-links. And if you want to keep your workflow short and concise instead of adding this flag for each and every fragment that accepts this option, you are free to introduce a new property and use this one directly or via the auto-inject option. Please refer to the updated documentation if you want to learn more about the MWE2 language.

Customizing Error Messages in Xtext 1.0

As the Eclipse Helios Release is around the corner I'll use the chance to describe some nice odds and ends in Xtext 1.0. One of which is a newly introduced API that allows to customize the default messages for parse errors and linking diagnostics.

If you try to read erroneous models, you'll get annotations for syntax errors and linking problems in the editor by default. These error messages are not always the most meaningful ones from a user's point of view. Furthermore it was not possible to i18n them. This was changed with Xtext 1.0 by means of two newly introduced customization hooks.

Users who want to alter the default messages that the parser produces may implement a so called ISyntaxErrorMessageProvider which allows to change the wording of syntax errors and messages from the value converter.

<br /> interface ISyntaxErrorMessageProvider {<br /> SyntaxErrorMessage getSyntaxErrorMessage(<br /> IParserErrorContext context);<br /> SyntaxErrorMessage getSyntaxErrorMessage(<br /> IValueConverterErrorContext context);<br /> }<br />
The implementor has the current state of the parser at hand to produce a meaningful message for the user. Basically any information is available to compute the SyntaxErrorMessage. She is free to use the intermediate semantic model, the attached node model that reflects the parsed tokens, or the really low level stream that the parser works with.

<br /> interface IErrorContext {<br /> String getDefaultMessage();<br /> EObject getCurrentContext();<br /> AbstractNode getCurrentNode();<br /> }<br /> interface IParserErrorContext<br /> extends IErrorContext {<br /> RecognitionException<br /> getRecognitionException();<br /> String[] getTokenNames();<br /> }<br /> interface IValueConverterErrorContext<br /> extends IErrorContext {<br /> ValueConverterException<br /> getValueConverterException();<br /> }<br />
Especially handy is the option to assign an issue code to a syntax error message. This is the constant that a quick fix refers to, to solve your models syntactic problems easily. Assuming you evolve the syntax of your language, you may want to define such quick fixes that migrate old files to the new notation. Of course there are more use cases such as mandatory keyword sequences at the end of an element.

There is another component that produces default error messages: If the linker cannot resolve an EMF proxy when navigating a cross reference, it will create an error diagnostic, too. We used the same pattern to allow clients to customize this message.

<br /> interface ILinkingDiagnosticMessageProvider{<br /> DiagnosticMessage getUnresolvedProxyMessage(<br /> ILinkingDiagnosticContext context);<br /> }<br /> interface ILinkingDiagnosticContext {<br /> EObject getContext();<br /> EReference getReference();<br /> String getLinkText();<br /> }<br />
Again, every piece of information is available in the context, that is passed to the message provider and of course it is possible to assign an issue code to the linking diagnostic. Even more, clients may choose to reduce the severity of the linking problem to a warning. Languages that support a kind of dynamic invoke like C# may establish real links for the statically bound features but ignore unresolved references for dynamic parts.

Both APIs may come quite handy if you want to i18n your applicaton or tailor the error messages that are presented to the user.

EclipseCon 2010 - A Review

This year was the first time that I attended the EclipseCon in Santa Clara and it was a really great event. However, the very first nice impression had actually nothing to do with the EclipseCon itself but with the nice weather in California. As the journey started in the rainy northern Germany, it was very pleasing to arrive in San Francisco with nearly 70F. To fight the jet lag back we rent some bikes and explored the vicinity on Sundays. It was a very enjoyable trip through the Bay Area. Nevertheless the night was rather short for me.

EclipseCon 2010

The first session on Monday morning was the Xtext meets e4 tutorial. It was especially interesting for me because Jan, Moritz and Heiko demonstrated a more user friendly approach to workbench modeling than editing tree editors and property sheets. I hope that at least some parts of the textual approach to store workbench information will make it into e4 as I personally do not like working with xmi-based persistence and form editors. However, up to now nobody wanted to jump in and shape up the prototypic Xtext editor. Hint, hint!

The next session was about The Twenty Modeling Things. Kenn and Stephan talked basically about some missing pieces in the Eclipse Modeling Project that will be covered by the Sphinx proposal and showed some of the IDE concepts that will be available. Afterwards I wanted to learn something about B3 from Henrik, but unfortunately the room was totally overcrowded. People seem to be aware of the fact, that automated builds that work the same way on the integration server as in the local Eclipse workspace are crucial to reliability and testability. Because building has been kind of a pain point for us in the past, I am especially happy about the new Buckminster build for Xtext on the Athena server, that Thomas set up last week.

On Tuesday, Sven and I presented some the new and especially the noteworthy things in Xtext. It is always fun to talk about the things that we have been busy with in the past months. The next event for modeling enthusiasts was the Modeling Runway. Although there were some technical ... inconveniences at the beginning, they turned into entertainment when Kenn advanced the slides or Sven's Xtext fast forward demo.

Wednesday was the day of THE keynote. Jeff Norris from the NASA played with all those things that boys dream of - and most likely still do as men. He remote controlled an 8 feet robot on the NASA area. It requires a lot of confidence in the software that you built when you perform those things live on stage. In the afternoon I gave my lightning talk about the new MWE tooling based on Xtext and a runtime layer that got an ultra slim diet.
The BoF about IDE tooling for the e4 development model was the last session on Wednesday. It was quite interesting and insightful to discuss the various ideas and approaches towards a steeper learning curve when developing plugins for e4. Tonny summarized the results in his blog and created or commented a few bugzillas that reflect the ideas.

Thursday went by as fast as it came. It was a day of some short discussions and I used the chance to do session hopping and hear something about basically everything in parallel. In the late afternoon, we returned the bikes and had a Mexican dinner before the traditional European Reception took place.

Conclusion

Although I expected the EclipseCon to take place in the Convention Center, I think it was a good decision to use the conference rooms in the Hyatt Hotel instead. Unfortunately they were a little too small for some of the talks and too low-ceiling so the canvas was probably not fully visible from the rows in the back. However, this minor drawback was fully compensated by the nice atmosphere during the conference. People ran literally into each other anytime and there were plenty of opportunities to chat about various topics. It was really easy to get in touch.

What I personally took along was that the development of custom language support is becoming kind of mainstream. It seems to me that people have learned that XML is not the best way to store human readable and editable information. The Xtext framework provides the necessary abstractions to easily develop the infrastructure to parse, process and edit files in a very convenient way. Projects like B3, EMF Query or AMP already adopted Xtext to provide tooling for programming or configuration languages and others decided after in-depth discussions that this might be a approach worthwhile to explore. Many people came to the itemis booth to talk about their proprietary solutions and we sketched many ideas on how to supersede existing implementations with EMF and Xtext. It was great fun to feel the vibes around Xtext.

A sneak peek at the new MWE editor

The Modeling Workflow Engine (MWE) basically allows to assemble and orchestrate Java objects to use the resulting instances. A major use case is the composition of different Eclipse Modeling components to create an automated workflow for model based development approaches. Model readers, tranformation and validation engines or code generators can be plugged together to create a complete chain from the edited model to a runnable application or at least a bunch of refinable source code. It is even possible to use MWE in your own application to configure it externally.

One drawback of the current version of MWE is the lack of tool support when you have to edit the workflow configuration files. Although the XML dialect is quite optimized and even the simple XML editor helps a lot with support for paired tags, syntax validation and highlighting, it does not provide more sophicated features like MWE specific content assist, constraint checking or navigation.

Therefore, improved tooling is a major item on the roadmap to MWE 0.8.0 which will be part of the Eclipse Helios release. We currently discuss a refinement of the syntax for workflow files. It should be more concise and we want to provide a state of the art editor with advanced features. That's why we created a first prototype for a new MWE language at the end of the last year based on the Xtext framework.

Support for externalized properties



MWE files allow users to externalize configuration values into properties to make them reusable and improve the readability of a workflow. Properties may be concatenated into another property, declared in the workflow itself or in a properties file. All the rich editor features like content assist, validation and go-to declaration are available for properties and composed properties as well.

Content assist



As MWE uses reflection to assemble Java object graphs, the configuration files contain a lot of references to Java types and their methods. Based on the type compatibility constraints of the language, the editor will propose all the valid Java types. The necessary information will be derived from the classpath configuration of your Java projects and even the types in your source folders are available in your MWE file.

Class imports



As we leverage many powerful features of the Xtext framework, it was easy to introduce import statements for Java types to keep the syntax even more compact. Even the content assist is aware of existing imports and will automatically insert a fully qualified name if necessary and a simple name if possible.

Navigate to types and methods



Advanced navigation features like "Go To Declaration" make it easier to understand the configuration itself as well as the configured components. The new MWE editor makes it possible to jump directly to the appropriate members or types that your workflow refers to.

Conclusion

The new concrete syntax for the MWE language is easier to read and edit then the existing XML format. Nevertheless, the ideas behind MWE will remain stable. The language consists of a small number of orthogonal, seperated concepts which make it easy to understand and yet powerful. The new syntax focusses on those clean core concepts to make them easier to grasp while providing sophisticated editor features and tool support.

What's next

The prototype is already available in the Eclipse CVS although it still lacks some features and - more important - a connector to the runtime of MWE. We are currently planning to make it available in Summer 2010 in line with the Eclipse Helios release.