Xtend2 - What's In The Pipeline?

In the last weeks, the Xtext team has been busy with creating Xtend2. Xtend2 is the successor for Xtend and Xpand and implemented on top of Xbase¹. I think we made really great progress both within Xbase - which is a reusable expression language that can be embedded into any language that is developed with Xtext - and with Xtend2 itself. 

Xtend2 provides some unique language features and great Eclipse based tooling. It's really a pleasure to see how the well choosen orthogonal concepts interact with each other and how their total creates a language that feels as fantastic as Sven predicted.

• The extension concept of Xtend2 allows to import extension methods in a static and in a dynamic manner. Dynamically imported extensions are injected into your instances by means of dependency injection. It is really straight forward to create a customizable and adoptable code generator simply be using well defined components that are assembled by a DI container such as Google Guice.
• Polymorphic dispatching allows to invoke overloaded methods according to the runtime type of their arguments instead of a static decision. This is especially useful if you want to process heterogenous data structures such as EMF models. It is even possible to rely on this runtime binding when you use Xtend2 classes from Java code.
• Even though Xtend2 compiles to java code which is not dependent on JDK7 (or later), we provide full support for closures. Writing own higher order functions that take closures as arguments or return function types is possible, too.
• Type inference works for expression trees and across method boundaries. It is usually not necessary to define the type of closure parameters, local variables or the return type of methods since the type system will derive it automagically. It works great with generics, too.
• Xtend2's rich strings allow to write code generator easily. We came up with a neat solution for the classical whitespace problem in template languages. The IDE support that includes validation of template indentation, quick fixes and semantic highlighting helps to write code generators or compilers in a readable way while the output looks nice, too.
• While we're at IDE support, did I mention the refactoring capabilities, content assist, hover information or the other features that are available in the Xtend2 editor because?

If you are at EclipseCon and as eager to get your hands dirty with Xtend2 as I am, make sure to attend the tutorial that Jan, Sven and I will give there.

[1] The Xbase development is sponsored by itemis and the Federal Ministry of Education and Research.

 

My Eclipse Summit Europe 2010

This years Eclipse Summit in Ludwigsburg is around the corner and I'm eagerly looking forward to it: The program promises to be interesting and a conference is always a great opportunity to meet nice people - especially if they owe me a beer :-)

Sven and I will give a live demo on the development of an Xtext based CSS editor that leverages the expressivness of Xbase. You'll see how easy it is to create behavioral languages with Xbase. It was really a pleasure to prepare the demo and I think we have something awesome in the pipeline. Make sure that you attend our session on Thursday, 11:30 am.

However, the conference starts on Tuesday so first things first: The first item on my personal conference schedule is the tutorial on Android development powered by Eclipse Modeling. Jan, Holger and Dennis will help you to create a language with Xtext to model your first Android app. A team from Obeo will assist you in the second part of the tutorial with Acceleo to actually generate the application from your model. Tuesday afternoon is symposia time. I'll attend the modeling symposium which promises to hold some interesting lightning demos and discussions. There, I will show how to integrate models with existing Java code by means of the JVM Types Ecore model.

On Wednesday, Sven will give a talk about dependency injection and I'd advise every developer who does not yet use and breath DI to attend this session as it's about a really a great programming model. After you've been injinfected you'll not want to miss it - especially if you use Google's DI framework Guice. Furthermore I'm looking forward to the sessions on GIT, B3 or the Code Recommenders project. I think they cover exciting technologies and projects.

Besides my own session on Thursday I can recommend Heiko's and Peter's talk about mobile development with Eclipse Modeling. I've heard this talk (or a slightly other version of it) before and it's entertaining and technically interesting.

Altogether, the odds are good that it's going to be a great conference. So in case you are one of the lucky guys who registered in time for this years ESE: See you there!

Fun with Xtext: Language Mixins

There are rumors that Xtext does only have rudimentary support for language libraries or language reuse and does not allow to create 'advanced stuff'. And it is actually true, that Xtext has only first class support for extending only one other language. But with some creativity and customizing, it is easily possible to create really nice editors for intermixed languages. I've produced a small screencast that demonstrates how an editor for template proposals (read: a template language) may look like when it's aware of your target syntax.



Besides the smart editor for template proposals there is another interesting feature in this screencast: As template proposals are usually edited on a preference page, I implemented a small prototype that shows how a form field may leverage the features of a full blown Xtext editor.

Both prototypic implementations are only based on Xtext 1.0 APIs and not yet publicly available. But I'ld assume that at least parts of them will be moved into the Xtext Indigo stream. Stay tuned!

How to Reduce Memory Consumption of the Xtext Builder?

As the Xtext builder loads every affected resource into a single resource set to update its cached state and validate the changes, it tends to require a huge amount of memory for a clean build of a really large project. If there are thousands of files involved it may happen that the machine simply runs out of available physical memory. As a side-effect performance degraded significantly due to paging and the vm crashed with an OutOfMemoryError.

During the past weeks we worked hard to fix this for Xtext SR1 which is due end of August. And the good news is - we could tackle the problem at its root. Finally we were able to build huge projects with a reasonable heap size that previously refused to build with 8Gb++.

The solution was a (more or less) simple divide and conquer algorithm: We split the job into two well defined phases and clustered each phase. This enables the builder to release memory early and thereby drastically reduce the peak amount of required heap.

To enable the clustering Xtext builder in your IDE, you have to override some central functionality that drives the builder for every language in your running IDE. Simply add an extension to your ui plugin that overrides the common configuration.
<br /> <extension point=<br /> "o.e.x.ui.shared.overridingGuiceModule"><br /> <module class=<br /> "..builder.clustering.ClusteringModule"/><br /></extension><br />
The second step is to override a binding in the ui module of your language to be communicate with the clustered builder. This implies that each Xtext based language that is installed in the IDE, has to expect the clustering builder instead of the default implementation. Otherwise you'll be confronted with an exception due to incompatible settings. The bindind in your ui module replaces the component that is used by the scope provider in the builder context:
<br /> @Override<br />public void<br /> configureIResourceDescriptionsBuilderScope(<br /> Binder binder) {<br /> binder.bind(IResourceDescriptions.class)<br /> .annotatedWith(<br /> Names.named(AbstractGlobalScopeProvider<br /> .NAMED_BUILDER_SCOPE))<br /> .to(CurrentDescriptions.ResourceSetAware<br /> .class);<br />}<br />
Besides this global change to reduce the memory consumption, you may want to optimize some language specific implementations. It is always a good idea to review the number of exported objects. There is usually no need to export any object that has a name. E.g. a local variably is only reachable from inside the resource, it can simply be skipped when the exported objects are computed. Another good candidate is the fragment provider. The default fragments for resources are quite generic and somehow verbose. It will not hurt if you can come up with a shorter, resource specific representation. Both customization points will usually not only make your language consume less memory but even have potential to improve the performance as well because you can use language specific information in your implementation.

How to Deploy Xtext to a Headless Plain-Java Environment?

Yesterday, Sven and I were at the Eclipse Democamp in Hannover which was organized by brox IT-Solutions and Bredex - thanks for the nice event. I met interesting people and we discussed various topics. One particular question about Xtext was raised several times from different sides: How do I use my language in an environment without Eclipse, e.g. on a build server? As this was already possible from the first minute even with the former oAW version of Xtext and with Xtext 0.7.0, too, I was kind of suprised. In fact, any execution of the generator workflow uses Xtext in a standalone environment - without OSGi, Equinox or any dependency on Eclipse UI. The problem seems to be another one: How can I determine all the required libraries to deploy my Xtext-driven generator?

It turns out that this task is suprisingly easy with Eclipse. We'll use a minor trick to simplify the necessary steps. First of all, we have to create a new Run Configuration for a Java Application. Actually this one will never be executed, but we need to refer to this one in the next step. Just create a new configuration for your generator-project and use the org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher as the main class:

Now that we have a run configuration for the generator project, let's export the project as a runnable jar. Choose Export... -> Runnable JAR file from the context menu of the generator project. The wizard will ask for a launch configuration and that's where we use the one that we created a second ago:

Select the options of your choice, point to a target file and hit the Finish-button. Eclipse will collect required libraries and add the necessary entries to the MANIFEST.MF in the runnable JAR file. As I already said, it's suprisingly simple.

Last but not least, I recommend to create a batch file or shell script that starts a virtual machine, points to the exported archive and refers to the module name of the generator workflow, e.g.:
java -jar exported.jar workflow.MyDslGenerator
It's probably a good idea to wrap the invocation of the Mwe2Launcher in a custom main class to encapsulate the name of the generator workflow and pass additional arguments to the workflow.

New In Xtext: Case Insensitive Languages

A quick glimpse at the New and Noteworthy features of Xtext 1.0 reveals that the framework allows to develop languages that are case insensitive.

To mark upper and lower cases meaningless for keywords or cross references, you just have to replace two default fragments in the MWE2 workflow, add an option and hit the Run button:
<br /> ..<br />fragment = AntlrGeneratorFragment {<br /> options = {<br /> // ignore case keywords<br /> ignoreCase = true<br /> }<br />}<br />..<br />fragment =<br /> scoping.ImportNamespacesScopingFragment {<br /> // ignore case cross references<br /> ignoreCase = true<br />}<br />..<br />fragment =<br /> ContentAssistParserGeneratorFragment {<br /> options = {<br /> // ignore case keywords<br /> ignoreCase = true<br /> }<br />}<br />..<br />
The o.e.x.g.parser.antlr.ex.rt.AntlrGeneratorFragment and o.e.x.g.parser.antlr.ex.ca.ContentAssistParserGeneratorFragment replace the two fragments, that are configured by default: o.e.x.g.parser.antlr.XtextAntlrGeneratorFragment and o.e.x.g.parser.antlr.XtextAntlrUiGeneratorFragment.

Xtext provides an API that allows to define custom, local scopes that are ignore case as well. Have a look at the utility class o.e.x.resource.ignorecase.IgnoreCaseScopes to learn more about it. The configured generator fragments will take care of the rest, e.g. the default bindings for the global scopes, the parser and the serializer.

As you can see it is possible to define case insensitivity independently for keywords and cross-links. And if you want to keep your workflow short and concise instead of adding this flag for each and every fragment that accepts this option, you are free to introduce a new property and use this one directly or via the auto-inject option. Please refer to the updated documentation if you want to learn more about the MWE2 language.

Customizing Error Messages in Xtext 1.0

As the Eclipse Helios Release is around the corner I'll use the chance to describe some nice odds and ends in Xtext 1.0. One of which is a newly introduced API that allows to customize the default messages for parse errors and linking diagnostics.

If you try to read erroneous models, you'll get annotations for syntax errors and linking problems in the editor by default. These error messages are not always the most meaningful ones from a user's point of view. Furthermore it was not possible to i18n them. This was changed with Xtext 1.0 by means of two newly introduced customization hooks.

Users who want to alter the default messages that the parser produces may implement a so called ISyntaxErrorMessageProvider which allows to change the wording of syntax errors and messages from the value converter.

<br /> interface ISyntaxErrorMessageProvider {<br /> SyntaxErrorMessage getSyntaxErrorMessage(<br /> IParserErrorContext context);<br /> SyntaxErrorMessage getSyntaxErrorMessage(<br /> IValueConverterErrorContext context);<br /> }<br />
The implementor has the current state of the parser at hand to produce a meaningful message for the user. Basically any information is available to compute the SyntaxErrorMessage. She is free to use the intermediate semantic model, the attached node model that reflects the parsed tokens, or the really low level stream that the parser works with.

<br /> interface IErrorContext {<br /> String getDefaultMessage();<br /> EObject getCurrentContext();<br /> AbstractNode getCurrentNode();<br /> }<br /> interface IParserErrorContext<br /> extends IErrorContext {<br /> RecognitionException<br /> getRecognitionException();<br /> String[] getTokenNames();<br /> }<br /> interface IValueConverterErrorContext<br /> extends IErrorContext {<br /> ValueConverterException<br /> getValueConverterException();<br /> }<br />
Especially handy is the option to assign an issue code to a syntax error message. This is the constant that a quick fix refers to, to solve your models syntactic problems easily. Assuming you evolve the syntax of your language, you may want to define such quick fixes that migrate old files to the new notation. Of course there are more use cases such as mandatory keyword sequences at the end of an element.

There is another component that produces default error messages: If the linker cannot resolve an EMF proxy when navigating a cross reference, it will create an error diagnostic, too. We used the same pattern to allow clients to customize this message.

<br /> interface ILinkingDiagnosticMessageProvider{<br /> DiagnosticMessage getUnresolvedProxyMessage(<br /> ILinkingDiagnosticContext context);<br /> }<br /> interface ILinkingDiagnosticContext {<br /> EObject getContext();<br /> EReference getReference();<br /> String getLinkText();<br /> }<br />
Again, every piece of information is available in the context, that is passed to the message provider and of course it is possible to assign an issue code to the linking diagnostic. Even more, clients may choose to reduce the severity of the linking problem to a warning. Languages that support a kind of dynamic invoke like C# may establish real links for the statically bound features but ignore unresolved references for dynamic parts.

Both APIs may come quite handy if you want to i18n your applicaton or tailor the error messages that are presented to the user.