Eszetts blog

Xtend @ EclipseCon Europe 2011

2011-11-03T16:41:00.000+01:00

The second day of the EclipseCon Europe 2011 is almost over and so far it's been a very nice conference. The sessions that I attended were quite interesting and the 10th birthday party was very nice. The folks from the Eclipse community are really sociable!

Now I'm looking forward to my session about Xtend - which I'm very excited about. Xtend is a programming language that is made for Java developers. With support for closures, extension methods or template expressions it allows to use some of the pending Java 8 features right now. In that sense it serves as a complement to the state of the art programming language - and integrates nicely with all the existing libraries and APIs. But first and foremost it's a real pleasure to write implementation code again, especially for scenarios where Java doesn't shine too bright. If you want to learn more about Xtend, please come to my session in the Theater Stage on Friday, 14:00.

Customizing Content Assist with Xtext

2011-06-19T22:27:00.000+02:00

As basically every part of Eclipse Xtext, the content assist API provides a reasonable default behavior and is greatly customizable by means of dependency injection. This blog post demonstrates, how easy it is to create own completion proposals with this API.

For the sake of simplicity, I'll use a grammar that fits in a single line:

It allows to define exactly one color literal and the syntax is not really strict about the literal's format. Valid input files are considered to look like color "0,0,0" or color "255,255,255".

Now comes the fun part: The proposal list for the rgb string should provide some frequently used values, e.g. literals for black or white.

Therefore I have to customize the proposal provider. Xtext already provides an empty stub implementation that will take the custom code. In order to change the proposals for the rgb-value in the parser rule Color, the method completeColor_Rgb(..) has to be overridden. JDT's content assist or the quick outline in the proposal provider implementation helps to find all suitable entry points.

Two lines of code later, I can already insert literals for black or white:

However, there are far more possibilities than simply creating plain vanilla proposals. The overloaded factory method createCompletionProposal(..) returns an instance of the ConfigurableCompletionProposal which has a quite powerful API. It allows to change the appearance of the display string, I can set additional hover information, enable linked edit modes for the custom proposal or make it auto-insertable to name only a few.

The next example uses an own ReplacementTextApplier to hook into the actual moment, when the proposal is applied:

You'll guess that already from the code: This implementation allows to pick the color by means of a color chooser after the proposal has been selected, but see yourself:

There are plenty of other interesting APIs and services around the code completion in Xtext that are worthy of talking about (e.g. the prefix matcher, which Alex covered in his recent post), but I'll leave them for another post.

Xdoc: A Documentation Language For Eclipse Frameworks

2011-06-14T23:59:00.000+02:00

Xdoc is a small markup language that allows to create code centric documentation in a very convenient way. The documents can be rendered to HTML, to Eclipse Help files and to a printable PDF.
Besides the usual markup tags to emphasize text, add images and tables or create structures such as chapters, sections and the like, there are two unique features of Xdoc that I want to highlight.

First of all, it is possible to refer to Java types in a way that allows to statically validate these references. This feature comes very handy as soon as you want to describe important hooks of a framework or codebase. The references will be rendered as hyperlinks to the online Javadoc and if there is a public source repository, a second link points directly to the code.

The second interesting feature is related to code snippets in the documentation. It is often valuable to document some usage patterns for a certain class or interface by example. However, making these snippets easy to read and achieving the familiar look of an Eclipse editor with syntax coloring and the like often requires a lot of effort. Xdoc simplifies this greatly. You just define a set of keywords for a language and the renderer will take care of the highlighting for your code snippets in a consistent manner. This is a big improvement when it comes to the appearance of your help files.

To provide just a few technical details: Xdoc was build with Xtext. That is, the Eclipse editor supports all the nice features that you are already familiar with from other editors. The code generator which creates the Eclipse Help files on the fly is written in Xtend which in turn is implemented on top of Xtext 2.0. Xdoc itself was used to write the Eclipse Help for Xtext and Xtend. Actually, the Xtend Help was written before Xtend was implemented, but that's another story. The chicken or the egg? Things get a little blurry these days... Did I mention that Xtext was used to implement Xtext?

New Features In The Xtext Grammar Editor

2011-06-07T23:46:00.001+02:00

Besides the huge new features like the Xbase library, the new Xtend and the cross language refactoring capabilities, we made sure that we still find some time to implement small refinements and improvements in Xtext's core. Some of them may be not too obvious but I'm pretty sure they will actually have some impact on the overall user experience when building new languages with the framework. This blog post is about two nice additions in the grammar editor that are available in Xtext 2.0 with Eclipse Indigo.

Improved content assist

You'll often want to create a concrete syntax for an existing EPackage which was somewhat cumbersome in the past since it involved copy and paste of the namespace uri. The good news is:
Adding imported packages was greatly simplified. It's now far easier to find the actual EPackage that you want to use in your grammar because content assist matches for the more selective last segments of the namespace uri, too:

Speaking about imported packages: If you want to override a rule from another grammar, you can now use content assist to create a stub for that. The editor will take care of necessary import clauses:

Quick Fixes

If your workflow is similar to mine, you'll usually develop a grammar top-down. This implies that your grammar often contains a bunch of unresolved rule calls since the invocation of a parser rule appears before the rule itself is written. A new quick fix allows to create stubs for unknown rules as soon as you refer to a rule which does not yet exist:

I hope you like these small improvements as much as I do!

Xtend2 - What's In The Pipeline?

2011-02-28T09:33:00.001+01:00

In the last weeks, the Xtext team has been busy with creating Xtend2. Xtend2 is the successor for Xtend and Xpand and implemented on top of Xbase¹. I think we made really great progress both within Xbase - which is a reusable expression language that can be embedded into any language that is developed with Xtext - and with Xtend2 itself.

Xtend2 provides some unique language features and great Eclipse based tooling. It's really a pleasure to see how the well choosen orthogonal concepts interact with each other and how their total creates a language that feels as fantastic as Sven predicted.

The extension concept of Xtend2 allows to import extension methods in a static and in a dynamic manner. Dynamically imported extensions are injected into your instances by means of dependency injection. It is really straight forward to create a customizable and adoptable code generator simply be using well defined components that are assembled by a DI container such as Google Guice.
Polymorphic dispatching allows to invoke overloaded methods according to the runtime type of their arguments instead of a static decision. This is especially useful if you want to process heterogenous data structures such as EMF models. It is even possible to rely on this runtime binding when you use Xtend2 classes from Java code.
Even though Xtend2 compiles to java code which is not dependent on JDK7 (or later), we provide full support for closures. Writing own higher order functions that take closures as arguments or return function types is possible, too.
Type inference works for expression trees and across method boundaries. It is usually not necessary to define the type of closure parameters, local variables or the return type of methods since the type system will derive it automagically. It works great with generics, too.
Xtend2's rich strings allow to write code generator easily. We came up with a neat solution for the classical whitespace problem in template languages. The IDE support that includes validation of template indentation, quick fixes and semantic highlighting helps to write code generators or compilers in a readable way while the output looks nice, too.
While we're at IDE support, did I mention the refactoring capabilities, content assist, hover information or the other features that are available in the Xtend2 editor because?

If you are at EclipseCon and as eager to get your hands dirty with Xtend2 as I am, make sure to attend the tutorial that Jan, Sven and I will give there.

[1] The Xbase development is sponsored by itemis and the Federal Ministry of Education and Research.

My Eclipse Summit Europe 2010

2010-10-31T00:00:00.006+02:00

This years Eclipse Summit in Ludwigsburg is around the corner and I'm eagerly looking forward to it: The program promises to be interesting and a conference is always a great opportunity to meet nice people - especially if they owe me a beer :-)

Sven and I will give a live demo on the development of an Xtext based CSS editor that leverages the expressivness of Xbase. You'll see how easy it is to create behavioral languages with Xbase. It was really a pleasure to prepare the demo and I think we have something awesome in the pipeline. Make sure that you attend our session on Thursday, 11:30 am.

However, the conference starts on Tuesday so first things first: The first item on my personal conference schedule is the tutorial on Android development powered by Eclipse Modeling. Jan, Holger and Dennis will help you to create a language with Xtext to model your first Android app. A team from Obeo will assist you in the second part of the tutorial with Acceleo to actually generate the application from your model. Tuesday afternoon is symposia time. I'll attend the modeling symposium which promises to hold some interesting lightning demos and discussions. There, I will show how to integrate models with existing Java code by means of the JVM Types Ecore model.

On Wednesday, Sven will give a talk about dependency injection and I'd advise every developer who does not yet use and breath DI to attend this session as it's about a really a great programming model. After you've been ~~inj~~infected you'll not want to miss it - especially if you use Google's DI framework Guice. Furthermore I'm looking forward to the sessions on GIT, B3 or the Code Recommenders project. I think they cover exciting technologies and projects.

Besides my own session on Thursday I can recommend Heiko's and Peter's talk about mobile development with Eclipse Modeling. I've heard this talk (or a slightly other version of it) before and it's entertaining and technically interesting.

Altogether, the odds are good that it's going to be a great conference. So in case you are one of the lucky guys who registered in time for this years ESE: See you there!

Fun with Xtext: Language Mixins

2010-08-29T22:22:00.002+02:00

There are rumors that Xtext does only have rudimentary support for language libraries or language reuse and does not allow to create 'advanced stuff'. And it is actually true, that Xtext has only first class support for extending only one other language. But with some creativity and customizing, it is easily possible to create really nice editors for intermixed languages. I've produced a small screencast that demonstrates how an editor for template proposals (read: a template language) may look like when it's aware of your target syntax.

Besides the smart editor for template proposals there is another interesting feature in this screencast: As template proposals are usually edited on a preference page, I implemented a small prototype that shows how a form field may leverage the features of a full blown Xtext editor.

Both prototypic implementations are only based on Xtext 1.0 APIs and not yet publicly available. But I'ld assume that at least parts of them will be moved into the Xtext Indigo stream. Stay tuned!

How to Reduce Memory Consumption of the Xtext Builder?

2010-08-14T14:55:00.017+02:00

As the Xtext builder loads every affected resource into a single resource set to update its cached state and validate the changes, it tends to require a huge amount of memory for a clean build of a really large project. If there are thousands of files involved it may happen that the machine simply runs out of available physical memory. As a side-effect performance degraded significantly due to paging and the vm crashed with an OutOfMemoryError.

During the past weeks we worked hard to fix this for Xtext SR1 which is due end of August. And the good news is - we could tackle the problem at its root. Finally we were able to build huge projects with a reasonable heap size that previously refused to build with 8Gb++.

The solution was a (more or less) simple divide and conquer algorithm: We split the job into two well defined phases and clustered each phase. This enables the builder to release memory early and thereby drastically reduce the peak amount of required heap.

To enable the clustering Xtext builder in your IDE, you have to override some central functionality that drives the builder for every language in your running IDE. Simply add an extension to your ui plugin that overrides the common configuration.

The second step is to override a binding in the ui module of your language to be communicate with the clustered builder. This implies that each Xtext based language that is installed in the IDE, has to expect the clustering builder instead of the default implementation. Otherwise you'll be confronted with an exception due to incompatible settings. The bindind in your ui module replaces the component that is used by the scope provider in the builder context:

Besides this global change to reduce the memory consumption, you may want to optimize some language specific implementations. It is always a good idea to review the number of exported objects. There is usually no need to export any object that has a name. E.g. a local variably is only reachable from inside the resource, it can simply be skipped when the exported objects are computed. Another good candidate is the fragment provider. The default fragments for resources are quite generic and somehow verbose. It will not hurt if you can come up with a shorter, resource specific representation. Both customization points will usually not only make your language consume less memory but even have potential to improve the performance as well because you can use language specific information in your implementation.

How to Deploy Xtext to a Headless Plain-Java Environment?

2010-06-22T23:16:00.005+02:00

Yesterday, Sven and I were at the Eclipse Democamp in Hannover which was organized by brox IT-Solutions and Bredex - thanks for the nice event. I met interesting people and we discussed various topics. One particular question about Xtext was raised several times from different sides: How do I use my language in an environment without Eclipse, e.g. on a build server? As this was already possible from the first minute even with the former oAW version of Xtext and with Xtext 0.7.0, too, I was kind of suprised. In fact, any execution of the generator workflow uses Xtext in a standalone environment - without OSGi, Equinox or any dependency on Eclipse UI. The problem seems to be another one: How can I determine all the required libraries to deploy my Xtext-driven generator?

It turns out that this task is suprisingly easy with Eclipse. We'll use a minor trick to simplify the necessary steps. First of all, we have to create a new Run Configuration for a Java Application. Actually this one will never be executed, but we need to refer to this one in the next step. Just create a new configuration for your generator-project and use the org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher as the main class:

Now that we have a run configuration for the generator project, let's export the project as a runnable jar. Choose Export... -> Runnable JAR file from the context menu of the generator project. The wizard will ask for a launch configuration and that's where we use the one that we created a second ago:

Select the options of your choice, point to a target file and hit the Finish-button. Eclipse will collect required libraries and add the necessary entries to the MANIFEST.MF in the runnable JAR file. As I already said, it's suprisingly simple.

Last but not least, I recommend to create a batch file or shell script that starts a virtual machine, points to the exported archive and refers to the module name of the generator workflow, e.g.:
java -jar exported.jar workflow.MyDslGenerator
It's probably a good idea to wrap the invocation of the Mwe2Launcher in a custom main class to encapsulate the name of the generator workflow and pass additional arguments to the workflow.

New In Xtext: Case Insensitive Languages

2010-06-20T14:20:00.001+02:00

A quick glimpse at the New and Noteworthy features of Xtext 1.0 reveals that the framework allows to develop languages that are case insensitive.

To mark upper and lower cases meaningless for keywords or cross references, you just have to replace two default fragments in the MWE2 workflow, add an option and hit the Run button:

The o.e.x.g.parser.antlr.ex.rt.AntlrGeneratorFragment and o.e.x.g.parser.antlr.ex.ca.ContentAssistParserGeneratorFragment replace the two fragments, that are configured by default: o.e.x.g.parser.antlr.XtextAntlrGeneratorFragment and o.e.x.g.parser.antlr.XtextAntlrUiGeneratorFragment.

Xtext provides an API that allows to define custom, local scopes that are ignore case as well. Have a look at the utility class o.e.x.resource.ignorecase.IgnoreCaseScopes to learn more about it. The configured generator fragments will take care of the rest, e.g. the default bindings for the global scopes, the parser and the serializer.

As you can see it is possible to define case insensitivity independently for keywords and cross-links. And if you want to keep your workflow short and concise instead of adding this flag for each and every fragment that accepts this option, you are free to introduce a new property and use this one directly or via the auto-inject option. Please refer to the updated documentation if you want to learn more about the MWE2 language.

Customizing Error Messages in Xtext 1.0

2010-06-17T20:30:00.003+02:00

As the Eclipse Helios Release is around the corner I'll use the chance to describe some nice odds and ends in Xtext 1.0. One of which is a newly introduced API that allows to customize the default messages for parse errors and linking diagnostics.

If you try to read erroneous models, you'll get annotations for syntax errors and linking problems in the editor by default. These error messages are not always the most meaningful ones from a user's point of view. Furthermore it was not possible to i18n them. This was changed with Xtext 1.0 by means of two newly introduced customization hooks.

Users who want to alter the default messages that the parser produces may implement a so called ISyntaxErrorMessageProvider which allows to change the wording of syntax errors and messages from the value converter.

The implementor has the current state of the parser at hand to produce a meaningful message for the user. Basically any information is available to compute the SyntaxErrorMessage. She is free to use the intermediate semantic model, the attached node model that reflects the parsed tokens, or the really low level stream that the parser works with.

Especially handy is the option to assign an issue code to a syntax error message. This is the constant that a quick fix refers to, to solve your models syntactic problems easily. Assuming you evolve the syntax of your language, you may want to define such quick fixes that migrate old files to the new notation. Of course there are more use cases such as mandatory keyword sequences at the end of an element.

There is another component that produces default error messages: If the linker cannot resolve an EMF proxy when navigating a cross reference, it will create an error diagnostic, too. We used the same pattern to allow clients to customize this message.

Again, every piece of information is available in the context, that is passed to the message provider and of course it is possible to assign an issue code to the linking diagnostic. Even more, clients may choose to reduce the severity of the linking problem to a warning. Languages that support a kind of dynamic invoke like C# may establish real links for the statically bound features but ignore unresolved references for dynamic parts.

Both APIs may come quite handy if you want to i18n your applicaton or tailor the error messages that are presented to the user.

EclipseCon 2010 - A Review

2010-04-01T16:02:00.000+02:00

This year was the first time that I attended the EclipseCon in Santa Clara and it was a really great event. However, the very first nice impression had actually nothing to do with the EclipseCon itself but with the nice weather in California. As the journey started in the rainy northern Germany, it was very pleasing to arrive in San Francisco with nearly 70F. To fight the jet lag back we rent some bikes and explored the vicinity on Sundays. It was a very enjoyable trip through the Bay Area. Nevertheless the night was rather short for me.

EclipseCon 2010

The first session on Monday morning was the Xtext meets e4 tutorial. It was especially interesting for me because Jan, Moritz and Heiko demonstrated a more user friendly approach to workbench modeling than editing tree editors and property sheets. I hope that at least some parts of the textual approach to store workbench information will make it into e4 as I personally do not like working with xmi-based persistence and form editors. However, up to now nobody wanted to jump in and shape up the prototypic Xtext editor. Hint, hint!

The next session was about The Twenty Modeling Things. Kenn and Stephan talked basically about some missing pieces in the Eclipse Modeling Project that will be covered by the Sphinx proposal and showed some of the IDE concepts that will be available. Afterwards I wanted to learn something about B3 from Henrik, but unfortunately the room was totally overcrowded. People seem to be aware of the fact, that automated builds that work the same way on the integration server as in the local Eclipse workspace are crucial to reliability and testability. Because building has been kind of a pain point for us in the past, I am especially happy about the new Buckminster build for Xtext on the Athena server, that Thomas set up last week.

On Tuesday, Sven and I presented some the new and especially the noteworthy things in Xtext. It is always fun to talk about the things that we have been busy with in the past months. The next event for modeling enthusiasts was the Modeling Runway. Although there were some technical ... inconveniences at the beginning, they turned into entertainment when Kenn advanced the slides or Sven's Xtext fast forward demo.

Wednesday was the day of THE keynote. Jeff Norris from the NASA played with all those things that boys dream of - and most likely still do as men. He remote controlled an 8 feet robot on the NASA area. It requires a lot of confidence in the software that you built when you perform those things live on stage. In the afternoon I gave my lightning talk about the new MWE tooling based on Xtext and a runtime layer that got an ultra slim diet.
The BoF about IDE tooling for the e4 development model was the last session on Wednesday. It was quite interesting and insightful to discuss the various ideas and approaches towards a steeper learning curve when developing plugins for e4. Tonny summarized the results in his blog and created or commented a few bugzillas that reflect the ideas.

Thursday went by as fast as it came. It was a day of some short discussions and I used the chance to do session hopping and hear something about basically everything in parallel. In the late afternoon, we returned the bikes and had a Mexican dinner before the traditional European Reception took place.

Conclusion

Although I expected the EclipseCon to take place in the Convention Center, I think it was a good decision to use the conference rooms in the Hyatt Hotel instead. Unfortunately they were a little too small for some of the talks and too low-ceiling so the canvas was probably not fully visible from the rows in the back. However, this minor drawback was fully compensated by the nice atmosphere during the conference. People ran literally into each other anytime and there were plenty of opportunities to chat about various topics. It was really easy to get in touch.

What I personally took along was that the development of custom language support is becoming kind of mainstream. It seems to me that people have learned that XML is not the best way to store human readable and editable information. The Xtext framework provides the necessary abstractions to easily develop the infrastructure to parse, process and edit files in a very convenient way. Projects like B3, EMF Query or AMP already adopted Xtext to provide tooling for programming or configuration languages and others decided after in-depth discussions that this might be a approach worthwhile to explore. Many people came to the itemis booth to talk about their proprietary solutions and we sketched many ideas on how to supersede existing implementations with EMF and Xtext. It was great fun to feel the vibes around Xtext.

A sneak peek at the new MWE editor

2010-01-12T11:28:00.006+01:00

The Modeling Workflow Engine (MWE) basically allows to assemble and orchestrate Java objects to use the resulting instances. A major use case is the composition of different Eclipse Modeling components to create an automated workflow for model based development approaches. Model readers, tranformation and validation engines or code generators can be plugged together to create a complete chain from the edited model to a runnable application or at least a bunch of refinable source code. It is even possible to use MWE in your own application to configure it externally.

One drawback of the current version of MWE is the lack of tool support when you have to edit the workflow configuration files. Although the XML dialect is quite optimized and even the simple XML editor helps a lot with support for paired tags, syntax validation and highlighting, it does not provide more sophicated features like MWE specific content assist, constraint checking or navigation.

Therefore, improved tooling is a major item on the roadmap to MWE 0.8.0 which will be part of the Eclipse Helios release. We currently discuss a refinement of the syntax for workflow files. It should be more concise and we want to provide a state of the art editor with advanced features. That's why we created a first prototype for a new MWE language at the end of the last year based on the Xtext framework.

Support for externalized properties

MWE files allow users to externalize configuration values into properties to make them reusable and improve the readability of a workflow. Properties may be concatenated into another property, declared in the workflow itself or in a properties file. All the rich editor features like content assist, validation and go-to declaration are available for properties and composed properties as well.

Content assist

As MWE uses reflection to assemble Java object graphs, the configuration files contain a lot of references to Java types and their methods. Based on the type compatibility constraints of the language, the editor will propose all the valid Java types. The necessary information will be derived from the classpath configuration of your Java projects and even the types in your source folders are available in your MWE file.

Class imports

As we leverage many powerful features of the Xtext framework, it was easy to introduce import statements for Java types to keep the syntax even more compact. Even the content assist is aware of existing imports and will automatically insert a fully qualified name if necessary and a simple name if possible.

Navigate to types and methods

Advanced navigation features like "Go To Declaration" make it easier to understand the configuration itself as well as the configured components. The new MWE editor makes it possible to jump directly to the appropriate members or types that your workflow refers to.

Conclusion

The new concrete syntax for the MWE language is easier to read and edit then the existing XML format. Nevertheless, the ideas behind MWE will remain stable. The language consists of a small number of orthogonal, seperated concepts which make it easy to understand and yet powerful. The new syntax focusses on those clean core concepts to make them easier to grasp while providing sophisticated editor features and tool support.

What's next

The prototype is already available in the Eclipse CVS although it still lacks some features and - more important - a connector to the runtime of MWE. We are currently planning to make it available in Summer 2010 in line with the Eclipse Helios release.

Xtext 0.7.2 has been released

2009-08-12T14:17:00.003+02:00

Almost four weeks after the first service release for TMF Xtext, we are proud to make Xtext 0.7.2 available for download. Nearly 40 issues have been fixed or improved in this release. Among them are some bugs with the serialization algorithm that occured in rare situations or fixes with the computation of available template proposals. Furthermore we improved the immediate user feedback in the Xtext grammar editor. It comes with more semantic highlighting and better validation for erroneous grammars. Internally we refactored some code to make it easier to customize because the default implementation may not suit your very own needs in each and every scenario. Some restrictions have been removed in this field to match even more use cases.

The probably most noteable changes affect a functionality, that has been available from the very beginning of Xtext in the 0.7 release but did not get to much attention although it may be quite interesting for you. Xtext ships with a generator fragment for new DSL projects, the SimpleNewProjectWizard. Although it is disabled by default - which means you have to uncomment it in your workflow file that generates your language - it may be quite handy for you. The fragment registers an Eclipse wizard for new projects. Such a project will know about all the required dependencies, contain a sample model file and a sample workflow file. It acts as a jump start for language users. The language designer can customize the initial content of the project and has a lot of flexibility. This feature may be really helpful for many Xtext users.

The easiest way to get Xtext is the update site at itemis which will install all required projects and dependencies in one shot. The updated full distribution with Xtext, MWE and Xpand 0.7.2 and support for Subversion, EMF Compare and all this fancy stuff will be available in the next hours.

Xtext meets Athena

2009-07-31T22:34:00.005+02:00

The weather reports look pretty good for TMF Xtext: The sun has been shining for a few builds and hopefully this trend will continue.

Due to the fact, that the emft buildserver will be down permanently from August, 5th, Dennis migrated the Xtext build to the modeling integration server, which is more powerful and faster then the previous integration system. This worked out pretty well, but we ended up with the very same problem as before: One of our test projects tends to break the build because of an untrapped exception in the compiler. As we have no idea about the reasons of this sudden breakdown - the build was successful for a long time and broke one day without any recognizable modification or log entry, we decided to look out for other possibilities to run our integration tests.

One promising option was the Eclipse build server. After we got the build slots for the Athena common build, which is powered by the Hudson continuous integration engine, we gave it a serious try. Dennis did some night shifts to create the configuration for the Xtext build, Xpand and MWE and finally all three projects were green. That was a very good feeling.

It was straight forward to define dependencies on required projects that will trigger dependent builds and to configure schedules for nightly builds. My overall impression is, that this infrastructure is easier to handle than the modeling build, it's definitly a step forward. What I like most - besides the easy configuration and the fact that it just builds our project - are the neat statistics and the weather report icons that indicate the trend of a project. Maybe we finally heralded the summer ?

Xtext 0.7.1 is in the starting blocks

2009-07-14T22:22:00.006+02:00

After an exciting Eclipse Galileo release, some celebrating and catching up a lot of sleep we decided to have a bugfix release short after Xtext 0.7.0 to incorporate some fixes that we had in the pipeline but that accidently didn't make it into the final Galileo build. The 0.7.1 release of TMF Xtext is scheduled for Friday, 17th of July, 2009 and it will contain more than 20 fixes and a few minor features that will improve the usability for both mouse and keyboard addicted users.

Thanks for all the valuable feedback that helped us to decide about the important issues that existed in the first release of TMF Xtext. Without the various complaints, questions and suggestions of our users in the newsgroup, the reports in the bugtracker and the responses via twitter, it would have been much more difficult to know where the shoe pinches.

Special thanks go to Alex from itemis Leipzip who visited us for one week in July to learn about the new version Xtext. Thereby he provided much feedback and told us about somewhat unexpected use cases. We could cover many topics about Xtext and beyond and it was great fun to learn about other people's experiences with the framework and to provide immediate help whenever possible.

Xtext Corner #4 – Customizing the generator

2009-03-18T20:24:00.012+01:00

TMF Xtext M6 has been released on Tuesday, 17 May 2009, and it comes with a bunch of handy new features. One of the most remarkable changes since milestone 5 is the redesigned generator infrastructure. A highly configurable, declarative framework has superseded the monolithic implementation, that we used previously.

The core idea behind the shiny new generator are so called generator fragments. They allow to contribute entries to various places during the generation process. It is possible to add extensions to the plugin.xml, declare exported packages for the MANIFEST.MF or list guice bindings. Furthermore and properly most important: generator fragments provide a mighty mechanism to create any kind of output files utilizing the Xpand template language.

Because the interface of fragments has been designed to be composable, it is easily possible to implement reusable components. Xtext itself ships with a neat set of default and often required fragments.

The generated grammar access provides strongly typed accessors for the various elements of your concrete syntax such as alternatives, assignments, cross references and keywords.
The Xtext parser that is responsable for reading text and instantiating a model. You'll get optionally an ANLTR-Parser, too.
A serializer, that is capable to transform the semantic model into its native textual form.
Xtext comes with a fragment, that creates a Check based validation stub. It allows to implement constraints for model instances declarativly. As there exists another fragment for java based validation, it is possible to use both and combine the expressivness of Check with the performance boost of native Java code for expensive validation.
A Java API for content assist is generated for the editor of your DSL. You can even choose to use an Xtend based API.
You'll get a custom EMF resource factory, that will create and configure your Xtext resources.
And the fragment that covers functionality that has not been in Xtext before, will generate Java classes for the infered metamodels. It uses EMF GenModels to output their corresponding Java API. Working with your semantic model is far more comfortable when there is no nead to deal with dynamic EMF models.

To create a library of generator fragments, we had to provide a simple way to parameterize the generator. Instead of instantiating objects the hard way and write a lot of Java code with constructor calls, getters and setters, we decided to use the Modeling Workflow Enging (MWE) to configure it. Clients plug the fragments they want to use, simply together by listing them in a workflow file. There is no need to write a single line of Java code to customize the behaviour or to execute the Xtext generator.

If someone has special needs for his language generation, he can implement a generator fragment that serves his very own interests well. Is your solution matching potential requirements of other parties, too? We will be happy to accept patches with reusable, customizable fragments to be packaged with Xtext.

Xtext Corner #3 – M5: What's in the pipeline?

2009-01-28T15:07:00.015+01:00

Some cool things will come up with the next milestone of TMF Xtext on February, 6th 2009.

The outline view has been redesigned. From now on it is possible to create a representation of your source file's structure, that does not necessarily map the semantic model one to one. You are free to include virtual nodes to emphasize aspects of special interest or to group the objects in another way as it is done in the textual source file.
Xtext resources observe their referenced models and can reload them to reflect recent changes transparently. This improves the overall user experience and provides faster feedback.
We will come up with a first draft of an Xtend API for the new Xtext. It is not as powerful as the corresponding Java API, but still very useful especially for early prototyping. Most notably Xtend is very convenient when you have to work with dynamic EMF models. Furthermore it comes with a nice collection API.

There is one more thing ...

Due to an IP issue with Antlr we are not allowed to use this mature parser generator if we want to be part of the next Eclipse release with TMF Xtext. Unfortunatly we are not satisfied with the eclipse compatible alternatives. That's why we decided to build our own parser generator based on the packrat algorithm. Besides the effort with implementing yet another parser generator, there are some positive side effects:

We learned a lot about minimizing dependencies and therefore made the parser more pluggable. If you don't like our home grown packrat parser and even don't want to use Antlr, you can theoretically take any generator of your choice and use it.
Attention: This feature comes without warranty.
We found a nice way to define terminal symbols. With Xtext M4 you could already write your own lexical rules in a somewhat awkward syntax. The whole body of the rule was a pure string without any syntactical check at design time. Xtext M5 comes with terminal rules. At a first glance, they seem to be like any other parser rule. The clue is, that they allow to define kind of a lexer body with a rich syntax and known semantics. But instead of plain parser rules, they will produce exacly one (leaf-)node in the parsed tree and may not be interrupted by any whitespace or comment.
 // java like block comments terminal ML_Comment: '/*' -> '*/'; // java like line comments terminal SL_Comment: '//' !('\n'|'\r')* ('\r'? '\n')?; 
Terminal rules will supersede the old school lexer rules.

Hidden tokens per rule were introduced: You can define terminal tokens as hidden on a per rule basis. If you do not want to have whitespaces between your fully qualified names, you can disallow them easily.
 Entity hidden( WS, ML_COMMENT, SL_COMMENT): 'entity' name=FQN '{' property+=Property* '}' // Property inherits hidden terminals // from calling rule Entity Property: /* something */; // FQN overrides hidden terminals FQN returns EString hidden(): ID ('::' ID)*;

Xtext Corner #2 - Linking and scoping

2009-01-19T08:22:00.009+01:00

I've been very busy these days with all this christmas stuff, the M4 milestone of TMF Xtext and obviously with the next milestone, which will be released mid Feb. That's why I had the rather long delay since my first post about linking in Xtext. In december I discussed some difficulties with multi value references in combination with the default behavior of the Linker. As I promised in my previous post the workaround is not necessary in Xtext M4: You don't have to avoid multi value (non containment) references in metamodels anymore for the ease of validation.

Sven wrote already a very nice blog post about the concept of scopes that we introduced in the last milestone. The crucial point is to draw a distinction between visible objects and name matching when linking the model. Therefore we split the default linking service implementation into two main stages.

The first parts is about identifying all visible objects and creating the so called scope. A scope is an abstraction from linkable EObjects and the algorithm to calculate their valid string representation. Any scope provides all reachable objects paired with information about the valid textual representation – the name – that will match this object starting from a given point in your model, the so called context. Important are two facts:

Scopes can be nested into each other and thus can hide their inherited elements.
A scope does not export two objects with the same name in the same scope-level.

If you think about java, a local variable foo might be declared in a method body, which will make the same named field unreachable by its simple name but require the „this“ qualifier to gain access to it. A well known tool that has to deal with scopes is the eclipse compiler. It will not compile, read „link“, any two fields with the same name in a given class. None of these two fields would be on the scope.

Because we can have fields and methods with the same name, that would both be visible in contrast to equally named variables, the scope provider has to take the type of the objects into account, that should be retrieved. Sven described the semantics of scopes more detailled and provided a neat example.

The second part of the linking-service' job is to compare the input string with the object names of the scoped elements that were retrieved from the scope provider. That's very easy in most cases because a simple string match is sufficient.

But it is not necessarily an object, that was retrieved from the scope service, that should be linked. In the Xtext grammar itself, we link imported metamodels to EPackages by a given URI. It is rather impossible to come up with a scope implementation, that iterates each and every metamodel, that can be reached via any given URI. But the linking-service is still able to establish this cross reference. This indicates, that we found some nice abstractions for the common cases, but did not introduce too tight restrictions for the seldom ones.

Let's come back to the example from my previous post. It is now possible to write the grammar for the given example language in the most intuitive way using the M4 milestone of TMF Xtext. By default, the linker will only cross-reference objects, that have a unique name. Even for multi value references, we will get very good error indicators for free. The last thing to do is to provide a check, that every object is linked only once for a reference, that means the list contains distinct values. Should really be a no brainer with Xtend ...

Xtext Corner #1 - Linking the model

2008-11-30T14:11:00.002+01:00

One of the core features of Xtext is the ability, to create a semantic model from a set of input files. While loading the resource, Xtext transforms the parsed AST into a domain model and trys to cross link the model elements.

From an information theory point of view this cannot be solved generally for every imaginable language, but we try to provide reasonable defaults with Xtext. A quick start with a language, that fulfils common usecases, should be as easy as possible. So we based the default linking on some kind of heuristics: Most likely, one would reference a model element by some kind of name, so we look for name features in the model elements and try to match them with the input text for a reference.

Lets consider a small example language for clarification:

The following input file would certainly match the given concrete syntax:

If we take a look at the lines two and three, we notice two objects in the container, that have the same name, but different indexes. The default linking implementation that ships with the M3 Milestone of TMF Xtext would try to link both 'Obj2' for the reference in line one. This seems to be perfectly ok, because our objects can refer to many other objects (note the '+=' notation in the concrete syntax of our language).

But in many cases, the user would expect, that the reference points to a unique object, even if the metamodel allows multiple values for a feature.
So how can we identifiy these ambiguities without implementing the whole linking service ourself? The simple answer is: In TMF Xtext M3, we cannot. But there is a workaround.

What we have to do is to slightly modify our grammar and therefore derive a different metamodel:

The refined 'SampleLang' uses a combination of containment and single value references instead of multi value references. At a first glance, we have greater efforts when writing functions that use our models. So what is the benefit? The default linking implementation works out of the box and implementing constraints on top of that to check our new kind of multi value references is pretty straight forward.

But whats the good news? This workaround does its job. And even better: It will not be required in TMF Xtext M4 which will be released at the end of the year. We thought again about linking and will come up with another default implementation, that is suitable for many more cases, which cannot be handled by the default language services in M3.

Optimistic Locking Revised

2008-11-12T22:39:00.022+01:00

When you implement concurrency control in a business application, you have to decide if you use pessimistic or optimistic locking. For some types of applications optimistic locking is sufficient, because in the most common cases users would modify different resources, thus only a few of them would rant about lost changes. If you use optimistic locking, subsequent conflicting changes will be denied and the user has to update his view on the data and redo his modifications prior to saving the resources.

Basically there exist two ways to implement optimistic locking. Both of them have advantages and disadvantages.

One solution is to check the original state of every modified property against the current value in the database and only update the affected columns. This implies, that almost every update statement is a custom statement. You cannot use prepared statements to perform batch updates. This can be a major performance drawback.

The statements would look like this:

 UPDATE table_name SET data1 = :data1, data2 = :data2 WHERE pk = :pk AND data1 = :old_data1 AND data2 = :old_data2
The second possibility is to introduce some kind of update counter in your table. Before you update a record in the database, the current value of the update counter is checked against the base value, that was read by the client. If both values are equal, the update is performed and the counter increased. The update counter is either an integer or a timestamp.

This is a typical example for an update statement using an update counter:

 UPDATE table_name SET data1 = :data1, data2 = :data2, data3 = :data3, blob = :blob, update_count = update_count + 1 WHERE pk = :pk AND update_count = :update_count 

The obvious advantage of this technique is the possibility to use prepared statements because the structure and the parameter list is the same for any given modified record in a table. But the chance is, that modifications are refused, even if they are not conflicting. The new data could be the same as in the persistet record, or only distinct properties could have been modified compared to the common base version. In both cases this solution would be quite frustrating or at least confusing from a users point of view.

If you implement optimistic locking with on of these patterns, you have to check against the number of modified records in your code, when you execute such a query. In pseudo code with a java like syntax this would look like the following snippet:

Neither the first nor the second obvious implementation is satisfying because of the mentioned drawbacks. Let's try to combine the advantages of both worlds at the expense of a slightly more complex pattern for update statements:

This implementation reduces the risk for wrongly refused updates to a minimum. It even allows special treatment of blob fields, that cannot be compared directly. Additionally you have the chance, to define special semantics for field groups, e.g. if any value in a group has been modified, the whole update can be denied, even if the concrete field that changed was not previously edited by another user.

I don't know about any implementation of this idea, but I am very interested in experience from the real world. Especially reports about performance differences and the influence on the overall usability are welcome.

Here we go ...

2008-11-07T12:07:00.008+01:00

This is the mandatory first blog post in and about this blog. On an irregular basis I will write about random topics like software development and abritrary banalities. Stay tuned.