Xtext 0.7.2 has been released

Almost four weeks after the first service release for TMF Xtext, we are proud to make Xtext 0.7.2 available for download. Nearly 40 issues have been fixed or improved in this release. Among them are some bugs with the serialization algorithm that occured in rare situations or fixes with the computation of available template proposals. Furthermore we improved the immediate user feedback in the Xtext grammar editor. It comes with more semantic highlighting and better validation for erroneous grammars. Internally we refactored some code to make it easier to customize because the default implementation may not suit your very own needs in each and every scenario. Some restrictions have been removed in this field to match even more use cases.

The probably most noteable changes affect a functionality, that has been available from the very beginning of Xtext in the 0.7 release but did not get to much attention although it may be quite interesting for you. Xtext ships with a generator fragment for new DSL projects, the SimpleNewProjectWizard. Although it is disabled by default - which means you have to uncomment it in your workflow file that generates your language - it may be quite handy for you. The fragment registers an Eclipse wizard for new projects. Such a project will know about all the required dependencies, contain a sample model file and a sample workflow file. It acts as a jump start for language users. The language designer can customize the initial content of the project and has a lot of flexibility. This feature may be really helpful for many Xtext users.

The easiest way to get Xtext is the update site at itemis which will install all required projects and dependencies in one shot. The updated full distribution with Xtext, MWE and Xpand 0.7.2 and support for Subversion, EMF Compare and all this fancy stuff will be available in the next hours.

Xtext meets Athena

The weather reports look pretty good for TMF Xtext: The sun has been shining for a few builds and hopefully this trend will continue.

Due to the fact, that the emft buildserver will be down permanently from August, 5th, Dennis migrated the Xtext build to the modeling integration server, which is more powerful and faster then the previous integration system. This worked out pretty well, but we ended up with the very same problem as before: One of our test projects tends to break the build because of an untrapped exception in the compiler. As we have no idea about the reasons of this sudden breakdown - the build was successful for a long time and broke one day without any recognizable modification or log entry, we decided to look out for other possibilities to run our integration tests.

One promising option was the Eclipse build server. After we got the build slots for the Athena common build, which is powered by the Hudson continuous integration engine, we gave it a serious try. Dennis did some night shifts to create the configuration for the Xtext build, Xpand and MWE and finally all three projects were green. That was a very good feeling.

It was straight forward to define dependencies on required projects that will trigger dependent builds and to configure schedules for nightly builds. My overall impression is, that this infrastructure is easier to handle than the modeling build, it's definitly a step forward. What I like most - besides the easy configuration and the fact that it just builds our project - are the neat statistics and the weather report icons that indicate the trend of a project. Maybe we finally heralded the summer?

Xtext 0.7.1 is in the starting blocks

After an exciting Eclipse Galileo release, some celebrating and catching up a lot of sleep we decided to have a bugfix release short after Xtext 0.7.0 to incorporate some fixes that we had in the pipeline but that accidently didn't make it into the final Galileo build. The 0.7.1 release of TMF Xtext is scheduled for Friday, 17th of July, 2009 and it will contain more than 20 fixes and a few minor features that will improve the usability for both mouse and keyboard addicted users.

Thanks for all the valuable feedback that helped us to decide about the important issues that existed in the first release of TMF Xtext. Without the various complaints, questions and suggestions of our users in the newsgroup, the reports in the bugtracker and the responses via twitter, it would have been much more difficult to know where the shoe pinches.

Special thanks go to Alex from itemis Leipzip who visited us for one week in July to learn about the new version Xtext. Thereby he provided much feedback and told us about somewhat unexpected use cases. We could cover many topics about Xtext and beyond and it was great fun to learn about other people's experiences with the framework and to provide immediate help whenever possible.

Xtext Corner #4 – Customizing the generator

TMF Xtext M6 has been released on Tuesday, 17 May 2009, and it comes with a bunch of handy new features. One of the most remarkable changes since milestone 5 is the redesigned generator infrastructure. A highly configurable, declarative framework has superseded the monolithic implementation, that we used previously.

The core idea behind the shiny new generator are so called generator fragments. They allow to contribute entries to various places during the generation process. It is possible to add extensions to the plugin.xml, declare exported packages for the MANIFEST.MF or list guice bindings. Furthermore and properly most important: generator fragments provide a mighty mechanism to create any kind of output files utilizing the Xpand template language.

Because the interface of fragments has been designed to be composable, it is easily possible to implement reusable components. Xtext itself ships with a neat set of default and often required fragments.

• The generated grammar access provides strongly typed accessors for the various elements of your concrete syntax such as alternatives, assignments, cross references and keywords.
• The Xtext parser that is responsable for reading text and instantiating a model. You'll get optionally an ANLTR-Parser, too.
• A serializer, that is capable to transform the semantic model into its native textual form.
• Xtext comes with a fragment, that creates a Check based validation stub. It allows to implement constraints for model instances declarativly. As there exists another fragment for java based validation, it is possible to use both and combine the expressivness of Check with the performance boost of native Java code for expensive validation.
  
• A Java API for content assist is generated for the editor of your DSL. You can even choose to use an Xtend based API.
  
• You'll get a custom EMF resource factory, that will create and configure your Xtext resources.
• And the fragment that covers functionality that has not been in Xtext before, will generate Java classes for the infered metamodels. It uses EMF GenModels to output their corresponding Java API. Working with your semantic model is far more comfortable when there is no nead to deal with dynamic EMF models.

To create a library of generator fragments, we had to provide a simple way to parameterize the generator. Instead of instantiating objects the hard way and write a lot of Java code with constructor calls, getters and setters, we decided to use the Modeling Workflow Enging (MWE) to configure it. Clients plug the fragments they want to use, simply together by listing them in a workflow file. There is no need to write a single line of Java code to customize the behaviour or to execute the Xtext generator.

If someone has special needs for his language generation, he can implement a generator fragment that serves his very own interests well. Is your solution matching potential requirements of other parties, too? We will be happy to accept patches with reusable, customizable fragments to be packaged with Xtext.

Xtext Corner #3 – M5: What's in the pipeline?

Some cool things will come up with the next milestone of TMF Xtext on February, 6th 2009.

• The outline view has been redesigned. From now on it is possible to create a representation of your source file's structure, that does not necessarily map the semantic model one to one. You are free to include virtual nodes to emphasize aspects of special interest or to group the objects in another way as it is done in the textual source file.
• Xtext resources observe their referenced models and can reload them to reflect recent changes transparently. This improves the overall user experience and provides faster feedback.
• We will come up with a first draft of an Xtend API for the new Xtext. It is not as powerful as the corresponding Java API, but still very useful especially for early prototyping. Most notably Xtend is very convenient when you have to work with dynamic EMF models. Furthermore it comes with a nice collection API.

There is one more thing ...

Due to an IP issue with Antlr we are not allowed to use this mature parser generator if we want to be part of the next Eclipse release with TMF Xtext. Unfortunatly we are not satisfied with the eclipse compatible alternatives. That's why we decided to build our own parser generator based on the packrat algorithm. Besides the effort with implementing yet another parser generator, there are some positive side effects:

• We learned a lot about minimizing dependencies and therefore made the parser more pluggable. If you don't like our home grown packrat parser and even don't want to use Antlr, you can theoretically take any generator of your choice and use it.
  Attention: This feature comes without warranty.
• We found a nice way to define terminal symbols. With Xtext M4 you could already write your own lexical rules in a somewhat awkward syntax. The whole body of the rule was a pure string without any syntactical check at design time. Xtext M5 comes with terminal rules. At a first glance, they seem to be like any other parser rule. The clue is, that they allow to define kind of a lexer body with a rich syntax and known semantics. But instead of plain parser rules, they will produce exacly one (leaf-)node in the parsed tree and may not be interrupted by any whitespace or comment.
  <br />// java like block comments<br />terminal ML_Comment: <br /> '/*' -> '*/';<br /><br />// java like line comments<br />terminal SL_Comment: <br /> '//' !('\n'|'\r')* ('\r'? '\n')?;<br />
  Terminal rules will supersede the old school lexer rules.
  


• Hidden tokens per rule were introduced: You can define terminal tokens as hidden on a per rule basis. If you do not want to have whitespaces between your fully qualified names, you can disallow them easily.
  <br />Entity hidden(<br /> WS, ML_COMMENT, SL_COMMENT):<br /> 'entity' name=FQN '{' <br /> property+=Property* <br /> '}'<br /><br />// Property inherits hidden terminals<br />// from calling rule Entity<br />Property: /* something */;<br /><br />// FQN overrides hidden terminals<br />FQN returns EString hidden():<br /> ID ('::' ID)*;<br />

Xtext Corner #2 - Linking and scoping

I've been very busy these days with all this christmas stuff, the M4 milestone of TMF Xtext and obviously with the next milestone, which will be released mid Feb. That's why I had the rather long delay since my first post about linking in Xtext. In december I discussed some difficulties with multi value references in combination with the default behavior of the Linker. As I promised in my previous post the workaround is not necessary in Xtext M4: You don't have to avoid multi value (non containment) references in metamodels anymore for the ease of validation.

Sven wrote already a very nice blog post about the concept of scopes that we introduced in the last milestone. The crucial point is to draw a distinction between visible objects and name matching when linking the model. Therefore we split the default linking service implementation into two main stages.

The first parts is about identifying all visible objects and creating the so called scope. A scope is an abstraction from linkable EObjects and the algorithm to calculate their valid string representation. Any scope provides all reachable objects paired with information about the valid textual representation – the name – that will match this object starting from a given point in your model, the so called context. Important are two facts:

1. Scopes can be nested into each other and thus can hide their inherited elements.
   
2. A scope does not export two objects with the same name in the same scope-level.

<br />public interface IScopeProvider <br /> extends ILanguageService {<br /><br /> IScope getScope(EObject context, <br /> EReference reference);<br /><br />}<br /><br />public interface IScope { <br /><br /> IScope getOuterScope();<br /><br /> Iterable<IScopedElement> getContents();<br /><br /> /* .. */<br />} <br /><br />public interface IScopedElement {<br /> <br /> String name();<br /> <br /> EObject element();<br /> <br /> /* .. */<br />}<br />

If you think about java, a local variable foo might be declared in a method body, which will make the same named field unreachable by its simple name but require the „this“ qualifier to gain access to it. A well known tool that has to deal with scopes is the eclipse compiler. It will not compile, read „link“, any two fields with the same name in a given class. None of these two fields would be on the scope.

<br />class Bar {<br /> int foo;<br /> void setFoo(int foo) {<br /> this.foo = foo;<br /> }<br />}<br /><br />class Outer {<br /> int foo;<br /> <br /> void foo(int param) { /*.. */ }<br /> class Inner {<br /> int foo; // error<br /> int foo; // error<br /> void bar() { <br /> foo(foo); // ok <br /> // same as <br /> // foo(Outer.this.foo);<br /> }<br /> }<br />}<br />

Because we can have fields and methods with the same name, that would both be visible in contrast to equally named variables, the scope provider has to take the type of the objects into account, that should be retrieved. Sven described the semantics of scopes more detailled and provided a neat example.

The second part of the linking-service' job is to compare the input string with the object names of the scoped elements that were retrieved from the scope provider. That's very easy in most cases because a simple string match is sufficient.

But it is not necessarily an object, that was retrieved from the scope service, that should be linked. In the Xtext grammar itself, we link imported metamodels to EPackages by a given URI. It is rather impossible to come up with a scope implementation, that iterates each and every metamodel, that can be reached via any given URI. But the linking-service is still able to establish this cross reference. This indicates, that we found some nice abstractions for the common cases, but did not introduce too tight restrictions for the seldom ones.

Let's come back to the example from my previous post. It is now possible to write the grammar for the given example language in the most intuitive way using the M4 milestone of TMF Xtext. By default, the linker will only cross-reference objects, that have a unique name. Even for multi value references, we will get very good error indicators for free. The last thing to do is to provide a check, that every object is linked only once for a reference, that means the list contains distinct values. Should really be a no brainer with Xtend ...