2010 07 16 making it hard to implement language compatible linq providers

Published on July 16th, 2010 at 12:51

Making it hard to implement language-compatible LINQ providers

There are two things compiler writers can do in order to give us LINQ provider writers significant headaches:

• Firstly, they can make certain kinds of expressions much more common than other languages do. For example, they can use overloads of query operators or produce expression combinations that are otherwise rarely used.
• Secondly, they can embed completely custom operators in their expression trees, masked as MethodCallExpressions to some compiler service methods.

The first issue is annoying because missing features that aren't a problem for other languages suddenly get really important. Implementing a LINQ provider is a LOT of work, so there’ll always have to be some feature prioritizing. One of my favorite quotes on this is by Frans Bouma, author of LLBLGen Pro’s LINQ provider:

Writing a Linq provider is a lot of work which requires a lot of code. If you're dealing with a Linq provider which is just, say, 32KB in size, you can be sure it will not support the majority of situations you will run into. However, the O/R mapper developer likely simply said 'We have Linq support', and it's even likely the provider can handle the more basic examples of a single entity type fetch with a Where, an Order By or even a Group By. But in real life, once you as a developer have tasted the joy of writing compact, powerful queries using Linq, you will write queries with much more complexity than these Linq 101 examples. Will the Linq 'provider' you chose be able to handle these as well? In other words: is it a full Linq provider or, as some would say, a 'toy' ?

Usually, there is a time limit when implementing a LINQ provider, so you have to decide what features to implement, what query methods to support – and what query methods not to support. You’d usually implement the most important ones first, and if you decide one query method is not that important because you don’t know another language requires it, that might be a problem.

However, it could be argued that these features also provide benefits for users of the other languages. For example, while the C# compiler won't specify a result selector in the GroupBy calls it generates, C# users can of course manually call the respective method overloads. So, implementing support for that overload of GroupBy is not a language-specific feature, it's just the priority that differs between the languages.

The second issue, on the other hand, the one of embedding custom operators in the expression tree, is the real problem of LINQ provider compatibility. You need to support these operators only for the sake of one programming language. You won't even know these operators exist unless you perform extensive testing in that language. The programming language might just as well construct expression trees with self-defined nodes instead of using the standard ones – the effect would be similar: there’s no way the LINQ provider could handle them unless it was built with explicit support for that language.

That’s a problem. Especially since VisualBasic.NET, of course, does both of these things.

VB.NET does make use of expressions uncommon in, say, a C# program; for example, it uses the GroupBy overloads taking a result selector, it generates additional, unnecessary Select (x => x) calls, and so on. These things you thought no programmer would commonly use in a LINQ query – the VB.NET compiler does use them, all too often.

And VB.NET does define its own operators. Expression trees coming from VB.NET may contain calls to the Microsoft.VisualBasic.CompilerServices.LikeOperator’s methods because VB does have a LIKE operator – which can’t be represented using standard expression types. And even string comparisons, which are usually represented as BinaryExpressions, cause special operators to be used.

Of course, there are good arguments as of why VB needs those operators: There is no standard expression type for LIKE, and VB has some specific string comparison semantics. Still, I wonder if the LINQ teams at Microsoft shouldn’t have thought about these problems a little harder, a little longer before releasing VB.NET’s LINQ support as is.

At the very least, some documentation of these VB-specifics would have saved a few headaches. Definitely.

- Fabian