Running this in LinqPad against Northwind

from o in Order
group o by o.OrderDate.Value.Year into orderYear
select orderYear

throws InvalidCastException

Unable to cast object of type 'SD.LLBLGen.Pro.LinqSupportClasses.ExpressionClasses.DbFunctionCallExpression' to type 'SD.LLBLGen.Pro.LinqSupportClasses.ExpressionClasses.EntityFieldExpression'.

while this runs fine

from o in Order
group o by o.OrderDate.Value.Year into orderYear
select orderYear.Key

as does this

from o in Order
group o by o.OrderDate.Value into orderYear
select orderYear

version 4.1.13.1114

Stack trace

at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.QueryExpressionBuilder.CoerceGroupByToFullQueryForProjection(GroupByExpression referencedGroupBy)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.QueryExpressionBuilder.HandleSetReferenceExpression(SetReferenceExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.QueryExpressionBuilder.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.QueryExpressionBuilder.HandleProjectionExpression(ProjectionExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.QueryExpressionBuilder.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleSelectExpression(SelectExpression expressionToHandle, SelectExpression newInstance)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleSelectExpression(SelectExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.QueryExpressionBuilder.HandleSelectExpression(SelectExpression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.GenericExpressionHandler.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.ExpressionHandlers.QueryExpressionBuilder.HandleExpression(Expression expressionToHandle)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.HandleExpressionTree(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.System.Linq.IQueryProvider.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProQuery`1.System.Collections.IEnumerable.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

Looks like it misses a check for a situation which can occur. We'll look into it.

from o in Order
group o by o.OrderDate.Value.Year into orderYear
select orderYear

This query will never work, I'm afraid. The reason is that the query is actually a nested query: one for the grouped value + a placeholder value, and one for the grouped set which is merged as a nested query in memory. And herein lies the problem: the .Year value used in the group by isn't a value in the order entity. This means that the merge logic can't find a value in order which will match with a value in the grouped value set, the first (outer) query.

So grouping on values other than real values available in the set to group will never work. This is also the reason I think the code to handle a function call at that point isn't there as it is of no use. I admit the error is cryptic and should be better, however if I add code to handle a function call there, it will crash later on that it can't find a correlation predicate.

This query is a special case query, where a linq query is fetched as a nested query, even though there's no nested query specified. The query which does a select orderYear.Key isn't a nested query and works properly.

You obviously ran into this with a much bigger query. If you need help rewriting it to a form which works, please let me know. It is possible to do this though:

first create a custom projection on order with the Year field as extra field then group that set on the year field. as it is now present in the grouped set, merge can be done in memory with the nested set. Of course the nested set isn't a set of entities in this case.

I came across that as I was decomposing another problem, I don't think I would ever have need to bring it back locally just as a IQueryable<IGrouping<int, OrderEntity>>. Now to the real problem, if I may. In Linqpad again

  public class GroupByYear
  {
    public int Year { get; set; }
    public int Month { get; set; }
        public string Group { get; set; }
  }

void Main()
{
this.CustomFunctionMappings=Northwind.DAL.SqlServer.DataAccessAdapter.StaticCustomFunctionMappings;
    var ordersGroupedByYearAnon = Order.GroupBy(o => new  {Group=o.ShipCity});
    ordersGroupedByYearAnon.Select(o => new {Group = string.IsNullOrEmpty(o.Key.Group) ? "empty" : o.Key.Group,count = o.Count()}).ToList().Dump();

    var ordersGroupedByYearNamed = Order.GroupBy(o => new GroupByYear {Group=o.ShipCity});
    ordersGroupedByYearNamed.Select(o => new {Group = string.IsNullOrEmpty(o.Key.Group) ? "empty" : o.Key.Group,count = o.Count()}).ToList().Dump();
}

The first block runs OK but the second fails with: ORMQueryConstructionException-The parameter at position 0 is of an unsupported type: MemberAccess the only difference being, the first block has an anonymous key class, while the second uses class GroupByYear.

I can rewrite to avoid calling string.IsNullOrEmpty on the DB but...What gives?

StaticCustomFunctionMappings.Add(new FunctionMapping(typeof(string), "IsNullOrEmpty", 1, "{0} IS NULL OR LEN({0}) = 0"));

The problem with the named class is that o.Key.Group is unknown what it is mapped on, i.e. what value is stored there. It might look obvious in the code, but the query doesn't have that at its disposal: it doesn't know what value .Group as a property returns, or better: what value from the DB is stored there, so it can replace the .Group property reference in the query with the actual db source field that's stored there. As it can't do that, the member access is never resolved

Otis wrote:

The problem with the named class is that o.Key.Group is unknown what it is mapped on, i.e. what value is stored there. It might look obvious in the code, but the query doesn't have that at its disposal: it doesn't know what value .Group as a property returns, or better: what value from the DB is stored there, so it can replace the .Group property reference in the query with the actual db source field that's stored there. As it can't do that, the member access is never resolved

But how is it any different with an anonymous class?

FWIW My end game is doing dynamic GroupBy.

TomDog wrote:

Otis wrote:

The problem with the named class is that o.Key.Group is unknown what it is mapped on, i.e. what value is stored there. It might look obvious in the code, but the query doesn't have that at its disposal: it doesn't know what value .Group as a property returns, or better: what value from the DB is stored there, so it can replace the .Group property reference in the query with the actual db source field that's stored there. As it can't do that, the member access is never resolved

But how is it any different with an anonymous class?

That is a good point indeed. I do recall there's a difference between anonymous types and full types in the linq provider but I have to look it up what the cause of this might be. I'll check it out.

FWIW My end game is doing dynamic GroupBy.

Anything dynamic with linq is a struggle. Queryspec makes this very very easy, why doing it the hard way?

The exception has nothing to do with the function mapping, that works fine. The thing is that the parameter of the function call isn't handled properly. This is because the parameter is a member init, which is unexpected.

When adding MemberInit to the clause which was already there for New (which is doing the same thing: pulling the fields from the expression to use as elements in the query), it works.

Running the other tests now to see whether it broke anything.

Works. Fixed in next build. Still, it's recommended to use queryspec for dynamically created queries. Next build is scheduled to be uploaded later today.

Otis wrote:

Works. Fixed in next build. Still, it's recommended to use queryspec for dynamically created queries. Next build is scheduled to be uploaded later today.

Thanks, that sorted it. While I don't doubt the advantages of QuerySpec, we have used Linq everywhere and it would be quite some task to convert all our Linq to it.

You don't have to convert everything, really you can use them side by side, e.g. only use queryspec when you have to do things dynamically.

Otis wrote:

You don't have to convert everything, really you can use them side by side, e.g. only use queryspec when you have to do things dynamically.

They may work side by side but the problem is they don't work together, we have a query screens which drive linq filters which we then project one way or the other. To use queryspec to do the projections, such as the dynamic grouping, we would have to have a parallel queryspec filter pipe.

BTW ORMQueryConstructionException: The parameter at position 0 is of an unsupported type: MemberAccess comes back if I switch to a constructor e.g

public GroupByYear(int year, int month, string @group)
    {
      Year = year;
      Month = month;
      Group = @group;
    }
Order.GroupBy(o => new GroupByYear(o.OrderDate.Value.Year,o.OrderDate.Value.Month,o.ShipCity) );    

TomDog wrote:

Otis wrote:

You don't have to convert everything, really you can use them side by side, e.g. only use queryspec when you have to do things dynamically.

They may work side by side but the problem is they don't work together, we have a query screens which drive linq filters which we then project one way or the other. To use queryspec to do the projections, such as the dynamic grouping, we would have to have a parallel queryspec filter pipe.

true, that is indeed a limitation.

BTW ORMQueryConstructionException: The parameter at position 0 is of an unsupported type: MemberAccess comes back if I switch to a constructor e.g

public GroupByYear(int year, int month, string @group)
    {
      Year = year;
      Month = month;
      Group = @group;
    }
Order.GroupBy(o => new GroupByYear(o.OrderDate.Value.Year,o.OrderDate.Value.Month,o.ShipCity) );    

That is indeed something which isn't solveable. The thing is that a ctor parameter has no correlation with a property, a property might have different code to obtain its value, so if a value is passed into a ctor, we can't do a thing about it: we can't map it to a property, as it doesn't know it should. If a value is stored in a property first (like your previous query) we know the db value relates to the property, so if the property is referenced later on, we know we can replace it with the db value which was stored inside it. With a ctor, we can't.

So you have to use the property initializers here, there's no other way.

I get another error now: InvalidOperationException - Code supposed to be unreachable

public class GroupByYear
  {
    public int Year { get; set; }
    public int Month { get; set; }
    public Object Group { get; set; }
    public string GroupString { get; set; }
        public int Count { get; set; }
        
        public GroupByYear(int year, int month)
    {
      Year = year;
      Month = month;
    }
        
    public GroupByYear(int year, int month, object group)
      : this(year, month)
    {
      Group = group;
    }

    public GroupByYear(int year, int month, string groupString)
      : this(year, month)
    {
      GroupString = groupString;
    }

    public GroupByYear()
    {
    }
        
  }

void Main()
{
    var ordersGroupedByYearNamed = Order.GroupBy(o => new GroupByYear(o.OrderDate.Value.Year,o.OrderDate.Value.Month){Group=o.ShipCity} );
  ordersGroupedByYearNamed.Select(o => new GroupByYear(o.Key.Year, o.Key.Month) {Group = o.Key.Group}).ToList().Dump();
}

at System.Linq.Expressions.Compiler.StackSpiller.RewriteExpression(Expression node, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.RewriteMemberExpression(Expression expr, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.RewriteExpression(Expression node, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.MemberAssignmentRewriter..ctor(MemberAssignment binding, StackSpiller spiller, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.BindingRewriter.Create(MemberBinding binding, StackSpiller spiller, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.RewriteMemberInitExpression(Expression expr, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.RewriteExpression(Expression node, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.RewriteUnaryExpression(Expression expr, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.RewriteExpression(Expression node, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.RewriteExpressionFreeTemps(Expression expression, Stack stack)
   at System.Linq.Expressions.Compiler.StackSpiller.Rewrite[T](Expression`1 lambda)
   at System.Linq.Expressions.Expression`1.Accept(StackSpiller spiller)
   at System.Linq.Expressions.Compiler.LambdaCompiler.Compile(LambdaExpression lambda, DebugInfoGenerator debugInfoGenerator)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProvider2.SetupProjectionElementsForExecution(QueryExpression toExecute)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProvider2.ExecuteValueListProjection(QueryExpression toExecute)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.ExecuteExpression(Expression handledExpression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProProviderBase.System.Linq.IQueryProvider.Execute(Expression expression)
   at SD.LLBLGen.Pro.LinqSupportClasses.LLBLGenProQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)

but if I add a cast to the last line

  ordersGroupedByYearNamed.Select(o => new GroupByYear(o.Key.Year, o.Key.Month) {Group = (string)o.Key.Group}).ToList().Dump();

it works fine

It seems LINQ to NHibernate has a similar problem: http://connect.microsoft.com/VisualStudio/feedback/details/557075/code-supposed-to-be-unreachable-exception-from-system-linq/

There's not a lot I can do, other than posting on the connect issue and hope ms will fix it. We simply compile the lambda after replacing the elements referring to db elements with array lookups, leaving everything else in tact. We do this as the projection can contain whatever code you like, so the only way to handle that is to compile it to IL and replace the db elements with array lookups so they can pull the data from the db from the array and it looks like the lambda specified is executed on the db.

I've posted on twitter and in the codeplex issue and a Microsoft engineer I know has taken the time to look into this, and has asked me to provide a small repro of this to see whether they can reproduce it. I don't have time for that today, but will try tomorrow (friday). https://twitter.com/KirillOsenkov/status/413454576436518912

If you are on twitter and can provide him with an easy repro before tomorrow, please reply to that tweet above

Otis wrote:

I've posted on twitter and in the codeplex issue and a Microsoft engineer I know has taken the time to look into this, and has asked me to provide a small repro of this to see whether they can reproduce it. I don't have time for that today, but will try tomorrow (friday). https://twitter.com/KirillOsenkov/status/413454576436518912

If you are on twitter and can provide him with an easy repro before tomorrow, please reply to that tweet above

Thanks Frans, I'm not on twitter and don't have time to make repro project but to simplify what I've found: in linqpad:

Order.GroupBy(o => new {Group=(object)o.CustomerId} ).Select(o => new  {Group = o.Key.Group}) //Fails

Order.GroupBy(o => new {Group=(object)o.CustomerId} ).Select(o => new  {Group = (object)o.Key.Group}) //With cast to object in projection is OK

The failing query works fine in both LinqToSQL and LinqToObjects.

A cast to object sorts it, that's a good enough workaround for now.

Thanks for testing on linq to sql and linq to objects and that simplified crash query. I've created a simple repro with this and will send that to MS for them to look at Interesting that linq to sql doesn't crash on this.

I tried to grab the lambda that gets compiled and when I did so, I noticed the issue: an unevaluated expression. What I found strange is that when I looked at the one which did work, the expression was properly evaluated. So the lambda compilation crashes when it finds an expression which it can't resolve. There might still be a problematic issue somewhere in the C# compiler though, as the cast to object does make the linq provider be able to properly convert the linq expression. I have to look into what the expression looks like in both.

Explanation:

The root cause is an expression in the lambda to compile which isn't evaluated properly.

Query which fails: var q = metaData.Order.GroupBy(o => new { Group = (object)o.CustomerId }).Select(o => new { Group = o.Key.Group });

Query which succeeds: var q = metaData.Order.GroupBy(o => new { Group = (object)o.CustomerId }).Select(o => new { Group = (object)o.Key.Group });

The lambda 'o => new { Group = o.Key.Group }' in the projection is rewritten at execution time, so that values coming from the db are consumed by the lambda. The linq provider rewrites this lambda to: (values, indices) => new { Group = values[indices[x]] }

where 'x' is a hardcoded value in the indices array for the field replaced with the array indices. When the lambda compile crashes the lambda looks like: .Lambda #Lambda1<SD.LLBLGen.Pro.ORMSupportClasses.ProjectionValueProducerFunc>( System.Object[] $values, System.Int32[] $indices) { (System.Object).New <>f__AnonymousType137`1System.Object }

When the lambda compile succeeds, it looks like: .Lambda #Lambda1<SD.LLBLGen.Pro.ORMSupportClasses.ProjectionValueProducerFunc>( System.Object[] $values, System.Int32[] $indices) { (System.Object).New <>f__AnonymousType137`1System.Object }

Clearly the issue is that there's an unevaluated expression in the lambda which causes the crash (IMHO the crash itself is unclear, so the error reporting should be way better in this case).

As the expression gets evaluated when there's a cast in front of it, I've looked into how the expression looks like before evaluation, so at the start of the query. The only difference is that the one which works has o.Key.Group wrapped in a Convert expression, due to the cast.

I've added that to the connect issue, so more info is given there too. The error message given by the lambda compiler should be better, and we should fix that issue.

Fixed runtime is attached. This avoids the mistake of leaving an expression in the lambda because the types don't exactly match but are convertible to each other (so the compiler accepted it). It can still do this though, if it can't determine whether the type is convertible. However that's in theory. In practice I don't think I can think of a query which results in that situation. (I'll leave that to you, as you've managed to come up with those in the past )

(found a better fix, have uploaded a better fix, it's now attached to this post.)