- Home
- LLBLGen Pro
- LLBLGen Pro Runtime Framework
Take method just doesn't work
I had a query which I instructed to get the Top 3 records. There were some joins where clauses etc, all which I have removed and distilled down to this very simple line:
meta.Offer.OrderBy(p => p.EndDate).Select(p=>new {p.Title}).Take(5).ToList()
Which results in this atrocity:
SELECT [LPLA_1].[Title] FROM [InfiniteRewards].[dbo].[Offer] [LPLA_1] ORDER BY [LPLA_1].[EndDate] ASC
W.T.F. I still get 3 records in my IList. The "take" functionality is happening in memory. Got a whole website that doesnt' work because of this.
I upgraded to 13th october runtimes and in my sample query Take is now working.
I went back to my original query and it still doesn't work, it seems that if you join no "top" is output and the list is shortened in memory.
I tried taking the query generated by llbl and put "DISTINCT TOP 3" in it. It then became apparent what the problem was, I have image/ntext columns which meant distinct couldn't be used and because I was joining that meant TOP couldn't be used (effectively).
Makes sense...
worldspawn wrote:
I tried taking the query generated by llbl and put "DISTINCT TOP 3" in it. It then became apparent what the problem was, I have image/ntext columns which meant distinct couldn't be used and because I was joining that meant TOP couldn't be used (effectively).
Makes sense...
Indeed. You may try to re-accommodate your query (by not joining, and excluding some fields on the main entity) or live with the in-memory approach.
Yeh that's what I did:
var offers = (from c in meta.Offer
join o in meta.AuditOffer on c.OfferId equals o.OfferId
where o.MemberId == memberId && c.IsActive == true
orderby c.EndDate descending
select c).ExcludeFields(p=>p.CompanyOverview, p=>p.Voucher, p=>p.Image, p=>p.Image1, p=>p.Image2).Take(3)
.WithPath(path => path.Prefetch<RedemptionLevelEntity>(p => p.RedemptionLevelUsingOfferTypeId).Prefetch<AlliancePartnerEntity>(p => p.AlliancePartnerUsingAlliancePartnerId)).ToList();
Works like a charm now
This is still frustrating me. I had a bug in my query. I was ordering by c.EndDate and I should have been ordering by o.DateCreated.
As soon as I flipped that ordering over to other field it was fail city again.
"ORDER BY items must appear in the select list if SELECT DISTINCT is specified."
Yet another PoS sql rule. Because DateCreated isn't in the select list LLBL has reverted to doing the Take operation in memory. This is annoyingly delicate.
Is there someway I can add o.DateCreated to the select list? I'd like to avoid doing something like:
select new { offer = new OfferEntity(){ OfferId = c.OfferId /* etc etc etc */}, o.DateCreated }
This would be a giant pain in the ass. All I want is ", DateCreated" added to the select list, I don't need the value though.
So I tried the nightmare query:
var offers = (from c in meta.Offer
join o in meta.AuditOffer on c.OfferId equals o.OfferId
where o.MemberId == memberId && c.IsActive == true
orderby o.DateCreated descending
select new { Offer = new OfferEntity() { OfferId = c.OfferId, Title = c.Title, RedemptionLevelId = c.RedemptionLevelId, RedemptionMethodId = c.RedemptionMethodId, AlliancePartnerId = c.AlliancePartnerId, LifestyleCategoryId = c.LifestyleCategoryId } }).Select(p => p.Offer).IncludeFields(p => p.OfferId, p => p.Title, p => p.RedemptionLevelId, p => p.RedemptionMethodId, p => p.AlliancePartnerId, p => p.LifestyleCategoryId).Take(number)
.WithPath(path => path.Prefetch<RedemptionLevelEntity>(p => p.RedemptionLevelUsingOfferTypeId).Prefetch<AlliancePartnerEntity>(p => p.AlliancePartnerUsingAlliancePartnerId)).ToList();
I am prefetching alliance partner and redemption method so it made sense that I needed to include those columns. For some bizarre reason I also had to include LifestyleCategoryId (otherwise sql error).
Anyway it didn't work. ORDER BY wasn't output into the SQL. As LLBL is so fond of 'detecting' issues and modifying the query accordingly, how about making it emit any order by columns to the select list if a distinct is needed??
I have another query that is doing a group by operation and then ordering the resultset by an aggregrate column returned from the group by subquery. It's failing for same reason because it ends with "select c" which doesn't output the aggregate column in the select statement.
Query I think I need is:
SELECT
[OfferId]
,[AlliancePartnerId]
,[RedemptionMethodId]
,[RedemptionLevelId]
,[LifestyleCategoryId]
,[Priority]
,[Title]
,[BriefDescription]
,[TermsConditions]
,[StartDate]
,[EndDate]
,[ClickNSaveUrl]
,[WebsiteUrl]
,[IsDraft]
,[IsApproved]
,[IsActive]
,[UpdatedBy]
,[UpdatedOn]
,[CreatedBy]
,[CreatedOn]
FROM
(
SELECT TOP 3
o.[OfferId]
,[AlliancePartnerId]
,[RedemptionMethodId]
,[RedemptionLevelId]
,[LifestyleCategoryId]
,[Priority]
,o.[Title]
,[BriefDescription]
,[TermsConditions]
,[StartDate]
,[EndDate]
,[ClickNSaveUrl]
,[WebsiteUrl]
,[IsDraft]
,[IsApproved]
,[IsActive]
,[UpdatedBy]
,[UpdatedOn]
,[CreatedBy]
,[CreatedOn]
, ao.DateCreated
FROM
Offer o
JOIN (SELECT TOP 3 ao.OfferId, MAX(DateCreated) [DateCreated]
FROM
AuditOffer ao
JOIN Audit a on a.AuditId = ao.AuditId
JOIN Offer o on ao.OfferId = o.OfferId
WHERE ao.MemberId = @memberId AND o.IsActive = 1 AND o.IsApproved = 1
GROUP BY ao.OfferId
ORDER BY MAX(DateCreated) DESC) [ao] ON o.OfferId = ao.OfferId
ORDER BY ao.DateCreated DESC) [x]
How can I express this in LINQ?
var audit = (from c in meta.AuditOffer
join o in meta.Offer on c.OfferId equals o.OfferId
where c.MemberId == 3374815 && o.IsActive == true && o.IsApproved == true
group c by c.OfferId into g
orderby g.Max(p=>p.DateCreated) descending
select new { OfferId = g.Key, DateCreated = g.Max(p => p.DateCreated) }).Take(3);
This output:
exec sp_executesql N'SELECT [LPA_L1].[OfferId], [LPA_L1].[LPAV_1] AS [DateCreated] FROM (SELECT [LPA_L4].[OfferId], MAX([LPA_L3].[DateCreated]) AS [LPAV_], MAX([LPA_L3].[DateCreated]) AS [LPAV_1] FROM (( [InfiniteRewards].[dbo].[Audit] [LPA_L3] INNER JOIN [InfiniteRewards].[dbo].[AuditOffer] [LPA_L4] ON [LPA_L3].[AuditId]=[LPA_L4].[AuditId]) INNER JOIN [InfiniteRewards].[dbo].[Offer] [LPA_L5] ON [LPA_L4].[OfferId] = [LPA_L5].[OfferId]) WHERE ( ( ( ( ( [LPA_L4].[MemberId] = @MemberId1) AND ( [LPA_L5].[IsActive] = @IsActive2)) AND ( [LPA_L5].[IsApproved] = @IsApproved3)))) GROUP BY [LPA_L4].[OfferId]) [LPA_L1] ORDER BY [LPA_L1].[LPAV_] DESC',N'@MemberId1 int,@IsActive2 bit,@IsApproved3 bit',@MemberId1=3374815,@IsActive2=1,@IsApproved3=1
But I wanted somethign along the lines of:
SELECT TOP 3 ao.OfferId, MAX(DateCreated) [DateCreated]
FROM
AuditOffer ao
JOIN Audit a on a.AuditId = ao.AuditId
JOIN Offer o on ao.OfferId = o.OfferId
WHERE ao.MemberId = @memberId AND o.IsActive = 1 AND o.IsApproved = 1
GROUP BY ao.OfferId
ORDER BY MAX(DateCreated) DESC
Why are there two select statements? I would expect 1. **The "TOP" also was not output. ** I put TOP 3 into the query, didn't an error but I got exactly the result I wanted. WHY IS THE TAKE BEING DONE IN MEMORY? The first select seems to do the same as the second except the second is selecting the max datecreated twice for some reason.
Really getting frustrated with this.
I think your query can be shortened to the following:
SELECT
o.[OfferId]
,[AlliancePartnerId]
,[RedemptionMethodId]
,[RedemptionLevelId]
,[LifestyleCategoryId]
,[Priority]
,o.[Title]
,[BriefDescription]
,[TermsConditions]
,[StartDate]
,[EndDate]
,[ClickNSaveUrl]
,[WebsiteUrl]
,[IsDraft]
,[IsApproved]
,[IsActive]
,[UpdatedBy]
,[UpdatedOn]
,[CreatedBy]
,[CreatedOn]
FROM
Offer o
JOIN (SELECT TOP 3 ao.OfferId, MAX(DateCreated) [DateCreated]
FROM
AuditOffer ao
JOIN Audit a on a.AuditId = ao.AuditId
JOIN Offer o on ao.OfferId = o.OfferId
WHERE ao.MemberId = @memberId AND o.IsActive = 1 AND o.IsApproved = 1
GROUP BY ao.OfferId
ORDER BY MAX(DateCreated) DESC) [ao] ON o.OfferId = ao.OfferId
-
The inner most query is grouped by offerId, and thus i will return 3 records with unique values of offerId.
-
When you join it to the direct top query on offerId, you will then get only 3 records as offerId is the PK of the Offer table as it seems.
-
Then you don't need to specify Top 3 again, as you will end up with only 3 records.
-
Accordingly you don't need to Order By the DateCreated in the top query.
-
Also you don't need to return the dateCreated, so it will be safe to omit the Order By clause from the top query.
Please try the above query and see if it satisfies the requirements.
Hi Walaa,
Ok i've expressed your revised query as this:
ar audit = (from c in meta.AuditOffer
join o in meta.Offer on c.OfferId equals o.OfferId
where c.MemberId == 3374815 && o.IsActive == true && o.IsApproved == true
group c by c.OfferId into g
orderby g.Max(p=>p.DateCreated) descending
select new { OfferId = g.Key, DateCreated = g.Max(p => p.DateCreated) }).Take(3);
var recentOffers = (from c in meta.Offer
join a in audit on c.OfferId equals a.OfferId
select c).ExcludeFields(p => p.CompanyOverview, p => p.Voucher, p => p.Image, p => p.Image1, p => p.Image2);
//audit.ToList();
recentOffers.ToList();
Which was output as:
SELECT DISTINCT [LPA_L2].[OfferId], [LPA_L2].[AlliancePartnerId], [LPA_L2].[RedemptionMethodId], [LPA_L2].[RedemptionLevelId], [LPA_L2].[LifestyleCategoryId], [LPA_L2].[Priority], [LPA_L2].[Title], [LPA_L2].[BriefDescription], [LPA_L2].[TermsConditions], [LPA_L2].[StartDate], [LPA_L2].[EndDate], [LPA_L2].[ClickNSaveUrl], [LPA_L2].[WebsiteUrl], [LPA_L2].[IsDraft], [LPA_L2].[IsApproved], [LPA_L2].[IsActive], [LPA_L2].[UpdatedBy], [LPA_L2].[UpdatedOn], [LPA_L2].[CreatedBy], [LPA_L2].[CreatedOn] FROM ( (SELECT [LPA_L3].[OfferId], [LPA_L3].[LPAV_1] AS [DateCreated] FROM (SELECT [LPA_L5].[OfferId], MAX([LPA_L4].[DateCreated]) AS [LPAV_], MAX([LPA_L4].[DateCreated]) AS [LPAV_1] FROM (( [InfiniteRewards].[dbo].[Audit] [LPA_L4] INNER JOIN [InfiniteRewards].[dbo].[AuditOffer] [LPA_L5] ON [LPA_L4].[AuditId]=[LPA_L5].[AuditId]) INNER JOIN [InfiniteRewards].[dbo].[Offer] [LPA_L6] ON [LPA_L5].[OfferId] = [LPA_L6].[OfferId]) WHERE ( ( ( ( ( [LPA_L5].[MemberId] = @MemberId1) AND ( [LPA_L6].[IsActive] = @IsActive2)) AND ( [LPA_L6].[IsApproved] = @IsApproved3)))) GROUP BY [LPA_L5].[OfferId]) [LPA_L3] ORDER BY [LPA_L3].[LPAV_] DESC) [LPA_L1] INNER JOIN [InfiniteRewards].[dbo].[Offer] [LPA_L2] ON [LPA_L2].[OfferId] = [LPA_L1].[OfferId])
SQL rejected this: The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP or FOR XML is also specified.
Also it doesn't anywhere have my TOP (Take) in it...
Also here's full trace of it. Note the extra, useless and unwanted subquery that gets created. Its sooo hard to read this sql, but it looks like in the first sub query the ORDER BY isn't output. So no ORDER BY and no TOP. I tried running the top sub query with a top and order by and it worked fine. Just LLBL isn't working.
Generated Sql query: Query: SELECT [LPA_L5].[OfferId], MAX([LPA_L4].[DateCreated]) AS [LPAV_], MAX([LPA_L4].[DateCreated]) AS [LPAV_1] FROM (( [InfiniteRewards].[dbo].[Audit] [LPA_L4] INNER JOIN [InfiniteRewards].[dbo].[AuditOffer] [LPA_L5] ON [LPA_L4].[AuditId]=[LPA_L5].[AuditId]) INNER JOIN [InfiniteRewards].[dbo].[Offer] [LPA_L6] ON [LPA_L5].[OfferId] = [LPA_L6].[OfferId]) WHERE ( ( ( ( ( [LPA_L5].[MemberId] = @MemberId1) AND ( [LPA_L6].[IsActive] = @IsActive2)) AND ( [LPA_L6].[IsApproved] = @IsApproved3)))) GROUP BY [LPA_L5].[OfferId] Parameter: @MemberId1 : Int32. Length: 0. Precision: 10. Scale: 0. Direction: Input. Value: 3374815. Parameter: @IsActive2 : Boolean. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: True. Parameter: @IsApproved3 : Boolean. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: True.
Method Exit: CreateSelectDQ Method Exit: CreateSubQuery Generated Sql query: Query: SELECT [LPA_L3].[OfferId], [LPA_L3].[LPAV_1] AS [DateCreated] FROM (SELECT [LPA_L5].[OfferId], MAX([LPA_L4].[DateCreated]) AS [LPAV], MAX([LPA_L4].[DateCreated]) AS [LPAV_1] FROM (( [InfiniteRewards].[dbo].[Audit] [LPA_L4] INNER JOIN [InfiniteRewards].[dbo].[AuditOffer] [LPA_L5] ON [LPA_L4].[AuditId]=[LPA_L5].[AuditId]) INNER JOIN [InfiniteRewards].[dbo].[Offer] [LPA_L6] ON [LPA_L5].[OfferId] = [LPA_L6].[OfferId]) WHERE ( ( ( ( ( [LPA_L5].[MemberId] = @MemberId1) AND ( [LPA_L6].[IsActive] = @IsActive2)) AND ( [LPA_L6].[IsApproved] = @IsApproved3)))) GROUP BY [LPA_L5].[OfferId]) [LPA_L3] ORDER BY [LPA_L3].[LPAV] DESC Parameter: @MemberId1 : Int32. Length: 0. Precision: 10. Scale: 0. Direction: Input. Value: 3374815. Parameter: @IsActive2 : Boolean. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: True. Parameter: @IsApproved3 : Boolean. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: True.
Method Exit: CreateSelectDQ Method Exit: CreateSubQuery Generated Sql query: Query: SELECT DISTINCT [LPA_L2].[OfferId], [LPA_L2].[AlliancePartnerId], [LPA_L2].[RedemptionMethodId], [LPA_L2].[RedemptionLevelId], [LPA_L2].[LifestyleCategoryId], [LPA_L2].[Priority], [LPA_L2].[Title], [LPA_L2].[BriefDescription], [LPA_L2].[TermsConditions], [LPA_L2].[StartDate], [LPA_L2].[EndDate], [LPA_L2].[ClickNSaveUrl], [LPA_L2].[WebsiteUrl], [LPA_L2].[IsDraft], [LPA_L2].[IsApproved], [LPA_L2].[IsActive], [LPA_L2].[UpdatedBy], [LPA_L2].[UpdatedOn], [LPA_L2].[CreatedBy], [LPA_L2].[CreatedOn] FROM ( (SELECT [LPA_L3].[OfferId], [LPA_L3].[LPAV_1] AS [DateCreated] FROM (SELECT [LPA_L5].[OfferId], MAX([LPA_L4].[DateCreated]) AS [LPAV], MAX([LPA_L4].[DateCreated]) AS [LPAV_1] FROM (( [InfiniteRewards].[dbo].[Audit] [LPA_L4] INNER JOIN [InfiniteRewards].[dbo].[AuditOffer] [LPA_L5] ON [LPA_L4].[AuditId]=[LPA_L5].[AuditId]) INNER JOIN [InfiniteRewards].[dbo].[Offer] [LPA_L6] ON [LPA_L5].[OfferId] = [LPA_L6].[OfferId]) WHERE ( ( ( ( ( [LPA_L5].[MemberId] = @MemberId1) AND ( [LPA_L6].[IsActive] = @IsActive2)) AND ( [LPA_L6].[IsApproved] = @IsApproved3)))) GROUP BY [LPA_L5].[OfferId]) [LPA_L3] ORDER BY [LPA_L3].[LPAV] DESC) [LPA_L1] INNER JOIN [InfiniteRewards].[dbo].[Offer] [LPA_L2] ON [LPA_L2].[OfferId] = [LPA_L1].[OfferId]) Parameter: @MemberId1 : Int32. Length: 0. Precision: 10. Scale: 0. Direction: Input. Value: 3374815. Parameter: @IsActive2 : Boolean. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: True. Parameter: @IsApproved3 : Boolean. Length: 0. Precision: 0. Scale: 0. Direction: Input. Value: True.
Method Exit: CreateSelectDQ Method Exit: CreatePagingSelectDQ: no paging.
I tried moving the orderby linq clause out to a .OrderByDescending method call before the Take call. Like this:
var audit = (from c in meta.AuditOffer
join o in meta.Offer on c.OfferId equals o.OfferId
where c.MemberId == 3374815 && o.IsActive == true && o.IsApproved == true
group c by c.OfferId into g
select new { OfferId = g.Key, DateCreated = g.Max(p => p.DateCreated) }).OrderByDescending(p=>p.DateCreated).Take(3);
That just ended up creating yet another useless subquery.
Couple of questions and a couple of remarks
questions: 1) what are 'useless subqueries' ? With linq the provider is slowly working through the expression tree to make it a sql executable query. Linq isn't 1:1 projectable onto sql, so conversions have to be made. In the end things might have been optimized abit, but with the subqueries the query should work.
2) when you ordered on a field not in the select list, did you still get an exception that ordering isn't allowed?
A couple of remarks: - when ordering on a field not in the select list, distinct can't be used. The reason is this: the RDBMS adds silently the field(s) ordered to the select list, then sorts and filters for distinct. As the ordering fields are extra columns, they could make a row 'unique' even though they're not in the select list. This is a problem with many databases, it's how they optimize the pipeline. LLBLGen Pro provides an abstraction layer for the database, so it's not the problem of the developer if a query using method ABC is going wrong, so it detects problems and switches to an alternative so you get the results you wanted.
-
Adding the extra columns to the select list is not going to help as you then still get duplicates which could give wrong results and thus TOP is producing the wrong subset.
-
To help you further, it's best if you try to limit down the query to the point where problems arise and from there, please describe the problems in 1 thread at a time. So if you run into multiple problems, please open multiple threads if possible, so the discussion in this thread is easy to follow
Hi Frans,
This first sub query has MAX(DateCreated) in the select list. (i don't see why)
[LPA_L5].[OfferId], MAX([LPA_L4].[DateCreated]) AS [LPAV_], MAX([LPA_L4].[DateCreated]) AS [LPAV_1]
The second one removes the extra datecreated field. [LPA_L3].[OfferId], [LPA_L3].[LPAV_1] AS [DateCreated]
When i tried moving the order by i ended up with another one of these queries that don't seem to do anything useful. That's what I meant by 'useless subquery'.
The behaviour I have observed is when ordering on a field in the select list the query behaves as expected (more or less). When ordering on a field NOT in the select list then it struggles and ends up performing the distinct and take in memory which results in a vastly slower query.
The last query revisement i wrote after Walaa's response does throw a sql exception saying "The ORDER BY clause is invalid in views, inline functions, derived tables, subqueries, and common table expressions, unless TOP or FOR XML is also specified."
I posted the SQL i'd like and LINQ that I have written, is there something wrong with my LINQ statement? What is causing the second subquery? How can I force Take and ORDER BY operations to occur where I want them?
If we focus only on the first part of the query:
var audit = (from c in meta.AuditOffer
join o in meta.Offer on c.OfferId equals o.OfferId
where c.MemberId == 3374815 && o.IsActive == true && o.IsApproved == true
group c by c.OfferId into g
orderby g.Max(p=>p.DateCreated) descending
select new { OfferId = g.Key, DateCreated = g.Max(p => p.DateCreated) }).Take(3);
Which outputs this:
exec sp_executesql N'SELECT [LPA_L1].[OfferId], [LPA_L1].[LPAV_1] AS [DateCreated] FROM (SELECT [LPA_L4].[OfferId], MAX([LPA_L3].[DateCreated]) AS [LPAV_], MAX([LPA_L3].[DateCreated]) AS [LPAV_1] FROM (( [InfiniteRewards].[dbo].[Audit] [LPA_L3] INNER JOIN [InfiniteRewards].[dbo].[AuditOffer] [LPA_L4] ON [LPA_L3].[AuditId]=[LPA_L4].[AuditId]) INNER JOIN [InfiniteRewards].[dbo].[Offer] [LPA_L5] ON [LPA_L4].[OfferId] = [LPA_L5].[OfferId]) WHERE ( ( ( ( ( [LPA_L4].[MemberId] = @MemberId1) AND ( [LPA_L5].[IsActive] = @IsActive2)) AND ( [LPA_L5].[IsApproved] = @IsApproved3)))) GROUP BY [LPA_L4].[OfferId]) [LPA_L1] ORDER BY [LPA_L1].[LPAV_] DESC',N'@MemberId1 int,@IsActive2 bit,@IsApproved3 bit',@MemberId1=3374815,@IsActive2=1,@IsApproved3=1
The first problem is "TOP 3" is not anywhere in that query. How can I solve that? It's a group by query so the results are already 'distinct', if I edit the sql and put TOP 3 i get no errors and I get the results I want. If i change to DISTINCT TOP i get that error about need the order by column in the select list. Only it is in the select, it's just been duplicated with a different name. Order by is on [LPA_L1].[LPAV_] and the select list has [LPA_L1].[LPAV_1]. Both are exactly the same value. Again I blame this extra sub query.
Some success!
I changed the audit part of the query to this:
var audit = (from c in meta.AuditOffer
join o in meta.Offer on c.OfferId equals o.OfferId
where c.MemberId == 3374815 && o.IsActive == true && o.IsApproved == true
group c by c.OfferId into g
//orderby g.Max(p=>p.DateCreated) descending
select new { OfferId = g.Key, DateCreated = g.Max(p => p.DateCreated) }).OrderByDescending(p=>p.DateCreated).Distinct().Take(3);
I had tried moving the order by out of linq into a OrderByDescending method call before but I didn't try it with Distinct() as well. This apparently makes all the difference. This way it looks like LLBL is able to identify that the group by aggregrate and the orderby are referencing the same field thus allowing a top, distinct order by query. Cue balloons and party hats!
I must say this LINQ stuff is... obscure