Generated SQL Performance Issues

Posts   
 
    
jader201
User
Posts: 33
Joined: 20-Mar-2007
# Posted on: 19-Sep-2008 17:26:09   

I have a question regarding the logic behind the generated SQL. Specifically, I have a query that is timing out because it seems to be using strange logic to build the query.

Basically, I have two tables which are joined via a composite key. I have two filters on the query, one on one table, and one on the other. But when LLBL creates the SQL from the query, it is using strange logic to join the tables and filter the results.

Here is the code for the data source:

  <llblgenpro:LLBLGenProDataSource2 ID="ContractDS" runat="server" EnablePaging="true" LivePersistence="false"
    DataContainerType="EntityCollection" 
    AdapterTypeName="Components.DataAccessLayers.MyDatabase.DatabaseSpecific.DataAccessAdapter, Components.DataAccessLayers.MyDatabaseDBSpecific"
    EntityFactoryTypeName="Components.DataAccessLayers.MyDatabase.FactoryClasses.ContractMasterEntityFactory, Components.DataAccessLayers.Servicer"
    OnPerformSelect="ContractDS_PerformSelect" OnPerformGetDbCount="ContractDS_PerformGetDbCount" 
    >
  </llblgenpro:LLBLGenProDataSource2>

Here is the code for the data source's PerformSelect:

    protected void ContractDS_PerformSelect(object sender, SD.LLBLGen.Pro.ORMSupportClasses.PerformSelectEventArgs2 e)
    {
        RelationPredicateBucket filter = new RelationPredicateBucket();
        filter.Relations.Add(ContractMasterEntity.Relations.CustMasterEntityUsingCustomerIdDataSource, JoinHint.Left);

        filter.PredicateExpression.Add(ContractMasterFields.DealerId == "0015392");
        filter.PredicateExpression.Add(CustMasterFields.LastName == LastName.Text);

        using (DataAccessAdapter adapter = new DataAccessAdapter())
        {

            PrefetchPath2 path = new PrefetchPath2((int)EntityType.ContractMasterEntity);
            path.Add(ContractMasterEntity.PrefetchPathCustMaster);

            adapter.FetchEntityCollection(e.ContainedCollection, filter,
                    e.MaxNumberOfItemsToReturn, e.Sorter, path, e.PageNumber, e.PageSize);
        }
    }

Here is the resulting SQL query for the customer data (I replaced column names with "*" fro simplicity):

SELECT  *
FROM    [MyDatabase].[dbo].[Cust_Master]
WHERE
(
    [MyDatabase].[dbo].[Cust_Master].[Customer_ID] IN 
    (
    SELECT  [MyDatabase].[dbo].[Contract_Master].[Customer_ID] AS [CustomerId]
    FROM    (
        [MyDatabase].[dbo].[Cust_Master]
        RIGHT JOIN [MyDatabase].[dbo].[Contract_Master]
        ON  [MyDatabase].[dbo].[Cust_Master].[Customer_ID]=[MyDatabase].[dbo].[Contract_Master].[Customer_ID]
        AND [MyDatabase].[dbo].[Cust_Master].[Data_Source]=[MyDatabase].[dbo].[Contract_Master].[Data_Source]
        )
    WHERE   ( 
            ( 
                (
                    [MyDatabase].[dbo].[Contract_Master].[Dealer_ID] = @DealerId1
                AND [MyDatabase].[dbo].[Cust_Master].[Last_Name] = @LastName2
                )
            )
        )
    )
AND [MyDatabase].[dbo].[Cust_Master].[Data_Source] IN
    (
    SELECT  [MyDatabase].[dbo].[Contract_Master].[Data_Source] AS [DataSource]
    FROM    (
        [MyDatabase].[dbo].[Cust_Master]
        RIGHT JOIN [MyDatabase].[dbo].[Contract_Master]
        ON  [MyDatabase].[dbo].[Cust_Master].[Customer_ID]=[MyDatabase].[dbo].[Contract_Master].[Customer_ID]
        AND [MyDatabase].[dbo].[Cust_Master].[Data_Source]=[MyDatabase].[dbo].[Contract_Master].[Data_Source]
        )
    WHERE   (
            (
                (
                    [MyDatabase].[dbo].[Contract_Master].[Dealer_ID] = @DealerId3
                AND [MyDatabase].[dbo].[Cust_Master].[Last_Name] = @LastName4
                )
            )
        )
    )
)

The above query takes two minutes to execute. However, the following query is much simpler, but is logically the same query, and takes a couple seconds to execute (uses indexes properly). Keep in mind, there are indexes on both [Contract_Master].[Dealer_ID] and [Cust_Master].[Last_Name].

SELECT  *
FROM    [MyDatabase].[dbo].[Cust_Master]
JOIN    [MyDatabase].[dbo].[Contract_Master]
ON  [MyDatabase].[dbo].[Cust_Master].[Customer_ID] = [MyDatabase].[dbo].[Contract_Master].[Customer_ID]
AND [MyDatabase].[dbo].[Cust_Master].[Data_Source] = [MyDatabase].[dbo].[Contract_Master].[Data_Source]
WHERE   [MyDatabase].[dbo].[Contract_Master].[Dealer_ID] = @DealerId1
AND [MyDatabase].[dbo].[Cust_Master].[Last_Name] = @LastName2

So my question is, what is causing LLBL to use this overly complicated and much less performant query? Does it have something to do with how the data is related (composite key), or is there something else going on? Is there anything I can do to help it use a better query?

Thanks in advance. Jerad

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39861
Joined: 17-Aug-2003
# Posted on: 19-Sep-2008 18:15:50   

LLBLGen Pro version? Runtime lib build nr? (see guidelines topic in this forum)

Frans Bouma | Lead developer LLBLGen Pro
jader201
User
Posts: 33
Joined: 20-Mar-2007
# Posted on: 19-Sep-2008 19:22:21   

Sorry about that. Here you go:

LLBL Gen Pro V. 2.5, Final Build Runtime = 2.5.0.0 Database = MS SQL 2005 (Build 3790: Service Pack 2)

Let me know if you need anything else.

Thanks again. Jerad

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39861
Joined: 17-Aug-2003
# Posted on: 19-Sep-2008 20:46:23   

Runtime library build nr would be great simple_smile

The runtime library version is obtainable by rightclicking the SD.LLBLGen.Pro.ORMSupportClasses.NETxy.dll in windows explorer and then by selecting properties and the version tab. The version is then enlisted at the top as the fileversion. It has the typical format as 2.x.0.YYMMDD, or starting in 2007, the format 2.x.YY.MMDD

To be sure, please be sure you are using the latest v2.5 build (see customer area)

Frans Bouma | Lead developer LLBLGen Pro
jader201
User
Posts: 33
Joined: 20-Mar-2007
# Posted on: 19-Sep-2008 21:05:56   

Dang, sorry. I accidentally pulled the version from the CF20.dll file instead of the NET20.dll file.

Here you go: 2.5.7.1019

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39861
Joined: 17-Aug-2003
# Posted on: 20-Sep-2008 11:59:13   

The query you posted is caused by the prefetch path. As it's a compound key, it has to filter the rows (children) to fetch based on the set of PK values of the set previously fetched (parents), through 1 subquery per field.

The join isn't the same resultset: it could lead to a LOT of duplicates which could severily bog down the performance as well, because for example the duplicates have to be filtered out on the client. It also has other problems: if you have a set of joins on the rows to fetch, the rows (parents) to base the filter on don't necessarily belong in the join set, as that could ruin LEFT/RIGHT joins or give false positives.

You can tweak this though: using the ParameterizedPrefetchPathThreshold flag on the adapter. If the # of parent entities (ContractMaster) is for example 150, set it to 300. (as you're using a compound key). It then should produce an IN query with solely the values of the parents in parameters, which is much faster.

Your query is the edge case btw where things go slow in the prefetch path system: a join in the parent and a compound PK-FK relation and more parent entities than the threshold is set to, which means you get the join in each subquery for each field.

In v2.6 we tried to optimize this with an EXIST query, which is the right way to solve this. We ran out of time unfortunately so this was postponed, as most of the time people don't run into this. I don't think I can backport a fix to v2.6 to v2.5 in this area unfortunately as other code was changed in v2.6, though I'll try to make an attempt to add EXISTS queries for this in v2.6 on monday. v2.6 is a free upgrade for you, and we made things faster, less memory consuming etc. so I'd upgrade to v2.6 if it's not a problem for you. (your runtime lib is also very old).

I'll report back in this thread on monday after I've spend time on this.

Frans Bouma | Lead developer LLBLGen Pro
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39861
Joined: 17-Aug-2003
# Posted on: 22-Sep-2008 10:53:42   

rcp-software, if you're wondering where your post went: I've removed it to keep this thread clean of off-topic material, as this thread is about something completely different than your problem. Please open a new thread per request.

PLease see: http://www.llblgen.com/tinyforum/Messages.aspx?ThreadID=7722

A separate thread is therefore required for your problem. This forum software doesn't have the facility to split off threads from other threads.

A possible answer to your question: just pass a string, all method parameters are converted to the proc parameter they represent.

Frans Bouma | Lead developer LLBLGen Pro
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39861
Joined: 17-Aug-2003
# Posted on: 22-Sep-2008 11:56:06   

jader201, there's a tiny problem... Postgresql 7.4/8.0, two database versions supported by llblgen pro can't deal with alias references in subqueries. The problem is though: the code creating the prefetch path subqueries is generic code shared among all databases. It's at that point also not detectable what database is targeted... I.o.w.: adding support for this means those databases will run into an exception in the DB due to a not runnable query.

It's not possible to toss out the aliases, as it is required to tie the subquery to the outside query. In Linq we also do this a lot, but users of those databases can then at least fall back onto our native api. if we change this in the middle of a version, they have to re-write applications or decide not to upgrade to a later build of the runtime.

Postponed.

Frans Bouma | Lead developer LLBLGen Pro