Hey guys,

I'm investigating a huge performance issue and found that when we FetchEntity passing context it takes significantly more than if we do not pass it.

Members is a M:N collection contains 250,000 records, with context ~10 mins and without context is ~6 seconds.

            using (var adapter = new DataAccessAdapter(@"..."))
            {
                var group = new SysGroupEntity(groupId);

                var context = new Context();
                adapter.FetchEntity(group, new PrefetchPath2(EntityType.SysGroupEntity) { SysGroupEntity.PrefetchPathMembers }, context);

 .....

We use LLBLGen 5.1.1.0, adapter framework.

Table structure is like this:

table sysSubject ID guid Name nvarchar(250)

table sysGroup inherited from sysSubject by FK ID guid Icon nvarchar(100)

table sysGroupMembership GroupID guid SubjectID guid

Thanks, Anton

Hi Anton,

When you add entities to a Context, their PKs are added to the Context's internal cache, so it makes sense that it takes longer to complete. It's a big difference though.

Have you evaluated the performance of the whole routine? ORMProfiler would be great here, to identify the bottleneck.

Also, Could you please describe your actual scenario in which you need to user the Context class?

The problem is not in the context but when you use it. It actually sets property DoNotAddIfPresent for entity collection which causes it to check if item is already in the collection or not, when item is being added. As you know, internally it is a list and it is non indexed operation to do Contains (n^2 operation) - lots of calls to Equals etc. Please take a look and investigate what can be done to fix it. It is really critical for us.

Thanks, Anton

If you know the data is new (i.e. fetching into an empty or new collection) set the DoNotPerformAddIfPresent property of the collection to false. This way the collection won't make a linear search each time an entity is added, which should speed things up.

Yeah that was my initial thought but no luck because it is being reset when context is set for entitycollection, I.e when I call Fetch passing context :-(

Anton

Walaa wrote:

If you know the data is new (i.e. fetching into an empty or new collection) set the DoNotPerformAddIfPresent property of the collection to false. This way the collection won't make a linear search each time an entity is added, which should speed things up.

The fetch already does that for you An entity collection fetch sets that flag to false and performs a hash check with existing entities if the collection isn't empty. It doesn't leave it to Contains() anymore (it did in the early days, pre, v2) to check for duplicates as that can be slow.

Looking into what's going on with context and the flag now.

A profile shows indeed that it uses Contains. The reason the flag is set is that because the entities in the collection are unique, adding a duplicate one isn't what you want. It currently relies on a simple seek using Contains() which is OK for general use. There's really not much else there's to do without introducing a memory hungry index, as the context can't be used to determine whether the entity to add is present in the collection (see below)

The problem occurs during merge of the fetched prefetch path, which uses Add().

I have to ask... why fetch 250,000 entities at all? A m:n relationship might contain many more entities over time (e.g. millions), which your code will fetch all then too. This isn't sustainable, you should work with smaller sets in memory.

In any case, the flag can't be switched off by default, that's a breaking change which we won't make in a fix as some code might rely on that. It's a problematic situation, as there's no fix which doesn't break anything or changes the templates (which we don't want to do either as many people have made subtle changes to the entity templates and we like to keep them as-is for now and won't change them mid-release in a fix).

The main thing is that what's in the collection is in the context, but what's in the context isn't necessarily in the collection (your root entity is in the context, not in the m:n collection for example). So although the context contains a set of pk hashes for fast checking whether an entity is present in it, it can't be used to test whether an entity is present in the collection. As this is a merge resulting from a fetch, it is highly likely the entity added to the collection isn't present, but the code used for that doesn't know (it uses the regular 'Add()' method, not a shortcut).

Though there's a workaround. Change your code to:

using (var adapter = new DataAccessAdapter(@"..."))
{
    var group = new SysGroupEntity(groupId);
    group.ActiveContext = new Context();
    group.Members.DoNotPerformAddIfPresent = false;
    adapter.FetchEntity(group, new PrefetchPath2(EntityType.SysGroupEntity) { SysGroupEntity.PrefetchPathMembers });
    group.Members.DoNotPerformAddIfPresent = true; // or not, if you don't need this functionality.

Here you assign the context to the entity before the fetch. It's the same as passing it to the method, as it gets assigned to the entity in the method you called. Assigning it before the fetch and then accessing the collection will create the collection and add the context to it, setting the flag. You can then reset it as you don't want it to be set. Your fetch then doesn't need to receive the context (as the entity fetched already contains the context so it's used) and as the flag isn't set, Members.Add() should be fast.

We'll look into solving this problem in the next release, as that's a way to introduce some changes needed for this which we can't do mid-release in a patch.

In any case, even though the workaround will make your fetch faster, you really shouldn't fetch that many entities in a normal query, unless you are processing them for something but even then it often is much faster do bring the processing to the data than vice versa

Hope this helps.

Completely agree with you and was also a bit surprised when I saw that client has one group with 250K of members in it i.e. other groups or employees. It is just unmanageable - we haven't expected that either.

Currently our code have to load it, using Prefetch and we do use context, having one instance is pretty important to us. Plus, we do use DoNotPerformAddIfPresent, so that no duplicates can be added to the collection. We obviously do not need/want to check it when we Fetch because we do have a unique index in DB. I will try to turn it off for the Fetch duration.

I think it would be awesome if this got fixed in the next version, because add operations for such a bit collections are just nightmare.

Anton

yes, it should be intuitive, as in: you'd assume it's fast and not slow because some flag is set. We'll try to come up with a solution for this in v5.3

Otis wrote:

yes, it should be intuitive, as in: you'd assume it's fast and not slow because some flag is set. We'll try to come up with a solution for this in v5.3

Cool, thank you!

Anton

Implemented in v5.3 (post EAP)