Hi,

using 5.5.1 hotfix 19/12 with Access DB.

I have to convert Btw percentages in Holland.

I'm still making use of DataScopes and I'm having problems with changes to multiple aggregates (Entity with children and grand children etc). One of the grandchildren has a price which has to be updated or a new one must be added.

Problem is, sometimes there are updates and/or additions, but the DataScope does not commitchanges. It just returns false and no SQL statements are written.

I don't know where the problem is. But I tried to narrow it down to a specific single entity, perhaps something is wrong somewhere in it. But was unsuccesful to find it. My attempt was to batch transactions in the datascope by 100 aggregates, then 10, then 5, then 2, then 1. With 1 it Always succeeds. But when using 2 or more, sometimes it succeeds, sometimes not. the larger the set, the lesser the chance.

I just don't know where to look. The Datascope has given me some nightmares before, it's hard to look for errors when no exceptions are thrown.

Maybe this has something to do with the Context again? Cant find it though. It's a rather large object hierarchy, so I don't know where to look either. I could export a JSON of two a set of two projects in which the Scope does nothing, but has dirty entities. With Object exporter. Maybe you can see something on these objects?

I don't know where to start either We need some a code snippets at least.

DataScope is a Context on steroids which creates a Unit of Work for you and can do work for you you otherwise have to do manually. There's not much more to it.

To determine what's in the Unit of work, you can do so, using CommitChanges(func): https://www.llblgen.com/Documentation/5.5/LLBLGen%20Pro%20RTF/Using%20the%20generated%20code/gencode_datascopes.htm#commiting-changes

If the unit of work is empty, no changes of entities inside the context are detected. If the unit of work contains entities, call ConstructSaveProcessQueues. This will give you the insert and update queue, in GetInsertQueue() and GetUpdateQueue(). If these are empty, no entities reachable are changed.

You say there are changes, but how do you know? If you clearly have a reference to an entity that's dirty/changed and has as context the datascope's context set (ActiveContext is set to that instance) and obtaining the Unit of work doesn't include that entity, then something odd is going on, but as Walaa said, we need a piece of code we can use to track down the issue.

Before you're going that route: I'm not going to stare at a lot of JSon data, sorry. The issue has to be reproducible with a simple test.

this works 1-by-1

        public async Task<List<BtwConversieReportItem>> UpdateAllProjectenAsync(BtwDto btwOud, BtwDto btwNieuw, DateTime ingangsdatum)
        {
            var projectIds = await _projectRepository.GetAllProjectIdsAsync();
            var reportItems = new List<BtwConversieReportItem>();
            var uow = _uowFactory.Invoke();
            uow.KeepAdapterOpen = true;
            //projectIds = new List<int>{6013};
            foreach (var batch in projectIds.BatchesOf(300))
            {
                var projecten = await _projectRepository.GetAggregatesByProjectIdsAsync(batch.ToList());
                foreach (var project in projecten)
                {
                    uow.Add(project);
                    foreach (var u in project.Uitgaven)
                    {
                        UitgavePrijzenConverteren(u, btwOud, btwNieuw, ingangsdatum);
                    }
                    var reportItemsProject = GetReportItems(project);
                    if (reportItemsProject.Count > 0)
                    {
                        reportItems.AddRange(reportItemsProject);
                        var success = await uow.CommitChangesAsync();
                        if (!success)
                            throw new Exception("UoW Commit niet succesvol terwijl wel verwacht");
                    }
                    uow.Reset();
                }
            }
            uow.Dispose();
            return reportItems;
        }

and this doesn't:

     public async Task<List<BtwConversieReportItem>> UpdateAllProjectenAsync(BtwDto btwOud, BtwDto btwNieuw, DateTime ingangsdatum)
        {
            var projectIds = await _projectRepository.GetAllProjectIdsAsync(); 
            var reportItems = new List<BtwConversieReportItem>();
            var uow = _uowFactory.Invoke();
            uow.KeepAdapterOpen = true;
            //projectIds = new List<int>{6013};
            foreach (var batch in projectIds.BatchesOf(300))
            {
                var projecten = await _projectRepository.GetAggregatesByProjectIdsAsync(batch.ToList());
                uow.Add(projecten);
                foreach (var project in projecten)
                {

                    foreach (var u in project.Uitgaven)
                    {
                        UitgavePrijzenConverteren(u, btwOud, btwNieuw, ingangsdatum);
                    }
                }
                var reportItemsProject = GetReportItems(projecten);
                if (reportItemsProject.Count > 0)
                {
                    reportItems.AddRange(reportItemsProject);
                    var success = await uow.CommitChangesAsync();
                    if (!success)
                        throw new Exception("UoW Commit niet succesvol terwijl wel verwacht");
                }
                uow.Reset();
            }
            return reportItems;
        }

The function GetReportItems(…) goes into the aggregate to find updated and newly added UitgavePrijsEntity's. These are grandchildren of Project in hierarchy Project - Uitgave(n) - UitgavePrijs (-zen). Just looking for .IsDirty (and .IsNew but these are dirty anyways).

So the aggregate is attached to the DataScope upfront and has changes, but somehow it is not picked up.

The CommitChangesAsync is

        public async Task<bool> CommitChangesAsync(DataScopeRefetchStrategyType refetchStrategy = DataScopeRefetchStrategyType.AlwaysRefetch)
        {
            var work = BuildWorkForCommit(); //for testing, check some property?
            base.RefetchStrategy = refetchStrategy;

            using (var adapter = _adapterFactory.CreateDataAccessAdapter())
            {
                return await this.CommitChangesAsync(adapter);
            }
        }

I dont use the work, but for testing it might give some clues?

Note: either reusing (and resetting) a DataScope descendend (uow i named it) or newing up each time, does not make any difference.

From where can I call ConstructSaveProcessQueues in the Datascope?

Please re-read what I wrote: you have to obtain the unit of work object using the CommitChanges(func) method as described in the docs I linked to. Then call the method on the unit of work object.

Please help me in this endeavour, I cant find it. this is what I have:

        public async Task<bool> CommitChangesAsync(DataScopeRefetchStrategyType refetchStrategy = DataScopeRefetchStrategyType.AlwaysRefetch)
        {
            //var work = BuildWorkForCommit();
            base.RefetchStrategy = refetchStrategy;

            return base.CommitChanges(CommitUnitOfWorkFunc);
            //using (var adapter = _adapterFactory.CreateDataAccessAdapter())
            //{
            //  return await this.CommitChangesAsync(adapter);
            //}
        }

        private bool CommitUnitOfWorkFunc(IUnitOfWorkCore uow)
        {
            uow. //doesnt have ConstructSaveProcessQueues method

            throw new System.NotImplementedException();
        }

Cast uow to UnitOfWork2, that one has the method. Then look in the debugger what the contents is or call GetInsertQueue()/GetUpdateQueue().

Like I said as well: if an entity is in there and has its ActiveContext set to the context of the datascope, it should be in the datascope. However it might be the entity is marked for deletion and have changes, then it will be triggered by your code as IsDirty but won't be persisted (so shouldn't be in the queues)

Also, all this might be a red herring, you base the conclusion something's wrong on the fact you get 'there's some entities which are changed and have to be persisted' from some code but the bug might be in there. E.g. the code might traverse a relationship towards an entity which isn't in the scope (has its ActiveContext property set to something else) or might be marked for deletion or something else.

I also don't know what you mean with batches of 2 etc. as access doesn't support batching, it has no effect.

WIth batches I mean, that I permit only so many projects in a DataScope as I configure. Then I call this a batch. These will be updated and Committed. Then I clean up and start over with another set of projects. Running with only 1 project in a DataScope at a time, it all works well over the total set. But when I add multiple projects in the DataScope, do the work and COmmit, then it says it has nothing to do, but there is.

fyi: i dont delete anything. I only update and/or add one particular Entity type.

When adding, I use the construct:

 var prijs = uitgave.UitgavePrijzen.AddNew(); //from ORMSupportClasses . this is the default implementation, no overriding here. Should expect the context to be set, and it does.

And I also don't add any other entities which are not in the scope. Simply said. I load the Entity's with all its children, put it in the scope, make changes, and commit.

Finished the code BTW:

        private bool CommitUnitOfWorkFunc(IUnitOfWorkCore iuow)
        {
            var uow = (UnitOfWork2)iuow;
            uow.ConstructSaveProcessQueues();

            var inserts = uow.GetInsertQueue();
            var updates = uow.GetUpdateQueue();

            using (var adapter = _adapterFactory.CreateDataAccessAdapter())
            {
                var count = uow.Commit(adapter);// await uow.CommitAsync(adapter);
                if (count == 0)
                    throw new Exception("Aantal verwacht!");
            }

            return true;

        }

It returns 0 in both inserts and updates. But when running it with only 1 project aggregate at a time, it runs ok, ie there are inserts or updates>0.

Maybe there's another edge case in which an ActiveContext is set to null. Do you have some code that I can use to traverse all entity nodes and check for the right ActiveContext, or is that not something we must persue?

Don't call AddNew(), that's a databinding related element from the old version when there wasn't any entityview available yet. Simply create a new entity using its constructor, then add it to the collection.

Might not make the difference, but AddNew() isn't something you should use in your code.

The ObjectGraphUtils offer a way to traverse the graph, e.g. : https://www.llblgen.com/Documentation/5.5/ReferenceManuals/LLBLGenProRTF/html/6DCAEC3E.htm

Pass in an entity in the graph and empty dictionaries for the rest, you'll get the graph's adjacency lists back in the dictionary passed as 2nd argument. That should contain all entities in the graph reachable from the entity you specified. You can then traverse that construct and check whether the context is the same.

You check for the # of elements returned. However are there any queries ran? I.e. maybe there's some issue in the # of elements returned from commit: if there are 2 insert queries run (enable tracing or use the ormprofiler) and you get 0 back then it's that.

Also, the queues in the unit of work created by the scope have the expected entities, namely the ones you changed / added? You didn't report on that.

I shall change AddNew by New.

i will try the util you referenced.

i dont do anything else as mentioned. so no additional queries whatsoever. The ORM profiler only starts a transaction and then immediately commits, but has nothing to do. no lines in between. didnt mention it before but i realize you need all info you can get.

and for the numbers i mentioned, yes indeed, they are the numbers from the two results of the variables i declared: updates and inserts. so not something else.

So the queues do contain the expected entities? But committing that uow does not result in any queries executed on the DB? Forget the # returned for now, check the queries that are actually executed.

Also, if 2 projects in your app make it fail, perhaps try 2 projects with 1 entity?, change that and you can debug it, traversing things with just 4 entities shouldn't take that much time...

It returns 0 in both inserts and updates. But when running it with only 1 project aggregate at a time, it runs ok, ie there are inserts or updates>0.

With this i mean:

Scenario 1: with multiple projects in the DataScope the results are:

private bool CommitUnitOfWorkFunc(IUnitOfWorkCore iuow)
        {
            var uow = (UnitOfWork2)iuow;
            uow.ConstructSaveProcessQueues();

            var inserts = uow.GetInsertQueue(); // == 0 even though some entities are .IsNew
            var updates = uow.GetUpdateQueue(); // == 0 even though some entities are .IsDirty

            using (var adapter = _adapterFactory.CreateDataAccessAdapter())
            {
                uow.Commit(adapter);// await uow.CommitAsync(adapter);
            }

            return true;

        }

Scenario 2: with only 1 project in the DataScope the results are:

private bool CommitUnitOfWorkFunc(IUnitOfWorkCore iuow)
        {
            var uow = (UnitOfWork2)iuow;
            uow.ConstructSaveProcessQueues();

            var inserts = uow.GetInsertQueue(); // == real number of inserts, can be 0 or more
            var updates = uow.GetUpdateQueue(); // == real number of updates, can be 0 or more

            using (var adapter = _adapterFactory.CreateDataAccessAdapter())
            {
                uow.Commit(adapter);// await uow.CommitAsync(adapter);
            }

            return true;

        }

Please check if the entities (which are new/dirty) are in the unit of work (it contains additional data structures for entities added for save, which aren't taken into account if the system thinks they're not dirty, hence the question)

And if possible, try a small set of entities and debug the creation of the unit of work in the datascope class (which is in the ormsupport classes).

I have some results of testing

Project 62 21253502 Uitgave 113 21253502 UitgavePrijs 673 21253502 False False UitgavePrijs 674 21253502 False False UitgavePrijs 1291 21253502 False False UitgavePrijs 5084 21253502 False False UitgavePrijs 6609 21253502 False False UitgavePrijs 7181 21253502 False False UitgavePrijs 7182 21253502 False False UitgavePrijs 7412 21253502 False False UitgavePrijs 9349 21253502 False False UitgavePrijs 0 21253502 True True Uitgave 1866 21253502 UitgavePrijs 3553 21253502 False False UitgavePrijs 0 21253502 True True Project 63 21253502 Uitgave 58 21253502

As you can see there are 2 new entities for one of the aggregates.

But do they have any fields set? that they're new is ok, but if they don't have any fields set no row will be inserted.

yep. Fields are set.

I can imagine you need proof?

But you already know that when I commit these aggregates One-by-one these additions are insert into the db...

Puser wrote:

yep. Fields are set.

I can imagine you need proof?

But you already know that when I commit these aggregates One-by-one these additions are insert into the db...

I don't need 'proof' this isn't a court. I can't do much more than give hints where to look, so now that everything seems to be there, you should debug the creation of the unit of work object by following CommitChanges() on the datascope. It will call the BuildWorkForCommit() method which will ask the internal context to create the unit of work.

It will end up in DataScopeContext.CreateFilledUnitOfWork(..) If there's an error you should see it there, as it does all its collecting there. The method is quite small so it shouldn't be that complex to do.

I don't see errors. I have debugged up to the DetermineActionQueues.

DetermineActionQueues is called for every element in elementsToProcess in ConstructSaveProcessQueues in UnitOfWork2.

List _entitiesToInsert is updated as variable queue in ObjectGraphUtils. Items are added. And at the end the list is cleared and then have to be added again from elementsPerType from sortedTypesList.

But in my second run of this call the queue has 3 items, are added to elementsPerType. queue is cleared, and in the last for each it is not again added.

Then I think the sortedTypesList is produced from the current Entity in ProduceTopologyOrderedList.

My guess is that the sortedTypesList can vary from entity to entity(type) when not all of the same types exist in the hierarchies. If so, then the code below will only add those new entities if the entity in the last run of DetermineActionQueues also has all these entities. A bit cryptive maybe, sorry for that, but I dont know how else to describe all of this else. So if the sortedTypesList does not contain one (or more) types in the queue, the queue is not complete or even empty, in my case.

            // now we have to sort the insert queue based on the sorted types list, so all entities belonging to a given type have to be placed in the slot of the type.
            // meaning, if we have as sorted types: Customer, Employee, Order, and as entities in the insert queue: Customer1, Order1, Customer2, Order2, Employee1, Order3,
            // the insert queue will become: Customer1, Customer2, Employee1, Order1, Order2, Order3. 
            // we can skip this step if there's just 1 type or we're processing updates as we'll leave those alone. 
            if(sortedTypesList.Count > 1 && forInserts && queue.Count > 1)
            {
                var elementsPerType = new Dictionary<Type, List<ActionQueueElement<TEntity>>>();
                foreach(var e in queue)
                {
                    var entityType = e?.Entity.GetType();
                    if(entityType == null)
                    {
                        continue;
                    }
                    var elements = elementsPerType.GetValue(entityType);
                    if(elements == null)
                    {
                        elements = new List<ActionQueueElement<TEntity>>();
                        elementsPerType.Add(entityType, elements);
                    }
                    elements.Add(e);
                }
                // clear the destination as we're going to restore it with the elements in the right order
                queue.Clear();
                foreach(var type in sortedTypesList)
                {
                    var elementsOfType = elementsPerType.GetValue(type);
                    if(elementsOfType == null)
                    {
                        continue;
                    }
                    queue.AddRange(elementsOfType);
                }
            }

Could you shed your light on this?

My guess is that the sortedTypesList can vary from entity to entity(type) when not all of the same types exist in the hierarchies.

Why guess? debug it, please. While debugging, did you find the sortedTypesList missing any type of the entities to be inserted?

Also, you're debugging the wrong code. I deliberately asked you to debug: DataScopeContext.CreateFilledUnitOfWork(..) as the UoW which goes wrong (i.e. inserts 0 rows) is empty, and that method (CreateFilledUnitOfWork) builds the UoW which has to have the entities to insert (but is empty in your case). The method is rather small so shouldn't be that hard to check. The first entity it misses is already enough.

What you're debugging is the code to calculate the queues, but if the unit of work is empty, the queues are empty as well, so you're debugging the one which does insert entities? IF not, you managed to make things more confusing and that's not going to help, sorry. (Besides, your assumptions would suggest no batching query would work, which isn't true, they work fine. For Access they're ignored anyway, as batching isn't used)

If I ask you to do things, please do those things. You ask for help, I try to do that, but if you then ignore that and look elsewhere, while admirable, it won't make things more clearer from our PoV.

Well I thought I done this and tried to help further. As I stated earlier:

I don't see errors. I have debugged up to the DetermineActionQueues.

If you think I'm ignoring then help me with debugging this. I clearly cant and was thinking I followed instructions.

And just like that I think I see the issue.

With the unit of work, it passes the queue it has itself for multiple graphs to the objectgraphutils. As you've seen correctly it clears the queue, but there are already elements in that queue from the previous call to the objectgraphutils. We have to test this with a test with 2 separate graphs inserting elements using 1 unit of work where the second graph has types not in the first one.

[Test]
public void MultipleDistinctGraphsInSingleUoWInsertTest()
{
    CustomerEntity customer = EntityCreator.CreateNewCustomer(1);
    customer.TestRunId = _testRunID;
    OrderEntity order1 = new OrderEntity();
    order1.Customer = customer;
    order1.TestRunId = _testRunID;
    order1.OrderDate = DateTime.Now;
    AddressEntity address = EntityCreator.CreateNewAddress(1);
    address.TestRunId = _testRunID;
    var uow = new UnitOfWork2();
    // first address, as the situation we're testing needs to have a 2+ entity graph in the second element
    uow.AddForSave(address);
    uow.AddForSave(customer);
    // calculate queues, there should be 3 elements in the insert queue
    uow.ConstructSaveProcessQueues();
    var insertQueue = uow.GetInsertQueue();
    Assert.AreEqual(3, insertQueue.Count);
}

Reproduces it: it contains 2 elements in the insert queue, not 3 as it should have: address is missing.

Fixing...

Could you please try the attached hotfix build if it fixes your issue? If so, we'll push the fix to nuget/site. Thanks!

Normally we'd push to nuget for the hotfix immediately but it's not sure if we've fixed it with this change (I think so, but just to be sure) and we can't push a new fix the same day.