Please attach a picture as a picture, not as a pdf. We don't open pdfs.
With performance problems, there's actually nothing else to do but first to profile what's going on. We can debate all day what is likely happening, but it might not be the bottleneck. What I can tell you is that loading a massive amount of entities into a UI and binding these to controls could be slowing down by any number of reasons, which are unknown until it's profiled (e.g. is it in the entity view's indexing, or is it in the grid which tries to resort all rows?).
Meindert wrote:
At the moment I have not the capability to profile but please try to explain me what happens in the ORM layer.
Situation (see also diagram):
- ControlModules are bound to the datagrid/treelist (DevExpress);
- ControlModuleAC are prefetched on ControlModule;
So we're talking 750.000+ entities live in memory ? that's an insane amount, sorry.
- One particular ControlModule is saved and re-fetched NOT the whole collection is saved;
- Wait 3-4 seconds before Grid/Tree becomes active again.
...which can be anything. To give you an idea: when you change in the designer a field mapping of a field, if we wouldn't do anything, it would become a slow experience where you'd have to wait for 2-3 seconds, even though the property change is instantly. Weird, right?
The main issue was that because of a change, containing objects notified observers that they were changed, and containing objects of those containing objects did the same, which ended up a LOT of objects notifying observers something was changed and they better repaint/update themselves. 99.9% of this work is redundant: even if a node in the tree would need to be updated, it would so just once, not a lot of times. So after profiling everything we saw the vast majority of the performance was in event handlers doing the same thing over and over again. So we added a simple event throttler object between event and control, which only re-raised the events further every x ms, and filtering out duplicates. Everything is then fast and smooth. There were many of these 'event floods', which start simple and logical, and due to multiple paths through the object graph, can bubble upwards to a lot of events which would all result in 'hey, repaint the grid!', which of course will slow down things tremendously.
That's why profiling is essential for all these kind of discussions, full stop. Guesswork just wastes anyone's time, trust me
Why I think LLBGen is my concern and not the grid/tree:
- When I remove the prefetchpath to ControlModuleAC the grid reacts instantly. This means that the IListChanged is not a concern regarding the datagrid/treelist update!
How do you know? You cut the # of entities by a massive number too doing that, that might be the problem too, who knows?
Questions:
- If I change one entity and write/re-fetch this entity then the ORM layer is processing 3-4 seconds (No profiler outcome but test scenario made that clear to me);
what does 'the orm layer is processing' mean, if I may ask, and how did you measure this? with a stopwatch call around SaveEntity() or unit of work commit?
- What happens if one entity is re-fetched and recursive is true;
If recursive is true, all entities reachable through that entity are examined whether they're 'dirty' or have dirty related entities which can sync FK values with them (so they become dirty). This is done using a fast graph traversal algorithm, but as you can imagine, if every entity is reachable in your 750.000 entities in-memory it will still take a while.
After that's done, all entities found to be persisted are saved with an insert/update query. After each entity has been saved, and refetch isn't switched off (see my post earlier), it's refetched and set as Fetched. I checked the codebase whether it signals any events, and it just raises AfterSave, which isn't bound by our own code afaik.
- What happens not if one entity value is written to the database and re-fetch is false (besides out of sync).
then the entity graph is still traversed, if recurse is true (default in adapter), though after the insert/update query is ran, nothing is done further.
Seriously, I'm not advising you to do weird things, I'm dealing with performance oriented code for a long time, there are really no other ways to look at things than this, so please:
1) seriously cut down the # of entities loaded into a UI. Use paging instead. No human is able to look at 100's of rows, let alone 1000s. Load data on-demand, not all at once. If you need all rows in your database in-memory for aggregates, calculate them in the DB. The runtime isn't slow, but massive amounts of data in a UI will make UIs in general simply slow to work with unless you take serious measures. E.g. adding a new entity to a collection with 10000 entities will take more time than adding it to an empty collection, simply because it will check if the entity is already there. Profiling will show you that. This is caused by a feature which in general is 'great to have' but in large sets can be a burden, so it can be switched off. But until you know that's the cause, switching it off is just trying to switch off a light by trying every switch in the building.
2) profile, profile, profile. Only then you know where the bottleneck is and you can do something about it, if that's required. E.g. loading 750000 rows in a grid hierarchy will take more time than loading 30 rows in a grid. Profiling that will show you a 'bottleneck' which isn't really there, it's caused by a massive amount of data: it might even be 3-4 seconds is very fast compared to what you'd expect (I have no idea, just an example) with the amount of data at hand.