I'm parsing a file and creating entities and adding them to a new parent entity. The parent entity is of type TraceSetEntity, and children are of type TraceEntity.

Then, in a wizard I can elect not to import one or more of these child entities; in which case I remove it from the parent collection using EntityCollection.Remove(). I've verified that this is happening as intended.

If I remove one of the children, I get an exception when I save. I ran SQL Server Profiler and was surprised to see an insert statement for the entity that I removed. Though not surprisingly, I suppose, the insert statement did not include the parent FK field, which of course cannot be null, thus the exception.

This seems very puzzling to me. Of course, if the entity was already in the database, that would be a different story. But here I have not yet persisted anything. I create an entity, add it to the parent, then decide I don't want it and remove it from the parent. It's as if GenPro "remembers" this entity, and only nulls out the FK field, leaving it in a state where it cannot legally be inserted into the database.

Also curious is that my save logic saves the top-level entity (WellEntity). Everything hangs off that. The insert statement that generates the exception contains NO fields that relate to the top level entity, the original parent entity, or any other entity.

The profile looks something like this (TraceSetID is parent FK):

INSERT INTO Traces (TraceID, TraceSetID, Name, Description, Origin)
// (repeat several times...)
INSERT INTO Traces (TraceID, Name, Description, Origin)
// notice no TraceSetID, thus exception, but this entity shouldn't be persisted anyway, right?

Hi, Jim. My bet is that you have an object reference to the new entity still lingering somewhere. You're doing a graph save, right? That entity is still in the graph, but probably connected through some other entity. Check all your object/entity assignments.

Jeff...

That's good thinking, but the only thing I do with this entity is add it to the parent's collection, then remove it. It's not associated with any other entity!

Could you check that the reference to the traceset entity is null in the trace entity after it's removed from the collection ?

Yes, before the removal the TraceSet property points to the parent, and after the removal it is null.

Do you think you could post some of the code that is causing the problem?

The thing is that an entity never ends up in the save queue if it's not referenced somewhere by an entity or collection which is examined, like the root entity you're saving.

So if the entity is removed from the queue and you're just saving the entity containing the collection recursively, and the entity removed isn't referenced by any other entity, it shouldn't end up in the queue. Do you use a unitofwork perhaps?

If you pass the TraceSetEntity's Traces collection to ObjectGraphUtils.DetermineActionQueues(IEntityCollection2 entityCollectionToSave, ref ArrayList insertQueue, ref ArrayList updateQueue, bool refetchAfterAction)

does the entity end up in one of the 2 queues?

Otis wrote:

The thing is that an entity never ends up in the save queue if it's not referenced somewhere by an entity or collection which is examined, like the root entity you're saving.

So if the entity is removed from the queue and you're just saving the entity containing the collection recursively, and the entity removed isn't referenced by any other entity, it shouldn't end up in the queue.

Thanks, Frans, that's what I thought and so I was really puzzled by this.

Do you use a unitofwork perhaps?

No (not yet perhaps I should say). This trace entity is not referenced anywhere else after I remove it.

Following your suggestion, I loaded a file with 10 traces, and flagged one of them, "BIT" for deletion. It appears to be successfully removed from the collection, but it's still in the insertQueue afterwards. Here's the code:

            ObjectGraphUtils utils = new ObjectGraphUtils();
            ArrayList a1 = new ArrayList();
            ArrayList a2 = new ArrayList();

            // only trace flagged for removal is "BIT", so this loop only executes once
            for (int x = 0; x < toDelete.Count; x++)
            {
                a1.Clear();
                a2.Clear();
                utils.DetermineActionQueues(traceSet.Traces, ref a1, ref a2, true);
                
                System.Diagnostics.Trace.WriteLine("PRIOR TO REMOVAL");
                System.Diagnostics.Trace.WriteLine("traceSet.Traces.Count: " + traceSet.Traces.Count.ToString());
                System.Diagnostics.Trace.WriteLine("a1.Count: " + a1.Count.ToString());
                System.Diagnostics.Trace.WriteLine("a2.Count: " + a2.Count.ToString());
                foreach (ActionQueueElement element in a1)
                {
                    TraceEntity trace = element.Entity as TraceEntity;
                    if (trace != null)
                        System.Diagnostics.Trace.WriteLine(trace.Name);
                    else
                        System.Diagnostics.Trace.WriteLine(element.Entity.ToString());
                }

                // toDelete is an EntityCollection that holds the TraceEntity "BIT"
                traceSet.Traces.Remove(toDelete[x]);

                a1.Clear();
                a2.Clear();
                utils.DetermineActionQueues(traceSet.Traces, ref a1, ref a2, true);
                
                System.Diagnostics.Trace.WriteLine("AFTER REMOVAL");
                System.Diagnostics.Trace.WriteLine("traceSet.Traces.Count: " + traceSet.Traces.Count.ToString());
                System.Diagnostics.Trace.WriteLine("a1.Count: " + a1.Count.ToString());
                System.Diagnostics.Trace.WriteLine("a2.Count: " + a2.Count.ToString());
                foreach (ActionQueueElement element in a1)
                {
                    TraceEntity trace = element.Entity as TraceEntity;
                    if (trace != null)
                        System.Diagnostics.Trace.WriteLine(trace.Name);
                    else
                        System.Diagnostics.Trace.WriteLine(element.Entity.ToString());
                }
            }

and the output (I've added a couple of blank lines and the comment):

PRIOR TO REMOVAL

traceSet.Traces.Count: 10
a1.Count: 13
a2.Count: 0
Chevron.RMA.DAL.EntityClasses.TraceSetEntity
ZDEN
CNCF
CVOL
BVOL
Chevron.RMA.DAL.EntityClasses.LASFileSectionEntity
BIT
CAL
WTBH
Chevron.RMA.DAL.EntityClasses.LASFileSectionEntity
DEPT
GR
DT

AFTER REMOVAL

traceSet.Traces.Count: 9
a1.Count: 13
a2.Count: 0
Chevron.RMA.DAL.EntityClasses.TraceSetEntity
ZDEN
CNCF
CVOL
BVOL
Chevron.RMA.DAL.EntityClasses.LASFileSectionEntity
BIT // still there!
CAL
WTBH
Chevron.RMA.DAL.EntityClasses.LASFileSectionEntity
DEPT
GR
DT

It would appear that it is a) getting removed from parent entity's collection, and b) getting parent FK value nulled out (see original post). But it is also still in that queue after the removal, and I suppose that is why the attempted insert is happening?

I could post the code that originally creates the TraceEntity and adds it to the parent, but I just double-checked that, it is all very straightforward; again, I don't see where any other entity is referenced by this one, or vice versa. Remember that the insert statement doesn't contain fields from any other table (it couldn't; TraceSetID is the only FK in the Traces table).

BTW I'm running 1.0.2005.1 March 31st.

You know, just for kicks, I wonder what would happen if you changed

traceSet.Traces.Add(trace);
//
//
traceSet.Traces.Remove(trace);

to

trace.TraceSet = traceSet;
//
//
trace.TraceSet = null;

Jeff...

The traces get added to the set in one module (file parser), and removed in another (the import wizard). For sure, I could rearrange my code to get around this problem somehow, for example stick all the traces into an ArrayList and then only after the user completes the wizard do I add the desired traces to the parent entity. But I shouldn't have to do that...

After traceSet.Traces.Remove(toDelete[x]);

is the count of traces one less?

If you do: ArrayList collections = traceSet.GetMemberEntityCollections(); collections will contain the Traces collection, does it indeed contain the BIT entity? the graph routine uses that routine to determine the contained entities and which ones to traverse further .

Otis wrote:

After

traceSet.Traces.Remove(toDelete[x]);

is the count of traces one less?

Yes, you can see that in my output:

traceSet.Traces.Count: 9

Otis wrote:

If you do:

ArrayList collections = traceSet.GetMemberEntityCollections();

collections will contain the Traces collection, does it indeed contain the BIT entity?

I get 2 collections, first is the other related entities, the 2nd one has the 9 remaining trace entities; BIT is not in there - which I guess jives with traceSet.Traces.Count == 9 after the removal, not 10.

But as per my previous post, DetermineActionQueues still puts it in the insert queue.

JimFoye wrote:

The traces get added to the set in one module (file parser), and removed in another (the import wizard). For sure, I could rearrange my code to get around this problem somehow, for example stick all the traces into an ArrayList and then only after the user completes the wizard do I add the desired traces to the parent entity. But I shouldn't have to do that...

Yea, I understand. I was just curious to see if that would actually fix it, the idea being to troubleshoot down to the cause. If changing the attachment mechanism resolved the problem, it may tell us something. What, you say? I dunno...

Jeff...

I can't reproduce it:

/// <summary>
/// Tests the insertion of a Customer entity and an VisitingAddress entity + it inserts 2 orders, it creates 3 and removes one before the save.
/// </summary>
[Test]
public void InsertCustomerWithOrdersTest()
{
    DataAccessAdapter adapter = new DataAccessAdapter();

    try
    {
        CustomerEntity newCustomer = EntityCreator.CreateNewCustomer(1);
        newCustomer.TestRunId = _testRunID;
        AddressEntity newAddress = EntityCreator.CreateNewAddress(1);
        newAddress.TestRunId = _testRunID;
        newCustomer.VisitingAddress = newAddress;
        newCustomer.BillingAddress = newAddress;

        // create 3 orders.
        OrderEntity newOrder = new OrderEntity();
        newOrder.OrderDate = DateTime.Now;
        newOrder.TestRunId = _testRunID;
        newCustomer.Orders.Add(newOrder);

        newOrder = new OrderEntity();
        newOrder.OrderDate = DateTime.Now;
        newOrder.TestRunId = _testRunID;
        newCustomer.Orders.Add(newOrder);

        newOrder = new OrderEntity();
        newOrder.OrderDate = DateTime.Now;
        newOrder.TestRunId = _testRunID;
        newCustomer.Orders.Add(newOrder);

        Assert.AreEqual(3, newCustomer.Orders.Count);

        // remove newOrder
        newCustomer.Orders.Remove(newOrder);

        Assert.AreEqual(2, newCustomer.Orders.Count);

        bool result = adapter.SaveEntity(newCustomer, true);
        Assert.AreEqual(true, result);
        Assert.AreEqual(EntityState.Fetched, newAddress.Fields.State);
        Assert.AreEqual(EntityState.Fetched, newCustomer.Fields.State);
        Assert.IsTrue((((int)newAddress.Fields[(int)AddressFieldIndex.AddressId].CurrentValue)>0));
        Assert.IsTrue((((int)newCustomer.Fields[(int)CustomerFieldIndex.CustomerId].CurrentValue)>0));
        int addressID = newAddress.AddressId;
        int customerID = newCustomer.CustomerId;
        Assert.IsTrue((newCustomer.VisitingAddressId==addressID));
        Assert.AreEqual(newAddress, newCustomer.BillingAddress);
        Assert.IsTrue((newAddress==newCustomer.BillingAddress));
        Assert.AreEqual(newAddress.ObjectID, newCustomer.VisitingAddress.ObjectID);
        Assert.AreEqual(newAddress.ObjectID, newCustomer.BillingAddress.ObjectID);

        Assert.AreEqual(2, newCustomer.Orders.Count);

        Assert.AreEqual(newCustomer.CustomerId, ((OrderEntity)newCustomer.Orders[0]).CustomerId);
        Assert.AreEqual(newCustomer.CustomerId, ((OrderEntity)newCustomer.Orders[1]).CustomerId);

        CustomerEntity fetchedCustomer = new CustomerEntity(customerID);
        adapter.FetchEntity(fetchedCustomer);
    }
    finally
    {
        adapter.Dispose();
    }
}

It perfectly removes the 3rd order entity, it inserts just 2, customerid in order isn't nullable, so I don't get phantoms inserted.

You say there are 2 collections in the membercollections arraylist. the second collection, what's in there?

The second collection contains the two LASFileSectionEntity instances that also belong to the TraceSetEntity instance. You can see their types in the output I included on my 4th post. I added this line before the loop to get rid of them

traceSet.LASFileSections.Clear();

and ran the code again. Same results.

I want to point out again that if you look at the insert statement sent to SQL Server, there are no fields that are FKs to other tables. The only possible related table for a TraceEntity is its parent TraceSetEntity. Once I remove it from the parent collection, and its TraceSetID is nulled out, it can't possibly be reside in some other entity's graph.

yes, it can, another entity could have an FK to traceentity (in theory)

As I can't repro it here, could I ask you to run your code with a debug build of the ormsupportclasses and see how the trace-entity ends up in the queue? The routine to examine is:

public void ProduceAdjacencyLists(IEntity2 entityToExamine, Hashtable adjacencyLists, Hashtable recursed)

in ObjectGraphUtils. In there, the entities related to a given entity, as seen from that entity, are examined. IF 'BIT' is seen from a given entity reachable from the root of the graph, you'll notice it there.

Otis wrote:

yes, it can, another entity could have an FK to traceentity (in theory)

Yes you are right, I just meant that Traces table contained no other FKs and so shouldn't be referenced by anything else. However, your comment makes me think of something. TraceEntity is a supertype. This "BIT" trace is actually a NumericTraceEntity. Could this be relevant?

Jim

No that should be irrelevant, also because it's not in the collection (as you tested), so it shouldn't appear in the graph thus also not in the queue.

That's the weird part.

Ok, I've got something that may help clear this up (got to thinking about this more after my last post).

When I create these trace entities, I set their related entities, because I need these in the wizard UI.

The "BIT" trace is actually a NumericTraceEntity, which is a subtype of TraceEntity. A NumericTraceEntity has a property called TraceTypeEntity. As I said, I set this related property before showing the wizard (one of the things the user has to do in the wizard is to correct this if I'm wrong, because it's partially a guess based on what I found in the file).

So I added this code to get the TraceTypeEntity that is attached to the BIT trace prior to removal from its parent, and output its own collection of NumericTraceEntity instances:

                NumericTraceEntity nt = (NumericTraceEntity)toDelete[x];
                TraceTypeEntity ty = nt.TraceType;
                foreach (NumericTraceEntity ntx in ty.NumericTrace)
                    System.Diagnostics.Trace.WriteLine(ntx.Name);

As expected, this TraceTypeEntity does have in its collection of NumericTraceEntity instances the BIT trace which in turn has this type as its own TraceType property.

Now I remove the BIT trace from its parent. We've already seen the parent's collection count drops by one, and it on longer has any reference to the BIT trace. But when I repeat the above code, I still find the BIT trace referenced by that TraceTypeEntity.

So here's what I'm thinking: I create a NumericTraceEntity, assign it a TraceTypeEntity, add it to the parent, remove it from the parent, but when I save, that TraceTypeEntity is somewhere in the graph, and it has a reference to the NumericTraceEntity I've removed, and an insert is attempted for this object?

I realize I'm cross-posting with you this morning.

Sure enough, if I null out the related properties on that trace prior to removal, the problem goes away.

Wow! This is one of those cases where you really have to think about the all the ramifications of working with these entities. I think the behavior here is, strictly speaking, correct. But I think you can see why once I removed the trace from its parent, I thought it was gone forever and would not reappear when I saved the topmost object!

JimFoye wrote:

I realize I'm cross-posting with you this morning.

Sure enough, if I null out the related properties on that trace prior to removal, the problem goes away.

Wow! This is one of those cases where you really have to think about the all the ramifications of working with these entities. I think the behavior here is, strictly speaking, correct. But I think you can see why once I removed the trace from its parent, I thought it was gone forever and would not reappear when I saved the topmost object!

Hah! I knew there was a reference there somewhere. I was bit by this too from a seemingly innocuous object reference a while back. Took me 2 days to figure it out and track it down, so I feel your pain.

Entity graph saves have their challenges, which I think Frans was aware of when he first implemented it. The big problem is that the semantic intention we have when we take certain actions (like entity removals) can't be fully realized with a simple object graph. The problem is that, even in a directed graph, there can be multiple paths to the same object. What we want to execute is: "remove this entity from consideration", but pratically, this would require that all references to the object be broken. This can't be accomplished with a single action now, with a simple "EntityCollection.Remove", so we're left with no simple means of executing what we want to happen, hence your issue.

That being said, I still think the functionality is vital and useful. There are predictable and reproducible ways of avoiding the problems, though the difficulty increases exponentially with the depth of the graph.

Jeff...

It's indeed a sideeffect of having a dual relation (A -> B and B -> A) where just 1 is used (A -> B). It's then often better to hide the opposite side so this won't happen.

it's not the first time this happens to someone, but as said, it's also something which is beneficial in other situations. It's just one of those things which can bite you if you're overlooking a relation in a graph, like in this situation. Glad it's cleared

Ah...instead of worrying about clearing those properties prior to removal, I should just turn off the relation....

Thanks for the help everybody. Good comments, Jeff.

Uh...sorry to bring this back from the dead, but....

I finally got around to cleaning up my code. I wanted to turn off the relationships on one side and then delete from my code where I am setting related properties to null, so as to keep everything tidy and proper.

But when I hide the relationship on one side, I have a new problem. Setting an entity's related entity does not actually set the FK value now. The FK stays 0.

Here I am loading a TraceSetEntity entity from a file and setting its TraceSetType property to a TraceSetTypeEntity with ID of 1 and description of 'Other'. (Here GetTraceSetType() is just a function that finds the requested type in an already loaded collection of TraceSetTypeEntity objects).

        traceSet.TraceSetType = GetTraceSetType(TraceSetType.Other);

If both sides of the relationship are NOT hidden, then this works fine. Not only traceSet.TraceSetType set to an instance of the "Other" TraceSetTypeEntity, but traceSet.TraceSetTypeID will be set to 1. Of course, I want to hide the relationship on the side of TraceSetType, because otherwise if the user cancels the loading of this set after this code, I will have to manually set traceSet.TraceSetType = null (see the first 20 or so posts of this thread).

But, when I do that, the above assignment behaves differently! Now, traceSet.TraceSetType is still set to an instance of the "Other" TraceSetTypeEntity, BUT traceSet.TraceSetTypeID is still 0, not 1!

I'm sure there is a perfectly good explanation for this behavior, and someone just needs to explain it to me.

Assuming I am not imagining things here, regardless of what the explanation is, it seems I have two choices. Either unhide the relationship and restore my code to carefully null out related entities prior to removing from parent collections, OR perhaps do this this kind of thing as necessary?

        traceSet.TraceSetType = GetTraceSetType(TraceSetType.Other);
        traceSet.TraceSetTypeID = traceSet.TraceSetType.TraceSetTypeID;

Man that's fugly!

v1 doesn't sync from PK to FK if the relation from PK side to FK side is hidden. V2 does. V1 couldn't be changed because of this, as it would change behavior of existing code which made it a breaking change. (it also took a lot of code changes, so it wasn't changed mid-1.0.2005.1)

If you're using v2, please let me know. In v1, this issue is unfortunately part of the way it works. The thing is that the reasoning behind it is that if PK -> FK is hidden, the PK doesn't 'see' the FK side and therefore won't sync.

However, as everything is implemented with the observer pattern, it's a little backwards and confusing, so we changed it in v2.