I'm using v2.5.7.827 runtime, I found that the RemovedEntitiesTracker is null after deserialization, is this behavior normal?

Yes.

The idea is this: - in the UI or client process you track actions

for remoting: - you then gather the actions and add them to a unit of work - you serialize the unit of work.

for webservices: - you then gather the actions and create messages - you send the messages.

The collection is for tracking actions on the containing collection, so you have it easier when you want to formulate the actions to perform by the unitofwork (as you can add the collection directly to the unit of work).

It would otherwise create a mess because entities would be serialized multiple times.

We are using remoting, but not using unit of work. And we try to handle RemovedEntitiesTracker in SerializeOwnedData() and DeserializeOwnedData().

However, we found that it's impossible, RemovedEntitiesTracker will be reset even we deserialize it in DeserializeOwnedData(). Because the collection is empty at DeserializeOwnedData(), I guess the collection is deserialized after DeserializeOwnedData(), and therefore it reset the value of RemovedEntitiesTracker .

Any workarounds in my situation?

As Frans have explained before, I think you should try to serialize a UoW instead.

Barry wrote:

We are using remoting, but not using unit of work. And we try to handle RemovedEntitiesTracker in SerializeOwnedData() and DeserializeOwnedData().

However, we found that it's impossible, RemovedEntitiesTracker will be reset even we deserialize it in DeserializeOwnedData(). Because the collection is empty at DeserializeOwnedData(), I guess the collection is deserialized after DeserializeOwnedData(), and therefore it reset the value of RemovedEntitiesTracker .

Any workarounds in my situation?

You have overrides of these methods in the EntityCollection<T> class I pressume? If you first call base.SerializeOwnedData(..) and then serialize the tracker collection in CommonEntityBase, you also have to deserialize it in that order in DeserializeOwnedData.

The collection isn't reset anywhere, it's not initialized (so it is null up front) when deserializing.

I override those methods in entity class, the following is the coding I handle RemovedEntitiesTracker, RemovedEntitiesTracker is null after serialization, anything is wrong in my coding?

public partial class SalesOrderEntity
{
    protected override void SerializeOwnedData(SerializationWriter writer, object context)
    {
        base.SerializeOwnedData(writer, context);
        writer.Write(this.Items.RemovedEntitiesTracker);
    }
    
    protected override void DeserializeOwnedData(SerializationReader reader, object context)
    {
        base.DeserializeOwnedData(reader, context);
        this.Items.RemovedEntitiesTracker = (EntityCollection<ItemEntity>)reader.ReadObject(); // this.Items.Count is 0, it should have some entities
    }
}

Will check it out. It might be you need to call different methods in the serializers, I'll do a test.

I think you should call WriteObject(collection) instead of Write(collection). WriteObject will then use a binaryformatter IF necessary. This is then correctly handled by the ReadObject again.

It doesn't fast serialize the tracked entities, which isn't really a problem as they're chopped off from the graphs anyway (so they won't pull related entities into the data stream)

I've modified my code, but it still does not work.

The following is the new code, and commented lines are the data I captured at the break points. I can deserialize RemovedEntitiesTracker at DeserializeOwnedData(), but it is reset to null after the whole deserialization process is completed.

public void TestSerialization(SalesOrderEntity salesOrder)
{
    SalesOrderEntity clonedSalesOrder = null;
    BinaryFormatter formatter = new BinaryFormatter();
    
    // salesOrder.Items.Count = 10
    // salesOrder.Items.RemovedEntitiesTracker.Count = 2

    using (MemoryStream memStream = new MemoryStream())
    {
        formatter.Serialize(memStream, salesOrder);
        memStream.Seek(0, SeekOrigin.Begin);
        clonedSalesOrder = (SalesOrderEntity)formatter.Deserialize(memStream);
    }
    
    // clonedSalesOrder.Items.Count = 10
    // clonedSalesOrder.Items.RemovedEntitiesTracker = null
}

public partial class SalesOrderEntity
{
    protected override void SerializeOwnedData(SerializationWriter writer, object context)
    {
        base.SerializeOwnedData(writer, context);
        // this.Items.Count = 10
        // this.Items.RemovedEntitiesTracker.Count = 2
        writer.WriteObject(this.Items.RemovedEntitiesTracker);
    }
    
    protected override void DeserializeOwnedData(SerializationReader reader, object context)
    {
        base.DeserializeOwnedData(reader, context);
        // this.Items.Count = 0
        // this.Items.RemovedEntitiesTracker = null;
        this.Items.RemovedEntitiesTracker = (EntityCollection<ItemEntity>)reader.ReadObject();
        // this.Items.Count = 0
        // this.Items.RemovedEntitiesTracker.Count = 2
    }
}

Barry wrote:

I've modified my code, but it still does not work.

I think I can explain why you are seeing 'correct' data in DeserializeOwnedData (I don't have access to the source code at the moment): When you set your breakpoint in your overridden DeserializeOwnedData method and inspect this.Items, LLCoolJ is lazy-instantiating a new collection, hence the Count==0. It then goes on to deserialize the RemovedEntitiesTracker and you get your 2 entities back as expected.

However, FastSerialization then goes on to recreate the member collections according to the entity's flags and as part of this creates the required member collections. Thus the collection which was lazy-instantiated is then overwritten with a new one. This is then populated with the 10 entities but RemovedEntitiesTracker is null.

There are a couple of issues here: 1) The SerializeOwnedData/DeserializeOwnedData methods were designed to allow fast serialization of private additional data on the object itself - the intention was really for primitive values, specifically not data/references to/from another object (even if that object is owned by the parent)

2) The RemovedEntitiesTracker could be considered 'owned' by the member collection itself and it is there that it would ideally be better serialized. But even then, 'ownership' could be ambiguous because it isn't strictly private data since the object is assigned to the collection.

I think Frans mentioned elsewhere that he envisaged the RemovedEntitiesTracker being used with the UnitOfWork object so it is basically ignored for FastSerialization as you have found.

There is another forum thread discussing why it might be advantageous to be able to let the member collection decide whether or not it need serializing (rather than the current situation which only serializes it if has a non-zero count). Maybe the next version release could incorporate this and then you could move your custom SerializeOwnedData/DeserializeOwnedData code to the collection class being used for a RemovedEntitiesTracker .

Cheers Simon

In your test, you don't set the Fast serialization setting to 'Fast', is that perhaps the problem? (I haven't written a unittest yet to see what's the prob, so I'm guessing here)

I think Simon hits the nail on the head.

I also think that the collection is better serialized in code inside a partial class of EntityCollection<T> (which also has SerializeOwnedData() etc.) One caveat there is that if the collection is empty, the whole collection is ignored (however there IS data).

Serializing the tracker collection is an option, it could be added in internal code (add flag to flag set of collection, let the collection serializer call SerializeOwnedData and let it serialize itself, etc. etc. but that will also take a lot of extra testing), however extra caution is then required to have exactly the same BUILD at either end. If there's something I don't like, it's having features depend on build numbers as it's always leading to trouble sooner or later. Also adding it NOW causes a lot of work, and I'm sorry to say it but after more than 2 months of beta-testing I'm not really fond of adding new things, there was plenty of time before.

Otis wrote:

Also adding it NOW causes a lot of work

I agree with you - thats why I mentioned "next version release" rather than "next build"

Wot, no "LLBLGEN Pro v2.6 feature discussions feed" created yet?? (I have a couple of suggestions brewing for optimization...)

Cheers Simon

simmotech wrote:

Otis wrote:

Also adding it NOW causes a lot of work

I agree with you - thats why I mentioned "next version release" rather than "next build"

heh But, the next version will be v3.

Wot, no "LLBLGEN Pro v2.6 feature discussions feed" created yet?? (I have a couple of suggestions brewing for optimization...) Cheers Simon

Uhoh . V3 will have the focus on a new designer, so no v2.6 is planned. The thing is that you always have to have a beta of more than a month, so to do a v2.6, it's not really that efficient, time-wise.

Though any optimization ideas to add for v3 are always welcome of course. I'll start working on Linq for llblgen pro in a week or so, and after that will work on v3.

I try to serialize the tracker by overriding methods in EntityCollection<TEntity>, it only works if the collection has data.

Any workarounds? or I better not to use the tracker and handle the removed entities by myself?

Barry wrote:

I try to serialize the tracker by overriding methods in EntityCollection<TEntity>, it only works if the collection has data.

Any workarounds? or I better not to use the tracker and handle the removed entities by myself?

What you ran into is exactly this thread: http://www.llblgen.com/TinyForum/Messages.aspx?ThreadID=11007

It's really unfortunate though... If it was related to client side stuff only it wouldn't be a big problem, but it is related to code which requires the same version on both sides, and that thus depends on build numbers.

Another problem is that if the trackers are serialized by default, it's extra data for the users who use unitofworks. This isn't that useful.

What you could do is add the tracker to a unitofwork for deletion, add the entities to process to the unitofwork for save and send that over the wire. I'm not sure if this meets your architecture.

Unitofwork is not suitable for our application, I'll continue to use my old ways to handle removed entities at this moment, I hope the next version of LLBLGen would improve it.

Barry wrote:

Unitofwork is not suitable for our application, I'll continue to use my old ways to handle removed entities at this moment, I hope the next version of LLBLGen would improve it.

We're currently discussing to add this change to the API, as it is a change we all benefit from and we're quite early after release. This means that it's very unlikely that there are already production systems with v2.5 out there with remoting AND the developers run into problems where the build numbers differ.

Otis wrote:

Barry wrote:

I try to serialize the tracker by overriding methods in EntityCollection<TEntity>, it only works if the collection has data.

Any workarounds? or I better not to use the tracker and handle the removed entities by myself?

What you ran into is exactly this thread: http://www.llblgen.com/TinyForum/Messages.aspx?ThreadID=11007

It's really unfortunate though... If it was related to client side stuff only it wouldn't be a big problem, but it is related to code which requires the same version on both sides, and that thus depends on build numbers.

Another problem is that if the trackers are serialized by default, it's extra data for the users who use unitofworks. This isn't that useful.

What you could do is add the tracker to a unitofwork for deletion, add the entities to process to the unitofwork for save and send that over the wire. I'm not sure if this meets your architecture.

I've been thinking a little about this problem and think I have a solution - albeit in two parts (I've not tested this BTW - just looked at the source so there may be a problem I haven't foreseen but I'm reasonably confident )

Firstly, if the next build could make one small library change, we could solve the problem of letting developers have an opportunity to serialize the tracker themselves manually.

The code changes is just to remove the " && memberEntityCollections[i].Count != 0" condition in FastSerializer.WriteEntityMemberCollections.

This change would have no impact on existing users except that a non-null but empty member collection would take a few extra bytes in the stream (I really mean a few - typically 2 for the collection flags, 1 for a string token plus the bytes required to store the string field name once). Build mismatches are not a problem because only the 'what' has changed, not the 'how'.

A change would also need to be made in the template (only for the developers wanting to manually serialize the tracker - not for general release) that creates the HasPopulatedMemberEntityCollections() method - just change so that the "Count > 0" condition is removed (or maybe leave it and add "&& _<membercoll>.RemovedEntitiesTracker != null && _<membercoll>.RemovedEntitiesTracker.Count > 0") However, since the collection is now being serialized, mikeg and Barry now should have control in SerializeOwnedData and can deal with the tracker. This would require the same build on both client and server but I'm sure that won't be a problem for mikeg and Barry anyway. (BTW rather than using the WriteObject method as mentioned in this thread, you could probably create a new FastSerializer, serialize the tracker and then store the resulting byte[] into the stream of the passed SerializationWriter - nested FastSerialization if you will!)

The second part suggested is for the next version. This would involve changing the interface to add a bool property allowing the object to say whether it should be serialized or not - default implementation for EntityCollectionBase2 would then be "return Count != 0" thus getting back the saving on the few extra bytes from part one. Plus template changes as mentioned above.

Then there is the issue of whether the tracker should be serialized by default or not. My opinion is that it probably should since I believe that serialization/deserialization should restore the entire object (with the exception of UnitOfWork since it has a flag giving the developer an option).

I understand the point about the tracker data being serialized twice if it is involved with a UnitOfWork but I believe this is easily got around by making the tracker an integral part of the collection - ie a bit flag to say whether one is present or not and adding a bit of code to ReferencedEntityMap to identify the tracker's entities as being referenced. That way, when incorporated into a UnitOfWork, only one instance will be stored with two references to it but if just in the collection, they are serialized inline.

Cheers Simon

Otis wrote:

Barry wrote:

Unitofwork is not suitable for our application, I'll continue to use my old ways to handle removed entities at this moment, I hope the next version of LLBLGen would improve it.

We're currently discussing to add this change to the API, as it is a change we all benefit from and we're quite early after release. This means that it's very unlikely that there are already production systems with v2.5 out there with remoting AND the developers run into problems where the build numbers differ.

I didn't see this message when I wrote my message BTW.

Cheers Simon

simmotech wrote:

The code changes is just to remove the " && memberEntityCollections[i].Count != 0" condition in FastSerializer.WriteEntityMemberCollections.

We're going to look into this and make the change if this can be done without problems.

After discussing this, we decided that there will be a v2.6. It will only contain one added feature, LINQ support, and any fixes that involve API changes.

It will only contain one added feature, LINQ support, and any fixes that involve API changes.

So this implies support for dotnet 3.5 and maybe vs2008 beta2 in v2.6?

arschr wrote:

It will only contain one added feature, LINQ support, and any fixes that involve API changes.

So this implies support for dotnet 3.5 and maybe vs2008 beta2 in v2.6?

Everything necessary to make llblgen pro usable with Linq (and thus .net 3.5). This will likely include one or more templates, a separate dll and perhaps a special build of the ormsupportclasses dll, but that's not certain at this point. For the rest, it's already usable today on beta2.

Simon: I implemented the stuff you suggested for this build except for the test for a remove tracker in the templates, as that would make it limited, because if you had other data, you then would run into hte same problem.

Ok, here's my test:

[Test]
public void FastSerializationWithTrackerCollection()
{
    SerializationHelper.Optimization = SerializationOptimization.Fast;
    CustomerEntity c = new CustomerEntity("CHOPS");
    PrefetchPath2 path = new PrefetchPath2(EntityType.CustomerEntity);
    path.Add(CustomerEntity.PrefetchPathOrders);
    using(DataAccessAdapter adapter = new DataAccessAdapter())
    {
        Assert.IsTrue(adapter.FetchEntity(c, path));
    }
    Assert.AreEqual(8, c.Orders.Count);
    Assert.IsNull(c.Orders.RemovedEntitiesTracker);

    // first simple serialization without tracker.
    MemoryStream stream = new MemoryStream();
    BinaryFormatter formatter = new BinaryFormatter();
    formatter.Serialize(stream, c);
    stream.Seek(0, SeekOrigin.Begin);
    CustomerEntity deserializedC = (CustomerEntity)formatter.Deserialize(stream);
    stream.Close();

    Assert.AreEqual(8, deserializedC.Orders.Count);
    Assert.IsNull(deserializedC.Orders.RemovedEntitiesTracker);

    // add tracker to c.Orders
    c.Orders.RemovedEntitiesTracker = new EntityCollection<OrderEntity>();
    List<OrderEntity> toRemove = new List<OrderEntity>(c.Orders);
    Assert.AreEqual(8, toRemove.Count);
    foreach(OrderEntity o in toRemove)
    {
        c.Orders.Remove(o);
    }

    Assert.AreEqual(0, c.Orders.Count);
    Assert.AreEqual(8, c.Orders.RemovedEntitiesTracker.Count);

    foreach(OrderEntity o in toRemove)
    {
        Assert.IsTrue(c.Orders.RemovedEntitiesTracker.Contains(o));
    }

    stream = new MemoryStream();
    formatter.Serialize(stream, c);
    stream.Seek(0, SeekOrigin.Begin);
    deserializedC = (CustomerEntity)formatter.Deserialize(stream);
    stream.Close();
    Assert.AreEqual(0, deserializedC.Orders.Count);
    Assert.IsNotNull(deserializedC.Orders.RemovedEntitiesTracker);
    Assert.AreEqual(8, deserializedC.Orders.RemovedEntitiesTracker.Count);

    SerializationHelper.Optimization = SerializationOptimization.None;
}

And here's my extension to EntityCollection<T> (partial class in generated code)

public partial class EntityCollection<TEntity> : EntityCollectionBase2<TEntity>
    where TEntity : EntityBase2, IEntity2
{
    /// <summary>
    /// Method which restores owned data - i.e. considered private to this collection
    /// and not shared with any external object
    /// </summary>
    /// <param name="writer">SerializationWriter</param>
    /// <param name="context">The serialization flags (previously constructed)</param>
    protected override void SerializeOwnedData(SerializationWriter writer, object context)
    {
        base.SerializeOwnedData(writer, context);
        byte[] trackerData = new byte[0];
        if((this.RemovedEntitiesTracker != null) && (this.RemovedEntitiesTracker.Count>0))
        {
            // serialize tracker
            FastSerializer serializer = new FastSerializer();
            trackerData = serializer.Serialize(this.RemovedEntitiesTracker).ToArray();
        }
        writer.Write(trackerData);
    }

    /// <summary>
    /// Method which restores owned data - i.e. considered private to this entity
    /// and not shared with any external object
    /// </summary>
    /// <param name="reader">The SerializationReader containing the serialized data</param>
    /// <param name="context">The serialization flags (previously read)</param>
    protected override void DeserializeOwnedData(SerializationReader reader, object context)
    {
        base.DeserializeOwnedData(reader, context);
        byte[] trackerData = reader.ReadByteArray();
        if(trackerData.Length > 0)
        {
            // tracker data read, deserialize it to a real tracker collection
            EntityCollection<TEntity> trackerCollection = new EntityCollection<TEntity>();
            FastDeserializer deserializer = new FastDeserializer();
            deserializer.Deserialize(trackerData, trackerCollection);
            this.RemovedEntitiesTracker = trackerCollection;
        }
    }
}

This is available in the next build, released later today

Otis wrote:

Simon: I implemented the stuff you suggested for this build except for the test for a remove tracker in the templates, as that would make it limited, because if you had other data, you then would run into hte same problem.

Thats excellent (and the template stuff was only really for mikeg and Barry anyway)

simmotech wrote:

Otis wrote:

Simon: I implemented the stuff you suggested for this build except for the test for a remove tracker in the templates, as that would make it limited, because if you had other data, you then would run into hte same problem.

Thats excellent (and the template stuff was only really for mikeg and Barry anyway)

So, there is no test for collection serialization? Are all collections serialized regardless of their count?