Note: Removing an entity from a collection doesn't necessarily mean it should be deleted from the databse.
In your case you may maintain a parallel collection only for deleted entities, so when you remove an entity from the main collection, add it to the to-Be-Deleted collection, and in your IsDirty method, you may check on the count of the to-Be-Deleted collection.
You may also a Unit of Work.