This is sadly a side effect of how things are implemented. 'By design' is a dirty word as it's not really intentional, but not really solveable either.
The issue is that if you do:
order.Customer = GetCustomer(order.CustomerId); // A
order.Customer = GetCustomer(order.CustomerId); // B
on line B the entity instance assigned in line A is dereferenced first. It is done without looking at the instance which might be set. So here it will reset the FK field to null. Then the instance set on line B is assigned as related entity and the FK is synced again, which means a change.
It doesn't keep track of the values the FK field has had in the past, so e.g. you go from 1 to null to 1, it will see it as a change, not as a 'revert to original value. Hence the IsDirty flag being set.
You run into problems with this particular side effect?