Extra research for this is planned, but when the results of that are implemented is not clear at this point.

Otis wrote:

Extra research for this is planned, but when the results of that are implemented is not clear at this point.

Any news on this?

I ran into this behavior as well a couple of times, what I would prefer is to have an option that always throws an exception if client side filtering might occur.

Obviously having the possibility to navigate to page 3500 of an active data set might indicate other design issues in your application but especially in automated processes running on very large data sets this could be a showstopper/app killer if a small change in ordering can enable this behavior.

Thanks,

Wiebe

Client side paging might be a bit slower, but it's often not doable to rewrite it in a server-side paging query, as sorting is essential for paging. Using a different sorting isn't going to give you the same data.

Throwing an exception when client-side paging occurs... is that really what you want? It's very hard to test for that, as you have to test your paging queries (which might be triggered through databinding, so no code written!) beyond page 1. Every single one of them.

IMHO far better is preventing a user to page to page 100+, as it makes no sense.

Otis wrote:

IMHO far better is preventing a user to page to page 100+, as it makes no sense.

Uhmmm I think that was what I said? At least that's what I meant.

The point was, in a current app we have many batch processes on large data sets, if I want to export 1 mil records in page sizes of 1000 I would definitely like an exception if it would end up trying to retrieve 900k+ records on every page at the end of the operation.

I'd prefer explicit errors any time over this potentially dangerous implicit behavior.

wtijsma wrote:

Otis wrote:

IMHO far better is preventing a user to page to page 100+, as it makes no sense.

Uhmmm I think that was what I said? At least that's what I meant.

The point was, in a current app we have many batch processes on large data sets, if I want to export 1 mil records in page sizes of 1000 I would definitely like an exception if it would end up trying to retrieve 900k+ records on every page at the end of the operation.

I'd prefer explicit errors any time over this potentially dangerous implicit behavior.

But how are you going to solve this? What if there's an exception which says 'switched to client side paging, aborted' ? Then the only way to solve it is to change the sorting but that might not work (as you already had a sorting outside the projection) or better: you might have a blob/ntext field in the projection which makes distinct not usable.

In batch processes, isn't it better to do this: - You create a custom projector. Use DataProjectorToEntityCollection as your starting point (the sourcecode that is). Instead of adding a newly read entity to the collection, you send it to the code which saves the entity to the file/ export destination - You fetch the data as a projection. Distinct filtering is build in, so you get real unique rows from it. Use your custom projector with the projection. Every entity fetched from the database is then passed to the export destination (I'd use a different thread for that, so pass the object to the other thread. This is safe as nothing is done with the entity by the fetch code)

No paging needed, and you have a forward only cursor over the data: you read the data and at the same time you export the data.

One caveat: if you use inheritance, this doesn't work as the projector doesn't know about inheritance. In v3.5 this is fixable however as the code to materialize entities from a row is then 1 call and doable from outside code.

Does this solve your problem?

Otis wrote:

But how are you going to solve this? What if there's an exception which says 'switched to client side paging, aborted' ? Then the only way to solve it is to change the sorting but that might not work (as you already had a sorting outside the projection) or better: you might have a blob/ntext field in the projection which makes distinct not usable.

In batch processes, isn't it better to do this: - You create a custom projector. Use DataProjectorToEntityCollection as your starting point (the sourcecode that is). Instead of adding a newly read entity to the collection, you send it to the code which saves the entity to the file/ export destination - You fetch the data as a projection. Distinct filtering is build in, so you get real unique rows from it. Use your custom projector with the projection. Every entity fetched from the database is then passed to the export destination (I'd use a different thread for that, so pass the object to the other thread. This is safe as nothing is done with the entity by the fetch code)

No paging needed, and you have a forward only cursor over the data: you read the data and at the same time you export the data.

One caveat: if you use inheritance, this doesn't work as the projector doesn't know about inheritance. In v3.5 this is fixable however as the code to materialize entities from a row is then 1 call and doable from outside code.

Does this solve your problem?

Well, my point here is not how to solve it, because that's not the problem (but thanks for the leads, will definitely have a look at those!), but merely I think that the framework should indicate that there IS a problem and you have to find a different solution, instead of silently failing.

Thanks!

W

It doesn't fail, it switches to another approach to fulfill your request. Other O/R mappers fail, we don't (as you get the results you asked for, not duplicate rows like with NHibernate or an exception as with EF).

But I see your point. Unfortunately the features for v3.5 are selected and the list is frozen. However I'll give you pointers to where to alter the sourcecode of the support classes to add the exception throwing, so you can add it now without waiting for a version beyond 3.5. This will be on monday however.

Otis wrote:

It doesn't fail, it switches to another approach to fulfill your request. Other O/R mappers fail, we don't (as you get the results you asked for, not duplicate rows like with NHibernate or an exception as with EF).

But I see your point. Unfortunately the features for v3.5 are selected and the list is frozen. However I'll give you pointers to where to alter the sourcecode of the support classes to add the exception throwing, so you can add it now without waiting for a version beyond 3.5. This will be on monday however.

Yes it's more like Silently succeeding' or 'Graceful degradation'...

Thanks!

n.b. What and when can we expect 3.5? The road map on the side is a bit limited

-w

wtijsma wrote:

Otis wrote:

It doesn't fail, it switches to another approach to fulfill your request. Other O/R mappers fail, we don't (as you get the results you asked for, not duplicate rows like with NHibernate or an exception as with EF).

But I see your point. Unfortunately the features for v3.5 are selected and the list is frozen. However I'll give you pointers to where to alter the sourcecode of the support classes to add the exception throwing, so you can add it now without waiting for a version beyond 3.5. This will be on monday however.

Yes it's more like Silently succeeding' or 'Graceful degradation'...

Paging always comes at a cost, even server-side paging. Client-side paging isn't the most efficient way to do it, but at least it works. You make it sound like it's worse than having the nhibernate/EF way of doing things, namely return duplicates, which is utterly wrong, as you can end up with a page with 1 entity, as all are duplicates.

I find it strange that people prefer that incorrect way of dealing with data over a system which actually works. I know for your particular situation it's not handy, but for almost all situations it is. (paging on access is only possible on the client btw).

n.b. What and when can we expect 3.5? The road map on the side is a bit limited -w

We're aiming for the end of january 2012. What you can expect? lots of stuff - all settings/configuration for a project now in 1 dialog - rule based attribute assignment, so you can define an attribute on the project level and define a rule (with expressions) on which element to add that attribute, e.g. only string based fields of length 42 - edmx import - queryspec - wcf data services support - built-in type converters for many types (so most of them work out of the box) - refactorings internally so selfservicing and adapter share way more code - entity return in projections in linq (from c in metadata.Customer select new { c}; ) - quickstart guides inside the designer, which are easy to follow walkthroughs like creating a project, adding meta-data, creating entities etc. - .NET 4.0 client profile compatible runtime - COM+ classes made mandatory deprecated (so you can't use them anymore, use system.transactions) - manipulate settings for elements in bulk (so change a code gen setting X for the entities you select in 1 click). - and many more smaller things, like easier to use code gen dialog. We focused on making things easier to use in v3.5.

About your problem at hand. In the ORM Support Classes, Query\RetrievalQuery.cs, line 208, the setter of RequiresClientSidePaging. if that flag is set to true, throw the exception there. This should catch the situation where client-side paging is performed at runtime. Not recommended as I said above, but if you test it well enough, you should be ok. We've added a workitem for adding the setting you requested in the future, but we don't know when/if it will be added (not v3.5, that's frozen)

Otis wrote:

I find it strange that people prefer that incorrect way of dealing with data over a system which actually works. I know for your particular situation it's not handy, but for almost all situations it is. (paging on access is only possible on the client btw).

Ah don't get me wrong, I definitely like your way of paging in most cases, but I'd prefer having to enable it explicitly (maybe by mentioning 'you have to enable [this] option' in the thrown exception) so people know what's going and are forced to think about this choice.

Looking forward to 3.5!

Thanks,

-W