Performance comparison LLBLGen - dOOdads

Posts   
 
    
flo1227
User
Posts: 8
Joined: 16-Sep-2004
# Posted on: 05-Sep-2006 18:18:58   

Hi Otis,

for a couple of projects I'm evaluating a change from dOOdads to LLBLGen due to the richer functionality. However I'm not happy with the performance of LLBLGen. For a comparison test I read a) 1000 and b) 10000 tuples from the database into collections of the respective business objects of dOOdads and LLBLGen and measure the runtime.

For any amount of data, dOOdads seems to be 5 to 7 times faster than LLBLGen. I know they are using DataReader on the Load() method. There is a DataReader functionality in LLBLGen 2.0 as well but it doesnt seem to be as convenient to use.

Is there any way to gain the speed of dOOdads with a LLBLGen-generated data access layer?

Best regards, Florian

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 05-Sep-2006 18:44:16   

I doubt it will be 5 times faster, simply because my performance tests show that LLBLGen Pro entity fetches have an on-par speed or faster than nhibernate fetches. (for example fetching 50,000 entities with strings, guids etc.). So you want to sell me the idea doodads are 5 times faster than nhibernate too?

Also, doesn't doodads use datatables, so they simply fetch a datatable? And how did you test? First a query using llblgen pro , loading all the client dlls, the db tables into memory, query caches in the db etc, and then you ran the doodads stuff?

(edit) they use a datatable internally. So a set of employees is simply a datatable and an employee is a datarow

A datatable has way less overhead than an entitycollection set. The reason is that a datatable runs into the one time overhead of setting up the columns. Every next row is simply pushing an object[] into the datatable's row list.

This also means that you don't have separate objects. Though I still doubt it's 5 times faster.

Frans Bouma | Lead developer LLBLGen Pro
mikeg22
User
Posts: 411
Joined: 30-Jun-2005
# Posted on: 05-Sep-2006 21:40:24   

The performance diffence could have been because of serialization. Datatables (datasets?) serialize much faster than LLBLGEN graphs.

flo1227
User
Posts: 8
Joined: 16-Sep-2004
# Posted on: 06-Sep-2006 09:15:32   

My comparison works as follows:


        static void Main(string[] args)
        {
            Stopwatch sw = new Stopwatch();

            Console.WriteLine("Start dOOdads...");
            sw.Start();
            TestdOOdads();
            sw.Stop();
            Console.WriteLine("Elapsed time: " + sw.GetElapsedTimeSpan().ToString());

            sw.Reset();

            Console.WriteLine("Start LLBLGen...");
            sw.Start();
            TestLLBLGen();
            sw.Stop();
            Console.WriteLine("Elapsed time: " + sw.GetElapsedTimeSpan().ToString());
        }

        static void TestLLBLGen()
        {
            CustomerCollection c = new CustomerCollection();
            
            // Filter
            PredicateExpression filter = new PredicateExpression(CustomerFields.Country == "NL");
            c.GetMulti(filter, 10000);
       }

        static void TestdOOdads()
        {
            Customers c = new Customers();

            // Filter
            c.Where.Country.Value = "NL";
            c.Where.Country.Operator = WhereParameter.Operand.Like;

            c.Query.Top = 10000;
            c.Query.Load();
        }

The result is as follows:


Start dOOdads...
1000 records found.
Elapsed time: 00:00:00.1490000
Start LLBLGen...
1000 records found.
Elapsed time: 00:00:00.8110000

Whether to start LLBLGen or dOOdads first does not make any difference. If you have any suggestion how to avoid overhead in case I just need a flat table without relations, please let me know.

Btw: is serialization beeing done before I explicitly request that (e.g. by WriteXml())?

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 06-Sep-2006 09:41:11   

If you just need a flat table and you don't need entities, just a table, use a dynamic list or typedlist. You can also do a projection fetch onto a datatable or custom classes if you want (llblgen pro v2).

There's no serialization done in the code you posted.

Frans Bouma | Lead developer LLBLGen Pro
mihies avatar
mihies
User
Posts: 800
Joined: 29-Jan-2006
# Posted on: 06-Sep-2006 09:43:01   

Hi flo1227,

You are not testing correctly IMO. You should run this code once at least and stopwatch second iteration. That would eliminate JITing and assembly loading. Also, a better testing approach would be using a performance profiler that measures time spent in methods (so one can eliminate connection time for example).

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 06-Sep-2006 11:03:53   

Good point, the reading of the config file alone takes 0.3sec initially (which is a hit you always will have once in your llblgen pro application).

I also see you use a compare value predicate query in llblgen pro and a like query in doodads?

(edit), I can't get good testresults from my benchmark database right now, as the server is backupped at the moment. A testrun suggested that the entitycollection fetch was actually faster than a datatable fetch, so I think I've to wait till the backup is done. Stay tuned.

Frans Bouma | Lead developer LLBLGen Pro
flo1227
User
Posts: 8
Joined: 16-Sep-2004
# Posted on: 06-Sep-2006 11:51:57   

Hi Otis,

I tested the TypedList and it performs way better than the EntityCollection, so many thanks for the hint!

The comparison might not be perfect, I didnt use a config file though, so that can't have an impact. I also run the whole test in a loop of several queries, so the dll loading time should not affect the result as well. I guess it is just the overhead of the object model of the EntityCollection.

The TypedList is only 10% slower than the dOOdads object which is a negligible value.

Florian

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 06-Sep-2006 12:19:55   

flo1227 wrote:

Hi Otis,

I tested the TypedList and it performs way better than the EntityCollection, so many thanks for the hint!

The comparison might not be perfect, I didnt use a config file though, so that can't have an impact.

Where does the connectionstring come from then? wink LLBLGen Pro always reads from the configfile in the static constructor of the DQE and ormsupportclasses, so tracelisteners and name overwriting settings can be setup IF available.

I also run the whole test in a loop of several queries, so the dll loading time should not affect the result as well. I guess it is just the overhead of the object model of the EntityCollection.

The TypedList is only 10% slower than the dOOdads object which is a negligible value. Florian

I'll post my test results very soon.

Frans Bouma | Lead developer LLBLGen Pro
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 06-Sep-2006 12:22:35   

Ok, DEBUG build tests (release build runtimes)

.NET 2.0: results:

Reading 10000 rows into a datatable took 00:00:01.0316394 Reading 10000 rows into a datatable took 00:00:01.0003776 Reading 10000 rows into a datatable took 00:00:01.0316394 Reading 10000 rows into a datatable took 00:00:01.0003776 Reading 10000 rows into a datatable took 00:00:01.0160085 Reading 10000 rows into a datatable took 00:00:01.0003776 Reading 10000 entities into an entity collection took 00:00:01.7662917 Reading 10000 entities into an entity collection took 00:00:01.2192102 Reading 10000 entities into an entity collection took 00:00:01.1410557 Reading 10000 entities into an entity collection took 00:00:01.0785321 Reading 10000 entities into an entity collection took 00:00:01.1097939 Reading 10000 entities into a collection of custom classes took 00:00:01.6725063 Reading 10000 entities into a collection of custom classes took 00:00:01.4693046 Reading 10000 entities into a collection of custom classes took 00:00:01.4380428 Reading 10000 entities into a collection of custom classes took 00:00:01.4224119 Reading 10000 entities into a collection of custom classes took 00:00:01.4067810

Code:


class Program
{
    private const int Amount = 10000;

    static void Main( string[] args )
    {
        DataTableBencher dtBencher = new DataTableBencher();
        dtBencher.Bench( Program.Amount );  // dummy, to get sqlserver fired up.
        int runs = 5;

        for(int i = 0; i < runs; i++)
        {
            dtBencher = new DataTableBencher();
            dtBencher.Bench(Program.Amount);
        }

        for( int i = 0; i < runs; i++ )
        {
            AdapterBencher aBencher = new AdapterBencher();
            aBencher.Bench( Program.Amount );
        }

        for( int i = 0; i < runs; i++ )
        {
            AdapterProjectorBencher apBencher = new AdapterProjectorBencher();
            apBencher.Bench( Program.Amount );
        }

        Console.WriteLine( "Press enter to exit" );
        Console.ReadLine();
    }
}


public class DataTableBencher
{
    public void Bench(int amount)
    {
        DataTable toFill = new DataTable();
        SqlConnection con = new SqlConnection("data source=nerd;initial catalog=BenchmarkData;integrated security=SSPI;persist security info=False;packet size=4096");
        SqlCommand cmd = new SqlCommand( string.Format( "SELECT TOP {0} * FROM RandomData", amount ), con );
        SqlDataAdapter adapter = new SqlDataAdapter( cmd );
        DateTime start = DateTime.Now;
        adapter.Fill( toFill );
        DateTime end = DateTime.Now;
        Console.WriteLine("Reading {0} rows into a datatable took {1}", toFill.Rows.Count, (end-start));
    }
}


public class AdapterBencher
{
    public void Bench(int amount)
    {
        EntityCollection<RandomDataEntity> randomData = new EntityCollection<RandomDataEntity>( new RandomDataEntityFactory() );
        randomData.Capacity = amount;
        DateTime start = DateTime.Now;
        using( DataAccessAdapter adapter = new DataAccessAdapter( ) )
        {
            adapter.FetchEntityCollection( randomData, null, amount );
        }
        DateTime end = DateTime.Now;
        Console.WriteLine( "Reading {0} entities into an entity collection took {1}", randomData.Count, (end - start) );
        //Console.WriteLine( "Press enter to exit routine." );
        //Console.ReadLine();
    }
}


public class AdapterProjectorBencher
{
    public void Bench( int amount )
    {
        List<RandomDataClass> randomData = new List<RandomDataClass>(amount);
        List<IDataValueProjector> valueProjectors = new List<IDataValueProjector>();
        valueProjectors.Add( new DataValueProjector( "Id", 0, typeof( long ) ) );
        valueProjectors.Add( new DataValueProjector( "GuidVal", 1, typeof( Guid ) ) );
        valueProjectors.Add( new DataValueProjector( "NullVal", 2, typeof( string) ) );
        valueProjectors.Add( new DataValueProjector( "IntVal", 3, typeof( int  ) ) );
        valueProjectors.Add( new DataValueProjector( "Varchar1Val", 4, typeof( string ) ) );
        valueProjectors.Add( new DataValueProjector( "Varchar2Val",5, typeof( string ) ) );
        valueProjectors.Add( new DataValueProjector( "ShortVal", 6, typeof( short ) ) );
        valueProjectors.Add( new DataValueProjector( "DateVal", 7, typeof( DateTime ) ) );
        valueProjectors.Add( new DataValueProjector( "Dummy1", 8, typeof( int ) ) );
        valueProjectors.Add( new DataValueProjector( "Dummy2", 9, typeof( int) ) );
        valueProjectors.Add( new DataValueProjector( "Dummy3", 10, typeof( int ) ) );
        valueProjectors.Add( new DataValueProjector( "Dummy4", 11, typeof( string ) ) );

        DataProjectorToCustomClass<RandomDataClass> projector = new DataProjectorToCustomClass<RandomDataClass>( randomData );
        IEntityFields2 fields = new RandomDataEntityFactory().CreateFields();
        DateTime start = DateTime.Now;
        using( DataAccessAdapter adapter = new DataAccessAdapter( ) )
        {
            adapter.FetchProjection(valueProjectors, projector, fields, null, amount, true);
        }
        DateTime end = DateTime.Now;
        Console.WriteLine( "Reading {0} entities into a collection of custom classes took {1}", randomData.Count, (end - start) );
        //Console.WriteLine( "Press enter to exit routine." );
        //Console.ReadLine();
    }
}


/// <summary>
/// Summary description for RandomDataClass.
/// </summary>
public class RandomDataClass
{
    #region Class Member Declarations
    private string _nullVal, _varchar1Val, _varchar2Val, _dummy4;
    private int _intVal, _dummy1, _dummy2, _dummy3;
    private long _id;
    private short _shortVal;
    private DateTime _dateVal;
    private Guid _guidVal;
    #endregion


    public RandomDataClass()
    {
    }

/// ... properties declarations, snipped.

}


RELEASE BUILD results:

Reading 10000 rows into a datatable took 00:00:01.0155795 Reading 10000 rows into a datatable took 00:00:00.9999552 Reading 10000 rows into a datatable took 00:00:01.0155795 Reading 10000 rows into a datatable took 00:00:01.0155795 Reading 10000 rows into a datatable took 00:00:00.9999552 Reading 10000 rows into a datatable took 00:00:00.9999552 Reading 10000 entities into an entity collection took 00:00:01.3124412 Reading 10000 entities into an entity collection took 00:00:01.2030711 Reading 10000 entities into an entity collection took 00:00:01.1405739 Reading 10000 entities into an entity collection took 00:00:01.0780767 Reading 10000 entities into an entity collection took 00:00:01.0937010 Reading 10000 entities into a collection of custom classes took 00:00:01.4530599 Reading 10000 entities into a collection of custom classes took 00:00:01.4374356 Reading 10000 entities into a collection of custom classes took 00:00:01.4374356 Reading 10000 entities into a collection of custom classes took 00:00:01.4061870 Reading 10000 entities into a collection of custom classes took 00:00:01.4374356

Frans Bouma | Lead developer LLBLGen Pro
Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 06-Sep-2006 12:28:43   

As you can see, entitycollection fetches are less than 10% slower than a datatable fetch of the same data.

5 times slower means that it should have taken EntityCollection fetches to take about 5 seconds or more to read 10000 entities, but it's far less, very close to a datatable fetch.

I've to add: the less columns you have in your table, the faster the datatable will be. My testentity is described in the projector code above, and has 12 fields, strings, guids, ints etc. some of them NULL. It's completely random data.

Sorry to be so anal about this, but I'm pretty sick and tired of FUD coming from the mygeneration corner that llblgen pro is slow compared to a tiny/narrow datatable fetch.

Frans Bouma | Lead developer LLBLGen Pro
mihies avatar
mihies
User
Posts: 800
Joined: 29-Jan-2006
# Posted on: 06-Sep-2006 12:57:37   

Otis wrote:

As you can see, entitycollection fetches are less than 10% slower than a datatable fetch of the same data.

5 times slower means that it should have taken EntityCollection fetches to take about 5 seconds or more to read 10000 entities, but it's far less, very close to a datatable fetch.

I've to add: the less columns you have in your table, the faster the datatable will be. My testentity is described in the projector code above, and has 12 fields, strings, guids, ints etc. some of them NULL. It's completely random data.

Sorry to be so anal about this, but I'm pretty sick and tired of FUD coming from the mygeneration corner that llblgen pro is slow compared to a tiny/narrow datatable fetch.

Interestingly OP states that the difference is 10% when using typed lists. So, something is not right at his corner I suppose.

MatthewM
User
Posts: 78
Joined: 26-Jul-2006
# Posted on: 07-Sep-2006 01:01:27   

Otis wrote:

Sorry to be so anal about this, but I'm pretty sick and tired of FUD coming from the mygeneration corner that llblgen pro is slow compared to a tiny/narrow datatable fetch.

Interestingly to get the buy off for LLBL I had to do a bench against a few other products including doodads. I thought I did a fair bench, and doodads was about 30% faster on reads, which nearly cost the decision.

However, for myself it is a simple matter that doodads does not provide actual objects nor does it provide the relationships. A wrapper around a datatable, imo, was not a DAL solution.

I chalked the extra time up to object creation and initialization, and told my boss that there was no way we could do this project without something having the capability of LLBL. Accurate or not, 30% is just simply no big deal.

IMO using doodads would just make for horrific code (forgive me if I oversimplify this trainwreck):


Customers customers = new customer(1);
Orders orders = new Orders();
customers.AddNew();
orders.AddNew();
orders.CustomerID = customers.CustomerID;  // WRONG!!  customers now points to a different row!

Midnight
User
Posts: 14
Joined: 07-Sep-2006
# Posted on: 07-Sep-2006 03:36:45   

I am a small software developer, my clients are small businesses. Performance is not an issue for me because the size of the databases is also relatively small. However, the results of Frans's tests are more than adequate. Before buying LLBLGen Pro I tried MyGeneration and DooDads. I really wanted to like it because the price was right. However, as in most things in life you get what you pay for. There is just no comparison in functionality between LLBLGen and MyGeneration. Although, the MyGeneration crowd are pretty good at self promotion.

Richard

flo1227
User
Posts: 8
Joined: 16-Sep-2004
# Posted on: 07-Sep-2006 08:41:30   

I just wanted to state that I'm a happy customer of LLBLGen for years already and just want to convince my new company of itwink Btw. what's a FUD?disappointed

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 07-Sep-2006 09:21:29   

They're indeed good in selfpromotion, but that's ok simple_smile

FUD == "FUD is an abbreviation for Fear, Uncertainty, and Doubt, a sales or marketing strategy of disseminating negative but vague or inaccurate information on a competitor's product. The term originated to describe misinformation tactics in the computer software industry and has since been used more broadly. " http://en.wikipedia.org/wiki/FUD

It's not you, Flo1227. I've read posts/blogs about how 'fast' doodads is compared to llblgen pro and all they did was test a 2-field fetch. It's annoying, but part of life I guess. simple_smile

Fetches were a bit of a problem in the 1.0.200x.y architecture, as there were some bottlenecks in the pipeline I couldn't get rid of due to architectural changes which were necessary. I removed them in v2.0 and profiled the living daylights out of the fetch code, also for memory consumption. There's still a small bottleneck in the code, but I can't remove that one, I couldn't find a solution for that. (it's about overriden fields in a subtype, the aliasses have to be changed).

In the beginning I looked at the datatable as the primary storage container for entities, but dropped it because the datarow cant live on its own, and it has the severe disadvantage that you can't add a property to the entity without also adding a column. So I went for the small bit of overhead and opted for classes instead.

Frans Bouma | Lead developer LLBLGen Pro
Devildog74
User
Posts: 719
Joined: 04-Feb-2004
# Posted on: 08-Sep-2006 03:54:11   

Why not comission an independent 3rd party to do a benchmark of the various OR tools out there? Dont PC magazines do this on various types of products from time to time?

Also, I thought FUD == "F*#@!# Up Data"

Answer
User
Posts: 363
Joined: 28-Jun-2004
# Posted on: 08-Sep-2006 04:17:26   

Just out of curiosity, but has anyone benched the time it takes to load say a single entity compared to a single row in a datatable?

Cuase i never really load more then 10-20 entities at a time, and if i do more then that its almost always a typed list with around 25-200 rows. My guess would be the time taken would be so insignificant between the two (using classes vs datatables) that it basically doesnt matter, go with the design that gives more flexibilty and functionality out of the box simple_smile

Otis avatar
Otis
LLBLGen Pro Team
Posts: 39588
Joined: 17-Aug-2003
# Posted on: 08-Sep-2006 09:31:01   

Devildog74 wrote:

Why not comission an independent 3rd party to do a benchmark of the various OR tools out there? Dont PC magazines do this on various types of products from time to time?

Benchmarks are a marketing tool, because it is hard to do them in a fair test AND use real-life tests. There's one benchmark test for o/r mappers, on Java that is, which counts the # of queries executed on the DB. Also not that great, I mean: what if the queries are slow?, but it tests something. So the saying goes: Lies, D*mn Lies and Benchmarks. wink

Answer wrote:

Just out of curiosity, but has anyone benched the time it takes to load say a single entity compared to a single row in a datatable?

Yes, datatable loads are slower, because the time to setup a datatable with the columns is significant now.

Cuase i never really load more then 10-20 entities at a time, and if i do more then that its almost always a typed list with around 25-200 rows. My guess would be the time taken would be so insignificant between the two (using classes vs datatables) that it basically doesnt matter, go with the design that gives more flexibilty and functionality out of the box simple_smile

Of course simple_smile The bigger picture contains more than just a single fast 2-column fetch simple_smile . That's also a reason why benchmarks have to be taken with a grain of salt.

Frans Bouma | Lead developer LLBLGen Pro
sailu_tp
User
Posts: 6
Joined: 09-Sep-2006
# Posted on: 09-Sep-2006 20:46:57   

I feel it is not a valid comparison between Doads and LL because Doads generates data layer objects and not business objects. LL generates business objects. After generating data objects using Doads you need to build business objects using them. How much time does it take to build these and compare these against LL and then it is a more valid comparison. Just my 2 cents. Sailu

flo1227
User
Posts: 8
Joined: 16-Sep-2004
# Posted on: 12-Sep-2006 08:36:58   

Its correct that LL builds business objects unlike dOOdads. But if you just need to fill a flat table with values without any relations or whatever overhead, you would be happy if there is a fast way to do this. Luckily LL is flexible enough to provide a method. TypedLists are the perfect way to do this kind of work.