ORMBattle.NET

We would love to be included in your score card. Just an FYI ...

Ok - thanks

Will you submit the test code, or we should write our own and send it to you for review?

We'll write the test code for you guys and then you can look it over to make sure we did it appropriately. We are heavy on next release but will come up for air soon, the EntitySpaces Q3 Beta is on Aug 31st and ships on Sept 28th so it will be either shortly after that or maybe even before, I don't imagine it will take us long at all to do, possibly after we get the Beta out the door on August 31st ...

- Mike

I cannot believe you guys withdrew your performance stats? That's too bad, and still other ORM vendors it appears had theirs removed. We are eagerly looking forward to submitting our tests. Sure, we could score low in certain areas but if we do we will know where to make improvements. There are many more criteria than just sheer performance stats when considering an ORM, everybody knows that.

We applaud your attempt here. Don't let the initial outcry discourage you guys. Our fear is now that you've pulled your stats why would you maintain such a site? It only advertises your competitors. You guys MUST republish your results and let the chips fall where they may. And what’s with the new scale? Come on guys, leave the politically correct stuff behind. Land Mark to Best Result, what happened to Worst Result to Best Result? We ought to be big boys by now.

We really think you guys should consider republishing your performance stats. If there are groups that don't like the tests let them pull their stats. You've done all the work on this, you should have your stats published.

Anybody ever heard of the Transaction Processing Performance Council ? It provides a very valuable service and we need such a thing for ORM solutions too. Maybe we vendors need to agree on a set of common crud tests. For instance, we saw a SubSonic post about performance and wondered how we would do, so we duplicated them and posted the results here, these types of tests seemed to be sensible.

www.tpc.org/tpcc/default.asp

The EntitySpaces Team
www.entityspaces.net

Mike, we are going to implement TPC-C or TPC-E performance tests where all framework features will be allowed because it's business logic bench with huge DB workload.
First stage is to make infrastructure for these tests (different implementations for particular ORMs should be rather similar). What test do you prefer (TPC-C or TPC-E - new)? Please reply.

Thanks for your reply. We did this mainly to reduce the heat over this web site. We understand it's temporarily. But before doing this, we must prove people may believe us and these tests. Since there is not enough info about this here, it was the best thing we could do: make it less suspicious. If public won't believe our tests, they aren't interesting at all anyway.

So it was not a withdraw. Yes, Oren won the first match - we decided to remove DO from this web site. But the next one was lost: NH is still one of the worst products there. After all the optimizations they suggested (no more proposals just because they're stuck now - their test looks perfectly).

So yes, we've lost one match, but the battle isn't finished

Keep tracking our blog, we're working on improving the web site & tests.

And thanks for your support.

Err... admin = Alex Yakunin

Mike, I've just implemented performance tests for EntitySpaces. Please checkout it from code.google.com/p/ormbattle/source/checkout. If you would like to modify your tests, provide us with your google account name (my email: alexis.kochetov@gmail.com) and I will add you to project commiters.

We will not publish the results without your explicit premission.

Oh, I see how to access it, but how did you add me as a committer, I see some things that need changing, you never have to declare a transaction in your tests, our Collection in Save() declare an esTransactionScope and protect it. So you're doing nested transactions in that case. Also, we can do link queries, see

www.entityspaces.net/blog/2008/05/11/Ent...ext+Beta+Part+2.aspx

You can create a link query and ask ES to load it (sneaky but it works)

We would like to make a few minor tweaks, but I need to figure out how to login, we will stay true to your tests though

See my post above, okay, I have access, will sync via SVN tonight and make some tweaks, and fill in the LINQ methods (because we can do that). In most cases in your tests we do not require a transaction, our collections create one for you automatically in the Save() method so you are creating nested transactions.

Also, not sure why you are creating transactions for things that just read but that's okay I guess. We just want to make a few minor tweaks, good job by the way so for so quickly figuring out how to use EntitySpaces.

If you would could you send me a PDF of the timings, I'm biting my fingers nails off here send them to my email address, you should have it as we have sent email back and forth on this already.

Anyway, let us make some quick tweaks and we'll email you when complete, and then you can review our changes, run the tests, and publish your findings.

Hey guys, we still support your efforts but we are asking for some changes that we think are very reasonable, make sense, and disallow specific optimizations that do not reflect real world usage anyway, see ...

www.entityspaces.net/blog/2009/08/22/ORMBattleNET.aspx

The EntitySpaces Team

Thanks - I'm keeping track of this thread, but can't do anything right now. I got a huge debate with Oren again, and hopefully, it's going to its end now (see his blog).

We'll definitely implement everything you wrote on Monday.

P.S. Just watched "Inglourious Basterds" - very good, and "District 9" - simply must see. Almost sure I never seen better science fiction film so far.

Just checking, did you see my post above?

www.entityspaces.net/blog/2009/08/22/ORMBattleNET.aspx

We are going to hold off participating for now, if you address the issues above we will gladly participate and think these are reasonable tests, in fact, you could probably change all of the tests to do this in a single day. This would go a long way towards eliminating alot of the debate.

Sorry, I really didn't read it yet. I thought it's about the same post one of my co-workers told me. I just looked it up. Comments:

1. "Single" tests aren't finished yet, and their results are not shown at all. We added them by request of NH community (to show another edge case and compute "average" better). So don't pay much attention to them. That's why there are no tests for OpenAccess for this. We simply had no time to implement them yet.

2. Our use of transactions is described here: ormbattle.net/index.php/performance-tests.html . Quote:

"There is one test per each tested operation per each ORM. Each of such tests executes nearly the following sequence:

- Open a session
- Start time measurement
- Open a transaction
- Perform N tested operations, currently N = 1000, 5000 and 30000
- Commit transaction
- Finish time measurement
- Close session"

And I don't see anything suspicious here:
- We open and commit a transaction just once per test to ensure all the operations have really happened.
- If we'd run multiple transactions (e.g. one per each command), we'd measure transaction per second rate. Which is nearly equal to HDD seek time on any system: SQL Server must flush the tail of transaction log to ensure the durability. So we'd get ~ 80-200 op/sec rate on any of our tests on almost any PC with regular HDD (not SSD!).

Resume: having a single transaction is necessary. And this is optimal and honest case.

3. I didn't get the part about collections. We never use collections in tests of other frameworks (except IQueryable, , which is in fact IEnumerable, but not a collection).

Ah, I got... Yes, by some reason Alexey used collection (SimplestCollection) in your tests. I'll ask him, why. May be because he didn't find a better way to load all the objects from the database...

Anyway, if you can advise how to implement this properly, please do this.

Mike, *Single tests implementation isn't finished yet; these results aren't published at Scorecard page. The main idea behind these tests is to measure the performance of CUD operations when they're intentionally executed one-by-one (i.e. no any batching).

On the other hand, it seems your proposal (fetch then update) has practical meaning. It would be great if you could provide us with exact test sample / methodology and description for this scenario.

Here is what the single tests should do, and not in a transaction, as you know, I think the use of the transactions in the tests (except for collections) is unwarranted.




This would provide useful information about reading and writing single records, and would make these tests consistant, everything has to happen within the for(), no re-use of objects, no transactions, I also think the use of transactions in the query/read tests is being used for statement preperation and sql server specific techniques which aren't valid and will not apply in web scenarios or even fat clients where reads and fetches are not done all within a single transaction. They also do not make the timing more accurate, they only make others skeptical of the testing.

> no re-use of objects

We ensure this in our tests.

> no transactions

Let's make it clear that "no transactions" = "one transaction per each operation". Dealing with RDBMS, you always involve transactions - implicitly or explicitly.

This mean we'll see 80...200 op/sec rate on any PC with "normal" HDD (i.e. without SSD), since the most costly operation in this case is transaction commit (transaction log flush). That's why frequently the only way to increase transaction rate is to use HDD/RAID controllers with memory (caching the writes) or SSD for transaction log.

To commit a transaction, RDBMS _must_ flush the tail of transaction log. Now imagine, you have 15K RPM hard drive. 15000/60 = 250 RPS (rotations per second). So you'll get ~ 250 log appends per second with this drive at max, if nothing special is used. Since we use regular 7200 RPM HDD on tests, we'll get ~ 100 op/sec.

Compare this with record insertion rate: it is ~ 1/200000 op/sec or even higher on our test.

Hopefully, this explains why we measure many of such operation per a single transaction.

> I also think the use of transactions in the query/read tests is being used for statement preparation

We don't use this with DO4. Although e.g. OpenAccess does this. On the other hand, I don't see any "criminal" here: since this happens transparently for developers, it's ok.

Moreover, prepared commands are intact (i.e. can be re-executed) while their connection is alive, rather than transaction.

> and sql server specific techniques which aren't valid and will not apply in web scenarios or even fat clients where reads and fetches are not done all within a single transaction.

What exactly do you mean here? Just unclear.

Sorry, I made a mistake: we run all the tests on workstation, where all write caches are enabled. I.e. both operating system and HDD may delay transaction log flushes to do this efficiently. On one hand, this is not honest: commit does not mean the data is really committed; on the other hand, usage of RAID controller with true write cache (independently powered RAM, etc.) would give nearly the same result.

So transaction rate on machine we run tests on can reach ~ 5000 transactions per second.