Sun's Conflated Benchmark
Bill Roth's Blog |
September 25, 2007 11:56 AM
|
Comments (5)
File this in the "Just So You Know" category. Sun recently announced a new benchmark from their app server. It is an interesting announcement, but you need to read what they say carefully. First off, the are conflating Glassfish with their commercial application server. Developers should be aware that there are significant differences between the two. The Sun Application Server result uses the Sun 6.0 JVM while our older BEA WLS 9.2 result used the Sun 5.0 JVM. When pressed, I am sure Sun will admit that there are many performance improvements in their 6.0 JVM. We believe the JVM accounts for most of the difference between our app server scores. We have improved our server benchmark tuning since our T2000 publication last December 2006. Interestingly, Sun has not published any server benchmark results on Glassfish. The published results they refer to are on the commercial version. Sun like's to blur the distinction between Glassfish and the commercial version and use them interchangeably. If Glassfish can run the server benchmarks, why hasn't Sun submitted results on Glassfish for review and publication? It should be noted that all of the results using Sun's application server are at relatively low throughputs. Published results on WLS show that we scale all the way up to the highest result ever published on the most popular benchmark. Technorati Tags: BEAS, BEA, BEA Systems, Java, J2EE, JEE, benchmark, WebLogic Server, Glassfish, Sun App Server, JVM, Jrockit
Comments
Comments are listed in date ascending order (oldest first) | Post Comment
-
FYI
Posted by: jclingan on September 27, 2007 at 10:09 AM
-
A quick check of the published FDR for the Sun Java System Application Server 9.1 result confirms the "DataDirect Connect for JDBC 3.6" driver was used to obtain this result. See
http://www.spec.org/jAppServer2004/results/res2007q3/jAppServer2004-20070619-00069.html#JDBCDataDirect_Connect_for_JDBC_3.60
A JDBC driver is certainly not performance neutral and therefore can be significant in a performance benchmark. Glassfish V2 didn't earn that SPECjAppServer2004 result because it doesn't include the JDBC driver used to achieve that performance. If Glassfish can achieve the same performance without those drivers, why not simply submit a result with the Glassfish drivers to prove it. Otherwise, stop claiming a result that wasn't earned.
It's true that SPEC results require supported products, but all that would require is listing a 3rd party willing to provide the required support for Glassfish. Real Glassfish results without third party JDBC drivers CAN be published, so please don't hide behind that smokescreen.
Steve Realmuto
Posted by: realmuto on September 28, 2007 at 11:30 AM
-
Mr. Clingan, your response?
Posted by: wgroth2 on September 28, 2007 at 12:06 PM
-
3rd party drivers are not performance neutral and I responded to a similar posted comment yesterday on my blog post. Sun's model is to commercialize GlassFish as the Sun Java System Application Server 9.1. These are the same bits except for the differences I've posted in the referenced FAQ (installer, Data Direct drivers, HADB). According to SPEC rules, we can't speculate what the results would be with GlassFish and 3rd party Data Direct drivers. With GlassFish + 3rd party Data Direct drivers, the only relevant difference in this case would be the installer (HADB wouldn't apply). Running the benchmark twice on the same bits would not be an efficient use of resources.
In the end, Sun recommends customers deploy their applications to the both WebLogic and then on GlassFish and/or the Sun Java System Application Server. Then run the ROI with the Sun Java System Application Server 9.1 at $4,500 for 4 sockets.
Posted by: jclingan on September 28, 2007 at 1:58 PM
-
Whups, just noticed I mis-read the prior JDBC driver comment. GlassFish only ships with JDBC drivers for JavaDB (Derby re-branded). For customers to use GlassFish V2 with IBM DB2 (the database used in the benchmark), they would still have to use a 3rd party driver. Customers would typically choose between either the driver supplied by IBM or, as in the Sun Java System Application Server 9.1, the Data Direct drivers for improved performance. The Data Direct drivers are not free.
So, in this use case, apply the cost of the Data Direct driver (or IBM DB2 driver) + cost of GlassFish V2 (free). Run the ROI. Or, if support is required, then purchase the Sun Java System Application Server 9.1 with the included Data Direct Drivers (or IBM DB2 driver). Run the ROI again. If the JDBC drivers are the same, then you'll get the same performance between GlassFish V2 and the Sun Java System Application Server 9.1 because the bits are the same. Now, compare the ROI of what you are running today ...
For complete disclosure, neither GlassFish V2 nor the Sun Java System Application Server ship with the T2000's used in the benchmark :-)
Posted by: jclingan on September 28, 2007 at 2:58 PM
|
