Dmitri Maximovich's Blog

Dmitri Maximovich's Homepage
Dmitri Maximovich is an independent consultant specializing in software design, development, and technical training. He has more than twelve years of industry experience and has been involved with J2EE since its inception. His job profile includes designing and developing mission critical applications for financial and pharmaceutical industries.

JMS messaging with Spring...nice but there is no magic

While article by itself can serve as reasonably good introduction to JMS support in Spring, there are some important moments which author didn't cover. The fact that Spring being positioned as 'J2EE framework' combined with recent announcement of BEA official support of Spring in WebLogic may lead some developers to believe that code from the article could be transfered without changes into your J2EE application running in WebLogic (probably instead of file-based JDNI most would prefer to use WebLogic startup class to map MQ ConnectionFactories and queues into WebLogic JNDI namespace or map it through support for Foreign JMS providers).

So what we would get if we transfer code from the article as is into J2EE application running in WebLogic? Well, as expected it's going to work - messages getting send and delivered, no exceptions flying around so everything is good at first sight. Until you try to use it in CMT or BMT transactions, say like in the code below, from a session bean:

/**
 * @ejb.bean
 *   type="Stateless"
 *   name="SpringTest"
 *   view-type="local"
 *   transaction-type="Container"
 *
 * @ejb.transaction
 *   type="RequiresNew"
 */
public class SpringTestBean implements SessionBean {
...
  /** @ejb.interface-method */
  public void sendMessage() throws Exception {
    JmsSender jmsSender =  
      (JmsSender)springContext.getBean("jmsSender");
    jmsSender.sendMesage("test");
    // rollback CMT transaction
    sessionContext.setRollbackOnly();
  }
...  
}

One would expect that after transaction rollback message won't be in the MQ but it will. The reason for this is pretty simple - WebLogic need to use special wrappers around MQ ConnectionFactory object to ensure proper resource enlistment into XA transaction context. Just putting object into WebLogic JNDI is not enough. Developer should declare ConnectionFactory via resource-ref element in EJB deployment descriptors:

<resource-ref>
  <res-ref-name>myQcf</res-ref-name>
  <res-type>javax.jms.QueueConnectionFactory</res-type>
  <res-auth>Container</res-auth>
  <res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>
...
<resource-description>
  <res-ref-name>myQcf</res-ref-name>
  <jndi-name>mq.qcf</jndi-name>
</resource-description>

And than, in Spring context definition, QueueConnectionFactory should refer to name mapped via resource-ref:

    <bean id="jmsQueueConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiName" value="java:comp/env/myQcf"/>
    </bean>

Note that factory looked up in java:comp/env namespace and not in the global JNDI scope. This will ensure that WebLogic uses properly wrapped ConnectionFactory object which is going to participate in global transaction and our example above would work as expected.

Although our example now works this is not very nice, because now all your operations with Spring's JMS objects should be originated from some EJB with properly defined resource-ref. This means that for example developer should be careful not to inject JMSSender as dependency into some class which could be executed not as a part of EJB call sequence (for example scheduler of some sort) or when access to more than one ConnectionFactory is needed. Alternative approach would be to extend Spring's JndiObjectFactoryBean class to enable creation of required wrapper. The problem with this is the wrapper API is not documented as far as I know.

I guess the conclusion is that it's best not just assume that framework would do 'magic' stuff for you and expect everything to work, but always test to make sure it actually works.

Oracle nightmare? I think not yet.

it could be the beginning of the end of Oracle's business model. If you really think about it, open source databases are getting very good. Better than good enough.

So is it really that open source databases ready to replace Oracle in as Enteprise database and we all soon should stop buying that expensive Oracle releases?

Let's see. How do you like this issue in MySql for example (note status 'Not a bug'):

By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate, and due to the design of the MySQL network protocol is easier to implement. If you are working with ResultSets that have a large number of rows or large values, and can not allocate heap space in your JVM for the memory required, you can tell the driver to 'stream' the results back one row at-a-time.

Now I'd like to see how's "US Democratic National Committee is using MySQL with a 1.5 billion row database for its voting analytics" - selecting all rows in memory or fetching one by one? I'm not saying that it's impossible, sure there is workarounds for this but this is not the only one problem.

To name a few more - how about read locking? Oracle employ multi-versioning for read consistency so your read requests are never blocked. Most (all?) other databases use shared read locks (which will lead to deadlocks and blocking) - good luck with dealing with these problems in shared enterprise environment.

Now how about robust distributed transactions support? It's just planned for the first time for next version of PostgreSql (not sure about MySql) and it will take them a while to iron it out. Enterprice application without distributed transactions? Sure...again, probably possible in most cases with workarounds and so on but the list can go on and on.

Now don't get me wrong, I'm using PosgreSql in many projects and this is excellent database as I'm sure MySQL and some others and there is definitely progress in this area so eventually it has a potential to be compared to Oracle, but I don't think this time has come just yet.

It's very simple to create a connection pool in WebLogic as long as database has a JDBC driver, but I wouldn't rush to conclude based on this fact that the database is ready for a prime time in the Enterprice as Oracle replacement.

What to do if your database doesn't support XA transactions?

Deepak Vohra wrote an article about configuring and using PostgreSQL from WebLogic Server. While it's certainly good introduction to get you started on using PosgreSql it fails to describe significant limitations that this database have when it comes to Enterprise-level development. Don't get me wrong, I personally think that PosgreSql is great database and is excellent free alternative even to leading commercial RDBMS to certain kind of projects. I personally used PostgreSql for some of my projects in the past and it's certainly in the first line of databases I would recommend to consider when you choose database for your next project. It's easy to install and manage, reliable, have good feature set and standards compliance and even has sequences support which makes it conveniently compatible with Oracle when it comes to PK generation strategies and so on.

But I also believe that developers should understand limitations of tools they're using, probably even better than benefits that these tools provides. Being excellent database overall PosgreSql lack support for distributed (XA) transactions, at least not in latest version 8. I'm not monitoring PosgreSql development closely so I'm not aware if adding XA support in the list for any future release (if anyone is better informed please post reply here, would be interesting to know). As a matter of fact, I don't think that there is any free open-source database with support for XA currently available, is there? Would be glad to learn that I'm wrong.

The point is that you need to analyze your project's requirements to understand if XA support is important for your particular project or not. I personally found that it's quite important feature for most Enterprise projects I've been worked on. Pretty much any time when your application have to work with more than one Resource Manager at once (think for example about MDB updating database) it's good to know that you could use XA transaction to get highest level of Quality of Service . Some people would argue that most of applications doesn't need (or could manage without) XA, which could be true to certain extent but almost always adds at least to development efforts (detecting duplicates in above example) or otherwise leads to degradation of QoS. Sometimes you just have to have XA to deliver functionality required by your use case.

Anyway, back to the topic of this post - what if you're using database that doesn't have support for XA transactions but you need it for one reason or another. Besides from manual strategies for emulating or avoiding XA transactions there is a transaction optimization technique known as "Last resource commit" which could be found in most modern Transaction Manager implementations. The idea is that it's possible for a single resource that is not XA aware (e.g. can only commit or rollback with no support for prepare phase), to be enlisted in a transaction with any number of two-phase commit aware resources. Transaction Manager then threats non-XA resource slightly differently, in that it executes the prepare phase on all (XA) resources first, and if it then decide to commit the transaction it passes control to non-XA resource. If that resource commits, then Manager logs decision to commit and attempts to commit the other resources as well (second phase of commit protocol). If commit on non-XA resource fails for any reason, Transaction Manager executes rollback to previously prepared XA resources. As you can imagine, there are failure scenarious where loss of atomicity is possible here but overall they should be pretty rare in practice.

WebLogic Server, starting from version 9, supports Last Resource Commit optimization (BEA called it Logging Last Resource Transaction Option or LLR). It's worth to add that besides from allowing you to enlist non-XA resource in XA transaction this feature could be viable performance-optimization technique as well, because XA drivers are generally less efficient compared to non-XA drivers and XA protocol itself adds some overhead in terms of network traffic and disk IO.

LLR feature in WebLogic 9 should not be confused with "Emulate Two-Phase Commit", which was available as an option in JDBC Connection Pools level in previous versions. There are subtle but significant differences between them. According to BEA's documentation for WebLogic 8:

When the Emulate Two-Phase Commit for non-XA Driver option is selected (EnableTwoPhaseCommit is set to true), the non-XA JDBC resource always returns XA_OK during the XAResource.prepare() method call. The resource attempts to commit or roll back its local transaction in response to subsequent XAResource.commit() or XAResource.rollback() calls. If the resource commit or rollback fails, a heuristic error results. Application data may be left in an inconsistent state as a result of a heuristic failure.

As you see the behaviour is completely different and should produce more heuristic outcomes when used.

There are many different approaches developer can use to deal with distributed transactions and it's good to have a choice as always. It pays to know your database and Transaction Manager features. I'd also recommend excellent book "Java Transaction Processing : Design and Implementation" by Mark Little et al. if you need to get a good background on Transaction processing theory and practice including JTA.

Optimal configuration of EJB Entity cache in WebLogic

If you read my previos post you probably agree that choosing right size for the Entity cache is vitally important, and not only from performance optimization point of view. From one hand if cache size is too large your application memory consumption would be unnecessary high but if you go to other extreme and configure cache size too small we're risking running into CacheFullException. So how to choose optimal cache size for all your Entity beans?

If you don't specify cache size for Entity bean explicitly via max-beans-in-cache element in deployment descriptor, WebLogic would use default size of 1000. This could be sufficient for some beans where it's know a-priori that number of instances not going to be very large, for example if bean represents lookup table in database, such as 'country' or 'state' where upper number of bean instances are well known. In this case it's perfectly ok not to specify cache size and let WebLogic use default value because there is no memory impact if cache is not used to full capacity. On a side note it's good idea to use ReadOnly concurrency strategy for the beans that is not changed or changed infrequently - this will not only eliminate unnecessary ejbLoad() calls but also limit number of instances with the same PK to one in Entity cache for that bean (multi-versioning is not necessary) therefore preserving memory and improving performance.

Things are little bit more complicated for beans where maximum number of instances which could be accessed simultaneously are not known. You need to analyze and estimate maximum number of beans returned from finder methods and accessed within one transaction then multiply by maximum number of simultaneous transactions with such operation that your application is expect to handle (this is usually limited by maximum number instances of entry points into your application - session beans and/or MDBs). This will give you rough estimate of minimum cache capacity needed for that particular Entity bean.

Analyzing and configuring each bean's cache could be cumbersome in case if there are many Entity beans used in your application. Especially tricky is to estimate number of bean instances returned from 'detail' side of 'master-detail' relationship, for example if your application executes finder on 'order' table and each order has collection of 'items' which could vary from one to potentially very high number. Another problem is that since every Entity bean has separate cache memory is not utilized in a most efficient way.

Probably realizing limitations of 'single cache per bean' model BEA added support for Application-Level Caching for Entity beans in WebLogic starting from version 7. This allows multiple entity beans in the same J2EE application to share a single runtime cache.

Application-level caching offers the following advantages:

• Reduces the number of entity bean caches, and hence the effort to configure the cache.
• Better utilization of memory and heap space, because of reduced fragmentation. For example, if a particular EJB experiences a burst of activity, it can make use of all memory available to the combined cache, while other EJBs that use the cache are paged out. If two EJBs use different caches, when one bean's cache becomes full, the container cannot page out EJBs in the other bean's cache, resulting in wasted memory
• Simplifies management; combined caching enables a system administrator to tune a single cache, instead of many caches.
• Provides better scalability

Using Application-level caching is viable alternative to specifying individual caches per every bean. There are no restrictions on the number of different Entity beans that may reference an individual cache as well as on number of caches defined. cache size could be specified in terms of number of bean instances (approach similar to one cache per bean) or in max memory size. Using memory size could seems attractive from management prospective but be aware than WebLogic is not caclulating actual amout of memory consumed by beans in the cache (that would probably be too expencive operation to perform) but merely divides amout of memory specified by average size of a bean which could be specified in the weblogic-ejb-jar.xml deployment descriptor. If size is not specified a bean is assumed to have an average size of 100 bytes. Therefore I think it's more transparent to specify cache size in number of bean instances.

EJB Entity cache and transactions

Everybody know that EJB Container supports Entity Beans caching. Usually cache size is configured per Bean in vendor-specific deployment descriptor. In WebLogic it's weblogic-ejb-jar.xml file, max-beans-in-cache element. If value of this element is not specified, default of 1000 is used.

Probably because it's described in documentation purely as performance-optimization parameter not all developers realize that there are some other implications which could dictate selection of proper cache size.

One of the dependencies, which sometime not immediately obvious, is that when your transaction uses Entity beans they're actually loaded and 'pinned' in Entity cache for the duration of transaction even if your code is not modifying Entity beans but just reading values from it.

For example, imagine that code in Session or MDB Bean is executing finder method on Entity bean Person and then iterates over returned Collection:

    ...
    Collection persons = personLocalHome.findSomething();
    
    for (Iterator iter = persons.iterator(); iter.hasNext();) {
        PersonLocal person = (PersonLocal)iter.next();
        Long id = person.getId();
        // do something: log, send email, etc
        ...
    }
    ...

If findSomething() method returns more objects than value specified in max-beans-in-cache your application will get unpleasant (and most likely unexpected) weblogic.ejb20.cache.CacheFullException exception when Iterator gets to the N+1 object (where N is the current Entity cache size).

This could seems like not such big of a deal and one could argue that finders should not return very large collections anyway, etc. but don't forget that first of all by default WebLogic Entity cache is multiversioned, which means if multiple transactions requests the same Entity bean multiple versions (one for each transaction) is created in the cache which could effectively lower cache capacity in terms of unique objects and second is that it's normal to have multiple simultaneous transactions running in container at the same time. Imagine if code above getting called from Session or MDB bean which deployed with high value in max-beans-in-free-pool parameter (which by default is 1000 as well) and there are say 50 simultaneous client requests coming in. This will leave every transaction with just 1000/50 = 20 slots in Entity cache to use and if finder returns more than 20 objects, well some of transactions are going to fail.

This is just something to keep in mind while designing operations with large number of Entity beans. Situation also worsened the fact that developers often works with pretty small database sizes where this problem could not introduce itself at all until code is deployed against production-size database. As a safeguard measure I would suggest in development not to use default settings for cache size but set it at artificially low value (10-100) so cache-related problems could be discovered and fixed at early stages of development.

October 2007

Search this blog:

Archives

October 2005
September 2005
May 2005
April 2005

Categories

Recent Entries

JMS messaging with Spring...nice but there is no magic

Oracle nightmare? I think not yet.

What to do if your database doesn't support XA transactions?

Articles

High Performance Message Processing with BMT Message-Driven Beans and Spring
Addressing the traditional MDB-based message consumption model, Dmitri Maximovich and Eugene Kuleshov show how this scenario can be refactored to increase performance with non-transactional message retrieval while retaining once-and-only-once quality of service. Jan. 30, 2006

Peak performance tuning of CMP 2.0 Entity beans in WebLogic Server 8.1 and 9.0
Tuning CMP 2.0 EJBs is as much an art as it is a science. In this detailed article, Dmitri Maximovich takes the reader on a tour from concurrency strategies and caching between transactions, to the read-mostly pattern and choosing an optimal cache size. Nov. 14, 2005

Parallel task execution in J2EE using the Work Manager specification
As it stands, the J2EE specification provides no easy way to initiate the execution of parallel tasks. JSR 237, the Work Manager for Application Servers specification, changes this. In this article, Dmitri Maximovich introduces the specification, and provides an example of how to use it. His code runs on the current beta of WebLogic Server 9.0. May. 16, 2005

All articles by Dmitri Maximovich »