merge problem creates runtime trouble

Bookmark Blog Post

	del.icio.us
	Digg
	DZone
	Furl
	Reddit

Jeremiah Johnson's Blog | February 27, 2007  11:09 AM | Comments (0)

I generally feel pretty good about refactoring and big merges because I am always pushing unit testing. If a suite of unit and / or integration tests exists, then we can feel confident that our app either broke or didn't during changes and can get an idea of where problems might be. So this past week, we did a big merge and then some code cleanup and one of our applications started failing tests.

To trouble-shoot, we usually start with test output; however, there was a special kind of problem in this case: some of our Web services were Oneway so they don't give responses and we rely on logging and checks against the database to verify things went the way the were supposed to. For these particular Web services, the database wasn't getting any changes and we were getting half of our logging. Here is a clip from the broken Web service:

@SOAPMessageHandlers( { @SOAPMessageHandler(className = "feedback.util.FeedbackLogHandler") })
public class SoapListener implements ServiceLifecycle {
...
    public void init(Object context) {
        log.debug("init invoked");
        servletEndpointContext = (ServletEndpointContext) context;
    }
...
    @Oneway()
    @WebMethod()
    public void accept(CrashDetail crashDetail) {
        log.debug("accept invoked");

The init was getting invoked as expected and we could verify that in the logs. The handler was getting invoked as expected which we also verified in the logs. Interestingly, 'accept invoked' wasn't appearing in the logs. This was a first for me.

There were no errors on the console, no application errors, nothing. In cases like this, you have to turn to the ultra-mega-verbose debugging of WSEE. Don't ever run with this in production unless you have some beefy monster of a machine and lightning-fast IO. I restarted WebLogic with the additional flag of -Dweblogic.wsee.verbose=*

I deployed a version of the EAR before any recent changes to see what the WSEE output should look like, ran my tests, and then deployed a 'broken' version of the EAR. Here's what WSEE showed me:

<WSEE>Processing AuthorizationHandler...  <HandlerIterator.handleRequest:118>
<WSEE>Processing ContainerHandler...  <HandlerIterator.handleRequest:118>
<WSEE>Failed to invoke handler<HandlerIterator.handleRequest:141>
<WSEE>weblogic.wsee.jws.container.InvokeException: Unable to Load JWS<HandlerIterator.handleRequest:141>
weblogic.wsee.jws.container.InvokeException: Unable to Load JWS
        at weblogic.wsee.jws.container.Container.init(Container.java:135)
        at weblogic.wsee.jws.container.Container.<init>(Container.java:79)
        at weblogic.wsee.jws.container.ContainerFactory.createContainer(ContainerFactory.java:51)

Honestly, that wasn't so helpful. Another first: usually, when I have any surprises in Web services, I turn on WSEE debugging and I find my problem quickly - well, as quickly as one can when searching through thousands of lines of debugging output.

Finally, I unpacked the good and bad EARs and found the problem by comparing all the files. I found that the weblogic-application.xml file had lost the weblogic-controls-1.0 library during the recent merge. Interestingly, the library wasn't needed for builds, so I could build and export without trouble. At runtime, I wasn't seeing any errors - probably because of the Onewayness of my service in this case.

Hope my steps for trouble-shooting my service will be helpful to someone out there.

Comments