Hussein Badakhchani's Blog

Hussein Badakhchani's Homepage
Hussein Badakhchani is an independent consultant and middleware analyst with over 10 years experience in software design, development, management and support. He now specializes in trouble-shooting live enterprise applications for financial institutions and continues to provide independent advice on application and infrastructure architecture as well as build process management. Hussein is also a regular contributer to dev2dev newsgroups posting under the name of hoos.

Help me write a schema file: server.xsd

I have finally got back to work on a code share project I had been neglecting for sometime due to distractions at work. Now that I have aligned my work with the project the pressure is on again and work on Genesis (not to be confused with BEA's miserable application of the same name) has started with a vengence.

I will be mopping up the codeshare repository and blowing away the cob webs from the source over the next few days. Anyway in the spirit of Choom the predecessor of Genesis I wanted to invite the dev2dev community to help me write 'proper' code by criticizing, ridiculing and poking fun at my efforts, so here goes.

I need to write an XML schema file to represent the logical and physical properties associated with a server. For instance the kind of information I want in an XML instance of the schema would be the servers Manufacturer, architecture, supplier, network interface cards etc, etc.

This is what I have come up with so far:

<!--
  The Genesis schema for a Server. This file defines both the
  physical and logical properties of a server.
 -->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="Server">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="manufacturer" type="xs:Manufacturer" />
        <xs:element name="supplier" type="xs:Supplier" />
        <xs:element name="model" type="xs:Model" />
        <xs:element name="arch" type="xs:Architecture" />
        <xs:element name="os" type="xs:OperatingSystem" />
        <xs:element name="memory" type="xs:Memory" />
        <xs:element name="nic" type="xs:NetworkInterfaceCard" />
        <xs:element name="sic" type="xs:SanInterfaceCard" />
        <xs:element name="function" type="xs:Function" />
      </xs:sequence>
    </xs:complexType>
  </xs:element>

  <xs:complexType name="Manufacturer">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="Supplier">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="Model">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="Architecture">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="OperatingSystem">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="Memory">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="NetworkInterfaceCard">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="SanInterfaceCard">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="Function">
    <xs:sequence>
      <xs:element name="Name" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

</xs:schema>

The kind of things I would like feedback on are:

• XML coding standards, white spaces, comments, casing etc.
• Defining my complex types and suggestions for new types to include in the schema for server.
• Exising examples of similar shcemas.

I know I should post this to the forums put I felt I had to displace blog enteries on SOA at dev2dev.

Middleware Strategy

If you Google "Middleware Strategy" the top link is to Sun World article published in 1998. Middleware has not been a very fashionable term in the last few years, eclipsed by acronyms beginning with 'S', preferred by the majority of attention deficit suffering tech writers. However much frustrated Google developers tried to wish it away Middleware has, for better or worse, survived. Now with the advent of Oracle's buy out of BEA I have found myself being contacted by said vendors to discuss their Middleware Strategy and being asked about my own.

A blog entry is not the best place to elaborate my plans, but the areas that I will be focusing on are (in no particular order):

• Provisioning and management
• Serviceability
• SOA/SCA support
• Security and auditing
• and erm, three tier architecture...

Using Appc with Maven

In a previous blog entry I suggested some simple rules for deliverables and mentioned that Maven had some big issues with appc. Well I don't trust anything developers tell me, in fact I don't believe anyone who woks in the I.T industry, and I couldn't believe that installing the MavenWebLogic plugin could be so problematic. I decided to have a crack at the problem myself. About eight hours latter I had deployed my EAR file, fully compiled with appc on to a WLS domain and returned the home page, success! Not bad, I thought, considering the last time I used maven it was to evalutate it for use in one of my projects a few years ago. I decided then that it was too immature and overkill for my needs and wrote my own build using Ant like everybody else, except for some crazy schemers who, it seemed, had developed a loathing of Ant.

Anyway here's how I configured the plug-in, I should point out that the WLS version I am working with is 9.2.2:

Step 1: Download and install Maven.

You can download it here. I used version 2.0.6 the current stable release.

Step 2: Create the project.

I followed the instructions on this page, this is the command in the document:

mvn archetype:create -DgroupId=com.mycompany.app -DartifactId=my-app

Here's the directory structure:

Step 3: Install depencies into your repository.

This step is quite vulnerable to mistakes. What documentation I found was incomplete. I issued the following commands to install the jars from my WebLogic distribution:

>mvn install:install-file -DgroupId=weblogic -DartifactId=xbean -Dversion=9.2 -Dpackaging=jar -Dfile=C:\bea\weblogic92\server\lib\xbean.jar

>mvn install:install-file -DgroupId=weblogic -DartifactId=wlxbean -Dversion=9.2 -Dpackaging=jar -Dfile=C:\bea\weblogic92\server\lib\wlxbean.jar

>mvn install:install-file -DgroupId=weblogic -DartifactId=bcel -Dversion=5.1 -Dpackaging=jar -Dfile=C:\bea\weblogic92\javelin\lib\bcel-5.1.jar

>mvn install:install-file -DgroupId=weblogic -DartifactId=wls-langx -Dversion=9.2 -Dpackaging=jar -Dfile=C:\bea\weblogic92\server\lib\wlw-langx.jar

>mvn install:install-file -DgroupId=weblogic -DartifactId=wlw-util -Dversion=9.2 -Dpackaging=jar -Dfile=C:\bea\weblogic92\common\lib\wlw-util.jar

>mvn install:install-file -DgroupId=weblogic -DartifactId=weblogic -Dversion=9.2 -Dpackaging=jar -Dfile=C:\bea\weblogic92\common\lib\weblogic.jar

>mvn install:install-file -DgroupId=com.sun -DartifactId=tools -Dversion=1.5.0_11 -Dpackaging=jar -Dfile="C:\Program Files\Java\jdk1.5.0_11\lib\tools.jar"

If you install these binaries manually like I did there are a couple of caveats you need to avoid. First make sure you use "" on Windows machines if the jar file is installed in a directory with a space in its name. Make sure you use the wlw-util.jar file under .../common/lib in you WLS distribution not the ones in .../workshop/lib. Finally I noticed an incosistency in the name of the wls-langx artifact. Even though the jar file with the relevant class is wlw-langx.jar the plugin references it as wls-langx.

Step 4: Configure the plugin.

I added the fillowing stanzas to my settings.xml

    <profile>
      <id>Snapshots</id>
      <activation>
        <property>
          <name>java.vendor</name>
          <value>Sun Microsystems Inc.</value>
       </property>
     </activation>
      <repositories>
        <repository>
          <id>Codehaus Snapshots</id>
          <url>http://snapshots.repository.codehaus.org/</url>
          <snapshots>
            <enabled>true</enabled>
          </snapshots>
          <releases>
            <enabled>false</enabled>
          </releases>
       </repository>
     </repositories>
     <pluginRepositories>
       <pluginRepository>
         <id>Codehaus Snapshots</id>
         <url>http://snapshots.repository.codehaus.org/</url>
         <snapshots>
           <enabled>true</enabled>
         </snapshots>
         <releases>
           <enabled>false</enabled>
         </releases>
       </pluginRepository>
     </pluginRepositories>
    </profile>
  
   ...

   <activeProfiles>
    <activeProfile>Snapshots</activeProfile>
   </activeProfiles>

And here is my complete pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.monitor</groupId>
  <artifactId>monitor-app</artifactId>
  <packaging>ear</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>monitor-app</name>
  <url>http://maven.apache.org</url>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>maven-plugins</groupId>
      <artifactId>maven-weblogic-plugin</artifactId>
      <version>1.0.0</version>
      <scope>package</scope>
    </dependency>
      <dependency>
        <groupId>com.sun</groupId>
        <artifactId>tools</artifactId>
        <version>1.5.0_11</version>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>wls-langx</artifactId>
        <version>9.2</version>
        <scope>package</scope>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>wlxbean</artifactId>
        <version>9.2</version>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>xbean</artifactId>
        <version>9.2</version>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>weblogic-container-binding</artifactId>
        <version>9.2</version>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>wlw-util</artifactId>
        <version>9.2</version>
      </dependency>
    <dependency>
      <groupId>weblogic</groupId>
      <artifactId>javelinx</artifactId>
      <version>9.2</version>
    </dependency>
    <dependency>
      <groupId>weblogic</groupId>
      <artifactId>wlw-langx</artifactId>
    <version>9.2</version>
    </dependency>
    <dependency>
      <groupId>weblogic</groupId>
      <artifactId>bcel</artifactId>
      <version>5.1</version>
    </dependency>
    <dependency>
      <groupId>weblogic</groupId>
      <artifactId>weblogic</artifactId>
      <version>9.2</version>
    </dependency>
  </dependencies>
  <build>
  <plugins>
  <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>weblogic-maven-plugin</artifactId>
    <version>2.9.0-SNAPSHOT</version>
    <configuration>
      <inputArtifactPath>C:\monitor\monitor-app\target\monitor-app-1.0-SNAPSHOT.ear</inputArtifactPath>
      <outputArtifactPath>C:\monitor\monitor-app\target\monitor-app-1.0.ear</outputArtifactPath>
      <verbose>true</verbose>
    </configuration>
    <executions>
      <execution>
        <phase>package</phase>
        <goals>
          <goal>appc</goal>
        </goals>
        <inherited>false</inherited>
        <configuration>
          <verbose>true</verbose>
          <debug>true</debug>
          <projectPackaging>ear</projectPackaging>
        </configuration>
      </execution>
    </executions>
    <dependencies>
      <dependency>
        <groupId>com.sun</groupId>
        <artifactId>tools</artifactId>
        <version>1.5.0_11</version>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>wlw-langx</artifactId>
        <version>9.2</version>
        <scope>package</scope>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>wlxbean</artifactId>
        <version>9.2</version>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>xbean</artifactId>
        <version>9.2</version>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>weblogic-container-binding</artifactId>
        <version>9.2</version>
      </dependency>
      <dependency>
        <groupId>weblogic</groupId>
        <artifactId>wlw-util</artifactId>
        <version>9.2</version>
      </dependency>
    <dependency>
      <groupId>weblogic</groupId>
      <artifactId>javelinx</artifactId>
      <version>9.2</version>
    </dependency>
    <dependency>
      <groupId>weblogic</groupId>
      <artifactId>wlw-langx</artifactId>
    <version>9.2</version>
    </dependency>
      <dependency>
      <groupId>weblogic</groupId>
      <artifactId>bcel</artifactId>
      <version>5.1</version>
    </dependency>
    </dependencies>
  </plugin>
  </plugins>
  </build>
</project>

As you can see I had to duplicate all the dependencies, once for the application and again for the plugin, I could not get it working otherwise. This is horrible, does anyone know a way of declaring the dependencies in one place?

Step 5: Run the build.

To run the build you execute:

mvn package

Here's the build output:

[jspc] Overriding descriptor option 'keepgenerated' with value specified on command-line 'true'
[jspc]  -webapp specified, searching . for JSPs
[jspc] Compiling /error.jsp
[jspc] Compiling /index.jsp
Compilation completed successfully.
[INFO] Weblogic APPC processing of C:\monitor\monitor-app\target\monitor-app-1.0-SNAPSHOT.ear successful
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 10 seconds
[INFO] Finished at: Sat Sep 08 06:05:06 BST 2007
[INFO] Final Memory: 11M/21M
[INFO] ------------------------------------------------------------------------

Sweet as a nutnut. The JSPs were compiled under WEB-INF/classes/jsp_servlet the default location. The final EAR includes all the jar files used in the compilation process which is a pain. Can anyone suggest the best way of filtering these files and other Maven build artefacts from the final EAR?.

To round off we are currently looking for a build monkey to fill a contract role (based in Greater London), your first assignment will be to implement my packaging specifications as I'm too busy to do it myself. If you're interested please get in touch with me, money is no object and we pay in sterling, check your exchange rates ;).

IT Infrastructure Nomenclature: Environment names

Building on my last blog on nomenclature I want to discuss the naming of environments. By an environment I mean all the infrastructure that belongs to a functional group of a project, for instance the User Acceptance Test environment or the Production environment.

A typical enterprise stack may consist of a Web Server, WebLogic server, perhaps a Digital Signature Messaging System or some Message Oriented Middleware like MQ and a database. Lets assume for simplicity that our environments have dedicated stacks, so my Functional Test environment is not sharing it's database or any other infrastructure with my User Acceptance Test environment. What are the options for naming my environment? Well you could just give it an arbitrary name based on an abbreviation of it's function and number to keep it unique. This is a fairly common practice and we end up with names like UAT01 and PRD01. Such a naming scheme is perfectly fine if you have a single project and you don't reuse the environment for other projects but what if you have multiple projects and multiple environments, what problems can you expect with your basic naming scheme?

Well for a start there is no way of telling which environments belong to which projects from the name alone, "So what," you might think and as you're probably a miserable developer, who has lived inside an enclipse shell for the last couple of years producing very buggy code, your limited vision is unsurprising. Lets demonstrate the cost to the project whith this simple dialogue:

• Support Caller: Can we change the apache configuration for the Micky Mouse project.
• Support Engineer: Erm, ok what environment are you talking about?
• Support Caller: The test environment.
• Support Engineer: Which test environment, Functional Test or UAT?
• Support Caller: Functional Test.
• Support Engineer: We have 10 FT environments which one?
• Support Caller: Erm, I'll get back to you on that.

Ok, this example maybe a little contrived and you're probably thinking why doesn't the defect tracker/ticketing system have all this information in it but it is a real scenario and a common one in meduim sized businesses. The point is we haven't considered encoding the project name as metadata in the environment name and this piece of information is a key requirement not just for the support team but for anyone in the project that uses the environments.

As with naming your machines the first step in naming your environments is to identify your metadata requirements. In a recent exercise for one client we agreed on the following requirements:

1. The function of the environment should be present as a two character abbreviation e.g pd = Production, ft=Functional Test, ut = User Acceptance Test.
2. All environments that belong to the "Hafez" programme will be identified by the numeral 1.
3. The two projects in the Hafez programme, "Enterprise security" and "Processing engine" will be identified by the numerals 1 and 2 respectively.

We can express these requirements in Backus-Naur form:

<Environment Name> ::= <Environment Function><Programme ID><Project ID>
<Environment Function> ::= "FT" | "UT" | "PD"
<Programme ID> ::= "1"
<Project ID> ::= "1" | "2"

These basic rules allowed us to name all the environments for the programme consistently, uniquely and in and easily identifiable fashion, for instance PD12 is the production environment for the processing engine project which is part of the Hafez programme.

While an environment naming convention is useful it really becomes poweful when we start to derive the names of key infrastructure artificates from our environment names. Lets list some of these artefacts and their derived names, we will stick with WebLogic artefacts for now. You may want to try wrtiting out the general rule for each artefact in Backus-naur form for yourself:

As you can see we have now consistently named listen addresses (hostnames), key stores and key aliases all from our environment name. Anyone can work out the name of the admin server for a given enironment based on our nomeclature. If you want to inspect the identity key you know what it's alias should be. We can also derive names for database schemas, unix users and host of other infrastructure artefacts from the seed name, the environment name.

The benefits of a well thought out nomenclature are numerous and include:

1. Simplifies environment proving and automated validation of environments.
2. Simplifies the task of communicating and understanding the infrastructure architecture.
3. Maintains naming consistency across the programme.
4. Quicker creation of standardised specifications for new environments.
5. Facilitates automation of environment provisioning.

The last two points are where my interests lie, but if your going around touting yourself as an I.T Architect, I.T Technologist, I.T Architectural Technologist, guru, software imagineer, system-medic, Creative Management I.T Dude or some other exotic, bogus job title you should consider all the benefits before making up that random naming scheme on the plane back from some dodgy industry do or in a 10 minute teleconference with the offshore development team.

Some simple rules for deliverables.

My first thoughts were OMFGBBQ. Then I settled down to see exactly what the 60mb EAR file that I opened had inside it. To cut a long story short I'll list what I expect to see in and EAR and what I don't want to find. The said EAR broke every rule. I wont elaborate too much on the reasoning behind the rules except to say that they improve the stability, performance, security and operational agility:

• No static content. I don't want image files, html files or any other static content in an EAR. All static content should be seperated an deployed to the Web servers. This is a standard requirment for both security and performance reasons. It also greatly reduces the size of EAR files which means faster and more stable deployment.
• No JSPs, uncompiled annotations or any other source files. Having a compiler configured on live servers is a security risk, as is having tools.jar on the system classpath. Therefore all artifacts in the EAR must be compiled. Use appc and apt respectively.
• No duplicate jar files. I don't want to see xxxx.jar under WEB-INF/lib in a WAR file and then again under APP-INF/lib in the parent EAR.
• No jar files that are supplied by containers system classpath. I found database drivers, Servlet APIs and a bunch of other jars that where "cleverly" included by their Maven build process which are supplied by WLS and any other Java EE containter for that matter.
• Ensure the EAR has weblogic-application.xml and WARs/EJBs also have their respective weblogic deployment descriptors, these must be well formed and functional.
• No Maven files. I don't want to see any pom.xml files.
• Configuration files must be consistanly named and located. I don't want to see environment.properties, env.properites, envi.properties or myapp.prop in ten different locations in the EAR or WAR file. Stick to a consistent naming convention and location for application configuration files.

Anyway it seems Maven has big issues with appc and apt apart, from that the build team seem confident that they can meet the criteria I have stipulated for them. If anyone else has any good rules I could add to my list I would like to hear them.

IT Infrastructure Nomenclature: Machine names

Those of you who read my last entry will know that I had started a debate with our Unix team in order to create a coherent set of naming conventions for our machines. I have worked in a lot places where little thought is given to the naming of infrastructure artifacts with predictable results.

The benefits of an enterprise wide nomenclature for infrastructure artifacts are manifold and include:

• All artefacts are named in an consistent and identifiable fashion.
• Enhanced clarity in cases of ambiguity. All stakeholders will be able to communicate information regarding the environments without risk of confusion.
• Avoids naming collisions.
• Simplifies the understanding of infrastructure architecture.
• Lays the foundations for automating infrastructure environment provisioning.
• Provides additional information, metadata, about the named artefact.

Encoding the correct metadata in the name of an artifact is the probably the most important step when creating your nomenclature. If you get this step right all the other benefits will follow, so take your time and make sure you include all the stakeholders in the requirements gathering process. Here are a few suggetions for metadata that you may decide to encode in a machine name:

• Programme or project identifier.
• Machine function.
• Environment identifier.
• Machine identifier.

For instance, say you have purchased six machines for a project code named "Ferdowsi". Each machine fullfils a function in your n-tier architecture so you have a Web server, application server and database server. These machines will be deployed to your test and production environments, lets look at one possible way of naming them. We will start by expressing our metadata in Backus-Naur form.

<Machine name>::=<Programme Id><Machine function><Environment Id><Machine ID>

<Programme ID>::="fr"

<Machine function>::=="ws"|"ap"|"db"

<Environment ID>::="1"|"2"

<Machine ID>:==<digit>

Lets describe the metadata in more detail as well as some of the constraints we want to place on them. The programme ID is two characters in length and idicates that these machines belong to the "Ferdowsi" project. If we had machines that were shared between programmes/projects we would drop this identifier from the machine name. Including the machine function in the name of the machine is a common and useful practice. Again we used a two length abbreviation of the full function name, so for Web server we derive "ws" and so on. The environment id is number that translates to environment to which the machine is deployed so. In our case 1="Production" and 2="Test". Finally the machine id is a digit assigned to identify the machines within there environments, so our machines are named as follows:

I don't want this entry to be interpreted as a prescriptive piece, I'm not telling you how to name your infrastrcture artifacts, rather it should help the reader to consider moving away from arbitrary naming practices and open possibilities for deriving the names of all the other artificates across the enterprise such as hostnames, file names for certificates, ketstores, logs, weblogic domain names, etc, etc.

Too many architects overlook nomenclature and as a result they inject and unhealthy dose of entropy into their systems. Finally with the nomeclature in place you can begin to automate the provisioning of your enterprise stack, from networks and operating system to applications.

Recovering from O'RAC'LE outages

It's a quiet day and all seems well with the world and them suddenly a node in the Oracle RAC dies without warning. Phones ring, text messags fly and the DBAs assemble togehter. Pretty quickly they manage to restart the damn thing but all the connections to the RAC have failed over to the other node. While there is no immediate issue you will need to rebalance the connections at some point during day so what are your options?

Here's what you can do from the WLS8.1 management console in order of greatest invasivness to service delivery:

1. Restart the WLS server. Yes, this will rebalance your connection pool in a known period of time. No its not very nice to your customers and stop touting as a 24/7 service provider. This operation is also the slowest way to reset your connection pools.
2. Untarget and retarget your connection pool. This has the effect of closing and reopening all your connections which will rebalance them accross all nodes. All users of the pool on a particular target (server or cluster) will lose service. It is a faster operation than restating your server but if your not careful you could run into JNDI errors if you retarget the pool too quickly.
3. Force suspend the connection pool and then resume it. Upon resumption the pool recreates all the connections that were closed and in effect connections are rebalanced against all defined nodes. This operation can be done on per server basis and limits service degridation to users of the connection pool. It is also the fastest way of rebalancing your connections out of the option I am putting forward.
4. Reset the connection pools. WLS will wait for a connection to become free before closing and reopening them. This can take some time if the site is very active and infact if the connection does not become free after a given timeout period it seems that WLS skips it and so the outcome of resetting connections cannot be well predicted if the site is busy. However this is the least invasive approach in terms of service delivery.

I think the order in which you define the your tns listeners in the in the JDBC connection URL will also have an impact on balancing of connections, especially in the case of resetting the connetion pools. Rather than defining the listerners on one node and then the next node it may be better to alternate nodes. I don't have any evidence that this will improve the rebalancing but it is something I plan to test soon.

If you know of better options for rebalancing those pesky connections pools let me know, if it wasn't for that meddling o'rac'le database I could have gotton away with it.

Get your hands off me, you damn, dirty Console!

Lets start with the only good feature of the WebLogic 9.2 console, it's blue. Gone is the gaudy green colour of past versions and instead we have the liquid blue enforced by BEA's marketing mandarins. But as soon as you browse to the console URL alarm bells start to ring, why did it take a good 5-10 seconds to load up and where is the miserable applet tree on the left hand panel? Then the first real clue of how bad things are going to get, you notice the URL: http://localhost:7001/console/console.portal? thats right "portal". The immortal line from planet of the apes rings through my monkey head "They did it! The b*******s finally did it! Damn you all! Damn you all to hell!".

It seems the console has been reimplemented using some portal widgets and large dose of, erm, design. Almost all the old features I used to use have been rendered unusable or have been completely removed, here's the list:

• All the functionality provided by the old applet that was rendered with a right mouse click now takes at least three clicks, each click taking a minimum of 3 seconds. Things like viewing the JNDI tree, Server logs, thread queues etc.
• No performance applet which indicated heap usage at a glance.
• All basic operations such as deployment require endless clicks to confirm your actions.
• Everything now runs considerably slower.

I can only assume the people who designed this application have never had to use the console in their Rationally Unified lives. Perhaps they thought they could make end users more agile by getting them to double or triple the number of mouse clicks it took to untarget a connection pool. Another theory I have is that the WLST team sabotaged the console in order to ensure all WLS managment was scripted. I take my hat off to you WLST team, mission accomplished!

I know the WLS 10 preview is now available, I'm going to down load it at the end of February, this will give the console team some time to fix the defects I have raised. My bug report will be posted to this blog so stay tuned. Now then, what was the URL for the JBoss console...

When is a BUG and good thing?

When it's, err, a BEA User Group! A week after the event I just want let everyone know how it went.

Attendance was in double figures, a great turnout for first meeting of London Middleware as an official dev2dev BUG. While this was a social event we inevitably started to talk shop. As the crowed was mostly made up of mission critical support and build engineers it was not surprising that BEA Guardian cropped as well as the topic of environment provisioning. Another interesting area of discussion was using syslog and Splunk. Logging is one of my bugbears right now, it was great to have a few hours free expert consultancy provided by London Middleware members on this subject.

BEA Guardian received good reviews, the consensus indicated it was well worth using and did pretty much what it said on the tin. BEAs mission crtitcal support customers can use this tool as part of the support package.

It's always good to talk face to face with people who are in the front line of implementing and supporting I.T. For me the few hours we spent at the Borough Bar gave me, among other things, the opportunity to have a few of my own ideas and approaches informally "peer reviewed". We are planning to hold a meeting every two months, I hope to see more faces at the next one.

London Middleware at the Borough Bar!

London Middleware are holding a social event at the Borough Bar on Friday the 2nd of February. This will be an opportunity for Dev2Dev members to put faces to names and exchange ideas, opinions and views in an open and relaxed environment.

If you're interested in attending please post a comment on this entry, that should give me a rough idea of the number of attendees. We have a modest budget for refreshments so get there early!

On behalf of London Middleware I want to thank Jon Mountjoy the editor-in-chief of dev2dev for his support of this event, and I hope to see you all in February.

WebLogic Server Debug Environment Variables

Here are all the WLS debug flags I have collected over the years from BEA and from grepping the internet. Simply add them to the wls java command using the -D option e.g.: -Dweblogic.debug.DebugMessaging=true

These debug parameters include:

IIOP
weblogic.iiop.ots
weblogic.iiop.transport
weblogic.iiop.marshal
weblogic.iiop.startup

JDBC/Data Sources
weblogic.JDBCConn
weblogic.JDBCSQL
weblogic.JDBCConnStackTrace

JMX/Deployment
weblogic.commoAdmin
weblogic.commoProxy
weblogic.deployerRuntime
weblogic.MasterDeployer
weblogic.deployTask
weblogic.deployHelper
weblogic.MasterDeployer
weblogic.OamDelta
weblogic.OamVersion
weblogic.slaveDeployer.semaphore
weblogic.slaveDeployer
weblogic.ConfigMBean
weblogic.ConfigMBeanEncrypt
weblogic.ConfigMBeanSetAttribute
weblogic.management.DynamicMBeanImpl
weblogic.management.DynamicMBeanImpl.setget
weblogic.mbeanProxyCache
weblogic.mbeanDelete
weblogic.mbeanQuery
weblogic.MBeanInteropList
weblogic.mbeanProxy
weblogic.registerMBean
weblogic.getMBeanInfo
weblogic.getMBeanAttributes
weblogic.addDependenciesRecursively
weblogic.MBeanListener
weblogic.application
weblogic.deployer
weblogic.appPoller
weblogic.appManager
weblogic.BootstrapServlet
weblogic.fileDistributionServlet

Application Deployment
weblogic.J2EEApplication
weblogic.application
weblogic.appPoller
weblogic.appManager

JTA
weblogic.JTAGateway
weblogic.JTAGatewayStackTrace
weblogic.JTA2PC
weblogic.JTA2PCStackTrace
weblogic.JTAHealth
weblogic.JTAPropagate
weblogic.JTARecovery
weblogic.JTAXA
weblogic.JTAXAStackTrace
weblogic.JTAResourceHealth
weblogic.JTAMigration
weblogic.JTARecoveryStackTrace
weblogic.JTANaming
weblogic.JTATLOG
weblogic.JTALifecycle

EJB
weblogic.ejb.cache.debug
weblogic.ejb.cache.verbose
ejb.enableCacheDump
weblogic.ejb20.cmp.rdbms.debug
weblogic.ejb20.cmp.rdbms.verbose
weblogic.ejb20.persistence.debug
weblogic.ejb20.persistence.verbose
weblogic.ejb20.compliance.debug
weblogic.ejb20.compliance.verbose
weblogic.ejb.deployment.debug
weblogic.ejb.deployment.verbose
weblogic.ejb20.dd.xml
weblogic.ejb.deployer.debug
weblogic.ejb.deployer.verbose
weblogic.ejb.verbose.deployment
weblogic.ejb20.ejbc.debug
weblogic.ejb20.ejbc.verbose
weblogic.ejb.runtime.debug
weblogic.ejb.runtime.verbose
weblogic.ejb20.jms.poll.debug
weblogic.ejb20.jms.poll.verbose
weblogic.ejb20.security.debug
weblogic.ejb20.security.verbose
weblogic.ejb.locks.debug
weblogic.ejb.locks.verbose
weblogic.ejb.bean.manager.debug
weblogic.ejb.bean.manager.verbose
weblogic.ejb.pool.InstancePool.debug
weblogic.ejb.pool.InstancePool.verbose
weblogic.ejb.swap.debug
weblogic.ejb.swap.verbose
weblogic.j2ee.dd.xml

General
weblogic.debug
weblogic.kernel.debug
weblogic.debug.DebugConnection
weblogic.debug.DebugRouting
weblogic.debug.DebugMessaging
weblogic.debug.isLogRemoteExceptionsEnabled
weblogic.StdoutDebugEnabled

WLI
wlc.debug.signature
wli.bpm.client.security.debug
wli.bpm.studio.timeprocessor.debug
wli.bpm.studio.debug
wli.bpm.server.common.timedevent.debug
wli.bpm.server.common.xmltemplate.debug
wli.bpm.server.eventprocessor.addrmsgdebug
wli.bpm.server.eventprocessor.debug
wli.bpm.server.jms.debug
wli.bpm.server.plugin.debug
wli.bpm.server.workflow.debug
wli.bpm.server.businesscalendar.debug
wli.bpm.server.busop.debug
wli.bpm.server.workflow.action.taskduedate.debug
wli.bpm.server.workflow.timedevent.debug
wli.bpm.server.xml.debug
wli.bpm.server.xslt.debug
wli.bpm.server.workflow.start.debug
wli.bpm.server.workflowprocessor.debug
wli.common.server.errorlistener.debug

Building and deploying your WebLogic domain Part 2

I'll start this entry by briefly recapping the two metrics I used to measure the performance of our old build process discussed in my last entry:

• Build time. If it takes you longer than a few minutes to build your domain, you should be asking why?
• Work time. If it takes you longer than a few minutes to make a configuration change to your domain, you should be asking why?

I have yet to hear a valid reason why a standard WLS domain (e.g. not a Portal domain) should take longer than a few minutes to build, if you think you've got one please post a comment on this entry. The main reason I come across is that the build process is "Dynamic" i.e the domain is created using WLSHELL or WLST. In my experience these domains take between 10 - 40 minutes to build. If you use these tools to build your domains please post a comment on the time it takes you. If you're happy with this time and it meets your requirements, well thats marvelous. After about the fourth time I rebuilt and deployed a domain of ours to fix multiple configuration issues, a process that took about 90 minutes (after two days spent trying to figure out exactly where to make the changes), It was plain to me that a 20 minute cycle was not acceptable for us. So there you have it, this is why I believe these two metrics are important. As the old cliche mandates time is money.

The third metric I want to discuss falls under the category of complexity, specifically the number of files in your build process that it takes to build your domain. I appreciate that measuring complexity in this way is blunt to say the least but we have start somewhere and in my experience it is a worthwhile measure. Latter on I'll discuss another form of complexity that is can't be measured easily but can be avoided. In my original rant I noted that our build process had no less than 69 files to describe a our WLS domain, thats property files and templates. How many files does your build process use?

In an effort to cap the number of files required to build our domain we came up with the following directory structure for Choom (the version on Choom that will be released in codeshare will be even simpler):

Choom stores property files in the properties directory and templates in, erm, the templates directory, no surprises there. In my last entry I described the hierarchical nature of the properties, this hierarchy is observed in the directory structure.

Directly under the properties directory we find common.properties. You guessed it, these properties are common to ALL our domains. Now there aren't many properties in this file but this does not diminish its importance, you see, the more properties you can define in this file, the more standardised your domans will be and standardised domains mean less maintenance effort.

Under the domains directory we store the properties for the various domains. Each domain type is named, for instance, we can see depicted the mydomain type. Typically the domain type is named after the project it belongs to. Domain level properties are stored under the domain type directory. As before the more poperties we store at this level as opposed to the environment level the more standardised our domains become.

The templates generally follow the same naming and oragnisational conventions. If you read my last entry on this subject you may have noticed that the use nomenclature is a key tool in reducing complexity. By adhering to some simple conventions for naming files, properties and tokens it becomes easy to locate and make changes to configuration values.

When we first embarked on creating our new build process I wanted to make sure that we stuck to just that, creating a process to build our domains rather than an application that builds our domains. This is why I really wanted to use Ant rather than a programming language. I have come across home made build applications written in Java, Lisp, Scheme and of course Jython/WLST and in almost every instance you find a build team that is not only responsible for environment provisioning but also has to support their miserable build application, yet another reinvented square wheel. This is a surefire way to inject a huge amount of complexity into your build process and I maintain it is completely unnecessary for the vast majority of WLS domains. I should qualify this by saying that Jython and WLST have an extremely valuable role in build verification and environment prooving and my next entry will describe how we exploited these features in choom.

BEA's presence in Iran

I recently received an email asking for help to tackle an HttpClusterServlet configuration issue. Nothing interesting about that except that the request was sent to my personal email address and was from Iran. I had no idea WLS was being used in Iran. I was in Tehran last March and as far as I could tell Linux, PHP, Tomcat (Java, JSP, Servlets) and to a smaller extent JBoss and of course MS where common tools being used. It's a bit of a surprise to me that WLS was being used. Can anyone tell me if BEA have any kind of presence in Iran, or as Iran does not recognise US copyrights is this a possible indication of using hacked licenses?

If the emails keep comming in I will have to consider setting up a consultancy/support operation in Tehran :).

Building and deploying your WebLogic domain

Those of you who read my blog would be aware that around late November of last year I decided our existing build process had to go. You can read my original rant here. Well I am please to report that after much discussion we bit the bullet a decided to scrap our build process and embark on creating a new process that we hoped would meet the increasing demands being made on the infrastructure team to build and deploy the numerous WLS domains that we manage. We have been using the new build process (which we call choom) for almost a year. In that time the demands to create new environments and support new versions of WebLogic, Apache and other middleware applications have only increased while our team has dropped in size from six to two people. If you are considering different approaches to building and deploying WLS domains I hope you find this and the next few entries useful. If you think these ideas are rubbish I want to know why?

To start with lets define some of the key requirments for our build process. The build process must:

• Create a complete WLS domain and scripts to distribute and install the domain onto multiple machines.
• Be fast. It should take no longer than a few minutes to build, distribute and install a WLS domain. This a key measure of the performance of a build process. The vast majority of WLS domains I have seen can be built and distrbuted in under a miniute on a desktop pc.
• Enforce architectural and design decisions and verify the configuration by checking the completed domain before it is distributed.
• Be simple to use. Anyone with basic unix skills should be able to build and deploy a domain after a 5 minute tutorial.
• Be able to build Apache, WebLogic and any other application maintained by the support team.
• Be simple to maintain. The build process should not require a team to support it. The process of adding new environments or applications must require no special skills.

With these requirements in place we invited BEA to help us evaluate different build approaches. The debates were always heated as everyone involved knew we had to get this right or we would end up in a mess. To outsiders it may well have seemed that the team was about to crack but in my experience civil and plesant technical discussion is often a sign that participants don't know what they are talking about or don't care. We whittled down the contenders to a) using WLST to dynamically create the domain and b) Create template domains and use ANT to "copy filter" the templates. We opted for ANT option for many reasons but the deciding reason was simple, WLST can only build WLS, the ANT option gave us a consistant method for building any application we wanted.

The next step was to create a template domain and define the properties. The customer had two sites, a live site and a contingency site. These sites were home to the production environment, a domain that spanned two machines. We would supply arguments to the build script which would determine which property files were loaded and used to create the domain. We had to group related properties together in their own files and we came up with the following files:

• common.properties - Used by all builds prefix=Common.
• mydomain.properties - Domain specific prefix=Domain
• production-live.properties - Production environment and live site prefix=Environment.
• production-cont.properties - Production environment and contingency site prefix=Environment.
• machine-live1.properties - Machine 1 in the live site prefix=Machine.
• machine-live2.properties - Machine 2 in the live site prefix=Machine.
• machine-cont1.properties - Machine 1 in the contingency site prefix=Machine.
• machine-cont2.properties - Machine 2 in the contingency site prefix=Machine.

To build the domain one would execute the following command:

ant -Dproduct=weblogic92 -Dsite=live -Ddomain=mydomain -Denv=production

The build loads the correct set of properties based on these arguments. This is a pretty standard approach familiar to anyone that uses ant, it looks like this:

<property name="properties.dir" location="${basedir}/properties"/>
<property file="${properties.dir}/common.properties"/>
<property file="${properties.dir}/domains/${domain}/${domain}.properties"/>
<property 
file="${properties.dir}/domains/${domain}/environments/${env}/${product}/${env}-${site}.properties"/>

I'll explain the directory structure in my next entry, to round off this entry I want to explain one of the problems with this standard ant approach and how we resolved it. If you have ever had to locate a property to change it's value you have probably had a few hours of your life wasted by having to locate that property because it is defined in multiple files and ant can be configured to inherit properties. Our old build process didn't have a consistant hierarchy which compounded the problem. It could take days to make a change to a single property! We were burned so badly by this that we decided there would be no inheritance of property files in Choom and that every property name would conform to a nomenclature that would indicate where that property was located in the build file system. For instance, if you were looking at a template of the config.xml you would see something like this:

    <ssl>
      <name>@Domain.admin.Server.Name@</name>
      <enabled>@Environment.admin.Server.SSL.Enabled@</enabled>

Any property prefixed with Domain corresponds to the domain level properties defined in mydomain.properties. If it's prefixed with Environment you know it has to be in the production-${site}.property file for that site. Now we can locate and change the value of any property in seconds. For me this is a key metric to use in determining the performance of your build process. If you find it takes longer than a couple of minutes to make a simple configuration change in your build, alarm bells should start ringing.

In my next entry I will discuss the directory structure of choom and the templates. We gave alot of thought to the directiry structure of the build process. We wanted it to be intuative, consistant and to reflect how we perceivethe architecture our domains.

My BEA World in Prague

I managed to attend one day of BEA world in Prague and it was well worth it. I arrived at the Hilton at about 22:00 on the Tuesday and was pleasantly surprised by the accommodation. My colleagues were at restaurant courtesy of BEA so I decided to have a couple of brandies at the bar and smoke a cigar in preparation for a busy day ahead.

I got talking to a few people who had attended the days sessions and was informed that one of the talks that generated a lot of interest was Steve Grigg's presentation "Ultra-High Volume Secure Transactions and the Single European Payments Area (SEPA) ". It was a shame that I couldn't attend the presentation as I have an interest in this field and by all accounts it was a good presentation.

I also managed to discuss another hot topic, that of the dev2dev newgroups with my new found drinking partners. In between the coulds of rich Montecristo No. 2 smoke I puffed at them we dicussed how to make the newgroups more useful. I'm still gathering more suggestions on this subject and if anyone in wants to meet and tell me what they think of the newsgroups or anything else, I and a few colleagues are meeting up in London and your more than welcome.

The next day was pretty good, I ran into a bunch of old faces and before I new it, it was time for the BEA Mission Critical Support Partners lunch. It was a fine meal and I managed to get a my "Mission Critical Support - Out To Lunch" card. I met a few people from H.S.B.C and Citigroup and we decided talking shop was not acceptable.

While most the crowd left that evening I hung around for one more night and enjoyed a great meal at Restaurant Zvonice, the wild boar was fantastic as was the bottle of vintage Saint-Emilion, I was congratulated by the sommelier for a good combination. By the end of the day I understood what it meant to "Think Liquid" and had a good grasp on the fundamentals of "Aqua Logic".

April 2008

Search this blog:

Archives

April 2008
January 2008
September 2007
August 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
February 2006
November 2005
October 2005
September 2005
August 2005
July 2005

Categories

Recent Entries

Help me write a schema file: server.xsd

Middleware Strategy

Using Appc with Maven

Articles

Best Practices for Building Production Quality EAR Files
This article examines the options available for packaging and deploying Java EE applications. It also describes the purpose of packaging specifications for application archives, and provides examples for implementing them. Feb. 13, 2008

Introduction to Trading Partner Integration, Part 1
Automating business processes requires knowledge about your partners and the services they offer. In this article, Hussein Badakhchani introduces the trading partner integration aspect of BEA WebLogic Integration. Sep. 19, 2005

Introduction to RosettaNet
This article introduces RosettaNet and the benefits it brings to business-to-business (B2B) supply-chain trading partners. We will start by looking at the origins of RosettaNet, the business processes it deals with, and how it fits in with other B2B integration initiatives. Digging deeper we will examine RosettaNet's specification structure, usage scenarios, and design patterns for implementing RosettaNet solutions. Dec. 6, 2004

All articles by Hussein Badakhchani »