There are many good reasons for using mainframe resources as part of WebLogic application development and integration projects. High on the list are leveraging existing logic and data, increasing the value of current investments, and focusing development effort on solving new business problems. However, there are also many reasons why organizations fear the prospect of mainframe integration; among the most serious are proprietary interfaces, radically different processing environments, lack of support for standard development APIs, and the fact that the people who created the applications have long since "moved on."
For users of the BEA WebLogic Platform who are building high-value applications that depend on IBM mainframe data and applications, getting this integration right means avoiding the headaches associated with a deluge of extra costs and specialized training.
Mainframe Integration Problems Start at "The Line"
There's a line of demarcation that divides WebLogic development organizations and mainframe systems management organizations. You can't see it, but you know it's there. From a human perspective, it's the boundary between comfort zones. WebLogic developers don't want to learn about the mainframe, and breaking out of their development environment is decidedly unattractive. Meanwhile, the mainframe "glass house" personnel are focused on maintaining the security and stability of their tightly controlled production environment and would prefer to avoid interaction with the chaotic world of open systems applications.
From a technological perspective, "the line" is an effectiveness threshold. Each computing environment has strengths and weaknesses. When a solution pushes the capabilities of a platform too far, problems are inevitable.
Fear and frustration often surround the idea of mainframe integration because most solutions force someone to cross "the line." The results are scalability and failover problems, limited feature set support, additional administrative headcount requirements, ongoing training, and maintenance rollout problems. From a business perspective, the results are development delays, a backlog of support calls, higher Total Cost of Ownership, and, more importantly, dissatisfied internal and external customers.
Using Components To Solve The Problem
One solution to this problem is the product Shadow from NEON Systems. It addresses this problem by placing a communications component on each side of "the line." This allows Shadow to present each organization with an interface that looks and acts in a manner consistent with the environment in which it operates. Shadow integrates with standard WebLogic development tools and allows mainframe resources to look and behave like distributed systems relational databases. For mainframe systems administrators, Shadow provides enterprise-class monitoring, management, and control facilities to maintain system availability and ensure minimal utilization of CPU and other operating system resources. Finally, Shadow integrates with mainframe security and native workload balancing facilities.
One of the advantages of NEON's approach with Shadow is that it provides mainframe access without a major overhaul of existing application architecture. In addition, NEON's Shadow supports both application integration or data integration, a key requirement. This is significant because many organizations need access to programs running under IMS/TM or CICS, and they also need access to the underlying data stored in DB2, IMS/DB, VSAM, and other databases. NEON's Shadow provides the WebLogic Platform with standard API access to the most important mainframe transaction managers and databases.
Component Architecture
An architectural overview of NEON's Shadow provides insight into how the product works. There are three fundamental software components that constitute Shadow. Each component has an important role to play in providing integration and maintaining the line of demarcation between computing systems. These components include:
- Shadow Client
- Shadow Server
- Shadow Console
The Client
Shadow Client is the primary interface, integrating with the BEA WebLogic Workshop and other leading development environments. "Under the covers" and transparent to the WebLogic platform, the Shadow Client exploits native OS features for performance optimization and addresses the translation and connectivity issues that can be handled from the distributed platform. Additional optimizations and connectivity issues that cannot be handled efficiently from the Shadow Client are addressed by the Shadow Server, the mainframe component.
Shadow supports the full range of metadata requests for table schemas and stored procedures. To developers using Shadow, mainframe databases appear as distributed systems relational databases, and mainframe programs look and behave like stored procedures. In production, the Shadow Client coordinates with the Shadow Server to support connection pooling and two-phase commit (2PC) transactions through extensions compatible with Java Transaction Service (JTS) and the Java Transaction API (JTA). The Shadow Client also gathers diagnostic and performance data that are made available to the Shadow Diagnostic Facility and the Shadow Client for debugging and troubleshooting purposes.
The Shadow Client includes a choice of J2CA, JDBC and ODBC adapters and an agent that gathers information useful in troubleshooting. The Shadow Client installs on the dominant UNIX environments, including UNIX System Services (USS) on z/OS, Linux, and all Windows 32bit platforms. Through its support of JDBC, J2CA and Web Services standards, the Shadow Client allows developers to build effective applications with no knowledge of the mainframe or the underlying implementation of Shadow's mainframe access technology.
In addition, the Shadow Client provides a single interface that provides mainframe access for virtually all components of the WebLogic Platform. The Shadow Client can be used in conjunction with the WebLogic Workshop and other development environments, the WebLogic Server, WebLogic Integration, and WebLogic Portal. This allows organizations to leverage a single technology across multiple projects.
Supported JDBC Integration
The Shadow JDBC Adapter uses a typical JDBC Type 2 installation procedure to integrate with the WebLogic Platform. It is compliant with JDBC 2.0 and is certified by Sun Microsystems for JDK 1.1+, J2EE 1.2, and J2EE 1.3.
Supported J2CA Integration
The Shadow J2CA Adapter uses a standard J2CA installation procedure to integrate with the WebLogic Platform. Through Shadow, all available mainframe subsystems look and behave like a single EIS. Shadow J2CA Adapter extensions are consistent with the proposed J2CA 1.5 specification and existing BEA WebLogic Integration extensions.
Supported JMS Integration
For more loosely coupled inbound integration that requires no Shadow component installation on the server, organizations can exploit the IBM WebSphere MQ JMS provider. Message Drive Beans (MDB) can register interest in the arrival of messages on certain queues or topics. These messages can originate from events captured by Shadow's event detection capabilities. In this scenario, event detection can be configured by the Shadow Systems Administrator or via the J2CA 1.5 inbound communications extensions. In the latter case, the inbound destination would be a WebSphere MQ topic or queue, rather than a J2CA machine.
Supported Web Services Integration
For organizations wishing to use Web Services within the WebLogic Platform to make Simple Object Access Protocol (SOAP) requests to the mainframe, Shadow can be configured to accept inbound SOAP requests from TCP/IP at the mainframe and map the requests to a particular CICS or IMS/TM application. The SOAP request can be built in WebLogic Workshop, based on WSDL stored in the Shadow metadata repository. This capability allows organizations to begin creating and exploiting Web Services that access the mainframe in a manner that maintains the value of existing investments.
The Server
Shadow Server is the mainframe footprint that simplifies monitoring, management, and control of composite applications for mainframe systems administrators. The Shadow Server installs quickly and runs "out of the box" to deliver rapid integration of mainframe resources. While some organizations would prefer a solution that did not include a component running on the mainframe, the reality is that this component is absolutely required in order to ensure high performance, scalability to meet evolving business and technological needs, and a manageable operational environment. In addition, a mainframe component is an aspect of virtually all native IBM solutions. However, the mainframe components of other solutions are developed and maintained independent of each other, are scattered throughout multiple mainframe subsystems, and do not provide a consistent or comprehensive integration infrastructure for the mainframe. In contrast, the Shadow Server simplifies and unifies mainframe integration.
The Shadow Server exploits all the mainframe "bells and whistles" that have made this system the de facto standard for security, availability, and reliability over the last 30 years. The Shadow Server extends these features to composite applications, providing maximum reliability and minimum resource utilization without compromising Shadow's ability to support thousands of users and thousands of transactions per second. Most importantly, Shadow masks the complexity of the mainframe. Shadow's sophisticated, pre-configured functions make mainframe integration easy for the implementation team and completely transparent for developers. Advanced configuration is only necessary in special circumstances.
The Shadow Server is designed to integrate with the mainframe in ways that are normal and natural for both the systems and the administrators who maintain them. The Shadow Server is a native z/OS started task, managing each interaction with composite applications through a separate z/OS thread known as a TCB. Exploiting TCBs provides administrators with control over resource utilization and ensures that each process or transaction is isolated to protect the other workloads running on the mainframe. Running as a started task allows the Shadow Server to exploit z/OS facilities including Sysplex support, Systems Measurement Facility (SMF) logging, Resource Recovery Services (RRS), and automated Workload Management (WLM).
Organizations can run multiple Shadow started tasks concurrently on a z/OS image to support separate development, quality assurance, and production environments. A single Shadow Server can support hundreds of concurrent, persistently connected users. Larger numbers of users can be supported through the Shadow Virtual Connection Facility (VCF) or by running multiple production Shadow Servers and utilizing Shadow Load Balancing. To provide ultimate availability, the Shadow Server also supports the z/OS Sysplex Distributed Dynamic Virtual IP Addressing (DVIPA) and Workload Manager facilities.
Additional features of the Shadow Server include:
- Global two-phased commit of application and data resources
- Full exploitation of the multiprocessing and multiprogramming capabilities of z/OS
- Security through RACF, ACF2, and Top Secret
- SSL encryption
- Advanced recovery and failover facilities
- Metadata and exclusive "pseudo-stored procedures," providing relational access to all mainframe resources
- Internationalization and DBCS support
- Exclusive diagnostic and automation facilities
- Automated resource controls for DB2, CICS, IMS, and Adabas
Monitoring Console
There is one aspect of composite applications where it is absolutely required that developers and mainframe systems personnel step over "the line" – application debugging and problem resolution. In this situation, developers and systems administrators must work together to find the cause of application or performance problems. The process normally entails gathering and reviewing trace logs from the application, any mid-tier servers, the mainframe integration technology, and the mainframe resource itself.
NEON Systems' exclusive Shadow Console significantly simplifies the process by providing a graphical user interface that displays information about all Shadow components and an interleaved trace log that shows application calls and mainframe responses in a single display. Using the combination of Shadow's standard management features and the Shadow Console, simple problems can be corrected automatically, and complex problems can be located and corrected more easily with Shadow than with any other integration technology.
The Shadow Console tree display shows all Shadow Servers, Shadow Clients, and applications across the enterprise that access mainframe resources through Shadow. This interface provides a single point of control for the entire Shadow environment, allowing administrators to change operational parameters, and providing "drill down" capability that displays response time and other critical information that helps administrators maintain service levels and detect problems.
The Shadow Console is also a tremendous benefit to application developers. Diagnostic data from each node can be viewed individually or interleaved with relevant information from other connected nodes. The interleaved displays present a time-sequenced view of events from the WebLogic Platform, to the mainframe and back. With the Shadow Console, developers can troubleshoot and tune applications with little or no assistance from mainframe systems administrators. This significantly reduces the time required to find and fix problems and allows both developers and systems administrators to remain productive and working within their comfort zones.
Exploring Mainframe Integration
This article has explained conceptually how Shadow simplifies the development and operation of composite applications that access mainframe resources. Of course, the real test is how it works in your environment. Following are instructions you can use to rapidly define and use sample CICS and IMS/TM programs, along with DB2, VSAM, and Adabas data running on a mainframe located at the NEON Systems headquarters.
The following test case demonstrates the typical Java programmer's view of Shadow used in conjunction with WebLogic Server. To quickly test Shadow in your own environment, download the Shadow Client and test case created for this article from the URL included in the following instructions. A sample JSP is provided at the end of this article. The sample applications being called in this example are provided with the base Shadow product, and other test cases are available if you wish to continue experimenting. For the sake of simplicity, the test case uses the JDBC API so that the only requirement besides the Shadow Client is to have a working copy of WebLogic Server.
Installation and deployment of Shadow JDBC driver with BEA WebLogic Server
- Download the Shadow Client JDBC driver from http://www.neonsys.com/downloads/drivers/download.asp?Product=NEON_JDBC_Drivers
- Install the Shadow Client JDBC driver, following the prompts.
- Create a new system environment variable NEONTRACE and set it to value TYPE2. Below is an example of how to specify a system environment variable on Windows.
- Modify the command file used to start the WebLogic Server so that it can find the Shadow Client JDBC driver within its CLASSPATH. In this example, the command file is in C:\bea7\weblogic700\samples\server\config\examples\startExamplesServer.cmd
This command file needs to have the Shadow JAR file included in the WebLogic Server CLASSPATH. (In a Windows installation, the Shadow JAR files are in the C:\SHADOW\JDBC directory. Choose SCJD12TS.JAR – the other driver supports debugging, which you will not use in this test case.)
The modified command file line for WebLogic Server should look like this:
setCLASSPATH=C:\bea7\jdk131_02\lib\tools.jar;%POINTBASE_HOME%\lib\pbserver42ECF172.jar;%POINTBASE_HOME%\lib\pbclient42ECF172.jar;%CLIENT_CLASSES%;%SERVER_CLASSES%;%COMMON_CLASSES%;%CLIENT_CLASSES%\utils_common.jar;c:\shadow\jdbc\scjd12ts.jar
- Start the WebLogic Server.
- You can copy the JSP for this test case to the directory for the DefaultWebApp (in this case: C:\bea7\weblogic700\samples\server\config\examples\applications\DefaultWebApp).
http://houscbeawl.neonsys.com/JdbcTable.jsp
The output of the JSP should look like the following sample.
Remove Fear from Mainframe Integration
While the JSP used in the test case is quite simple and certainly does not demonstrate the full capabilities of the product, it does illustrate some important aspects of Shadow. One important aspect shown very effectively is that the CALL and SELECT statements used are very similar, even though the JSP accesses radically different mainframe resources, including DB2 and VSAM data and CICS and IMS programs. This is important since the last thing a developer wants to encounter is another API when balancing between application and data integration components.
In a very real sense, Shadow takes the fear and frustration out of mainframe integration. This is done through the Shadow Server's comprehensive and technically superior mainframe integration footprint, designed specifically for mainframe administrators, and the Shadow Client's standards based, feature rich, user-friendly environment, essential for today's developer. These components, when combined with the monitoring, management, and debugging capabilities of the Shadow Console, provide an unmatched environment for the development and management of composite applications.
Because Shadow does not force any organization to cross "the line," developers and systems personnel work well within their comfort zones, computing systems work effectively within standard application implementations, and the overall organization gains increased productivity and business agility. As you now see, with Shadow, there's nothing to fear about mainframe integration.
Mainframe Integration Test Case JSP
The following JSP accesses several mainframe data sources, and can be used for testing.
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;CHARSET=iso-8859-1">
<meta name="description" content="BEA WebLogic Server">
<meta name="keywords" content="BEA WebLogic Server">
<title>JDBC Table Servlet</title>
<LINK REL="stylesheet"
TYPE="text/css"
HREF="wls_examples.css"
TITLE="BEA WebLogic Server">
</head>
<body bgcolor="#ffffff" link="#3366cc" vlink="#9999cc" alink="#0000cc">
<!-- top intro paragraph tables -->
<!-- RED LINE -->
<table cellspacing="0" cellpadding="0" border="0" width="100%">
<tr>
<td width="100%" bgcolor="#ff0000" height="1">
<p class="small"> </p>
</td>
</tr>
</table>
<table border=0 cellspacing="18" cellpadding="0">
<tr>
<td valign="top">
<h3>Using JSP to retrieve database data with JDBC</h3>
</td></tr>
</table>
<table border=0 cellspacing="10" cellpadding="0">
<tr>
<td valign="top">
<h4>Make a selection</h4>
Choose a JDBC driver from the drop down list below.
<form method="post" name="JdbcTable" action="JdbcTable.jsp">
<table border=0 cellspacing=2 cellpadding=2 width=80%>
<tr>
<td width=30%><font face="Helvetica"><b>JDBC driver :</b></td>
<td><font face="Helvetica"><select name="jdbcDriver">
<option value="com.neon.jdbc.Driver">com.neon.jdbc.Driver</option>
</select></td>
</tr>
<tr>
<td width=30%><font face="Helvetica"><b>SQL Query :</b>
<strong>Enter SQL or Shadow CALL statement:</strong> <br>
<i>(or click on button below for sample SQL or CALL statements)</i><br></td><td>
<INPUT TYPE="SUBMIT" value=" DB2 "
OnClick="sqlQuery.value='SELECT * FROM Q.STAFF';return false; ">
<INPUT TYPE="SUBMIT" value="CICS/TS"
OnClick="sqlQuery.value='CALL CICSEX.NEONFILA(2,\'FILEA
\', \'0000005\')';return false;">
<INPUT TYPE="SUBMIT" value="IMS/TM "
OnClick="sqlQuery.value='CALL SHADOW_IMS(\'OTM\',\'PART\',\'*\'
,\'MAP(NAME(PARTALL))\')' ;return false;">
<INPUT TYPE="SUBMIT" value="IMS/DB "
OnClick="sqlQuery.value='CALL SDCOIM';return false;">
<INPUT TYPE="SUBMIT" value=" VSAM "
OnClick="sqlQuery.value='CALL SHADOW_VSAM(\'SELECT * FROM
<INPUT TYPE="SUBMIT" value="ADABAS "
OnClick="sqlQuery.value='CALL SHADOW_ADABAS(\'SELECT
FIRST_NAME, LAST_NAME, SEX FROM EMPLOYEE WHERE LAST_NAME LE AZ\')';return false;">
<BR><TEXTAREA name="sqlQuery" COLS="55" ROWS="8"
WRAP="VIRTUAL"></TEXTAREA>
</tr>
<tr>
<td><font face="Helvetica"><input type="Submit" value="Submit Query"
name=" Submit"></td></tr>
</table>
</form>
<hr width=80%>
<%@ page import="
javax.naming.*,
java.util.*,
java.sql.*,
weblogic.common.*
" %>
<%
if ("POST".equals(request.getMethod())) {
String jdbcDriver = (String) request.getParameter("jdbcDriver");
String dbURL = "jdbc:neon:DSN=UNKNOWN;HOST=mkt.neonsys.com;PORT=1200;
AUST=NO;APPL=ODBC;CPFX=SYSIBM;
LINK=TCPIP;CD=NO;PLAN= SDBC1010;DBTY=DB2;SUBSYS=DSN1";
String sqlQuery = (String) request.getParameter("sqlQuery");
String username = "demo01";
String passwd = "demo01";
%>
<h2>Results from previous query:</h2>
Here are the results from the previous SQL query using the these parameters:
<ul>
<li> JDBC Driver: <%= jdbcDriver==null?"No driver specified.":jdbcDriver %>
<li> Database URL: <%= dbURL==null?"No URL specified":dbURL %>
<li> SQL query: <%= sqlQuery==null?"No SQL query":sqlQuery %>
</ul>
<p>
<%
Connection conn = null;
Statement stmt = null;
ResultSet rs = null;
try {
Driver myDriver = (Driver) Class.forName(jdbcDriver).newInstance();
if ((username != null) && (passwd != null)) {
Properties props = new Properties();
props.put("user", username);
props.put("password", passwd);
conn = myDriver.connect(dbURL, props);
}
else {
conn = myDriver.connect(dbURL, null);
}
stmt = conn.createStatement();
rs = stmt.executeQuery(sqlQuery);
ResultSetMetaData rsmd = rs.getMetaData();
int numCols = rsmd.getColumnCount();
%>
<p>
<center>
<table border=1 cellspacing=2 cellpadding=0 width=400>
<tr>
<%
for (int i = 1; i <= numCols; i++) {
%>
<td><font face="Helvetica"><b><%= rsmd.getColumnLabel(i) %></b></td>
<%
}
%>
</tr>
<%
while (rs.next()) {
%>
<tr>
<%
for (int i = 1; i <= numCols; i++) {
%>
<td><font face="Helvetica"><%= rs.getString(i) %></td>
<%
}
%>
</tr>
<%
}
}
catch (Exception e) {
%>
<p><b>There was an error executing or processing the query:</b>
<br>
Exception: <%= e %>
<pre><%= getStackTraceAsString(e) %></pre>
<%
}
finally {
try {
rs.close();
stmt.close();
conn.close();
}
catch (Exception e) {
out.print("<b>There was an error closing the database connection</b>");
out.print("<br>Exception: " +e);
out.print("<br><pre>"+getStackTraceAsString(e)+"</pre>");
}
}
}
%>
</table>
</center>
</td>
</tr>
</table>
<br>
<!-- RED LINE -->
<table cellspacing="0" cellpadding="0" border="0" width="100%">
<tr>
<td width="100%" bgcolor="#ff0000" height="1">
<p class="small"> </p>
</td>
</tr>
</table>
</body>
</html>
<%!
String getStackTraceAsString(Exception e)
{
// Dump the stack trace to a buffered stream, then return it's contents
// as a String. This is useful for printing the stack to 'out'.
ByteArrayOutputStream ostr = new ByteArrayOutputStream();
e.printStackTrace(new PrintStream(ostr));
return(ostr.toString());
}
%>
|
