Figure 1. Eclipse integration of TestNG (click the image for a full-size screen shot)
Published on dev2dev (http://dev2dev.bea.com/)
http://dev2dev.bea.com/pub/a/2006/09/testng-categorization.html
See this if you're having trouble printing code examples
by Andy Glover
11/08/2006
TestNG is an annotation-based testing framework, which draws on some of the shortcomings of JUnit by adding features such as flexible fixtures, test categorization, parametric testing, and dependent methods, to name a few. What's more, TestNG runs on both Java 5.0 (via annotations) and 1.4 (via JavaDoc style comments). Through TestNG's ability to easily categorize developer tests into unit, component, and system groups, build times can be kept manageable. Via the use of the groups annotation and multiple Ant tasks, test groups can be run at varying frequencies whether on a workstation or in a continuous integration environment.
This article examines the concept of test categories and demonstrates how to incorporate TestNG's groups annotation tied with flexible fixtures to facilitate running tests at different frequencies via specific Ant targets. It assumes knowledge of TestNG.
TestNG supports an intuitive mechanism for grouping test classes and associated test methods. At its most basic level, the grouping feature of TestNG is enabled via the groups parameter of the test annotation, which can be attached to classes or individual methods. As the name implies, a class or individual method can belong to 1...n groups.
For example, the following class contains two public methods, which by default are labeled as tests and further categorized as belonging to group one:
/**
* @testng.test groups="one"
*/
public class SimpleGroupedTest {
private String fixture;
/**
* @testng.before-class = "true"
*/
private void init(){
this.fixture = "Hello World";
}
public void verifyEquality(){
Assert.assertEquals("Hello World", this.fixture);
}
public void verifySame(){
String value = this.fixture;
Assert.assertSame(this.fixture, value);
}
}
In contrast, the next class defines two test methods. However, one method belongs to two different groups—one and two. Accordingly, any associated fixture logic must be associated with a desired group. In this case, the init() method must be configured to run before either group one or two is executed.
public class SimpleGroupedTwoTest {
private String fixture;
/**
* @testng.before-class = "true" \
* groups = "one, two"
*/
private void init(){
this.fixture = "Hello World";
}
/**
* @testng.test groups="one, two"
*/
public void verifyEqualityAgain(){
Assert.assertEquals(this.fixture, "Hello World");
}
/**
* @testng.test groups="two"
*/
public void verifySameAgain(){
String value = this.fixture;
Assert.assertSame(value, this.fixture);
}
}
TestNG supports running desired groups in a number of ways, ranging from specifying them via the TestNG Eclipse plug-in to listing them in the TestNG Ant task.
Performing a build (that is, compiling source code and executing tests) is one of the easiest ways to verify working software; therefore, it should come as no surprise that long-running builds are a drag on developer productivity. There are few annoyances that are worse than having to wait for an excessively long build to complete. Encountering an unexpectedly blue screen of death reboot in the middle of coding ranks up there too, but at least we can easily do something about long builds.
The test execution step is almost always the reason for long builds (unless the build compiles millions and millions of .java files). Plus, the aggregate time to execute a test suite has the tendency to lengthen when there are extensive set-up steps, such as configuring a database or deploying an EAR file, to name a few. Therefore, crafting a test categorization strategy and running categories at prescribed intervals will facilitate manageable build durations.
Categorizing tests, however, requires that we define specific categories, which means refining the generic term unit tests. Unit testing, as it turns out, is just one piece in a three-sliced pie, with the two remaining pieces being component tests and system tests. The next section examines the various types of tests that are typically written by developers, such as unit tests, component tests, and system tests. Subsequently, these test types will be implemented in TestNG and integrated into an Ant build script.
Unit tests verify the behavior of isolated objects; however, due to class coupling, they can also verify the behavior of related objects. For example, the following unit test, which verifies object identity and is implemented in TestNG, solely focuses on one type: PartOfSpeechEnum.
/**
* @testng.test
*/
public class PartOfSpeechEnumTest {
public void verifyNotEquals() throws Exception{
assert PartOfSpeechEnum.ADJECTIVE !=
PartOfSpeechEnum.NOUN: "NOUN == ADJECTIVE?";
}
public void verifyEquals() throws Exception{
assert PartOfSpeechEnum.ADJECTIVE ==
PartOfSpeechEnum.ADJECTIVE "ADJECTIVE != ADJECTIVE";
}
}
Sometimes, a unit test verifies the behavior of more than one object. These objects, however, have few other outside dependencies. For example, the following test verifies two objects: Dependency and DependencyFinder.
//imports removed...
public class DependencyFinderTest {
private DependencyFinder finder;
/**
* @testng.test
*/
public void verifyFindDependencies() throws Exception{
String targetClss = "test.com.sedona.frmwrk.dep.Finder";
Filter[] filtr = new Filter[] {
new RegexPackageFilter("java|org")};
Dependency[] deps =
finder.findDependencies(targetClss, filtr);
Assert.assertNotNull(deps, "deps was null");
Assert.assertEquals(deps.length, 5, "should be 5 large");
}
/**
* @testng.before-class = "true"
*/
protected void init() throws Exception {
this.finder = new DependencyFinder();
}
}
The key thing to remember is that unit tests do not rely on outside dependencies such as databases, which have the tendency to increase the amount of time it takes to set up and run tests. Unit tests do not have a configuration cost (measured in terms of time), and the resource consumption to run them is negligible.
Because unit tests run so quickly, they should be run any time a build is run, including in continuous integration environments, where builds are usually run if the source repository (such as CVS) changes.
|
Component test have a few aliases, such as subsystem tests or integration tests. Regardless of the desired term, these tests verify portions of an application and may even require a fully installed system or a more limited set of external dependencies, such as databases, file systems, or network endpoints. These tests, in essence, verify that different components interact to produce the expected combined behavior.
A typical component test requires a seeded database; moreover, the test itself may verify behavior across architectural boundaries. Because large amounts of code are exercised by component tests, more code coverage is achieved; however, these tests have the tendency to take longer to run than unit tests.
Because component tests have a cost associated with them—dependencies have to be put in place and configured—they should not be run every time a build is executed. Instead, they should be run at regular intervals. Remember, while these tests alone may only take a few seconds, as more component tests are added to a suite the total test time increases, often quite quickly.
For example, the component test below utilizes DbUnit to seed an underlying database. The set-up alone in this test case takes longer than most unit tests should take to run.
//imports removed...
public class WordDAOImplTest {
private WordDAO dao = null;
/**
* @testng.before-method = "true"
*/
private void setUp() throws Exception {
final ApplicationContext context =
new ClassPathXmlApplicationContext(
"spring-config.xml");
this.dao = (WordDAO) context.getBean("wordDAO");
}
/**
* @testng.before-class = "true"
*/
public void oneTimeSetUp() throws Exception{
final IDatabaseConnection conn =
this.getConnection();
final IDataSet data = this.getDataSet();
try{
DatabaseOperation.CLEAN_INSERT.execute(conn, data);
}finally{
conn.close();
}
}
/**
* @testng.test
*/
public void createWord() {
final IWord word = new Word();
word.setSpelling("pabulum");
word.setPartOfSpeech(
PartOfSpeechEnum.NOUN.getPartOfSpeech());
final IDefinition defOne = new Definition();
defOne.setDefinition("food");
defOne.setWord(word);
final Set defs = new HashSet();
defs.add(defOne);
word.setDefinitions(defs);
try{
this.dao.createWord(word);
}catch(CreateException e){
Assert.fail("CreateException was thrown");
}
}
private IDataSet getDataSet()
throws IOException, DataSetException {
return new FlatXmlDataSet(
new File("test/conf/words-seed.xml"));
}
private IDatabaseConnection getConnection()
throws ClassNotFoundException, SQLException {
final Class driverClass =
Class.forName("org.gjt.mm.mysql.Driver");
final Connection jdbcConnection = DriverManager.
getConnection("jdbc:mysql://localhost/words",
"words", "words");
return new DatabaseConnection(jdbcConnection);
}
}
Component tests don't always depend on databases, however. For example, relying on file systems creates a coupling that increases configuration complexity and, in some cases, time. For instance, the component test below uses XMLUnit to verify generated XML. Note that the test relies on a file system path to compare two XML documents.
//imports removed...
public class BatchDepXMLReportValidationTest {
/**
* @testng.before-class = "true"
*/
protected void configure() throws Exception {
XMLUnit.setControlParser(
"org.apache.xerces.jaxp.DocumentBuilderFactoryImpl");
XMLUnit.setTestParser(
"org.apache.xerces.jaxp.DocumentBuilderFactoryImpl");
XMLUnit.setSAXParserFactory(
"org.apache.xerces.jaxp.SAXParserFactoryImpl");
XMLUnit.setIgnoreWhitespace(true);
}
private Filter[] getFilters(){
return new Filter[] {
new RegexPackageFilter("java|org"),
new SimplePackageFilter("junit.")
};
}
private Dependency[] getDependencies(){
return new Dependency[] {
new Dependency("com.vanward.resource.XMLizable"),
new Dependency("com.vanward.xml.Element")
};
}
/**
* @testng.test
*/
public void assertToXML() throws Exception{
BatchDependencyXMLReport report =
new BatchDependencyXMLReport(
new Date(9000000), this.getFilters());
report.addTargetAndDependencies(
"com.vanward.test.MyTest", this.getDependencies());
report.addTargetAndDependencies(
"com.xom.xml.Test", this.getDependencies());
Diff diff = new Diff(new FileReader(
new File("./test/conf/report-control.xml")),
new StringReader(report.toXML()));
Assert.assertTrue(
diff.identical(),"XML was not identical");
}
}
While component tests shouldn't be executed every time a build is run, it's a good bet to run them before committing code into a repository. In a continuous integration environment it's probably a good idea to run these at frequent intervals, such as hourly.
|
System tests exercise a complete software application, verifying external interfaces like Web pages, Web service end points, or GUIs, work end-to-end as designed. Because these tests exercise an entire system, they are often created towards the latter cycles of development. These tests also have the tendency to have prolonged execution times, in addition to extended set up and configuration times.
For example, the following test utilizes jWebUnit to test the login functionality of a Web site. Note that a number of assumptions are made in this test, such as the URL being available and the user "tst" actually having a valid account, not to mention a trading history. These implicit dependencies usually require a configuration step preceding a test run.
public class LoginTest {
private WebTester tester;
/**
* @testng.before-class = "true"
*/
protected void init() throws Exception {
this.tester = new WebTester();
this.tester.getTestContext().
setBaseUrl("http://stocktrader.com");
}
/**
* @testng.test
*/
public void verifyLogIn() {
this.tester.beginAt("/");
this.tester.setFormElement("user", "tst");
this.tester.setFormElement("psswd", "t@t");
this.tester.submit();
this.tester.assertTextPresent("Logged in as tst");
}
/**
* @testng.test dependsOnMethods = "verifyLogIn"
*/
public void verifyAccountInfo() {
this.tester.clickLinkWithText("History", 0);
this.tester.assertTextPresent("$89.00, sold");
}
}
Developers should run these tests locally on an as-needed basis and in a continuous integration environment. Executing these tests nightly (if they can be pulled off in an automated fashion) is a good strategy. Running them more frequently can tax system resources, especially in larger environments. With proper hardware planning and sophisticated automation, however, teams can run these tests more frequently.
Categorizing TestNG tests into these three categories is as easy as using the group annotation described above. Having a test class with methods spanning various test granularities is unusual, therefore tagging can be effectively applied at the class level.
For example, the following class has been tagged as belonging to the unit test group. Note how HierarchyBuilderTest verifies the behavior of the Hierarchy class by depending on the HierarchyBuilder type. As this relationship ends at HierarchyBuilder, which doesn't depend on a file system or database, you can effectively consider this a unit test:
import org.testng.Assert;
/**
* @testng.test groups="unit"
*/
public class HierarchyBuilderTest {
private Hierarchy hier;
/**
* @testng.before-class = "true" groups="unit"
*/
private void init() throws Exception{
this.hier =
HierarchyBuilder.buildHierarchy(Vector.class);
}
public void validateIsntNull() throws Exception{
Assert.assertNotNull(this.hier,
"should be something!");
}
/**
* @testng.test dependsOnMethods="validateIsntNull"
*/
public void validateValues() throws Exception{
Assert.assertEquals(
this.hier.getHierarchyClassNames().length,
2, "should be 2");
}
}
Similarly, individual methods in a system test can be tagged with a system group identification as demonstrated here:
public class LoginTest {
private WebTester tester;
/**
* @testng.before-class = "true" groups="system"
*/
protected void init() throws Exception {
this.tester = new WebTester();
this.tester.getTestContext().
setBaseUrl("http://acme.com:8080/ppo/");
}
/**
* @testng.test groups="system"
*/
public void verifyView() {
this.tester.beginAt("/");
this.tester.setFormElement("isbn", "900930390");
this.tester.submit();
this.tester.assertTextPresent("Book in stock");
}
}
Before checking in code to a content management system, it's paramount to run tests locally, either through a build or through an environment such as an IDE. Running categorized tests via the TestNG Eclipse plug-in is amazingly simple. By selecting the groups option in the TestNG Create, manage, and run configurations dialog, as shown in Figure 1, the list of available groups is presented with checkboxes to facilitate selecting one or more. After the desired group or groups have been selected, hit the Run button, and watch that green bar go!
Figure 1. Eclipse integration of TestNG (click the image for a full-size screen shot)
Running categorized TestNG tests via a build becomes a matter of defining the appropriate Ant targets for each group. For example, to run all tests belonging to the component group, the TestNG Ant task is defined with the component group specified:
<target name="testng-component"
depends="testng-init">
<mkdir dir="${testng.output.dir.comp}"/>
<testng groups="component"
outputDir="${testng.output.dir.comp}"
sourceDir="${testng.source.dir}"
classpath="${testclassesdir};${classesdir}">
<classfileset dir="${testng.source.dir}">
<include name="**/*Test.java"/>
</classfileset>
<classpath>
<path refid="build.classpath"/>
</classpath>
</testng>
</target>
Accordingly, using this strategy, at least four test targets would be created. There would be three corresponding to unit, component, and system tests, and a final target that could run all three.
TestNG facilitates test categories quite easily and this is probably one of TestNG's most exciting benefits. What's more, TestNG's group annotation also helps place tests into other categories, such as batch tests, acceptance tests, and even performance tests. In fact, it seems this feature may have had an influence on the newest version of JUnit, which is planning to support test groups as well!
The next time you blindly code a new test case, consider the long-term implications on build execution times. Build scalability starts with a test categorization strategy that is run at various frequencies, and an effective test categorization strategy starts with TestNG.
Andy Glover is the President of Stelligent Incorporated and is based in Reston, Virginia. He actively blogs about software quality at www.thediscoblog.com.
Return to dev2dev.