Figure 1. Storing information of a new employee in a database
Published on dev2dev (http://dev2dev.bea.com/)
http://dev2dev.bea.com/pub/a/2007/06/mock-shortcomings.html
See this if you're having trouble printing code examples
by Alex Ruiz
08/22/2007
Writing unit tests is not easy. Most of the time, we need to test code that uses complex collaborators such as databases, application servers, or software modules that have not been written yet. We also may need to deal with conditions that are difficult to generate in a test environment. Setting up these dependencies may require a considerable amount of time and energy, counteracting the benefits of having automated tests. This article looks at Mock Objects, a testing technique from the XP community that offers a way to test our code in isolation by simulating those external dependencies. As with any other tool, we need to be careful and avoid overusing them.
In recent years, developers have revived the benefits of writing their own tests, agreeing that finding and fixing defects in software is expensive. As a result, Unit Testing has become an essential part of the software development process, as a way to find coding mistakes and to help identify system requirements. The main goal of unit testing is to test in isolation a single unit of work, which is usually a class. Testing code in isolation is difficult, especially when dealing with dependencies that are not easy or quick to set up in our tests. The more difficult it is to write and maintain unit tests, the greater the chances that developers will feel discouraged and stop writing tests.
Tim Mackinnon, Steve Freeman, and Philip Craig introduced the idea of mock objects in their paper "Endo-Testing: Unit Testing with Mock Objects," which they presented at XP2000. Mock objects, or mocks, simulate difficult-to-use and expensive-to-use collaborators and provide a way to:
close in java.sql.Connection was called at a specific moment)SQLException being thrown by the JDBC driver)Although useful, mocks are not a panacea and their overuse can bring more harm than good to our projects.
Let's look at a few items that the mock programmer needs to be aware of.
This is especially true if we test code using mocks exclusively, without writing integration tests.
Consider the example in Figure 1.
Figure 1. Storing information of a new employee in a database
The EmployeeBO class offers business services related to Employees and uses EmployeeDAO to persist data in a relational database using JDBC. Testing EmployeeBO would imply setting up a database and populating it with test data.
Proponents of mock objects suggest that we can save a significant amount of time and effort by mocking EmployeeDAO, avoiding the overhead of setting up and using a real database in our tests. Mocks can effectively speed up creation and execution of unit tests, but they cannot give us the confidence that the system, as a whole, works correctly. Mock testing may hide bugs or defects in the collaborators that are being mocked. To find those defects, our test suites need to include integration tests. In our example, the system being tested uses a database to store employee information. Mock testing is limited to verify that the interaction between EmployeeBO and EmployeeDAO is correct—that is, that EmployeeBO calls only the expected methods from EmployeeDAO at the expected time. Only integration tests can help us discover problems, such as bugs in the JDBC driver or in the database itself, that should not be present when the application goes to production.
The following code listing uses EasyMock to test that EmployeeBO uses EmployeeDAO to store information of new employees and to update information of existing employees.
@Before public void setUp() {
mockEmployeeDAO = createMock(EmployeeDAO.class);
employeeBO = new EmployeeBO(mockEmployeeDAO);
employee = new Employee("Alex", "CA", "US");
}
@Test public void shouldAddNewEmployee() {
mockEmployeeDAO.insert(employee);
replay(mockEmployeeDAO);
employeeBO.addNewEmployee(employee);
verify(mockEmployeeDAO);
}
@Test public void shouldUpdateEmployee() {
mockEmployeeDAO.update(employee);
replay(mockEmployeeDAO);
employeeBO.updateEmployee(employee);
verify(mockEmployeeDAO);
}
The test method shouldAddNewEmployee verifies correct interaction between the object under test (employeeBO) and the mock (mockEmployeeDAO). It expects employeeBO to call the method insert in mockEmployeeDAO, passing the same instance of Employee that it receives. Although simple, the method shouldAddNewEmployee has the following code that does not communicate its purpose, adding clutter to our test:
replay to notify EasyMock that all the expectations have been setverify to notify EasyMock that the expectations should be verifiedUsage of mocks usually follows this pattern:
Such a pattern introduces duplication in test code, which is evident in the test methods shouldAddNewEmployee() and shouldUpdateEmployee(). The following template class, EasyMockTemplate, can help reduce both code clutter and duplication:
/**
* Understands a template for usage of EasyMock mocks.
* @author Alex Ruiz
*/
public abstract class EasyMockTemplate {
/** Mock objects managed by this template */
private final List<Object> mocks = new ArrayList<Object>();
/**
* Constructor.
* @param mocks the mock objects this template will manage.
* @throws IllegalArgumentException if the list of mock objects is null or empty.
* @throws IllegalArgumentException if the list of mock objects contains a null value.
*/
public EasyMockTemplate(Object... mocks) {
if (mocks == null) throw new IllegalArgumentException("The list of mock objects should not be null");
if (mocks.length == 0) throw new IllegalArgumentException("The list of mock objects should not be empty");
for (Object mock : mocks) {
if (mock == null) throw new IllegalArgumentException("The list of mocks should not include null values");
this.mocks.add(mock);
}
}
/**
* Encapsulates the common pattern followed when using EasyMock.
* <ol>
* <li>Set up expectations on the mock objects</li>
* <li>Set the state of the mock controls to "replay"</li>
* <li>Execute the code to test</li>
* <li>Verify that the expectations were met</li>
* </ol>
* Steps 2 and 4 are considered invariant behavior while steps 1 and 3 should be implemented by subclasses of this template.
*/
public final void run() {
setUp();
expectations();
for (Object mock : mocks) replay(mock);
codeToTest();
for (Object mock : mocks) verify(mock);
}
/** Sets the expectations on the mock objects. */
protected abstract void expectations();
/** Executes the code that is under test. */
protected abstract void codeToTest();
/** Sets up the test fixture if necessary. */
protected void setUp() {}
}
Now, let's refactor our test to use EasyMockTemplate:
@Test public void shouldAddNewEmployee() {
EasyMockTemplate t = new EasyMockTemplate(mockEmployeeDao) {
@Override protected void expectations() {
mockEmployeeDAO.insert(employee);
}
@Override protected void codeToTest() {
employeeBO.addNewEmployee(employee);
}
};
t.run();
}
@Test public void shouldUpdateEmployee() {
EasyMockTemplate t = new EasyMockTemplate(mockEmployeeDao) {
@Override protected void expectations() {
mockEmployeeDAO.update(employee);
}
@Override protected void codeToTest() {
employeeBO.updateEmployee(employee);
}
};
t.run();
}
EasyMockTemplate brings the following benefits:
expectations and codeToTest are abstract, EasyMockTemplate forces developers to specify both expectations and the code to test, minimizing programming errors.replay and verify anymore.
You can download the latest version of EasyMockTemplate here.
Mock testing is glass box testing, which requires intimate knowledge of class internals. This is a side effect of the interaction-based nature of mocks. Justifiable changes in the implementation of a method may break tests using mocks, even if the result of executing such a method is still the same.
In our example, EmployeeBO interacts with EmployeeDAO to store employee information in a database using plain JDBC. Let's say we change the way we store information in the database—from JDBC to JPA, for example—by replacing EmployeeDAO with EmployeeJPA, storing exactly the same information in exactly the same database tables. We would expect our existing tests to pass, since the outcome (storing data in the database) has not changed. Unfortunately, our tests using mocks will fail simply because the interaction between EmployeeBO and EmployeeDAO no longer exists: EmployeeBO now uses EmployeeJPA to store data in the database, as shown in Figure 2.
Figure 2. Changing database persistence strategy breaks existing tests using mocks
Conversely, black box testing (for example, functional testing) is less likely to break if the internals of the system change while the outcome stays the same, because the testing is based solely on the knowledge of system requirements.
In our example, integration tests that verify the correctness of the data being stored in the database (instead of verifying how data was stored) will not fail after replacing EmployeeDAO with EmployeeJPA.
|
Some mock object frameworks like EasyMock and JMock provide extensions that use cglib to enable this practice. cglib generates proxies at runtime for a given class by subclassing it and overriding the methods of interest. As a result, the class to mock and the methods involved in the expectations cannot be final, which limits our design decisions.
At the same time, a specific constructor signature (using reflection) may be necessary to instantiate a class-based mock, tightly coupling that constructor to one or more tests. Consequently, classes used as mocks are difficult to maintain, and tests are even more fragile, since it is not trivial to refactor code when usage of reflection is present, even with modern Java IDEs.
Regular mocks, the ones based on interfaces, have their expectations set on public APIs, which can be considered more or less stable. In contrast, expectations on class-based mocks may depend on protected or package-protected methods, which represent implementation details of a class. Such implementation details can (and will) change at any time, changing the interaction between the code under test and mocks, increasing the chances of breaking existing tests.
interface overuse
A possible side effect of mock abuse is the unnecessary creation of Java interfaces, for the sole purpose of mock creation (trying to avoid the problems related to class-based mocks.) Typical examples include creation of interfaces that will have one and only one implementation, such as utility or helper classes. This practice is often justified by a misinterpretation of the principle "Program to an interface, not an implementation." This principle refers to the concept of interface, a supertype used to exploit polymorphism, not the Java construct interface. It is possible to program to an interface, implemented using a Java interface or an abstract class.
Creating interfaces to aid mock testing increases maintenance costs (because there is more code to maintain), which usually outweighs any benefit that mocks may bring.
On the bright side, mock objects can be useful when used with discretion. The following are some possible good use cases for mocks.
Having a fast-running test suite, to be executed by each developer before committing her local changes to the source control repository, can (obviously) speed up development. Mock objects can be used to build this test suite, as long as these tests can give us the confidence that our local changes are not going to accidentally introduce bugs to the code base. A classic example is testing Servlets in isolation, using mocks for the HttpServletRequest, HttpServletResponse, and HttpSession objects, which is significantly faster and easier to set up than a real application server.
We can use mocks in our test suite as long as we keep in mind that these tests may be fragile, and at some point (for example, in the continuous integration build) we need to execute integration and functional tests as well.
Mocks can be useful when integration of complex components is expected to occur in the future. For example, it would make sense for one team to use mock testing while they wait for a second team to finish their component. To minimize any integration problems, the second team can build and provide the mock to the first team. Once the second team has finished their work, integration between the components from both teams can take place, hoping that testing using mocks got them as close as possible to the expected system behavior.
At this point, mocks have served their purpose and, because of their potential shortcomings, should be removed, even for future testing.
In the previous example, how EmployeeBO stores employee information in the database was irrelevant, as long the data was stored correctly. In the case of decorators, correct interaction between them and the decorated object is as important as the end result of such interaction. Consider the following simple (but unrealistic) example depicted in Figure 3.
Figure 3. Class diagram of a cache manager system
Figure 3 illustrates a cache management system, with the responsibility of storing frequently used objects in a cache as way to improve the performance of a system. The cache management system is composed of an interface CacheManager and two implementations, DistributedCacheManager and EmbeddedCacheManager, which are expected to be used in Web applications and rich-client applications, respectively.
Let's say we need to introduce a way to configure how the caching system handles Exceptions. If the system is in production, errors in the caching system should not stop the execution of such system. In another words, any Exception thrown by the cache should be ignored (and perhaps logged). On the other hand, for integration testing we need to catch all Exceptions thrown by the cache to diagnose and fix any implementation defect in the caching system.
This problem can be easily solved using the Decorator Pattern. We can easily attach exception handling to any implementation of CacheManager dynamically, as a flexible alternative to inheritance.
public final class IgnoreExceptionsCacheManagerDecorator implements CacheManager {
private static final Object NULL = new Object();
private static Logger logger = Logger.getAnonymousLogger();
private final CacheManager decorated;
public IgnoreExceptionsCacheManagerDecorator(CacheManager decorated) {
this.decorated = decorated;
}
public Object getFromCache(String key) {
try {
return decorated.getFromCache(key);
} catch (Exception e) {
logger.log(SEVERE, "Unable to retrieve an object using key \"" + key + "\"", e);
}
return NULL;
}
public void putInCache(String key, Object o) {
try {
decorated.putInCache(key, o);
} catch (Exception e) {
logger.log(SEVERE, "Unable to store the object " + o + " using key \"" + key + "\"", e);
}
}
}
To prevent any errors in the caching system to stop the execution of any application in production, we simply need to use IgnoreExceptionsCacheManagerDecorator:
CacheManager cacheManager = new IgnoreExceptionsCacheManagerDecorator(new DistributedCacheManager());
The following code listing illustrates how we can use mocks (and EasyMockTemplate) to test IgnoreExceptionsCacheManagerDecorator:
public class IgnoreExceptionsCacheManagerDecoratorTest {
private IgnoreExceptionsCacheManagerDecorator decorator;
private CacheManager decoratedMock;
@Before public void setUp() {
decoratedMock = createMock(CacheManager.class);
decorator = new IgnoreExceptionsCacheManagerDecorator(decoratedMock);
}
@Test public void shouldNotPropagateExceptionFromCache() {
final String key = "name";
final RuntimeException exception = new RuntimeException();
EasyMockTemplate t = new EasyMockTemplate(decoratedMock) {
@Override protected void expectations() {
expect(decoratedMock.getFromCache(key)).andThrow(exception);
}
@Override protected void codeToTest() {
try {
decorator.getFromCache(key);
} catch (Exception e) {
if (e == exception) fail("Should not propagate exception thrown by the cache");
}
assertExceptionWasLogged(exception);
}
};
t.run();
}
}
In our decorator pattern example, mock objects proved to be a good testing aid because:
get(String, Object) from IgnoreExceptionsCacheManagerDecorator is calling get(String, Object) from the decorated objectTesting decorators is, in fact, one of many possible use cases of mock testing when the interaction between two or more objects is as important as the end result of such interaction. Usage of mock testing must be determined carefully on a case-by-case basis (for example, when testing certain implementations of the Adapter pattern.) Decorators are just a special case where it is safe to introduce mocks.
Testing code in isolation is a challenge. Non-trivial code usually depends on collaborators that are not easy or quick to set up in tests. Developers, even the most motivated ones, can be discouraged after spending large amounts of time and energy writing, maintaining, and executing tests. To prevent testing from decreasing, mock objects provide a mechanism to test code in complete isolation, by simulating those real-world, hard-to-use, and expensive-to-use dependencies. Although mocks can simplify creation of unit tests, they are not a replacement for functional and integration tests. Mocks need to be used carefully; overusing them may introduce problems such as hidden integration issues, clutter, and duplication in test code, unnecessary code, and test fragility.
EasyMockTemplateAlex Ruiz is a Software Engineer in the development tools organization at Oracle.
Return to Dev2Dev.