Regression Testing

Smoke Tests

A Smoke Test is a series of basic tests of the application software to verify that the software performs its basic tasks correctly. The Smoke Test often includes various metrics such as memory use and time to execute, to enable the Smoke Test to identify serious errors in algorithms. Smoke Tests are typically run for each full build of the software. Any test that fails then identifies the software changes added to the build since the last build – thus enabling easy identification of the parts of the software that may have led to the Smoke Test failure.

Smoke Tests are described in Chapter 18 of Steve McConnell’s book Rapid Development, published by Microsoft Press.

ISBN:1-55615-900-5

Regression Testing

Regression Testing is an extension of the Smoke Test concept to a much more extensive set of tests of the software. Whereas the Smoke Test is designed to be quick to perform and the be performed on a daily (or faster) basis, Regression Test suites may take a day or week to perform. Such Regression Tests gather large amounts of data about the application being tested.

Regression Tests allow the software testers to automatically perform many tests. The results of these tests ensure that the application performs the appropriate task correctly. In addition, other collected metrics allow the testing software to automatically spot when memory usage deviates from expected parameters, execution time increases or decreases greatly and when code coverage changes dramatically.

If the Smoke Test suite and/or the Regression Test suite cover enough parts of the application software, performing Code Coverage using the Regression Test suite is very useful. This can provide a daily or weekly audit of code coverage, allowing management and software developers to track which parts of the code are being routinely exercised and which parts are not being routinely exercised. This information can be used to inform the quality assurance and testing groups about authoring of new tests.

Coverage Validator does not provide direct support for Regression Testing. However the HTML export, XML export and Cobertura XML export features of Coverage Validator can be used to export a session which can then be processed by an HTML or XML processor. The results of this can be saved in a format useful to your organisation for comparison with a future run of the Regression Tests.

The results can highlight trend growth in product size, code coverage, new areas covered and so forth. Large deviations in the trend graph indicate events to be investigated.