The strategy and tactics of cleanroom testing are fundamentally different from conventional testing approaches. Conventional methods derive...
The strategy and tactics of cleanroom testing are fundamentally different from conventional testing approaches. Conventional methods derive a set of test cases to uncover design and coding errors. The goal of cleanroom testing is to validate software requirements by demonstrating that a statistical sample of use-cases have been executed successfully.
Statistical Use Testing
The user of a computer program rarely needs to understand the technical details of the design. The user-visible behavior of the program is driven by inputs and events that are often produced by the user. But in complex systems, the possible spectrum of input and events (i.e., the use-cases) can be extremely wide. What subset of usecases will adequately verify the behavior of the program? This is the first question addressed by statistical use testing.
Statistical use testing “amounts to testing software the way users intend to use it” . To accomplish this, cleanroom testing teams (also called certification teams) must determine a usage probability distribution for the software. The specification (black box) for each increment of the software is analyzed to define a set of stimuli
(inputs or events) that cause the software to change its behavior. Based on interviews with potential users, the creation of usage scenarios, and a general understanding of the application domain, a probability of use is assigned to each stimuli.
Test cases are generated for each stimuli according to the usage probability distribution. To illustrate, consider the SafeHome security system discussed earlier . Cleanroom software engineering is being used to develop a software increment that manages user interaction with the security system keypad. Five stimuli have been identified for this increment. Analysis indicates the percent probability distribution of each stimulus. To make selection of test cases easier, these probabilities are mapped into intervals numbered between 1 and 99 and illustrated in the following table:
Program Stimulus Probability Interval
Arm/disarm (AD) 50% 1–49
Zone set (ZS) 15% 50–63
Query (Q) 15% 64–78
Test (T) 15% 79–94
Panic alarm 5% 95–99
To generate a sequence of usage test cases that conform to the usage probability distribution, a series of random numbers between 1 and 99 is generated. The random number corresponds to an interval on the preceding probability distribution. Hence, the sequence of usage test cases is defined randomly but corresponds to the appropriate probability of stimuli occurrence. For example, assume the following random number sequences are generated:
Selecting the appropriate stimuli based on the distribution interval shown in the table, the following use-cases are derived:
The testing team executes these use-cases and verifies software behavior against the specification for the system. Timing for tests is recorded so that interval times may be determined. Using interval times, the certification team can compute mean-timeto- failure. If a long sequence of tests is conducted without failure, the MTTF is low and software reliability may be assumed high.
The verification and testing techniques discussed earlier in this chapter lead to software components (and entire increments) that can be certified. Within the context of the cleanroom software engineering approach, certification implies that the reliability (measured by mean-time-to-failure, MTTF) can be specified for each component.
The potential impact of certifiable software components goes far beyond a single cleanroom project. Reusable software components can be stored along with their usage scenarios, program stimuli, and probability distributions. Each component would have a certified reliability under the usage scenario and testing regime described. This information is invaluable to others who intend to use the components.
The certification approach involves five steps :
1. Usage scenarios must be created.
2. A usage profile is specified.
3. Test cases are generated from the profile.
4. Tests are executed and failure data are recorded and analyzed.
5. Reliability is computed and certified.
Steps 1 through 4 have been discussed in an earlier section. In this section, we concentrate on reliability certification.
Certification for cleanroom software engineering requires the creation of three models :
Sampling model. Software testing executes m random test cases and is certified if no failures or a specified numbers of failures occur. The value of m is derived mathematically to ensure that required reliability is achieved.
Component model. A system composed of n components is to be certified. The component model enables the analyst to determine the probability that component i will fail prior to completion.
Certification model. The overall reliability of the system is projected and
At the completion of statistical use testing, the certification team has the information required to deliver software that has a certified MTTF computed using each of these models.