The overriding goal of software engineering is to produce a high-quality system, application, or product. To achieve this goal, software en...
The overriding goal of software engineering is to produce a high-quality system, application, or product. To achieve this goal, software engineers must apply effective methods coupled with modern tools within the context of a mature software process. In addition, a good software engineer (and good software engineering managers) must measure if high quality is to be realized.
The quality of a system, application, or product is only as good as the requirements that describe the problem, the design that models the solution, the code that leads to an executable program, and the tests that exercise the software to uncover errors. A good software engineer uses measurement to assess the quality of the analysis and design models, the source code, and the test cases that have been created as the software is engineered. To accomplish this real-time quality assessment, the engineer
must use technical measures to evaluate quality in objective, rather than subjective ways.
The project manager must also evaluate quality as the project progresses. Private metrics collected by individual software engineers are assimilated to provide projectlevel results. Although many quality measures can be collected, the primary thrust at the project level is to measure errors and defects. Metrics derived from these measures provide an indication of the effectiveness of individual and group software quality assurance and control activities. Metrics such as work product errors per function point, errors uncovered per review hour, and errors uncovered per testing hour provide insight into the efficacy of each of the activities implied by the metric. Error data can also be used to compute the defect removal efficiency (DRE) for each process framework activity.
An Overview of Factors That Affect Quality
Over 25 years ago, McCall and Cavano defined a set of quality factors that were a first step toward the development of metrics for software quality. These factors assess software from three distinct points of view:
(1) product operation (using it),
(2) product revision (changing it), and
(3) product transition (modifying it to work in a different environment; i.e., "porting" it).
In their work, the authors describe the relationship between these quality factors and other aspects of the software engineering process:
First, the framework provides a mechanism for the project manager to identify what qualities are important. These qualities are attributes of the software in addition to its functional correctness and performance which have life cycle implications. Such factors as maintainability and portability have been shown in recent years to have significant life cycle cost impact . . .
Secondly, the framework provides a means for quantitatively assessing how well the development is progressing relative to the quality goals established . . .
Thirdly, the framework provides for more interaction of QA personnel throughout the development effort . . .
Lastly, . . . quality assurance personal can use indications of poor quality to help identify standards to be enforced in the future.
It is interesting to note that nearly every aspect of computing has undergone radical change as the years have passed since McCall and Cavano did their seminal work in 1978. But the attributes that provide an indication of software quality remain the same.
What does this mean? If a software organization adopts a set of quality factors as a “checklist” for assessing software quality, it is likely that software built today will still exhibit quality well into the first few decades of this century. Even as computing architectures undergo radical change (as they surely will), software that exhibits high quality in operation, transition, and revision will continue to serve its users well.
Secondly, the framework provides a means for quantitatively assessing how well the development is progressing relative to the quality goals established . . .
Thirdly, the framework provides for more interaction of QA personnel throughout the development effort . . .
Lastly, . . . quality assurance personal can use indications of poor quality to help identify standards to be enforced in the future.
It is interesting to note that nearly every aspect of computing has undergone radical change as the years have passed since McCall and Cavano did their seminal work in 1978. But the attributes that provide an indication of software quality remain the same.
What does this mean? If a software organization adopts a set of quality factors as a “checklist” for assessing software quality, it is likely that software built today will still exhibit quality well into the first few decades of this century. Even as computing architectures undergo radical change (as they surely will), software that exhibits high quality in operation, transition, and revision will continue to serve its users well.
Measuring Quality
Although there are many measures of software quality, correctness, maintainability, integrity, and usability provide useful indicators for the project team. Gilb suggests definitions and measures for each.
Correctness. A program must operate correctly or it provides little value to its users. Correctness is the degree to which the software performs its required function. The most common measure for correctness is defects per KLOC, where a defect is defined as a verified lack of conformance to requirements. When considering the overall quality of a software product, defects are those problems reported by a user of the program after the program has been released for general use. For quality assessment purposes, defects are counted over a standard period of time, typically one year.
Maintainability. Software maintenance accounts for more effort than any other software engineering activity. Maintainability is the ease with which a program can be corrected if an error is encountered, adapted if its environment changes, or enhanced if the customer desires a change in requirements. There is no way to measure maintainability directly; therefore, we must use indirect measures. A simple time-oriented metric is mean-time-tochange (MTTC), the time it takes to analyze the change request, design an appropriate modification, implement the change, test it, and distribute the change to all users. On average, programs that are maintainable will have a lower MTTC (for equivalent types of changes) than programs that are not maintainable.
Hitachi has used a cost-oriented metric for maintainability called spoilage—the cost to correct defects encountered after the software has been released to its end-users. When the ratio of spoilage to overall project cost (for many projects) is plotted as a function of time, a manager can determine whether the overall maintainability of software produced by a software development organization is improving. Actions can then be taken in response to the insight gained from this information.
Integrity. Software integrity has become increasingly important in the age of hackers and firewalls. This attribute measures a system's ability to withstand attacks (both accidental and intentional) to its security. Attacks can be made on all three components of software: programs, data, and documents.
To measure integrity, two additional attributes must be defined: threat and security. Threat is the probability (which can be estimated or derived from empirical evidence) that an attack of a specific type will occur within a given time. Security is the probability (which can be estimated or derived from empirical evidence) that the attack of a specific type will be repelled. The integrity of a system can then be defined as
integrity = summation [(1 – threat) x (1 – security)]
where threat and security are summed over each type of attack.
Usability. The catch phrase "user-friendliness" has become ubiquitous in discussions of software products. If a program is not user-friendly, it is often doomed to failure, even if the functions that it performs are valuable. Usability is an attempt to quantify user-friendliness and can be measured in terms
of four characteristics:
(1) the physical and or intellectual skill required to learn the system,
(2) the time required to become moderately efficient in the use of the system,
(3) the net increase in productivity (over the approach that the system replaces) measured when the system is used by someone who is moderately efficient, and
(4) a subjective assessment (sometimes obtained through a questionnaire) of users attitudes toward the system.
Defect Removal Efficiency
A quality metric that provides benefit at both the project and process level is defect removal efficiency (DRE). In essence, DRE is a measure of the filtering ability of quality assurance and control activities as they are applied throughout all process framework activities.
When considered for a project as a whole, DRE is defined in the following manner:
DRE = E/(E + D)
where E is the number of errors found before delivery of the software to the end-user and D is the number of defects found after delivery.
The ideal value for DRE is 1. That is, no defects are found in the software. Realistically, D will be greater than 0, but the value of DRE can still approach 1. As E increases (for a given value of D), the overall value of DRE begins to approach 1. In fact, as E increases, it is likely that the final value of D will decrease (errors are filtered out before they become defects). If used as a metric that provides an indicator of the filtering ability of quality control and assurance activities, DRE encourages a software project team to institute techniques for finding as many errors as possible before delivery.
DRE can also be used within the project to assess a team’s ability to find errors before they are passed to the next framework activity or software engineering task. For example, the requirements analysis task produces an analysis model that can be reviewed to find and correct errors. Those errors that are not found during the review of the analysis model are passed on to the design task (where they may or may not be found). When used in this context, we redefine DRE as
DREi = Ei/(Ei + Ei+1)
where Ei is the number of errors found during software engineering activity i and Ei+1 is the number of errors found during software engineering activity i+1 that are traceable to errors that were not discovered in software engineering activity i. A quality objective for a software team (or an individual software engineer) is to achieve DREi that approaches 1. That is, errors should be filtered out before they are passed on to the next activity.
