To convince a customer that software is high-quality, the engineer needs to build an assurance case: a convincing argument that the code meets its requirements and specifications. This argument is documentation that should be delivered to anyone evaluating the software. If the customer is the government, an assurance case is a requirement, and will include documentation about many aspects of the software engineering process, including testing.
For testing to be an effective part of the assurance case, it is important to document testing strategy. We have discussed many different ways to approach testing. As engineers, you should think about what is the most effective way to test your code, and justify your decisions as part of your testing strategy document. When you design test cases, the testing strategy document should describe how test cases were produced and why they provide useful coverage. The test cases and the results of testing can also be part of this assurance case.
Extreme programming (XP) is a collection of ideas about how to structure team programming projects. It emphasizes incremental development of code: there is always something that can be tested. Testing is not something that happens after implementation; instead, continuous testing is used to catch errors early. Thus, it is important to develop unit tests immediately when the code is written. Automating test suites is crucial so that continuous testing requires essentially no effort.
Extreme programming emphasizes bottom-up development of code modules for which units tests can be developed. Extreme programming also emphasizes social process on programming teams. Programmers work closely, communicating more verbally than with formal design documents. Specifications are short and programmers rely on common knowledge among the development team. Extreme programming is a good way to quickly development software, but it has been criticized for not producing high-level design documents. In the long run a project developed this way might be hard to manage, because the code is wedded to the engineers.
An extremely effective way to find faults in software systems is code review, in which the programmer goes through the code and documentation to explain how it works and why it is correct. It is the programmer's job to convince an expert review team (usually 1–3 people) who act as skeptical devil's advocates.
There are different ways to use code review effectively:
In the walkthrough approach, the programmer presents the documentation and code to the reviewing team, and the team gives comments. This is an informal process. The focus is on the code rather than the coder, so hurt feelings are easier to avoid. However, the team may not get as much assurance that the code is correct.
Here, the review team drives the code review process. Some, though not necessarily very much, team preparation beforehand is useful. They define goals for the review process and interact with the coder(s) to understand where there may be quality problems. Again, making the process as blameless as possible is important.
The most informal approach to code review is through pair programming, in which code is developed by a pair of engineers: the driver who writes the code, and the observer who watches. The role of the observer is be a critic, to think about potential errors, and to help navigate larger design issues. It's usually better to have the observer be the engineer with the greater experience with the coding task at hand. The observer reviews the code, serving as the devil's advocate that the driver must convince. When the pair is developing specifications, the observer thinks about how to make specs clearer or shorter.
Pair programming has other benefits. It is often more fun and educational to work with a partner, and it helps focus both partners on the task. If you are just starting to work with another programmer, pair programming is a good way to understand how your partner thinks and to establish common vocabulary. It is a good idea for partners to trade off roles, too. Pair programming is an excellent approach for CS 3110, and we recommend trying it.
Code review can be remarkably effective. In one study conducted at IBM (Jones, 1991), code inspection found 65% of the known coding errors and 25% of the known documentation errors, whereas testing found only 20% of the coding errors and none of the documentation errors. The more formal inspection process may be more effective than walkthroughs. One study (Fagan, 1976) found that code inspections resulted in code with 38% fewer failures, compared to code walkthroughs.
Through code review can be expensive, however. Jones found that preparing for code inspection took one hour per 150 lines of code, and the actual inspection covered 75 lines of code per hour. Having up to three people on the inspection team improves the quality of inspection; beyond that, more inspectors doesn't seem to help. Spending a lot of time preparing for inspection did not seem to be useful, either. Perhaps this is because much of the value of inspection lies in the interaction with the coders.
These notes borrow from material originally developed by Michael Clarkson.