Quality Assurance Plan
Todd Plessel, Lockheed Martin / US EPA
Contents
The purpose of this plan is to briefly summarize the notion of quality as it pertains to software development and define the development process and product standards that will be applied to achieve high-quality software products.
What is Software Quality?
Software qualities can be associated with either the process or the product:
Software Process: The human activity associated with software development.
Software Product: The software and artifacts resulting from that process.
Software Process: Principles of Quality Improvement
The Problem:
People are imperfect, programmers are people, therefore programmers are imperfect.
Programs must be near-perfect (have minimal defects) to yield desired results.
How can imperfect programmers create near-perfect products, especially since programmers are often blind to their imperfections and those of the products they create?
The Solution:
By either improving their personal abilities to create less-defective products or improving the detection and correction of defects in the products they produce. These are the a priori / personal / long-term vs a posteriori / impersonal / short-term approaches. Ideally, both should be employed.
Personal Improvement:
There is strong evidence that personal ability is the most significant factor that determines the quality of the resulting product. Personal improvement is the first and largest quality filter. How can it be maximized?
Education and Training
Professional software developers owe it to themselves, their employer, their profession and the users of their products to improve their own personal abilities to yield higher quality products. One way is through continuing education and training in the relevant skills. This includes both theoretical and empirical results. Some portion of their professional life should be dedicated to such activities, for example, taking appropriate courses, reading books and journals, attending conferences and experimenting.
Learn from personal experience.
Take time to look back on and examine personal experiences in developing software and identify important lessons learned. Write them down and share them with your colleagues. After all, "Those that ignore history are doomed to repeat it."
Learn from the experience of others.
"Experience is a good teacher. Some would have no other." It is not enough to only consider one's (relatively limited) personal experience. It is better to learn from the mistakes of others than to repeat them yourself. It is better to make leaps and bounds toward best practices than eventually (maybe) discover them yourself. Seek out the valuable lessons - of what didn't work and what worked well - and, after analysis and possible modification, adopt them.
Measure your progress and make changes that yield improvements.
Humphrey's Personal Software Process is an example framework for guiding such process improvements. Note that this is a personal process that is individualized and private. But if you discover approaches that yield major improvements for you, you may want to share them with others.
Product and Process Improvement:
No matter what level of personal abilities has been achieved, the resulting products will still contain defects. This is where tools and processes can be applied to further filter product defects before they reach the end-user.
Tools include: static analyzers (e.g., syntax-aware editors, compilers, lint-like utilities, metrics) and dynamic analyzers (e.g., instrumentation utilities such as Purify, pixie, prof). Processes include requirements analysis, user-centered design, peer review, testing, and measurement and evaluation of results. These tools and processes will act as 'back-end' quality filters that yield higher-quality products than would otherwise result.
Generic Improvement Process:
An important goal is to maximize productivity of the development staff. This will be achieved, in part, by streamlining the entire development process as much as possible without sacrificing the (high) targeted quality standards. For SQA processes, this means streamlining 'traditional' SQA activities by minimizing the human-intensive bureaucratic aspects and maximizing automation by applying CASE tools where possible and needed. Specifics are outlined below.
Software product qualities derived from the applied software process:
Timeliness: The ability of a software system to be made available when or before its users want it.
Affordability: The financial cost of developing or acquiring and using the software.
Software Product: Quality Factors, Modularity Criteria and Design Principles
Software product qualities can be external or internal [Meyer]:
External Quality Factors: Quality properties observable from executing a program.
These include: correctness, robustness, efficiency, ease of use, functionality.
Internal Quality Factors: Quality properties perceptible by programmers reading the source text.
These include: correctness, robustness, efficiency, functionality, extendibility, compatibility, portability, understandability and testability.
Primary Product Qualities:
Correctness: The ability of software to perform their exact tasks as defined by their specification.
Robustness: The ability of software systems to react appropriately to abnormal conditions.
Efficiency: The ability of a software system to place as few demands as possible on hardware resources such as processor time, internal and external memories and communication bandwidth.
Ease of use: The ease with which people of various backgrounds and qualifications can learn to use software products for their intended role.
Functionality: The extent of possibilities provided by a system. The applicability of software to solve a range of problems.
Extendibility: The ease of adapting software to changes of specification.
Compatibility: The ease of combining software elements with others.
Portability: The ease of transferring software products to various hardware and software environments.
Reusability: The ability of software elements to serve for the construction of many different applications.
Understandability: The ease with which a programmer can understand the source code of a software system.
Testability: The ease of testing a software system for correctness and robustness.
Derived Product Qualities:
Integrity: The extent to which the software protects against data corruption, loss and information privacy.
Security: The extent to which the software protects against unauthorized access and denial of service while still providing service to authorized users.
Accuracy/Precision: The degree of fidelity in which information is represented by the software.
Reliability: The probability of the software to execute without failure for a specific period of time under specific operating condition.
Availability: The percentage of time that the software system is fully functional. MTTF / (MTTF + MTTR).
Adaptability: See Extendibility.
Flexibility: See Extendibility.
Readability: See Understandability.
Interoperability: See Compatibility.
Integrability: See Compatibility (above) and Composability (below).
Performance: The responsiveness of the system - the time required to respond to stimulus or the number of events processed in some interval of time. See Efficiency.
Maintainability: The ease of changing the software to correct defects or meet new requirements. (Understandability x Extendibility x Portability x Testability.)
Learnability: The ease of which a person can learn to use the software. See Ease of Use.
Memorability: The ease of which a person can remember how to use the software. See Ease of Use.
Error Avoidance: The extent to which the software prevents the user from using it incorrectly. See Ease of Use.
Error Recovery: The extent to which the software helps the user recover from using it incorrectly. See Ease of Use.
Consistency: The extent to which the software exhibits uniformity throughout with respect to a given choice of implementation.
Satisfiability: The extent to which the software satisfies the user. The product of all external factors.
Buildability: See Decomposability and Composability (below).
Completeness: The extent to which the software meets its requirements.
Interoperability: See Compatibility.
Cohesion: The extent to which software modules contain only strongly-related data and operations.
Decoupling: The extent to which software modules exhibit minimal functional interdependence.
Replaceability: The extent to which a software module can be replaced by another. See Decoupling.
Modularity: The extent to which the software is structured into clearly-defined units of composition/decomposition. See below.
Modularity Criteria, Rules and Principles [Meyer]
Related to software quality as a whole, are qualities associated with the structure of the software.
Decomposability: The ability of a software construction method to help in decomposing a software problem into a small number of less complex subproblems, connected by a simple structure, and independent enough to allow further work to proceed separately on each of them.
Composability: The ability of a software construction process to yield modules that can be freely combined with others to produce new systems.
Understandability: The ease with which a programmer can understand a given module without having to examine many others.
Continuity: A small change in specification requires only a small change in implementation (a small number of modules).
Protection: The ability of modules that, at runtime, limit the propagation of failures to few, if any, other modules.
Direct Mapping: The modular structure for modeling a problem should be reflected in the modular structure of the software that solves the problem.
Few Interfaces: Every module should communicate with as few others as possible.
Small Interfaces (weak coupling): If two modules communicate, they should exchange as little information as possible.
Operand Principle: The arguments of a routine should only include operands (no options - "modes of operation").
Explicit Interfaces: Whenever two modules A and B communicate, this must be obvious from the text of A or B or both.
Information hiding: A module must allow only a subset of its properties, called the public interface, to be accessible to client modules while ensuring that the rest remains secret (unknowable and inaccessible).
Law of Demeter: Never retrieve part of an object and then perform an operation on that part, but rather have the object perform the operation itself.
Linguistic Modular Units Principle: Modules correspond to syntactic units in the language used. (Module = Type = Class.)
Self-Documentation Principle: Modules contain canonical minimal and complete documentation on their use. (To facilitate automatic extraction of such information by software tools.)
Uniform Access Principle: All services offered by a module should be available through a uniform notation, which does not betray whether they are implemented through storage or computation.
Command-Query Separation Principle:
Features should be of two logical types:
Commands: Perform an action - conceptually changing the object state - but do not return a value.
Queries: Answer a question without abstract side-effect (i.e., without changing conceptual state).
Open-Closed Principle: Modules are 'closed' - available for immediate use, as is, by clients and 'open' for extension by descendants inheriting from it.
Liskov Substitution Principle : derived classes must usable through the base class interface. This is polymorphism implemented by dynamic binding.
Single-Choice Principle: Whenever a software system must support a set of alternatives, at most only one module in the system should know their exhaustive set. (E.g., table-driven approaches.)
Dependency Inversion Principle: High-level modules should not depend on low-level modules. Both should depend on abstractions. Abstractions should not depend upon details. Details should depend upon abstractions.
Quality Trade-off Relationships [Wiegers and McConnell]
|
How increasing the factor below affects the factors to the right |
Availability | Efficiency | Flexibility | Integrity | Interoperability | Maintainability | Portability | Reliability | Reusability | Robustness | Testability | Usability | Correctness | Adaptability | Accuracy |
| Availability | + | + | + | ||||||||||||
| Efficiency | + | - | - | - | - | - | - | - | - | ||||||
| Flexibility | - | + | - | + | + | + | + | ||||||||
| Integrity | - | + | - | - | - | - | |||||||||
| Interoperability | - | + | - | + | + | ||||||||||
| Maintainability | + | - | + | + | + | + | |||||||||
| Portability | - | + | + | - | + | + | + | - | |||||||
| Reliability | + | - | + | + | + | + | + | + | |||||||
| Reusability | - | + | - | + | + | + | - | + | + | ||||||
| Robustness | + | - | + | + | + | ||||||||||
| Testability | + | - | + | + | + | + | + | ||||||||
| Usability | - | + | - | + | |||||||||||
| Correctness | + | + | + | -? | + | + | |||||||||
| Adaptability | - | + | + | ||||||||||||
| Accuracy | - | + | -? | + | - | + |
Meyer's Quality First approach will be used for long-term research projects with a large potential impact. Described in more detail elsewhere, it essentially works like this: while funding remains available, functionality is prioritized and then incrementally and carefully implemented without degrading other qualities. That is, the software product qualities are ranked such that functionality is grown while other qualities (correctness, robustness, extendibility, etc.) are sustained.
Refactoring is another term that describes the commitment to identifying and making the necessary changes to ensure that each time a new increment of functionality is added, the resulting version of the software is not degraded. Sometimes this requires substantial effort and/or changes to the system to uphold the other qualities and this is accepted as preferable to "quick and dirty" hacks to achieve timely functionality at the expense of all other qualities. ("The problem with quick-and-dirty is the quick is forgotten long after the dirty remains.")
Besides the application of superior methods and tools (read Eiffel), other time-honored effective processes such as coding standards, peer reviews, unit testing and system testing will be employed as essential defect-removal filters.
Level I. Essential
Failure to meet these qualities could result in termination of the project.
Level II. Important
Failure to meet these qualities could result in dissatisfaction with the project.
Some qualities are non-negotiable meaning that, for this project, such qualities are not allowed to be sacrificed (consciously by design) for the sake of improving other qualities. In general, work on achieving and upholding such qualities will be higher priority than work on achieving and upholding all other qualities - "Quality First". This means that, for example, discovered Level I defects should be removed before pursuing improvements to other qualities ("fixes" before "enhancements"). Occasionally, however, extenuating circumstances may require postponing such work temporarily to meet other goals and obligations.
Some qualities are considered binary meaning they are subject to an objective pass/fail test that determines whether or not the quality is achieved. This achievement is not in an "absolute global" sense but rather only with respect to the given pre-specified test.
Others are considered continuous in nature but may still be subject to an objective pass/fail test to determine if the quality has been (adequately) achieved.
Note that the ranking of the qualities does not imply that lower-ranking qualities need only be considered later in the development process, but only that their impact (of having been achieved or not achieved) will be revealed later - like saving for retirement. The time/phase in which to consider a quality is orthogonal to its ranking. For example, a relatively low-ranking quality such as Extendibility, must be considered during early design phases - not postponed until later - since it cannot be retrofitted after coding has already been done.
Put another way, each phase of development activity must be done with an awareness of all of the desired ranked qualities so that appropriate trade-offs are consciously made. However, once appropriate consideration has been given, a phase activity will naturally focus on some subset of the qualities. For example, the Unit Testing activity is focused on the qualities: Correctness, Robustness, Efficiency and, indirectly, Targeted Platform Portability and Targeted Component Compatibility and Testability, while the Architectural Design activity is primarily focused on Extendibility, Reusability and their derived qualities.
Another important point is that the impact relationship between the qualities is orthogonal to their ranking and the time phase of their impact. For example, increasing Efficiency is well-known to negatively impact most other qualities regardless of how we choose to rank them or when we choose to focus on them. See the quality trade-off relationship matrix showing the positive and negative impacts (on each other quality) of increasing each given quality.
Level I. Continuous Impact
Failure to achieve these qualities would likely result not only derail the project, but could also have significant far-reaching negative impact. (E.g., quietly corrupting existing, widely-used and distributed data used for decision-making due to incorrect program logic. Or halting significant work on sub-projects due to unavailable or incompatible needed components.)
Level II. Immediate Impact
Failure to achieve these qualities would seriously degrade the overall effectiveness of using or building the component. (E.g., too awkward or slow to use or too hideous to readily comprehend.)
Level III. Imapct Soon
Failure to achieve these qualities will inevitably have a significant cumulative negative impact on the long-term effectiveness of using or maintaining the system. (E.g., over its lifetime, a legacy system may eventually become non-cost-effective to maintain - "geriatric software maintenance".)
Level IV. Impact By End Of Project
Failure to achieve these qualities means the project is not a complete success. E.g., it does not do everything that was specified or does not do it as well as required.
Level V. Long-Term Impact
Failure to meet these qualities means decreased cost-effectiveness of the developing organization on future projects.
The idea of the levels is to provide a framework for evaluating design and implementation decisions for their impact on various qualities. For example, a change that would increase a quality like Optimal Efficiency (IV.2) but decrease a higher-ranking quality like Targeted Platform Portability (I.2) should usually be rejected. Likewise a change that would increase, say, Extendibility (III.1) at the expense of Good Enough Efficiency (II.1) should also be rejected. Of course, there are few absolutes and each case must be carefully considered for its specific impact, but it is essential to have an a priori agreed-upon framework to perform the weighted pro-con analysis so that there is an increased chance of upholding architectural integrity in such a large-scale long-term group project.
Techniques For Achieving Qualities
| Quality | Technique | Benefit |
| Correctness and Robustness | Strong Static Typing | No runtime type mismatch failures. |
| Automatic Initializations | No uninitialized data reads. | |
| Automatic Memory Management | No incorrect memory allocation, deallocation or accesses. | |
| Design By Contract | Defines and documents correct usage of interfaces. Traps interface usage errors. | |
| Disciplined Exceptions | Traps ignored, bypassed or insufficient handling of runtime resource acquisition or usage failures. | |
| Coding Standards | Highlight conventions important to ensure correctness and robustness. | |
| Code Review | Traps higher-level design defects (e.g., inconsistencies, omissions). | |
| Unit Testing | Traps lower-level code defects (e.g., flawed control-flow, data-flow, sequencing, and efficiency). | |
| Testability | Development Environment | Provides tools to assist in (CASE tools for metrics evaluating the testability of and testing) and testing developed components. |
| Coding Standards | Highlight conventions important to ensure testability. | |
| Code Review | Identifies potential testability problems in code. | |
| Unit Testing | Identifies actual testability problems. | |
| Portability and Compatibility | Development Language and Environment (supporting automatic dependency analysis, adapting and renaming external interfaces, etc.) | Consistency and ease of building applications on different platforms and in combining independently developed components (e.g., assists in adapting and components.) |
| Coding Standards | Highlight conventions important to ensure portability and compatibility. | |
| Code Review | Identifies potential portability and compatibility problems in code. | |
| Unit Testing | Traps non-portable and incompatible code at compile-time (on each supported platform). | |
| Usability | Coding Standards | Highlight conventions important to ensure ease of use. |
| Code Review | Identifies potential usability problems in design or code. | |
| User-Centered Design (UCD) | Obtains early and continuous input from the actual users. | |
| GOMS Analysis | Provides an objective measure of the complexity (number and magnitude of mental and physical actions required) of using an application. | |
| (User) System Testing | Uncovers usability problems with applications during development rather than during production use. | |
| Learnability | Built-in hands-on tutorials. | Provides at-a-glance walkthroughs of application features. |
| UCD | Obtains early and continuous input from the actual users. | |
| GOMS | Provides an objective measure of the complexity (number and magnitude of mental and physical actions required) of using an application. | |
| Use of 'real world' concepts/objects/metaphors. | Leverages existing user knowledge. | |
| Memorability | UCD | Obtains early and continuous input from the actual users. |
| GOMS | Provides an objective measure of the complexity (number and magnitude of mental and physical actions required) of using an application. | |
| Use of 'real world' concepts/objects/metaphors. | Users will perceive the system as natural and intuitive. | |
| Minimize the number of 'objects' that must be understood. | Users will perceive the system as simpler. | |
| Error Avoidance | UCD | Obtains early and continuous input from the actual users. |
| GOMS | Provides an objective measure of the complexity (number and magnitude of mental and physical actions required) of using an application. | |
| Built-in metadata to allow automatic detection and prevention of misuse. (E.g., gray-out inapplicable menu selections.) | Avoids misuse. | |
| Error Recovery | UCD | Obtains early and continuous input from the actual users. |
| GOMS | Provides an objective measure of the complexity (number and magnitude of mental and physical actions required) of using an application. | |
| Watchdog/Heartbeat/time-outs. | Detect possible failures and correct them (e.g., restart). | |
| Consistency | UCD | Obtains early and continuous input from the actual users. |
| GOMS | Provides an objective measure of the complexity (number and magnitude of mental and physical actions required) of using an application. | |
| Nested Design: repeat design aspects through the different layers. | Enforces architectural integrity. | |
| Chief Architect with authority and responsibility for overall design and constraints. | Enforces architectural integrity. | |
| Understandability (of source code) | Readable Language Syntax | Comprehension needed to evaluate code for other qualities. |
| Simple Language Semantics | Learnability, memorability and consistency of representation. Minimized arbitrary complexity by elimination of unstructured control flows and removal of redundant unnecessary variation. | |
| Coding Standards | Consistency of form and content to minimize the cognitive load (distraction) during reading. | |
| Code Review | Identifies hard-to-understand code for rewriting. | |
| Extendibility and Reusability | Information Hiding | Limits the locality of impact of changes. |
| Abstract Data Types | Provides systematic structure for implementing Information Hiding. | |
| Separation of Concerns | Ensures cohesion and provides guidance for allocating a work breakdown structure. | |
| Layering | Limits the locality of impact of changes. Enables/enhances portability and reusability. | |
| Balanced Distribution of Responsibility | See Separation of Concerns. | |
| Multiple Inheritance | Leverages existing code implementations and provides substitutability of components i.e., decomposability and composability. | |
| Genericity | Factors out common code while allowing variation by type. | |
| Design Patterns | Yield designs that are optimized for various design goals under certain forces and constraints. | |
| Framework Architecture Design | Reuse control flow while allowing variation in computation. | |
| Coding Standards | Highlight conventions important to ensure modifiability. | |
| Code Review | Identifies potential modifiability problems in code. | |
| Generalization / Refactoring process step | Uncovers and corrects actual modifiability deficiencies. | |
| Functionality | Development Process and Project Management | Systematically identifies, prioritizes, controls, tracks and validates incrementally growing functionality. |
| Development Environment (software libraries) | Provides pre-built high-quality reusable components. | |
| Efficiency | Knowledge of algorithms and computer architecture | Leads to the selection of appropriately efficient implementations for the particular problem at hand. |
| Coding Standards | Highlight conventions important to ensure efficiency. | |
| Code Review | Identifies potential efficiency problems in code. | |
| Unit Testing | Identifies actual measured efficiency problems. | |
| Timeliness and Affordability | Good Development Infrastructure (contracts, staff, facilities) | High productivity relative to cost. |
| Balanced economical and technical constraints | Ensures stakeholder commitment. | |
| Development Environment (hardware, software) | Productivity enhanced by availability and dedicated use of good tools. |
2. References
Some relevant literature used as a basis for this quality development process is listed below. See Project Lead for copies of these and other relevant publications in our in-house library.
Pressman, Roger S., Software Engineering: A Practitioner's Approach, McGraw-Hill, New York, 1997.
Davis, Alan M., 201 Principles of Software Development, McGraw-Hill, New York, 1995.
Humphrey, Watts S., A Discipline For Software Engineering, Addison Wesley, Reading MA, 1995.
Gould, John D., "How To Design Usable Systems", Handbook of Human-Computer Interaction (Martin Helander Ed.), North-Holland: Elsiver, 1988, pp 757-789.
Johnson, Philip M., "Reengineering Inspection", Communications of the ACM, vol. 41, no. 2, February 1998, pp 49-52. See also, Internet Formal Technical Review Archive.
Meyer, Bertrand, Object-Oriented Software Construction, Second Edition, Prentice Hall, Upper Saddle River, New Jersey, 1997.
___, "Practice To Perfect: The Quality First Model," IEEE Computer, vol. 30, no. 5, May 1997, pages 102-106.
Plessel, Todd, Design By Contract: A Missing Link In The Quest For Quality Software.
Plessel, Todd, Why Eiffel.
Dromey, R.G., "A Model For Software Product Quality," Technical Report, Software Quality Institute, Griffith University, October, 1994.
Booch, Grady, Object-Oriented Analysis And Design With Applications, Second Edition, Benjamin Cummings, Redwood City, CA, 1994.
Bass, Len, et al., Software Architecture In Practice, Addision Wesley, Reading, MA, 1998.
Gamma, Erich, et al., Design Patterns: Elements of Reusable Object-Oriented Software, Addision-Wesley, Reading MA, 1995
Martin, Robert, "OO Design Quality Metrics," September, 1995.
Musa, John, Software Reliability Engineering, McGraw-Hill, 1998.
Beizer, Boris, Software Testing Techniques, Second Edition, Thompson Computer Press, London, 1990.
Binder, Robert V., "Modal Testing Strategies For OO Software", IEEE Computer, vol. 29, no. 11, November 1996, pp 97-99.
___, Testing Object-Oriented Systems: Models, Patterns and Tools, Addision Wesley, Reading, MA, 2000.
Cargill, Tom, "Short Tour Testing", C++ Report, vol 7, no. 2, February 1995, pp 60-62.
Cargill, Tom, C++ Programming Style, Addison Wesley, Reading, MA, 1992.
Meyer, Scott, Effective C++: 50 Specific Ways to Improve Your Programs and Designs, Addison Wesley, Reading, MA, 1992.
Meyer, Scott, More Effective C++: 35 New Ways to Improve Your Programs and Designs, Addison Wesley, Reading, MA, 1996.
Coplien, James, O., Advanced C++: Programming Styles and Idioms, Addision Wesley, Reading, MA, 1992.
Ellemtel C++ Style Guide (See exceptions below)
Sutter, Herb, Exceptional C++, Addison-Wesley, 1999. (Includes PeerDirect, Inc. C++ Coding Standards.)
Muller, Harald M., "Ten Rules For Handling Exception Handling Successfully", C++ Report, vol. 8, no. 1, January 1996, pp 23 - 36.
McConnell, Steve, Code Complete, Microsoft Press, Redmond, WA, 1993.
McConnell, Steve, Software Project Survival Guide, Microsoft Press, Redmond, WA, 1998.
Van Vleck, Tom, "Three Questions About Each Bug You Find", ACM SIGSOFT Software Engineering Notes, vol 14 no 5 July 1989 pages 62-63.
Weigers, Karl E., Software Requirements, Microsoft Press, Redmond, WA, 1999.
Software Program Managers Network: Principle Best Practices.
3. Management Organization, Tasks and Responsibilities
This section lists and illustrates exemplars of the documentation artifacts that are produced (and retained) during the development process.
4.2.1 Management Documentation: see Project Plan.
4.2.2 Technical Documentation: see Development Plan. which describes the production of quality-reviewed artifacts from the activity phases:
Periodically the following documents are produced by the Project Lead to assist in process evaluation.
5. Standards, Practices and Conventions
5.1 Purpose
This section briefly lists the quality assurance processes applied during development and reference standards which artifacts are evaluated against.
5.2 Practices
The following quality assurance practices are employed during development:
FORTRAN-90 Coding Standard
TBD. (But see the C / C++ Coding standard in the meantime. Also, use ASSERTIONS.H.)
A good start might be the ARPS F90 Coding Standard.
Java Coding Standard
TBD. (But see the C / C++ Coding standard in the meantime. Also, apply iContract for DBC.)
5.4 Failure/Defect Classifications
Failures, defects and errors are defined as follows:
Failure: an undesired effect of running the software. This is reported by users of the software - including peers conducting system (application) testing during development. Failures are externally observable without access to the software source code. There are six severity levels with more serious ones assigned higher numbers to facilitate simple summing. In addition, each has one or more quality factors that it is derived from. These are fixed (if there is only one choice) or must be specified by the user.
Note that failures are indications that the software contains defects and so should be fixed. This sense of failures excludes cases of failure handling in software that robustly detects and reasonably copes with exceptional runtime conditions that are beyond its control, namely, invalid user or data input and temporary inability to acquire shared resources (e.g., memory, disk space, network connection, etc.).
In fact, software that exhibits such failure handling behavior during "stress testing" is highly desirable and the goal and definition of robustness. But if frequent under "normal operational conditions", such failures can be viewed as failures of the system (human-hardware-software) though in such cases, it is not the software part of the system that needs to be corrected.
Defect: a fixable problem in the software's source code that caused (or could cause) a failure. This is determined by software engineers analyzing the source code (and other artifacts). Defects are the internal root cause of externally observed failure(s). There are three severity levels of the grouped quality factors:
Error: a human mistake that might result in defective software (or derived artifacts). Errors can be of three types: commission, omission or mission with corresponding resulting defects such as incorrect, missing or unneeded code. Errors are the root cause of defects.
Beizer's Bug Taxonomy
Failures are classified and recorded in the component's review.log and, if discovered by users during operation, logged via email and/or GNATS tracking system.
The purpose of reviews is: (1) increase the quality of the produced software artifacts (particularly source code) by identifying defects and improvements that could not be accomplished otherwise, and (2) to increase staff knowledge of how to produce high-quality software designs and implementations by learning better techniques from the review process.
1. Any and all potential improvements are game.
Traditional reviews such as Fagen inspections focus only on identifying (primarily correctness and robustness) defects in a software artifact - usually source code of the implementation. Our goal is broader. Requirements, designs, interfaces, implementations, and their artifacts are all reviewed and reviews are not limited to just identifying defects, but to identifying any and all feasible improvements.
2. No improvement is too small.
Consider the product (e.g., source code) as writing first and foremost for human consumption. It should be evaluated as such - just like essays are in a literature class. Minor improvements such as adjusting whitespace or changing an identifier name are akin to minor grammatical defects and style improvements. By definition, a minor improvement should require a minor effort and so is not worth wasting staff time arguing over.
3. Specific suggestions for alternatives are welcome.
Specific suggestions for defect corrections and improvements is a key part of the review process (unlike Fagen inspections). This aspect presents an opportunity for team members to learn about good designs and implementations so that they can become more effective producers of high-quality software.
4. Streamline by eliminating minimally-productive meetings.
Unlike Fagen inspections, our scheduled reviews are asynchronous and concurrent meaning that reviewers are not required to hold face-to-face structured meetings. This minimizes time lost to staff training (to learn how to perform meeting roles) and also "air-time" while awaiting your turn to present results in a meeting. See Reengineering Inspection for a discussion of this approach.
This is predicated on the idea that the main benefit of reviews - the specific list of suggested improvements - is developed prior to any meeting so why not obtain that by email, analyze it when convenient and eliminate the overhead of coordinating and conducting a meeting? (Studies at AT&T confirm that formal review meetings are overrated and 90% of the defects are found prior to the meeting.)
Instead use email to notify participants and clarify minor items. Use one-on-one meetings between the author and a single reviewer to clarify major items. Arrange for group meetings only as a last resort to discuss high-impact and/or controversial items.
5. Be prepared - don't waste staff time.
The submitter of the product to be reviewed (design, code interface, code implementation, etc.) is expected to have applied the available tools to statically analyze the product and correct any reported defects prior to submission for review. For example, code interfaces, such as header files, are expected to have been compiled (with maximal warnings enabled), linted (make lint), generated metrics (make metrics) and reported defects corrected prior to review. The author must be able to justify any remaining warnings indicated by the static analysis tools.
6. Management has final say on controversial decisions.
When a proposed change or major decision is high-impact and/or controversial, a meeting will be held with all concerned developers and each will be given the opportunity to present their view. These individual views will be recorded, discussed and analyzed as objectively as practical. Then a vote will be taken to see if there is consensus on a decision. If there is still no consensus, the appropriate lead or manager will have the final say on the matter. See organizational authority hierarchy.
7. The review process is amenable.
The review process is (like all processes here) amenable to improve its effectiveness, subject to scheduling and ultimate approval by the QA Lead.
6.3. Requirements Reviews
It is the nature of our scientific research environment that requirements are usually quite vague. Our users don't know exactly what they want in great enough detail to be specified in a form suitable for some traditional approaches. Often all we have to go by are sections of the Task Order. Contractual restrictions prohibit detailed discussions that would be required for serious cost/schedule estimation methods.
The best we can hope for is to have one-on-one discussions with at least one scientist (user) and try to briefly scope-out the work and negotiate the features and schedule for the "first-cut". The time frame in this research and development environment simply does not permit many of the traditional approaches for "nailing down" requirements. Therefore the basic operating principle is "deliver something for the users to try out and solicit and be very responsive to feedback". This is not a license to deliver quick-and-dirty software, but simply a recognition that timeliness is important and functional requirements are subject to considerable change.
Requirement scoping and specification is developed by the assigned Software Engineer through dialog with the designated User(s) and/or Customer. These documents are subject to review by the Project Lead and Chief Architect.
6.4. Analysis and Design Reviews
Analysis and design artifacts are reviewed by peers and/or the Chief Architect for conformance to the selected project architecture and technical approaches. However, for relatively simple / already understood components, analysis artifacts are not required and design artifacts may consist of just the minimal requirement: deferred / stubbed classes with contracts.
6.5. Code Verification and Validation Reviews
Source code (the most important artifact) is reviewed by peer Software Engineers and the Quality Assurance Lead for support of quality factors and possible improvements.
6.6 Unit Testing Reviews
The Quality Assurance Lead assesses the completeness of the unit testing process with respect to the achievement of the project standards including: 100% predicate coverage and evidence of a variety of testing techniques such as ADUP, domain / boundary-value, etc.
6.6. Functional Audit
Peer software engineers perform system testing (of applications) including achievement of functionality and performance called for in the requirements specification and ease of use.
6.7. Physical Audits
The Quality Assurance Lead verifies the correctness and completeness of the official repository and installability of platform-specific runtime releases.
6.8. In-process Audits
There are reviews performed at the end of each "development step" by both peers and leads.
6.9. Management Reviews
The Project Lead conducts periodic (monthly) metrics collection of data summarizing the project status.
7. Testing
Scope and Process
Two levels of testing are employed: unit and system. Unit testing is performed on each component by its author while system testing is performed (only) on applications and is done by peers. User validation testing of applications (not currently formalized) can occur after (frequent) releases. Note: integration testing is considered unit testing of the component that "integrates" the other components. Components are implemented and tested bottom-up, that is, lower-level components needed by others are implemented and unit tested before those components that need them are built. Thus unit testing assumes that components used are correct and focuses only on code that is implemented in the component itself. Defects found in lower-level components are forwarded to their authors for correction. Authors of each component are expected to be responsive to needed corrections which will generally take priority over other activities.
This document contains descriptions of the unit testing process and system testing process. Which includes the use of testing CASE tools and the generation (and retention) of testing artifacts. And there are references to other testing resources.
8. Problem Reporting and Corrective Action
Defects encountered during development and post-release are recorded in a review.log file associated with each cluster/library to facilitate subsequent process analysis (e.g., defect removal efficiency).
9. Tools, Techniques and Methodologies
The complete list of CASE tools used are listed elsewhere, but include a variety of language-specific (C/C++, Eiffel) development tools such as: compilers, static checkers, dynamic checkers, coverage and profiling testing utilities, forward and reverse engineering design tools, document generators, text and HTML editors, metrics utilities, spreadsheets and other drawing applications.
Object-oriented Eiffel techniques are used including Design By Contract and other important notions described in OOSC2. C/FORTRAN development will use object-based approaches (lacking inheritance) for encapsulation, but will to the extent possible adhere to principles of the Eiffel method.
We work within a scientific research and development agency and devote a considerable amount of effort in researching, evaluating and applying new leading technologies for visualization, supercomputing and programming.
10. Code Control
The Project Librarian is responsible for maintaining the repository of artifacts that have been integrated.
11. Media Control
The main host for the repository is backed-up daily. Additional copies of the repository also exist on alternate development hosts located in different physical locations.
12. Supplier Control
We have no official contracts to control vendors of the hardware and software we use (beyond basic maintenance agreements).
13. Records Collection, Maintenance and Retention
The Project Librarian is responsible for maintaining the repository of artifacts that have been integrated. The Project Lead is responsible for tracking and analyzing project summary data.
14. Training
Staff have considerable experience in the domain (scientific visualization) and are continuing to receive training in relevant areas through graduate-level courses, reading and attending conferences. Additional software engineering and other relevant training courses may become available through local organization services.
15. Risk Management
See RMMM Plan.
See above definitions of Process and Product Qualities and also definitions and classifications of Failure, Defect and Error.
Other Glossary terms [Meyer]:
Abstract Data Type (ADT): A set of mathematical elements specified by listing the functions applicable to all these elements and the formal properties of these functions.
Attribute: the description of a field present in all instances of a class.
Component: an element of software that can be reused in many applications.
Class: a partially or totally implemented abstract data type. Serves both as a module and as a type (or type pattern if the class is generic).
Class invariant: An assertion which must be satisfied on creation of every instance of a class, and preserved by every exported routine of the class, so that it will be satisfied by all instances of the class whenever they are externally observable.
Client: a class that uses the features of another, its supplier, on the basis of the supplier's interface specification (contract).
Cluster: a group of related classes or, recursively, of related clusters.
Deferred Class (aka Abstract or Virtual Base Class): a class which has at least one deferred feature.
Deferred Feature (aka pure virtual): a feature which, in a certain class, has a specification but no implementation.
Design By Contract: a discipline that applies the business contracting metaphor enforced by inherited assertions to the specification, documentation and verification of software component interfaces. Contracts precisely define correct usage of a component through its routine preconditions, postconditions and class invariant.
Effective Class (aka Concrete Class): a class that is not deferred.
Effective Feature (aka non-pure-virtual): a feature that is implemented either as a routine or an attribute.
Field: one of the values making up an object.
Function: a routine that returns a result. Queries do not change an object's state but return a result. Queries are implemented either as functions (if the result is computed) or attributes (if the result is stored).
Genericity: Parameterization of classes by other class types. For example, generic container classes that are parameterized by the type of the contained items. E.g., class LIST [EMPLOYEE], LIST [BOOK].
Interface: The assumptions that components can make of each other. Signature + Contract (+ Documented Constraints + Undocumented Constraints...).
Module: a unit of software decomposition. In OO, module = type = class.
Object: a run-time instance of some class.
Object-Based: built from Abstract Data Types (ADTs).
Object-Oriented: built from classes, assertions, genericity, inheritance, polymorphism and dynamic binding. ADTs + Inheritance.
Operator Overloading: the ability to create user-defined routines for implementing operators. For example operator + may be defined for a class MATRIX, thus allowing clients to express matrix addition with more mathematical syntax: m1 := m2 + m2 rather than explicit routine call syntax: m1 := m1.plus (m2).
Overloading: the ability to let a feature name denote two or more operations. For example, operator + may be defined in more than one class or within the same class but for more than one type, for example, multiplying matrices and matrices vs matrices and vectors: m1 := m1 *m2; v1 := m1 * v2.
Postcondition: An assertion attached to a routine, which must be guaranteed by the routine's body on return from any call to the routine if the precondition was satisfied on entry..
Precondition: An assertion attached to a routine, which must be guaranteed by every client prior to any call to the routine.
Procedure: a routine that does not return a result. Commands may change an object's state and are implemented by procedures.
Routine: a computation defined n a class, and applicable to the instances of that class. Along with attribute, one of the two forms of feature.
Signature of a feature: The type part of the feature's interface specification. (The contract is the other part.) For an attribute or function, includes the return type; for a routine, includes the number, sequence and type of arguments.