Volume 6, Number 2 - Software Quality Assurance

Lessons Learned in Software Quality Assurance

by Dr. Linda H. Rosenberg, Goddard Space Flight Center, NASA


Over the years, NASA has become increasingly reliant on software to provide the functionality of the systems it develops and uses. Software Quality Assurance (SQA) is critical to the success of every project, but the roles and responsibilities are often misunderstood. SQA covers all phases of the Software Development Process including safety, reliability, Independent Verification and Validation (IV&V), and metrics. However, it is often difficult for projects to understand the relationships and apply appropriate quality assurance at an affordable cost.

This article contains excerpts from a recently published book, Managing Software Engineering Knowledge [1]. Twelve lessons learned in Software Quality Assurance are presented here. The author learned these lessons through 10 years of working in the Quality Assurance Directorate at NASA’s Goddard Space Flight Center (GSFC).

Lesson #1

Project Managers and Software Developers need to understand what "Software Quality Assurance" is and how their project can benefit by its application.

Software Quality is defined in The Handbook of Software Quality Assurance in multiple ways, but concludes with the definition "Software Quality is the fitness for use of the software product" [6]. This definition implies the evaluation of Software Quality related to the specification and application of software quality. While this definition seems to be clear and unambiguous, the concept of quality is not. Kitchenham states quality is "Hard to define, impossible to measure, easy to recognize" [3]. Gilles states, "Quality is generally transparent when present, but easily recognized in its absence" [2]. While quality in theory can be defined, in practice and use, an absolute definition is elusive. This is part of the knowledge that needs to be captured - how to apply these abstract SQA concepts to real projects.

Lesson #2

Software Quality Assurance implementation is a balancing activity that must be tailored as project appropriate.

No project in the history of Software Development at NASA has had "enough" money, especially when it comes to implementing Software Quality Assurance activities. It is not possible to achieve all aspects of quality because of interrelationships. Software Quality Assurance engineers must determine, based on experience, what trade-offs are to be made within the specific project since each project has different objectives. Gilles defined some of these relationships between the quality assurance criteria e.g., an inverse relationship between usability and efficiency; a direct relationship between maintainability and reusability [2]. To make the best trade-offs, all relationships must be understood and experience used to interpret the impact.

Lesson #3

Software Quality Assurance must evaluate the process as well as the products.

Within NASA, SQA tends to focus on the products produced, such as the requirements documents, design, code listing, and test plans. But the focus of SQA is to monitor continuously throughout the Software Development Life Cycle to ensure the quality of the delivered products. This requires monitoring both the processes and the products. In process assurance, SQA provides management with objective feedback regarding compliance to approved plans, procedures, standards, and analyses. Product assurance activities focus on the changing level of product quality within each phase of the life cycle, such as the requirements, design, code, and test plan. The objective is to identify and eliminate defects throughout the life cycle as early as possible, thus reducing test and maintenance costs.

Lesson #4

There must be a Software Assurance Plan.

Most Project Managers feel they have too many plans, and the suggestion of another one for Software Quality Assurance might be the proverbial straw that breaks the camel’s back! But the success of any undertaking is to know what one is trying to achieve and how one expects to accomplish it, hence, the necessity of a plan for SQA. The plan will specify the goals, what is to be performed, standards against which the development work is to be measured, all relevant procedures, and the organizational structure of the Quality Assurance within the project. At NASA a software assurance plan is required.

Lesson #5

Software Quality Assurance must span the entire Software Development Life Cycle.

Software Development starts at the concept phase and continues through maintenance. SQA activities and funding should also start at the concept definition and continue through the entire life cycle. At each phase, there are activities that could be performed [4]. The project and the quality assurance office must work together to negotiate what activities are appropriate at each phase based on risks and resources. The Quality Assurance Plan should indicate what activities will be performed, the decision basis, and the trade-offs made.

Lesson #6

Requirements, the birthplace of successful projects.

Although SQA is performed across the entire life cycle, success of a project can often be determined by the attention paid to requirements. The earlier in the life cycle potential risks are identified, the easier it is to eliminate or at least manage the conditions that introduce that risk. Problems not found until the testing phase are approximately 14 times more costly to fix than if found and fixed earlier in the requirements phase. The first tangible representation of the capabilities produced is the requirements specification document, be they system, hardware, software, or operational requirements. The document also serves to establish the basis for all the project’s engineering management and assurance functions. If the quality of the requirements specification is poor, the project is at risk even before work begins. [7] Therefore, a specific lesson in SQA is the emphasis on requirements.

Lesson #7

Software Quality Assurance does NOT Equal Testing.

Too often project managers assume they have Quality Assurance since they are doing testing already, or believe they don’t need quality assurance until testing. These are incorrect assumptions based on lack of experience, understanding and/or knowledge. From the aspect of quality assurance, the purpose of testing is to:

Lesson #8

Metrics are a necessity.

Software Metrics are often ignored during the early Software Development Life Cycle phases and not actively associated with SQA - but should be. For SQA practitioners, with their responsibility for assuring both the processes and the products of the Software Development, measurement is critical. Throughout each of the life cycle phases metrics have proven invaluable in the evaluation of the quality of the products and processes [5].

Lesson #9

Safety and Reliability are critical aspects of SQA.

Safety is a team effort and everyone’s responsibility. Software within NASA is a vital part of the system and can impact the safety of the mission. Project managers, Systems Engineers, Software Leads and engineers, Software Assurance or Quality Assurance, and System Safety Personnel all play a part in creating a safe system. Reliability and safety cannot be tested at the end of a project; they must be built in as the software is being Developed.

Reliability impacts safety - a system cannot be deemed safe if it is not reliable.

Lesson #10

Independent Verification and Validation (IV&V) is an important tool within SQA.

NASA defines Software IV&V as a Systems Engineering Process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the Software Development Life Cycle. Without SQA, IV&V is expensive and often not as effective. SQA is a broad "blanket" across the project, overseeing all process and product activities, including software. IV&V focuses on only those processes and products determined to have the highest risk, doing an in-depth evaluation of them. IV&V should not be used on all projects, but instead as a tool to increase reliability and the probability of mission success! [4]

Lesson #11

Hardware Does NOT Equal Software!

The influence of Hardware Quality Assurance is evident in the Software Quality Assurance practitioner community, where hardware-intensive systems and typical hardware-related concerns predominate. Two issues which dominate discussions about hardware are time and operating conditions. Software however, is built with different constraints and considerations. NASA has grappled with these differences and the best approach for recognizing the differences (yet similarities) of hardware and software quality assurance. At GSFC, and within other NASA Centers, hardware and software QA are in one department. This allows for them to work together, since both are needed for a successful project, yet are still separate in other areas where needed.

Lesson #12

Risk Management is NOT Optional.

Risk is a daily reality on all projects, and Continuous Risk Management should become just as routine. It should be ongoing and comfortable and neither imposed nor forgotten. Like any good habit, it should seamlessly fit into the daily work.

Software Risk Management is important because it helps avoid disasters, rework, and overkill. The objectives of Software Risk Management are to identify, address, and eliminate software risk items before they become threats to success or major sources of rework. Good project managers are also good managers of risk. It makes good business sense for all software development projects to incorporate risk management as part of project management.


Software Quality Assurance (SQA) is an important aspect of Software Engineering. An aspect that is best learned not from a book, but through the experiences of those who have practiced it. SQA is itself comprised of many areas of Software Engineering, such as life cycle development, metrics, safety, and reliability. In all of these areas, extensive research has been done and theories provided. But SQA and its supporting areas must be applied based on the experiences of those within the industry where it is being applied. This article discussed lessons learned within the NASA community on areas of Software Engineering within Software Quality Assurance. These lessons are applicable to any Software Assurance activities, but must be tailored appropriately.

About the Author

Dr. Rosenberg serves as the Acting Assistant Director for Information Sciences / Chief Information Officer at NASA’s Goddard Space Flight Center. She is matrixed from her position as Chief Scientist for Software Assurance for Goddard Space Flight Center (GSFC), NASA in the Office of Systems Safety and Mission Assurance Directorate. She is also the former division chief of the Software Assurance Technology Office (SATO).

Dr. Rosenberg is a recognized International expert in the areas of software assurance, software metrics, requirements and reliability and serves on IEEE program committees in those areas. She has presented papers/tutorials and chaired sessions at many international conferences, including those sponsored by NASA, DoD, and AIAA. Dr. Rosenberg holds a Ph.D. in Computer Science, an M.E.S. in Computer Science, and a B.S. in Mathematics. She is a member of the Electrical and Electronic Engineers (IEEE), the IEEE Computer Society, the Association for Computing Machinery (ACM) and Upsilon Pi Epsilon.

Author Contact Information

Dr. Linda H. Rosenberg
Chief Sci. for Software Assurance
Goddard Space Flight Center, NASA
Greenbelt, MD 20771
Previous Table of Contents Next