Mission Ready Software

Since the 1980’s there has been an “either-or” mentality to software testing. Either you test the requirements or you test the design/code. It is a common myth that testing the requirements is all that is needed.  The fact is that if requirements based testing were sufficient there would be no failed projects and no world events due to software failures. Clearly this popular testing approach is not working. The facts[1] show that organizations that develop reliable software on time do both requirements and design/code testing.  This is why.

Testing only requirements may- at best -cover 40% of the code. Requirements based testing won’t cover:

Why is this approach so popular? Answer:

So how does one change this popular but ineffective approach? Requirements management tools such as DOORS present obstacles for testing anything but requirements. Some alternatives include:

There’s an old adage in software development that states “you can’t test in quality.”  Others state “software doesn’t break.”  Both of these miss the point as they relate to testing of code in development.  Highly Accelerated Life Testing (HALT) and Highly Accelerated Stress Screening (HASS) for electro-mechanical systems demonstrate that you can, and must, utilize testing which forces components to failure to identify and correct inherent design weaknesses and to objectively characterize the reliability of the system as a whole.  While it’s arguable that software does not “break” in the classic sense of a physical change which no longer conforms to original specifications, it does “fail” when it operates in a manner that was unintended with consequences which may range from undetectable by the user to catastrophic depending on the nature of the failure and the effect that failure has on application or system operation.  Just like the breakage of a mechanical component, software failures often require a repair function although it may be in the form of an application restart or system reboot as opposed to component reconditioning or replacement.  A comprehensive approach to testing the software is necessary for ensuring software quality and reliability and requirements-based testing alone is insufficient to meet these goals.

Requirements based testing is, by definition, only as good as the written requirements on which it is based.  Functions that may have been implemented but not explicitly defined in the requirement set may not be tested at all while those which are ambiguous or lack sufficient definition of edge cases may be only partially tested.  In a complex application or system, there is often a one-to-many or many-to-many relationship of requirements to test cases.  In these situations, completeness of the test cases is dependent on the level of understanding and degree of rigor of the engineers developing and executing the test cases.To ensure user satisfaction, it is important that a subset of the requirements reflect the user needs of the application or system from which lower-level requirements can then be derived.  The US Food and Drug Administration (FDA) refers to the demonstration of fulfillment of user needs as Validation with a demonstration of conformance to requirements defined as Verification.  The inclusion of actual or representative users during Validation testing is an important element in demonstrating that their needs are met with the application as developed.

The only way to fully understand the completeness of testing is the use of coverage tools that monitor the application in operation to determine which lines of code are being executed during the test cycle.  Even full structural coverage is not enough as the code may operate differently based on conditional flow and the variables which determine the flow. The Federal Aviation Administration (FAA) has adopted a standard for code coverage requirements based on the severity of consequence in the event of a software failure.  DO-178B/C defines five levels of risk for software components as shown in the table below.