Tips for testers

11 November

If you are lucky enough to have specifications to base your testing on there’s always a strong possibility that the author hasn’t captured the requirement as well as they could have.

Can you spot anything wrong with the following statements?

  • March planned for Next August
  • Quarter of a Million Chinese Live on Water
  • Panda Mating Fails; Veterinarian Takes Over
  • Squad Helps Dog Bite Victim
  • Stolen Painting Found by Tree
  • Red Tape Holds up NewBridge
  • New Study of Obesity Looks for Larger Test Group
  • Astronaut Takes Blame for Gas in Spacecraft
  • Kids Make Nutritious Snacks

Even though these amusing statements are not likely to be found in software specifications they do help illustrate the problem of ambiguity.

The following checklist was prepared for the U.S. Air Force a long time ago by MITRE Corporation, but the principles hold true. The authors suggest being on the look out for:

1. Incomplete lists ending with “etc.,” “and/or,” and “TBD.”

2. Vague words and phrases such as “generally,” “normally,” “to the greatest extent,” and “where practicable.”

3. Imprecise verbs such as “supported,” “handled,” “processed,” or “rejected.”

4. Implied certainty, flagged by words such as “always,” “never,” “all,” or “every.”

5. Passive voice, such as “the counter is set.” (By whom or what?)

6. Every pronoun, particularly “it” or “its.” Each should have an explicit and unmistakable reference.

7. Comparatives, such as “earliest,” “latest,” “highest.” Words ending in “or” or “est” should be suspect.

8. Words and phrases that cannot be quantified, such as flexible, modular, achievable, efficient, adequate, accomplish, possible (or possibly), correct (or correctly), minimum required, minimum acceptable, better, higher, faster, less, slower, infrequent, to the extent specified, to the extent required, be compatible, to be associated with.

9 Words and phrases whose meaning can be disputed between developer and customer, such as instantaneous, simultaneous, achievable, finish, degraded, a minimum number of, nominal/normal/average, minimum, steady-state, coincident, adjacent, synchronous.

10. Contractually troublesome phrases:
a. “Design goal.” The developer will spend money and other resources with no guarantee of goal accomplishment.
b. “To the extent practicable.” A decision in the eyes of the developer.
c. “Where applicable.” There are no criteria for judgment.
d. “Shall be considered.” The developer will think about.
e. “A minimum of X.” The developer will provide exactly X.

Most of the difficulty with the fuzzy requirements addressed by the MITRE report arises from the imprecision of the English language as written by most people.

To clarify requirements, the specification author should equations and logical relations to express constraints and computational requirements.

“The system shall do A. After completing A, the system shall do B. After completing B, the system shall do C. After completing C, the system shall do D. Upon completing D, the system shall do E.” If the sequential relationship between tasks A, B, C, D, and E is other than linear, then use a logic diagram: “The system shall perform tasks A, B, C, D, and E as shown in Figure X.”

Finally, a word about the often abused phrase “To Be Determined” or “‘TBD.” TBD is used as a placeholder for requirements that haven’t been finalized. TBDs are meant to be conspicuous and easy to spot. They’re a message to readers that” there’s something missing, but we haven’t forgotten it.” When used that way, TBDs serve a very useful purpose. Specification reviewers who categorically reject specifications containing TBDS are simply inviting developers to submit specifications with much more subtle TBDs which may end up being missed anyway.

I hope this list helps you in your quest to improve the project documentation in your organisation and to find and fix those defects earlier, saving both time and money for the project.


Post by Toby Thompson