News

This article is in split into two parts. The first part discusses acceptance criteria. The second part discusses how to get the “right” acceptance criteria. The paper argues that the terms “acceptance criterion” and “requirement” do not have the same meaning, but are closely related. 

Some people elicit requirements because it is their job to get them met.  Other people want to have their requirements met. The people talk to each other and eventually agree on a set of requirements that have acceptance criteria for new or modified products.

Let’s define a product as anything that has been produced. A produced thing could be any from the following list:

• New software
• Modified software
• New business process
• Modified business process
• New automated process
• Modified automated process
• New organisation structure
• Modified organisation structure
• New business strategy
• Modified business strategy

The list above is not exhaustive, it gives an idea of the almost limitless possibilities of what a product could be.

Ideally, the involved people work as a team and with varying degrees of formality to produce a product. Elicitors use varying degrees of formality to document the acceptance criteria for that product as defined by the staters.

Part 1: What are Acceptance Criteria

Testers should test a product before it is released for use by its users. The testers test the product to know whether it meets its functional and non-functional requirements. 

The functional requirements describe what a product must be able to do or be used for:  e.g. the car must be usable to make journeys. 

The non-functional requirements describe how well the product must meet its functional requirements: e.g. the car shall consume less than 7 litres of fuel per 100 kilometres travelled. 

A functional requirement is a statement of intent. 

A non-functional requirement is a measure of success.  There should be at least one and usually several of these per functional requirement.

See the standard ISO/IEC 25010 for a guide to functional and non-functional requirements.

When running tests, the testers use acceptance criteria based on requirements to validate and verify that the product or service meets its:

Functional requirement acceptance criteria:

• example criterion -- has the car been used to make journeys?

And non-functional requirement acceptance criteria:

• example criterion -- was the car’s fuel consumption less than 7lt/100k?

Each acceptance criterion is either met or not met – nothing in between.  No acceptance criterion is ever part met.

Validation vs Verification

Figure 1: Validation v Verification

The customers [1] of a product/service want to be sure that it is both valid and verified.

One part of the Agile Manifesto [2] says “we value customer collaboration over contract negotiation”.  What does that mean? An interpretation is that it is more important that users of a product or service see it as being fit for its intended purpose (valid) than that the purchaser agrees that the product or service meets its contracted obligations (verified).  Of course, in an ideal world, the users and the purchaser would both be satisfied.

In a nutshell, users want the right product built right: purchasers (those paying for the product’s development) seek completion on time, within spend limit, and compliance with contract clauses.

Because the product buyer is not always a user of the product, employees of organisations sometimes must use products or services that have been bought by people who don’t have to use those products or services. 

When a product is perceived by is users as not valid (even if verified), the outcome can be loss of morale: the product doesn’t get used:  is “worked around”: or a combination of all three. See Figure 1: Validation v Verification.  In some cases, the purchasing organisation loses customers because the customers are dissatisfied. 

The buyer of the product or service negotiates a contract with the supplier (internal or external).  The product is bought, or is built, or is changed, or is bought and changed. 

Testers test it to make sure that it follows the contract.  Users use it, then grumble that it is not fit for its intended purpose: “It doesn’t work for me”. The product is verified but invalid: a classic case of wrong product built right and is a result of not enough collaboration amongst the buyers, suppliers, and users. 

Validation is the process of making sure that the users have the right product or service.  Verification is the process of making sure that the product or service follows its contract.

What we look for is the right product built right.

Acceptance Criterion

An acceptance criterion is used to confirm that a product is fit for its intended purpose and or follows any contracted obligation. Usually, there is more than one acceptance criterion. The plural of criterion is criteria.  To show that a product is fit for purpose (valid) and meets its contract (verified), acceptance testers test it against a set of acceptance criteria. See Figure 2: Iterative Agreement below: some but not all types of acceptance criterion are shown.

Example Acceptance Criteria

• Business Rule 504 is obeyed
• For 95% of all instances of enquiry X, the response is provided within 5 seconds of the search argument being input
• The house conforms to District Plan 504
• The vehicle has a current Warrant of Fitness
• The refrigerator does keep an interior temperature of 2^oC+/- 0.5^oC regardless of exterior temperature
• Customer queuing time for service in the bank averages 4 minutes or less from the time the customer enters the bank to the time a teller starts to serve them
• The cost of sale per unit sold is $5.00 or less
• 98% of customers have scored our service as excellent

Part 2: Getting Good Acceptance Criteria 

Figure 2: Iterative Agreement

Get “good” acceptance criteria from a combination of people who will be affected by the new or changed product, process, or structure.  The elicitor or elicitation team is not likely to be able to talk to everyone so will need to talk with representatives. Some of those representatives will be future users, others will be purchasers, yet others will be developers, organisation architects, organisation change managers, testers, trainers… the list can go on and on.

If you want “good” acceptance criteria, have lots of conversations with the people that really matter – the ones that will use the product or service.

Ensure that every acceptance criterion, functional and non-functional, is specific and testable i.e.has a measurement and has been recorded.  The records can be informal, perhaps post-it notes on a wall, or formal, perhaps in a database or specification document of some kind.

If you want to know what a horse needs, then ask that horse.  Only that horse knows what that horse needs and different horses have different needs: so, it’s no good asking just one horse, you need to ask some horses to gain a consensus.

Asking the jockey, owner, punter, stable hand, or anyone else what the horse wants is a waste of time.  Horses know what horses want.  Workers know what workers want not their bosses.

In the end, of course, the horses will tell you if the received product is acceptable.

There are many kinds of horses. Do get the acceptance criteria straight from the horses’ mouths and not from some proxy.  At the very least, describe a typical horse of a type and then try to second guess what that “typical” horse would want.

Avoid the so-called “waterfall”

A logical approach is in sequence to get requirements, then design a solution, then build/buy that solution, then test it, and finally deploy it.  Some organisations wanting to reduce the risk of time and cost over-runs attempt to freeze the outcome from each step by not allowing any changes.  Such organisations try to ignore the fact of change.  The eventual product/service/process/structure meets the contract but is unfit for purpose.  Do not try to deny or stop change. Change will happen.

To misquote Sir Winston Churchill, “trying to stop change is like a person standing in a bucket and trying to lift themselves up by tugging on the bucket’s handle.”

Do use an Iterative, Incremental, Evolutionary, Collaborative, Change-welcoming Approach

Please see Figure 2: Iterative Agreement above.  As an integrated team work together to specify, design, develop, test, and deploy a little, early, and often. 

1. Minimise bureaucracy.
2. Avoid top-down command and control. 
3. Do not micro-manage
4. Work as a self-managing team 
5. Produce no more documentation than is needed to ensure the right product gets built
1. The customer wants to eat a hamburger, not a hamburger recipe
2. But the recipe does form the basis of the acceptance criteria
3. The only true measure of progress is delivery of something that works – as perceived by its users. 

This is an approach that has been understood and used by successful teams for many thousands of years. 

The idea is to get it “righterer soonerer” – appalling grammar but it illustrates the principle that the sooner you show your customer something the sooner they can say “that’s not quite what I meant.”  If you would like to call this approach “Agile” then please do, but just remember that it is not new.

Post by John Watson