News

I was recently at Nashville attending Agile 2013 and had the great pleasure of finally meeting Janet Gregory. I have been tweeting with Lisa Crispin for years and it was fabulous to meet the Janet part of the “dynamic duo” that I had not met before. Why the dynamic duo? Lisa and Janet wrote a book called “Agile Testing” and I love it! It has been the cornerstone of my testing and Agile testing thinking framework for ages.

Janet was part of an open questions session and I took the opportunity to discuss with her my interpretation of the “Agile Testing Pyramid” that is referenced in their book. (p 277 Agile Testing, Lisa Crispin & Janet Gregory, Addison-Wesley, 2009).

In the session, Janet drew the pyramid on the board, and as she was doing so, referred to it as a triangle instead, and I asked if I could explain how I use this as a pyramid start point for explaining strategy as I do training and strategy sessions with people. Step 1 is basic explanation of the pyramid and what each layer means, and as Janet said in her talk, and which is clearly outlined in the book, what each layer is designed to test (which aids inappropriate test design for the team members).

For me, step 2 is adding the next (right) side to the pyramid which encompasses the tools that we can use to execute these tests. Each tool is chosen for the test type and the test executor. For example, at the unit test level, the tools depend on the technologies that the code is using and how well that integrates into the Continuous Integration environment. At the Acceptance Test Level (level 2) we are looking at System Testing – behind the screens and making sure the fundamental functionality of each of the architectural elements works and works together. At the GUI level, we are looking for User Acceptance Testing. This means that we need to have the right type of tool for the executors of the testing at that level to use.

This expansion allows people to understand that there is no one tool that will do everything and that when we are automating we need to match the tool to the tester as well as the testing. I like this model as people can see how the tool types and tool users change depending on where inside the application or architecture we are focusing on. This also allows the team to spread the load of work, helping make sure that test coverage is achieved but also achieved by the most effective people at the most effective time in the lifecycle.

Then I pushed the friendship and moved onto the left side of the pyramid, where I extend the now 2 sided pyramid to have 3 sides, and this third side is where I overlay test type.  I love Brian Marick’s Agile  Test Matrix (http://www.exampler.com/old-blog/2003/08/22/#agile-testing-project-2) which has been used by Lisa and Janet in the Agile Testing book (http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/) when thinking about this, or I like to use ISO25010 or ISO9126 as my checklist for system attributes because they are quick and easy to understand. James Bach has a fabulous list as well! (http://www.satisfice.com/tools/htsm.pdf). These allow us to be sure that we consider all the aspects of the solution that need to be tested.

This is where we can start to see who is going to focus on what type of testing, for example, functional versus non-functional and why these tests should be run at various levels. This then allows us to check and see if the tool selection and the tests have been designed for coverage of the system attribute we are trying to evaluate.

Once the test attribute is assigned to the test level, people in the team begin to understand how much testing may be involved and we start to talk about data sets, reuse of data, test reuse, test design for specific attributes and test design for reuse. We also allow technical team members to share their knowledge on how to test for the more technical attributes at the higher levels. Non-technical team members start to understand why test automation and test design is important for coverage and for building the testing resources for regression.

Once I mentioned the “R” word, some of the audience seemed to groan, but this is where I take the discussion when we are using the pyramid as part of the strategy session and expand it to include regression testing. I explain that if we think and plan well, we can reuse what we create as the initial test execution activities as our ongoing regression suite.

Visually it is pretty hard to represent that 4^th side of the pyramid, so I just like to leave the regression tests as lines running down the 3^rd side, indicating that the regression can be on any system attribute that has been considered as part of the testing. I use this when I am talking about automation in general, not just in the Agile framework. But it is very useful when considering testing is done by the whole team, consistent and reusable test and data sets, and explaining the need for different types and levels of testing to prove that each of the system components, elements and attributes hang together in a solution that “works” in a way that we expect.

Janet was very kind to me, and nodded a lot, but did leave me with wise words….” be careful about making it too complicated” which is something that can happen very quickly when talking about test automation. I totally understand that! Our final diagram (see below) would take some time to understand. The principle is there though – just in the finished state. I would always use the pyramid to begin the discussion, and then leverage the pyramid to extend the discussion about test strategy in the direction that we needed to go. I really like the pyramid analogy as it is a self-supporting structure that cannot exist without each of its sides. Each has a different focus (or direction) but each tie into the whole unit.