Test Automation Helpful Tips

Introduction

Test Automation is a controversial topic with many differing thoughts, ideas and opinions. Does automation speed up the development process? Should developers create the automation in all testing levels? Should QA’s upskill and write UI level automation?

Automation, like most things in testing, is context dependent. While one organisation is able to utilise automation to ship software rapidly with a level of confidence, other organisations struggle to get the benefit for the amount of effort and resources that are put in.

I work in an organisation where automation is one of the vital components we use to successfully ship software rapidly. My belief is that automation is used as a tool to aid testing efforts. Automation does not take away the need for other testing activities.

In this blog I will talk about the:

  • Test Automation Pyramid,
  • The Importance of Automating as a Team,
  • The Importance of a Trustworthy Automation Suite,
  • and Testing vs Checking.

 

The Test Automation Pyramid

There are many discussions and opinions out there about the test automation pyramid, is it a good representation? is it something that teams should follow as part of good agile practices.

Richard Bradshaw and John Stevenson raise some good points and things to consider when you are thinking about using the test automation pyramid model.

For the purpose of this post, I’m referring to the test automation pyramid as a guide on what you should aim for when creating automation. I’m not looking at using the test automation pyramid as a way to work out your full test strategy, what belongs where.  One of the modified test pyramids has exploratory testing at the top of the pyramid, as Richard and Steven point out,  you use exploratory testing to work out what you need to automate in the first place, so does that really belong at the top of the pyramid?

In my own opinion the test automation pyramid isn’t intended to be a hard and fast rule of what tests belong in what level of the testing pyramid. The test automation pyramid is setup as a guideline, to help show where certain automation could be created.

The test pyramid leans towards having more tests in the Unit level due to the granularity of tests, unit testing looks to test the smallest testable parts within your application, the further up the test pyramid you go, the less granular and more complex the tests become. 

The test automation pyramid is a concept developed by Mike Cohn. Described in his book “Succeeding with Agile”.  

Test Automation Pyramid

The above image shows a pyramid with ‘Unit’ testing at the bottom, then ‘Service’ followed by ‘UI’ at the top of the pyramid. As we go up the pyramid the sections get smaller. Mike Cohn’s main point with this testing pyramid illustration suggests there should be more tests in the lower level of the pyramid than the upper levels i.e more tests in Unit level than in the UI level. You could relate this to the “Food Pyramid” where the foods we should eat most of appear at the bottom, compared to the food we should eat least of appearing in the smallest section at the top.

When organisations start building up an automation suite they can fall into the trap of creating too many tests in the upper levels of the pyramid. When you have more tests in the UI level than the Unit level, your testing pyramid becomes the Ice Cream Cone Model.

 

Ice Cream Cone Test Pyramid

Why the Ice Cream Cone model is Bad

The further up the testing pyramid we automate, the following occurs; the tests are more brittle, are harder to create, take a lot longer to run. Any changes to the application can mean big and time consuming re-work to the UI Automation suite.

UI Automation which interacts with a browser has challenges that need to be worked through in order to get a reliable robust UI automation suite.

Some of the problems include:

  • Slow Environments: This can cause tests to fail unexpectedly due to an element not displaying within a set time frame.
  • Element locators changed: Moving an element on the page, or change in ID will cause tests to fail due to not being able to find the element located on the page.
  • Hard to locate Elements: The best way to locate and use an element is by using the ‘id’, unfortunately many times you are forced to use complex XPath or CSS locators to interact with an element.
  • Error handling: When a test fails, or the web page fails to load, you need your automation suite to recover in order to run the next test in the list.
  • Maintainability: There is always going to be a maintenance cost. How this is managed needs to be established in the beginning.

 

Working as a Team when Automating

Whether you are a QA involved in the coding aspects of automation, or not, it  can really help to be involved in all levels of the testing pyramid.

That isn’t to say you need to be sitting there watching the developer code unit tests, but it is essential to know what is being created and why. The ‘why’ is really important to ensure you are developing tests at the right level of the testing pyramid.

If you code UI level automated tests, then you need to be aware of what tests are being developed at the other levels of the testing pyramid, if this information is lacking then there will be duplicated effort.

If you are a QA that doesn’t code and the developer has created the automation throughout the testing pyramid, then it is important for you to be involved in working out what tests are automated. QA’s have a focus on testing, we have a set of skills, tools and models that we use to work out what tests are useful and how we can exercise the application to find bugs.

I like to create Acceptance Criteria and Acceptance Tests for each task that is created. This gives us the opportunity to speak about what tests we are likely to automate and at what level the tests should live in the testing pyramid before the feature has been developed. It’s important to note that as always, things can change and you often think of more tests as you go, so continue to have conversions with your team throughout the process.

 

The importance of a trust worthy Automation Suite

It is important that the automation you create is trustworthy. No matter what level in the test pyramid the automation is created, if it isn’t trusted it won’t give you the benefit that automation can provide. A couple of ‘flaky’ tests and incorrect failures in the test suite, due to environmental or test related issues, can have disastrous consequences. You will find that it’s easy for people to start blaming these failing tests or tests packs as flaky tests, tests that “sometimes” fail, while missing critical bugs.  

The automation suite needs the attention and effort that the application gets, to ensure the tests are robust.That’s why it’s important to also ensure that the right tests are in the right level of the testing pyramid.

 

Testing vs Checking

An important concept to understand is what automation is doing, what benefits it gives you and also what it is not doing for you.

Testing Is Exploration and Learning

Testing is something that we do with the motivation of finding new information. Testing is a process of exploration, discovery, investigation, and learning. (Michael Bolton, Testing vs Checking)

Checking Is Confirmation

Checking is something that we do with the motivation of confirming existing beliefs. Checking is a process of confirmation, verification, and validation. (Michael Bolton, Testing vs Checking)

You can’t fully automate testing. Testing is an analytical, cognitive, human activity. Automation serves as a checker, it checks that expected behaviour, that has been programmed into the automated test, works. You have to know in advance what it is you are going to test for. It doesn’t learn and discover along the way.

Applications change rapidly, what is expected and known behaviour one day, can be different the next.

Testing is done by humans, we use our brain to think, to follow our intuition, to question and discover things.  A person uses automation to help with their testing efforts.

 

Articles

Testing vs Checking

James Bach Testing vs Checking Refined

Bolton and Bach on Testing vs Checking

 

Test Automation Pyramid

Introducing Software Testing Cupcake Anti Pattern – Fabio Pereira

Test Pyramid – Martin Fowler

The Forgotten Layer of the Test Automation Pyramid – Mike Cohn

 

7 Comments

  • Jim Hazen

    January 12, 2016

    You miss the key reason/point of the Automation Pyramid in its structure, that is the natural distribution of said tests. Meaning that the level/focus/granularity (or atomicity) of Unit Tests means there will be more of them because they are very specific and more of a one-to-one relationship with the code itself. In order to cover code statements (lines) and segments (chuncks of code such as logic blocks or function/method call/called pairs) you will need to build more individual tests, thus more of them. This serves two purposes; first, code coverage by using assertion type tests to validate code functionality and second, to improve code stability by testing basic and specific functionality to ferret out basic errors before they propogate down stream to other levels/testing efforts.

    As you go higher in the pyramid you will naturally need less automation because your tests become more complex and less granular. They test things more in conglomeration of functionality.

    That is something Mike Cohn has said in later posts about the pyramid. It is to be a guideline (not a hard rule) to push left the testing effort and to catch errors earlier, thus reducing rework costs. It doesn’t mean upper level automated tests are “bad”, it means that if you do employ a testing model that uses all 3 layers that you will “natrually” get the distribution.

    Reply
    • Daniel Donbavand

      January 15, 2016

      Thanks Jim for the useful comments. I have made a couple of changes to my blog.

      Reply
  • Pingback: Testing Bits – 1/10/16 – 1/16/16 | Testing Curator Blog

  • Pingback: Five Blogs – 18 January 2016 | 5blogs

  • Pingback: Java Testing Weekly 3 / 2016

  • Pingback: Java Web Weekly 108 | Baeldung

  • Jack Martin

    March 11, 2016

    Very informative post and it was quite helpful to me. I also wrote something on similar lines on test automation – http://bit.ly/1Xh1xNN

    Reply

Leave a Reply