Test Automation Helpful Tips
Test Automation is a controversial topic with many differing thoughts, ideas and opinions. Does automation speed up the development process? Should developers create the automation in all testing levels? Should QA’s upskill and write UI level automation?
Automation, like most things in testing, is context dependent. While one organisation is able to utilise automation to ship software rapidly with a level of confidence, other organisations struggle to get the benefit for the amount of effort and resources that are put in.
I work in an organisation where automation is one of the vital components we use to successfully ship software rapidly. My belief is that automation is used as a tool to aid testing efforts. Automation does not take away the need for other testing activities.
In this blog I will talk about the:
- Test Automation Pyramid,
- The Importance of Automating as a Team,
- The Importance of a Trustworthy Automation Suite,
- and Testing vs Checking.
The Test Automation Pyramid
There are many discussions and opinions out there about the test automation pyramid, is it a good representation? is it something that teams should follow as part of good agile practices.
Richard Bradshaw and John Stevenson raise some good points and things to consider when you are thinking about using the test automation pyramid model.
For the purpose of this post, I’m referring to the test automation pyramid as a guide on what you should aim for when creating automation. I’m not looking at using the test automation pyramid as a way to work out your full test strategy, what belongs where. One of the modified test pyramids has exploratory testing at the top of the pyramid, as Richard and Steven point out, you use exploratory testing to work out what you need to automate in the first place, so does that really belong at the top of the pyramid?
In my own opinion the test automation pyramid isn’t intended to be a hard and fast rule of what tests belong in what level of the testing pyramid. The test automation pyramid is setup as a guideline, to help show where certain automation could be created.
The test pyramid leans towards having more tests in the Unit level due to the granularity of tests, unit testing looks to test the smallest testable parts within your application, the further up the test pyramid you go, the less granular and more complex the tests become.
The above image shows a pyramid with ‘Unit’ testing at the bottom, then ‘Service’ followed by ‘UI’ at the top of the pyramid. As we go up the pyramid the sections get smaller. Mike Cohn’s main point with this testing pyramid illustration suggests there should be more tests in the lower level of the pyramid than the upper levels i.e more tests in Unit level than in the UI level. You could relate this to the “Food Pyramid” where the foods we should eat most of appear at the bottom, compared to the food we should eat least of appearing in the smallest section at the top.
When organisations start building up an automation suite they can fall into the trap of creating too many tests in the upper levels of the pyramid. When you have more tests in the UI level than the Unit level, your testing pyramid becomes the Ice Cream Cone Model.
Why the Ice Cream Cone model is Bad
The further up the testing pyramid we automate, the following occurs; the tests are more brittle, are harder to create, take a lot longer to run. Any changes to the application can mean big and time consuming re-work to the UI Automation suite.
UI Automation which interacts with a browser has challenges that need to be worked through in order to get a reliable robust UI automation suite.
Some of the problems include:
- Slow Environments: This can cause tests to fail unexpectedly due to an element not displaying within a set time frame.
- Element locators changed: Moving an element on the page, or change in ID will cause tests to fail due to not being able to find the element located on the page.
- Hard to locate Elements: The best way to locate and use an element is by using the ‘id’, unfortunately many times you are forced to use complex XPath or CSS locators to interact with an element.
- Error handling: When a test fails, or the web page fails to load, you need your automation suite to recover in order to run the next test in the list.
- Maintainability: There is always going to be a maintenance cost. How this is managed needs to be established in the beginning.
Working as a Team when Automating
Whether you are a QA involved in the coding aspects of automation, or not, it can really help to be involved in all levels of the testing pyramid.
That isn’t to say you need to be sitting there watching the developer code unit tests, but it is essential to know what is being created and why. The ‘why’ is really important to ensure you are developing tests at the right level of the testing pyramid.
If you code UI level automated tests, then you need to be aware of what tests are being developed at the other levels of the testing pyramid, if this information is lacking then there will be duplicated effort.
If you are a QA that doesn’t code and the developer has created the automation throughout the testing pyramid, then it is important for you to be involved in working out what tests are automated. QA’s have a focus on testing, we have a set of skills, tools and models that we use to work out what tests are useful and how we can exercise the application to find bugs.
I like to create Acceptance Criteria and Acceptance Tests for each task that is created. This gives us the opportunity to speak about what tests we are likely to automate and at what level the tests should live in the testing pyramid before the feature has been developed. It’s important to note that as always, things can change and you often think of more tests as you go, so continue to have conversions with your team throughout the process.
The importance of a trust worthy Automation Suite
It is important that the automation you create is trustworthy. No matter what level in the test pyramid the automation is created, if it isn’t trusted it won’t give you the benefit that automation can provide. A couple of ‘flaky’ tests and incorrect failures in the test suite, due to environmental or test related issues, can have disastrous consequences. You will find that it’s easy for people to start blaming these failing tests or tests packs as flaky tests, tests that “sometimes” fail, while missing critical bugs.
The automation suite needs the attention and effort that the application gets, to ensure the tests are robust.That’s why it’s important to also ensure that the right tests are in the right level of the testing pyramid.
Testing vs Checking
An important concept to understand is what automation is doing, what benefits it gives you and also what it is not doing for you.
Testing Is Exploration and Learning
Testing is something that we do with the motivation of finding new information. Testing is a process of exploration, discovery, investigation, and learning. (Michael Bolton, Testing vs Checking)
Checking Is Confirmation
Checking is something that we do with the motivation of confirming existing beliefs. Checking is a process of confirmation, verification, and validation. (Michael Bolton, Testing vs Checking)
You can’t fully automate testing. Testing is an analytical, cognitive, human activity. Automation serves as a checker, it checks that expected behaviour, that has been programmed into the automated test, works. You have to know in advance what it is you are going to test for. It doesn’t learn and discover along the way.
Applications change rapidly, what is expected and known behaviour one day, can be different the next.
Testing is done by humans, we use our brain to think, to follow our intuition, to question and discover things. A person uses automation to help with their testing efforts.
Testing vs Checking
Test Automation Pyramid