All validation (test) activities cost money. So, every activity to test a system should be executed as efficient as possible (see also the ‘testing in sprint’ series on this blog). Adding automation to the execution of these validation saves execution time, which saves money. But, specifying, creating and maintaining these automation cost money. So, the hard question is where is the balance, where the break-even point of test automation.
Not that easy to answer question but lets give it a try. All quadrants in the ‘Agile Testing quadrant’ have some kind of Visual Studio tool support (No automation with out tool support).
For the technology focused tests, the ROI of test automation is a kind of clear. Developers create unit tests conform a specific code coverage level, proven the implementation is still correct, which give the team some comfort and quality level. For the load tests its of no discussion that there is a ROI.
But, when we look at the ‘top - business facing’ tests, ROI of test automation is a bigger challenge. We have to think of how long is the validation member of our project (for example not all functional tests aren’t executed every sprint, only a sub set, only the most important, see this post ‘only meaningful tests’) and how many times is the validation executed (how many times over time but also on different environments). This gives us indications how much effort we must put in our test automation.
Basic there are three test automation levels:
- No Automation
- Record and Playback
- Test Scripts
Visual Studio adds two other levels to it.
(slide from the testing roadshow)
All these automation levels have an invest, and a maintainability level. The better you can maintain a test case the longer you can use it for you ever evolving software system. That is the connection between ‘how long’ and ‘how well maintainable’, another connection is the effort it takes to create the automation. the resulting benefit is, you can execute your script over and over again.
The ideal situation: a test script with very small investment to create, for a test which needs to be executed the whole life of the application which doesn’t change overtime. No investment, no maintainability issues, maximum amount of executions. Result: maximum ROI.
Too bad, we’re not living in the ideal world. So some tradeoffs need to make.
| ||No automation. |
No need for maintainable test scripts, no automation investment. I have customers who use MTM for Test Case Management only, and they are happy with it. maintaining thousands of test cases and their execution gathering information about the test coverage of the implemented functionality.
In most situations, this is an ideal starting point for adopting MTM and starting to look at test case automation. As a test organization getting used to the benefit of integrated tools the support of different ALM scenario's.
| ||Shared Steps with Action Recording | Record Playback parts. |
Collecting an action recording takes some effort. You have to think upfront what you want to do, and often you have to execute the test case several times to get a nice and clean action recording. So there is some investment to create an action recording which you can reuse over and over again.
You can’t maintain an action recording. When a application under tests changes or when the test cases changes you have to record every steps again. A fragile solution for automation.
Using Shared Steps (reusable test steps) with their own action recording solves this a bit.
Find the test steps which appear in every test case, make a shared step of these steps and add an action recording to it. Optimize this action recoding and reuse the shared step in every test case. This definitely improves the ROI. Now you can fast forward all the boring steps and focus on the real test.
The good thing, when a shared steps changes, you only have to record that one again.
Creating multiple shared steps with action recoding and compose a test case is also a good scenario.
After the zero investment, this is a good next step. You get used to the behavior of action recordings and have the benefit of reusing them throughout the project.
Action recordings of shared steps keep their value the whole project, there is some effort to create and maintain them but you will execute them for every test case, a good ROI.
| ||Test Cases with Action Recordings | Full Test Case Record Playback. |
The same activity as for the Shared Steps action recordings. But, you will use the action recording less and it is harder to maintain (more test steps). So defiantly the ROI is much lower as in the Shared Steps situation.
The scenario where you create the action recording and execute often, for example on many different environment will bring some benefits. MTM action recordings can be recorded on one environment and playback on another environment.
Another reason you might want to go with this scenario, is that you want to reuse the action recording for test script generation. See next step.
| ||Generate test script from action recording. |
A really nice scenario for quickly creating test automation scripts.
See this video for a quick how to:
The maintainability of the generated code is hard. There are some tools in place to react on UI changes, which make it easier. Also challenging are some common development practices like branching, merging and versioning of the test script. See this presentation
Conclusion, the creation of test scripts generated from action recordings is really fast but hard to maintain. This together with the recording of the action recording (number 2) has its influences on the investment.
| ||Write your own Test Script ( by using the Coded UI Framework ). |
Write the test automation script your self, following all the good coding principles of maintainability and reusability. like: Separation of concerns, KISS principle, Don't repeat yourself, etcetera.
Code First API Library is a good starting point.
This automation scenario is completely opposite of the generated test script (3). This one is hard to create, it will take some effort, but is (if implemented well) very maintainable. and you can follow all the coding practices and versioning strategies.
So, MTM with Coded UI support different test automation scenario’s. From fast creation with hard maintainability (2 and 3) to hard to create with good maintainability (4). It is a good to think up front about test automation before random start using the tools.
My rules of thumb are:
- Use 2 and 3 in a sprint and maybe in a release timeframe, but not longer. Maintainability will ruin the investment.
- Use 4 for system lifetime tests.
They run as long as the system code runs and should be treated and have the same quality as that code. Don’t use it for tests you only run in a sprint, the effort will be to big.
- Use 0 and 1 always when ever you can. it supports several ALM scenario’s and the shared steps action recording really is a good test record-playback support with some good ROI.
After a small investigation on a test tool conference I made the conclusion that no tool which supports test script generation (or something similar) all have some kind of workaround to make it maintainable. Only a good test automation framework (for example the Coded UI Framework or Selenium )will make test automation (I’m still talking about business facing tests) code maintainable. Test script generation is nice but you don’t want to maintain it.