There are multiple ways to execute your test automation effort, you can run them from and on Microsoft Test Manager, from and on Visual Studio, during the build and on the Build Server or during the build on Test Agents configured with Test Controllers or … many flavors. so, where should you execute your automated tests, and how should you configure your test infrastructure.
Where to run your automated tests, and how to configure the test infrastructure?
The different flavors
Flavor A: Execution from VS2010… | |
 | Purpose: Real developer test / test automation dry run.
A valid scenario is when Developers create a codedUI test from a test case which found a bug. To reproduce the bug, developers generate the CodedUI add the assertion and verify the result when they fixed the bug. Triggered by: right mouse click – run tests, in test project, or Test List Editor, run checked tests. Information: http://msdn.microsoft.com/en-us/library/dd286580.aspx Pro
Easy setup
Debug-able test automation
No additional configuration needed* Con
No collection of test results in TFS/ Test Manager
Run on developer environment * Data Diagnostic Settings in the Local.Testsettings file are configurable. |
| | |
Flavor B: Execution during build with Build controller | |
 | Purpose: Part of Build Verification Tests (its strange to run UI tests on a build and have a build service running in interactive mode) Triggered by: build Information:
Set the Build Service to run in interactive mode.
http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx
Configure the build to run the tests
http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType Pro
Easy setup
Test results in build result Con
No collection of test results in Test Manager (no reporting on test points)
Build Service needs to run in interactive mode (what happens after a restart)
Tests are executed on build environment |
| | |
Flavor C: Execution during build with Test controller | |
 | Purpose: Part of Build Verification Tests, distribute tests over multiple test agents. (Preferred configuration above flavor B) Triggered by: build Information:
Configure Test Controller (don’t register it with a project collection )
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers
Configure
Test Agents on clients (interactive mode)
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents
Configure *.Testsettings file in solution to use Test Controller and Test Agents
Configure the build to run the tests (see B) Pro
Tests run distributed over multiple environments
Test Results in Build result Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments
Test Settings from VS |
| | |
Flavor D: Execution from Microsoft Test Manager | |
 | Purpose: Part of Regression Tests (other type of test than BVT). Triggered by: MTM user, right mouse click on test case, run Information:
Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller and create test ‘agent’ physical environment.
http://msdn.microsoft.com/en-us/library/ee390842.aspx
http://msdn.microsoft.com/en-us/library/dd293551.aspx
Associate CodedUI test with WI Test Case from VS.
http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM
Full control by the tester Con
Test Controller needs to be configured with a Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure
Hard to see which test case is automated |
| | |
Flavor E: Execution from MTM during Build… | |
 | Purpose: Part of BVT. Preferred configuration above flavor C. Flavor D and E can be configured together. Triggered by: Build Information:
Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
Associate CodedUI test with WI Test Case from VS.
Create Build task to run TCM or MSTEST task for Test Plan
http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx
How to: Run Test Cases with Automation from the Command Line Using Tcm
http://msdn.microsoft.com/en-us/library/dd465192.aspx Pro
Test run distributed over test environments.
Tests can be configured to run on different configured environments
Test Result in MTM and TFS
Triggered by build
Test Settings from MTM Con
Hard to configure
maintenance of TCM commands in build |
Enough flavors to choose from… the pro and cons I wrote down can differ per situation. My preferred flavors are E and D configured together. The reason, test settings and test results are where the belong, in the testers hands. And, the TCM command line give much more flexibility what to run where during the build.
Lab Management for virtual environments have different configurations, pros and cons… will make a post for this later…
<UPDATE>
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution from Colin Dembovsky it’s a XAML build template which lets you easily configure the Build-Deploy-Test workflow for physical environments. one drawback from flavor E will be minimized when using this template, the maintenance of the TCM commands.