There are multiple ways to execute your test automation effort, you can run them from and on Microsoft Test Manager, from and on Visual Studio, during the build and on the Build Server or during the build on Test Agents configured with Test Controllers or … many flavors. so, where should you execute your automated tests, and how should you configure your test infrastructure.

Where to run your automated tests, and how to configure the test infrastructure?
The different flavors
Flavor A: Execution from VS2010… | |
 | Purpose: Real developer test / test automation dry run. A valid scenario is when Developers create a codedUI test from a test case which found a bug. To reproduce the bug, developers generate the CodedUI add the assertion and verify the result when they fixed the bug. Triggered by: right mouse click – run tests, in test project, or Test List Editor, run checked tests. Information: http://msdn.microsoft.com/en-us/library/dd286580.aspx Pro Easy setup Debug-able test automation No additional configuration needed* Con No collection of test results in TFS/ Test Manager Run on developer environment * Data Diagnostic Settings in the Local.Testsettings file are configurable. |
| |
Flavor B: Execution during build with Build controller | |
 | Purpose: Part of Build Verification Tests (its strange to run UI tests on a build and have a build service running in interactive mode) Triggered by: build Information: Set the Build Service to run in interactive mode. http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType Pro Easy setup Test results in build result Con No collection of test results in Test Manager (no reporting on test points) Build Service needs to run in interactive mode (what happens after a restart) Tests are executed on build environment |
| |
Flavor C: Execution during build with Test controller | |
 | Purpose: Part of Build Verification Tests, distribute tests over multiple test agents. (Preferred configuration above flavor B) Triggered by: build Information: Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B) Pro Tests run distributed over multiple environments Test Results in Build result Con No collection of test results in Test Manager Harder to configure Need for specific test client environments Test Settings from VS |
| |
Flavor D: Execution from Microsoft Test Manager | |
 | Purpose: Part of Regression Tests (other type of test than BVT). Triggered by: MTM user, right mouse click on test case, run Information: Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode) Configure Lab Center in MTM to use test controller and create test ‘agent’ physical environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html Pro Test run distributed over test environments Test Result in MTM Test Settings from MTM Full control by the tester Con Test Controller needs to be configured with a Project Collection (one controller per collection) Manually triggered by Tester (or pro) Hard to configure Hard to see which test case is automated |
| |
Flavor E: Execution from MTM during Build… | |
 | Purpose: Part of BVT. Preferred configuration above flavor C. Flavor D and E can be configured together. Triggered by: Build Information: Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx How to: Run Test Cases with Automation from the Command Line Using Tcm http://msdn.microsoft.com/en-us/library/dd465192.aspx Pro Test run distributed over test environments. Tests can be configured to run on different configured environments Test Result in MTM and TFS Triggered by build Test Settings from MTM Con Hard to configure maintenance of TCM commands in build |
Enough flavors to choose from… the pro and cons I wrote down can differ per situation. My preferred flavors are E and D configured together. The reason, test settings and test results are where the belong, in the testers hands. And, the TCM command line give much more flexibility what to run where during the build.
Lab Management for virtual environments have different configurations, pros and cons… will make a post for this later…
<UPDATE>
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution from Colin Dembovsky it’s a XAML build template which lets you easily configure the Build-Deploy-Test workflow for physical environments. one drawback from flavor E will be minimized when using this template, the maintenance of the TCM commands.