Agile Test practices with Microsoft Visual Studio 2010 [TMap and Scrum]

Often the testing effort in an agile environment is called ‘Agile Testing’. This sounds a bit strange, it almost looks like testing is a separate thing, something special, something you can do or don’t in agile projects. But, testing is nothing more or less a part of the process. Tasks covering testing activities are executed during a sprint, are planned during a sprint planning meeting and have to follow the basic rules of Scrum.

The Visual Studio 2010 Scrum Process Template support agile work, but when teams think about testing as a separate effort, success is far away even when adopting the Scrum Process Template. So, testing practices like those in the TMap Test Approach need to be embedded. This is even more important when you want to use the full capabilities of the Visual Studio 2010 Application Lifecycle Management Tool Suite, where test support is a first class citizen.

This article explains how TMap testing processes and practices integrate in the Scrum Process Template and how they will be executed within Visual Studio 2010.


Three topics come together in this article; Scrum, TMap and Visual Studio 2010. It is recommended having a basic knowledge of all these topics. In this paragraph you can read a short introduction of these topics with an external link with recommended reading.


Taken from; Scrum is based on industry-accepted best practices, used and proven for decades. It is then set in an empirical process theory. As Jim Coplien once remarked to Jeff, “Everyone will like Scrum; it is what we already do when our back is against the wall.”


Figure 1 Scrum Process

Scrum, which is grounded in empirical process control theory, employs an iterative, incremental approach to optimize predictability and control risk. Three pillars uphold every implementation of empirical process control.

  • The first leg is transparency
  • The second leg is inspection
  • The third leg is adaptation

Recommended reading: Scrum guide.

Visual Studio 2010 Scrum Process Template

With the release of the Microsoft Visual Studio 2010 Scrum 1.0 Process Template the basic rules of Scrum are supported within Visual Studio 2010 Application Lifecycle Management (VS2010 ALM) Tool Suit. As can be read on the download site:

“Scrum is an iterative, incremental framework for project management and agile software development. This template includes work items and reports to help your Scrum team be successful.“

VS2010 ALM supports different kind of tasks in the software delivery process. It supports engineering tasks, architectural tasks and project management task, but also test tasks. This wide support, covering multiple ALM roles, makes it easier for teams to collaborate, work together with tasks and on artifacts.

Work items, reports, and team queries in process templates help teams to track information about Product Backlog Items and corresponding tasks, supporting the agile process.


Figure 2: Sprint Burndown (Scrum)

External link: Visual Studio Scrum 1.0

TMap, testing approach

Testing is one of the most important actions an organization can take to control risks. Testing provides insight into the quality of the software and the risks associated with insufficient quality. Based on this insight, organizations can make informed decisions about whether to take software into operation. With proper testing, organizations can make better decisions and effectively manage risk.


Figure 3: TMap logo

The 4 essentials of TMap.

  1. TMap is based on a business-driven test management (BDTM) approach.
  2. TMap describes a structured test process.
  3. TMap contains a complete tool box.
  4. TMap is an adaptive test method.

External link:

Test Practices

Just as there are engineering practices in agile environments, like test drive development [TDD], continues integration [CI] and architectural practices like emergent architectures and not doing Big Design Up Front [BDUF], there are also testing practices in agile environments. All these different practices are helping the Scrum Team to deliver a product following the Product Owners priority, acceptance criteria and conform the definition of done in the smartest way.

Examples of Agile test practices is for example the well-known; test early and often, a common practice but not always that easy to execute. Other practices are: automate as soon as possible, no risk no test and more (which can be found in the test approach TMap).
But, how do you execute these test practices in the most efficient way, specific with Visual Studio 2010 as a tools set, together with a Scrum and Risk Based mindset.

Additional reading: 10 Suggestions for the Architect of an Agile Team

Test Early and Often

The best know Agile testing practice is ‘Test Early and Often’. Get started testing as fast as possible. Most often people think about these practices in the context of engineering practices, thinking about TDD. But, Test Early and Often also covers other testing efforts. Efforts like defining the risk, setting up the environments, create test plans, scripts and more.

Test early and often, covers the whole testing effort. It wants to say; start working on testing the moment you start to think about a system. Within a Scrum Team this means, have testing knowledge available at the Product Planning Meeting, at the Sprint Planning meeting. Essentially have testing knowledge available throughout the complete project.

Test during the Release Planning meeting.

Testing knowledge during the Release Planning meeting cover mostly tasks which support the Product Owner. During the Release Planning the Product Owner defines, prioritizes, and maintains Backlog Items, and grooms it during the project (actually this knowledge support is necessary throughout the project). The main knowledge support is about the risk, the major risks for the product need to be defined..

“The release plan establishes the goal of the release, the highest priority Product Backlog, the major risks, and the overall features and functionality that the release will contain.”
Scrum Guide


Within the TMap test approach product risk analysis is an important technique. Determine risk analyses are part of the proposed activities in the Master Test Plan of TMap: ‘Analysing the product risks’. It not only supports the Product Owner to make the right decisions it also the Team benefits in a later stage, this information is invaluable while defining the right test case design techniques for the Product Backlog Item.

“The focus in product risk analysis is on the product risks, i.e. what is the risk to the organization if the product does not have the expected quality? ”

Having full product risk analysis for every Product Backlog Item during the Release Planning meeting is slightly overdone, but the major risks should be found. Determine product risks at this stage will also provide input for the Definition of Done list.


Figure 4: TMap Product Risk Analyses

Within the Visual Studio Scrum 1.0 Process Template Product Backlog Items are written down in the Work Item Type ‘Product Backlog Item’. This Work Item Type hasn’t got a specific field for risk classifications. Adding a risk field is preferable (TFS Powertools makes this an easy task) so you can query on this property, or you can make the risk analyses a more product generic property.


Figure 5: VS2010 Scrum 1.0 Product Backlog Item

Not only risk classification, defining the major risks, is a task with which test knowledge can help during Release Planning meeting. Also for defining and formulating ‘Acceptance Criteria’ information is testing knowledge a must have.

A practice which gets more and more attention in Agile projects is Acceptance Test Driven Development (ATDD). In this practice Acceptance tests are formulated, written and automated prior to the code.

“Advanced practices of test-driven development can lead to Acceptance Test-driven development (ATDD) where the criteria specified by the customer are automated into acceptance tests, which then drive the traditional unit test-driven development (UTDD) process.[5] This process ensures the customer has an automated mechanism to decide whether the software meets their requirements. With ATDD, the development team now has a specific target to satisfy, the acceptance tests, which keeps them continuously focused on what the customer really wants from that user story.” (from Wikipedia )

Parts of this ATDD practice can be executed during the Release Planning meeting, define the acceptance criteria. Other parts need to be executed after the Product Backlog Item is selected in a Sprint, and a create Test Cases tasks is defined by the Team. Within the Product Backlog Item there is a tab where you can link test case or create new linked test cases to a Product Backlog Item, these new test cases should be edited in Microsoft Test Manager.


Figure 6: Microsoft Test Manager new Test Case

Test during the Sprint Planning meeting

Sprints contain and consist of the Sprint Planning meeting, the development work, the Sprint Review, and the Sprint Retrospective. During the first part of the Sprint Planning meeting Product Backlog Items which will be realized are selected, during the second part these are broken-down in tasks and estimated (if no already done). So, during the second part decisions are made by the Team ‘How’ to realize the Backlog Items, tasks are created and committed to.


Within the TMap lifecycle model this is the first step, the Planning Phase. This step covers the responsibilities to describe tasks ‘How’ to test the functionality that will be realized. This ‘How’ contains decisions based on the risk (this information can be found in the Product Backlog) and covers test design techniques how test case should be created, and every other task necessary to be conform the Definition of Done.


Figure 7 TMap Test Lifecycle

So, the Team describes ‘How’ the system will be made and ‘How’ it will be tested during the Sprint Planning meeting.

During this meeting Work Items of the type Task are created within Visual Studio. Remaining work is estimated so the Sprint Burndown can be tracked. These tasks can cover different types of activities, when using the ‘activity’ field for this; queries can be made to track this information.


Figure 8 VS2010 Scrum 1.0 Task Work Item Type

Visual Studio has the knowledge of linked Work Items, which makes it possible to link tasks to Product Backlog Items. This linking creates the capability to track the progress of work that has occurred to complete the Backlog Item.


Figure 9 Link Task to Product Backlog Item

Information gathered in these Work Items can easily transform in Burndown and Velocity charts. See below and .


Figure 10 Sprint Burndown


Figure 11 Velocity


Figure 12 Release Burndown

Test Basis Finding

Another ‘Test Early and Often’ practice is described in the TMap lifecycles Preparation Phase. During this phase the team member who committed to execute the ‘create test cases’ task, starts preparing his testing effort. One proposed action for this preparation by TMap is; Asses the Test Basis, check if the information necessary for the creation of test cases (i.e. documentation, models, and Backlog Item description) is testable. Testability here means completeness, consistency, accessibility and translatability into test cases.

This testing practice helps to tackle defects as early as possible (‘Test Early and Often’), because the team members who realize the functionality also are using this same information for their tasks. Incompleteness or inconsistencies can be discussed during the Daily Scrum, or in the Team room.


If you should file a test basis finding within a VS2010 Bug Work Item can be part of a discussion. Filing surely helps the learning cycle and this information can be used during the Sprint Retrospective. Adding a specific ’bug type’ property would be helpful for queries and reports.


Figure 13 Bug Work Item Type

The Specification, Execution and Completion phases of the TMap Lifecycle model are all executed (as they are defined as tasks) during the development of the sprint. So, testing early and often not only covers unit testing but every part of the process.

Only test when there is a business risk

An important mindset of Scrum is ‘only execute tasks who make an addition to the product are worthy to execute’, this not only counts for engineering tasks or architecture tasks, but definitely also for testing tasks. One of the essentials of TMap is that testing should be ‘business driven’, a Business Driven Test Management (BDTM) approach. With the simple statement ‘No Risk, No Test’, when there isn’t any business risk you don’t have to test the system.

Business Driven Test Management (BDTM)

The total test effort is related to the risks of the system to be tested for the organization.

Choices must be made in what is tested and how thoroughly. Such choices depend on the risks that an organization thinks it will incur, the available quantities of time and money, and the result the organization wishes to achieve. The fact that the choices are based on risks, result, time and cost is called business-driven and constitutes the basis for the BDTM approach.

Steps in the BDTM process are (see figure):

  1. Formulating the assignment and gathering test goals.
  2. Determining the risk class for each combination of characteristic and object part.
  3. Determining whether a combination of characteristic and object part must be tested thoroughly or lightly.
  4. An overall estimate is then made for the test and a planning set up.
  5. Allocating test techniques to the combinations of characteristic and object part.
  6. Throughout the test process, the test manager provides the client and other stakeholders with adequate insight into and control options over testproces and testobject.


Figure 14 BDTM Steps

Not every step fits within a Scrum project, but the mind set of BDTM fits perfect. When tasks are defined for the testing effort it is efficient to take a good look at BDTM.

Within Visual Studio 2010 Scrum process template you can look at the field ‘business value’ and ‘risk’ (when this one is added) to decide if this Product Backlog Item must be tested thoroughly or lightly.


Figure 15: WIT Product Backlog Item with added Risk field

When adding the risk field queries can be made to capture all Product Backlog Items with a high risk classification, with high priority test cases. This gives a good overview of the quality of the testing effort.

BDTM test practices and Visual Studio 2010 capabilities to customize Work Item Types and create custom queries can give projects the right know-how to execute the testing effort with a Scrum mindset.

Automate as soon as possible

Automate test as soon as possible gives the team the benefit to focus on quality, they know the system under development still runs as accepted and get notifications when not. It also gives the product owner a benefit because he can easily see what functionality is realized and runs as he accepted.

Visual Studio 2010 has a perfect mechanism to automate test case, named CodedUI tests. For the procedure to create automated tests see on MSDN : Testing the User Interface with Automated UI Tests and this YouTube video: Create CodedUI.

Just automate every test case would be a bit overdone, some are very hard to automate or are too costly to automate, and useless. Every task within a Scrum project should have an added value to the product, so automating just everything is not done. What test to automate is a challenging task, for sure you want automated tests for the main scenario’s which you want to run every sprint. Actually the tests selected for regression testing need to be automated.


Which tests to select for this regression set is described in the TMap test lifecycle. During the completion phase test cases are selected for regression.

Also the query capabilities of Visual Studio 2010 can be used to select candidate test cases for automation. For example a query like described in the previous paragraph “high risk PBI with high prio Test Cases” is an interesting selection for automation.

High Quality Test Cases

Not a real agile test practice, more a common sense test practice. But, in agile environments having quality test scripts created and available for every team member, are a must.

Many test scripts are candidate for automation, see previous topic. So, just having a test script with one step mentioning ‘test the screen’ doesn’t help well for automation. Well described tests which are broken down in clear steps are much better. Not only for automation also when the guy who writes the test case isn’t the same team member who runs the test case. Well written test cases also explain what risk they cover, see paragraph Business Driven Test Management.

..building test cases really comes down to the simplest of concepts. Understanding your feature well enough and with enough detail such that when you look over your test cases, you have a high degree of confidence that you mitigated customer risk and provided your staff with a well defined set of criteria by which they can assume they have done their job. Good engineering is a team effort, the role of you or the person developing test cases is engagement and building confidence in how you test. This will go far in helping bring a high quality product to market.
Developing high quality test cases

Designing test cases is often targeted as a big challenge in Agile environments because the lack of documentation and changing specifications. The close collaboration with the Product Owner and the binding with engineering tasks will cover the necessary knowledge. To get the necessary flexibility in the test cases a good test case management system like Microsoft Test Manager is a must. Beside test case management Microsoft Test Manager also has the feature shared steps.

Shared steps

Visual Studio 2010 has the notion of Shared Steps. These are test steps which can be re-used for different test cases (See: How to: Share Common Test Case Steps Using Shared Steps). Making use of these shared test steps not only cuts down the amount of work it takes to create test cases, because many test cases contain some of the same steps, it also helps in cutting down re-writing work.


Automating test cases with shared steps will result in a clean separation of common test methods. Shared steps are generated as separated methods which makes the maintaining of these automations a lot more painless task.

So, within the context of Visual Studio it is a true benefit when focusing on the quality of the test case.

Undone End to End Testing

End to end testing, is not an agile specific testing practice, it’s a common sense testing practice. Create test cases to detected defects anywhere in the chain. This type of testing has been growing the couple of years enormous due to the rise of Service Oriented Architectures and more recent Cloud Computing.  as the TMap end-To-End Testing books describesketentesten%20EN_tcm9-57123

“End-to-end testing is a form of testing in which the (business) process is tracked through an interconnected series of systems, which may be based in different organizations, aiming to answer the following questions:
- Does the IT process support the business process adequately? - Are the process and systems integrated with each other in the right way?” End-To-End Testing

End-to-End testing is seen as a challenges in Agile environments, with the most important challenge that systems under development aren’t ready for End-to-End testing. Not every service is finished to test the complete end-to-end business process. Stubs are a solution, but can these be made in the current sprint and are stubs good enough to say something about the quality of the system. And, when the system are ready and a change in a service is needed, how and can we re-run all the end-to-end tests, is there time enough.

All these challenges with end-to-end testing in agile projects can be summarized in two topics:

  • Not Ready Systems in a Sprint
  • Not Enough Time in a Sprint

tipScrum solves this problem, this problem of not able to be compliant on the definition of done at sprint finisch, by defining ‘Undone work’. the scrum guide mentions this in a TIP and in the final thoughts of the scrum guide

Some organizations are incapable of building a complete increment within one Sprint. They may not yet have the automated testing infrastructure to complete all of the testing. In this case, two categories are created for each increment: the “done” work and the “undone” work. The “undone” work is the portion of each increment that will have to be completed at a later time. The Product Owner knows exactly what he or she is inspecting at the end of the Sprint because the increment meets the definition of “done” and the Product Owner understands the definition. “Undone” work is added to a Product Backlog item named “undone work” so it accumulates and correctly reflects on the Release Burndown graph. This technique creates transparency in progress toward a release. The inspect and adapt in the Sprint Review is as accurate as this transparency.


Within the Visual Studio 2010 Scrum process template this means that you have to link or create tasks to a specific ‘Undone Work’ product backlog item. this ‘Undone Work’ product backlog item accumulates the undone work regarding to End-to-End Testing (and others) over time and can be monitored just as other backlog items.

Understanding the Professional Scrum Developer Training and Certification Program [pptx]


Final thoughts

It is a must to incorporate test practices in an agile environment, beside that it makes the life of the Scrum Team easier. It also raises the quality bar of the system. In this article the some common testing practices for Scrum Teams are explained. For sure not all challenges teams have will be gone when following these practices, there are still a lot off glitches.

The Visual Studio 2010 Scrum Process Template support agile work, but when teams think about testing as a separate effort, success is far away even when adopting the Scrum Process Template. So, testing practices like those in the TMap Test Approach need to be embedded. This is even more important when you want to use the full capabilities of the Visual Studio 2010 Application Lifecycle Management Tool Suite, where test support is a first class citizen.

Comments (1) -

Nice article. I think it will be much better if we can add IntelleTrace feature into backlog and test cases just in Microsoft Visual Studio TFS.

Add comment

Şarkı Sozleri