Tag Archives: Visual Studio

‘Insert Call to webtest’ and the ‘Inherit Web Test Settings” property

One of the nice things in Visual Studio’s load test framework is the ability for a webtest to include calls to other webtests. This allows you to compose complex scenarios from a bunch of simple .webtests. In fact my load test uses statistical modelling to dynamically decide which tests its going to call.

If your webtest includes User Name / Password properties then you need to set the ‘Inherit Web Test Settngs‘ to True. Otherwise the called webtest won’t use the same credentials that the calling test is using. Especially in situations where you are running with Controllers and Agents you will probably need to set this setting.

Lets assume we have Webtest1 and it includes several calls to Webtest2. This has a few consequences you need to be aware of and deal with:

  1. Any WebTestPlugins in Webtest1 will execute when Webtest1 starts and each time Webtest2 starts. You probably don’t want that. Many people use the ‘Generate GUID’ plugin to store a unique ID in the context for the test session. However in this case the context parameter’s value will change every time one of the webtests is started. In my opinion its best to avoid using any WebTestPlugins on webtests that call to other webtests.
  2. Any validation rules in Webtest1 will also be applied to all requests in Webtest2. This can be problematic if webtest2 needs less or different validation.
  3. If webtest2 includes a call to webtest3, then webtest3 will also show the above issues.

Loadtest summary shows many URLs for a single page

When you are running a load test against an MVC page you will see an enormous amount of different URLs reported. That’s because dynamic parameters such as IDs are frequently included in the URL of the page that is being accessed.

For example, here is an URL that will update an invoice whose ID is 129101 https://my-system.local/invoice/update/129101?SPHostUrl=..... If we update 5 different invoices, the load test summary will report 5 different URLS. That’s not useful at all.

The solution is to open the .webtest that sends the GET/POST requests and set the “Reporting Name” property. All the calls to that action will then be reported as one URL.There’s no need to have a different “Reporting Name” for the GET and POST requests to this URL, the load test summary will automatically report 2 separate summary lines for the URL.

Running your testcases concurrently

System- and integration tests can take a lot of time to complete. If you want to speed this up then you can choose to run multiple test cases at the same time in parallel.

Unit Testcases in Visual Studio


You can configure Visual Studio to run unit concurrent testcases on your local machine using the following procedure:

  1. Ensure your project is using a .testsettings file
  2. Open it in an XML editor (Notepad++ or visual Studio’s “Open With” command)
  3. Set the parallelTestCount attribute on the Execution entity some some integer value

This will only work for situations where all the following is true:

  • The testcases and related frameworks are thread-safe
  • The testcases are of type unit testcase. It really won’t work for coded-ui tests
  • Its NOT a Data-Driven Unit Test
  • There are 0 diagnostic adapters included in the .testsettings file
  • Your machine has multiple cores

Controller and Agents with Visual Studio


If you have enough Physical or Virtual Machines available, then its always possible to configure a Controller and multiple agents. That will allow you to distribute your testcases over the machines. There’s no limitation on the type of testcases here. The only thing to take into account that running testcases that interact with a User Interface will require specific configuration on part of the agent machines. I use this setup mainly for running a lot of performance-tests against my client’s SharePoint farm.

See Configuring Test Controllers and Test Agents for Load Testing for details.

SpecFlow / SpecRun


SpecRun is a test-runner from the makers of SpecFlow that will allow you to run many concurrent SpecFlow scenarios. Its easy to setup and doesn’t really have any significant limitations. I use a combination of SpecFlow, SpecRun and an on-premise Selenium Grid to run my 250+ automated regression tests.

How to get video recordings of your testcase

Recording the screen during test execution is very helpful in diagnosing issues. Here are a few ways you can achieve this.

Visual Studio’s Data Diagnostic Adapters

If you’re running your tests with Visual Studio and/or Test Controller/Agents, then you can add a .testsettings file to the test project and configure it to include the “Video Data Diagnostic Adapter” That will record the screen while each testcase is executing and save the results in the output directory specified in the .testsettings file. I find this to be very useful when I’m authoring new testcases on my machine or when I’m running through a Controller/Agent for an app that interacts with the desktop.

Selenium Grid Servlets

If you’re running on a Selenium Grid setup, then there are various ways of getting the nodes to record the screen. Take a look at Tuenti VideoRecorderService or Selenium Video Node.

Screen recording software

You can use some kind of screen recording software like Camstudio or VNC. For me this isn’t preferred as it requires manual action on behalf of the tester.

Installing SpecFlow in Visual Studio 2013 Premium and up

In this post we go through the steps needed to get SpecFlow up-and-running in Visual Studio 2013. I’ve done this using the Premium edition and it should work on most editions except for Express.

Install SpecFlow

Click on Tools -> Extensions and updates:

Extensions and updates

Click on ‘Online’ and type in SpecFlow in the searchbox:

Extensions and updates

Click on the ‘ Download’ button and Visual Studio will download SpecFlow:

Download of the extension

When the download is complete, Visual Studio will show the following window:

SpecFlow's license agreement

Just click on the ‘Install’ button. Once Visual Studio has completed the install, it will show the following window. Click on ‘Restart now’ to restart Visual Studio:

Restarting Visual Studio after SpecFlow installation

Create a new Test project in Visual Studio

From Visual Studio’s menu click on File->New->Project. Then select the ‘Test’ template and choose ‘Unit Test Project’. Enter the name and location of the project conform your usual habits.

Creating a Test project

You now have a Solution that contains a Unit Test project which in-turn contains one empty test case.

Empty testcase

Setup your project to use SpecFlow

In earlier steps we installed the SpecFlow add-on. However, each project that wants to use this add-on, needs to set its references correctly. The NuGet package manager can do this for you. Use the following command in the Package Manager console to achieve this:
Install-Package SpecFlow -ProjectName SpecFlowAndTestRailIntegration
Using NuGet to setup your project's references

This has added an App.config  to our project:

Your App.config has been updated

And it has setup the references for us:

References have been updated

Add a SpecFlow testcase to the project

SpecFlow testcases, also known as ‘scenarios’ are contained within a feature file. One file can contain many testcases.

Right click on your project in the Solution Explorer, choose ‘Add->New Item’. This will show the following dialog, where you can see three types of SpecFlow related files. In this case, we will add a ‘ SpecFlow Feature File’

Adding a new .feature file

Click on ‘Add’  and the feature file will be added to the project:

.feature file has been added

Delete ‘UnitTest1.cs’

This file was generated by Visual Studio when we created the project, we don’t need it, so just delete ‘UnitTest1.cs’

Bindings: Getting the testcases to actually do something

When you add a new .feature file, SpecFlow fills it with a small default testcase:

SpecFlow's default content

In the above example, you can see that some lines contain purple text. This is SpecFlow’s way of indicating that these steps in the testcase don’t do anything yet and that you need to create some code that actually does what the step needs. This code is known as a ‘Binding’. Right click on the following line:

Given I have entered 50 into the calculator

and a context menu will appear:

Popup when right clicking in a .feature file

Select ‘Generate Step Definitions’

Generating step definitions

In this dialog window, SpecFlow allows us to control for which steps we want to create bindings and what the name of the C# class and file will be. For sake of this example, just accept the defaults and press ‘Generate’.

Saving the generated stepdefinitions

Press ‘Save’ and we now have a new .cs file in our project:

Project contains a new .cs file

This file contains the following not-so-useful code. We will complete this code later on.

using System;
using TechTalk.SpecFlow;

namespace SpecFlowAndTestRailIntegration
{
    [Binding]
    public class SpecFlowFeature1Steps
    {
        [Given(@"I have entered (.*) into the calculator")]
        public void GivenIHaveEnteredIntoTheCalculator(int p0)
        {
            ScenarioContext.Current.Pending();
        }

        [When(@"I press add")]
        public void WhenIPressAdd()
        {
            ScenarioContext.Current.Pending();
        }

        [Then(@"the result should be (.*) on the screen")]
        public void ThenTheResultShouldBeOnTheScreen(int p0)
        {
            ScenarioContext.Current.Pending();
        }
    }
}

Configuring which unittest framework to use

If you compile the solution at this point in time, then you would probably get a bunch of the following compilation errors:
The type or namespace name 'NUnit' could not be found (are you missing a using directive or an assembly reference?)
This is due to the fact that the default App.config for SpecFlow, wants to use the NUnit framework which we don’t have in this example. Instead, modify the App.Config to use Microsoft’s test framework as shown below:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow" />
  </configSections>
  <specFlow>
    <unitTestProvider name="MsTest" />
    <!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config -->
  </specFlow>
</configuration>

When we save changes to the App.config, SpecFlow detects this and will ask you the following question:

SpecFlow asking to regenerate codebehinds

It’s almost always fine to just press ‘Yes’. Now when we build the solution, we will only get 3 informational messages that can be ignored:

Three remaining messages

Finishing the code in Binding

To actually get the teststeps to do something useful, we need to implement the different steps in the binding. Replace the content of the .cs file with the following:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using TechTalk.SpecFlow;

namespace SpecFlowAndTestRailIntegration
{
    [Binding]
    public class SpecFlowFeature1Steps
    {
        private List<int> inputs;
        private int result;

        public SpecFlowFeature1Steps()
        {
            this.inputs = new List<int>();
        }

        [Given(@"I have entered (.*) into the calculator")]
        public void GivenIHaveEnteredIntoTheCalculator(int p0)
        {
            this.inputs.Add(p0);
        }

        [When(@"I press add")]
        public void WhenIPressAdd()
        {
            this.result = 0;
            foreach(int number in this.inputs)
            {
                result += number;
            }
        }

        [Then(@"the result should be (.*) on the screen")]
        public void ThenTheResultShouldBeOnTheScreen(int p0)
        {
            if(p0 != this.result)
            {
                Assert.Fail(String.Format("Actual result {0} does not match expected result {1}",
                    this.result, 
                    p0));
            }
        }
    }
} 

Run the test

Build the solution, this should now build without any errors and open the Test Explorer window (Test->Windows->Test Explorer) and click on ‘Run All’

Testcase has run successfully

Congratulations, you’ve just run a SpecFlow test!