Monthly Archives: April 2014

Testcase management – my favorite tools

What’s the state of the product? Which areas need more work? Is it good enough for go-live?

These are some of the most important questions every tester and testmanager needs to be able answer right now!

If you’re working on a small project then a few worksheets in Excel with functions such as COUNTIF(), DSELECT() and DSUM() go a long way. But what if your project has lots of user stories, testcases and multiple testers? Then we need some tools to help us keep track of user stories, testcases, testresults and bugs.

Microsoft Team Foundation Server and Visual Studio

When you’re working with Microsoft technology, then there’s a big chance that the team is using TFS and Visual Studio for development. These tools are great for testing and testcase management too.

What does it cost?

Assuming you’re not eligible for volume licensing. You’ll need Team Foundation Server, priced at €646.00 for 5 users, or a Visual Studio Online Services subscription priced at €30.00 to €60.00 per user per month. Also each team member needs one of the following editions of Visual Studio with a MSDN subscription, even when you’re using Visual Studio Online Services.

  • Test Professional (€2,810.00)
  • Premium (€7,927.00)
  • Ultimate (€17,228.00)

Watch-out though: The ‘Express’ and ‘Professional’ (without ‘test’) editions aren’t suitable.

What does it provide?

This setup includes:

  • Creation of user stories, testplans, testsuites, testcases, testresults, bugs
  • Maintaining the relationships between these types of work items
  • Running your manual and automated tests
  • Reports such as defect and testcase status per user story

TestRail

TestRail is a web-based tool that you can either install on your own machine or let TestRail host it on their cloud infrastructure.

What does it cost?

The standalone version costs $239.00. The hosted version costs $25.00 per user per month. This excludes the cost for the 3rd party tools mentioned later in this post.

What does it provide?

TestRail includes

  • Creation of testsuites, testcases, test results
  • Maintaining the relationships between these types of work items
  • Reports such as defect and testcase status per user story
  • An HTTP based API for integration with your automated testcases

TestRail can maintain the relationships between testcases and user stories/requirements. However, it assumes you have a 3rd party web-based tool that stores the the user stories and that it can link to these using an URL such as

http://server/view.html?%id_of_user_story%

TestRail is also able to integrate with loads of 3rd party bugtrackers. It contains plug-ins to integrate with most popular bugtrackers such as
JIRA, FogBugz, OnTime, Fixx, Lighthouse, Gemini, Youtrack, Bitbucket, Rally, Microsoft Team Foundation Sever (TFS), BugZilla, Vault Pro, Mantis, Trac, Bugtracker.NET, Redmine, Pivotal Tracker, GitHub and Assembla

If you are using another bugtracker, then TestRail also supports using a simple URL scheme such as

http://server/view_bug?%id_of_bug%

TestLodge

TestLodge is a web-application available as a hosted instance in the cloud.

What does it provide?

TestLodge includes

  • Creation of requirements/userstories, testplans, testsuite, testcases, testresults
  • Maintaining the relationships between these types of work items
  • Reports such as defect and testcase status per user story

It integrates with loads of bugtrackers such as FogBugz, Redmine, BugZilla, Assembla, Basecamp, GitHub, Trello, JIRA, Lighthouse, Mantis, Unfuddle, Pivotal Tracker, Redbooth

Unfortunately there’s no API to integrate your automated testcases with TestLodge. They are currently still workin to deliver one. So you’ll have to manually set the results of automated testcases through the web-interface. If you’re working in an environment without automated tests, then TestLodge is in my opinion a very good tool, if you have lots of automated tests, then maybe TestRail is a better choice.

What does it cost?

The pricing is $14 – $119 per month depending on the number testplans, testcases and testruns. The number of users doesn’t matter.

Logo of Techdays 2014

Techdays 2014 – Automated crossbrowser testing with CodedUI tests

Presented by Marcel de Vries. Was an interesting presentation with a overview of CodedUI tests and the UIMap. There’s loads of information available on-line, so I wont repeat that here. Some bits that did stick in my mind.

CodedUI webtests use Selenium as driver for Internet Explorer. You’ll need at least Visual Studio 2012 with update 1. You can also get support for Firefox or Chrome by installing the “Selenium components for Cross Browser Testing” Visual Studio Add-on/Extension. This Add-on will require you to install .Net 3.5 first.

You can select which browser you want to use with the following code

BrowserWindow.CurrentBrowser = "IE";
BrowserWindow.CurrentBrowser = "Chrome";
BrowserWindow.CurrentBrowser = "Firefox";

When you run your tests, you might be surprised that by default they run quite slow. This is due to the fact, that the framework will wait for all outstanding events and AJAX requests on the webpage to be fully complete. Classic Selenium only waits for a pageload to be complete and you’d explicitly have to write code using WebDriverWait to ensure your test isn’t going faster than your webpage can update itself. Microsoft’s framework can be configured to wait as much or as little as you want

//Wait until all actions are complete
Playback.PlaybackSettings.WaitForReadyLevel = WaitForReadyLevel.AllThreads;

//Only wait for render of the initial DOM 
Playback.PlaybackSettings.WaitForReadyLevel = WaitForReadyLevel.UIThreadOnly; 

//Only wait for pageload to be complete
Playback.PlaybackSettings.WaitForReadyLevel = WaitForReadyLevel.Disabled;   

The Page Object pattern is a good way of keeping your testcases maintainable. It introduces a nice separation between what a testcase wants to do and how it finds and uses the various controls such as text boxes, links and images.

Logo of Techdays 2014

Techdays 2014 – The road to continuous delivery

Presented by René van Osnabrugge at Techdays 2014. An interesting presentation. See slideshare (Dutch) for the slides.

Due to pressures and opportunities in the market, businesses feel the need to get new features available as fast as possible. They don’t know up-front which features or when. But as soon as they do know, they don’t want to wait for a project to complete a bunch of sprints just to get that one killer feature out there. So they feel the need to release faster, more often and into various environments running from test, acceptance and maybe all the way into production.
Enter ‘Continuous Delivery’

A collection of Processes, tools and techniques for rapid, reliable and continuous development and delivery of software.

In short: We want software to always be suitable for deployment and deployment should be doable with just a push of a button. This needs a few foundations to be successful.

Agile. Your organization needs to embrace the agile way of thinking and acting.
Quality. QA needs to have a strategy based on test early and test often. Doing continuous delivery of features that don’t work isn’t going to be a success.
Automation. Automate a much as you can. Builds, Unit / System / Acceptance Tests, deployments. And don’t forget the rollback scenarios.
Architecture and Configuration Management. Avoid huge monolithic systems, architect your system into smaller components that can be developed and deployed separately.
DevOps. Ensure releases are automated, have instrumentation in the software to measure how it’s used by the end-users.

Some interesting bits that happened to grab my attention were:
– Microsoft Release Management with TFS and workflows to manage the release path and validation/authorization over various environments;
– Using feature flags to ensure that unfinished features in the code don’t interfere with that killer feature you want to release right now;
– It’s not just the source-code that’s managed, it’s the builds too. After all, a build is what gets propagated to the various environments. We don’t have to create a new build for each environment

Logo of Techdays 2014

Techdays 2014 – Desired State Configuration (DSC) with PowerShell

Presented by Jeff Wouters at Techdays 2014. Using PowerShell we can define what configuration we want and the DSC framework will make sure that this configuration is enforced on a multitude of machines. Very much like Chef and Puppet.

Some examples of configuration settings you can define are Installing windows features, MSI’s, registry settings, services that must be started/stopped etc etc.

Jeff’s tips were:
You need WMF v4.

Install KB2883200. This is a must-have, don’t even try to use DSC without it.

Take a look at TechNet for overview and details.

Want to see what resources you have to use for DSC? Run the Get-DSCResource cmdlet.

Take a look at PowerShell DSC community resources.

Installing SpecFlow in Visual Studio 2013 Premium and up

In this post we go through the steps needed to get SpecFlow up-and-running in Visual Studio 2013. I’ve done this using the Premium edition and it should work on most editions except for Express.

Install SpecFlow

Click on Tools -> Extensions and updates:

Extensions and updates

Click on ‘Online’ and type in SpecFlow in the searchbox:

Extensions and updates

Click on the ‘ Download’ button and Visual Studio will download SpecFlow:

Download of the extension

When the download is complete, Visual Studio will show the following window:

SpecFlow's license agreement

Just click on the ‘Install’ button. Once Visual Studio has completed the install, it will show the following window. Click on ‘Restart now’ to restart Visual Studio:

Restarting Visual Studio after SpecFlow installation

Create a new Test project in Visual Studio

From Visual Studio’s menu click on File->New->Project. Then select the ‘Test’ template and choose ‘Unit Test Project’. Enter the name and location of the project conform your usual habits.

Creating a Test project

You now have a Solution that contains a Unit Test project which in-turn contains one empty test case.

Empty testcase

Setup your project to use SpecFlow

In earlier steps we installed the SpecFlow add-on. However, each project that wants to use this add-on, needs to set its references correctly. The NuGet package manager can do this for you. Use the following command in the Package Manager console to achieve this:
Install-Package SpecFlow -ProjectName SpecFlowAndTestRailIntegration
Using NuGet to setup your project's references

This has added an App.config  to our project:

Your App.config has been updated

And it has setup the references for us:

References have been updated

Add a SpecFlow testcase to the project

SpecFlow testcases, also known as ‘scenarios’ are contained within a feature file. One file can contain many testcases.

Right click on your project in the Solution Explorer, choose ‘Add->New Item’. This will show the following dialog, where you can see three types of SpecFlow related files. In this case, we will add a ‘ SpecFlow Feature File’

Adding a new .feature file

Click on ‘Add’  and the feature file will be added to the project:

.feature file has been added

Delete ‘UnitTest1.cs’

This file was generated by Visual Studio when we created the project, we don’t need it, so just delete ‘UnitTest1.cs’

Bindings: Getting the testcases to actually do something

When you add a new .feature file, SpecFlow fills it with a small default testcase:

SpecFlow's default content

In the above example, you can see that some lines contain purple text. This is SpecFlow’s way of indicating that these steps in the testcase don’t do anything yet and that you need to create some code that actually does what the step needs. This code is known as a ‘Binding’. Right click on the following line:

Given I have entered 50 into the calculator

and a context menu will appear:

Popup when right clicking in a .feature file

Select ‘Generate Step Definitions’

Generating step definitions

In this dialog window, SpecFlow allows us to control for which steps we want to create bindings and what the name of the C# class and file will be. For sake of this example, just accept the defaults and press ‘Generate’.

Saving the generated stepdefinitions

Press ‘Save’ and we now have a new .cs file in our project:

Project contains a new .cs file

This file contains the following not-so-useful code. We will complete this code later on.

using System;
using TechTalk.SpecFlow;

namespace SpecFlowAndTestRailIntegration
{
    [Binding]
    public class SpecFlowFeature1Steps
    {
        [Given(@"I have entered (.*) into the calculator")]
        public void GivenIHaveEnteredIntoTheCalculator(int p0)
        {
            ScenarioContext.Current.Pending();
        }

        [When(@"I press add")]
        public void WhenIPressAdd()
        {
            ScenarioContext.Current.Pending();
        }

        [Then(@"the result should be (.*) on the screen")]
        public void ThenTheResultShouldBeOnTheScreen(int p0)
        {
            ScenarioContext.Current.Pending();
        }
    }
}

Configuring which unittest framework to use

If you compile the solution at this point in time, then you would probably get a bunch of the following compilation errors:
The type or namespace name 'NUnit' could not be found (are you missing a using directive or an assembly reference?)
This is due to the fact that the default App.config for SpecFlow, wants to use the NUnit framework which we don’t have in this example. Instead, modify the App.Config to use Microsoft’s test framework as shown below:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="specFlow" type="TechTalk.SpecFlow.Configuration.ConfigurationSectionHandler, TechTalk.SpecFlow" />
  </configSections>
  <specFlow>
    <unitTestProvider name="MsTest" />
    <!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config -->
  </specFlow>
</configuration>

When we save changes to the App.config, SpecFlow detects this and will ask you the following question:

SpecFlow asking to regenerate codebehinds

It’s almost always fine to just press ‘Yes’. Now when we build the solution, we will only get 3 informational messages that can be ignored:

Three remaining messages

Finishing the code in Binding

To actually get the teststeps to do something useful, we need to implement the different steps in the binding. Replace the content of the .cs file with the following:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using TechTalk.SpecFlow;

namespace SpecFlowAndTestRailIntegration
{
    [Binding]
    public class SpecFlowFeature1Steps
    {
        private List<int> inputs;
        private int result;

        public SpecFlowFeature1Steps()
        {
            this.inputs = new List<int>();
        }

        [Given(@"I have entered (.*) into the calculator")]
        public void GivenIHaveEnteredIntoTheCalculator(int p0)
        {
            this.inputs.Add(p0);
        }

        [When(@"I press add")]
        public void WhenIPressAdd()
        {
            this.result = 0;
            foreach(int number in this.inputs)
            {
                result += number;
            }
        }

        [Then(@"the result should be (.*) on the screen")]
        public void ThenTheResultShouldBeOnTheScreen(int p0)
        {
            if(p0 != this.result)
            {
                Assert.Fail(String.Format("Actual result {0} does not match expected result {1}",
                    this.result, 
                    p0));
            }
        }
    }
} 

Run the test

Build the solution, this should now build without any errors and open the Test Explorer window (Test->Windows->Test Explorer) and click on ‘Run All’

Testcase has run successfully

Congratulations, you’ve just run a SpecFlow test!