Monthly Archives: May 2014

Sometimes reality rocks

Today one of my testers logged a bug that made me smile: “When I hit the ‘Save As’ button, my desktop is rotated left by 90 degrees”

It was reproducible.

This is why I love my job.

Setting the language for your .feature files

The syntax of the .feature files is meant to be readable by your users, so of course its logical that we should be able to create .feature files in many different languages.

How to set the language to use

Setting the language for all .feature files

If you want to set the language for all .feature files, then you can specify this in the App.config:

    <language feature="nl-NL" />

Setting the language per .feature file

If you want set the language for a single .feature file then you can use a comment in the header of your feature file:

#language: nl-NL
Functionaliteit: Optellen van getallen

How to find the syntax for your language

Now that you’ve set your desired language, you can start creating .feature files using the keywords in your user’s language. How do we find out what these keyword are? Take a look at the Gherkin i18n file and you’ll be able to figure it out.

"lv": {
"name": "Latvian",
"native": "latviešu",
"feature": "Funkcionalitāte|Fīča",
"background": "Konteksts|Situācija",
"scenario": "Scenārijs",
"scenario_outline": "Scenārijs pēc parauga",
"examples": "Piemēri|Paraugs",
"given": "*|Kad",
"when": "*|Ja",
"then": "*|Tad",
"and": "*|Un",
"but": "*|Bet"
"nl": {
"name": "Dutch",
"native": "Nederlands",
"feature": "Functionaliteit",
"background": "Achtergrond",
"scenario": "Scenario",
"scenario_outline": "Abstract Scenario",
"examples": "Voorbeelden",
"given": "*|Gegeven|Stel",
"when": "*|Als",
"then": "*|Dan",
"and": "*|En",
"but": "*|Maar"

Basic integration of automated testcases with TestRail

TestRail's logo
In a previous post I talked about TestRail and using it for testcase management. In this post we dive into getting this up-and-running using a combination of SpecFlow and TestRail. SpecFlow is the tool that will run our testcases. TestRail is where we maintain lists of testsuites, testcases and testresults.

Setup a project on your TestRail account

First we start by creating a new project in TestRail just for the purpose of this post:
The created project in TestRail

The created project in TestRail

Create a testsuite in TestRail

Within the project we define a testsuite that will contain our automated testcases. Depending on your specific situation you might want to have multiple suites.
Image showing creation of a new suite

Creating a new suite

Image showing the created suite

Testsuite created

Create a testcase in TestRail

Now its time to create a testcases in TestRail that represents our automated testcase:
Creating a new testcase

Creating a new testcase

After saving this, we have 1 testsuite containing 1 testcase:
New testcase created

New testcase created

Create a test run in TestRail

A single testcase can be run multiple times and it might have a different result each time. Therefore, TestRail wants us to schedule the testcase in a testrun and we provide the testresult for the combination of the testcase within that testrun.
Testrun created

Testrun created, no results yet

In the above screenshot we see that we have created a testrun with id R1 and it includes a test with ID T1. Remember this ID as we will need it later when our automated test talks to your TestRail account.

Setup TestRail to allow access to the API

Before the API can be used, you need to enable it for your TestRail project. Go to your Dashboard and click on ‘Administration’
TestRail project administration page

TestRail project administration page

Click on ‘Manage site settings’, then on the ‘API’ tab and put a checkmark in the ‘Enable API’ checkbox:
API enabled for project

API enabled for project

Create a TestRail user to be used by the automated tests

Although not required, it is wise to have the automated testcases integrate with TestRail through a dedicated user. For this post I created a user with the following settings:

E-mail adresAn email that you are able to access and that is not yet in use by another user
LocaleEnglish (United States)

Download the TestRail API

TestRail provides a HTTP/JSON API that you can use to integrate your favorite testtool with your online TestRail projects. Its documented on

Download the API from the link in the TestRail documentation and extract the .zip into the directory where your solution is located.

Extracting API into your solution

Extracting API into your solution

Add the API to your solution

Insert the TestRail .csproj into your solution using File->Add->Existing Project:
Inserting Gurock.TestRail.csproj

Inserting Gurock.TestRail.csproj

Visual Studio might ask you which .Net framework to use. Change it to .Net 4.5

Visual Studio asking which .Net version to use

Visual Studio asking which .Net version to use

Build the Gurock project and add a reference in your testproject to the Gurock assembly.

Adding reference to Gurock.TestRail assembly

Adding reference to Gurock.TestRail assembly

Also add a reference to the Newtonsoft JSON library. We need this as the TestRail API returns JSON objects that we need to deal with in our C# code.

Adding reference to JSON.Net

Adding reference to JSON.Net

My installation of Visual Studio 2013 already had this library, if your installation doesn’t then the TestRail API docs provide a download link.

Create your automated testcase in SpecFlow

Our starting point will be the automated testcase we created earlier

Add the following using statements to your Binding:

using Gurock.TestRail;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

Add the following method to the Binding. This code is executed by SpecFlow whenever a testcase has been run and it will push the result to TestRail. Because this is just a very basic introduction, I have hardcoded the ID of the test that we remembered from earlier.

        public static void AfterScenario()
            Gurock.TestRail.APIClient client = new Gurock.TestRail.APIClient("");
            client.User = ""; //Put the e-mail of your user here
            client.Password = "SpecFlow"; //Put the password of your user here

            Dictionary<string, object> testResult = new Dictionary<string, object>();
            if(null != ScenarioContext.Current.TestError)
                testResult["status_id"] = "5"; //failed;
                testResult["comment"] = ScenarioContext.Current.TestError.ToString();
                testResult["status_id"] = "1"; //passed
            client.SendPost("add_result/1", testResult); //Here I am using a hardcoded test id. 

Note: Although the TestRail docs use a http:// address, you must use a https:// address if your site is hosted on TestRails cloud service, otherwise the API will throw exceptions containing HTTP 401 error messages.

Run the test and check your TestRail site

Now run the SpecFlow testcase and you will see that TestRail now reports that the test has passed:
TestRail showing passed testcase

TestRail showing passed testcase