Basic integration of automated testcases with TestRail

TestRail's logo
In a previous post I talked about TestRail and using it for testcase management. In this post we dive into getting this up-and-running using a combination of SpecFlow and TestRail. SpecFlow is the tool that will run our testcases. TestRail is where we maintain lists of testsuites, testcases and testresults.

Setup a project on your TestRail account

First we start by creating a new project in TestRail just for the purpose of this post:
The created project in TestRail

The created project in TestRail

Create a testsuite in TestRail

Within the project we define a testsuite that will contain our automated testcases. Depending on your specific situation you might want to have multiple suites.
Image showing creation of a new suite

Creating a new suite


Image showing the created suite

Testsuite created

Create a testcase in TestRail

Now its time to create a testcases in TestRail that represents our automated testcase:
Creating a new testcase

Creating a new testcase


After saving this, we have 1 testsuite containing 1 testcase:
New testcase created

New testcase created

Create a test run in TestRail

A single testcase can be run multiple times and it might have a different result each time. Therefore, TestRail wants us to schedule the testcase in a testrun and we provide the testresult for the combination of the testcase within that testrun.
Testrun created

Testrun created, no results yet

In the above screenshot we see that we have created a testrun with id R1 and it includes a test with ID T1. Remember this ID as we will need it later when our automated test talks to your TestRail account.

Setup TestRail to allow access to the API

Before the API can be used, you need to enable it for your TestRail project. Go to your Dashboard and click on ‘Administration’
TestRail project administration page

TestRail project administration page

Click on ‘Manage site settings’, then on the ‘API’ tab and put a checkmark in the ‘Enable API’ checkbox:
API enabled for project

API enabled for project

Create a TestRail user to be used by the automated tests

Although not required, it is wise to have the automated testcases integrate with TestRail through a dedicated user. For this post I created a user with the following settings:

UsernameSpecFlow
E-mail adresAn email that you are able to access and that is not yet in use by another user
PasswordSpecFlow
LanguageEnglish
LocaleEnglish (United States)
TimezoneUTC+1

Download the TestRail API

TestRail provides a HTTP/JSON API that you can use to integrate your favorite testtool with your online TestRail projects. Its documented on http://docs.gurock.com/testrail-api2/start

Download the API from the link in the TestRail documentation and extract the .zip into the directory where your solution is located.

Extracting API into your solution

Extracting API into your solution

Add the API to your solution

Insert the TestRail .csproj into your solution using File->Add->Existing Project:
Inserting Gurock.TestRail.csproj

Inserting Gurock.TestRail.csproj

Visual Studio might ask you which .Net framework to use. Change it to .Net 4.5

Visual Studio asking which .Net version to use

Visual Studio asking which .Net version to use

Build the Gurock project and add a reference in your testproject to the Gurock assembly.

Adding reference to Gurock.TestRail assembly

Adding reference to Gurock.TestRail assembly

Also add a reference to the Newtonsoft JSON library. We need this as the TestRail API returns JSON objects that we need to deal with in our C# code.

Adding reference to JSON.Net

Adding reference to JSON.Net


My installation of Visual Studio 2013 already had this library, if your installation doesn’t then the TestRail API docs provide a download link.

Create your automated testcase in SpecFlow

Our starting point will be the automated testcase we created earlier

Add the following using statements to your Binding:

using Gurock.TestRail;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

Add the following method to the Binding. This code is executed by SpecFlow whenever a testcase has been run and it will push the result to TestRail. Because this is just a very basic introduction, I have hardcoded the ID of the test that we remembered from earlier.

        [AfterScenario]
        public static void AfterScenario()
        {
            Gurock.TestRail.APIClient client = new Gurock.TestRail.APIClient("https://yoursite.testrail.com/");
            client.User = "user@domain.com"; //Put the e-mail of your user here
            client.Password = "SpecFlow"; //Put the password of your user here

            Dictionary<string, object> testResult = new Dictionary<string, object>();
            if(null != ScenarioContext.Current.TestError)
            {
                
                testResult["status_id"] = "5"; //failed;
                testResult["comment"] = ScenarioContext.Current.TestError.ToString();
            }
            else
            {
                testResult["status_id"] = "1"; //passed
            }
            
            client.SendPost("add_result/1", testResult); //Here I am using a hardcoded test id. 
        }

Note: Although the TestRail docs use a http:// address, you must use a https:// address if your site is hosted on TestRails cloud service, otherwise the API will throw exceptions containing HTTP 401 error messages.

Run the test and check your TestRail site

Now run the SpecFlow testcase and you will see that TestRail now reports that the test has passed:
TestRail showing passed testcase

TestRail showing passed testcase