In a previous post I talked about TestRail and using it for testcase management. In this post we dive into getting this up-and-running using a combination of SpecFlow and TestRail. SpecFlow is the tool that will run our testcases. TestRail is where we maintain lists of testsuites, testcases and testresults.

Setup a project on your TestRail account

First we start by creating a new project in TestRail just for the purpose of this post:

Create a testsuite in TestRail

Within the project we define a testsuite that will contain our automated testcases. Depending on your specific situation you might want to have multiple suites.

Create a testcase in TestRail

Now its time to create a testcases in TestRail that represents our automated testcase:

After saving this, we have 1 testsuite containing 1 testcase:

Create a test run in TestRail

A single testcase can be run multiple times and it might have a different result each time. Therefore, TestRail wants us to schedule the testcase in a testrun and we provide the testresult for the combination of the testcase within that testrun.
In the above screenshot we see that we have created a testrun with id R1 and it includes a test with ID T1. Remember this ID as we will need it later when our automated test talks to your TestRail account.

Setup TestRail to allow access to the API

Before the API can be used, you need to enable it for your TestRail project. Go to your Dashboard and click on ‘Administration’
Click on ‘Manage site settings’, then on the ‘API’ tab and put a checkmark in the ‘Enable API’ checkbox:

Create a TestRail user to be used by the automated tests

Although not required, it is wise to have the automated testcases integrate with TestRail through a dedicated user. For this post I created a user with the following settings:

Username	SpecFlow
E-mail adres	An email that you are able to access and that is not yet in use by another user
Password	SpecFlow
Language	English
Locale	English (United States)
Timezone	UTC+1

Download the TestRail API

TestRail provides a HTTP/JSON API that you can use to integrate your favorite testtool with your online TestRail projects. Its documented on http://docs.gurock.com/testrail-api2/start

Download the API from the link in the TestRail documentation and extract the .zip into the directory where your solution is located.

Add the API to your solution

Insert the TestRail .csproj into your solution using File->Add->Existing Project:

Visual Studio might ask you which .Net framework to use. Change it to .Net 4.5

Build the Gurock project and add a reference in your testproject to the Gurock assembly.

Also add a reference to the Newtonsoft JSON library. We need this as the TestRail API returns JSON objects that we need to deal with in our C# code.

My installation of Visual Studio 2013 already had this library, if your installation doesn’t then the TestRail API docs provide a download link.

Create your automated testcase in SpecFlow

Our starting point will be the automated testcase we created earlier

Add the following using statements to your Binding:

using Gurock.TestRail;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

Add the following method to the Binding. This code is executed by SpecFlow whenever a testcase has been run and it will push the result to TestRail. Because this is just a very basic introduction, I have hardcoded the ID of the test that we remembered from earlier.

        [AfterScenario]
        public static void AfterScenario()
        {
            Gurock.TestRail.APIClient client = new Gurock.TestRail.APIClient("https://yoursite.testrail.com/");
            client.User = "user@domain.com"; //Put the e-mail of your user here
            client.Password = "SpecFlow"; //Put the password of your user here

            Dictionary<string, object> testResult = new Dictionary<string, object>();
            if(null != ScenarioContext.Current.TestError)
            {
                
                testResult["status_id"] = "5"; //failed;
                testResult["comment"] = ScenarioContext.Current.TestError.ToString();
            }
            else
            {
                testResult["status_id"] = "1"; //passed
            }
            
            client.SendPost("add_result/1", testResult); //Here I am using a hardcoded test id. 
        }

Note: Although the TestRail docs use a http:// address, you must use a https:// address if your site is hosted on TestRails cloud service, otherwise the API will throw exceptions containing HTTP 401 error messages.

Run the test and check your TestRail site

Now run the SpecFlow testcase and you will see that TestRail now reports that the test has passed: