In a previous post I talked about TestRail and using it for testcase management. In this post we dive into getting this up-and-running using a combination of SpecFlow and TestRail. SpecFlow is the tool that will run our testcases. TestRail is where we maintain lists of testsuites, testcases and testresults.
Setup a project on your TestRail account
First we start by creating a new project in TestRail just for the purpose of this post:
The created project in TestRail
Create a testsuite in TestRail
Within the project we define a testsuite that will contain our automated testcases. Depending on your specific situation you might want to have multiple suites.
Creating a new suite
Create a testcase in TestRail
Now its time to create a testcases in TestRail that represents our automated testcase:
Creating a new testcase
After saving this, we have 1 testsuite containing 1 testcase:
New testcase created
Create a test run in TestRail
A single testcase can be run multiple times and it might have a different result each time. Therefore, TestRail wants us to schedule the testcase in a testrun and we provide the testresult for the combination of the testcase within that testrun.
Testrun created, no results yet
In the above screenshot we see that we have created a testrun with id R1 and it includes a test with ID T1. Remember this ID as we will need it later when our automated test talks to your TestRail account.
Setup TestRail to allow access to the API
Before the API can be used, you need to enable it for your TestRail project. Go to your Dashboard and click on ‘Administration’
TestRail project administration page
Click on ‘Manage site settings’, then on the ‘API’ tab and put a checkmark in the ‘Enable API’ checkbox:
API enabled for project
Create a TestRail user to be used by the automated tests
Although not required, it is wise to have the automated testcases integrate with TestRail through a dedicated user. For this post I created a user with the following settings:
|E-mail adres||An email that you are able to access and that is not yet in use by another user|
|Locale||English (United States)|
Download the TestRail API
TestRail provides a HTTP/JSON API that you can use to integrate your favorite testtool with your online TestRail projects. Its documented on http://docs.gurock.com/testrail-api2/start
Download the API from the link in the TestRail documentation and extract the .zip into the directory where your solution is located.
Extracting API into your solution
Add the API to your solution
Insert the TestRail .csproj into your solution using File->Add->Existing Project:
Visual Studio might ask you which .Net framework to use. Change it to .Net 4.5
Visual Studio asking which .Net version to use
Build the Gurock project and add a reference in your testproject to the Gurock assembly.
Adding reference to Gurock.TestRail assembly
Also add a reference to the Newtonsoft JSON library. We need this as the TestRail API returns JSON objects that we need to deal with in our C# code.
Adding reference to JSON.Net
My installation of Visual Studio 2013 already had this library, if your installation doesn’t then the TestRail API docs provide a download link.
Create your automated testcase in SpecFlow
Our starting point will be the automated testcase we created earlier
Add the following using statements to your Binding:
Add the following method to the Binding. This code is executed by SpecFlow whenever a testcase has been run and it will push the result to TestRail. Because this is just a very basic introduction, I have hardcoded the ID of the test that we remembered from earlier.
public static void AfterScenario()
Gurock.TestRail.APIClient client = new Gurock.TestRail.APIClient("https://yoursite.testrail.com/");
client.User = "email@example.com"; //Put the e-mail of your user here
client.Password = "SpecFlow"; //Put the password of your user here
Dictionary<string, object> testResult = new Dictionary<string, object>();
if(null != ScenarioContext.Current.TestError)
testResult["status_id"] = "5"; //failed;
testResult["comment"] = ScenarioContext.Current.TestError.ToString();
testResult["status_id"] = "1"; //passed
client.SendPost("add_result/1", testResult); //Here I am using a hardcoded test id.
Note: Although the TestRail docs use a http:// address, you must use a https:// address if your site is hosted on TestRails cloud service, otherwise the API will throw exceptions containing HTTP 401 error messages.
Run the test and check your TestRail site
Now run the SpecFlow testcase and you will see that TestRail now reports that the test has passed:
TestRail showing passed testcase