Testcase management – my favorite tools

What’s the state of the product? Which areas need more work? Is it good enough for go-live?

These are some of the most important questions every tester and testmanager needs to be able answer right now!

If you’re working on a small project then a few worksheets in Excel with functions such as COUNTIF(), DSELECT() and DSUM() go a long way. But what if your project has lots of user stories, testcases and multiple testers? Then we need some tools to help us keep track of user stories, testcases, testresults and bugs.

Microsoft Team Foundation Server and Visual Studio

When you’re working with Microsoft technology, then there’s a big chance that the team is using TFS and Visual Studio for development. These tools are great for testing and testcase management too.

What does it cost?

Assuming you’re not eligible for volume licensing. You’ll need Team Foundation Server, priced at €646.00 for 5 users, or a Visual Studio Online Services subscription priced at €30.00 to €60.00 per user per month. Also each team member needs one of the following editions of Visual Studio with a MSDN subscription, even when you’re using Visual Studio Online Services.

  • Test Professional (€2,810.00)
  • Premium (€7,927.00)
  • Ultimate (€17,228.00)

Watch-out though: The ‘Express’ and ‘Professional’ (without ‘test’) editions aren’t suitable.

What does it provide?

This setup includes:

  • Creation of user stories, testplans, testsuites, testcases, testresults, bugs
  • Maintaining the relationships between these types of work items
  • Running your manual and automated tests
  • Reports such as defect and testcase status per user story

TestRail

TestRail is a web-based tool that you can either install on your own machine or let TestRail host it on their cloud infrastructure.

What does it cost?

The standalone version costs $239.00. The hosted version costs $25.00 per user per month. This excludes the cost for the 3rd party tools mentioned later in this post.

What does it provide?

TestRail includes

  • Creation of testsuites, testcases, test results
  • Maintaining the relationships between these types of work items
  • Reports such as defect and testcase status per user story
  • An HTTP based API for integration with your automated testcases

TestRail can maintain the relationships between testcases and user stories/requirements. However, it assumes you have a 3rd party web-based tool that stores the the user stories and that it can link to these using an URL such as 

http://server/view.html?%id_of_user_story%

TestRail is also able to integrate with loads of 3rd party bugtrackers. It contains plug-ins to integrate with most popular bugtrackers such as
JIRA, FogBugz, OnTime, Fixx, Lighthouse, Gemini, Youtrack, Bitbucket, Rally, Microsoft Team Foundation Sever (TFS), BugZilla, Vault Pro, Mantis, Trac, Bugtracker.NET, Redmine, Pivotal Tracker, GitHub and Assembla

If you are using another bugtracker, then TestRail also supports using a simple URL scheme such as 

http://server/view_bug?%id_of_bug%

TestLodge

TestLodge is a web-application available as a hosted instance in the cloud.

What does it provide?

TestLodge includes

  • Creation of requirements/userstories, testplans, testsuite, testcases, testresults
  • Maintaining the relationships between these types of work items
  • Reports such as defect and testcase status per user story

It integrates with loads of bugtrackers such as FogBugz, Redmine, BugZilla, Assembla, Basecamp, GitHub, Trello, JIRA, Lighthouse, Mantis, Unfuddle, Pivotal Tracker, Redbooth

Unfortunately there’s no API to integrate your automated testcases with TestLodge. They are currently still workin to deliver one. So you’ll have to manually set the results of automated testcases through the web-interface. If you’re working in an environment without automated tests, then TestLodge is in my opinion a very good tool, if you have lots of automated tests, then maybe TestRail is a better choice.

What does it cost?

The pricing is $14 – $119 per month depending on the number testplans, testcases and testruns. The number of users doesn’t matter.