Category Archives: Software Development

Migrating from YouTrack to JIRA

Recently I wanted to migrate about 400 issues and 350 attachments from a YouTrack OnDemand instance to a JIRA InCloud instance. JIRA doesn’t provide an importer that is compatible with YouTrack, so I coded a quick .Net C# application that migrated the data for me.

I started with quick list of my must- and nice-to-haves:

Must-haves
Entity Information to migrate
Projects Name
Issues Title, description, state and priority
Issues Attachments belonging to the issue
Issues Comments, including date and author
Nice-to-haves
Entity Information to migrate
Issues Reporter and assignee
Issues Tags
Projects and Issues Components
Issues Affected and fixed version information
Issues Relationship between issues (duplicate/relates-to etc etc)
Issues Historical information such as when the issue was transitioned from one state to another

I didn’t want to migrate or convert between YouTrack’s WIKI formatting used in issue description/comments and the JIRA way of formatting those fields. In fact it turns out that these formats are very similar, so that was a pleasant surpise when I was finished.

The first choice…REST or Import plugin?

The first choice I had to make was between JIRA’s REST API or JIRA’s JSON Import plugin. I opted for the plugin because the REST-API tends to completely ignore information such as state, dates and users. Being able to control the content of these fields is really crucial for data migration.

Getting the issues out of YouTrack…YouTrack and YouTrackSharp challenges

I already blogged how to get issues and attachments out of YouTrack, so there weren’t too many surprises:

  • YouTrackSharp won’t return the description of a project
  • YouTrackSharp won’t return an issue’s tags (a.k.a. labels) or comments. You need to call IssueManager.GetIssue() for each issue returned from instead of IssueManager.GetAllIssuesForProject()
  • The fieldnames and their types are different between the IssueManager.GetIssue() and IssueManager.GetAllIssuesForProject() calls
  • Version numbers associated with an issue in fields affectedVersion and FixedVersion are stored as a CSV string, not as a ICollection in YouTrackSharp
  • The text of a comment is usually returned as the .Text member of the dynamic object. However, I’ve seen a few issues where its returned as a .text member. In C# this difference in case is significant. I used the following approach to handle both cases: 
    try {
    ExportComment.Text = Comment.Text;
}
catch (Microsoft.CSharp.RuntimeBinder.RuntimeBinderException) {
    ExportComment.Text = Comment.text;
}

Importing the issues into JIRA

JSON Import documentation

The JSON structure that JIRA can import is documented on Overview and details.

The import plug-in basically does the following. Firstly it will create all the users listed in the JSON. Secondly it will create the projects, components and versions. Thirdly it will import the issues into each project. If the issue contains an attachment, it will download it from the specified URL/webserver into your JIRA instance’s datastore and attach it to the issue. Finally it will create any links between the issues.

Don’t worry about the license limit on the number of users. The plug-in will create them all, but any user above your license limit wont be granted access to the JIRA application and wont count towards the license. Also, after the import is complete, its fine to delete the attachments from your URL.

So the requirements for my application were:

  1. Be able to list all distinct users that are referenced somewhere in an issue, comment or attachment
  2. Per project, be able to list all distinct components that are referenced somewhere in an issue
  3. Per project, be able to list all distinct versions that are referenced somewhere in an issue field
  4. Be able to list all distinct relationships between issues. These relationships could in theory be cross-project
  5. Per issue, be able to translate YouTrack’s values for fields into JIRA’s equivalent. Specifically the following:
    • YouTrack’s state field to JIRA’s state and resolution
    • YouTrack’s default 4 priority values to JIRA’s default 6 priority values
    • YouTrack’s name for Issue types to JIRA’s name for Issue types
  6. Be able to translate YouTrack’s usernames to JIRA’s usernames. I had a few users that existed in both systems with slightly different usernames
  7. Be able to place the downloaded attachments from YouTrack on webserver that JIRA can access and write that URL into the JSON datastructure.

Jira JSON import gotcha’s

  • You are not allowed to supply the resolved date in the issue object. created and updated are fine though
  • The JSON import documentation doesn’t make it clear that you can control what the key of an imported issue should be using the key property of an issue object. If you don’t supply this property, then JIRA will simply give each issue a key equal to the order in which its listed in the JSON. This will almost always be a problem as its very common for issues to in YouTrack to have been deleted
  • YouTrack can handle 1 issue containing multiple attachments with the same name. The JIRA JSON import will throw the following exception and will stop importing more attachments for the issue. I only had 1 issue that had 2 attachments with the same name in YouTrack and I removed one of them 
    com.atlassian.jira.plugins.importer.external.ExternalException: com.atlassian.jira.web.util.AttachmentException: Could not save attachment to storage: java.io.FileNotFoundException: /data/service/j2ee_jira/catalina-base/temp/jira-importers-plugin-downloader-2621864330368162205.tmp (No such file or directory)
	at com.atlassian.jira.plugins.importer.imports.importer.impl.ExternalUtils.attachFile(ExternalUtils.java:354)
	at com.atlassian.jira.plugins.importer.imports.importer.impl.DefaultJiraDataImporter.createIssue(DefaultJiraDataImporter.java:944)
	at com.atlassian.jira.plugins.importer.imports.importer.impl.DefaultJiraDataImporter.importIssues(DefaultJiraDataImporter.java:764)
	at com.atlassian.jira.plugins.importer.imports.importer.impl.DefaultJiraDataImporter.doImport(DefaultJiraDataImporter.java:390)
...
Caused by: com.atlassian.jira.web.util.AttachmentException: Could not save attachment to storage: java.io.FileNotFoundException: /data/service/j2ee_jira/catalina-base/temp/jira-importers-plugin-downloader-2621864330368162205.tmp (No such file or directory)
	at com.atlassian.jira.issue.managers.DefaultAttachmentManager.createAttachmentOnDisk(DefaultAttachmentManager.java:473)
...
  • YouTrack and Jira have a different interpretation of the direction of the Duplicate issue links. 
                        
                    Assume that in YouTrack the follwing link exists: YouTrack: XXX-27 is duplicated by XXX-1
                    Then the IssueLink object will look like this:
                            SourceId	"XXX-27"	string
		                    TargetId	"XXX-1"	string
		                    TypeInward	"duplicates"	string
		                    TypeName	"Duplicate"	string
		                    TypeOutward	"is duplicated by"	string
                    If we translate that to JIRA's JSON format 
                    {
                      "name": "Duplicate",
                      "sourceId": "XXX-27",
                      "destinationId": "XXX-1"
                    },
                    Then JIRA will report that XXX-27 duplicates XXX-1. Ergo,for the Duplicate type, we need to swap Source and Target
  • If you have a private installation of JIRA, then you can control the format of the project key. However, in OnDemand instances, the key is restricted to only upper- and lowercase letters, you cant change that. I had 2 projects in YouTrack whose key contained numbers. I could have written a few lines of code to replace the numbers with some letters, but in my case it was far easier to modify the project in YouTrack and remove the numbers.

Highlighting webelements from Selenium

When I’m using selenium to navigate through a web-app, I like to record the session and visually highlight the DOM elements that my testscripts are using.

The IWebDriver and IWebElement interfaces don’t supply a method for this….so we’ll have to build our own. The approach I use, is to inject some JavaScript into the webpage to modify the style of the DOM element.

Here’s the code:

        public void highlightElement(IWebElement element)
        {
            IJavaScriptExecutor js = this.Driver as IJavaScriptExecutor;
            string OriginalStyle   = element.GetAttribute("style");
            string HighlightedStyle = OriginalStyle + "border: 3px solid red;";
            
            for (int i = 0; i < 3; i++)
            {
                js.ExecuteScript("arguments[0].setAttribute('style', arguments[1]);", element, HighlightedStyle);
                System.Threading.Thread.Sleep(500);
                js.ExecuteScript("arguments[0].setAttribute('style', arguments[1]);", element, OriginalStyle);
                System.Threading.Thread.Sleep(500);
            }
        }

Say for example, that I have the following testscript:

Feature: SeleniumHighlightingOfWebElements
	I want to show how I can show a user which elements in the DOM my testcases are using

Scenario: DemonstrateHighlighting
	Given I navigate to 'http://en.wikipedia.org/wiki/Hello_world_program'
	Then there must be a link with 'text' 'computer program'
	And there must be an element with 'xpath' '//./div[@class='thumbinner' and .//a[@href and normalize-space() =  'Firefox'] ]'
	And there must be a link with 'text' 'display device'
	And there must be an element with 'xpath' '//./div[@class='thumbinner' and .//a[@href and normalize-space() =  'CNC'] ]'
	And there must be a link with 'text' 'programming languages'
	And there must be an element with 'xpath' '//./div[@class='thumbinner' and .//a[@href and normalize-space() =  'PlayStation Portable'] ]'

That will result in the following screen capture:

grepping in Powershell

Many unix/linux users are intimately familiar with regular expressions and using them with grep, sed or awk in a pipeline. A typical usage scenario is the following command:

ls | grep -i '.*[0-9].*\.dll' | sort

So how do we do grep in PowerShell? Well, we can use PowerShell’s operators:

Operator Description
-match Matches its input property agains a regular expression. By default Powershell uses case-insensitive matching. Thats different than .Net’s default behavior with the RegEx class
-imatch Case-insensitive version of Match
-cmatch Case-sensitive version of Match
-like This compares its input against a string that can contain a wildcard. So no regular expressions usable here
-contains Only tells you if the input does or doesn’t contain the exact value you supplied

For our specific example, the PowerShell equivalent is:

Get-ChildItem | where { $_.Name -match ".*[0-9].*\.dll" } | Sort-Object -Property Name

And gives us the following output:

    Directory: C:\Windows\System32

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        20-11-2010     22:29     640512 advapi32.dll
-a---         11-6-2011      3:58     138056 atl100.dll
-a---         14-7-2009      3:14      65024 avicap32.dll
-a---        20-11-2010     22:29      91648 avifil32.dll
-a---         14-7-2009      3:14      10752 bitsprx2.dll
...
...

Another approach is to directly use the RegEx class from .Net. An example:

$matchOptions = [System.Text.RegularExpressions.RegexOptions]::IgnoreCase
$matcher = new-object System.Text.RegularExpressions.Regex (".*[0-9].*\.dll",$matchOptions)
$files  = Get-ChildItem
$files | where { $matcher.IsMatch($_.Name)  }

How to programatically download attachments from YouTrack

In a previous post I showed how to retrieve your bugs, comments and attachments from YouTrack. However, actually downloading each attachment is a little more work. Assume that the variable Url contains a value like "http://xxxxx.myjetbrains.com/youtrack/_persistent/yyyyy.doc?file=xx-xxx&v=0&c=false"

The following raises a HTTP NotFound exception:

var foo = Connection.Get(Url); //YouTrackSharp.Infrastructure.Connection

As does this:

String basePath = new System.Uri(Url).PathAndQuery;
var foo = Connection.Get(basePath );

The following will raise a HTTP NotAuthorized exception because we are not supplying the authentication cookie to the web-server. 

byte[] file = new System.Net.WebClient().DownloadData(Url);

I looked at the YouTrackSharp.Infrastructure.Connection class and I didn’t see anything that would let me retrieve the authentication cookie.

Instead of writing code to manually implement the YouTrack REST API, I decided to take a different approach. I decided to modify YouTrackSharp itself to give me access to the authentication cookie.
I removed the NuGet package YouTrackSharp from my solution.
I downloaded the source of YouTrackSharp from GitHub and unzipped it next to my own project.
I included src\YouTrackSharp\YouTrackSharp.csproj into my solution
I modified src\YouTrackSharp\Infrastructure to add the public modifier as follows:

public CookieCollection _authenticationCookie;

I changed my own program to download the file as follows:

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(Url);
req.CookieContainer = new CookieContainer();
req.CookieContainer.Add(Connection._authenticationCookie);
req.Method = "GET";
WebResponse response = req.GetResponse();
System.IO.Stream Stream = response.GetResponseStream();
var fileStream = System.IO.File.Create(String.Format("C:\\tmp\\{0}", Name));
Stream.CopyTo(fileStream);
fileStream.Close();

Exporting your bugs out of YouTrack

If you need to export your bug reports from your YouTrack instance, this is how to get it done using C#. Its a small example that will print all the issues,comments and attachments for each project in your instance.

Don’t forget to enable the REST API in your YourTrack settings page.

Create a new project called YouTrackExport and use NuGet to install the YouTrackSharp package

PM> Install-Package YouTrackSharp
Attempting to resolve dependency 'EasyHttp (≥ 1.6.29.0)'.
Attempting to resolve dependency 'JsonFX (≥ 2.0.1209.2802)'.
Installing 'JsonFx 2.0.1209.2802'.
...
...

The following code will print out the information:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using System.Dynamic;
using YouTrackSharp.Infrastructure;
using YouTrackSharp.Projects;
using YouTrackSharp.Issues;
using YouTrackSharp.Admin;
using System.Net;

namespace YouTrackExport
{
    class Program
    {
        static void Main(string[] args)
        {
            String Username = "xxx";
            String Password = "yyy";
            String Site = "zzz.myjetbrains.com";

            YouTrackSharp.Infrastructure.Connection Connection;
            YouTrackSharp.Projects.ProjectManagement ProjectManager;
            IEnumerable<YouTrackSharp.Projects.Project> Projects;
            YouTrackSharp.Issues.IssueManagement IssueManager;
            IEnumerable<YouTrackSharp.Issues.Issue> Issues;
            IEnumerable<YouTrackSharp.Issues.Comment> Comments;


            Connection = new Connection(Site, 80, false, "youtrack");
            Connection.Authenticate(Username, Password);
                    
            ProjectManager = new ProjectManagement(Connection);
            Projects    = ProjectManager.GetProjects();
            foreach (Project project in Projects)
            {
               
                Console.WriteLine(String.Format("Found project {0} - {1} ", project.ShortName, project.Name));
                IssueManager = new IssueManagement(Connection);
                Issues = IssueManager.GetAllIssuesForProject(project.ShortName); 
                
                //An issue in youtrack can have many fields that we dont know at compile time.
                //Therefore its a .Net DynamicObject
                foreach (dynamic Issue in Issues) 
                {
                    Console.WriteLine(String.Format("\tFound issue {0}", Issue.Id));

                    Comments = IssueManager.GetCommentsForIssue(Issue.Id);
                    foreach (Comment Comment in Comments)
                    {
                        Console.WriteLine(String.Format("\t\tComment: {0} - {1} - {2}", Comment.Author, Comment.Created, Comment.Text));
                    }

                    foreach (dynamic Attachment in Issue.Attachments)
                    {
                        String Name = Attachment.name;
                        String Url = Attachment.url;
                        String Author = Attachment.authorLogin;
                        String Id = Attachment.id;
                        String Group = Attachment.group;
                        long Created = Attachment.created;
                        Console.WriteLine(String.Format("\t\tAttachment: {0} - {1} - {2}", Name, Author, Url));
                    }
                }
            }
        }
    }
}

Some gotcha’s that I encountered

The following code wont work if your YouTrack instance is hosted in the cloud:

var connection = new Connection("http://zzz.myjetbrains.com/youtrack"); 
var connection = new Connection("zzz.myjetbrains.com/youtrack"); 
var connection = new Connection("zzz.myjetbrains.com");

The following code will raise a HTTPException: NotFound Not Found exception. You need to to use the ShortName member:

Issues = IssueManager.GetAllIssuesForProject(project.Name);

PowerShell for recursive file and directory listings with MD5 hashes

The other day I needed to know if the files in a specific directory structure on different systems were identical or different. Many tools exist for doing file and directory comparisons but the challenge here was that each system was locked-down, no installation of software was possible. And even worse, we needed to compare at least 3 different machines that couldn’t reach each others drives or shares. Instead of hunting around to find which one of the many tools would work in this environment I thought that Windows Powershell should be able to do this quite easily. In fact, it turned out to be so easy that just finding another tool would have taken much longer.

The script uses the Get-ChildItem cmdlet to recursively get each file/directory in the form of a System.IO.FileInfo or System.IO.DirectoryInfo object. We calculate the MD5 hash using the ComputeHash() method of the System.Security.Cryptography.MD5CryptoServiceProvider class. We simply write the hash and some interesting fields as a CSV string to the output.

Here’s the bits of code glued into a single script.

function Get-MD5
{
  [CmdletBinding(SupportsShouldProcess=$false)]
  param
  (
    [Parameter(Mandatory=$true, ValueFromPipeline=$true, ValueFromPipelineByPropertyName=$true, HelpMessage="file(s) to create hash for")]
    [Alias("File", "Path", "PSPath", "String")]
    [ValidateNotNull()]
    $InputObject
  )
 
  begin
  {
     $hasher = new-object System.Security.Cryptography.MD5CryptoServiceProvider
  }
 
  process
  {
    $hashBytes = ''

    $item = Get-Item $InputObject -ErrorAction SilentlyContinue
 
    if ($item)
    {
      $InputObject = $item
    }
 
    if ($InputObject -is [System.IO.FileInfo])
    {
      $stream    = $null;
      $hashBytes = $null
 
      try
      {
        $stream     = $InputObject.OpenRead();
        $hashBytes  = $hasher.ComputeHash($stream);
      }
      finally
      {
        if ($stream -ne $null)
        {
          $stream.Close();
        }
      }
    }

 
    Write-Output ([BitConverter]::ToString($hashBytes)).Replace("-",'')
  }
}

function Get-FilesWithHash
{
    [CmdletBinding(SupportsShouldProcess=$false)]
    param
    (
        [string]$Directory,
        [switch]$Recurse
    )
 

    Get-ChildItem -Path $Directory -Recurse:$Recurse | foreach {
        $item = $_
        $hash = ''
        if (!$_.PSIsContainer) 
        {
            $hash = Get-MD5 $_    
        }
        else
        {
            $hash = ''
        }
        Write-Output "$($item.FullName),$hash,$($item.Length),$($item.CreationTime),$($item.LastWriteTime)" 
        
    }
}

Combining SpecFlow and Selenium

One of the cool things I like to do when testing webapps, is to define the testcases using SpecFlow (also known as Cucumber) and then use Selenium to actually execute the testcases against the web-application.

Here is an example testcase:

Feature: SpecFlowAndSelenium
	For demonstration purposes, I want to show how the human-readable
	SpecFlow testcases can be executed using Selenium to operate
	a real webbrowser

Scenario: SeleniumAndSpecFlow
	Given I navigate to 'http://www.google.com'
	Then there must be a control with 'id' 'gbqfq' 
	When I type 'Hello World!' into it
	Then there must be a control with 'id' 'gbqfb'
	When I click on it
	Then there must be a link with 'partialtext' 'Wikipedia' 
	When I click on it
	Then there must be a control with 'id' 'searchInput'
	When I type 'Selenium' into it
	Then there must be a control with 'id' 'searchButton' 
	When I click on it
	Then there must be a link with 'text' 'chemical element'
	And there must be a link with 'text' 'Selenium (software)'
	When I click on it
	Then there must be a link with 'text' 'web browsers'

And here you can see Selenium using Firefox to perform all the actions and checks:

Getting MS Access to remember the password for linked tables

If you’re using linked tables from MS Access to a database server (Oracle or SQL Server) using the MS Access GUI you can simply put a check mark in the ‘Remember password’ box. If you’re dynamically creating linked tables from VBA and supplying the username and password in the Connection member of the TableDef object then Access will still prompt for credentials when its restarted.

You can avoid this by using the dbAttachSavePWD attribute. Example:

Dim tDef As TableDef
Set tDef = CurrentDb.CreateTableDef("MyNewTable", dbAttachSavePWD)
tDef.Connect = ...
tDef.SourceTableName = ...

Creating linked tables in MS Access using VBA

Recently I ran into a system that used Oracle as back-end database and an MS Access database/app containing the forms as front-end for the users. The Access database connected to Oracle tables and views using ODBC linked tables. We had about 5 different Oracle servers for development, test, acceptance and production. Due to a lot of legacy code the table names in Access and Oracle weren’t always the same.

Frequently we needed to change to which Oracle server a specific instance of the Access app would talk to. Instead of manually removing and relinking the tables, I created a simple local table to define which tables should exist in the Access database and what the tables name on the Oracle server should be. A simple VBA subroutine actually establishes the links.

Here’s a short version of it:

Option Compare Database
Option Explicit
Private Const gDSNTemplate = "ODBC;DRIVER={Oracle in OraHome817};SERVER=${server};UID=${user};PWD=${password};DBQ=${server};DBA=W;APA=T;EXC=F;XSM=Default;FEN=T;QTO=T;FRC=10;FDL=10;L...(cant remember the rest)..."
 
Public Sub LinkTables(Server As String, username As String, password As String)
    Dim dsn As String
    Dim strAccessName As String
    Dim strOracleName As String
    Dim tDef As TableDef
    Dim rs As Recordset

    'Create the DSN for the requested environment
    dsn = gDSNTemplate
    dsn = Replace(dsn, "${server}", Server)
    dsn = Replace(dsn, "${user}", username)
    dsn = Replace(dsn, "${password}", password)

    Set rs = CurrentDb.OpenRecordset("SELECT * FROM tblLinkedTables")
    Do While Not rs.EOF
        strAccessName = rs!AccessName
        strOracleName = rs!OracleName
 
        'Remove the outdated linked table 
        '(ignore the error if doesn't exist)
        On Error Resume Next
        DoCmd.DeleteObject acTable, strAccessName
        On Error GoTo 0
 
        'Create the linked table
        Set tDef = CurrentDb.CreateTableDef(strAccessName, dbAttachSavePWD)
        tDef.Connect = dsn
        tDef.SourceTableName = strOracleName
        CurrentDb.TableDefs.Append tDef
        tDef.RefreshLink
 
        rs.MoveNext
    Loop
End Sub

Balsamiq a great tool for sketching your user interface

Often I’ll find myself wanting to show someone an idea for a user interface. A few years ago a colleague introduced to me to Balsamiq and I’ve loved it ever since. Its simple to quickly visualize my ideas and its easy enough that I can sit next to someone and we start moving stuff around until we’re happy with the look. It exports to .pdf, .png and the windows clipboard so its trivial to send the idea through mail or *gasp* print-it-out.

An image showing a basic design for a blog page in Balsamiq

Its easy to customize the contents of the various UI controls in the mock-up:
A image showing how a control in Balsamiq can be customized for each specific need

It has plenty of UI elements included by default and you’ll be well set for anything related to web, application, tablet or phone. There’s also lots of plugins available. Some eye-catchers are integration with Confluence and JIRA. The pricing for stand-alone and for the OnCloud versions can be found on Balsamiq’s pricing page

Here’s a video showing how to integrate Balsamiq with JIRA: