Property based testing in dotnet with FsCheck

In a previous prost, I explained property based testing. In this post we’ll see a simple example in dotnet with Fscheck

Lets assume we built code to transfer money from account A to account B. Some properties that comes to mind are:

  1. Successful transfer changes account balances.
  2. Successful transfer leaves source balance as non-negative.
  3. Failed transfer do not change account balances.

Our first implementation looks like this (don’t judge! … its supposed to be buggy)

class BuggyBank
{
    public int BalanceOfAccountA {get; set;} 
    public int BalanceOfAccountB {get; set;}
    
    public void TranserToAccountB(int Amount) 
    {
        BalanceOfAccountA -= Amount;
        BalanceOfAccountB += Amount;
    }
}

First we setup an empty dotnet unit test project like this:

mkdir FsCheckExample
cd FsCheckExample
dotnet new mstest
dotnet add package FsCheck --version 3.0.0-beta2

Then we add one test to validate all properties like this. Its vital to use QuickCheckThrowOnFailure() instead of QuickCheck() otherwise the test runner never reports failures.

    [TestMethod]
    public void MultiplePropertiesInOneTest()
    {
        Prop.ForAll<int, int, int>((StartBalanceForA, StartBalanceForB, AmountToTransfer) => {
            BuggyBank T = new()
            {
                BalanceOfAccountA = StartBalanceForA,
                BalanceOfAccountB = StartBalanceForB
            };
            try 
            {
                T.TranserToAccountB(AmountToTransfer);
            } 
            catch 
            {
                //The transfer failed
                bool BalancesUnchanged = (T.BalanceOfAccountA == StartBalanceForA && T.BalanceOfAccountB == StartBalanceForB);
                return BalancesUnchanged.Label("Failed transfer do not changes account balances.");
            }

            //Transfer succeeded
            bool BalancesChanged = T.BalanceOfAccountA != StartBalanceForA && T.BalanceOfAccountB != StartBalanceForB;
            bool NonNegativeBalanceOfA = T.BalanceOfAccountA >= 0;
            return
                NonNegativeBalanceOfA.Label("Successful transfer leaves source balance as non-negative")
                .And(BalancesChanged).Label("Successful transfer change account balances.")
            ;
            
        }).QuickCheckThrowOnFailure();
    }

When we run this test, FsCheck discovers the code allows transfers of 0 which violate the property that balances must change after successful transfer.

  Failed MultiplePropertiesInOneTest [674 ms]
  Error Message:
   Test method FsCheckExample.FsCheckExample.MultiplePropertiesInOneTest threw exception: 
System.Exception: Falsifiable, after 1 test (0 shrinks) (10302727913982160643,12688396740290863899)
Last step was invoked with size of 2 and seed of (10747404355721025928,11972757671557555113):
Label of failing property: Successful transfer change account balances.
Original:
(0, 0, 0)
with exception:
System.Exception: Expected true, got false.

After fixing that bug, FsCheck finds yet another bug! Transfers succeed even when the source account’s balance is insufficient:

  Failed MultiplePropertiesInOneTest [444 ms]
  Error Message:
   Test method FsCheckExample.FsCheckExample.MultiplePropertiesInOneTest threw exception:
System.Exception: Falsifiable, after 8 tests (4 shrinks) (16420426895032412258,4100991820053463935)
Last step was invoked with size of 9 and seed of (13117456436209885594,15897597155289983093):
Labels of failing property (one or more is failing):
Successful transfer change account balances.
Successful transfer leaves source balance as non-negative
Original:
(-2, -2, 2)
Shrunk:
(0, 0, 1)
with exception:
System.Exception: Expected true, got false.

Property based testing

Many test techniques rely on your assumptions where bugs are likely to be. In practice its really difficult to be complete and correct in your assumptions, so we end-up with defects in our product. For software used at scale, those defects are likely to cause real problems.

Property Based Testing avoids relying on assumptions. Instead it uses randomization to exhaustively check your product always meets some ‘property’ It often finds defects in the code as well as ambiguities in specifications.

Frequently used terminology

Term Meaning
Property Some fact or truth that always applies to your system. Some examples:

  • The total amount of money in a set of bank accounts is constant regardless of what transfers are executed.
  • The result of serializing and deserializing some object is identical to the initial object.
  • The result of squaring a number is zero or more.
Generator Code that generates random input suitable for your test. Some examples are:

  • Generate a random integer.
  • Generate a random string.
  • Generate an object whose members have random values suitable for their type.
  • Generate a HTTP GET request with a random url.
  • Generate a randomly ordered set of command BUT it never starts with command X
Shrinker Takes some generated input (usually one that caused a failure) and make it one size smaller. Test frameworks usually keep shrinking until they find the smallest input that still causes a failure. This really helps in analyzing the bug.

Are my tests still deterministic if we use randomization?

Yes, you control the source of randomness. As long as you use the same source with the same seed, tests on your machine run exactly the same as elsewhere.

What frameworks help me get this up and running?

Fscheck for DotNet.
quick (no Shrinking) or gopter (with Shrinking) for Golang.
fast-check for JavaScript / TypeScript.

Whats a good place to have these type of tests?

These tests need to run fast. You’ll frequently see them in the build/unit-test stages. Sometimes in the component- or integration test stages of a pipeline.

Playwright in CI/CD pipelines

I use Playwright for testing in .NET with Azure DevOps. A CI pipeline performs builds. A CD pipeline deploys our product and runs the tests.

Playwright needs the browser binaries available in %USERPROFILE%\AppData\Local\ms-playwright The documentation says to run bin\Debug\netX\playwright.ps1 install to download them. In my case playwright.ps1 did not exist and test were unable to run. I solved it in my CI pipeline like this. 

  - task: PowerShell@2
    displayName: Download Playwright Browsers
    inputs:
      targetType: inline
      script: >-
        cd <directory with the playwright .csproj>
        dotnet build
        dotnet tool install --global Microsoft.Playwright.CLI
        playwright install
...
  build remaining projects, code analysis etc. etc.
...
  - task: CopyFiles@2
    displayName: 'Copy Files to: $(build.artifactstagingdirectory)'
    inputs:
      SourceFolder: $(system.defaultworkingdirectory)
      Contents: '**\bin\$(BuildConfiguration)\**'
      TargetFolder: $(build.artifactstagingdirectory)
      CleanTargetFolder: true

This creates the script, downloads the browsers and includes the script into the build artifact for other stages and pipelines.

If your CD pipeline uses different machines or users, you need to run playwright.ps1 install in the CD pipeline before starting tests.

Comparing Playwright to Selenium

Playwright is a library for controlling web browsers similar to Cypress, Nightwatch, Selenium etc etc. Its modern, advanced and fast! Especially compared to Selenium and Selenium Grid from providers such as BrowserStack and SauceLabs.

Playwright supports videos, console and network logs out of the box. Even for headless browsers.

You can work with downloads, browser’s network-stack and console.

It has convenient ways of locating elements and easily combines different locator types into 1 locator. Selenium has ByChained, but is more cumbersome.

It automatically waits for elements to be actionable, while Selenium requires the tester to use constructs like: Wait.Until(ElementIsClickable()).Click()

Playwright does way less DOM access than Selenium. Here’s an somewhat extreme example to show the difference. If you do this in Selenium, then for each row, it will query the DOM to return a WebElement for the checkbox in the row. 

var rows = FindElements(By.ClassName("table-row"))
foreach(var row in rows)
{
   var checkbox = row.FindElement(By.ClassName("checkbox"))
}

Playwright won’t query the DOM for the checkbox. It returns a new locator (equivalent to Selenium’s By class) derived from the row web element to find that specific check-box.

Runs headless in CI/CD pipelines but still delivers video recordings and logfiles.

Although most tutorials use the default Playwright test runner, its works great with TypeScript and cucumber-js.

Using defineParameterType() with multiple regexes in cucumber-js

In a previous post I showed how to automatically replace parameter values before they get passed to your step definition code.

Now lets say you want that replacing/parsing/transformation for single- and double-quoted string like this in your feature file

When I print "Hello world!"
When I print 'its a beautiful day!'

Well…you’re in luck! The defineParameterType() method allows you to pass an array of regexes. We can use that to support both single- and double quoted strings with the same transformation function.

There’s a big gotcha here though. The the docs say this about the transformation function

A function or method that transforms the match from the regexp. Must have arity 1 if the regexp doesn’t have any capture groups. Otherwise the arity must match the number of capture groups in regexp.

In other words, when you use an array of regexes or if your regex has multiple capture groups, the function must have the same number of parameters as the total number of capture groups of all regexes in the array.

When cucumber calls the function, each parameter contains result from the corresponding capture group in the regex/array. If a regex does not match then cucumber passes an undefined value to the corresponding parameter number of the transform function So you’ll have to check each element if its undefined/null or not before using it.

defineParameterType({
    regexp: [/'([^']*)'/, /"([^"]*)"/],
    transformer: function (singleQ, doubleQ) {
        return singleQ ? replacePlaceholders(singleQ) : replacePlaceholders(doubleQ)
    },
    name: "a_string_with_replaced_placeholders"
});

Automatic type conversion of parameters in SpecFlow

In a previous post I showed that SpecFlow can change values of parameters. This mechanism is not just limited to transforming content of string values. It can also convert the literal string in your feature file to some complex object for your step-definition code.

Let say in your feature file you want to write steps like this:

Scenario: MyScenario
    When I print this list 'A,B,C,D'

Then you might be tempted to convert the string 'A,B,C,D' to a list like this 

[Binding]
public class MyBindings
{
    [StepDefinition(@"I print this list '([^']*)'") ]
    public void PrintList(string input)
    {
        var items = input.Split(',')
        foreach(var item in items)
        {
            Console.WriteLine(item)
        }
    }
}

Don’t do this…it causes the same problems as mentioned in the previous post.

Instead SpecFlow’s Step Argument Conversion lets us simplify our step definition code to this:

[Binding]
public class MyBindings
{
    [StepDefinition(@"I print this list '([^']*)'") ]
    public void PrintList(IEnumerable<string> items) 
    {
        foreach(var item in items)
        {
            Console.WriteLine(item)
        }
    }
   
    [StepArgumentTransformation]
    public IEnumerable<string> TransformStringToList(string input)
    {
        return input.Split(',');
    }
}

Automatic parsing of parameters in SpecFlow

Most implementations of cucumber can automatically replace parameter values with some run time value. Let say in your feature file you want to write steps like this:

Scenario: MyScenario
    When I print 'Hello from test case {test}' 
    When I save the value '{test}' to my database

Then you might be tempted to write these bindings (dont do this…keep reading!)

[Binding]
public class MyBindings
{
    [StepDefinition(@"I print '([^']*)'") ]
    public void PrintMessage(string Message)
    {
        string Name = ....
        Message = Message.Replace("{test}",Name)
        Console.WriteLine(Message);
    }

    [StepDefinition(@"I save '([^']*)' to my database") ]
    public void SaveToDatabase(string Message)
    {
        string Name = ....
        Message = Message.Replace("{test}",Name)
        Console.WriteLine(Message);
    }
}

This way of writing step definitions becomes a real problem as the size of your automation increases:

  1. we have to repeat the same logic in every step definition.
  2. Its easy to forget to repeat that code.
  3. New placeholders need you to find and update all places where this kind of parsing is done.
  4. Its code that’s needed but not relevant for the objective of the step definitions.

Luckily SpecFlow’s Step Argument Conversion helps us! We can simplify our binding code to this:

[Binding]
public class MyBindings
{
    [StepDefinition(@"I print '([^']*)'") ]
    public void PrintMessage(string Message) => Console.WriteLine(Message);

    [StepDefinition(@"I save '([^']*)' to my database") ]
    public void SaveToDatabase(string Message) => Console.WriteLine(Message);
    
    [StepArgumentTransformation]
    public string TransformParsedString(string input)
    {
        string Name = ...
        return input.Replace("{test}",Name);
    }
}

Other implementations of cucumber also implement this principle. Its frequently referred to using these buzzwords:

  1. Transforms / Transformations
  2. Parsing / Replacing
  3. StepArgumentTransformation

Automatically replacing/transforming input parameters in cucumber-js

Most implementations of cucumber provide a mechanism for changing literal text in the feature file to values or objects your step definition code can use. This is known as step definition or step argument transforms. Here’s how this works in cucumber-js.

Assume we have this scenario:

Scenario: Test
    When I print 'Welcome {myname}'
    And I print 'Today is {todays_date}'

And we have this step-definition.

defineStep("I print {mystring}", async function (this: OurWorld, x: string) {
    console.log(x)
});

Notice the use of {mystring} in the Cucumber expression

We can use defineParameterType() to automatically replace all placeholders.

defineParameterType({
    regexp: /'([^']*)'/,
    transformer: function (s) {
        return s
            .replace('{todays_date}', new Date().toDateString())
            .replace('{myname}', 'Gerben')
    },
    name: "mystring",
    useForSnippets: false
});

You can even use this to for objects like so:

defineParameterType({
    name: 'color',
    regexp: /red|blue|yellow/,
    transformer: s => new Color(s)
})

defineStep("I fill the canvas with the color {color}", async function (this: OurWorld, x: Color) {
    // x is an object of type Color
});

When I fill the canvas with the color red

How to dump the state of all variables in JMeter

To see the state of the variables and properties at a specific point in the test, you add a Debug sampler. This sampler dumps the information as response data into whatever result listener are configured.

If need the information in your own code to make decisions then you can use the following snippet of JSR223 code in a sampler or post processing rule:

import java.util.Map;
for (Map.Entry entry : vars.entrySet().sort{ a,b ->  a.key <=> b.key }) {
	log.info entry.getKey() + "  :  " + entry.getValue().toString();
}
for (Map.Entry entry : props.entrySet().sort{ a,b ->  a.key <=> b.key }) {
	log.info entry.getKey() + "  :  " + entry.getValue().toString();
}