Recipes for load and stability testing

The objective of load and stability testing is to determine if some System Under Test (SUT) is able to meet its performance and/or stability requirements. A secondary objective is to experiment how far we can push it before it breaks.

Some challenges to deal with are:

  1. How can we make sure the tests and the SUT use suitable data? You need some way of ensuring the SUT and test scripts have a common understanding of what data exists at start of the test.
  2. How can we deploy and start the SUT without its neighboring components being available or suitable for load testing?
  3. What about authentication blocking us from running automated tests?
  4. How can we test the impact of faults, caused by external components, when we don’t control those external components?

Here are some recipes ranging from complex to simple. None of these are perfect for all situations. Each has a best fit at some stage in the Software Development Life Cycle (SDLC)

End to end load tests

The entire product with all its software components, internal data and ‘customer’ data is deployed into some production like environment.

An example:

  1. Deploy all microservices onto a Kubernetes cluster.
  2. Use scripting to seed databases or use real-world (anonymized) databases and extract whatever content the tests need from those databases.
  3. Use tools and services such K6, JMeter, Locust, Gatling, Taurus, NeoLoad, BlazeMeter to generate load.
  4. Use your DevOps tooling to monitor and analyze the SUTs behavior.

Pros

Impact of resource limits, network set-up and caches are validated with this approach.

Unpredictable interactions between components become visible.

Impact of real world data set becomes visible.

Cons

Its costly and time consuming to arrange a suitable environment.

Deployment of components can be time consuming. Especially as the environment is likely to start in an undefined state due to previous test runs and deployments of feature branches.

Authentication and intrusion detection can block running these type of tests. Obtaining valid credentials might be impossible when Multi Factor Authentication is active..

Laws and regulations may block you from using real world data.

Bringing databases into a suitable starting state can be difficult and time consuming. Especially when the data is distributed over multiple services with own tech for data storage.

Component load/stability tests

A single component with its own internal data and fake, but large enough, ‘customer’ data is tested in isolation. It starts, gets tested and deleted.

A simple example is:

  1. Add a single micro service, the SUT, into a container image.
  2. Use frameworks such as Faker and Bogus to generate data for the SUT, mocks and repository databases.
  3. Add mock servers and their data into a container image. Something like WireMock or some simple custom built HTTP server.
  4. Use a real database server in a container to host your repository data.
  5. If you need to simulate intermittent faults on repository access, you can mock the repository classes to throw an exception x% of the time. In the sunny day scenario the mock simply forwards the request to the real repo class. Use frameworks such as Moq and Mockito to generate the mocks around the real repository classes.
  6. Add a load generator and the test scripts into the container.
  7. Configure the SUT to talk to the mock servers and repository / database server in the container.
  8. Run the container(s) as part of your CI/CD pipelines. Maybe using docker-compose on a single CI/CD agent or maybe in some temporary Kubernetes namespace.

If running everything in a few containers is not suitable, you can distribute the SUT, mocks, databases and load generator over multiple container hosts and/or external service providers. For example:

  1. Use providers of mocking tools to spin-up mock servers. Some providers are Mockoon, BlazeMeter Service Virtualization and many more.
  2. Same idea for database servers.
  3. Add the SUT into a container image and configure it to talk to the mock servers and databases.
  4. Start the SUT’s container image on some online platform (Kubernetes cluster, Azure Container Instances etc. etc.)
  5. Run the load test from some load test service provider like NeoLoad or BlazeMeter.

Pros

Having the SUT tested in isolation removes many blockers end-to-end tests suffer from. Its easier to build and maintain these tests. Less work to run and analyze them.

Tests act as an automated QA gate for a component. Failures are visible earlier in the SDLC.

Using mocks and custom repository classes makes it easy to control injection of faults . Its trivial to have a mock server or repository class simulate a failure in x% of the cases .

Cons

The SUT needs to be engineered in such a way that its possible to use your mocks, database-servers / repository classes. Legacy code might block this and need refactoring.

Usually you wont be testing with real resource limits that end-to-end test would run with.

Unit/integration test(s) running performance critical code in a tight loop

Use the unit test framework of the programming environment to run some critical code for a specific length of time or until a max number of iterations is reached.

An example would be to implement a NUnit/JUnit test to run some algorithm many times against various random data sets. Use frameworks such as Moq and Mockito to avoid depending on external systems and control what data the SUT gets back from the mocked dependencies. Use frameworks such as Faker, Bogus or Property Based test generators to generate the randomized inputs to the algorithm

Pros

Runs fast.

Does not depend on availability and state of external systems as they are mocked-away.

Integrates well with IDE’s.

Cons

Impact of interacting with external systems is not tested here. This will miss defects on topics such as connection pooling, buffering, latency and caching.

A tight loop is usually single threaded, it will miss defects caused by concurrency problems.

Failures during CI runs are hard to analyze as unit test frameworks are not generally geared to providing post-mortem analysis of time based metrics. Integration with a profiler of your programming language is likely to be needed.

Generating custom random inputs for your property based test in C# .Net with CsCheck

Here’s how you can create a custom generator for your property based test in C# using CsCheck.

Assume we need to generate instances of this class:

class MyClass
{
    public int MyInt { get; set; }
    public string MyString { get; set; }
}

For sake of this example, assume the generated instances must conform to these rules:

  1. The integer must be greater than 0.
  2. The string must be one of these: ❌, ✓, A string with 10 to 20 random characters.

Here’s what the generator looks like:

class MyGenerator
{
    public static Gen<MyClass> MyClass =>
         from i in Gen.Int where i > 0 
         from s in Gen.OneOf(Gen.String[10, 20], Gen.Const("❌"), Gen.Const("✓"))
         select new MyClass
         {
             MyInt = i,
             MyString = s,
         };
}

Here is how to use it:

    [Fact]
    public void EachInputMustBeCorrect()
    {
        MyGenerator.MyClass.List.Sample(Input =>
        {
            if(Input.Count == 0) { return; }
            Input.Should().AllSatisfy(MustBeCorrect);
        });
    }

    void MustBeCorrect(MyClass o)
    {
        o.MyInt.Should().BeGreaterThan(0);
        o.MyString.Should().Match(x => x == "❌"  || x == "✓" || ( x.Length >= 10 && x.Length <= 20));
    }

Loading container images into Kubernetes containerd

Do you need to run containers whose images are not available in a container registry? Here’s how…

Make sure you have the image available as a tar file. Here’s how to save one from a machine with docker:

docker save repository/image --output ./image.tar

Copy the tarfile to the target machine and run this command to load image into containerd:

sudo ctr -n k8s.io images import image.tar

To use this image in a Kubernetes deployment, make sure its marked as never pull:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: my-app
  labels:
    app: my-app

spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: xxxxx/my-app
          imagePullPolicy: Never

How to install Kubernetes onto physical machines for a home lab

On each machine: Install Ubuntu Server LTS 24.04

Ensure you can SSH into it and enable password less sudo

echo "$USER ALL=(ALL:ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/$USER

This helps in running commands on each machine in parallel.

On each machine: Install kubeadm

Based on Bootstrapping clusters with kubeadm

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | $ sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo swapoff -a

On each machine: Install containerd

Kubernetes recently deprecated usage of dockerd as the container runtime. So we’ll use containerd directly based on Anthony Nocentino’s blog: Installing and Configuring containerd as a Kubernetes Container Runtime

Configure the required kernel modules:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

Configure persistence across system reboots

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

Install containerd packages

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update && sudo apt-get install containerd.io

Create a containerd configuration file

sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

Set the cgroup driver to systemd. Kuberbetes uses systemd while containerd uses something else. They must both use the same setting:

sudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

Only on the first (master) machine

Initialize the K8s cluster. Save this output somewhere, you’ll need the kubeadm join ... part later.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

[init] Using Kubernetes version: v1.30.2
[preflight] Running pre-flight checks
...
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0629 19:20:06.570522   14350 checks.go:844] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.9" as the CRI sandbox image.
...
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.178.171:6443 --token v1flk8.wy9xyikw6kosevps \
        --discovery-token-ca-cert-hash sha256:e79a8516a0990fa232b6dcde15ed951ffe46880854fe1169ceb3b909d82fff00

On each machine: Follow the recommendation of kubeadmin to update the sandbox image.

Use a text editor to replace sandbox_image = "registry.k8s.io/pause:3.8" with sandbox_image = "registry.k8s.io/pause:3.9"

restart containerd

sudo systemctl restart containerd.service

On the master node: Ensure kubectl knows what cluster you work with

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

On each other machine: join them to the cluster:

kubeadm join 192.168.178.171:6443 \ 
    --token v1flk8.wy9xyikw6kosevps \
    --discovery-token-ca-cert-hash sha256:e79a8516a0990fa232b6dcde15ed951ffe46880854fe1169ceb3b909d82fff00

On the master node: Configure the POD network

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl apply -f kube-flannel.yml

Installation finished, check status:

kubectl get nodes

should give output like:

NAME        STATUS   ROLES           AGE     VERSION
optiplex1   Ready    control-plane   2d23h   v1.30.2
optiplex2   Ready    <none>          2d23h   v1.30.2
optiplex3   Ready    <none>          2d23h   v1.30.2

Using step definition code from Given and When contexts

Have you ever wanted to use a single step definition like I create file '(.*)' from Given and When contexts in a feature file like this?

Scenario: CreateFile
    When I create file 'hello.txt'
    Then ...

Scenario DeleteFile
    Given I create  file 'hello.txt'
    When I delete 'hello.txt'

Here’s how to have 1 method that implements both the Given and the When I create file '...'

Create a class like this:

public class GivenWhenAttribute : StepDefinitionBaseAttribute
{
readonly static StepDefinitionType[] types = new[] { StepDefinitionType.Given, StepDefinitionType.When };
public GivenWhen() : this(null)  {  }
public GivenWhen(string regex) : base(regex, types ) { }
public GivenWhen(string regex, string culture) : this(regex) { Culture = culture; }
}

Use it in in the [Binding] classes like this:

[GivenWhen("I create file '(.*)'")]
public void CreateFile(String Name) { File.Create(Name); }

Adding PACT Contract Testing to an existing golang code base

Here’s how to add a Contract Test to a Go microservice, a provider in pact terminology, using pact-go This post uses v1 of pact-go, as the V2 version is stil in beta.

Install PACT cli tools on your dev machine

A linux or dev-container based environment can just use the same approach as the CI pipeline documented later on in this post.

For a Windows dev machine, install the pact-cli tools like this:

  1. Use browser to download https://github.com/pact-foundation/pact-ruby-standalone/releases/download/v2.0.2/pact-2.0.2-windows-x86.zip
  2. unzip to c:\Repos
  3. Change PATH to include c:\repos\pact\bin
  4. Restart any editors or terminals
  5. Run go install gopkg.in/pact-foundation/pact-go.v1

Create a unit test to validate the service meets the contract/pact

Add unit test like below. Notice these settings and how the environment variables affect them. This helps when tests need to run both on the local development machine as well as on CI/CD machines.

Setting Effect
PublishVerificationResults Controls if pact should publish the verification result to the broker. For my local dev machine I dont publish. From the CI pipeline I do publish
ProviderBranch The name of the branch for this provider version.
ProviderVersion The version of this provider. My CI pipeline ensures variable PACT_PROVIDER_VERSION contains the unique build number. On my local machine its just set to 0.0.0 
package app
import (
    "fmt"
    "bff/itemrepository"
    "os"
    "strconv"
    "strings"
    "testing"

    "github.com/pact-foundation/pact-go/dsl"
    "github.com/pact-foundation/pact-go/types"
    "github.com/stretchr/testify/assert"
)

func randomItem(env string, name string) itemrepository.item{
    return itemrepository.item{
        Environment:               env,
        Name:                      name,
    }
}

func getEnv(name string, defaultVal string) string {
    tmp := os.Getenv(name)
    if len(tmp) > 0 {
        return tmp
    } else {
        return defaultVal
    }
}

func getProviderPublishResults() bool {
    tmp, err := strconv.ParseBool(getEnv("PACT_PROVIDER_PUBLISH", "false"))
    if err != nil {
        panic(err)
    }
    return tmp
}

func TestProvider(t *testing.T) {
    //Arrange: Start the service in the background.
    port, repo, _ := startApp(false)
    pact := &dsl.Pact{ Consumer: "MyConsumer", Provider: "MyProvider", }

    //Act: Let pact spin-up a mock client to verifie our service.
    _, err := pact.VerifyProvider(t, types.VerifyRequest{
            ProviderBaseURL:            fmt.Sprintf("http://localhost:%d", port),
            BrokerURL:                  getEnv("PACT_BROKER_BASE_URL", ""),
            BrokerToken:                getEnv("PACT_BROKER_TOKEN", ""),
            PublishVerificationResults: getProviderPublishResults(),
            ProviderBranch:             getEnv("PACT_PROVIDER_BRANCH", ""),
            ProviderVersion:            getEnv("PACT_PROVIDER_VERSION", "0.0.0"),
            StateHandlers: types.StateHandlers{
                "I have a list of items": func() error {
                    repo.Set("env1", []itemrepository.item{randomItem("env1", "tenant1")})
                        return nil
                },
            },
        })
    pact.Teardown()

    //Assert
    assert.NoError(t, err)
}

Ensure the CI pipeline runs the test and publishes verification results

For my CI builds, I run this to make sure the CI machine has the pact-cli tools installed
cd /opt
curl -fsSL https://raw.githubusercontent.com/pact-foundation/pact-ruby-standalone/master/install.sh | bash
export PATH=$PATH:/opt/pact/bin
go install github.com/pact-foundation/pact-go@v1
...
pipeline already runs the unit tests
...

Check `can-i-deploy` in the CD pipeline

Change the CD pipeline to

  • verify this version of the service has passed the contract test, Otherwise do not deploy to production
  • After successful deployment, inform broker which new version is running in production.

My CD pipeline runs on different machines, so it again has to ensure PACT is installed, just like the CI pipeline:

# pipeline variables
PACT_PACTICIPANT=MyProvider
PACT_ENVIRONMENT=production
PACT_BROKER=https://yourtenant.pactflow.io
PACT_BROKER_TOKEN=...a read/write token...preferably a system user to represent CI/CD actions...

# task to install PACT
cd /opt
curl -fsSL https://raw.githubusercontent.com/pact-foundation/pact-ruby-standalone/master/install.sh | bash
echo "##vso[task.prependpath]/opt/pact/bin"
...
...
# Task to check if the version of the build can be deployed to production
pact-broker can-i-deploy --pacticipant $PACT_PACTICIPANT --version $BUILD_BUILDNUMBER --to-environment $PACT_ENVIRONMENT --broker-base-url $PACT_BROKER --broker-token $PACT_BROKER_TOKEN
...
#Do whatever tasks you need to deploy the build to the environment
...
...
#Task to record the deployment of this version of the producer to the environment
pact-broker record-deployment --environment $PACT_ENVIRONMENT --version $BUILD_BUILDNUMBER --pacticipant $PACT_PACTICIPANT --broker-base-url $PACT_BROKER --broker-token $PACT_BROKER_TOKEN

Adding PACT Contract Testing to an existing TypeScript code base

I like Contract Testing! I added a contract test with PACT-js and Jest for my consumer like this:

Installing PACT

  1. Disable the ignore-scripts setting: npm config set ignore-scripts false
  2. Ensure build chain is installed. Most linux based CI/CD agents have this out of the box. My local dev machine runs Windows; according to the installation guide for gyp the process is:
    1. Install Python from the MS App store. This takes about 5 minutes.
    2. Ensure the machine can build native code. My machine had Visual Studio already so I just added the ‘Desktop development with C++’ workload using the installer from ‘Tools -> Get Tools and Features’ This takes about 15 – 30 minutes
    3. npm install -g node-gyp
  3. Install the PACT js npmn package: npm i -S @pact-foundation/pact@latest
  4. Write a unit test using either the V3 or V2 of the PACT specification. See below for some examples.
  5. Update your CI build pipeline to publish the PACT like this: npx pact-broker publish ./pacts --consumer-app-version=$BUILD_BUILDNUMBER --auto-detect-version-properties --broker-base-url=$PACT_BROKER_BASE_URL --broker-token=$PACT_BROKER_TOKEN

A V3 version of a PACT unit test in Jest

//BffClient is the class implementing the logic to interact with the micro-service.
//the objective of this test is to:
//1. Define the PACT with the microservice
//2. Verify the class communicates according to the pact

import { PactV3, MatchersV3 } from '@pact-foundation/pact';
import path from 'path';
import { BffClient } from './BffClient';

// Create a 'pact' between the two applications in the integration we are testing
const provider = new PactV3({
    dir: path.resolve(process.cwd(), 'pacts'),
    consumer: 'MyConsumer',
    provider: 'MyProvider',
});

describe('GET /', () => {
    it('returns OK and an array of items', () => {
        const exampleData: any = { name: "my-name", environment: "my-environment", };

        // Arrange: Setup our expected interactions. Pact mocks the microservice for us.
        provider
            .given('I have a list of items')
            .uponReceiving('a request for all items')
            .withRequest({method: 'GET', path: '/', })
            .willRespondWith({
                status: 200,
                headers: { 'Content-Type': 'application/json' },
                body: MatchersV3.eachLike(exampleData),
            });
        return provider.executeTest(async (mockserver) => {
            // Act: trigger our BffClient client code to do its behavior 
            // we configured it to use the mock instead of needing some external thing to run
            const sut = new BffClient(mockserver.url, "");
            const response = await sut.get()

            // Assert: check the result
            expect(response.status).toEqual(200)
            const data:any[] = await response.json()
            expect(data).toEqual([exampleData]);
        });
    });
});

A V2 version

import { Pact, Matchers } from '@pact-foundation/pact';
import path from 'path';
import { BffClient } from './BffClient';

// Create a 'pact' between the two applications in the integration we are testing
const provider = new Pact({
    dir: path.resolve(process.cwd(), 'pacts'),
    consumer: 'MyConsumer',
    provider: 'MyProvider',
});

describe('GET', () => {
    afterEach(() => provider.verify());
    afterAll(() => provider.finalize());

    it('returns OK and array of items', async () => {
        const exampleData: any = { name: "my-name", environment: "my-environment", };
        // Arrange: Setup our expected interactions. Pact mocks the microservice for us.
        await provider.setup()
        await provider.addInteraction({
            state: 'I have a list of items',
            uponReceiving: 'a request for all items',
            withRequest: { method: 'GET', path: '/',  },
            willRespondWith: {
                status: 200,
                headers: { 'Content-Type': 'application/json' },
                body: Matchers.eachLike(exampleData),
            },
        })

        // Act: trigger our BffClient client code to do its behavior 
        // we configured it to use the mock instead of needing some external thing to run
        const sut= new BffClient(mockserver.url, "");
        const response = sut.get()
        
        // Assert: check the result
        expect(response.status).toEqual(200)
        const data: any[] = await response.json()
        expect(data).toEqual([exampleData]);
    });
});

Contract Testing helps you deploy faster with more confidence

Many modern systems are made-up of lots of small components such a microservices and frontends. Each independently built, released and maintained by a bunch of different teams. This is really useful for scaling out your organisation and increasing the speed at which changes can be delivered to your customers.

For people involved in producing this kind of software, it poses a few challenges:

  1. Will a new version of component X still work with its neighboring components?
  2. What does component Y even expect of component X? What is the structure of the data X returns? What status codes and headers?
  3. What version of component X is running in which environment. What version of the interface does X have there?
  4. How do we start work on X even though we have no clue when Y will be available?
  5. If I write a mock for Y does it truly simulate its interface or am I blinded by my own assumptions?
  6. How can I quickly deploy and test X without having to spin up its neighboring components? Can I even achieve that in a large product?

Contract Testing solves these problems for you. Lets assume we have some micro-service X and a web front-end Y that communicates with it. The tools that implement Contract Testing let component Y document its expectations of the interface with X in a machine readable contract, frequently called a PACT.

How it works from Y’s point-of-view:

The unit/integration tests of Y define the PACT. Maybe its the same as the previous version, maybe it different. Both is fine.

They let the test framework dynamically generate a mock for X based on this PACT. Instead of needing to talking to a ‘real’ X, they talk to the mock.

The mock and Y’s own unit test together verify that Y really works according to the PACT. If the test fails, the team of Y has some fixing work to do and this process restarts. If the test passes, Y uploads the PACT to a broker and informs it which version of Y just validated successfully.

Whenever Y is deployed to production its CI/CD pipeline asks the broker if the current version of X in production has been validated against this PACT version. If not, Y’s deployment fails. If its fine, the pipeline informs the broker which version of Y is now deployed to production.

How it works from X’s point-of-view:

The unit/integration tests of X Start an instance of X running locally on some reachable network location.

It retrieves the PACT from the broker and uses the testing framework to generate a mock of Y. This mock sends requests to X based on the PACT and validates X’s responses match the PACT.

If the test fails, the team of X has some fixing work to do and this process restarts. If the test passed, X communicates to the broker that it was able to work with this version of the PACT.

Whenever X is deployed to production its CI/CD pipeline asks the broker if the current version of Y in production has been validated against this PACT version. If not, X’s deployment fails. If its fine, the pipeline informs the broker which version of X is now deployed to production.

Generating custom random inputs for your property based test in golang

Golang’s quick testing package is great for generating random input for your property based test. Sometimes you want to control how the input values are generated. Here’s how you can create a custom generator for the input parameters to your property based test.

Lets assume you want to test the function IsValid(string) with many random inputs …. but …. the input may only contain characters in range a-z, A-Z, and 0-9. Here’s how to do it:

func randomAlphaNumericString(output []reflect.Value, rnd *rand.Rand) {
    alphabet := []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789")
    size := rnd.Intn(8192)
    v := reflect.New(reflect.TypeOf("")).Elem()
    v.SetString(randStringOfLen(rnd, size, alphabet))
    output[0] = v
}

//Generates a string of len n containing random characters from alphabet
func randStringOfLen(rnd *rand.Rand, n int, alphabet[]rune) string {
    b := make([]rune, n)
    for i := range b {
        b[i] = alphabet[rnd.Intn(len(alphabet))]
    }
    return string(b)
}

func TestMyMethod(t *testing.T) {
    propertyTest := func(input string) bool { return true == IsValue(input) }
    c := quick.Config{MaxCount: 1000, Values: randomAlphaNumericString}
    if err := quick.Check(propertyTest , &c); err != nil {
        t.Error(err)
    }
}

Lets break it down into steps:

Firstly we define our propertyTest to check all input strings are valid. So far nothing special. This function takes a single string input parameter.

The quick.Config struct has the Values member. This member lets you supply a function to generate whatever parameters propertyTest needs. In our case the randomAlphaNumericString function does that job.

The randomAlphaNumericString function generates a suitable random string and it stores it as a reflect.Value in the output array at the position where the propertyTest expects to receive a string parameter.

TypeScript logo

Property based testing in TypeScript with fast-check

In a previous post, I explained property based testing. In this post we’ll see a simple example using fast-check

Let assume we’re building the same bank transfer code as described in the dotnet FsCheck earlier post. Here’s the TypeScript version of the test:

import BuggyBank from "../buggy-bank"
import * as fc from 'fast-check';

function transferProperties(startBalanceA: bigint
        , startBalanceB: bigint
        , amount: bigint) : void {

    const bank = new BuggyBank()
    bank.balanceOfA = startBalanceA
    bank.balanceOfB = startBalanceB
    try {
        bank.transfer(amount)
    } catch {
        //Transfer failed
        const balanceAUnchanged = bank.balanceOfA == startBalanceA
        const balanceBUnchanged = bank.balanceOfB == startBalanceB
        expect(balanceAUnchanged).toBeTruthy()
        expect(balanceBUnchanged).toBeTruthy()
        return
    }
    //Transfer succeeded
    const balanceAIsNonNegative = bank.balanceOfA >= 0
    const balanceAChanged = bank.balanceOfA != startBalanceA
    const balanceBChanged = bank.balanceOfB != startBalanceB
    expect(balanceAIsNonNegative).toBeTruthy()
    expect(balanceAChanged).toBeTruthy()
    expect(balanceBChanged).toBeTruthy()
}

test('properties of a bank transfer must be correct', () => {
    const config = { numRuns : 10000 }
    //use this if you need to control the seed for the random number generator
    //const config = { numRuns : 10000, seed:123 }
    const property = fc.property(fc.bigIntN(32)
            , fc.bigIntN(32)
            , fc.bigIntN(32)
            , transferProperties)
    fc.assert(property,config)
})

I created the quick-n-dirty example like this:

mkdir fast-check-example
cd fast-check-example
npm init --yes
npm install typescript ts-node
echo "{}" > tsconfig.json
npm install --save-dev jest ts-jest @types/jest
npm install --save-dev fast-check
mkdir src
#Start vscode
code

The first time I ran the test, it detected a defect: The code allowed transferring zero amounts:

Property failed after 1 tests
{ seed: -1444529403, path: "0:2:0:0:1:0:5:1:3:0:0:1:1:0:1:2:2:0:0:0:0:0", endOnFailure: true }
Counterexample: [0n,0n,0n]
Shrunk 21 time(s)
...stack trace to relevant expect() line in code ....

After fixing that bug it detected another defect: Transfers succeed even when the source account’s balance is insufficient:

Property failed after 4 tests
{ seed: 1922422813, path: "3:0:0:1:0", endOnFailure: true }
Counterexample: [0n,0n,1n]
Shrunk 4 time(s)
...