Tag Archives: pbt

Generating custom random inputs for your property based test in golang

Golang’s quick testing package is great for generating random input for your property based test. Sometimes you want to control how the input values are generated. Here’s how you can create a custom generator for the input parameters to your property based test.

Lets assume you want to test the function IsValid(string) with many random inputs …. but …. the input may only contain characters in range a-z, A-Z, and 0-9. Here’s how to do it:

func randomAlphaNumericString(output []reflect.Value, rnd *rand.Rand) {
    alphabet := []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789")
    size := rnd.Intn(8192)
    v := reflect.New(reflect.TypeOf("")).Elem()
    v.SetString(randStringOfLen(rnd, size, alphabet))
    output[0] = v

//Generates a string of len n containing random characters from alphabet
func randStringOfLen(rnd *rand.Rand, n int, alphabet[]rune) string {
    b := make([]rune, n)
    for i := range b {
        b[i] = alphabet[rnd.Intn(len(alphabet))]
    return string(b)

func TestMyMethod(t *testing.T) {
    propertyTest := func(input string) bool { return true == IsValue(input) }
    c := quick.Config{MaxCount: 1000, Values: randomAlphaNumericString}
    if err := quick.Check(propertyTest , &c); err != nil {

Lets break it down into steps:

Firstly we define our propertyTest to check all input strings are valid. So far nothing special. This function takes a single string input parameter.

The quick.Config struct has the Values member. This member lets you supply a function to generate whatever parameters propertyTest needs. In our case the randomAlphaNumericString function does that job.

The randomAlphaNumericString function generates a suitable random string and it stores it as a reflect.Value in the output array at the position where the propertyTest expects to receive a string parameter.

TypeScript logo

Property based testing in TypeScript with fast-check

In a previous post, I explained property based testing. In this post we’ll see a simple example using fast-check

Let assume we’re building the same bank transfer code as described in the dotnet FsCheck earlier post. Here’s the TypeScript version of the test:

import BuggyBank from "../buggy-bank"
import * as fc from 'fast-check';

function transferProperties(startBalanceA: bigint
        , startBalanceB: bigint
        , amount: bigint) : void {

    const bank = new BuggyBank()
    bank.balanceOfA = startBalanceA
    bank.balanceOfB = startBalanceB
    try {
    } catch {
        //Transfer failed
        const balanceAUnchanged = bank.balanceOfA == startBalanceA
        const balanceBUnchanged = bank.balanceOfB == startBalanceB
    //Transfer succeeded
    const balanceAIsNonNegative = bank.balanceOfA >= 0
    const balanceAChanged = bank.balanceOfA != startBalanceA
    const balanceBChanged = bank.balanceOfB != startBalanceB

test('properties of a bank transfer must be correct', () => {
    const config = { numRuns : 10000 }
    //use this if you need to control the seed for the random number generator
    //const config = { numRuns : 10000, seed:123 }
    const property = fc.property(fc.bigIntN(32)
            , fc.bigIntN(32)
            , fc.bigIntN(32)
            , transferProperties)

I created the quick-n-dirty example like this:

mkdir fast-check-example
cd fast-check-example
npm init --yes
npm install typescript ts-node
echo "{}" > tsconfig.json
npm install --save-dev jest ts-jest @types/jest
npm install --save-dev fast-check
mkdir src
#Start vscode

The first time I ran the test, it detected a defect: The code allowed transferring zero amounts:

Property failed after 1 tests
{ seed: -1444529403, path: "0:2:0:0:1:0:5:1:3:0:0:1:1:0:1:2:2:0:0:0:0:0", endOnFailure: true }
Counterexample: [0n,0n,0n]
Shrunk 21 time(s)
...stack trace to relevant expect() line in code ....

After fixing that bug it detected another defect: Transfers succeed even when the source account’s balance is insufficient:

Property failed after 4 tests
{ seed: 1922422813, path: "3:0:0:1:0", endOnFailure: true }
Counterexample: [0n,0n,1n]
Shrunk 4 time(s)
Golang logo

Property based testing in golang with quick

In a previous prost, I explained property based testing. In this post we’ll see a simple example using golang’s built-in quick package.

Let assume we’re building the same bank transfer code as described in the dotnet FsCheck earlier post.

Here’s the golang equivalent of the test:

package goquickcheckexample

import (

func TestProperties(t *testing.T) {
	bank := BuggyBank{}
	properties := func(StartBalanceA int, StartBalanceB int, TransferAmount int) bool {
		bank.BalanceOfAccountA = StartBalanceA
		bank.BalanceOfAccountB = StartBalanceB
		err := bank.Transfer(TransferAmount)
		if err != nil {
			//Transfer failed
			balancesChanged := (bank.BalanceOfAccountA != StartBalanceA) || (bank.BalanceOfAccountB != StartBalanceB)
			if balancesChanged {
				t.Log("Balances changed on failed transfer")
			return !balancesChanged
		//Transfer succeeded
		balanceAIsNonNegative := bank.BalanceOfAccountA >= 0
		balanceAChanged := bank.BalanceOfAccountA != StartBalanceA
		balanceBchanged := bank.BalanceOfAccountB != StartBalanceB
		if !balanceAIsNonNegative {
			t.Log("Balance of A ended negative")
		if !balanceAChanged {
			t.Log("Balance of A did not change")
		if !balanceBchanged {
			t.Log("Balance of B did not change")
		return balanceAIsNonNegative && balanceAChanged && balanceBchanged

	c := quick.Config{MaxCount: 1000000}
	if err := quick.Check(properties, &c); err != nil {

Note: If you want all test runs to use the same set of random numbers then use: c := quick.Config{MaxCount: 1000000, Rand: rand.New(rand.NewSource(0))}

When I ran the test, it detected a defect: Transfers succeed even when the source account’s balance is insufficient:

bank_test.go:28: Balance of A ended negative
bank_test.go:41: #2: failed on input 6319534437456565100, -3125004238116898490, 8226184717426742479

After fixing that bug, it detected a defect: The code allowed transferring negative amounts:

bank_test.go:34: Balance of A ended negative
bank_test.go:47: #22: failed on input 5995030153294015290, -7891513722668943486, -3464545538278061921

While analyzing this defect we notice yet another one: This code is not safe against integer overflow.

Property based testing

Many test techniques rely on your assumptions where bugs are likely to be. In practice its really difficult to be complete and correct in your assumptions, so we end-up with defects in our product. For software used at scale, those defects are likely to cause real problems.

Property Based Testing avoids relying on assumptions. Instead it uses randomization to exhaustively check your product always meets some ‘property’ It often finds defects in the code as well as ambiguities in specifications.

Frequently used terminology

Term Meaning
Property Some fact or truth that always applies to your system. Some examples:

  • The total amount of money in a set of bank accounts is constant regardless of what transfers are executed.
  • The result of serializing and deserializing some object is identical to the initial object.
  • The result of squaring a number is zero or more.
Generator Code that generates random input suitable for your test. Some examples are:

  • Generate a random integer.
  • Generate a random string.
  • Generate an object whose members have random values suitable for their type.
  • Generate a HTTP GET request with a random url.
  • Generate a randomly ordered set of command BUT it never starts with command X
Shrinker Takes some generated input (usually one that caused a failure) and make it one size smaller. Test frameworks usually keep shrinking until they find the smallest input that still causes a failure. This really helps in analyzing the bug.

Are my tests still deterministic if we use randomization?

Yes, you control the source of randomness. As long as you use the same source with the same seed, tests on your machine run exactly the same as elsewhere.

What frameworks help me get this up and running?

Fscheck for DotNet.
quick (no Shrinking) or gopter (with Shrinking) for Golang.
fast-check for JavaScript / TypeScript.

Whats a good place to have these type of tests?

These tests need to run fast. You’ll frequently see them in the build/unit-test stages. Sometimes in the component- or integration test stages of a pipeline.