Author Archives: Gerben

Comparing Playwright to Selenium

Playwright is a library for controlling web browsers similar to Cypress, Nightwatch, Selenium etc etc. Its modern, advanced and fast! Especially compared to Selenium and Selenium Grid from providers such as BrowserStack and SauceLabs.

Playwright supports videos, console and network logs out of the box. Even for headless browsers.

You can work with downloads, browser’s network-stack and console.

It has convenient ways of locating elements and easily combines different locator types into 1 locator. Selenium has ByChained, but is more cumbersome.

It automatically waits for elements to be actionable, while Selenium requires the tester to use constructs like: Wait.Until(ElementIsClickable()).Click()

Playwright does way less DOM access than Selenium. Here’s an somewhat extreme example to show the difference. If you do this in Selenium, then for each row, it will query the DOM to return a WebElement for the checkbox in the row.

var rows = FindElements(By.ClassName("table-row"))
foreach(var row in rows)
{
   var checkbox = row.FindElement(By.ClassName("checkbox"))
}

Playwright won’t query the DOM for the checkbox. It returns a new locator (equivalent to Selenium’s By class) derived from the row web element to find that specific check-box.

Runs headless in CI/CD pipelines but still delivers video recordings and logfiles.

Although most tutorials use the default Playwright test runner, its works great with TypeScript and cucumber-js.

Using defineParameterType() with multiple regexes in cucumber-js

In a previous post I showed how to automatically replace parameter values before they get passed to your step definition code.

Now lets say you want that replacing/parsing/transformation for single- and double-quoted string like this in your feature file

When I print "Hello world!"
When I print 'its a beautiful day!'

Well…you’re in luck! The defineParameterType() method allows you to pass an array of regexes. We can use that to support both single- and double quoted strings with the same transformation function.

There’s a big gotcha here though. The the docs say this about the transformation function

A function or method that transforms the match from the regexp. Must have arity 1 if the regexp doesn’t have any capture groups. Otherwise the arity must match the number of capture groups in regexp.

In other words, when you use an array of regexes or if your regex has multiple capture groups, the function must have the same number of parameters as the total number of capture groups of all regexes in the array.

When cucumber calls the function, each parameter contains result from the corresponding capture group in the regex/array. If a regex does not match then cucumber passes an undefined value to the corresponding parameter number of the transform function So you’ll have to check each element if its undefined/null or not before using it.

defineParameterType({
    regexp: [/'([^']*)'/, /"([^"]*)"/],
    transformer: function (singleQ, doubleQ) {
        return singleQ ? replacePlaceholders(singleQ) : replacePlaceholders(doubleQ)
    },
    name: "a_string_with_replaced_placeholders"
});

Automatic type conversion of parameters in SpecFlow

In a previous post I showed that SpecFlow can change values of parameters. This mechanism is not just limited to transforming content of string values. It can also convert the literal string in your feature file to some complex object for your step-definition code.

Let say in your feature file you want to write steps like this:

Scenario: MyScenario
    When I print this list 'A,B,C,D'

Then you might be tempted to convert the string 'A,B,C,D' to a list like this

[Binding]
public class MyBindings
{
    [StepDefinition(@"I print this list '([^']*)'") ]
    public void PrintList(string input)
    {
        var items = input.Split(',')
        foreach(var item in items)
        {
            Console.WriteLine(item)
        }
    }
}

Don’t do this…it causes the same problems as mentioned in the previous post.

Instead SpecFlow’s Step Argument Conversion lets us simplify our step definition code to this:

[Binding]
public class MyBindings
{
    [StepDefinition(@"I print this list '([^']*)'") ]
    public void PrintList(IEnumerable<string> items) 
    {
        foreach(var item in items)
        {
            Console.WriteLine(item)
        }
    }
   
    [StepArgumentTransformation]
    public IEnumerable<string> TransformStringToList(string input)
    {
        return input.Split(',');
    }
}

Automatic parsing of parameters in SpecFlow

Most implementations of cucumber can automatically replace parameter values with some run time value. Let say in your feature file you want to write steps like this:

Scenario: MyScenario
    When I print 'Hello from test case {test}' 
    When I save the value '{test}' to my database

Then you might be tempted to write these bindings (dont do this…keep reading!)

[Binding]
public class MyBindings
{
    [StepDefinition(@"I print '([^']*)'") ]
    public void PrintMessage(string Message)
    {
        string Name = ....
        Message = Message.Replace("{test}",Name)
        Console.WriteLine(Message);
    }

    [StepDefinition(@"I save '([^']*)' to my database") ]
    public void SaveToDatabase(string Message)
    {
        string Name = ....
        Message = Message.Replace("{test}",Name)
        Console.WriteLine(Message);
    }
}

This way of writing step definitions becomes a real problem as the size of your automation increases:

we have to repeat the same logic in every step definition.
Its easy to forget to repeat that code.
New placeholders need you to find and update all places where this kind of parsing is done.
Its code that’s needed but not relevant for the objective of the step definitions.

Luckily SpecFlow’s Step Argument Conversion helps us! We can simplify our binding code to this:

[Binding]
public class MyBindings
{
    [StepDefinition(@"I print '([^']*)'") ]
    public void PrintMessage(string Message) => Console.WriteLine(Message);

    [StepDefinition(@"I save '([^']*)' to my database") ]
    public void SaveToDatabase(string Message) => Console.WriteLine(Message);
    
    [StepArgumentTransformation]
    public string TransformParsedString(string input)
    {
        string Name = ...
        return input.Replace("{test}",Name);
    }
}

Other implementations of cucumber also implement this principle. Its frequently referred to using these buzzwords:

Transforms / Transformations
Parsing / Replacing
StepArgumentTransformation

Automatically replacing/transforming input parameters in cucumber-js

Most implementations of cucumber provide a mechanism for changing literal text in the feature file to values or objects your step definition code can use. This is known as step definition or step argument transforms. Here’s how this works in cucumber-js.

Assume we have this scenario:

Scenario: Test
    When I print 'Welcome {myname}'
    And I print 'Today is {todays_date}'

And we have this step-definition.

defineStep("I print {mystring}", async function (this: OurWorld, x: string) {
    console.log(x)
});

Notice the use of {mystring} in the Cucumber expression

We can use defineParameterType() to automatically replace all placeholders.

defineParameterType({
    regexp: /'([^']*)'/,
    transformer: function (s) {
        return s
            .replace('{todays_date}', new Date().toDateString())
            .replace('{myname}', 'Gerben')
    },
    name: "mystring",
    useForSnippets: false
});

You can even use this to for objects like so:

defineParameterType({
    name: 'color',
    regexp: /red|blue|yellow/,
    transformer: s => new Color(s)
})

defineStep("I fill the canvas with the color {color}", async function (this: OurWorld, x: Color) {
    // x is an object of type Color
});

When I fill the canvas with the color red

How to dump the state of all variables in JMeter

To see the state of the variables and properties at a specific point in the test, you add a Debug sampler. This sampler dumps the information as response data into whatever result listener are configured.

If need the information in your own code to make decisions then you can use the following snippet of JSR223 code in a sampler or post processing rule:

import java.util.Map;
for (Map.Entry entry : vars.entrySet().sort{ a,b ->  a.key <=> b.key }) {
	log.info entry.getKey() + "  :  " + entry.getValue().toString();
}
for (Map.Entry entry : props.entrySet().sort{ a,b ->  a.key <=> b.key }) {
	log.info entry.getKey() + "  :  " + entry.getValue().toString();
}

How to hide the caret cursor in Selenium

If you are using Selenium to take screenshots of your user Interface, then is wise to hide the cursor. Use the WebDriver to execute the following snippet of JS code and voila!

{ var el = document.activeElement; el && el.blur(); }

Migrating from Visual Studio load tests to JMeter

Microsoft recently announced:

Our cloud-based load testing service will continue to run through March 31st, 2020. Visual Studio 2019 will be the last version of Visual Studio with the web performance and load test capability. Visual Studio 2019 is also the last release for Test Controller and Test Agent

The time has come to find other technologies for load testing. JMeter is one of the alternatives and in this article I show how the various concepts in Visual Studio map to it.

Visual Studio concept	JMeter equivalent
Web requests	Samplers -> HTTP Request
Headers of web requests	Config -> HTTP Header Manager
Validation rules	Assertions
Extraction rules	Post Processors
Conditions / Decisions / Loops	Logic Controllers -> If, Loop and While controllers
Transactions	Logic Controllers -> Transaction Controller
Web Test	Test Fragment
Call to Web Test	Logic Controllers -> Module Controller
Context parameters	User Defined Variables along with the syntax `${myvariable}` wherever the value of the variable is needed
Data sources	Config Element -> CSV Data Set Config
Virtual users, Load patterns and duration	See the settings of the Thread Groups
Credentials	Config Element -> HTTP Authorization Manager
Web Test Plugins	Although its possible to write JAVA plugins, its probably easiest to add a JSR223 Sampler with a snippet of Groovy code inside a Test Fragment or Thread Group
Request plugins	Same here, except use a JSR223 Pre- or Post Processor

Free SSL for machines in your private network

If you are running servers in your private network that need SSL, you can use LetsEncrypt and Certbot to automatically obtain and renew certificates for free. Even if your machines are not accessible from the internet.

What you need:

A static IP in your internal network for the server, like 192.168.1.2
Own a domain like “example.com” In this post I assume the server is accessed using server.example.com
Certbot’s support for the nameserver of the domain. Even if you purchased your domain at some unupported provider, its usually no cost to change to a supported nameserver. In this post I am using Cloudflare

How to set it all up:

Get a global API key from Cloudflare and remember it.

Create /root/.secrets/cloudflare.ini and put the following content into it:

dns_cloudflare_email = "<mailadres of your cloudflare account>"
dns_cloudflare_api_key = "<the api key you remembered earlier>"

Ensure only root can read the directory and file

sudo sudo chmod 0700 /root/.secrets/
sudo chmod 0400 /root/.secrets/cloudflare.ini

Install Certbot and the plugins it needs to talk to Cloudflare. For my environment this boiled down to:

sudo apt-get install certbot -t stretch-backports
sudo apt-get install python3-certbot-dns-cloudflare -t stretch-backports

Tell Certbot to obtain a free certificate for server.example.com

sudo /usr/bin/certbot certonly \
    --dns-cloudflare \
    --dns-cloudflare-credentials /root/.secrets/cloudflare.ini \
    -d server.example.com \
    --preferCed-challenges dns-01

Voila! You now have a certificate stored in /etc/letsencrypt/live/server.example.com/fullchain.pem

Dealing with renewals:

Certificates from LetsEncrypt have a short expiry time, so we need to renew it before it expires. We don’t want to have to think about doing this, we want this to be automatic. A simple crontab entry solves that.
```
14 5    * * *   root    /usr/bin/certbot renew --quiet > /dev/null 2>&1
```

Doing something with the SSL Certificate:

After Certbot has obtained or renewed a certificate it executes scripts located in /etc/letsencrypt/renewal-hooks/post/
In my case I am running Ubiquity’s Unifi controller software and use this script to deal with the renewal:

#!/bin/sh
DOMAIN=unifi.example.com

# Backup previous keystore
cp /var/lib/unifi/keystore /var/lib/unifi/keystore.backup.$(date +%F_%R)

# Convert to PKCS12 format
openssl pkcs12 -export \
    -inkey /etc/letsencrypt/live/${DOMAIN}/privkey.pem \
    -in /etc/letsencrypt/live/${DOMAIN}/fullchain.pem \
    -out /etc/letsencrypt/live/${DOMAIN}/fullchain.p12 \
    -name unifi \
    -password pass:unifi

# Install certificate
keytool -importkeystore \
    -deststorepass aircontrolenterprise \
    -destkeypass aircontrolenterprise \
    -destkeystore /var/lib/unifi/keystore \
    -srckeystore /etc/letsencrypt/live/${DOMAIN}/fullchain.p12 \
    -srcstoretype PKCS12 \
    -srcstorepass unifi \
    -alias unifi \
    -noprompt

#Restart UniFi controller
service unifi restart

A real world example of digital signature checking

In this post we will see exactly how we can check if a SSL certificate hasn’t been tampered with.

We will use https://google.com as an example and we’re manually going to check that the certificate’s digital signature is valid. Other important steps such as traversing the entire chain is beyond the scope of this simple example. Certificates don’t remain valid forever, so today you will get different ones. For sake of reproduction. I’ve included the ones I used later on in this post.

When I browsed to Google, it returned 2 certificates to my browser:

Its own certificate
The certificate of the intermediate CA that signed Google’s certificate

We’re going to use the following approach to check the signature on Google’s certificate:

Retrieve the digital signature included in Google’s certificate.

Retrieve the intermediate CA’s public-key from the CA’s certificate.

Decrypt the digital signature in Google’s certificate using the public-key from the intermediate CA. Now we have the hash value that the intermediate CA calculated at the time when it signed Google’s certificate.

Calculate the hash value of Google’s certificate ourself

Compare the two hash values. If they are the same, then Google’s certificate has not changed since it was signed and therefore we consider it to be valid

Retrieve the signature from Google’s certificate

Google’s certificate is listed further on in this post. Its in the PEM format which is just a base64 encoded representation of a X.509 certificate. I decoded it back into ‘plain old’ bytes and then I had the ASN.1 DER encoded version of the certificate. Using an ASN.1 viewer I can see that the entire X.509 file has the following structure.

SEQUENCE(3 elem)
    SEQUENCE(8 elem) <-- Google's part of the certificate. It contains 8 things, which I'm not showing here
    SEQUENCE(2 elem) <-- 2 elements that say which algorithm the intermediate CA used to sign Google's part of the certificate. Its a SHA1 with RSA encryption
    BIT STRING(2048 bit) <-- Intermediate CA's signature

So the last 2048 bits (256 bytes) contain the signature of the certificate. Below is the hex representation of those bytes:

348B7D645A64085B1FF6D86DF35480F9D913EADB09210B7E7402B7779F730077C7C7926A7A953DCD814C35E30608C02586A220795F965AF0E97F3CE5C32E7234FD6259782E447BFF73F6319797CA8DB1EB8D0A58119FB0794EF83ACCD8E45895C91FDCA97BB82FB425811E8A4CF0D41594618A5663BF774AC9CE2DBB9798E6E5BB6C5CCEC68B80D93E8C6748394B3822DE437C4FB93BCF302723ACD4D9ECAC75FFA4993D559C12C2E17228AC917942B1666D9948C6C42FAD1B0EB8F78AB0B38A5B392F85E7BDBFE97FD7534269CBB8FE22B03EF305514668DCE491683B1DD6852DBEE9C21E9C9E955B41E7078ACB722B2555CECBDEAD60AEC4FDC1C9A9686BE8

By the way. If you're doing these steps too and using an ASN.1 viewer, you might have noticed that I skipped the first byte of the contents. That's because its a BITSTRING and the following quote from the ITU-T X.690 specification implies that the content starts with a byte thats not really part of the content

The initial octet shall encode, as an unsigned binary integer with bit 1 as the least significant bit, the number of unused bits in the final subsequent octet. The number shall be in the range zero to seven.

Retrieve the intermediate CA's public-key from the CA's certificate

The CA's public-key is stored somewhere in the the middle of its certificate (not Google's certificate). Here I used the same trick of using an ASN.1 viewer to figure out which part of the ASN.1 contained the key.

The modulo is

009C2A04775CD850913A06A382E0D85048BC893FF119701A88467EE08FC5F189CE21EE5AFE610DB7324489A0740B534F55A4CE826295EEEB595FC6E1058012C45E943FBC5B4838F453F724E6FB91E915C4CFF4530DF44AFC9F54DE7DBEA06B6F87C0D0501F28300340DA0873516C7FFF3A3CA737068EBD4B1104EB7D24DEE6F9FC3171FB94D560F32E4AAF42D2CBEAC46A1AB2CC53DD154B8B1FC819611FCD9DA83E632B8435696584C819C54622F85395BEE3804A10C62AECBA972011C739991004A0F0617A95258C4E5275E2B6ED08CA14FCCE226AB34ECF46039797037EC0B1DE7BAF4533CFBA3E71B7DEF42525C20D35899D9DFB0E1179891E37C5AF8E7269

There are 2 odd things about this modulo. I know that its a 2048 bit / 256 byte key. However I have 257 bytes. You might think that we're running into that BITSTRING thing again here, but that's not the case as the ASN.1 tag specifies that the modulo element is an INTEGER. Whats really going on is that the RSA modulo is a 2048 bit unsigned number and that's serialized with an extra leading byte to indicate that its unsigned.

The exponent is:

01 00 01

Decrypt the signature from Google’s certificate

We know the intermediate CA's public key and we know the bytes that contain the signature of the certificate. So now we can do an RSA decyption on those bytes and voila, we will have the hash that the intermediate CA calculated during the signing process.

I used the following snippet of Python to do this. But most languages should be able to do this:

#Decrypt the signature from the certificate using the intermediate CA's public RSA key
modulo    = 0x009C2A04775CD850913A06A382E0D85048BC893FF119701A88467EE08FC5F189CE21EE5AFE610DB7324489A0740B534F55A4CE826295EEEB595FC6E1058012C45E943FBC5B4838F453F724E6FB91E915C4CFF4530DF44AFC9F54DE7DBEA06B6F87C0D0501F28300340DA0873516C7FFF3A3CA737068EBD4B1104EB7D24DEE6F9FC3171FB94D560F32E4AAF42D2CBEAC46A1AB2CC53DD154B8B1FC819611FCD9DA83E632B8435696584C819C54622F85395BEE3804A10C62AECBA972011C739991004A0F0617A95258C4E5275E2B6ED08CA14FCCE226AB34ECF46039797037EC0B1DE7BAF4533CFBA3E71B7DEF42525C20D35899D9DFB0E1179891E37C5AF8E7269
exponent  = 0x010001
signature = 0x348B7D645A64085B1FF6D86DF35480F9D913EADB09210B7E7402B7779F730077C7C7926A7A953DCD814C35E30608C02586A220795F965AF0E97F3CE5C32E7234FD6259782E447BFF73F6319797CA8DB1EB8D0A58119FB0794EF83ACCD8E45895C91FDCA97BB82FB425811E8A4CF0D41594618A5663BF774AC9CE2DBB9798E6E5BB6C5CCEC68B80D93E8C6748394B3822DE437C4FB93BCF302723ACD4D9ECAC75FFA4993D559C12C2E17228AC917942B1666D9948C6C42FAD1B0EB8F78AB0B38A5B392F85E7BDBFE97FD7534269CBB8FE22B03EF305514668DCE491683B1DD6852DBEE9C21E9C9E955B41E7078ACB722B2555CECBDEAD60AEC4FDC1C9A9686BE8
IntermediateCAsHash = pow(signature, exponent, modulo)
bytesOfHash = IntermediateCAsHash.to_bytes(sys.getsizeof(IntermediateCAsHash),byteorder='big', signed=False)
print ( "%s" % ''.join(format(x, '02X') for x in bytesOfHash ))

Running this code, gave me the following output ( I manually added line breaks, so remove them if you ever copy/paste this somewhere):

00000000000000000000000000000000
00000000000000000000000000000000
00000000000000000001FFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
FFFFFFFF003021300906052B0E03021A
05000414F8F3D8AACF7E27B2F66A2231
C3240682A15ADFF6

The 0000...1FFF...FF00 part is an RSA Encryption Block Type 1 from the PKCS#1 standard and isn't really part of the data that the intermediate CA wanted to encrypt. We can ignore it and focus on the 3021300906052B0E03021A05000414F8F3D8AACF7E27B2F66A2231C3240682A15ADFF6 part. This part is an ASN.1 DER encoded data-structure defined in RFC2313 as:

DigestInfo ::= SEQUENCE {
     digestAlgorithm DigestAlgorithmIdentifier,
     digest Digest
}
DigestAlgorithmIdentifier ::= AlgorithmIdentifier
Digest ::= OCTET STRING

The AlgorithmIdentifier is defined in RFC 5280 as

AlgorithmIdentifier  ::=  SEQUENCE  {
    algorithm               OBJECT IDENTIFIER,
    parameters              ANY DEFINED BY algorithm OPTIONAL
}

So this means we should get:

SEQUENCE(2 elements)
    SEQUENCE(2 elements)
        OBJECT IDENTIFIER
        NULL (see RFC2313)
    OCTET STRING

And indeed when we use the ASN.1 decoder we get the following output:

SEQUENCE(2 elem)
    SEQUENCE(2 elem)
        OBJECT IDENTIFIER 1.3.14.3.2.26 sha1(OIW)
        NULL
    OCTET STRING(20 byte) F8F3D8AACF7E27B2F66A2231C3240682A15ADFF6

So, now we know that the hash value calculated by the intermediate CA is

F8F3D8AACF7E27B2F66A2231C3240682A15ADFF6

Calculate the hash value of Google's certificate ourself

Now we are going to repeat the same hash calculation that the intermediate CA did a long time ago. We will:

Need to extract the bytes that represents Google's part of the certificate.This may NOT include any of bytes that hold the digital signature itself.
Run a SHA1 hash calculation on it.

The following python code does that and when I run it, it prints

F8F3D8AACF7E27B2F66A2231C3240682A15ADFF6

So we conclude that Google's certificate has not been tampered with!

import base64
import hashlib
    
def showSha1HashOfCertificate(bashe64EncodedCert):

    #Before doing the base64 decoding, we need to remove the 1st and last lines
    certificateWithoutCommentLines = bashe64EncodedCert.replace("-----BEGIN CERTIFICATE----","").replace("----END CERTIFICATE-----","")
    bytesOfCertificate =  base64.b64decode(certificateWithoutCommentLines)
    
    #The hash is calculated over the bytes that resulted from DER encoding the part that the X.509 specs
    #refer as the 'tbsCertificate' field of the entire certificate. 
    #Using the ASN.1 viewer I see that the tbsCertificate (the first member of the sequence) starts at offset 4 and its length is 4 + 1453 bytes     
    bytesOftbsCertificatePart = bytesOfCertificate[4: 1461]
    sha1Hasher = hashlib.sha1()
    sha1Hasher.update(bytesOftbsCertificatePart)
    ourHash = sha1Hasher.digest();
    print ("%s" % ''.join(format(x, '02X') for x in ourHash ))

googlesBashe64EncodedCert = """
-----BEGIN CERTIFICATE-----
MIIGxTCCBa2gAwIBAgIIVGohyFSBd4owDQYJKoZIhvcNAQEFBQAwSTELMAkGA1UE
... I removed a lot of the lines for brevity
rsT9wcmpaGvo
-----END CERTIFICATE-----
"""

showSha1HashOfCertificate(googlesBashe64EncodedCert)

The certificates

Below is the certificate for Google (its a big one!)

And here we have the certificate of the intermediate CA that signed the above certificate:

Gerben’s blog

Musings on software development and quality assurance