The objective of load and stability testing is to determine if some System Under Test (SUT) is able to meet its performance and/or stability requirements. A secondary objective is to experiment how far we can push it before it breaks.
Some challenges to deal with are:
- How can we make sure the tests and the SUT use suitable data? You need some way of ensuring the SUT and test scripts have a common understanding of what data exists at start of the test.
- How can we deploy and start the SUT without its neighboring components being available or suitable for load testing?
- What about authentication blocking us from running automated tests?
- How can we test the impact of faults, caused by external components, when we don’t control those external components?
Here are some recipes ranging from complex to simple. None of these are perfect for all situations. Each has a best fit at some stage in the Software Development Life Cycle (SDLC)
End to end load tests
The entire product with all its software components, internal data and ‘customer’ data is deployed into some production like environment.
An example:
- Deploy all microservices onto a Kubernetes cluster.
- Use scripting to seed databases or use real-world (anonymized) databases and extract whatever content the tests need from those databases.
- Use tools and services such K6, JMeter, Locust, Gatling, Taurus, NeoLoad, BlazeMeter to generate load.
- Use your DevOps tooling to monitor and analyze the SUTs behavior.
Pros
Impact of resource limits, network set-up and caches are validated with this approach.
Unpredictable interactions between components become visible.
Impact of real world data set becomes visible.
Cons
Its costly and time consuming to arrange a suitable environment.
Deployment of components can be time consuming. Especially as the environment is likely to start in an undefined state due to previous test runs and deployments of feature branches.
Authentication and intrusion detection can block running these type of tests. Obtaining valid credentials might be impossible when Multi Factor Authentication is active..
Laws and regulations may block you from using real world data.
Bringing databases into a suitable starting state can be difficult and time consuming. Especially when the data is distributed over multiple services with own tech for data storage.
Component load/stability tests
A single component with its own internal data and fake, but large enough, ‘customer’ data is tested in isolation. It starts, gets tested and deleted.
A simple example is:
- Add a single micro service, the SUT, into a container image.
- Use frameworks such as Faker and Bogus to generate data for the SUT, mocks and repository databases.
- Add mock servers and their data into a container image. Something like WireMock or some simple custom built HTTP server.
- Use a real database server in a container to host your repository data.
- If you need to simulate intermittent faults on repository access, you can mock the repository classes to throw an exception x% of the time. In the sunny day scenario the mock simply forwards the request to the real repo class. Use frameworks such as Moq and Mockito to generate the mocks around the real repository classes.
- Add a load generator and the test scripts into the container.
- Configure the SUT to talk to the mock servers and repository / database server in the container.
- Run the container(s) as part of your CI/CD pipelines. Maybe using
docker-compose
on a single CI/CD agent or maybe in some temporary Kubernetes namespace.
If running everything in a few containers is not suitable, you can distribute the SUT, mocks, databases and load generator over multiple container hosts and/or external service providers. For example:
- Use providers of mocking tools to spin-up mock servers. Some providers are Mockoon, BlazeMeter Service Virtualization and many more.
- Same idea for database servers.
- Add the SUT into a container image and configure it to talk to the mock servers and databases.
- Start the SUT’s container image on some online platform (Kubernetes cluster, Azure Container Instances etc. etc.)
- Run the load test from some load test service provider like NeoLoad or BlazeMeter.
Pros
Having the SUT tested in isolation removes many blockers end-to-end tests suffer from. Its easier to build and maintain these tests. Less work to run and analyze them.
Tests act as an automated QA gate for a component. Failures are visible earlier in the SDLC.
Using mocks and custom repository classes makes it easy to control injection of faults . Its trivial to have a mock server or repository class simulate a failure in x% of the cases .
Cons
The SUT needs to be engineered in such a way that its possible to use your mocks, database-servers / repository classes. Legacy code might block this and need refactoring.
Usually you wont be testing with real resource limits that end-to-end test would run with.
Unit/integration test(s) running performance critical code in a tight loop
Use the unit test framework of the programming environment to run some critical code for a specific length of time or until a max number of iterations is reached.
An example would be to implement a NUnit/JUnit test to run some algorithm many times against various random data sets. Use frameworks such as Moq and Mockito to avoid depending on external systems and control what data the SUT gets back from the mocked dependencies. Use frameworks such as Faker, Bogus or Property Based test generators to generate the randomized inputs to the algorithm
Pros
Runs fast.
Does not depend on availability and state of external systems as they are mocked-away.
Integrates well with IDE’s.
Cons
Impact of interacting with external systems is not tested here. This will miss defects on topics such as connection pooling, buffering, latency and caching.
A tight loop is usually single threaded, it will miss defects caused by concurrency problems.
Failures during CI runs are hard to analyze as unit test frameworks are not generally geared to providing post-mortem analysis of time based metrics. Integration with a profiler of your programming language is likely to be needed.