What you need to know about software testing automation
A comprehensive collection of articles, videos and more, hand-picked by our editors
Rapid application updates are great, but only if organizations can easily test stability.
In the modern era of software development, companies all over the world compete to deliver products that evolve based on customer feedback. Amazon, Google, Microsoft and other big players are deploying changes and updates to their production applications weekly, and, in some cases, even multiple times per day.
By submitting your personal information, you agree that TechTarget and its partners may contact you regarding relevant content, products and special offers.
The key to making this possible is the implementation of a continuous delivery (CD) workflow, in which incremental changes are integrated into the application and proven ready for production at any given time.
One of the biggest challenges in adopting a CD model is putting in place a comprehensive series of tests. The test automation process allows teams to deploy changes to production with confidence that the releases are stable, even when coming at a rapid pace.
Automated testing in a CD workflow
When working in a continuous delivery model, code changes that include new features or services kick off a series of automated processes that validate the code and verify that it works properly in a production-like environment. These changes typically flow through a series of stages, also referred to as a pipeline. Automated tests within every phase are essential for teams to iterate quickly.
A typical test automation process for CD consists of the following:
Version control. Developers working on a production application make changes to code in a version-control repository. For example, a team of developers might be collaborating on a large project using a distributed version-control tool such as Git, and the code is available in a central location: a public or private GitHub repository. When it comes time to add a new feature or service to an application, a developer can modify the code, commit it to the master branch and then push those changes to the main repository shared with the other members of the team.
Automated build and unit testing. A continuous integration (CI) system can poll the repository for changes, and, when developers check in new code, an automated build runs to compile binaries and generate artifacts such as webpages, scripts and configuration files. Common CI systems include Jenkins, TeamCity and Travis CI. Automated unit tests are typically run during this stage using a test framework such as NUnit for .NET or JUnit for Java. Automated unit tests have no external dependencies, such as web servers, database servers or file systems. The entire CI build and unit test process runs quickly, so developers know right away if there is a problem.
Integration tests. With help from the CI system or another orchestration tool, the built code and artifacts deploy to either a new or existing environment for development and testing. One common approach is to implement infrastructure as code (IaC) during this stage to spin up an environment with all the application dependencies. This lets integration tests run to ensure that the application works once it's deployed. For example, an automated integration test can confirm that all of the components in the application, such as database access from a web application, function.
Functional/acceptance tests. Automated functional tests, also referred to as acceptance tests, can ensure that the application works for end users. Again, a stage within the pipeline can spin up a production-like environment to be used for staging or quality assurance. From there, an automated acceptance test -- using a framework such as Selenium -- can control a web browser and have it interact with a webpage to ensure the application behaves properly for users.
Performance tests. Automated load testing can ensure the application will handle anticipated activity, in a production-like staging environment. There are a number of open source and commercial load-testing tools available. Apache JMeter, an open source load tester, is a common choice, and many commercial products, including BlazeMeter and Hewlett Packard Enterprise's StormRunner Load, extend its functionality and provide integration with many CI systems.
Release to production. Once the team is satisfied with the results, it can deploy the new version of the application into the production environment.
There is a lot going on in this example. Any manual tasks required within this workflow would slow down the development process and make weekly or daily releases extremely difficult.
Benefits of the test automation process
Automated tests in a CD pipeline enable an IT team to move quicker, and the resulting software should be of higher quality.
Running automated builds and unit tests after every change to the application source code identifies problems right away. This allows the team to make fixes immediately and gives developers the chance to continue iterating on the application.
Automated tests can execute tasks in a fraction of the time that is takes to do them manually. Integration, acceptance and performance tests might take days or weeks if done manually. Automating these tests gives teams a competitive advantage, with more time for innovating and improving the application.
Without automated performance testing, it would be extremely difficult to simulate the load of a production application for the purpose of validating performance. This is especially true for cloud-based applications supporting thousands of users. These apps need to dynamically scale to meet the demands for unique usage patterns.
Also, applications developed with testing in mind tend to be more reliable. For example, test-driven software development patterns will ensure teams take a test-it-first approach to application architecture. The result will be an application that lends itself to automated testing and will likely implement a loosely coupled design that is easier to maintain and troubleshoot.
Automated CD patterns and test practices
There are a few patterns and practices you'll want to keep in mind as you implement a test automation process and build a CD pipeline.
First, be sure tests are included in your project's version-control repository. This will make them available to the continuous integration system for automation.
Place tests into separate containers or categories so they can be executed in the right order within the CD pipeline. This will make it easier to run faster tests early in the pipeline. For example, run unit tests first for quick feedback. Once the build and unit testing is complete, developers know their changes are valid; longer integration and performance tests can be executed in subsequent stages of the process.
To ensure predictability and reliability for an application, automate every stage in the CD pipeline. This is a best practice as long as the process or test you need to run can be automated in a reasonable amount of time and realistic amount of effort.
It's wise to settle on an automated testing approach so that everyone on the team knows how to proceed. For example, test-driven development requires that developers first write failing unit tests. Then they write code and refactor the application until the test passes. TDD typically results in an explicit and predictable code base.
Another approach is behavior-driven development, which requires teams to first define the behavior and specifications of the application before writing any code. BDD results in documentation based off existing specifications and a solid understanding of the intended behavior of the application.
Finally, acceptance test-driven development is a collaborative approach to writing tests. ATDD requires everyone on the team to understand each feature before any code is written, and the team's progress can be accurately measured until all tests are in a passing state.
Software development teams need to move fast to keep up with the rapid pace of innovation that is so common in today's technology landscape. Implementing a continuous delivery model with automated testing is a great way to achieve the same agility as some of the biggest software companies in the world.
To release new features and services rapidly, tap into the power of automation and foster a collaborative approach between development and operations teams. Still, be aware that there's no cookie-cutter approach. Every application is different and has its own unique set of requirements.
Mike Pfeiffer is an IT architect, consultant, writer and conference speaker with nearly 20 years of experience in the tech industry.
Getting started with container orchestration
Build automation moved to systems level
New automation product available with Chef