It's a fact of life that practically all the applications we use and develop today are in constant flux. Features are being added or tweaked, bugs are being fixed, and...ah, crap, that last update just created a bunch of new bugs, and reopened a couple that were already fixed. Here we go again.
We've gotten really good at the assembly line of product development, and regression testing is one of those things that make that possible. Every time your application's source code changes, you perform a regression test to check for new bugs and old, renewed bugs that could potentially bring your app to a grinding halt.
Ok, time for a quick 101 - Regression testing is essentially testing applications for bugs that were previously addressed in prior iterations before the next release. The main purpose is to ensure that defects that were fixed during previous releases, if regressed are identified for the development team to fix in time. It's also done to make sure that your recent changes behave as intended, and the changes did not inadvertently cause problems in functions that otherwise worked correctly before the changes.
However, regression testing historically has been largely restricted to the functional and performance aspects of an application. But changes in source code don't just affect the functional aspects of the application, they also affect its security posture.
Let's try to extend the concept of regression to security. What if previously identified automated and manual security vulnerabilities in an application are checked for regressions prior to release? Pretty cool, right?
It absolutely is! But if you want to know why, there's 2 reasons.
1. It magnifies the effect of security automation
As companies move towards DevOps and Agile frameworks, applications go through major code changes during its short development lifecycle. Plugging in security regressions along with an automated tool-chain and existing QA automation framework, pumps up your application’s test coverage by leaps and bounds.
The effect is that the app doesn't just functionally operate in the manner it was meant to, but also operate securely. This consequently minimises the risk of the application malfunctioning or experiencing downtime during production, reducing business loss and saving a ton of man-hours needed to fix the bugs if detected in later stages.
2. You don't need to manually test for bugs every single time
Pen-Testers don't just rely on automated tools to perform security sweeps, but spend valuable man-hours in dissecting deep-rooted logic flaws that can never be unearthed by point-and-click tools. Now, imagine if these vulnerabilities were to be manually checked prior to every major release.
Read more: How AppSec testing has changed over the years
Not only is this operationally inconvenient, but it's an absolute waste of manpower, plain and simple. Automated security regressions save critical man-hours of penetration testers that can be put to better use.
Convinced yet?
There are multiple reasons that stop companies from trying to do regression testing with security. Here's 3 of the biggest ones:
1. Security was not considered as a priority.
Functional defects are generally more observable and measurable than the security flaws. Let’s face it: realistically, how many companies out there would actually block a major release if there was a Medium severity bug that was found during a security sweep?
If the functionality faces issues, the potential costs are measurable, and you can directly address it as a critical part of the application. Security, on the other hand, is measured based on risk. If there's a flaw in the application, the first thing you ask yourself is, "What are the odds that someone will find it and try to exploit it?"
Read more: 10 questions you should ask yourself about your AppSec testing process
You can't really measure that, so why spend time and resources on something that may or may not affect the business operations of the company? Security is relegated to an accessory task instead of a priority. And since security wasn't a priority, 'security regression', and more so 'automated security regression' isn't something that companies think of integrating into their testing methodologies.
2. The misconception that ‘Security testing can only be performed by experts’
I mean, sure, until a few years ago, security testing was exclusively meant for experts. Developers lacked the tools and strategies to implement security during development, and AppSec testing was largely a manual (pentesting) and laborious process.
But things are different now. Today, automation technology (like Robot frameworks) can be used and extended to run security regressions. In addition, you can even write scripts that automate test cases as part of your existing CI/CD testing framework.
Check it out: Download we45's repository of free scripts and libraries
Even better, one could potentially re-use the entire stack of functional automation scripts to be used in a DAST automation pipeline. By utilising these tools and automation frameworks, you're not only improving the security aspect of your application but also reducing the pressure on the security teams with minimal additional effort.
3. Bridge the gap between QA and Security
Despite the availability of tools and technology, QA Engineering teams still need a certain level of understanding (knowledge and skill) of how application security works to be able to perform security regression. While they can utilise their functional regression knowledge for certain security automation scenarios, they still need to understand some key aspects of security such as threat modelling, how exploit scenarios work, and how to integrate security tools into CI/CD pipeline.
Learn more: Bringing Application Security closer to QA
While at first glance this may seem completely unrelated, just bear with me for a sec. In order to write security regression test-cases, QA engineers need to first understand how threat modelling works, and how to use it to write test cases. This knowledge can be applied to write security test cases for the purpose of integrating it into the CI/CD pipeline. QA engineers also need to understand how security testing scenarios work, at least on a conceptual level. This will help them understand how they can use security exploit scripts and walk-through scripts for security regression testing.
Furthermore, they require an understanding of how tooling works in the CI/CD pipeline. As much as your security vendors would love to help you with integrating regression testing into your testing frame so that you can plug-and-play, it isn't always reliable. Even a minor error may require you to reach out to your vendor repeatedly. With the understanding of tooling and how to integrate it into the CI framework, your QA engineers can change and adapt security regression testing as they see fits per your application development strategies.
Learn more: we45's unique approach to Threat Modelling
Fixing a security bug during production stage is ten times costlier than fixing it during UAT. By using security regression, security vulnerabilities can be found during the development cycle, thus enabling developers to fix them much earlier in the release cycle. This reduces the business impact of the application and greatly reduces the cost of fixing them on production as opposed to fixing them in the development cycle.
Due to the potential benefits from security regression testing, and the current availability of tools and automated frameworks, integrating security regression into your CI/CD pipeline is far more beneficial than to leave it to the experts or worse, ignoring it.
