The argument for testing in production
Accurate Attack Stimulation
One of the main goals of a penetration test is to provide a simulation of what a live attack would be without the risk and cost of a real security incident. Armed with that knowledge, your organization can take action to prevent future breaches.
Some pen testing projects are conducted in a closed environment. When testers aren’t given any information about the underlying system, including legitimate credentials, this is known as black-box testing.
This style of engagement is different from the grey box testing (which companies usually prefer) but is a more accurate simulation of an attacker coming in from the outside.
These types of tests have their time and place for web application APIs and even network tests.
If your organization has a mature Appsec program that’s ready for a scoped penetration test then you should test in production to get the most accurate attack simulation that you can.
Cost and Time Savings
You’d want to create the most secure software imaginable but your organization may not have the resources to make that happen.
Best practice dictates that security tests be done in a dedicated staging environment to remove any risk of impacting production.
Yet, not everyone has the workforce to align with every best practice immediately. Many organizations don’t have a separate environment dedicated to security testing.
Lower environments may easily become unstable due to frequent code changes from developers or due to testing from other quality assurance teams.
These instabilities can impede security and may make it difficult to reproduce findings.
In the world of DevOps and CI-CD pipelines, code enters production minutes. If your organization has limited resources for its security budget then testing in production can help you find issues without the cost of creating a separate testing environment.
The argument against testing in production
Complications with availability
The CIA triad is one of the foundational principles of cybersecurity. A secure system ensures that private information is confidential, preserves data integrity by ensuring information is accurate and complete and it’s available when it’s needed.
Pen testing activities such as some types of automated scanning can cause disruptions, especially in legacy systems. Some industries like healthcare, have systems that cannot experience any downtime or latency because of the real-time impact on patients.
Having an identical staging QA (Quality Assurance) environment would help ensure that none of the changes that have occurred within the pen test is reflected in a production system.
Complications with customer confidentiality
A regular pen test is important for Payment Card Industry (PCI) and Health Insurance Portability and Accountability Act (HIPAA) compliance. The goal of these standards is to safeguard customer information.
The standard practice for any cybersecurity company is to ensure that sensitive information like credit cards and healthcare records are purged from the pen testers systems after a test.
Trusting a third-party cybersecurity agency with your confidential data is an important consideration when choosing one.
If your organization wants to take an extra step to protect real customer data, you can perform a test in the staging environment, so as long as the data in the stage environment accurately reflects the type of data in production.
Shifting Right Instead of Shifting Left
Penetration testing is a specialized type of quality assurance testing. Functional Software is also secure software by definition, penetration testing focuses on finding specific types of bugs that impact the CIA Triad.
Testing in a QA environment than a production environment makes it easier for development and operation teams to roll out remediation.
Instead of rushing a hotfix through the pipelines into a production-facing system, teams can see how remediation works in context with other bug fixes and new features.
By doing your security in staging you also prevent regressions that would force teams to scramble to re-patch. You must make it a norm to do testing during the development life cycle and train developers to write secure code from the beginning.
This is what it means to shift left, if you want to mature your Appsec program beyond meeting compliance requirements then pen-testing earlier in a staging or development environment will help.
Key Takeaway
There are pros and cons of testing in a production environment versus in a staging or development environment.
We can safely perform a penetration test in production in most cases but you’d prefer to test in a stable stage or QA environment to protect the availability of production systems, the confidentiality of customers and to shift security left.
If you have a mature security program and need an accurate attack simulation, or if you’re just getting started and don’t have the resources for multiple environments, you should test in production.
