DevOps Lifecycle Assurance
Traditional penetration testing tends to focus on systems that are already in place, either in development or production environments. The shorter dev cycles and continuous delivery demands of DevOps requires a different approach to security testing, verification, and assurance.
The earlier in the process that security can be independently tested and verified, the more secure the end product will be. Ideally, security issues and attack vectors should be considered before the first line of code is written.
It also demands expert knowledge of AWS, Azure, and Google Cloud environments, both for ensuring secure deployment, and also as part of governance so that any unsanctioned public facing deployments can be identified and managed appropriately. As it’s so easy to spin-up services and products to the cloud it’s not uncommon to find rogue instances.
Automation technologies also need close attention.
The DevOps lifecycle can be broken down like this:
- Plan
- Code
- Test
- Build
- Release
- Deploy
- Operate
- Monitor
Security measures should be implemented in each of these phases, and we work with clients to add value to these phases in the following ways.
1. Plan
Before code has been written, developers should be considering the types of security threats that may face the environment once it is deployed. Pen Test Partners can help improve security in this area by offering the following services:
- Table-top exercises using threat-intelligence to highlight the types of expected attacks.
- Secure coding workshops which can be used to share the lessons we have learnt from multiple engagements to ensure the same mistakes are not being made.
If developers can start thinking like an attacker before developing code, then they will ideally plan to build in the security features that will result in a more secure deployment.
2. Code
During code development, the client should be using automated security features such as those built into several integrated development environments to highlight security flaws in real-time. This will help prevent common mistakes such as hard-coding sensitive information, or common input validation issues.
With infrastructure as code being so prevalent, this phase of testing would target both application and infrastructure code to ensure that both the application, and it’s hosting environment are configured in accordance with security best practices.
Whilst peer-reviewing of code is highly recommended at this stage, we can provide an independent third-party review to increase confidence of the code base.
We use a mixture of static code analysis tools and manual code reviewing to ensure that your automated processes are finding most of the issues during development.
Manual testing will always go above and beyond the capabilities of automated testing however and is therefore recommended at this stage. This includes checking both the code base developed in house, and the use of any third-party scripts or plugins that may introduce security risks into the codebase.
3. Test
Once an application and its associated infrastructure has passed all the phases above, it is then ready to be deployed to the development or staging area for it to be tested.
Configuration reviews of the hosting environment, such as on-prem or cloud should be carried out to identify any areas in which security best practices have not been adhered to. Pen Test Partners have a robust cloud testing methodology in which all resources would be tested for security issues. On-prem environments are tested in line with our internal testing methodologies to identify security issues in client owned infrastructure.
Traditional penetration testing of the applications and services should be employed at this stage to verify that no security issues have managed to get through the phases above. It may be that business logic issues arise during this phase of testing, as these tend to be harder to find during code analysis alone.
Security issues may also be found during this stage with regards to integrations with other systems.
Testing in development or staging is highly recommended as this allows the consultants to cover a larger range of test cases, knowing that genuine users of the system will not be affected. Testing in production, a consultant will never run test cases that may remove information or cause a denial of service. A genuine attacker has not such limitations and therefore these types of attacks should be carried out during this phase.
4. Build
Once the application, service, and associated infrastructure has been securely planned, and coded, the initial build will take place. This generally involves deploying code to a central repository, running CI/CD (Continuous Integration/Continuous Delivery) pipelines to automatically test, compile, and run code, and then moving the output into a development environment.
There are many areas of security that need to be addressed during this phase, and no two CI/CD pipelines are the same. However, the following is an example of some of the services that can be offered during this phase:
- Reviewing of code repository to ensure it has been securely configured, and that users, roles, groups, and projects are all correctly secured, and that access tokens are limited in scope and have expiration times.
- Testing to ensure that systems used in automation such as runners or similar are well secured and cannot be compromised.
- Reviewing CI/CD pipeline scripts to ensure that they are fit for purpose, advising on further checks that could be implemented, and that data is handled securely.
Unit testing should also be carried out during this phase. In this, automated tests are run to ensure that the application and services operate as intended. For example, a test case may submit a numeric value to a service expecting such an input and monitor the result to ensure the function is working and returns the correct result.
Security unit testing should be carried out during this phase too. To use the same example, test cases should be written to submit a letter, a single quote, 100 megabytes of A characters, null characters, and any other input that might break application logic in a way that would be advantageous to an attacker.
Pen Test Partners can help by reviewing these unit tests and advising on common attack types that may be missed.
Any code failing these tests should then be returned to the coding phase to mitigate the issues and the coding practices used to fix the issue should be shared amongst the entire development team.
5. Release
Now that code has been tested at the code, build, and test phases, there should be enough confidence in the code quality and hosting environment to release into a live environment, however this phase is the step before that happens and should be considered near-live.
This is the client’s opportunity to ensure that everything is ready for deployment and that any last minute alterations required are identified, documented, and if changes are required the steps above are carried out again against these specific feature changes.
This phase should consider any recommendations made in the previous phases and ensure that any changes made as a result have not broken functionality.
We can advise on the changes made during this phase and any potential impacts on security. For example, changing a graphic on a web application may be not cause for concern, however changing the single sign on provider or backend database vendor may have an impact that would need to be investigated before deployment into the live environment.
6. Deploy
Now that all phases have been completed, the code can be deployed into the live environment.
All test cases run in the test phase should be carried out against live to ensure that the environment is well secured and ready for the first users
Results from the test phase can be retested in live to verify recommendations have been implemented correctly.
It is recommended that all applications, services, and infrastructure is fully tested as this is the environment that will be exposed and therefore has the highest risk of being attacked.
Traditional penetration testing as well as configuration reviews against all elements would be carried out to ensure confidence in all elements of the platform.
7. Operate
In this phase, the platform is live and in active use. It is now the responsibility of the client to ensure the ongoing security of the platform.
It is recommended that ongoing penetration testing is carried out at regular intervals to identify if the actual operation of the environment has introduced security vulnerabilities.
The frequency of testing should be dictated by the criticality of the platform and the risk management policies of the client
8. Monitor
Continual monitoring of the environment aids security as if attacks are carried out, these should be spotted and acted on in real-time.
Pen Test Partners can aid in this process by assessing monitoring capabilities and advising on the key metrics that may indicate an attack or compromise of the platform.
If suspicious entries are found in logs, we can advise on the legitimacy of the attack, and using threat intelligence dictate whether attacks are background noise or serious threats to the environment.
For example, an internet facing application will likely be attacked hundreds of times a day by automated scanners and scripts and may be no cause for concern, however monitoring needs to be able to quickly identify actual threats against the system so security measures can be developed to prevent an attack in real time.