Technical application examples for automated security testing

In this article we have already looked at why investments in security testing are worthwhile and what the difference is between static and dynamic security testing (SAST & DAST). But how can automated security tests and scans be integrated into project cycles?

‍False negatives and false positives

For the success of security tests, the two metrics false positive rate and false negative rate are usually used. The false negative rate provides information on how many weaknesses an approach or a specific tool was unable to find. According to studies, a SAST approach can identify 50% of vulnerabilities. However, concrete results vary greatly here, depending on the tool and analysis approach used. The false positive rate, on the other hand, shows how many findings were incorrectly identified as weak points. SAST tools in particular are predestined for a higher false positive rate.

The results of automated security tests should always be evaluated and classified manually. For a comprehensive picture of an application's security, regular manual security checks (penetration tests) should also be carried out.

Before we go into specific tools for identifying vulnerabilities in your own code below, an important note: The exploitation of security gaps in third-party systems without prior permission is according to Section 202a StGB prohibited and punishable!

Example 1: Semgrep (SAST Approach)

‍Semgrep OSS is a free open-source application that enables static code analysis with relatively low configuration and setup costs. Semgrep can be run in automatic mode directly in the project folder without prior selection of the technologies and programming languages used. For its part, the tool analyses which test rules should be applied. As an example application, we use DVWA, which contains numerous security gaps for learning purposes.

Semgrep found 48 code points in our sample application which appear vulnerable and require manual testing. Each of these results is clearly presented with an excerpt of the code, the affected file and a link to further information. A look at the affected code shows that the above finding is actually a (critical) vulnerability that endangers both the confidentiality, availability and integrity of the data processed in our sample application. So we've just identified our first vulnerability using Semgrep without much effort! 🥳

However, Semgrep becomes particularly practical when it integrated in CI processes. Here, Semgrep can be configured so that only the current changes are analyzed. As a result, the analysis process does not take as long as a complete scan of the entire application source code. It also allows developers to specifically check their changes for potential security risks before they are actually used in the application.

Example 2: Dastardly (DAST Approach)

Dastardly by Portswigger is a scanner for vulnerabilities in web applications, which carries out tests dynamically and at the runtime of the application. Internally, the tool uses a modified Chrome browser to visit websites and carry out automated tests. As with Semgrep, using Dastardly is extremely simple: A dynamic vulnerability test can be started within a few moments using a preconfigured Docker container, which only needs to be handed over the initial URL and directory for the generated report:

The result of the scan is an XML report, which can be used as input to further tools:

As the excerpt from the Dastardly report above shows, the tool has identified a cross-site scripting vulnerability with high criticality. The excerpt was abridged for presentation reasons. In the original form, the report contains further information on how the vulnerability can be specifically reproduced, why and how the vulnerability can occur, and tips on how to fix it. Dastardly was also designed to be integrated in CI processes. Since Dastardly requires a running web application to perform a dynamic scan, integration is recommended at a later stage of the build process.

‍Conclusion

The false positive rate and false negative rate metrics are a central aspect of the success of automated tests. These metrics help to understand how many vulnerabilities are statistically overlooked by the methods used and how many are falsely reported as vulnerabilities. Tools such as Semgrep (SAST) and Dastardly (DAST) offer the opportunity to efficiently identify vulnerabilities while supporting development processes through their integration with CI/CD pipelines.

It is important to always manually check the results of automated tests and also carry out regular manual penetration tests. This ensures a comprehensive picture of an application's security. With the right combination of automated and manual testing, developers can significantly improve the security of their applications and identify and address potential risks early on.