Guest Blogger: Software Assurance: How can you tell?

Today we have a guest blogger – Steve Lipner, Partner Director of Program Management, Trustworthy Computing Security, Microsoft Corporation. Steve discusses security assessment of products by IT buyers.

Individuals and organizations often ask us how they can tell how secure a specific product or a vendor’s product family really is or how they can ensure that they are acquiring secure software. We’ve seen some organizations propose to take the direct approach by posting a procurement requirement that software be free from all vulnerabilities. As our companion blog by Eric Baize of EMC shows, that’s a completely impractical requirement. Real software does have vulnerabilities. The question is how good a job the developer does at minimizing their prevalence and effect.

As SAFECode members, we are confident that secure software results when the developer follows a process that’s designed to produce secure software. We all apply such processes, and our experience shows that they result in fewer vulnerabilities and reduced severity and increased difficulty of exploitation of the vulnerabilities that remain. Our SAFECode documents describe such processes in a general way – individual members apply processes that are tailored to their specific technologies and product offerings. But all share the elements that are reflected in our “Fundamental Practices” papers. And all are applied consistently: a secure development process only works if the developer actually implements it across the product family.

One effective way to determine whether your suppliers are building secure software is to ask them about their secure development process. They should have an answer, and ideally the answer should be definitive and specific: “we do this, we require that” rather than “we could do this and we might do that.” A supplier might be giving false answers, but if that’s the case, there are bigger problems.

What about third-party evaluations of secure development processes? Some member countries of the Common Criteria Mutual Recognition Agreement made an attempt a few years ago to develop a new version of the Common Criteria that would be based on evaluation of vendors’ development processes. They found that defining evaluation criteria that could be applied reliably by multiple evaluation organizations was very difficult and abandoned the effort. While there may be certifications for developer processes in the future, they’re likely to be high-level: they’ll tell you that the vendor has a process and actually applies it, but they’re not likely to say a lot about the details of the process or about how vendors’ processes compare.

We’ve also seen proposals for assessments of software security by consultants or contractors independent of both developer and customer. These assessments are often based on the use of tools that purport to tell how secure a piece of software is or isn’t. Some of these tools operate on the finished software product – and thus they’re limited in how much insight they can apply, while others assume access to vendor source code – and thus assume a lot of access to vendor intellectual property and an improbable level of compatibility with vendor development environments and tool suites. Unfortunately, there is no “magic tool” for measuring software security.

Comparing published vulnerability rates is another approach to telling “how secure” – and all of us in SAFECode pay attention to our external vulnerability experience because it’s an indication of how our process is working. But it’s hard to compare vulnerability rates – they may be indicating how easy it is for security researchers to get access to a product, how “popular” that product is with the researchers rather than how secure it is or isn’t. A better vulnerability-related metric to watch is how effectively the vendor responds to reported vulnerabilities: does the vendor accept reports and issue updates, or “blame the messenger” by condemning the security researchers who report problems.

It’s not easy to tell how secure a vendor’s software is, and anyone who offers an easy way to answer that question probably doesn’t “get it” – or is trying to sell something. From the perspective of the SAFECode members, the best way to understand the security of a vendor’s software is to review the vendor’s secure development process for technical soundness and clear commitment. If the vendor is clearly following a documented proven process that aligns with the SAFECode “Fundamental Practices for Secure Software Development”, that’s a good sign.