For the last few weeks, SAFECode has been discussing a number of government initiatives related to software security assurance. This is the first of several blogs that we will be publishing to share our perspective and recommendations for approaches that will help governments and other organizations gain confidence in the software that they acquire and use.
Since the release of Executive Order (EO) 14028 in 2021, SAFECode has been tracking the U.S. Government’s plans and statements about how the portions of the EO dealing with secure software development and supply chain security would be put into action. This brief paper provides a SAFECode perspective on attestation that we believe will be important for software development organizations, end users, and government policymakers.
As directed in the EO, the National Institute of Standards and Technology (NIST) updated its Secure Software Development Framework (SSDF) to conform to the requirements in the EO. The SSDF is the key document that defines the government’s requirements for secure software development and software supply chain security.[1] As directed by the EO, the Office of Management and Budget (OMB) issued Memorandum 22-18 directing government agencies to require their suppliers of software to attest that their development practices conform with the requirements in the SSDF. Also as required by the EO, OMB will move to formalize the requirements for attestation in the Federal Acquisition Regulations (FAR).
The SSDF is a framework, not a checklist. It identifies best practices for software security at a high enough level to be broadly applicable. This is a positive attribute of the SSDF: any software development organization can apply it. However, the price of that generality is that each software development organization must determine how best to implement the SSDF requirements given the specific software it develops. To take a concrete example, the SSDF refers to the use of static analysis tools to detect potential code-level vulnerabilities. A development organization must choose a particular static analysis tool that will be appropriate and effective for its chosen programming language(s) and platform. Next it must specify how to handle specific error reports or findings from the tool – which are “must fix” based on their having a low rate of false positives and a high correlation with potentially severe software vulnerabilities. The choices will be based on tests and the reputation of the tool and on experience of actual software vulnerabilities in the development organization’s and other organizations’ code.
A development organization’s attestation that their practices conform to the SSDF is a statement that the organization has done the analysis sketched above for each SSDF practice and task, and has integrated appropriate SSDF practices and tasks into its development lifecycle. The attestation also states that the development organization is ensuring that its development personnel are trained and are actually applying the tools and processes to the code it delivers. The evidence to support the attestation will vary based on the practice and may come from amongst other sources; databases, online documents for engineering staff, and test and other data in the development organization’s engineering system.
Creation of the evidence summarized above is a natural part of creating and executing a secure software development process: a vendor who fails to do the tasks outlined doesn’t really have a process. The artifacts that the process creates may be internal policies or procedures, bugs in a database, threat models, or source code. Some, such as Software Bills of Materials (SBOMs), may be public or sharable with customers under nondisclosure agreements, while others may be proprietary to the development organization. (Examples of the latter include artifacts that disclose development organization intellectual property or the details of proprietary tools or analysis techniques.) While artifacts will not be released to customers or the public in many cases, the development organization must retain all internally in order to execute their development process and to be able to maintain and enhance software. The organization’s attestation is a statement that the artifacts exist and align with the SSDF.
In specific cases, agencies may choose to integrate attestation as part of other security assessment regimes. Where the attestation becomes part of a regime that does an effective job of evaluating security, such as FedRAMP, this may be an effective approach. But we have seen proposals that standards compliance and/or third-party assessments might replace attestation to conformance with the SSDF. Such proposals are attractive to organizations looking for a “compliance box to check”, but we believe that they almost always provide vastly weaker assurance of software security than attestation. In the case of standards compliance, there are additional mappings (SSDF -> Standard -> Development organization’s standards compliance documents -> Development organization’s actual practices) that obscure what the development organization is actually doing. In the case of third-party assessments, only a very few assessors will have the competence to understand the development organization’s practices (are the tools and must-fix bugs the rights ones?) and their applications (were the error reports properly triaged and the correct bugs fixed?). In our experience (for example, with the International Common Criteria for Information Technology Security), assessors are likely to ask organizations to provide a document about their practices rather than getting their hands dirty by looking at tools, requirements, and code. Thus, we do not recommend substituting reliance on standards conformance or third-party assessments for development organizations’ attestation.
Although a development organization that attests to its conformance with the SSDF is not routinely subject to even the limited scrutiny associated with certification under a standard or a third-party assessment, attestation is far from a “free ride.” If a history of discovered vulnerabilities leads a government agency to believe that a development organization has released a misleading or false attestation, the agency can initiate an investigation under the Civil Cyber-Fraud Initiative[2]. Conducting an effective investigation requires highly qualified auditors who can understand the development organization’s tools, processes, and implementation at a detailed level. If such investigations are infrequent, it should be possible to find qualified personnel to conduct them. Thus, it makes sense for the government to centralize investigative practices and resources. If an investigation finds that an organization falsified an attestation, that organization can be subject significant penalties. The potential threat of an adverse investigation is likely to have a significant effect on ensuring that all development organizations take the SSDF – and software security – seriously.
[1] SAFECode has been involved in the development of the SSDF since its conception. For more details about the relationship between the SSDF and SAFECode’s guidance, see https://safecode.org/resource-publications/navigate-the-executive-order-14028-era-of-software-security/
[2] See Section 3.5 of the National Cybersecurity Strategy