OpenSSL Software Foundation, Inc.

FAQ - The I.G. 9.5 Issue

Last update 2013-03-14

Q1: What is the issue? Executive summary please.

From time to time the strange little world of FIPS 140-2 validation testing manages to conjure thrills and excitement from what is normally a very dull and boring topic. The latest such surprise is a new interpretation¹ of the "guidance" (mandatory requirement) in section 9.5 of the Implementation Guidance (I.G.) document, a key canon of the FIPS 140-2 scripture. The bureaucracy responsible for such interpretations is the CMVP.
This new interpretation mandates some very disruptive changes to software libraries like the OpenSSL FIPS Object Module ("FIPS module") and those derived from it (which is a large proportion of all validated software modules).
OSF will complete those private label validations we are currently working on, as best we can, but we are no longer accepting new private label validations for reasons that are discussed in more detail below. We are not optimistic that any new FIPS module that satisfies this new requirement, or an accompanying "FIPS capable" OpenSSL, will become available for the foreseeable future.
The existing OpenSSL FIPS Object Module 2.0 ("2.0 FIPS module"), validation certificate #1747, is unaffected by this new requirement and will continue to be available as a validated cryptographic module. We have already been contacted by several software vendors who had been planning on a private label validation and are instead sponsoring additional platforms for the 2.0 FIPS module.

Q2: We have a FIPS 140-2 validation currently in process. Are we affected?

If your formal test report submission has already occurred (on or before December 27, 2012) then you're in the clear. If not your validation is affected to an as yet unknown degree, and have presumably already been so informed by your test lab.
UPDATE 2013-03-14: Apparently there will be a transitional grace period of some sort with a formal report submission deadline of November 15, 2013. It is not yet clear that the terms of this transition are sufficient to allow completion of currently in-process validations without major recoding or retesting. These transitional validations will carry a caveat to the effect of "This module has no assurance of integrity in non-approved mode" that will have no practical impact on deployment or use of the module (the "assurance of integrity" is only of interest in the context of FIPS 140-2 in approved mode).
For our current private label validation customers caught in this mess, OSF will if necessary hack a custom version of the FIPS module to satisfy the new requirement, at our expense². However, that will take additional time (months perhaps) and will result in custom code that will no longer be compatible with official OpenSSL releases. For that we sincerely apologize; we'll do what we can to get something validated but it won't be the end result you were expecting and may not be usable for your original intended purposes. There is a substantial element of risk to any FIPS 140-2 validation as this latest incident illustrates with a vengeance.

Q3: Why is the new I.G. 9.5 requirement bad news?

This new interpretation requires that none of the function calls in our cryptographic library can return useful information until after the POST³ is performed. It also requires that this restriction must be enforced in the module, that is in the cryptographic library itself, and not merely stated as a condition to be satisfied by the calling application.
Sounds trivial, right ... just set a flag or something? If only that were so.
There are three inter-related issues that conspire to make this a nasty problem: 1) hacking the module, 2) validating the module, and 3) hacking the "FIPS capable" OpenSSL. All three are non-trivial undertakings that collectively are very unlikely to be accomplished in less than about one to two years, if ever.

Q4: What hacks to the FIPS module will be required?

First there is the problem that a library cannot choose which API functions are called or when they are called; that is decided by the calling applications. So, some mechanism must be used to disable all function calls. A simple global static variable won't work because the POST itself calls many of the same API functions, and if those functions are disabled the POST will see errors just as any external application would. The POST can't just flip the global static variable without introducing a race condition, as other threads could call those functions while the POST was executing, which is a no-no not only from a technical perspective (the code is no longer thread safe) but is also a FIPS 140-2 scripture violation.
Note that by "all" functions we assume the functions intended to be part of the external FIPS module interface, not all functions visible to the linker at build time. Since the FIPS module is static object code ("monolithic object module")⁴ it necessarily contains global symbols that could in principle be referenced by a calling application during the final link phase where the application executable file is produced. Disabling the interface functions only is at least theoretically possible; disabling all functions isn't possible even in principle, in any sort of platform portable way⁵ A mandate to disable all functions would effectively eliminate the possibility of validating software for platforms that don't support dynamic linking such as Apple iOS and many real-time operating systems.
For the POST to run while all functions are disabled we either need an elaborate locking scheme or public/private wrappers for all the necessary functions (many dozens to hundreds, see the FIPS module API summary in the last 50 pages or so of the FIPS module User Guide⁶ to get a feel for what that would entail). The FIPS module already has a horribly complicated scheme for avoiding symbol name conflicts; this would pile on yet more complexity.
All in all a very non-trivial proposition, just to prevent functions from working that are similar or identical to the uncontrolled implementations residing in the regular OpenSSL library⁷. A new validation and new OpenSSL (see below) would also be needed.

Q5: Will a new validation be required for the hacked FIPS module?

Yes, and that's a problem ... a big one.
The OpenSSL FIPS Object Module validations are few and far between for a reason: they are painful and expensive. All validations are expensive and time-consuming, with the open source validations especially so⁸. Commercial vendors don't have a significant incentive to pay for the open source based validations (though we have had some notable minority participation). All of the open source based validations to date have been funded in whole or in substantial part by U.S. government agencies, but only after very extended periods of discussion and consideration. No such discussions are in progress at this time, nor do we know of any prospects for such funding.
Even once fully funded an open source based validation can be expected to take a minimum of 18 months or so, as each time new issues seem to arise even when the basic functionality of the source is unchanged from previous validations. During that time a validation is at risk of the introduction of yet more new validation requirements⁹, so even once a validation is begun it isn't possible to predict the schedule and outcome with any degree of confidence.

Q6: What about the matching "FIPS capable" OpenSSL?

That will need to be modified to match the new FIPS module, of course.
The FIPS module by itself is of little use to most application software vendors. To our knowledge only one vendor has ever used the FIPS module directly without OpenSSL, and that was for a very special and specific application. The FIPS module was designed from the beginning to be used in conjunction with OpenSSL in a configuration we call the "FIPS Capable" OpenSSL, where the FIPS module is embedded in the OpenSSL shared library (or library plus application as an executable file) and all relevant cryptographic operations are seamlessly diverted to the FIPS module when the FIPS mode of operation is enabled. A major advantage of the "FIPS capable" OpenSSL is that vendors can ship the same software to all of their customers, those who want the FIPS 140-2 validated cryptography and those who do not¹⁰. For many software products and markets the latter group of users is the largest.
The "FIPS capable" OpenSSL redirects allowed cryptography to the embedded FIPS module and disables all other cryptography (those algorithm implementations not in the FIPS module), and to be conservative¹¹ we implemented the "FIPS capable" OpenSSL 1.0.1 to unconditionally redirect some cryptographic operations (SHA digests, for instance, which are the same for FIPS and non-FIPS implementations) to the FIPS module.
So, a FIPS module that unconditionally disables those cryptographic operations will break OpenSSL for anyone who doesn't immediately enable FIPS mode, which is most calling applications. Note that even aside from the fact that most users won't want the overhead of the POST, it isn't possible for a library to automatically perform POST processing in a portable way.
The proper treatment for a new FIPS capable OpenSSL for use with a disabled-by-default FIPS module will be to have OpenSSL ignore the FIPS module completely until and if it is referenced via a call to the FIPS_mode_set()¹² function.
The OpenSSL FIPS Object Module 2.0 (#1747) is widely used and will remain in widespread use for years, so even if a technical solution to accommodate the new hacked FIPS module was possible we could not implement any changes in the baseline OpenSSL that would affect use of the 2.0 FIPS module. At best some sort of special build-time option¹³ could be developed to accommodate a hacked FIPS module. We haven't studied that in detail other than to note it would be a substantial amount of work.

Q7: What are the best case prospects for a new I.G. 9.5 compatible FIPS module and matching OpenSSL?

Well, two things will need to happen:

A new FIPS module will need to be coded and validated. Even if we had ample funding now (we don't) that will probably take between one and two years.

A new OpenSSL release capable of dealing with a completely non-functional FIPS module will need to be developed and released. Per the existing OpenSSL versioning scheme¹⁴ for incompatible functional changes, the earliest version of OpenSSL that could accommodate the necessary changes would be release 1.0.2. That release is very tentatively planned for early 2014.

Note that the best-case scenario for a new "FIPS capable" OpenSSL solution presumes ample funding which is very unlikely to materialize, so for the foreseeable future the best option for most software vendors will be to leverage the existing 2.0 FIPS module (see next question). It also presumes that the necessary modifications to the code are technically feasible in the first place.

Q8: What is the worst case scenario?

If we don't find substantial funding for the coding and validation work required, and also find the enthusiasm for once again diverting a large proportion of our limited manpower to a goal of interest to only a narrow segment¹⁵ of the overall OpenSSL user community, then the OpenSSL FIPS Object Module 2.0 (#1747) will be the end of the line in that series of open source based validations. There will be no longer be a ready-made code base suitable as a model for future private label validations.
UPDATE 2013-02-28: With the exception of a temporary transitional phase for partially completed validations this scenario appears to be the current reality.

Q9: We need validated OpenSSL crypto now. What can we do?

At present your best -- in fact only -- option is to leverage the existing 2.0 FIPS module (#1747) by adding additional platforms as necessary. Over sixty platforms¹⁶ have been formally tested already and that number will continue to grow, and if your platform(s) of interest aren't already covered they can be added at a relatively modest cost in time and money (as compared to a full validation). Contact OSF for details on sponsoring additional platforms for the 2.0 FIPS module.
The 2.0 FIPS module is still relatively current¹⁷ so using it does not expose you to severe technical deficiencies of the sort that could be only be addressed in a new private label validation.
If you are really desperately eager to have a private label validation with a custom hacked FIPS module and OpenSSL then give us a call, but don't expect to receive much encouragement. If you would like to sponsor a supported open source based solution then we'd love to hear from you, but note that would be an expensive proposition¹⁸.

Q10: Any chance the CMVP will reconsider?

We're being asked this a lot. This impact of this requirement is widespread and we have heard from multiple sources that some accredited test labs and affected vendors have already communicated their concerns to the CMVP. A closed meeting between the test labs and CMVP occurred on February 27, at which this issue was a primary topic of discussion. The CMVP has re-affirmed that new guidance but subsequent preliminary feedback indicates they will permit a temporary transitional period to allow currently in-process validations to be completed. Details on the terms of that transitional period are as yet unknown.
If you wish to contact the CMVP with concerns or questions the CMVP contact info is on the NIST web site. Do keep in mind that there is no protest or appeals process; it's their sandbox and they make the rules. It it possible, likely even, that the scale and severity of the impact of this requirement was not fully anticipated by the CMVP, but upholding the sanctity of the FIPS 140-2 "scripture" is their primary responsibility even in the face of unintended consequences.

- Footnotes -

¹ We're told the CMVP is claiming this interpretation is not new and has been that way all along. However, that interpretation has been unknown to us through many validations with many test labs, and from numerous contacts with other software vendors such ignorance outside of the CMVP proper appears to have been widespread if not universal. If we had known of this requirement when design and development of the FIPS module began some 10 years ago we could have accommodated it as part of the evolution of that design which was after all driven by CMVP requirements.

² For our commercial clients we absorb as much of the risk of the validation process as we can, as that is part of the "valued add" for our validation services. In this case that means we will absorb the full cost of the substantial amount of labor that will be needed for recoding and retesting. We will also have to pay the lab fees for the formal algorithm and operational environment testing that will need to be repeated with the new code (or if the labs waive some or all of those fees then they take a loss, as their testing effort is also non-trivial). At present the losses we will incur promise to be a substantial portion of our annual revenues. Ouch. If we ever do commence doing private label validations again we will need to factor in this level of risk.

³ The Power Up Self Test (POST) is an elaborate set of initialization tests mandated by FIPS 140-2. Roughly speaking they consist of an integrity check to verify no bits have flipped in the executable code since it was created, and checks for the various algorithms which are basically complex mathematical tautologies (does 1+1=2?). The algorithm tests won't fail unless there is something really wrong with the computing environment (1+1≠2). So, for all practical purposes a POST failure will not occur and if it does you have far worse problems than a failed crypto module.

⁴ The monolithic object module is an invention born of the first validation attempt in order to satisfy the peculiar requirements of FIPS 140-2; see section 2.3 of the FIPS module User Guide. In essense it is a composite object module with the individual object modules forced in a specific order in terms of ascending file or memory offsets. That ordering is technically irrelevant to the behavior of the static linker during creation of the "fipscanister" object module and the final application executable containing that object module. All OpenSSL FIPS Object Module validations (and the many "private label" validations derived from them) have featured this monolithic object module that is statically linked into an executable application file. In practice that application often takes the form of the libcrypto.so shared library file (or libeay32.dll for Windows operating systems) for the "FIPS capable" OpenSSL, but for some platforms (e.g. Apple iOS) the FIPS module is statically linked into the application executable. Implementing the FIPS module for static linking exclusively was a strategic design decision from the beginning as there is no platform portable way to implement the integrity test mandated by FIPS 140-2.

⁵ In some very platform and architecture specific contexts hiding of selected global symbols in object code may be possible, using sophisticated compiler and linker voodoo. See for instance Drepper, 2011-12-10, section 2.2, for one specific object code format (ELF). Such voodoo is not an option one would chose absent any other recourse.

⁶ Appendix I of the User Guide for the OpenSSL FIPS Object Module v.20 was machine generated from the source code to satisfy a FIPS 140-2 documentation requirement that such documentation exist. That documentation would not have been prepared otherwise as it has minimal real-world utility; application programmers will not reference the FIPS module API directly except in unusual circumstances in which source code would be consulted as a matter of course anyway. That automation was developed to avoid the considerable labor costs of manually preparing documentation for previous validations. As such it documents every external function and is not a useful guide to the FIPS module interface. That interface was designed for compatibility with the "FIPS Capable" OpenSSL as described in that User Guide document, and not for direct reference by application programmers. The source code to OpenSSL itself is the best description of the FIPS module interface.

Should documentation prove to be a critical issue then it could be redone accordingly to reference only selected functions, either by machine generation from the "FIPS Capable" OpenSSL source and/or by filtering on the basis of the "blocking" mechanism as implemented in the FIPS module. Either approach would be more accurate than manually drafted documentation.

⁷ Almost invariably both identical or near identical cryptographic algorithm implementations, one validated and one not, will reside in the same executable or shared library file. The "FIPS capable" OpenSSL will preferentially use the non-validated algorithm implementations as those are actively maintained to incorporate bugfixes and performance improvements (unlike the FIPS module which effectively cannot be maintained). So, this blocking will be for the purpose of preventing use by an application when such access is undesirable anyway.

⁸ While the open source based validations use the same or very similar source code as many conventional proprietary binary validations they are subjected to a much higher level of scrutiny that significantly drives up the cost.

⁹ Requirement changes can impact ongoing validations up to the point where the formal test report is submitted to the CMVP by the accredited test lab; that is to say after all the design, implementation, and testing work has been completed. Until that point, a period that can span many months for a typical validation, new requirements can mean major rework and expense. That risk should be factored into the contingency planning for any validation effort.

¹⁰ Some users won't know or care one way or the other whether the products they purchase use FIPS 140-2 validated cryptography, but some specifically do not want validated cryptography: it is inherently less secure (in the real-world sense of resistance to malicious compromise) than the equivalent unvalidated implementations; a blanket restriction to the FIPS 140-2 approved algorithms can cause interoperability issues; and for some algorithms performance is inferior. Absent a specific policy mandate there is no reason to use validated cryptography; it is not "better" in any technical or functional sense and is much more difficult to maintain.

¹¹ As an aid to conversion of existing OpenSSL based applications to use of FIPS validated cryptography we put a lot of effort into having the "FIPS capable" OpenSSL API automatically provide the appropriate behavior. When FIPS mode is enabled the calling application is not supposed to invoke any non-validated cryptographic algorithm implementations. Due to the complexity of the legacy OpenSSL API, handling the redirection automatically and comprehensively is a bit tricky. Where there was a chance that redirection of a cryptographic operation to the validated implementation might fail we erred on the side of caution by unconditionally referencing the validated implementation where the FIPS module is present. That is done for instance for the SHA digests, where for the present at least the validated and unvalidated implementations are identical.

¹² The FIPS_mode_set() function is actually defined in the FIPS capable OpenSSL, not the FIPS module. It initializes the passthrough and disabling mechanisms in OpenSSL and calls the FIPS_module_mode_set() function defined in the FIPS module which actually enables the FIPS module of operation.

¹³ The tentative name for the associated macro is OPENSSL_FIPS_IG_9_5_BOLLOCKS. We do try to keep our sense of humor.

¹⁴ Briefly stated "letter" revisions (e.g. 1.0.1e) are for bugfixes only; the introduction of new features requires a "number" release (e.g. 1.0.2). The argument has been advanced that a new crippled FIPS module could be considered a bug, but before adding any such support to the 1.0.1 baseline we would have to be absolutely sure the existing #1747 validated module was not affected.

¹⁵ When working on FIPS 140-2 validation issues it's easy to forget that they are of interest to only some OpenSSL users. Use of FIPS 140-2 validated cryptography is mandated for the U.S. and Canadian governments, and is of some interest in selected commercial settings (financial, health care industries) in North America and Europe, but of little or no interest throughout most of the world and for very large sectors of the software industry. A valid argument can be made that too many scarce OpenSSL resources have been already consumed in accommodating the peculiar requirements of FIPS 140-2 instead of improving the OpenSSL product in general.

¹⁶ The #1747 validation includes more tested platforms ("Operational Environments") than any other validation. We believe it also includes the largest number of algorithms.

¹⁷ The current OpenSSL FIPS Object Module 2.0 (#1747) does not support GMAC, and the FIPS module with the associated "FIPS capable" OpenSSL 1.0.1 does not fully address the "Lucky Thirteen" TLS vulnerability. We have been adding GMAC support to recent private label validations. The TLS vulnerability could potentially, though not easily, be addressed for new private label validations but this latest issue makes that moot. As with GMAC support that vulnerability can't be fully addressed for the existing 2.0 module because of the practical impossibility of changing an existing validated module.

¹⁸ We've been asked this several times already and don't have a ready answer as the cost will depend on a more thorough analysis of the coding issues and external factors such as the accredited test lab fees. The coding work alone will probably cost a minimum of mid five figures (US$50,000+) and in the low-to-mid six figures at least for a formal FIPS 140-2 validation. That assumes compliance is even theoretically possible (see earlier footnote on the meaning of "all" functions).

- Update History -

2013-03-14: Q2: Note transition deadline of November 15
2013-03-01: More detail for new footnote #4. New footnote #9 added and all earlier footnotes #8 and obove incremented by one.
2013-02-27: Added preliminary feedback from 2013-02-27 manager's meeting
Added new footnote #4, #5, and #6 for Q4, all earlier footnotes #4 and above incremented by three.
2013-02-27: Multiple edits and new footnotes
2013-02-12: Initial draft