[cryptography] OpenBSD IPSEC backdoor(s)
marsh at extendedsubset.com
Sat Dec 18 05:12:47 EST 2010
On 12/16/2010 04:09 PM, Marsh Ray wrote:
After reviewing the code. Here are my opinions:
* Angelos Keromytis made huge contributions to OpenBSD by porting and
enhancing the early IPsec implementation of John Ioannidis. He also
contributed to the initial development of the OpenBSD crypto framework.
* In what is perhaps the sincerest form of flattery, this code has also
been incorporated into many other projects, some of which are closed
source and some are not derived from BSD.
* I didn't spot anything malicious or intentionally backdoored in the
IPsec ESP implementation code that I looked at.
* There was a serious vulnerability in ESP-mode IPsec shipped in OpenBSD
3.0 and 3.1 and silently patched before 3.2.
* Gregory Perry made allegations that were specific and testable enough
that they merited a little investigation and a bug was found that could
have made a very close match for his description. But upon closer
inspection, this particular bug is extremely ordinary.
* I primarily reviewed a small set of source files specific to ESP,
these only partially overlapped those of the developer Perry accused by
name (Jason Wright). Nevertheless, any credence which might have been
given to Perry's claims as a result of this bug should be reverted to
zero (or less).
* This bug doesn't sufficiently meet the criteria for a malicious backdoor:
- The bug does not leak key material or establish a covert channel, it
would require an active attack to exploit and even then would probably
need to be used in connection with some other defect in order to result
in meaningful unauthorized access. Yeah sorta it maybe could be used as
part of that, but not really its own.
- The bug is not hidden. There is nothing to suggest any attempt at
misdirection or obfuscation.
- The bug is not particularly subtle or even hard-to-find.
- Angelos is a recognized expert in low-level maliciousness. Surely he
would have come up with something better.
- The bug has a far simpler explanation (more on that later)
* There is little or nothing to suggest that Angelos was influenced by
money from NETSEC. To the contrary, judging by publications, Angelos
clearly had a plethora of research projects on his plate at the time he
moved on from OpenBSD in July of 2002 (shortly before the bug was patched).
* When Angelos moved on, the IPsec and associated crypto code were
adopted by Jason and other OpenBSD developers. But the transition
appears to have left some code changes in an unfinished state. For
example, the inverted conditional at the core of this problem looks like
it was introduced as part of an architectural enhancement to support
IPsec-enabled network cards which performed decryption and
authentication of the incoming packets right on the NIC itself. However,
no drivers of this type appeared in the source tree, so the new logic
probably went untested. The apparent work-in-progress code silently
became part of the 3.0 and subsequent release branches.
* OpenBSD did not live up to their stated principle of full disclosure.
They should have issued an advisory for this.
* OpenBSD's security auditing processes did not catch this bug, either
when it was introduced or in any subsequent review. In a follow-up email
to the CVS commit, Jason indicates that the fix was supplied by BSD guru
Sam Leffler, who was working on an optimized IPsec implementation for
FreeBSD about that time.
* Code coverage testing would have had a good chance of catching this.
* This bug has a far simpler explanation than Perry's, yet almost
equally grand: The combination of the inherent difficulty of testing
crypto protocol implementations and bizarre constraints placed on the
development process made it highly likely that the resulting software
would contain vulnerabilities.
Typically, protocol implementations are tested first for functionality
and interoperability, then secondarily for failure modes. In the early
years of any protocol (and especially in the data security middle ages
of the 1990s), attack tools with which to test a new implementation are
not publicly available. So the implementer is usually expected to create
all of the tools needed for the simulation of every attack himself,
adding yet more onto the schedule of competing development priorities.
When the release date approaches and inevitably something must be cut,
failure testing loses to functional testing nearly every time. Open
source appears to be no different from closed source commercial software
in this regard.
At the time, the US government was pursuing a policy of restricting
"export" of crypto by classifying it as a "munition". Aside from the
question of whether or not this was a sensible policy, it certainly made
the software development process more colorful. The bulk of the IPsec
and OCF source code was checked-in with notes such as "This software was
developed in Greece" in order to avoid the permanent stain of US
development. OpenBSD developers are regularly emigrating to Canada for a
weekend or a week in order to compress as much untained development as
possible into round-the-clock coding sessions called "hackathons".
Because of the need to document the non-US origin of this code, the code
produced from the hackathons needed be committed to the OpenBSD source
control system (CVS) before the developers returned to the US, whether
it was fully-baked or not. As OpenBSD consistently adhered to a
calendar-based release schedule (every six months), this effectively set
the clock ticking to get it in release shape.
Of course, the calendar and codebase integration are challenges to any
multi-developer software project. But the added challenge of having to
leave the country to develop on certain parts of it could not have
helped. One can't help but wonder if this energy spent "routing around
the damage" in the natural progress of software development would have
otherwise been spent on deeper testing.
So this IPsec bug is a concrete manifestation of a US export policy
which had the effects of driving the developers of data security
software out of the US (literally!) while at the same time causing users
within the US to experience a net reduction in their effective security.
* For a variety of reasons, perhaps the least of which being the
allegations of Gregory Perry, development groups which rely on code
derived from the OpenBSD IPsec, Crypto Framework, and associated drivers
of this era should probably review it again closely for security bugs.
OpenBSD reviewers will sometimes explicitly disclaim responsibility for
evaluating the exploitability of bugs, preferring to fix them and move
on. In the absence of proactive disclosure practices, CVS logs for the
individual files give invaluable clues about where bugs were fixed and
how they were found.
* Where there have been bugs found, there are likely more bugs.
* I spotted an old security bug and ran it to ground. I feel like I've
done my duty on this and need to get back to other stuff. It's not a
government backdoor, Angelo could have come up with tons of stuff way
better than that. Unless, of course, that's exactly what he wanted us to
think, hmm... oh give it up everyone, dig into more code review or move
marsh at extendedsubset.com
Here are the specifics on the IPsec ESP auth bypass in OpenBSD 3.0:
There are only two places that match 'bcmp' and 'auth', one in
netinet/ip_esp.c and the other in ip_ah.c. These are also the two places
with an "authentication failed" message other than an unrelated wifi
driver. So I conclude that those are probably the only places an IPsec
authenticator is being checked in software.
The code path to that check in ip_ah.c is controlled by the correct
"mtag == NULL" test.
The code path in ip_esp.c has the incorrect "mtag != NULL" test.
So the only way to have your IPsec ESP authenticator validated is to
have a non-NULL mtag in the esp_input_cb() function.
The esp_input_cb function sets mtag from tc->tc_ptr. tc is set from a
function parameter with one additional indirection through a tdb_crypto
esp_input_cb() is called directly from esp_input() if it has an mtag at
that point. Otherwise, esp_input exits with a tail call to
crypto_dispatch(crp), with crp pointing to a structure which contains a
pointer to esp_input_cb. In that case esp_input_cb might be thought of
as a completion routine on the asynchronous result of some crypto
Various crypto drivers, such as the hifn7751, will call to this
crp_callback pointer in one place. In the case of hifn7751.c, it will
call crp_callback directly from one place, dev/pci/hifn7751.c:1966,
where it is an error exit return. Otherwise it calls it directly through
a crypto_done() function defined in crypto/crypto.c.
But this really doesn't matter, because none of these driver files
access the tdb_crypto structure from which any potential mtag would be
extracted by esp_input_cb. The tdb_crypto structure is passed from
esp_input to esp_input_cb through any requested crypto processing in a
parameter named 'crp_opaque'.
So in OpenBSD 3.0, there is no driver which can add the mtag to a crp
crypto processing request. (Although the request for decrypting and
hashing the packet is still processed.) So that means the hifn card is
not enabling integrity.
That leaves just one possible way for an mtag to have been set: before
the call to esp_input(). It requres an input chain containing a mbuf of
type PACKET_TAG_IPSEC_IN_CRYPTO_DONE. Mbufs are created with the
Nothing I can find sets mtag type PACKET_TAG_IPSEC_IN_CRYPTO_DONE.
There's one function (ipsp_parse_headers in ip_ipsp.c) that could do it,
but nothing calls that function nor is its address taken. This dead code
function remains at least through 3.2, maybe later.
So it really doesn't look like anyone running IPsec ESP on OpenBSD 3.0
or 3.1 was protected from message manipulation, even with just the right
Without implying anything about anything, these are some things I'd
intended to follow up on in case anyone else is interested:
appears to remove the limits on the number of simultaneous crypto
sessions with that model accelerator card. But it's not immediately
obvious what limits remain to replace it, other than contiguous free
kernel RAM. Linear search, linear copy, grow-by-1 reallocation, could be
a nasty DoS, or could amount to nothing.
* Could there be some attacks on the IPsec code, especially if various
IP extension headers were sent out-of-order? What if ipcomp were
enabled? Some of that IPsec header mbuf manipulation code looked fragile.
* This commit message
says that some drivers "will do really nasty things like DMA to
completely random locations in memory" if you give them non-blocksize
input to block cipher operations. There were other messages suggesting
bugs had been fixed in that area. Are there more?
More information about the cryptography