[cryptography] OpenBSD IPSEC backdoor(s)

Marsh Ray marsh at extendedsubset.com
Sat Dec 18 05:12:47 EST 2010

On 12/16/2010 04:09 PM, Marsh Ray wrote:
> http://code.bsd64.org/cvsweb/openbsd/src/sys/netinet/ip_esp.c.diff?r1=1.74;r2=1.75;f=h

After reviewing the code. Here are my opinions:

* Angelos Keromytis made huge contributions to OpenBSD by porting and 
enhancing the early IPsec implementation of John Ioannidis. He also 
contributed to the initial development of the OpenBSD crypto framework.

* In what is perhaps the sincerest form of flattery, this code has also 
been incorporated into many other projects, some of which are closed 
source and some are not derived from BSD.

* I didn't spot anything malicious or intentionally backdoored in the 
IPsec ESP implementation code that I looked at.

* There was a serious vulnerability in ESP-mode IPsec shipped in OpenBSD 
3.0 and 3.1 and silently patched before 3.2.

* Gregory Perry made allegations that were specific and testable enough 
that they merited a little investigation and a bug was found that could 
have made a very close match for his description. But upon closer 
inspection, this particular bug is extremely ordinary.

* I primarily reviewed a small set of source files specific to ESP, 
these only partially overlapped those of the developer Perry accused by 
name (Jason Wright). Nevertheless, any credence which might have been 
given to Perry's claims as a result of this bug should be reverted to 
zero (or less).

* This bug doesn't sufficiently meet the criteria for a malicious backdoor:
- The bug does not leak key material or establish a covert channel, it 
would require an active attack to exploit and even then would probably 
need to be used in connection with some other defect in order to result 
in meaningful unauthorized access. Yeah sorta it maybe could be used as 
part of that, but not really its own.
- The bug is not hidden. There is nothing to suggest any attempt at 
misdirection or obfuscation.
- The bug is not particularly subtle or even hard-to-find.
- Angelos is a recognized expert in low-level maliciousness. Surely he 
would have come up with something better.
- The bug has a far simpler explanation (more on that later)

* There is little or nothing to suggest that Angelos was influenced by 
money from NETSEC. To the contrary, judging by publications, Angelos 
clearly had a plethora of research projects on his plate at the time he 
moved on from OpenBSD in July of 2002 (shortly before the bug was patched).

* When Angelos moved on, the IPsec and associated crypto code were 
adopted by Jason and other OpenBSD developers. But the transition 
appears to have left some code changes in an unfinished state. For 
example, the inverted conditional at the core of this problem looks like 
it was introduced as part of an architectural enhancement to support 
IPsec-enabled network cards which performed decryption and 
authentication of the incoming packets right on the NIC itself. However, 
no drivers of this type appeared in the source tree, so the new logic 
probably went untested. The apparent work-in-progress code silently 
became part of the 3.0 and subsequent release branches.

* OpenBSD did not live up to their stated principle of full disclosure. 
They should have issued an advisory for this.

* OpenBSD's security auditing processes did not catch this bug, either 
when it was introduced or in any subsequent review. In a follow-up email 
to the CVS commit, Jason indicates that the fix was supplied by BSD guru 
Sam Leffler, who was working on an optimized IPsec implementation for 
FreeBSD about that time.

* Code coverage testing would have had a good chance of catching this.

* This bug has a far simpler explanation than Perry's, yet almost 
equally grand: The combination of the inherent difficulty of testing 
crypto protocol implementations and bizarre constraints placed on the 
development process made it highly likely that the resulting software 
would contain vulnerabilities.

Typically, protocol implementations are tested first for functionality 
and interoperability, then secondarily for failure modes. In the early 
years of any protocol (and especially in the data security middle ages 
of the 1990s), attack tools with which to test a new implementation are 
not publicly available. So the implementer is usually expected to create 
all of the tools needed for the simulation of every attack himself, 
adding yet more onto the schedule of competing development priorities. 
When the release date approaches and inevitably something must be cut, 
failure testing loses to functional testing nearly every time. Open 
source appears to be no different from closed source commercial software 
in this regard.

At the time, the US government was pursuing a policy of restricting 
"export" of crypto by classifying it as a "munition". Aside from the 
question of whether or not this was a sensible policy, it certainly made 
the software development process more colorful. The bulk of the IPsec 
and OCF source code was checked-in with notes such as "This software was 
developed in Greece" in order to avoid the permanent stain of US 
development. OpenBSD developers are regularly emigrating to Canada for a 
weekend or a week in order to compress as much untained development as 
possible into round-the-clock coding sessions called "hackathons".

Because of the need to document the non-US origin of this code, the code 
produced from the hackathons needed be committed to the OpenBSD source 
control system (CVS) before the developers returned to the US, whether 
it was fully-baked or not. As OpenBSD consistently adhered to a 
calendar-based release schedule (every six months), this effectively set 
the clock ticking to get it in release shape.

Of course, the calendar and codebase integration are challenges to any 
multi-developer software project. But the added challenge of having to 
leave the country to develop on certain parts of it could not have 
helped. One can't help but wonder if this energy spent "routing around 
the damage" in the natural progress of software development would have 
otherwise been spent on deeper testing.

So this IPsec bug is a concrete manifestation of a US export policy 
which had the effects of driving the developers of data security 
software out of the US (literally!) while at the same time causing users 
within the US to experience a net reduction in their effective security.

* For a variety of reasons, perhaps the least of which being the 
allegations of Gregory Perry, development groups which rely on code 
derived from the OpenBSD IPsec, Crypto Framework, and associated drivers 
of this era should probably review it again closely for security bugs. 
OpenBSD reviewers will sometimes explicitly disclaim responsibility for 
evaluating the exploitability of bugs, preferring to fix them and move 
on. In the absence of proactive disclosure practices, CVS logs for the 
individual files give invaluable clues about where bugs were fixed and 
how they were found.

* Where there have been bugs found, there are likely more bugs.

* I spotted an old security bug and ran it to ground. I feel like I've 
done my duty on this and need to get back to other stuff. It's not a 
government backdoor, Angelo could have come up with tons of stuff way 
better than that. Unless, of course, that's exactly what he wanted us to 
think, hmm... oh give it up everyone, dig into more code review or move 


Marsh Ray
marsh at extendedsubset.com


Here are the specifics on the IPsec ESP auth bypass in OpenBSD 3.0:

There are only two places that match 'bcmp' and 'auth', one in 
netinet/ip_esp.c and the other in ip_ah.c. These are also the two places 
with an "authentication failed" message other than an unrelated wifi 
driver. So I conclude that those are probably the only places an IPsec 
authenticator is being checked in software.

The code path to that check in ip_ah.c is controlled by the correct 
"mtag == NULL" test.
The code path in ip_esp.c has the incorrect "mtag != NULL" test.

So the only way to have your IPsec ESP authenticator validated is to 
have a non-NULL mtag in the esp_input_cb() function.

The esp_input_cb function sets mtag from tc->tc_ptr. tc is set from a 
function parameter with one additional indirection through a tdb_crypto 

esp_input_cb() is called directly from esp_input() if it has an mtag at 
that point. Otherwise, esp_input exits with a tail call to 
crypto_dispatch(crp), with crp pointing to a structure which contains a 
pointer to esp_input_cb. In that case esp_input_cb might be thought of 
as a completion routine on the asynchronous result of some crypto 
processing operation.

Various crypto drivers, such as the hifn7751, will call to this 
crp_callback pointer in one place. In the case of hifn7751.c, it will 
call crp_callback directly from one place, dev/pci/hifn7751.c:1966, 
where it is an error exit return. Otherwise it calls it directly through 
a crypto_done() function defined in crypto/crypto.c.

But this really doesn't matter, because none of these driver files 
access the tdb_crypto structure from which any potential mtag would be 
extracted by esp_input_cb. The tdb_crypto structure is passed from 
esp_input to esp_input_cb through any requested crypto processing in a 
parameter named 'crp_opaque'.

So in OpenBSD 3.0, there is no driver which can add the mtag to a crp 
crypto processing request. (Although the request for decrypting and 
hashing the packet is still processed.) So that means the hifn card is 
not enabling integrity.

That leaves just one possible way for an mtag to have been set: before 
the call to esp_input(). It requres an input chain containing a mbuf of 
type PACKET_TAG_IPSEC_IN_CRYPTO_DONE. Mbufs are created with the

Nothing I can find sets mtag type PACKET_TAG_IPSEC_IN_CRYPTO_DONE.
There's one function (ipsp_parse_headers in ip_ipsp.c) that could do it, 
but nothing calls that function nor is its address taken. This dead code 
function remains at least through 3.2, maybe later.

So it really doesn't look like anyone running IPsec ESP on OpenBSD 3.0 
or 3.1 was protected from message manipulation, even with just the right 
hardware combination.


Without implying anything about anything, these are some things I'd 
intended to follow up on in case anyone else is interested:

This change
> http://code.bsd64.org/cvsweb/openbsd/src/sys/dev/pci/hifn7751.c.diff?r1=1.145;r2=1.146

appears to remove the limits on the number of simultaneous crypto 
sessions with that model accelerator card. But it's not immediately 
obvious what limits remain to replace it, other than contiguous free 
kernel RAM. Linear search, linear copy, grow-by-1 reallocation, could be 
a nasty DoS, or could amount to nothing.

* Could there be some attacks on the IPsec code, especially if various 
IP extension headers were sent out-of-order? What if ipcomp were 
enabled? Some of that IPsec header mbuf manipulation code looked fragile.

* This commit message
> http://code.bsd64.org/cvsweb/openbsd/src/sys/crypto/cryptodev.c#rev1.13

says that some drivers "will do really nasty things like DMA to 
completely random locations in memory" if you give them non-blocksize 
input to block cipher operations. There were other messages suggesting 
bugs had been fixed in that area. Are there more?

More information about the cryptography mailing list