[botan-devel] Deprecating public use of many Botan headers

Jack Lloyd jack at randombit.net
Tue Aug 6 07:42:51 EDT 2019

Currently Botan has 300 (!) public headers.

I've opened a PR (https://github.com/randombit/botan/pull/2061) which will
deprecate public access to many of these headers. Right now this means a warning
is issued if the header is included. In Botan 3 the headers will be made
internal and no longer accessible to applications.

The benefits of this change are:

 * Reducing the number of APIs which a developer must look at (which of the 300
   headers has the functionality you want?)

 * Making internal library evolution easier (fewer API contracts to maintain)

 * It will enable some useful optimizations that should particularly improve
   performance on systems with AES/GCM hardware. Right now, every time AES
   operation is performed a check of the CPUID fields is done. This requires an
   out-of-line data access, because MSVC doesn't support thread local storage in
   DLL-exported data. In the future, not only can the CPUID check be inline, but
   we won't need it on each call because instead we will again perform the
   implementation dispatch once at creation time and then use vtable lookups
   (see https://github.com/randombit/botan/issues/477 for why that doesn't
   happen currently - main answer is because AES_128 class is a public API).
   Currently CPUID dispatch is an alarmingly high % of the total cost of AES
   operations when hardware is available.

 * Reducing ABI surface. We typically have to increment the soname with each
   release, sometimes due to trivial reasons like a single struct changing
   size. Hiding as much as possible reduces soname churn. This in particular
   would make it easier for updates to get into Debian and downstream distros
   since they don't like soname bumps.

 * Reducing shared lib exports. If the compiler knows a symbol is not exported
   (and thus cannot be overridden eg via LD_PRELOAD) then it can optimize
   somewhat better. This last is a bit marginal since a lot of functionality is
   already exported (only) for tests which use them via the internal headers.

There are a few major categories of headers so deprecated:

 * Algorithm headers like aes.h, gcm.h, kdf2.h, etc. Instead the application
   should use X::create and use the operation through the base class
   interface. This does not apply to public key classes since there is some
   functionality only available via direct usage right now, namely keygen and
   accessing fields.

 * The EME/EMSA headers are entirely deprecated, even the base class - there
   doesn't seem to be any reason for an application to ever use these.

 * Utility headers like bswap.h, rotate.h, cpuid.h, parsing.h - these are used
   to implement the library but aren't part of our core functionality offering.

 * Implementation detail headers like xmss_common_ops.h, divide.h, keypair.h,
   tls_magic.h which an application has no obvious reason to use directly.

The full list of headers currently targetted for going internal is:

adler32.h aes.h aria.h atomic.h bcrypt_pbkdf.h blake2b.h blinding.h blowfish.h
bswap.h camellia.h cascade.h cast128.h cast256.h cbc.h cbc_mac.h ccm.h cfb.h
chacha20poly1305.h chacha.h charset.h cmac.h comb4p.h cpuid.h crc24.h crc32.h
ctr.h curve_gfp.h curve_nistp.h datastor.h des.h desx.h divide.h eax.h eme.h
eme_pkcs.h eme_raw.h emsa1.h emsa.h emsa_pkcs1.h emsa_raw.h emsa_x931.h gcm.h
gf2m_small_m.h ghash.h gmac.h gost_28147.h gost_3411.h hash_id.h hkdf.h hmac.h
http_util.h idea.h iso9796.h kasumi.h kdf1.h kdf1_iso18033.h kdf2.h keccak.h
keypair.h lion.h loadstor.h locking_allocator.h md4.h md5.h mdx_hash.h misty1.h
mode_pad.h mul128.h noekeon.h oaep.h ocb.h ofb.h par_hash.h parsing.h pbes2.h
pbkdf1.h pbkdf2.h pgp_s2k.h poly1305.h polyn_gf2m.h pow_mod.h prf_tls.h
prf_x942.h pssr.h rc4.h rfc6979.h rmd160.h rotate.h salsa20.h scan_name.h
scrypt.h secqueue.h seed.h serpent.h sha160.h sha2_32.h sha2_64.h sha3.h
shacal2.h shake_cipher.h shake.h siphash.h siv.h skein_512.h sm3.h sm4.h
sp800_108.h sp800_56a.h sp800_56c.h stl_compatibility.h stream_mode.h streebog.h
threefish_512.h tiger.h tls_algos.h tls_magic.h twofish.h uuid.h whrlpool.h
x919_mac.h xmss_common_ops.h xmss_hash.h xmss_index_registry.h xmss_tools.h
xtea.h xts.h

I expect the most common change that will be required is switching to using the
factory functions to create objects like AES or SHA-256. Before merge I'll be
checking open source projects that I know use a wide range of functionality
(like SoftHSM, KEA, and libQtShadowsocks) to see if/how wide breakage is.

If you think this is a terrible idea in general *or* you think there is some
specific header in the above list which should remain available to applications
in the future, now is the time to comment. The PR will remain open for comments
through early September. And of course, no actual breakage will occur until
Botan 3 is released.


More information about the botan-devel mailing list