[botan-devel] Deprecating public use of many Botan headers
jack at randombit.net
Tue Aug 6 07:42:51 EDT 2019
Currently Botan has 300 (!) public headers.
I've opened a PR (https://github.com/randombit/botan/pull/2061) which will
deprecate public access to many of these headers. Right now this means a warning
is issued if the header is included. In Botan 3 the headers will be made
internal and no longer accessible to applications.
The benefits of this change are:
* Reducing the number of APIs which a developer must look at (which of the 300
headers has the functionality you want?)
* Making internal library evolution easier (fewer API contracts to maintain)
* It will enable some useful optimizations that should particularly improve
performance on systems with AES/GCM hardware. Right now, every time AES
operation is performed a check of the CPUID fields is done. This requires an
out-of-line data access, because MSVC doesn't support thread local storage in
DLL-exported data. In the future, not only can the CPUID check be inline, but
we won't need it on each call because instead we will again perform the
implementation dispatch once at creation time and then use vtable lookups
(see https://github.com/randombit/botan/issues/477 for why that doesn't
happen currently - main answer is because AES_128 class is a public API).
Currently CPUID dispatch is an alarmingly high % of the total cost of AES
operations when hardware is available.
* Reducing ABI surface. We typically have to increment the soname with each
release, sometimes due to trivial reasons like a single struct changing
size. Hiding as much as possible reduces soname churn. This in particular
would make it easier for updates to get into Debian and downstream distros
since they don't like soname bumps.
* Reducing shared lib exports. If the compiler knows a symbol is not exported
(and thus cannot be overridden eg via LD_PRELOAD) then it can optimize
somewhat better. This last is a bit marginal since a lot of functionality is
already exported (only) for tests which use them via the internal headers.
There are a few major categories of headers so deprecated:
* Algorithm headers like aes.h, gcm.h, kdf2.h, etc. Instead the application
should use X::create and use the operation through the base class
interface. This does not apply to public key classes since there is some
functionality only available via direct usage right now, namely keygen and
* The EME/EMSA headers are entirely deprecated, even the base class - there
doesn't seem to be any reason for an application to ever use these.
* Utility headers like bswap.h, rotate.h, cpuid.h, parsing.h - these are used
to implement the library but aren't part of our core functionality offering.
* Implementation detail headers like xmss_common_ops.h, divide.h, keypair.h,
tls_magic.h which an application has no obvious reason to use directly.
The full list of headers currently targetted for going internal is:
adler32.h aes.h aria.h atomic.h bcrypt_pbkdf.h blake2b.h blinding.h blowfish.h
bswap.h camellia.h cascade.h cast128.h cast256.h cbc.h cbc_mac.h ccm.h cfb.h
chacha20poly1305.h chacha.h charset.h cmac.h comb4p.h cpuid.h crc24.h crc32.h
ctr.h curve_gfp.h curve_nistp.h datastor.h des.h desx.h divide.h eax.h eme.h
eme_pkcs.h eme_raw.h emsa1.h emsa.h emsa_pkcs1.h emsa_raw.h emsa_x931.h gcm.h
gf2m_small_m.h ghash.h gmac.h gost_28147.h gost_3411.h hash_id.h hkdf.h hmac.h
http_util.h idea.h iso9796.h kasumi.h kdf1.h kdf1_iso18033.h kdf2.h keccak.h
keypair.h lion.h loadstor.h locking_allocator.h md4.h md5.h mdx_hash.h misty1.h
mode_pad.h mul128.h noekeon.h oaep.h ocb.h ofb.h par_hash.h parsing.h pbes2.h
pbkdf1.h pbkdf2.h pgp_s2k.h poly1305.h polyn_gf2m.h pow_mod.h prf_tls.h
prf_x942.h pssr.h rc4.h rfc6979.h rmd160.h rotate.h salsa20.h scan_name.h
scrypt.h secqueue.h seed.h serpent.h sha160.h sha2_32.h sha2_64.h sha3.h
shacal2.h shake_cipher.h shake.h siphash.h siv.h skein_512.h sm3.h sm4.h
sp800_108.h sp800_56a.h sp800_56c.h stl_compatibility.h stream_mode.h streebog.h
threefish_512.h tiger.h tls_algos.h tls_magic.h twofish.h uuid.h whrlpool.h
x919_mac.h xmss_common_ops.h xmss_hash.h xmss_index_registry.h xmss_tools.h
I expect the most common change that will be required is switching to using the
factory functions to create objects like AES or SHA-256. Before merge I'll be
checking open source projects that I know use a wide range of functionality
(like SoftHSM, KEA, and libQtShadowsocks) to see if/how wide breakage is.
If you think this is a terrible idea in general *or* you think there is some
specific header in the above list which should remain available to applications
in the future, now is the time to comment. The PR will remain open for comments
through early September. And of course, no actual breakage will occur until
Botan 3 is released.
More information about the botan-devel