The cleanest way to discover that a piece of widely-deployed open source software has a quiet bug is to try to use a feature that almost nobody exercises in production. For us, that feature was generating a 521-bit elliptic curve key inside OpenStack’s Barbican key manager.
The reason almost nobody hits the bug is that almost nobody uses 521-bit EC keys. Most enterprise crypto runs on RSA-2048 or RSA-4096, sometimes P-256 for ECDSA TLS. The 521-bit NIST curve sits at the high end of the elliptic-curve range, with a 256-bit security level matching AES-256. It is the curve you reach for when policy requires it, when you want symmetry between your symmetric and asymmetric strength, or when you are designing a long-lifetime sovereign-cloud root of trust where every bit of margin matters.
We were doing the third one. Here is what happened, and what we did about it.
The setup
Barbican is OpenStack’s secrets and key management service. It is the backbone of any cloud that wants to offer customer-managed key encryption for Cinder volumes, certificate storage for Octavia load balancers, or asymmetric key generation for any tenant that needs it. Recent work on our platform wired Barbican into both Cinder and Nova as the key manager backend so that volume encryption uses customer-controlled keys rather than provider-controlled ones. That closes the SC-28 gap (the NIST 800-53 control covering protection of information at rest) and is the technical precondition for offering Bring-Your-Own-Key arrangements to enterprise customers.
We use a custom PKCS#11 plugin behind Barbican for the actual cryptographic operations. The plugin supports symmetric AES generation, RSA at 2048, 3072, and 4096 bits, and EC key generation across the three NIST curves: P-256, P-384, and P-521.
Eight of nine key types worked first time. The ninth, P-521, did not.
The bug
A POST to Barbican to create an asymmetric EC key order with bit_length=521 returned the same error every time:
UnsupportedField: Must be a positive integer that is a multiple of 8Two hundred fifty-six is divisible by eight. Three hundred eighty-four is divisible by eight. Five hundred twenty-one is not. The validator that gates bit_length on order creation enforces a multiple-of-eight rule that is correct for symmetric algorithms (AES key sizes are always multiples of eight) and incidentally correct for two of three NIST EC curves, but it rejects the third.
The relevant code in barbican/common/validators.py:
def _validate_bit_length(self, meta, schema_name):
bit_length = int(meta.get('bit_length'))
if bit_length % 8 != 0:
raise exception.UnsupportedField(field="bit_length", ...)P-521 fails the modulo check. P-256 and P-384 pass it for the same arithmetic reason that any number ending in three binary zero places is divisible by eight: coincidence, not design.
The fix is small and obvious in hindsight. Allow an exception for the known EC curve bit lengths when the algorithm field is EC:
_EC_BIT_LENGTHS = frozenset([256, 384, 521])
def _validate_bit_length(self, meta, schema_name):
bit_length = int(meta.get('bit_length'))
algorithm = meta.get('algorithm', '').lower()
if bit_length % 8 != 0:
if algorithm == 'ec' and bit_length in self._EC_BIT_LENGTHS:
return
raise exception.UnsupportedField(field="bit_length", ...)Six lines added to the validator, thirty-four lines of tests covering the three curves and the negative cases. Roughly a thirty-minute fix once the root cause was clear. The tests matter more than the code: they document the invariant (“these specific bit lengths are valid for EC orders even though they are not multiples of eight”) so a future refactor does not regress the bug.
Filing it upstream
We filed the bug on Launchpad as Barbican #2147221 with a clear reproducer, the expected and actual behavior, and the proposed fix. The patch is now in review on the OpenStack Gerrit at change 983320 under the canonical metadata:
Closes-Bug: #2147221
Change-Id: I61975ab6ca793cf83270c555ca93629e8043dee5The P-521 fix is the most recent in a run of upstream Barbican contributions from our team this past month. The pattern is consistent: encounter a real defect during deployment, isolate the minimal reproducer, write the test, propose the fix on Gerrit. Recently merged:
- 983290 – Fix incorrect
P11CryptoPluginExceptionimport in the KEK migration script. Broke the rotation tooling for any deployment using the PKCS#11 backend. - 983319 – Documentation fix for the
enforce_new_defaultsruntime override behavior. Subtle, but the kind of policy/RBAC documentation gap that costs operators hours. - 983328 – Fix
SimpleCryptoPluginTypeErrorwhen the KEK is unconfigured. Defensive failure path that previously crashed instead of returning a useful error.
Plus 983331 (remove a deprecated PKCS#11 key generation script) currently in review alongside the P-521 fix.
That is what running OpenStack in production looks like when you actually run it in production. You do not get to file the bug and wait. You write the patch, ship it locally so your customers are unblocked, and propose it upstream so the next operator who hits the same wall does not have to repeat the work. It is unglamorous, and it is exactly the kind of contribution that keeps OpenStack functioning as a project that vendors can deploy for customers without each vendor maintaining a private fork of every component forever.
Why customer-managed P-521 actually matters
The reason this bug had to be fixed, rather than worked around, is that “use a different curve” is not always an option for the customer who needs the key. Compliance regimes, security policies, and long-lived root keys all sometimes require P-521. A managed cloud that cannot generate the curve the customer’s policy demands is a managed cloud that the customer cannot use for that workload.
Customer-managed keys for Cinder volume encryption are part of how a managed OpenStack cloud closes the gap with the hyperscaler “BYOK” offerings. The customer controls the key. The provider controls the infrastructure. Compromise of one does not compromise the other. That is the architectural property that makes the arrangement worth the operational complexity, and it depends on the key manager actually being able to generate the keys the customer wants.
The smaller, related point: if the validator had silently accepted bit_length=512 (which is a multiple of eight and which a customer might plausibly choose as a “round number” near 521), Barbican would have generated a different key than the one the customer expected. The error here is loud, which is the right kind of failure. The fix preserves the loudness for genuinely invalid sizes while correctly accepting the three NIST EC curves.
What you can do with this
Three things, depending on who you are.
If you operate Barbican in your own OpenStack deployment, this bug affects you the moment you try to support EC P-521 keys. The patch is six lines. You can apply it locally today against your fork while the upstream review proceeds, and pin the corresponding test addition to make sure the invariant survives any future changes.
If you run workloads on a managed OpenStack cloud and the provider claims customer-managed key support, ask them which curves they actually generate end-to-end. The honest answer reveals whether they have done this work or whether they have stopped at “we run Barbican.”
If you are evaluating managed cloud options where sovereign cryptography matters, the test that separates the marketing from the engineering is whether the provider can hand you a P-521 EC key from their key manager, prove it was generated inside their HSM, and show you the audit trail. That is now a thing you can do on Open Edge Cloud.
If you want to compare notes on Barbican PKCS#11 plugin development, customer-managed-key architectures, or upstream OpenStack contribution mechanics, send a note to [email protected].