-------------------- Start of forwarded message -------------------- To: nisse@lysator.liu.se Subject: ML-KEM: encap/decap cannot reject invalid keys (FIPS 203 input checks) From: RatioLabs info@ratiolabs.eu Date: Fri, 20 Nov 2026 16:59:35 +0000
Hello,
We sent the message below to nettle-bugs, but the list only accepts posts from subscribers and rejected it. Your README invites mail, so we are sending it to you directly. Please feel free to forward it to the list.
----
Hello,
We ran the public Wycheproof ML-KEM vectors (C2SP/wycheproof at 65ca494727a5) against the ML-KEM code on the Nettle master branch (commit 10d33c6). All valid vectors give the expected result for ML-KEM-768 and ML-KEM-1024: key generation from a seed, encapsulation with fixed randomness, and decapsulation including implicit rejection.
The one thing we would like to raise, since the code is not released yet, is about the interface. ml_kem_*_encap() and ml_kem_*_decap() return void, so they have no way to refuse a key, and FIPS 203 asks for two input checks:
- Encapsulation key (section 7.2, modulus check). A key whose coefficients are not reduced modulo q is accepted: full_decode() in ml-kem-internal.c reduces the coefficients silently, while the hash of the key is computed over the original bytes. 112 of the invalid Wycheproof keys for ML-KEM-768 and 116 for ML-KEM-1024 go through.
- Decapsulation key (section 7.3, hash check). _ml_kem_decap() uses the stored H(ek) and the embedded ek without comparing them. The two Wycheproof vectors per parameter set with a corrupted hash or a corrupted embedded key go through.
The impact is small: no key material is disclosed. It is a conformance point, and it means Nettle can disagree with other implementations about which keys are valid. It seemed worth mentioning before 4.1, while a return value can still be added without breaking anyone.
This work was carried out with AI assistance (Claude Fable 5.2); we have run everything reported here ourselves.
RatioLabs https://www.ratiolabs.eu info@ratiolabs.eu -------------------- End of forwarded message --------------------
nettle-bugs@lists.lysator.liu.se