I just realized that supporting RSA with SHA-512 leads to an interface problem.
For RSA keys to work with PKCS#1 signatures, the modulo n has to be large enough. Up to now, I have used two defines, RSA_MINIMUM_N_OCTETS and RSA_MINIMUM_N_BITS, which are determined so that pkcs#1 works with *all* supported hash function.
The rsa_public_key_prepare and rsa_private_key_prepare functions check that n is large enough and returns a success/failure value. Then no other rsa functions are expected to ever fail due to n being too small.
With sha256, the minimum was 62 octets or 489 bits. But for sha512, the minimum is 94 octets or 745 bits. So if I just bump the RSA_MINIMUM_N_* accordingly, support for 512 bits RSA keys disappears. (And for comparison, with md5, the minimum is 353 bits).
I really hope noone still uses 512 bit RSA for anything important, but I nevertheless don't think it's appropriate to just drop support for it. Some applications may have modest security requirements where small keys are somehow appropriate, and then there are legacy applications and old keys. It should still be *possible* to verify signatures made many years ago, using 512-bit RSA, even though they can no longer be considered very secure.
I haven't figured out yet how I want to handle this, and I see no solution without its own problems. Some alternatives:
* Discontinue the RSA_MINIMUM_N_*, or use some reasonable value which isn't large enough for rsa-sha512. Let signature functions abort if they encounter a too small key. Possibly define new hash-specific constants, RSA_<hash>_MINIMUM_N_BITS or so.
* Add some argument to the rsa_*_key_prepare functions to say what the key is to be used for, so that it can check if the key is large enough. I don't quite like this; it's against the spirit of the rest of the Nettle interface.
* Introduce a return value for the signing functions, to let them return a success/failure indication. This is analogous to the rsa_encrypt function, which checks if the key size is large enough for the current message size. If we do this, is the return value from the rsa_*_key_prepare functions still useful?
I think it's less of a problem with public key operations, since signature verification and rsa decryption have a return value and for too small keys they can easily return fail for all messages. The problem is the signing functions.
No matter how the problem is solved, it will break backwards compatibility in some way. Suggestions?
Regards, /Niels