Hello,
Nikos told me that there is a case where RSA-PSS signature verification leads to an assertion failure:
bignum.c:120: nettle_mpz_get_str_256: Assertion `nettle_mpz_sizeinbase_256_u(x) <= length' failed.
I thought it wouldn't be possible because 'x' is already rounded by the RSA modulus and 'length' is bound to the modulus.
However, actually 'length' is calculated as ((modBits - 1) + 7) / 8, i.e. one bit less than the original modulus. Thus, it would be possible that the octet length of 'x' exceeds 'length'.
I am attaching a patch for this.
Regards,