I've been made aware of a bug in Nettle's code to verify ECDSA signatures. Certain signatures result in the ecc point multiply function being called with out-of-range scalars, which may give incorrect results, or crash in an assertion failure. It's an old bug, probably since Nettle's initial implementation of ECDSA.
I've just pushed fixes for ecdsa_verify, as well as a few other cases of potentially out-of-range scalars, to the master-updates branch. I haven't fully analysed the implications, but I'll describe my current understanding.
I think an assertion failure, useful for a denial-of-service attack, is easy on the curves where the bitsize of q, the group order, is not an integral number of words. That's secp224r1, on 64-bit platforms, and secp521r1.
Even when it's not possible to trigger an assertion failure, it's easy to produce valid-looking input "signatures" that hit out-of range intermediate scalar values where point multiplication may misbehave. This applies to all the NIST secp* curves as well as the GOST curves.
To me, it looks very difficult to make it misbehave in such a way that ecdsa_verify will think an invalid signature is valid, but it might be possible; further analysis is needed. I will not be able to analyze it properly now, if anyone else would like to look into it, I can provide a bit more background.
ed25519 and ed448 may be affected too, but it appears a bit harder to find inputs that hit out of range values. And since point operations are inherently more robust on these curves, I think they will produce correct results as long as they don't hit the assert.
Advise on how to deal best with this? My current plan is to prepare a 3.7.2 bugfix release (from a new bugfix-only branch, without the new arm64 code). Maybe as soon as tomorrow (Wednesday, european time), or in the weekend.
Regards, /Niels