I have put up a web page at
http://josefsson.org/gss/gss-lsh.html
with some information and the current patch. It works fairly well, but some minor issues in the specification probably doesn't work.
I hope I have fixed the problems mentioned earlier. Let me know what you think. I'm going to test it with other GSS libraries now.
ObWarning: I didn't review it for logical security flaws. There could be major holes.
I suspect some extra logic is needed, but perhaps not much. Consider a client sending
SSH_MSG_USERAUTH "gssapi" (starting a gssapi "session") SSH_MSG_USERAUTH "none" (restarting authentication) SSH_MSG_USERAUTH_GSSAPI_TOKEN
Then the client should get a protocol error, but it might make contact with your old handler which is still installed. It might be simple to fix, by having the handler for SSH_MSG_USERAUTH reset all handlers for messages in the userauth range.
Yes, sounds reasonable.
One additional thing would be required: how do I make the GC invoke a function (do_gc_gssapi() in the patch above) when a gssapi_server_instance class instance is gc'ed? The GSS library allocates some resources that should be deallocated if an authentication attempt is aborted.
Can gss_acquire_cred (or other gss-api functions) block, for example for contacting a kerberos server? Then the entire lshd server will block too.
What is the worst problem this could cause anyway? I can only think of the lsh server core being delayed when responding to the client, but the client should kind of expect this anyway, when it requests a GSS mechanism that takes a very long time to finish.
What will happen is that the lshd server will block and stop responding on *all* connections, for all users. So it's a denial of service attack on the other users of the system.
Oops, I didn't think of this. OK, it seems fairly clear that GSS should be invoked from a different process, which communicate with the lsh core using some protocol, then. Still, since kerberos 5 GSS mechanism is fast, this is probably not a show-stopper. I think I'll leave this as an exercise.
Thanks for the guidance, Simon