Ok, this is a little embarassing... As some of you probably have noticed, I managed to get vacation set up wrong, and I've had some mail bounces. And the funny part is that mailman decided I shouldn't get any more lsh-bugs mail.
Anyway, I've now reenabled mail delivery, and I have read the TCP_NDELAY discussion in the mail archive. I'm not terribly familiar with Nagle, but I have some comments:
1. When having an interacting session, and sending two keystrokes "X" and "Y", one may end up with loads of IP packets:
Client Server ---> "X" First keystroke (a) <--- TCP ACK Ack of the data (b) <--- SSH ADJ SSH_MSG_WINDOW_ADJUST (c) ---> TCP ACK Ack of above message (d) <--- "X" The server pty echoes the data (e) ---> TCP ACK Ack of the echoed data (f) ---> SSH_ADJ SSH_MSG_WINDOW_ADJUST (g) <--- TCP ACK Ack of the above message. (h)
---> "Y" Next keystroke (i) ... Etc
Then TCP uses extra delays to try to get the acks and echoes into fewer IP packets, in order to waste less bandwidth. In general, that seems like a good thing to do, and ideally, (b), (c) and (e) should get lumped together into one IP packet, and so should (d), (f), (g) and (i).
2. I'm not sure which part of this trick is the "Nagle algorithm" , and disabled, and I'm not sure in which direction it is most important. To optimize this, one would need to look at some packet traces.
3. I'm not personally experiencing any lagginess, but perhaps I'm too tolerant, or use a too fast network (I don't use anything slower than adsl on a daily basis. At the moment, I'm typing over an interactive lsh/lshd connection over a 9 hop network, 10MB at my end point, and a 10ms roundtrip time). But I'm sure this is system and IP-stack dependent, so I'll be happy to add TCP_NDELAY call, if that helps for other people.
4. There's also one additional source of small delays in lsh: When setting the client tty into raw mode, lsh fiddles with the VMIN and VTIME values (exactly if and how has varied a little between versions). The intent is that if you type reasonably fast, there should not be a simple one-to-one correspondence between keystrokes and IP-packets, to make it a little harder for an eavesdropper to observe your typing pattern. Again, some packet traces would help to determine if this really works well. The code is in unix_interact.c:
static struct terminal_attributes * do_make_raw(struct terminal_attributes *s) { CAST(unix_termios, self, s); CLONED(unix_termios, res, self);
CFMAKERAW(&res->ios);
/* Modify VMIN and VTIME, to save some bandwidth and make traffic * analysis of interactive sessions a little harder. */ res->ios.c_cc[VMIN] = 4; /* Inter-character timer, in units of 0.1s */ res->ios.c_cc[VTIME] = 1;
return &res->super; }
This should result in an extra delay of MAX(0.1s, 3 * your intertyping time).
I'd be happy for corrections to this description, as well as experiments with changed TCP and TTY options. The patch that adds an setsockopt(..., TCP_NODELAY) to io.c:io_connect seems to hack at the right place for the client. There should probably be configurable so that it is used only when running under a tty. For the server side of things, the corresponding funtion seems to be io.c:do_listen_callback.
As some noticed, after editing io.c, the Makefile will want to recreate io.c.x, which will fail if you don't have guile or scsh installed. For these changes, it should be safe to just touch io.c.x to convince make that it is up-to-date.
Best regards, /Niels
Thanks Pontus and Niels, for your responses.
Niels Möller wrote...
- I'm not personally experiencing any lagginess, but perhaps I'm too tolerant, or use a too fast network (I don't use anything slower than adsl on a daily basis. At the moment, I'm typing over an interactive lsh/lshd connection over a 9 hop network, 10MB at my end point, and a 10ms roundtrip time). But I'm sure this is system and IP-stack dependent, so I'll be happy to add TCP_NDELAY call, if that helps for other people.
I have been testing two different arrangements - first is lsh to lshd on localhost, and second is using the Putty client on a Win32 box across the LAN to lshd (Putty has the Nagle algorithm disabled by default). Both lsh and Putty are using aes256-cbc. I don't think the network itself should be a factor. Also worth noting is that this is my first significant use of SSH2 - I've been using the original ssh package. I just now tested OpenSSH (across a T1) and found it to have some very minor jump/lag - not as smooth as the old ssh, but not nearly as uncomfortable as I'm seeing with lsh.
There's also one additional source of small delays in lsh: When setting the client tty into raw mode, lsh fiddles with the VMIN and VTIME values (exactly if and how has varied a little between versions). The intent is that if you type reasonably fast, there should not be a simple one-to-one correspondence between keystrokes and IP-packets, to make it a little harder for an eavesdropper to observe your typing pattern. Again, some packet traces would help to determine if this really works well. The code is in unix_interact.c:
static struct terminal_attributes * do_make_raw(struct terminal_attributes *s) { CAST(unix_termios, self, s); CLONED(unix_termios, res, self);
CFMAKERAW(&res->ios); /* Modify VMIN and VTIME, to save some bandwidth and make traffic * analysis of interactive sessions a little harder. */ res->ios.c_cc[VMIN] = 4; /* Inter-character timer, in units of 0.1s */ res->ios.c_cc[VTIME] = 1; return &res->super;}
This should result in an extra delay of MAX(0.1s, 3 * your intertyping time).
Interesting - thanks. I tried commenting out the lines which modify VMIN and VTIME, and that didn't seem to make any difference. So for the moment I'm going to assume this isn't a significant factor and leave it as you have it.
I'd be happy for corrections to this description, as well as experiments with changed TCP and TTY options. The patch that adds an setsockopt(..., TCP_NODELAY) to io.c:io_connect seems to hack at the right place for the client. There should probably be configurable so that it is used only when running under a tty. For the server side of things, the corresponding funtion seems to be io.c:do_listen_callback.
I applied the patch that Pontus offered, and it did not seem to make a difference :-/ I then looked further through io.c and found an existing setsockopt in io_bind_sockaddr - and added a similar call to setsockopt setting TCP_NODELAY. This made my Putty->lshd sessions nice and smooth, but my lsh->lshd sessions were still jumpy. Applying this change _and_ Pontus' suggestion had the same effect (ie, Putty->lshd smooth, lsh->lshd remained jumpy). This makes me wonder if perhaps the change I came up with in io.c:io_bind_sockaddr effectively modifies the behavior of lshd, but the behavior of lsh isn't being modified by my change or Pontus'. Does that make any sense? Here is the snippet of io_bind_sockaddr which I modified...
{ int yes = 1; int nodelay = 1; setsockopt(s, SOL_SOCKET, SO_REUSEADDR, (char*)&yes, sizeof yes); setsockopt(s, IPPROTO_TCP, TCP_NODELAY, (char*)&nodelay, sizeof nodelay); }
I would add the TCP_NODELAY option in io.c:do_listen_callback to test that, but frankly I'm not sure where and how to do it :-/
As some noticed, after editing io.c, the Makefile will want to recreate io.c.x, which will fail if you don't have guile or scsh installed. For these changes, it should be safe to just touch io.c.x to convince make that it is up-to-date.
Aha! Thanks - this'll allow me to do some testing without bothering the list :) I don't know if I mentioned this already, but I'm working with GCC 3.2.3 on a Solaris 9 host, in case that's significant.
-
nisse＠lysator.liu.se -
Pete Naylor