atty on Sun 3

[Guy Harris]

>Atty core dumps because it calls abort. It calls abort because an pty
>ioctl fails. The ioctl fails because it tries to set terminal modes
>on the master side of the pty instead of on the slave side.
>
>I don't know if that's possible on other systems but it won't work on
>SunOS 4.0 suns, see the enclosed program:

OK, here's the scoop.

	1) On the vanilla BSD pseudo-tty driver, you can do tty "ioctl"s
	   on either the master or the slave side.  This works because
	   the pseudo-tty driver can get at the slave side's line
	   discipline from the master side.

	2) The SunOS 4.0 driver does not make this easy, being
	   streams-based.  The streams-based slave-side driver doesn't
	   know anything about the modules pushed above it (except that
	   it expects them to understand a couple of specialized
	   messages to turn input processing on and off, so that remote
	   mode works).

	   However, there is a truly slimy kludge in the 4.0 pseudo-tty
	   driver that lets it work *most* of the time for the *BSD*
	   terminal "ioctl"s.  It does *not* work for the System V-style
	   terminal "ioctl"s, because, frankly, I didn't give a damn about
	   making them work on the master side; the only reason the kludge
	   is there is so that old binaries, and crufty programs written
	   for BSD that bang on the master side rather than the slave
	   side, work.

	   Furthermore, if the slave side isn't open, there *is* no
	   line discipline there for on which the "ioctl"s can even act,
	   so even the aforementioned kludge wouldn't be sufficient; it
	   might work with even more kludgery, but I figured I'd reached
	   the point of diminishing returns by then....

	   I've not yet run across any program that couldn't be fixed
	   to do the "ioctl"s on the slave side; I expect the System V
	   Release 4 pseudo-tty driver not to bother making them work
	   on the master side, so I suggest people not do tty "ioctl"s
	   on the master side in new code.  (I also expect the S5R4
	   pseudo-tty driver to have some hooks that will permit
	   programs like "atty" to be cleaned up a fair bit....)

The reason your test program fails is that it's doing the S5-style
"TCGETS" on the master side.  Were you to change it to do a BSD-style
"ioctl" such as TIOCSETP, it would fail because you don't have the slave
side open.

The first problem doesn't apply to "atty", since it doesn't use the
S5-style "ioctl"s.  The second problem does, because it tries to set the
modes on the master side before it even opens the slave side.

The following patch to "atty.c" seemed to fix the problem (I was able to
start "atty" up, and type at it a bit, at least); it moves the setting
of the initial pseudo-tty modes from code in the parent which bangs on
the master side to code in the child that bangs on the slave side after
it's opened):

*** atty.c.ORIG	Sat Jun  3 16:25:00 1989
--- atty.c	Sat Jun  3 16:34:32 1989
***************
*** 215,222 ****
  #endif
        if (ioctl(ptyfd, TIOCPKT, (char *)&one) < 0)
  	    fatal("TIOCPKT");
-       if (setttymodes(ptyfd, &ptymodes) < 0)
- 	    fatal("set pty modes");
        fillttymode();
        copyttysize();
        if ((childpid = fork()) == -1) {
--- 215,220 ----
***************
*** 311,316 ****
--- 309,316 ----
        close(2);
        dup(0);				/* file descriptor 1 = dup of 0 */
        dup(0);				/* file descriptor 2 = dup of 0 */
+       if (setttymodes(2, &ptymodes) < 0)
+ 	    fatal("set pty slave modes");
        if (ioctl(2, TIOCSPGRP, (char *)&mypid) < 0)
  	    printf("TIOCSPGRP failed, errno=%d\n", errno);
        basename = shell;