atty on Sun 3

[Guy Harris]

>I applied the patch to atty.c and re-compiled with gcc.  The result
>was that I'm able to get the initial prompt and type a few characters,
>but when I hit Return, I get the same core dump as before and,
>apparently, for the same reason.  Naturally, since atty(1) is not
>meant to run under System V and SunOS 4.0 is partially based on System
>V, it doesn't work too well here.

"Naturally" that has little, if anything, to do with the problem.  The
reasons "atty" won't run under S5 are that:

	1) it uses BSD-style tty "ioctl"s - the *documented* use of which
	   SunOS 4.0 supports;

	2) it uses BSD-style pseudo-ttys, and S5 didn't even *have*
	   pseudo-ttys until recently (I think S5R3.2 might have them,
	   albeit undocumented, for the benefit of e.g. "xterm" - but
	   not BSD-style ones) - but SunOS 4.0 has BSD-style pseudo-ttys;

	3) it may depend on other BSDisms - which SunOS 4.0 also has.

The SunOS 4.0 tty driver isn't based on the S5 one; parts of its
interface are, but the code isn't.  The SunOS 4.0 *pseudo*-tty driver is
what's at issue here, and it isn't based on the S5 one because S5 hasn't
*had* one until recently.  The problem is, again, that one particular
hack that wasn't ever *documented* as being supported by BSD-style
pseudo-ttys, namely the ability to do arbitrary tty "ioctl"s on the
master side, is being used; the fix is to do them on the slave side instead.

A patch that fixes, as far as I can tell, all of the instances where
it's doing tty "ioctl"s on the master side is:

*** atty.c.ORIG	Sat Jun  3 16:25:00 1989
--- atty.c	Mon Jun  5 11:57:49 1989
***************
*** 43,49 ****
  struct tty realtty;		/* tty modes of real tty */
  struct tty ptymodes;		/* modes of pty */
  int realfcntl;			/* flags for real tty file descriptor */
! int ptyfd;			/* file descriptor of pty */
  char ptydevice[12];		/* pty device */
  int childpid;			/* pid of child process */
  jmp_buf sigjmp;			/* where to jump on signal */
--- 43,50 ----
  struct tty realtty;		/* tty modes of real tty */
  struct tty ptymodes;		/* modes of pty */
  int realfcntl;			/* flags for real tty file descriptor */
! int ptyfd;			/* file descriptor of pty controller */
! int ttyfd;			/* file descriptor of pty slave */
  char ptydevice[12];		/* pty device */
  int childpid;			/* pid of child process */
  jmp_buf sigjmp;			/* where to jump on signal */
***************
*** 215,221 ****
  #endif
        if (ioctl(ptyfd, TIOCPKT, (char *)&one) < 0)
  	    fatal("TIOCPKT");
!       if (setttymodes(ptyfd, &ptymodes) < 0)
  	    fatal("set pty modes");
        fillttymode();
        copyttysize();
--- 216,225 ----
  #endif
        if (ioctl(ptyfd, TIOCPKT, (char *)&one) < 0)
  	    fatal("TIOCPKT");
!       if ((ttyfd = open(ptydevice, O_RDWR)) < 0) {
! 	    fatal("Can't open tty");
!       }
!       if (setttymodes(ttyfd, &ptymodes) < 0)
  	    fatal("set pty modes");
        fillttymode();
        copyttysize();
***************
*** 297,302 ****
--- 301,307 ----
        close(0);
        close(1);
        close(ptyfd);
+       close(ttyfd);
        fd = open("/dev/tty", O_RDWR);
        if (fd >= 0) {
  	    ioctl(fd, TIOCNOTTY, (char *)0);
***************
*** 790,796 ****
        int pgrp;
        int zero = 0;
  
!       if (ioctl(ptyfd, TIOCFLUSH, (char *)&zero) < 0)
  	    fatal("TIOCFLUSH");
        needclearin++;
        if (ioctl(ptyfd, TIOCGPGRP, (char *)&pgrp) < 0)
--- 795,801 ----
        int pgrp;
        int zero = 0;
  
!       if (ioctl(ttyfd, TIOCFLUSH, (char *)&zero) < 0)
  	    fatal("TIOCFLUSH");
        needclearin++;
        if (ioctl(ptyfd, TIOCGPGRP, (char *)&pgrp) < 0)
***************
*** 851,857 ****
        int changereal;
        int flags;
  
!       if (getttymodes(ptyfd, &ptymodes) < 0)
  	    fatal("get pty modes");
        noedit = 0;
  #ifdef notdef
--- 856,862 ----
        int changereal;
        int flags;
  
!       if (getttymodes(ttyfd, &ptymodes) < 0)
  	    fatal("get pty modes");
        noedit = 0;
  #ifdef notdef