tgetent() of libtermlib does not use private termcaps correctly.

[Karl Kleinpaste]

While working with the  tenex  csh  distributed  around  Oct 1983, it became
obvious  that  the vb capability advertised by the man page is not  in  fact
available at all times. Specifically, it does not deal with any capabilities
in  a  personalized termcap file referenced by a TERMCAP env  variable.  The
problem is  that  termcap.c  in  libtermlib.a  has  a  bogus  check for what
constitutes  a  valid  file descriptor, and inadvertently  throws  away  the
reference to one's personalized termcap file.  This bug report is  for  both
4BSD and USG5.0, but I believe it applies to all libtermlibs.

Subject: tgetent() of libtermlib does not use private termcaps correctly.

Index: /usr/src/usr.lib/libtermlib/termcap.c 4.2BSD

Description:
	Routine tgetent() of libtermlib has an incorrect check for whether
	or not the open of a personalized termcap file (referenced by an
	environment variable TERMCAP) succeeds: it checks against 0 as an
	invalid descriptor, when in fact 0 is a perfectly reasonable (if
	somewhat unusual) file descriptor for use with an interactive
	program as something other than stdin. This may sometimes cause
	the reference to the personalized termcap file to be ignored, and
	/etc/termcap will be substituted.

Repeat-By:
	% vi ~/.termcap
	[Create a termcap entry for the current terminal from a similar
	entry in /etc/termcap.  Add a vb capability to the entry; anything
	will do, it doesn't matter what you use.  Make sure that the
	original /etc/termcap version of the termcap entry does not have a
	vb capability, or at least that it's a different vb.]
	% setenv TERMCAP ~/.termcap
	% csh				[Use tenex csh, not /bin/csh.]
	[Subshell starts up...next prompt is from that csh.]
	% ^[				[Hit an ESC.]
	[The csh will beep at you, using ^G, instead of the vb capability
	just added above in ~/.termcap, and will then erase the ^[ as it's
	supposed to.]

	The problem is that the open of ~/.termcap showed up as file
	descriptor 0, which caused tgetent() to ignore it.  (Csh moves
	stdin, stdout, and stderr away from 0, 1, and 2 frequently.)

Fix:
	The routine tgetent() must be fixed so that it checks for file
	descriptors less than zero as an "invalid" condition.  Diffs follow.

	*** termcap.c~	Thu Dec 13 18:24:33 1984
	--- termcap.c	Thu Dec 13 18:26:50 1984
	***************
	*** 44,50
	  	int tf;
	  
	  	tbuf = bp;
	! 	tf = 0;
	  #ifndef V6
	  	cp = getenv("TERMCAP");
	  	/*
	
	--- 44,50 -----
	  	int tf;
	  
	  	tbuf = bp;
	! 	tf = -1;	/* fixed initialization: used to be 0. */
	  #ifndef V6
	  	cp = getenv("TERMCAP");
	  	/*
	***************
	*** 66,72
	  		} else
	  			tf = open(cp, 0);
	  	}
	! 	if (tf==0)
	  		tf = open(E_TERMCAP, 0);
	  #else
	  	tf = open(E_TERMCAP, 0);
	
	--- 66,72 -----
	  		} else
	  			tf = open(cp, 0);
	  	}
	! 	if (tf < 0)		/* fixed test: used to be "tf == 0". */
	  		tf = open(E_TERMCAP, 0);
	  #else
	  	tf = open(E_TERMCAP, 0);

Workaround:
	For those who lack appropriate privileges to edit/recompile the
	source files for libtermlib, the following is a (generally) suitable
	fix for the problem:

	All occurrences of
		if (tgetent(term, area) < 0)
		{
			/* recover from no-termcap-file error condition. */
		}
	can be bracketed thusly:
		{
			int dummyfile = -1;
			dummyfile = open ("/dev/null", 0);
			if (tgetent (term, area) < 0)
			{
				/* recover from error as before. */
			}
			close (dummyfile);
		}

	This makes sure that file descriptor 0 is occupied by some open
	file, since Unix always assigns the lowest-available file descriptor
	when a new file is opened.

	Note that this will not always work.  Since it occupies another file
	descriptor, it may cause programs needing large numbers of files to
	reach the EMFILE "too many open files" conditon.
-- 
>From the badly beaten keyboards of him who speaks    +---best address
in textured Technicolor *TyPe* f-O-n-T-s...          |
						     |
Karl Kleinpaste @ Bell Labs, Columbus   614/860-5107 +-----> cbrma!kk
                @ Ohio State University 614/422-0915   osu-eddie!karl