transient bug in vi nailed down

[Stephen J. Friedl]

Pre-note note:

	In this note, `function key' means a terminfo description
	for a function key, an arrow key, or any other kind of
	special key.

Description: 

	Vi produces a vgoto error or a core dump if a function-key
	sequence is interrupted for terminals that have a
	character-insert mode.  Is this strange or what?

Easily reproduced on:

	Machine		Operating System
	--------	------------------------------------
	3B2/310		System V Release 2.0.4
	3B2/400		System V Release 3.1
	3B2/600		System V Release 3.2.1
	3B15		System V Release 3.1.1
	6386E		AT&T System V Release 3.1
	6386 		Interactive System V/386  Rel 1.0.5
	AST Prem/286	Microport System V/AT Rel 2.3

	(enough?)

Terminals:

	ANSI terminals with (1) some function keys defined that
	begin with ESC and (2) any kind of character-insert mode
	(smir/rmir or ich).  Note that this usually does NOT
	include the vt100 because it lacks the character-insert
	capability.

How to reproduce:

	Put the following in a file.  Don't include the `cut here'
	lines, and the file should have 11 lines and 35 characters.

#------ cut here -----
x
x
x
x
x
x
x
x
x
x
xxxxxxxxxxxxxx
#------ cut here -----

     Now, go to the bottom line and enter insert mode.  As soon
as you hit ESCAPE to exit insert mode, *immediately* hit DELETE or
whatever is your local interrupt character.  On all systems tried,
we get a `vgoto error' from vi and get stuck in ex mode.

     One some machines (3B2/400 and Microport, at least) deleting
one of the `x' lines and doing the same things will cause a core
dump -- maybe this will help you developer-kind-of-folks track
this down.

     This problem goes away if there are no function key
sequences that start with ESCAPE (\E) or if the terminal has no
kind of character insert mode (either smir/rmir or ich1/ich).
Unfortunately, this is not a very useful fix for the ANSI
terminals that many of use :-(.

     The problem seems to relate to the confusion between ESCAPE
as the exit-insert-mode character and ESCAPE as the leading
character for function keys.  If vi sees an ESCAPE character, it
reads the next character with some kind of timeout (say, one
second).  If a character is read *before* this timeout
terminates, it is assumed to be part of a function key definition
and processed accordingly.  If the timeout does terminate, it
must have been a regular old ESCAPE key from the user, and it
usually terminates insert or append mode.

     The bug seems to show up if the DELETE key is used to
terminate a timeout read -- perhaps a signal handler is messing
up somewhere (where is `volatile' when you need it :-).

     If you are trying to track down this bug, there is an
additional different and *very* unfriendly manifestation of same.
Edit an empty file, enter append mode, hit ESCAPE and *quickly*
interrupt.  Vi will start spewing characters at high speed,
ignoring XOFF and interrupts, and will generally lock up the
terminal.  DO NOT DO THIS unless you have another terminal nearby
to kill your job.  It will sometimes run forever and will
sometimes core dump.

     If anybody tracks this down, I would love to hear about
the source of the problem.  If anybody is unable to reproduce
this on the abovementioned systems, I can send the terminfo
source that causes the problem.

     Steve

-- 
Stephen J. Friedl        3B2-kind-of-guy            friedl at vsi.com
V-Systems, Inc.        I speak for me only      attmail!vsi!friedl
Santa Ana, CA  USA       +1 714 545 6442    {backbones}!vsi!friedl
-------Nancy Reagan on