Slow [ NFS ] file update

[David Steffens]

In article <9767 at brazos.Rice.edu>
dz at cornu.ucsb.edu (Daniel James Zerkle) writes:
> A certain quirk with NFS is causing major headaches...
> Basically, my program uses three Unix machines:
> 1. The main program runs on a Sun 3/60.
> 2. The main program, via rsh, calls a secondary program on a Masscomp.
> 3. A Sun 3/160 (I think) is a file server for the above two machines.

Gasp! This is almost exactly the same configuration we are using
(substitute Sun4 for Sun3) and exactly the same problem we are having.  In
fact, I was just getting ready to submit something to SunSpots a couple of
weeks ago when I got distracted by other things.  IMHO, this is a Sun NFS
bug, since I can reproduce it using a simple pair of test programs, each
running on a Sun and using a third Sun as a server (i.e. _no_ Masscomp
involved).

> To summarize the operation:
> a. 1 calls program on 2
> b. 2 writes first 256 bytes of a file (open-write)
> c. 1 reads those 256 bytes (open-read-close)
> d. 2 writes rest of file (write-close-exit)
> e. 1 reads rest of file

Our paradigm is similar, but not identical:
a. 1 starts a program on 2 via rsh and pauses
b. 2 opens a file on 3 for writing and signals 1 that it is ready
c. 1 waits for 2 to signal ready then opens the same file on 3 for reading
d. 2 then continuously writes variable length hunks of data
   into the file on 3 and tells 1 how much was written each time
e. 1 loops reading and processing each hunk of data written by 2

> Right now, I have 1 do a periodic check to see if NFS has gotten it
> straight how big the file is.  In other words, 1 may wait about a minute
> while the file size gets straightened out.  This is totally unacceptable...

Some things which seem to improve the situation:
1. After writing a hunk of data on 2, it helps to have 2 do an fsync(2).
2. Before reading the data on 1, it helps to close the file, reopen it
   and then seek to the end of the data already read.

Kludge #1 is important for us but may not be for you.  Apparently, there
is some buffering on machine 2 for optimization of NFS writes since the
file on 3 always seems to grow by 8K increments no matter how much (or
little) is written on 2.  Therefore, Kludge #1 helps, but does not cure
the problem.

Kludge #2 seems to be very important also.  Apparently, 1 asks 3 for the
attributes of the file when the file is first opened and then caches the
results.  Since 1 now thinks that it knows  everything there is to know
about the file, it doesn't bother to interrogate 3 for the current file
attributes before each read, and thus doesn't see that the file has
changed size.  As a consequence of this, 1 won't ask 3 for any piece of
the file beyond the size it knows.  The close/open/seek seems to force 1
to check the attributes of the file against reality on 3 and update its
cache.  Just seeking around in the file is insufficient -- the file must
be closed and reopened for the seek to have the desired effect.

Kludge #2 improves the situation tremendously, although even both Kludges
taken together are not sufficient to guarantee success under all
conditions.  The residuum seems to be explainable by network delays,
however.

It is still a mystery to me what mechanism eventually causes 1 to catch up
with the state of affairs on 3 when 1 is left to its own devices.  I've
seen delays of a minute or more which don't seem to correlate with
anything.  As long as the program on 1 keeps the file open, the attributes
of the file remain incorrect, even as reported by standard utilities, e.g.
ls -l.

P.S. For the skeptics out there... source for the test suite is available
by mail from me.  If you can find something wrong with our approach or our
implementation, I'd like to hear about it!

{harvard,mit-eddie,think}!eplunix!das	David Allan Steffens
243 Charles St., Boston, MA 02114	Eaton-Peabody Laboratory
(617) 573-3748 (1400-1900h EST)		Mass. Eye & Ear Infirmary
	I'm a firm believer in learning from one's past mistakes...
	...but why should it take so many to get a good education?