Surprising Workstation Swap Behavior

[Paul Lutt]

We have approximately one hundred diskless Sun workstations in-house.
These systems are a mix of Sun-3/50, 3/60, 3/80, 4/60, and 4/65 systems,
with a few oddballs thrown in for extra flavor.  Our latest fileserver is
a Sun-4/390 serving twenty client workstations.  Our users are acquiring
workstations faster than we can acquire fileservers, so we would like to
increase the client load on this fileserver.  We decided to take a look at
swapping traffic to determine if a local hard disk for swapping would
reduce the fileserver load enough to support additional client machines.

We decided we needed to quantify what sort of load a client workstation
presented to our fileserver.  We traced NFS traffic on the network using a
network analyzer, which generated a packet trace file that we
post-processed on a workstation.

The configuration examined was a Sun-3/390 fileserver running SunOS 4.0.3
with twenty SparcStation 1 clients.  All of the workstations have between
8 MBytes and 16 MBytes of RAM.  We were interested in determining what
portion of the overall workstation traffic was devoted to swapping, as
opposed to binary file reads and user data file reads and writes.

Below are the results of a one hour sample of fileserver traffic.

                                           % of Total
		   READ	    % of Total     Read/Write
    Filesystem   Requests  Read Requests    Requests       Comments
   ------------  --------  -------------  -----------  ----------------
   /export/exec       6        0.4%           0.1%
   /export/root      73        4.4%           0.8%     client root
   /export/swap     547       32.7%           6.0%     client swap
   /orion/usr1      366       21.9%           4.0%     local binaries
   /orion/usr3       73        4.4%           0.8%     home directories
   /orion/usr4       42        2.5%           0.5%     home directories
   /usr             565       33.8%           6.2%     sun binaries
		      	     ------
		      	     100.0%

                                            % of Total
		  WRITE       % of Total    Read/Write
    Filesystem   Requests   Write Requests   Requests        Comments
   ------------  --------   --------------  -----------  ----------------
   /export/root     321          4.4%          3.5%      client root
   /export/swap    6678         90.5%         73.8%      client swap
   /orion/usr3      231          3.1%          2.6%      home directories
   /orion/usr4      144          2.0%          1.6%      home directories
                               ------
                               100.0%

We initially expected that there would be a sizable amount of swapping
traffic, given the diskless configuration of the workstations.  Indeed,
writes to the swapping files were by far the largest single source of
traffic to the fileserver.  Imagine our surprise, however, when we
discovered that only a fraction of the data written to the swap files was
ever read back by a workstation.  The ratio of write requests to read
requests was about 12 to 1.

The swapping behavior is very interesting.  It appears that the virtual
memory system in SunOS 4.0.3 tries to write out memory pages long before
there is a need to reclaim any pages.  Each time a new program is started,
there is a corresponding flurry of writes to the swap file.  On my 16MByte
SparcStation, I have seen cases where there have been hundreds of swap
writes without a single swap read.

Given the synchronous nature of NFS writes, this swapping traffic
adversely affects the performance of both the workstations and the
fileserver.

Upon seeing these results, we arranged to borrow an internal hard disk for
a SparcStation to test as a swapping disk.  The impression is that the
hard disk helps interactive performance significantly, while reducing the
overall network traffic.

The $64,000 question is:  Has Sun done anything in SunOS 4.1 to address
this problem?  One of the claims is that SunOS 4.1 has been "tuned" to
provide better performance on a memory starved Sun-3/50.  Does this mean
that it will improve the Sun-3/50 performance at the expense of the newer
systems?  Any one out there have the answer?  

Paul Lutt 
Domain: pwl at tc.fluke.COM