Benchmarking UNIX - bibliography (LONG)

[STUDENTS]

We recently put an inquiry on the net for information about benchmarking
UNIX systems, the response to that message brought us much information which
we are presently collating.

We have, in the meantime, built up a 'refer' data-base of the bibliography
of UNIX specific references which we now offer to the net for recommendations
and further inclusions.  If we have missed off your most prized work, we first
apologise and then ask you to get in touch and let us know.

We also take this opportunity to thank all those who mailed us and haven't
yet received a reply.

Jim Lyons for CSSD

UUCP ...!ukc!cheviot!cssd
ARPA cssd%cheviot.newcastle at ucl-cs.ARPA
JANET cssd at newcastle.cheviot

----------------------------------------------------------------------------

%A J. Feder
%T The Evolution of UNIX System Performance
%J AT&T Bell Laboratories Technical Journal
%V 63.8
%D October 1984
%P 1791
%O The definitive work on UNIX development from the team who built it.
A good article giving an historic view on benchmarking in the Bell Labs.
Shows the improvement gained over various issues of UNIX and improvement in
performance brought about by changing algorithms to suit new hardware.
 
%A D. F. Hinnant
%T Benchmarking UNIX systems
%J BYTE
%V 9.8
%D August 1984
%P 132
%O This is an article entirely about benchmarking UNIX systems.
The benchmarks measure performance at three different levels, the system
level, the command level and multi-user level.  The benchmarks are constructed
with the minimum of overhead and this article describes them and the results
achieved on various systems.
 
%A A. S. Tanenbaum
%T Two programs, many UNIX systems.
%J EUUG
%V 3.4
%P 24-26
%D Winter 1983
%O A comparison of two benchmarks run on many systems.
The article shows the results of two test programs. The two test programs
are run on a wide variety of machines from single-user to multi-users.
One test program measures CPU/memory speed and the other measures I/O speed.
Care must be taken when interpreting the results. Some vendors may have
carefully selected comparisons to show how fast a particular machine is.
The article actually said that "figures don't lie, but liars figure".
 
%A B. Clark
%T UNIX - the horse before the cart
%J Computer Systems
%V 4/11
%P 85-87
%D November 1984
%O Machine architectures built specifically to run Unix.
A description of a machine designed to run Unix, rather than
tailoring Unix to suit the hardware.  The article discusses hardware
features which complement Unix, especially the advantages of using a
RISC architecture with many registers for the Unix machine.  Some
crude performance figures are given of the RISC machine against a
VAX 11/780 and a Motorola 68000.
 
%A Bob Critchlow
%T Optimising Unix performance
%J Computer Systems
%V 3/11
%P 73-76
%D November 1983
%O Comparing Intel System 286/310 performances running Unix to other machines.
A review of the Intel System 286/310 computers which run many versions of
Unix. The results of six types of benchmarks, run on six different manufacturers
machines are presented. This article is generally a 'plug' for Intel - the
author Bob Critchlow is Intel's technical systems marketing manager.
 
%A Mike Pearce
%T Who needs high capacities
%J Computer Systems
%V 3/11
%I Techpress
%C Bromley
%D November 1983
%P 4
%O An article which primarily discusses the electro-mechanical features of a
range of Winchester disk drives.
It makes passing reference to Unix's need for a high performance disk
and plots transfer rate against access time for a range of disk systems to
show that disk access time is a critical factor.
 
%A M. K. McKusick
%A W. N. Joy
%A S. J. Leffler
%A R. S. Fabry
%T A Fast File System for Unix
%I Computer Systems Research Group, Dept of EECS, Berkeley, Ca.
%J Technical Report #8
%O This paper sets out to describe the differences in file systems between
version 4.1 and 4.2
This paper addresses itself to the enhancements and changes made to the
Berkeley UNIX version 4.1 to bring it to version 4.2,  it discusses changes to
block sizes and access algorithms and related subjects.
 
%A L. F. Cabrera
%T Benchmarking Unix - A comparative study
%B Experimental Computer Performance Evaluation
%I North-Holland Publishing Company
%P 205-215
%O Compares UNIX performance at the user level, performance is defined as
response time against number of processes.
Article explains the method and results of the high level benchmarking
of three UNIX systems. Three 'typical' jobs were devised and run periodically
on each system while that system is loaded with real processes.  Response times
for these jobs are marked against a number of processes.
This is a useful article. It also notes that it would be better to have a
load that is reproducible on any system, rather than rely on real loads whose
characteristics are subject to variation over the systems.
 
%A E. D. Lazowska
%T The Benchmarking, Tuning and Analytic Modelling of VAX/VMS
%J Performance Evaluation Review
%V 8.3
%D Fall 1979
%C Department of Computer Science, University of Washington, Seattle
%P 57-64
%O Benchmarking, tuning and modelling the DEC's VMS executive with a strong
emphasise in modelling.
This work arises from the performance problems which the author
encountered. For example, the response time for simple editing requests
takes several minutes; it takes between fifteen minutes to three hours
for compilation and the system capacity is four active users. An analytic
model was designed for performance prediction. The author has presented a
good analytic model as the predictions from the model are very close to the
actual results obtained.
 
%T Benchmark report : Datasystem 355
%J Performance Evaluation Review, Sigmetrics
%V 9.3
%D Fall 1980
%P 7-30
%O A report of the benchmark carried out on 13 users of DEC's 11/34 based
systems.
This article is about benchmarking the DEC Datasystem 355. It contains the
benchmark process, the benchmark results, details about the system and its
peripherals and a summary of user comments. Article looks into how well the
equipment performs in specific applications and that performance compares with
other computing systems. It also concerns with subjective factors like ease of
use, versatility and the support service. The benchmark measures speed and
response time against the number of terminals. The response time increases as
the number of terminals has increased.
 
%A Robert G. Estell
%T Benchmarks and Watermarks
%J Performance Evaluation Review, Sigmetrics
%V 9.3
%D Fall 1980
%O The article consists of the author's opinions and some points to consider
when benchmarking the performance measure of computers. Include some valuable
points.
 
%A S. Spanier
%T Sun-3 Benchmarks
%J Sun Microsystems internal memo
%D August 1985
%O A wide range of benchmarks results comparing the SUN-3 machine and other
machines like IBM PC, VAX, APOLLO.
This article characterises the performance of the SUN-3 machine and provides
an aid for people choosing a workstation. Benchmark, in this article, is
defined as a test of the performance of one or more elements of a computer
system. A wide range of benchmarks were carried out to show how good the SUN-3
performances are as compared with some other machines.
 
%A David Hough
%T Weitek 1164/5 Floating Point Accelerators
%O Article is about the Floating Point Accelerator (FLA) measured at the SUN.
It talks about the performance of SUN-3 Floating Point Accelerator.
Suggested to use the Linpack benchmark for measuring performance on scientific
and engineering floating point calculations because it measures the performance
of hardware and compiler in an indisputable way on a realistic computation, and
the Livermore loop benchmarks for larger inner loops.
 
%A T. Lang
%A J. P. Newbury
%A D. E. P. Watkins
%T Experiences in Benchmarking for the Selection of an Interactive System
%J Software - Practice and Experience
%V 12
%P 531-542
%O Report on how a benchmark was constructed to assist in the selection of a
general purpose interactive computing system.
The article describes the stages in constructing a benchmark to measure the
hardware and software performances on a number of computer systems. The
problems the authors encountered and their experiences gained in this exercise
are also described in the paper.
Provides some assistance to anyone starting on a benchmarking exercise for the
first time.
 
%A B.A. Wichmann
%A H.J. Curnow
%T A Synthetic Benchmark: Whetstone
%J Computer Journal
%V 19 No: 1
%D February 1976
%P 43
%O Presents the background to the Whetstone benchmark and how it was constructed
originally to measure the performance of the ALGOL compiler.
The article describes the stages in constructing a benchmark to measure the
performances of a High Level Language, such as ALGOL on a number of computer
systems. The benchmark takes a number of commonly used statements and checks
their performance (timed events). The benchmark has also been translated in
other High Level Languages. There are a set of four performance measures that
are used.
Regarded as a 'standard' test for any benchmark suite of programs.
 
%A R.P. Weicker
%T Dhrystone: A Synthetic systems programming benchmark
%J Communications of the ACM
%V 27 No: 10
%D October 1984.
%P 1013
%O Presents the background to the Dhrystone benchmark and how it was constructed
to measure the performance of executing different types of High Level Language
programming statements.
The article describes the stages in constructing a benchmark to measure the
performances of a High Level Language on a number of computer systems.
This is basically very similar to the Whetstone benchmark, but perhaps more
widely used as a measure of performance.
The benchmark takes a number of commonly used statements (a mix of statements)
then 'times' them to evaluate their performance. The actual benchmark can be
run in two modes, with and without the use of 'extra' registers at execution
time. It has been run on many machines using the UNIX operating system.
Regarded as a 'standard' test for any benchmark suite of programs.
 
%A P. Marvit.
%A M. Nair.
%T Benchmark Confessions.
%B The UNIX System - Encyclopaedia.
%I Yates Ventures.
%D 1984.
%P 63.
%O Presents the whole concept of benchmarking UNIX systems and what factors need
to be considered beforehand and sets guidelines on the evaluation of results.
The article describes the philosophy and practice behind the whole concept
of benchmarking UNIX systems. It states that benchmarking UNIX is, at best,
problematic, and at worst, a gross distortion of reality. It points out the
loop holes that can easily be entered by misinterpreting the results obtained
and warns against these by setting guidelines on how to use the results.
This methodology would be very useful in our own work when we come to analyse
our results. It also points out the h/w and s/w considerations to be taken into
account when constructing your benchmark.
 
%A P. Marvit
%T A UNIX Benchmark Blitzkrieg
%B The UNIX System - Encyclopaedia
%I Yates Ventures
%D 1984
%P 69
%O Presents a set of six different benchmarks for UNIX systems.
The article looks into the  benchmark tests and briefly comments on the result
obtained, including the 'Whetstone' benchmark. It shows a table of results
obtained on several machines. But otherwise it is just listings of the benchmarks
themselves.