SPEC Bench-a-Thon

[Robert E. Novak]

If you already saw this in comp.arch, skip it NOW!

During the week of June 5-9, the SPEC Bench-a-thon was held at MIPS
computers.  We had 7 vendors participating with 10 different architectures.
It was a grueling week for many of us.  In a single week we had to analyze 
24 different benchmarks and ensure that they would run on all of the
machines.  What we did not do was post results from running the benchmarks.

The participating vendors included:

Apollo				2 architectures
DEC					2 architectures and 2 Operating Systems
Hewlett-Packard		2 architectures
IBM
MIPS
Motorola
SUN

As a result of this meeting, the SPEC consortium produced an Alpha tape of
11 programs that could be run by all vendors.  This tape has been
distributed to all of the participants by now.  We are awaiting comments
from the participants to clean up any small flaws in the Alpha tapes and we
will generate a Beta release on June 30.  The Beta tape will undergo a 60
day evaluation period.  

This tape will be available to all companies that are members of 
the SPEC consortium.  The tape will be made publicly available at a to be
determined nominal cost at the end of the Beta period.

The tape contains the following benchmarks which have run on the machines
of the participants:

GCC 1.35	Integer intensive C compiler for CASE
espresso	C, Integer-intensive benchmark for ECAD
spice2g6	FORTRAN, Floating point intensive benchmark for ECAD
			applications (analog circuit simulation)
doduc		FORTRAN, Floating point intensive benchmark for ECAD
			applications (high-energy physics)[SIC].
nasker		FORTRAN, floating point intensive benchmark for scientific
			applications.
li			C, integer intensive benchmark for CASE
eqntott		C, integer intensive benchmark for ECAD
intmc		FORTRAN, integer intensive benchmark for scientific
			applications.
matrix300	FORTRAN, floating point intensive benchmark for wsceintific
			applications (Monte Carlo code that performs various matrix
			multiplications, including transposed Linpack routines).
fpppp		???
tomcatv		FORTRAN, floating point intensive benchmark for scientific
			applications.

These benchmarks are roughly 1/2 Fortran, 1/2 C, 1/2 integer and 1/2
floating point intensive.  Unfortunately, all of these benchmarks are
heavily CPU intensive and do very little to exercise the rest of a system.

The tape also contains a number of programs that are "Works in Progress."
These programs are candidates for future SPEC benchmark tapes.  They are as
follows:

Timberwolf
gbench		C, integer intensive benchmark for measuring X-terminal
			performance.
isajet		FORTRAN, floating point intensive (single prescision) benchmark
			for scientific applications (high energy physics
			applications).
mdljdp		FORTRAN, floating point intensive benchmark for scientific
			applications (particle simulation program--molecular dynamics).
wave		Maxwell's Equations?
spartan		Computational Chemistry

SPEC is actively soliciting for additional benchmarks.  There are several
requirements for the benchmarks:

1) They must be public domain or publicly available (e.g. GCC).
2) They should be highly portable.
3) They should run in a UNIX(tm) environment.
4) They should not use time() to seed the random number generator.
5) They should contain their own random number generator if they need one.
6) All calls to time() times() getrusage() et.al. calls should be removed.

SPEC is especially interested in benchmarks that exercise an entire system.
Areas that are under consideration include:
	disk I/O bandwidth
	network bandwidth
	DBMS performance in non-transaction environments (leave the
		transactions to TPC)
	Workstation support (i.e. how many workstations can a file/client
		server support?)

In all cases, it is highly desirable that the benchmarks be REAL
applications, not synthetic workloads.  It is okay if it is a crippled
version of a live application, but it should be exercising a real problem.

If you want more information about SPEC, contact Kim Shanley at Waterside
Associates, 39510 Paseo Padre Pkwy, Fremont, CA  94538, 415/792-2901

All of the usual disclaimers apply.  I speak only for myself... etc.
-- 
Robert E. Novak                                     MIPS Computer Systems, Inc.
{ames,decwrl,pyramid}!mips!rnovak      928 E. Arques Ave.  Sunnyvale, CA  94086
rnovak at abbott.mips.COM (rnovak%mips.COM at ames.arc.nasa.gov)       +1 408 991-0402