v03i061: xbench -- benchmark X servers, Part04/04

[Dan Heller]

Submitted-by: Claus Gittinger <unido!sinix!claus>
Posting-number: Volume 3, Issue 61
Archive-name: xbench/part04

#! /bin/sh
# This is a shell archive.  Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file".  To overwrite existing
# files, type "sh file -c".  You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g..  If this archive is complete, you
# will see the following message at the end:
#		"End of archive 4 (of 4)."
# Contents:  xbench.doc
# Wrapped by x11 at babel on Thu Mar 23 14:08:04 1989
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'xbench.doc' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'xbench.doc'\"
else
echo shar: Extracting \"'xbench.doc'\" \(14481 characters\)
sed "s/^X//" >'xbench.doc' <<'END_OF_FILE'
X.. @(#)xbench.doc	1.7 89/03/22
X.RP
X.TL
xbench - a set of benchmark tests for X servers
X.AF ""
X.AU
C. Gittinger
X.AI
Siemens AG Munich Dep. D ST SP 4
X.NH 1
How to run xbench
X.LP
XFor a first test, you should run:
X.nf
X.sp
X    \fBxbench -ask -sync -timegoal 5 -nruns 1\fP
X.fi
X.sp
to find out which tests work correctly on your
machine (look carefully at the output of the dashed-line test,
the arc tests and the tile/stipple rectangle fill tests).
X.br
If your server doesn't perform an operation correctly (or crashes),
it should
be excluded from the test. It wouldn't be fair, to compare a server
doing dashes with another one which draws solid lines instead.
X.sp
X.I Xbench
will ask you for some information about the client/server machine -
these are for documentation only and will appear in the output listing.
You may just enter <return> here if you're not going to publish the
benchmark - if you send it to some newsgroup it is recommended to
describe the server and client configuration here. (for example:
no FPU, graphic accelerator, 8mb main memory ...).
X.sp
You can also put everything into a script-file and feed this onto
X.I xbench.
The distribution contains a prototype-script called "script.run" -
change this according to your needs.
X.br
Then start the real
X.I xbench
run by calling:
X.nf
X.sp
X    \fBxbench -ask < script.run > xbench.out\fP
X.fi
X.sp
On some machines (those which draw nothing when screen-saver is active)
the screen saver should be turned off before:
X.sp
X.nf
X    \fBxset s off\fP
X.fi
X.sp
If your server turns off the cursor while drawing so that
you get different performance data depending in the position of the
cursor, you should probably leave the cursor in the draw area -
since this is the way you usually work with the system;
or run the test twice (first run with cursor in the upper left,
X2nd run with cursor in the lower right).
X.NH 1
XXbench tests
X.LP
XEach test in xbench has associated with it a
X.I level
number.
If you run
X.I xbench
without a level argument, only the level 1 tests are performed.
X.sp
XFor comparison of different server implementations, the level 1 tests
give you enough information, while the higher levels are mostly useful for
server tuners.
X.LP
The following chapters describe the individual tests. The tables
include the test-names, the level and a short description of what is done.
X.br
The last 2 cols of the tables are the reference value and the weight for
the 
X.I xstone 
rating - this will be described in more detail in the next chapter.
X.KS
X.NH 2
Solid lines
X.LP
The
X.B line*
benchmark draws horizontal, vertical and diagonal
lines of different length:
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
line2!2!vectors of len 2!-!0
line10!1!len 10!2616!300
line100!1!len 100!1631!400
line400!1!len 400!644!300
line500!2!len 500!-!0
X_
X ! ! ! ! 1000
X.TE
X.KE
X.KS
X.NH 3
Lines more detail
X.LP
Horizontal, vertical and diagonal lines can be tested separately with:
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
hline2!3!horizontal vectors of len 2!-!0
hline10!2!len 10!-!0
hline100!2!len 100!-!0
hline400!2!len 400!-!0
hline500!3!len 500!-!0
X=
vline2!3!vertical vectors of len 2!-!0
vline10!2!len 10!-!0
vline100!2!len 100!-!0
vline400!2!len 400!-!0
vline500!3!len 500!-!0
X=
sline2!3!sloped (diagonal) vectors of len 2!-!0
sline10!2!len 10!-!0
sline100!2!len 100!-!0
sline400!2!len 400!-!0
sline500!3!len 500!-!0
X=
X.TE
X.KE
X.KS
X.NH 2
Dashed lines
X.LP
The
X.B dline*
benchmark draws dashed lines.
The dashes are [4,4]-dashes meaning 4 dots on, 4 dots off.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
dline2!2!dashed vectors of len 2!-!0
dline10!1!len 10!819!50
dline100!1!len 100!172!150
dline400!1!len 400!49!100
dline500!2!len 500!-!0
X_
X ! ! ! ! 300
X.TE
X.KE
X.KS
X.NH 2
Wide lines
X.LP
The
X.B wline*
benchmark draws wide lines.
The linewidth is 5.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
wline2!2!wide vectors of len 2!-!0
wline10!1!len 10!78!50
wline100!1!len 100!19!100
wline400!1!len 400!6!50
wline500!2!len 500!-!0
X_
X ! ! ! ! 200
X.TE
X.KE
X.KS
X.NH 2
Rectangles
X.LP
The
X.B rects*
benchmark draws rectangles.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
rects2!2!2x2 rectangles!-!0
rects10!1!10x10!1115!150
rects100!1!100x100!797!250
rects400!1!400x400!414!150
rects500!2!500x500!-!0
X_
X ! ! ! ! 550
X.TE
X.KE
X.KS
X.NH 2
XFilled rectangles
X.LP
The
X.B fillrects*
benchmark draws solid filled rectangles.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
fillrects2!2!2x2 rectangles!-!0
fillrects10!1!10x10!1245!350
fillrects100!1!100x100!694!450
fillrects400!1!400x400!157!350
fillrects500!2!500x500!-!0
X_
X ! ! ! ! 1150
X.TE
X.KE
X.KS
X.NH 2
Tiled rectangles
X.LP
The
X.B tiledrects*
benchmark draws tiled rectangles.
The tile used is a 32x32 tile (which is most common).
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
tiledrects2!2!2x2 rectangles!-!0
tiledrects10!1!10x10!1387!150
tiledrects100!1!100x100!323!250
tiledrects400!1!400x400!48!150
tiledrects500!2!500x500!-!0
X_
X ! ! ! ! 550
X.TE
X.KE
X.KS
X.NH 2
Stippled rectangles
X.LP
The
X.B stippledrects*
benchmark draws stippled rectangles.
The stipple is a 32x32 stipple.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
stippledrects2!2!2x2 rectangles!-!0
stippleddrects10!1!10x10!835!50
stippleddrects100!1!100x100!153!100
stippleddrects400!1!400x400!35.6!50
stippleddrects500!2!500x500!-!0
X_
X ! ! ! ! 200
X.TE
X.KE
X.KS
X.NH 2
Inverting rectangles
X.LP
The
X.B invrects*
benchmark inverts rectangles on the screen.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
invrects2!2!2x2 rectangle invert!-!-!0
invrects10!1!10x10!3440!100
invrects100!1!100x100!938!150
invrects400!1!400x400!134.5!100
invrects500!2!500x500!-!0
X_
X ! ! ! ! 350
X.TE
X.KE
X.KS
X.NH 2
Arcs
X.LP
The
X.B arc*
benchmark draws arcs of varying angle.
The angle-range is 5 to 360 degrees.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
arcs2!2!2x2 box!-!0
arcs10!1!10x10!5.4!50
arcs100!1!100x100!1.2!100
arcs400!1!400x400!0.53!50
arcs500!2!500x500!-!0
X_
X ! ! ! ! 200
X.TE
X.KE
X.KS
X.NH 2
XFilled arcs
X.LP
The
X.B filledarc*
benchmark draws filled arcs of varying angle.
The angle-range is 5 to 360 degrees.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
filledarcs2!2!2x2 box!-!0
filledarcs10!1!10x10!25.3!50
filledarcs100!1!100x100!6.0!100
filledarcs400!1!400x400!2.7!50
filledarcs500!2!500x500!-!0
X_
X ! ! ! ! 200
X.TE
X.KE
X.KS
X.NH 2
XFilled polygons
X.LP
The
X.B filledpoly*
benchmark draws a filled polygon with 5 points.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
filledpoly10!2!vector len 10!-!0
filledpoly100!1!len 100!100!200
X_
X ! ! ! ! 200
X.TE
X.KE
X.KS
X.NH 2
BitBlt's
X.LP
A number of different bitblt tests are available:
X.br
Screen to Screen copy, its special case with srcX == dstX
X(i.e. scrolling), pixmap to screen copy using XCopyArea and
bitmap to screen using XCopyPlane (which is more often used on color
systems). For monochrome systems, pixmap to screen and bitmap to screen
is the same operation.
The scroll test copies a 640x400 window;
it has been added since the server may have a special-tuned 
function for this and perform scrolling faster than general bitblt's.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
screencopy2!2!screen->screen copy 2x2 rectangle!-!0
screencopy10!1!10x10!226!50
screencopy100!1!100x100!95!150
screencopy400!1!400x400!17.8!100
screencopy500!2!500x500!-!0
X=
scroll!1!screen->screen (640x400) scroll!68!1000
X=
pixmapcopy2!3!pixmap->screen copy 2x2 pixmap!-!0
pixmapcopy10!2!10x10 pixmap!-!0
pixmapcopy100!2!100x100 pixmap!-!0
pixmapcopy400!2!400x400 pixmap!-!0
pixmapcopy500!3!500x500 pixmap!-!0
X_
bitmapcopy2!2!bitmap->screen copyplane 2x2 bitmap!-!0
bitmapcopy10!1!10x10 bitmap!303!50
bitmapcopy100!1!100x100 bitmap!68.3!100
bitmapcopy400!1!400x400 bitmap!10.7!50
bitmapcopy500!2!500x500 bitmap!-!0
X_
X ! ! ! ! 1500
X.TE
X.KE
X.KS
X.NH 2
Drawing characters
X.LP
The
X.B imagestring*
benchmark draws a string. The font most likely to be found on
every machine (all releases) is "fixed".
The other fonts are propably not on every machine.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
imagestring:fixed!1!characters font=fixed!5636!3000
imagestring:6x12!2!font=6x12!-!0
imagestring:8x13!2!font=8x13!-!0
imagestring:9x15!2!font=9x15!-!0
X_
X ! ! ! ! 3000
X.TE
X.KE
X.KS
X.NH 2
Complex test
X.LP
All of the above tests are low level tests, measuring mainly the
performance of the graphics engine/ddx code.
To get a feeling of overall server performance when dealing with windows,
graphic contexts etc., this tests creates a window, creates a GC,
clears an area,
draws some text, scrolls the window and finally destroys the window.
X.TS
X.box, tab(!);
c | c | c | c | c | c
l | c | r | r | r | r.
name!level!description!ref.value!weight
X=
complex1!1!create/draw/destroy!18.6!600
X_
X ! ! ! ! 600
X.TE
X.KE
X.KS
X.NH 1
Output
X.LP
The output of
X.B xbench
consists of:
X.NH 2
Absolute performance data
X.LP
The absolute run time in seconds and the number of runs per second
are printed. Some tests (for example fillrects) also output the
number of pixels drawn per second.
X.NH 1
Statistics and xStones
X.LP
To compare different servers performances, the absolute performance
data (as output by
X.I xbench)
can be processed by some awk-scripts to compute:
X.NH 2
Relative performance
X.LP
The ratio of the tested servers performance with the performance
of an unloaded monochrome Sun3/50
running UNTUNED (i.e. original MIT code) X11R3 under OS 3.4.
X.sp
This machine has been choosen, since it is:
X.sp
X.nf
X    well known. 
X    available at our site.
X    it uses no graphics co-processor and plain MIT code.
X.fi
X.sp
The 
X.I ref.values
in the previous tables are the numbers obtained by running the benchmarks
on this reference machine. (BTW: thanks to my brother Juergen for
running the bench on his machine for me)
X.NH 2
Subjective performance
X.LP
XFor every group of tests the time it takes to execute it is
compared with the time it takes on the reference machine.
The individual tests are weighted in this computation according to
to
X.I weight
values in the tables listed above.
X.LP
The weights have been scaled to give a monochrome sun3/50 running untuned
MIT release 3 code an xstone rating of 10000
X(so that percentages are easy to compute).
X.LP
The weights are based on our experience on what is used often.
Since most of the time you're dealing with text its weight is very
high.
XXCopyPlane from bitmap to screen is used more often than
XXCopyArea from pixmap to screen (on a monocrome server they are
equivalent).
Solid lines/rectangles are used more often than dashed, tiled and stippled
ones.
Tiled rectangles are drawn more often than stippled ones (root window
background).
X.LP
If you look at the weights in the tables above, you will notice that
text performance counts 30%, bitblts 15% (with scroll alone 10%),
solid lines + rectangles 15% and filled + tile-filled rectangles 17%.
X.br
Since arcs, dashed- and wide lines are not very often drawn, their
weight together is only about 10%.
X.LP
Of course, depending on your applications, the rating
could differ from your personal feeling. If you are running a CAD
application doing only line-drawing, the line weight is too small
compared to text/bitblt rating.
If you are only working with xterm, the scroll/character weight are too
small compared to line rating (and so on ....).
X.LP
The final output consists of the individual
X.I stone
ratings for the line-, fill-, blit-, text-, arc- and complex tests and
the total
X.I xStone
rating.
X.br
XFor a server with an
X.I xStone
rating of 5000 you can expect that a graphical mix with 30% text,
X15% lines etc. takes twice the time to execute as it takes on the reference
machine.
X.br
If you are doing mostly text, look at the
X.I textStone
ratings; if you are doing mostly line-drawing, look at the
X.I lineStones.
X.LP
All benchmarks excluded (with -not option)
will be counted in the final 
X.I xstone
rating according to the rating of the other benchmarks - so that
a server showing double sun3 performance in all other tests
will get this rating also for the missing tests.
This number is of course wrong - but
how should a non-working server be compared to a working one ?
X.LP
The scripts to generate the ratings are in the "scripts"-directory.
To use them, create a subdirectory "results", put all your ".out"-files
into this directory and call:
X.sp
X.nf
X	make summary.ms detail.ms
X.fi
X.sp
This will create 2 files named "summary.ms" and "detail.ms"
which can be printed with:
X.sp
X.nf
X	tbl summary.ms detail.ms | troff -Txxx -ms
X    or
X	tbl summary.ms detail.ms | nroff -ms | your_printer
X.fi
X.KE
X.KS
X.NH 1
XErrors
X.LP
As
X.I xbench
measures times in 1 second deltas, therefore within each run,
the error (with timegoal=10) is about 10%.
Bigger timegoals lead to a smaller error but also increased runtime.
Also care should be taken, that the machine under test is load-free;
if possible disable daemons (like rwhod). If testing via ethernet
the measured time also depends on how fast the client-machine (the one on
which
X.I xbench
runs) can send the requests. If possible run the bench on different
client-machines and take the best result. Run the bench at times
X(lunch time or at night) when your ethernet-load is low.
X.NH 1
XFinal Note
X.LP
I'd better not included the weighting stuff -
I can see the flames on the net ....
X.br
Before you send me any mail on xbench please think about 2 things:
X.sp
X.nf
X    How fair/exact are other *stone tests ?
X.sp
X    What could/should be done better and why ?
X.fi
X.sp
If there are any suggestions, enhancements I will add them
into xbench and release a new version of it.
X.br
XFor now -
X.sp
X.sp
X\fB- Have fun\fP
X.sp
Claus Gittinger
X.br
Siemens Munich
X.br
Dep D ST Sp 4
X.br
D-8000 Munich/Neuperlach
X.br
Otto-Hahn-Ring
X.br
West Germany
X.sp
X(..!unido!sinix!claus)
X.KE
END_OF_FILE
if test 14481 -ne `wc -c <'xbench.doc'`; then
    echo shar: \"'xbench.doc'\" unpacked with wrong size!
fi
# end of 'xbench.doc'
fi
echo shar: End of archive 4 \(of 4\).
cp /dev/null ark4isdone
MISSING=""
for I in 1 2 3 4 ; do
    if test ! -f ark${I}isdone ; then
	MISSING="${MISSING} ${I}"
    fi
done
if test "${MISSING}" = "" ; then
    echo You have unpacked all 4 archives.
    rm -f ark[1-9]isdone
else
    echo You still need to unpack the following archives:
    echo "        " ${MISSING}
fi
##  End of shell archive.
exit 0