_UNIX_Today!_ hits a new benchmarking low
Ross M. Greenberg
greenber at utoday.UUCP
Sat Sep 2 15:03:00 AEST 1989
This is me in an official capacity for UNIX Today!.
First, in the future, please make sure to advise me of an ongoing
discussion regarding UNIX Today: our objective is to get better at
what we do. Sometimes we do a damned fine job. Sometimes we don't.
Based on what I'm reading here, I guess we blew the benchmarks on the
MPE. Main reason: we're small. We don't have all the hardware and
software we need in-house to get all the benchmarking we'd like. We're
addressing that most rapidly with some stuff I can't go into until the
deal is signed. Suffice it to say that I hereby promise you that
screw-ups as you believe happened will never happen again.
But...how did it happen, you might wonder? Well, those benchmarks
*were* done honestly. Nothing phoney baloney about it. They were
not done to make the MPE look good -- or bad. Those are the real
numbers that came out.
It is difficult to benchmark a new piece of hardware. Frankly, I don't
hold much faith in the <generic>stone tests, prefering an application
simulation myself. But, getting your hands on a piece of hardware and
then figuring the best benchmarks to run, and what to run them against
is a tough thing to do on a two week production cycle. Not a good
enough excuse, I know.
Now, we use a lot of freelancers. Bob Morien, the author of the MPE
review, happens to be one of them. Freelancers work *directly* under
my supervision. No review or techie stuff gets out into the paper without
my personal approval. I am the one ultimately responsible for the errors
in judgement in letting those particular benchmarks get printed. Due to
some heavy deadline pressures, *I* didn't review them as thouroughly
as I normally would have. I blew it. I screwed up. I should have
gone back to Bob and asked him to rerun the benchmarks with more current
and comparable stuff. Bob's actually a very good benchmarker. He simply
didn;t have all the hardware and software on-hand at the time to re-run
the benchmarks. He screwed up a little. I screwed up in a major way.
If it's any consolation to you, I noticed the problem when we got back
the proofs. It was too late at that point to do anything about it as
the paper was to be printed about two hours later.
We're working hard to make the paper better. We need your help. If we
screw up, send the mail directly to me, or advise me immediately what
newsgroup to join to take part in the discussion.
Anyway...we'll be reviewing quite a bit of hardware in the very near
future. You can be sure that the benchmarks for that hardware will
show a comparison with the MPE benchmark. If it's a dog, the readers
will know. If it's a shining light, they'll know, too.
Benchmarks are always a problem spot. In two or three issues, you'll
see what we're doing about it. I'll bet you all a drink at the next
UseNix that you'll be more than satisfied.
But you have to promise me that, once I drink all those drinks I win,
somebody helps carry me back to my room, ok?
Ross M. Greenberg
UNIX TODAY! 594 Third Avenue New York New York 10016
Review Editor Voice:(212)-889-6431 BBS:(212)-889-6438
uunet!utoday!greenber BIX: greenber MCI: greenber CIS: 72461,3212
To subscribe, send mail to circ at utoday with "Subject: Request"
More information about the Comp.unix.i386