Compressing to a tape drive
les at chinet.chi.il.us
Thu Mar 15 09:16:03 AEST 1990
In article <371 at mtndew.UUCP> friedl at mtndew.UUCP (Steve Friedl) writes:
>The adventurous might want to investigate another approach that
>might help the tape stream even more. The problem with the huge
>blocksizes is that it stops while the buffer refills, so why not
>use two processes sharing (at least) two buffers.
>The parent process reads into some huge buffer, then forks a
>child to write the buffer to the output place (the tape or
>whatever). While the child is writing the buffer, the parent is
>busy filling a second buffer. Once the second one is full, the
>parent waits for the child to finish, then forks again to let it
"afio" does this if you use the -f flag and it works pretty well.
>I have not actually done this, but I bet a donut that AT&T has.
>Their "ctccpio" command feeding the stupid 23MB tape unit on the
>3B2 has lots of process switches, and I think this is how they
>manage to keep the tape streaming. It trashes the machine for
>the duration, but it is >much< faster than a dd with even a large
>blocksize (it could also be because ctccpio turns on stream mode
>with some ioctl).
The second guess is correct - afio also knows how to stream the
3B2 tape if you compile the option in and use the -C flag. Turning
on streaming mode is not something an ordinary program like dd
should attempt because (a) it is wierd - see the code in afio, and
(b) writing the wrong sized buffer will crash the machine. The 2nd
process doesn't really gain that much because the tape controller
eats most of the CPU anyway (23M tape).
>You could probably do an even better job on this with a reader
>and a writer with a common set of shared-memory buffers. Then
>you get the speed of double buffering without the process
I started on this a long time ago, but after crashing the 3B2 I
moved the test code to a 3B1 and found that writing to a device
from shared memory would crash the machine and that you couldn't
malloc() anything after attaching a shared memory segment (3B1 only).
Again, the 3B1 floppy controller eats the CPU so there isn't a
big gain anyway.
I recently tried the same code on a 386 and found that it would
run about 50-70K/sec from a cpio of the (ESDI) disk to the (125M)
tape. The program uses an arbitrary number of arbitrary sized
blocks in shared memory and can copy over to local memory on the
device side if needed. The tricky part is that it tries to guess
when the tape has stopped and lets most of the shared memory fill
up before re-starting. It can usually do slightly better than
afio even though there is an extra pipe involved (from cpio).
I'll mail you a copy - if you have time you can clean it up and
post it or let me know what you think of the concept.
les at chinet.chi.il.us
More information about the Comp.unix.i386