sockets vs. STREAMS, also intermachine networking on SV

[Steven M. Bellovin]

In article <506 at gonzo.UUCP>, daveb at gonzo.UUCP (Dave Brower) writes:
> The main reason you want a
> stream pipe is so you can use poll() in your server, which is supposed
> to pay attention to a number of connected clients.

Also to use I_SENDFD, to hand off file descriptors.  That's not a trivial
point; see below.

> [about security concerns]  Is there a reasonable answer, or are we really
> supposed to wait for SVR4?

I'm afraid I just don't know; when I was using /dev/spx, I was doing research
and prototype development; it was enough for me to wave my hands and say
``assume a stream pipe''...  I wouldn't release production code with that
sort of attitude towards security, but I wasn't working on anything releasable.

> I will claim that lacking
> any documentation (or a source license for the stuff) it is kind of hard
> to tell!

Yes.  /dev/spx was never intended as a standard, supported feature; in
fact, in some sense you're not supposed to use it, any more than you're
supposed to call _doprnt() directly.  It's considered an internal interface
in SVR3 that may go away without notice at any time -- like if they introduce
real stream pipes.

> Lacking documentation and source, can someone answer these questions
> about /dev/spx? Imagine a server has opened /dev/spx, and arranged for
> the creation of  a device file for the returned minor device.  Client
> programs want to talk to the server by opening that device file.
> 
> 1. What happens if two client programs try to open the same minor device?
>    Would they get a multiplexed channel with a server on the other end,
>    or would the second fail?  (By comparison FIFO files multiplex, but
>    psuedo-tty's fail, I think).

It can't happen if the proper protocol is used.  You're supposed to open
/dev/spx only, and it's a ``clone device'' -- a driver that returns a
minor device that isn't in use.  In fact, as distributed I don't believe
there are any named minor devices.

FIFOs don't multiplex in the sense of getting a distinguishable connection
on the server end; anyone who opens a FIFO can read data written by the
server.  There's no way to direct it to just the proper consumer.  Nor
can the server distinguish between data written by multiple clients.

> 2. If the server is sitting in a poll() on a stream pipe, and the client
>    on the other end dies, does the poll return or does it hang?  If it
>    hangs, what can you do to recover?

An M_HANGUP message is sent upstream.

> 3. What _does_ RFS use /dev/spx for?  How does it arrange 
>    rendevouz addresses?

Let me warn you here, this is going to be a bit weird....

The server creates a stream pipe by doing two opens of /dev/spx, and then
an I_FDINSERT.  It then uses fstat() to determine the major and minor device
numbers of one end of the pipe, and uses mknod() to create another name for
one end of the pipe.  The server can then sit there poll()ing the other end
(i.e., the anonymous end).

Have your eyes popped back in yet?  It gets worse.

To contact the server, a client also creates a stream pipe, via /dev/spx and
I_FDINSERT.  It then opens the named portion of the server's pipe, and
uses I_SENDFD to hand off its file descriptor for one end of its pipe to
the server.  It then closes the fd handed off, and also the fd to the
named pipe; it retains only one fd for its anonymous stream pipe.  The
named pipe is not used for any data transfer; it's not a multiplexing
connection, there's no way to differentiate between different opens, one
can't even associate text with a transmitted file descriptor, etc.

The above description is a bit sketchy; I gave the same explanation, in
somewhat more detail, in my ``Session Tty Manager'' paper in the S.F.
Usenix proceedings.  I refer you to it.

It's obvious, I think, why this interface was never documented.  It's
ugly, cumbersome, usable only by root, limited in the number of stream
pipes and servers available (only 256 minor devices, after all), etc.
SVR4 will have much nicer ways to do the same thing, but I don't know if
I'm allowed to discuss just what they are.  I have no idea if /dev/spx
will still be present; it will be obsolete, and was never guaranteed, but
of course it was used, and compatibility is a major concern.

The ability to pass file descriptors around is very powerful.  4.2bsd has
a similar feature -- the ``access rights'' on UNIX domain sockets are in
fact file descriptors.  It was rather buggy in 4.2, but I believe it works
well in 4.3; however, I don't know of any Berkeley code that actually uses
it.

You should also check out the Presotto and Ritche paper ``Interprocess
Communication in the 8th Edition UNIX System'', in the proceedings of the
Portland (Summer '85) Usenix conference; they describe lots of other things
to do when you can pass around file descriptors.  For example, you can
have a ``connection server''; it makes calls for you, using whatever weird
conventions your network or autodialers want, and pass back the file
descriptor.  If you get a new network, or a new type of dialer, only
one process need be changed.

		--Steve Bellovin