Read from BaseStream after creating a StreamReader on a NamedPipeServerStream?

Question

I create a named pipe like so:

var server = new NamedPipeServerStream(pipename, PipeDirection.InOut, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous);

Then, after a client connects, I create a StreamReader:

namedPipeReader = new StreamReader(server);

I can use the namedPipeReader to read lines from the stream, but then I need to pass it to an object that reads binary data.

I pass namedPipeReader.BaseStream, and the object fails to read anything. If I call

namedPipeReader.BaseStream.Read(bytes, 0, length);

The program stalls, until new text data is sent from the client. If I call

namedPipeReader.Read();

I can read the binary data I expect, although it is incorrect (I would guess due to StreamReader trying to treat them as characters.)

Why is this happening? What do I need to do to read both text and binary data from a NamePipeServerStream?