How to call av_read_frame() when streamed data is not yet available?

Question

I am streaming audio data and using the ffmpeg/libav libraries to process it in C++. I create a CustomIOContext and a read_packet function to provide the streamed data to ffmpeg, and I create an AVFormatContext using that CustomIOContext.

I want to loop through all available data and extract packets using av_read_frame(), and then when there is not enough data left to create a packet I want to wait until I receive more data and then loop again through the new data. However, I can't figure out what to return in my read_packet function below to tell ffmpeg to wait until more data is available. Ffmpeg seems to require all data to be available now. When I have no data to return, ffmpeg seems to think it's the end of the file, when actually it just isn't ready yet. I have tried returning AVERROR(EAGAIN) but I still have problems. It seems like I get random decoder-specific error messages, because it thinks the input is invalid (when actually it is just not available yet), and then once I have the input and I try calling av_read_frame() again, it immediately returns a negative error code because it got into a bad state.

What do I need to do to be able to loop through all the available data using av_read_frame(), and then pause and wait until more data arrives? How do I keep ffmpeg happy?

// -------- Set up  --------
size_t ioContextBufferSize = 4096;
avioContextBuffer = avAllocateEmptyBuffer(ioContextBufferSize);
avioContext = avio_alloc_context(avioContextBuffer.ptr,
                         ioContextBufferSize,
                    0, // bWriteable (1=true,0=false)
                        &(opaque),
                        read_packet,
                        0, // Write callback function
                        0)}; // Seek function not provided

AVFormatContext* inContext = avformat_alloc_context();
inContext->pb = avioContext;
inContext->flags = AVFMT_FLAG_CUSTOM_IO;

avformat_open_input(&inContext, "", nullptr, nullptr);
inputPacket = av_packet_alloc();

// -------- Loop and read data -------
// (the second time we reach this while loop, it never enters the loop :(
//
while ((ret = av_read_frame(inContext, inputPacket)) >= 0) {
  .. do stuff with inputPacket
}

// -------- read_packet function -----
// copies data into ffmpeg's internal buffer. What do I return here???
//
static int read_packet(void* opaque, uint8_t* avioContextBuffer, int ioContextBufferSize) {
  OpaqueDataWrapper* streamedData = static_cast<OpaqueDataWrapper*>(opaque);

  size_t bytesToRead = std::min((size_t)ioContextBufferSize, streamedData->remainingBytes());

  if (!bytesToRead) {
    // We need to wait for more data to arrive! What do I return here to tell ffmpeg to wait a while? 
    if (streamedData->streamStillOpen()) {
      return AVERROR(EAGAIN);
    } else {
      return AVERROR_EOF;
    }
  }
  ... otherwise copy the next bit of data
}