First/single frame encoded with ffmpeg/libavcodec library cannot be immediately decoded

Question

I'm using libavcodec library and h264 codec to prepare the video stream on one end, transmit the encoded frames to the other PC and there decode it.

What I noticed after receiving very first packet (first encoded video frame) and feeding decoder with it, it is not possible to decode that frame. Only when I receive another frame the first one can be decoded but 'current' one not. So in the end I have constantly one frame delay on the decoder side.

I was trying different presets (focusing rather on 'ultrafast'), also 'zerolatency' tune, also whole variety of bit_rate values of AVCodecContext.

I also tried to flush (with nullptr packet) after injecting first frame data, just to check if it is maybe because of some internal buffers optimization - the frame still not decoded. Experimenting with other codecs (like mpeg4) gives even worse 'dalay' in number of frames to the point when when first frames can become decodable.

Is it normal, unavoidable because of some internal mechanisms? Otherwise how I can achieve real zero latency.

Supplementary setup information:

• max_b_frames set to 0 (higher value gives even more delay)
• pix_fmt set to AV_PIX_FMT_YUV420P

edit:

Answering some comment question:

(1) What is the decoder (or playback system)?

Custom decoder written using libavcodec, the decoded frames are later displayed on screen by OpenGL.

• initialization:

parser_ = av_parser_init(AV_CODEC_ID_H264);
codec_ = avcodec_find_decoder(AV_CODEC_ID_H264);
context_ = avcodec_alloc_context3(codec_);
context_->width = 1024;
context_->height = 768;
context_->thread_count = 1;
if ((codec_->capabilities & AV_CODEC_CAP_TRUNCATED) == 0)
{
  context_->flags |= AV_CODEC_FLAG_TRUNCATED;
}
if (avcodec_open2(context_, codec_, nullptr) < 0)
{
  throw std::runtime_error{"avcodec_open2 failed"};
}
avcodec_flush_buffers(context_);

• then player periodically calls of the method of decoder that suppose to check if the another frame can be retrieved and displayed:

auto result = avcodec_receive_frame(context_, frame_);
if (!buffer_.empty())
{  // upload another packet for decoding
  int used;
  if (upload_package(buffer_.data(), buffer_.size(), &used))
  {
    buffer_.erase(buffer_.begin(), buffer_.begin() + used);
  }
}
if (result == AVERROR(EAGAIN) || result == AVERROR_EOF || result < 0)
{
  return false;
}
yuv_to_rgb();
return true;

boolean return value informs if the decoding succeeded, and every time the buffer where the incomming packets are stored is checked and uploaded to libavcodec decoder

• and that is how the method that uploads the buffer looks like:

bool upload_package(const void* data, const std::size_t size, int* used)
{
  auto result = av_parser_parse2(parser_, context_, &packet_->data, &packet_->size, reinterpret_cast<const std::uint8_t*>(data), size, AV_NOPTS_VALUE, AV_NOPTS_VALUE, 0);
  if (result < 0)
  {
    return false;
  }
  *used = result;
  if (packet_->size != 0)
  {
    result = avcodec_send_packet(context_, packet_);
    if (result < 0)
    {
      return false;
    }
  }
  return true;
}

(2) If possible, save each one as a .bin file and then share the links with us for testing.

I will try to figure out something...

(3) Show example C++ code of your encoder settings for H264...

• initialization:

codec_ = avcodec_find_encoder(AV_CODEC_ID_H264);
context_ = avcodec_alloc_context3(codec_);

context_->bit_rate = 1048576; // 1xMbit;
context_->width = 1024;
context_->height = 768;
context_->time_base = {1, 30}; // 30 fps
context_->pix_fmt = AV_PIX_FMT_YUV420P;
context_->thread_count = 1;

av_opt_set(context_->priv_data, "preset", "ultrafast", 0);
av_opt_set(context_->priv_data, "tune", "zerolatency", 0);
avcodec_open2(context_, codec_, nullptr);

frame_->format = AV_PIX_FMT_YUV420P;
frame_->width = 1024;
frame_->height = 768;
av_image_alloc(frame_->data, frame_->linesize, 1024, 768, AV_PIX_FMT_YUV420P, 32);

• frame encoding:

rgb_to_yuv();
frame_->pts = frame_num_++;
auto result = avcodec_send_frame(context_, frame_);
while (result >= 0)
{
  result = avcodec_receive_packet(context_, packet_);
  if (result == AVERROR(EAGAIN) || result == AVERROR_EOF)
  {
    return;
  }
  else if (result < 0)
  {
    throw std::runtime_error{"avcodec_receive_packet failed"};
  }
  // here the packet is send to the decoder, the whole packet is stored on the mentioned before buffer_ and uploaded with avcodec_send_packet
  // I can also add that the whole buffer/packet us uploaded at once
  stream_video_data(packet_->data, packet_->size); 
}
av_packet_unref(packet_);
}

edit2:

I think I figured out the issue that I had.

For every incoming data packet (encoded frame) I was calling first av_parser_parse2, and then I was sending the data through avcodec_send_packet. And I was not recalling that procedure having empty buffer_, so for the first frame data the av_parser_parse2 was never called after uploading it through avcodec_send_packet, for the second frame it was called and the first frame was parsed, so it could be properly decoded, but for that (second) frame the parse2 was also not called, and so on ...

So the issue in my case was wrong sequence of av_parser_parse2 and avcodec_send_packet to handle the encoded data.