Lossless compression of 10bit images (stored as 16bit pngs) with ffmpeg - HEVC preferably

Question

I am trying to encode 10 bit images losslessly in a video format, preferably using HEVC encoding. The images are stored as 16 bit png files (but only use 10 bit) and I have been working with ffmpeg to create and read back the video files.

My best attempt so far is based on https://stackoverflow.com/a/66180140/17261462 but as mentioned there, I get some pixel intensity differences which may be due to rounding when converting between 10 and 16 bit representation. I tried a few different means (bit shifting, left bit replication, floating point based scaling) but haven't yet figured out how to get a trully lossless reconstruction.

Below is a small piece of code to replicate my issue. I probably am doing something wrong there so feedback would be appreciated.

import subprocess
import numpy as np
import matplotlib.pyplot as plt
import tempfile
import imageio

# Create simple image
bitdepth = 10
hbd = int(bitdepth/2)
im0 = np.zeros((1<<hbd,1<<hbd),dtype=np.uint16)
im0[:] = np.arange(0,1<<bitdepth).reshape(im0.shape)
print('im0',np.min(im0),np.max(im0),im0.shape,im0.dtype)
# tile it to be at least 64 pix
im0 = np.tile(im0, (2, 2))
print('im0',np.min(im0),np.max(im0),im0.shape,im0.dtype)
im0ref = im0
# bitshift it or rescale intensities
#im0 = (im0<<6)
#im0 = (im0<<6) + (im0>>4)
im0 = np.uint16(np.round(im0 * np.float64((1<<16)-1)/np.float64((1<<10)-1)))
print('im0',np.min(im0),np.max(im0),im0.shape,im0.dtype)

# Save it as png
tmp0 = tempfile.NamedTemporaryFile(suffix='.png', delete=False)
print(f'Using tmp file: {tmp0.name}')
imageio.imwrite(tmp0.name,im0)

# Encode with ffmpeg
tmp1 = tempfile.NamedTemporaryFile(suffix='.mkv', delete=False)
# note that adding the following doesn't seem to impact the results 
#  + ' -bsf:v hevc_metadata=video_full_range_flag=1' \
mycmd = f'ffmpeg -y -i {tmp0.name}' \
  + ' -c:v libx265 -x265-params lossless=1' \
  + ' -pix_fmt gray10be' \
  + f' {tmp1.name}'
print(mycmd)
p = subprocess.run(mycmd.split(), capture_output=True)
print( 'stdout:', p.stdout.decode() )
print( 'stderr:', p.stderr.decode() )

tmp2 = tempfile.NamedTemporaryFile(suffix='.png', delete=False)
mycmd = f'ffmpeg -y -i {tmp1.name}' \
  + ' -pix_fmt gray16be' \
  + f' {tmp2.name}'
print(mycmd)
p = subprocess.run(mycmd.split(), capture_output=True)
print( 'stdout:', p.stdout.decode() )
print( 'stderr:', p.stderr.decode() )

# Read back with ffmpeg
im1 = imageio.imread(tmp2.name)
print('im1',np.min(im1),np.max(im1),im1.shape,im1.dtype)

# Bitshift or scale back
im1pre = im1
#im1 = (im1>>6)
im1 = np.uint16(np.round(im1 * np.float64((1<<10)-1)/np.float64((1<<16)-1)))

# check the result
plt.figure()
plt.imshow(im0ref)
plt.colorbar()

plt.figure()
plt.imshow(im1)
plt.colorbar()

plt.figure()
plt.imshow(np.int32(im1)-np.int32(im0ref))
plt.colorbar()

print('err: ',np.linalg.norm((np.float32(im1)-np.float32(im0ref)).ravel()))

plt.show()

EDIT: I have now also posted my question on the FFmpeg-user list: http://ffmpeg.org/pipermail/ffmpeg-user/2021-November/053761.html

Also for convenience, a simple script is provided below to generate the different variants of using the 16 bits with 10bit data:

import numpy as np
import imageio

# Create simple image with  gradient from
# 0 to (2^bitdepth - 1)
bitdepth = 10
unusedbitdepth = 16-bitdepth
hbd = int(bitdepth/2)
im0 = np.zeros((1<<hbd,1<<hbd),dtype=np.uint16)
im0[:] = np.arange(0,1<<bitdepth).reshape(im0.shape)

# Tile it to be at least 64 pix as ffmpeg encoder may only work
# with image of size 64 and up
im0 = np.tile(im0, (2, 2))
print('im0',np.min(im0),np.max(im0),im0.shape,im0.dtype)

# Save it
imageio.imwrite('gradient10bit-lsb.png',im0)

# Bitshift the values to use most significant bits
im1 = (im0<<unusedbitdepth)
print('im1',np.min(im1),np.max(im1),im1.shape,im1.dtype)
imageio.imwrite('gradient10bit-msb.png',im1)

# Scale the values use all 16 bits
im2 = np.uint16(np.round(im0 * np.float64((1<<16)-1)/np.float64((1<<bitdepth)-1)))
print('im2',np.min(im2),np.max(im2),im2.shape,im2.dtype)
imageio.imwrite('gradient10bit-scaledto16bits.png',im2)

# Left bit replication as a cost-effective approximation of scaling
# See http://www.libpng.org/pub/png/spec/1.1/PNG-Encoders.html
im3 = (im0<<unusedbitdepth) + (im0>>(bitdepth-unusedbitdepth))
print('im3',np.min(im3),np.max(im3),im3.shape,im3.dtype)
imageio.imwrite('gradient10bit-leftbitreplication.png',im3)

As well as raw ffmpeg / image magick commands.

Encoding:

ffmpeg -y -i gradient10bit-scaledto16bits.png -c:v libx265 -x265-params lossless=1 -pix_fmt gray10be gradient10bit-scaledto16bits.mkv

Decoding back to png:

ffmpeg -y -i gradient10bit-scaledto16bits.mkv -pix_fmt gray16be recons-gradient10bit-scaledto16bits.png

Comparison:

magick compare -verbose -metric mae gradient10bit-scaledto16bits.png recons-gradient10bit-scaledto16bits.png diff-scaledto16bits.png

Many thanks,

Tom

Answer 1

If, as you say, you want to encode 10-bit images to video losslessly, you would surely do better to use a lossless format that is capable of storing such things - such as ffv1 - then you can store a full 16-bits without shifting/scaling or doing anything.

#!/bin/bash

# Generate 16-bit greyscale PNG
magick -size 1920x1080 xc:gray +noise random 1.png
magick 1.png -format "File: %f Unique colours: %k, Min: %[min], Max: %[max]\n" info:

# Encode to video
ffmpeg -v warning -y -i 1.png -c:v ffv1 -pix_fmt gray16le video.mkv

# Decode back to PNG
ffmpeg -v warning -y -i video.mkv 2.png
magick 2.png -format "File: %f Unique colours: %k, Min: %[min], Max: %[max]\n" info:

# Compare
magick compare -verbose -metric ae {1,2}.png null:

Output

File: 1.png Unique colours: 65536, Min: 0, Max: 65535
File: 2.png Unique colours: 65536, Min: 0, Max: 65535
1.png PNG 1920x1080 1920x1080+0+0 16-bit Gray 3.96256MiB 0.030u 0:00.029
2.png PNG 1920x1080 1920x1080+0+0 16-bit Gray 4161790B 0.020u 0:00.017
Image: 1.png
  Channel distortion: AE
    gray: 0
    all: 0
1.png=> PNG 1920x1080 16-bit Gray 3.96256MiB 0.760u 0:00.064

Answer 2

Thanks to Paul B Mahol on the ffmpeg-user mailing list, I have been able to solve this while using temporary rawvideo files. A solution without temporaries would nonethless be preferrable.

# convert png to rawvideo in 16 bits
ffmpeg -y -i gradient10bit-lsb.png -f rawvideo -pix_fmt gray16le gradient10bit-lsb.raw

# convert rawvideo to hevc-mkv in 10 bits by tricking the rawvideo demuxer
# into thinking the input is a 10 bit video
ffmpeg -y -f rawvideo -pixel_format gray10le -video_size 64x64 -i gradient10bit-lsb.raw -c:v libx265 -x265-params lossless=1 -pix_fmt gray10le gradient10bit-lsb.mkv

# delete tmp file
rm -f gradient10bit-lsb.raw

# convert hevc-mkv to rawvideo 10 bit
ffmpeg -y -i gradient10bit-lsb.mkv -f rawvideo -pix_fmt gray10le gradient10bit-lsb-postmkv.raw

# convert rawvideo back to png 16bits by tricking the rawvideo demuxer
# into thinking the input is 16 bits
ffmpeg -y -f rawvideo -pixel_format gray16le -video_size 64x64 -i gradient10bit-lsb-postmkv.raw -pix_fmt gray16be recons-gradient10bit-lsb.png

# delete tmp file
rm -f gradient10bit-lsb-postmkv.raw

# compare
magick compare -verbose -metric mae gradient10bit-lsb.png recons-gradient10bit-lsb.png diff-lsb.png