How to compile libav for AFLGo

Question

I need to compile libav library with AFLGO.

cd ~
export AFLGO=$PWD/aflgo
cd path/to/libav/folder
export SUBJECT=$PWD; export TMP_DIR=$PWD/obj-aflgo/temp
export CC=$AFLGO/afl-clang-fast
export CXX=$AFLGO/afl-clang-fast++
export COPY_CFLAGS=$CFLAGS
export COPY_CXXFLAGS=$CXXFLAGS
export ADDITIONAL="-targets=$TMP_DIR/BBtargets.txt -outdir=$TMP_DIR -flto -fuse-ld=gold -Wl,-plugin-opt=save-temps"
export CFLAGS="$CFLAGS $ADDITIONAL"
export CXXFLAGS="$CXXFLAGS $ADDITIONAL"
export LDFLAGS=-lpthread

./configure

it returns:

gcc is unable to create an executable file.
If gcc is a cross-compiler, use the --enable-cross-compile option.
Only do this if you know what cross compiling means.
C compiler test failed.

If you think configure made a mistake, make sure you are using the latest
version from Git.  If the latest version fails, report the problem to the
[email protected] mailing list or IRC #libav on irc.freenode.net.
Include the log file "avbuild/config.log" produced by configure as this will help
solving the problem.

I have checked config.log but it doesn't seem useful, these are the last lines:

....
gcc -targets=/home/ubuntu16/Documents/github-repos/libav/obj-aflgo/temp/BBtargets.txt -outdir=/home/ubuntu16/Documents/$
gcc: error: unrecognized command line option '-targets=/home/ubuntu16/Documents/github-repos/libav/obj-aflgo/temp/BBtar$
C compiler test failed.

why does this try compile it with gcc again? I want to instrument it with AFLGo fuzzer.

Answer 1

First, you should explicit specify the compiler and flags as follow:

./configure --cc=$CC --enable-lto --extra-ldflags="-flto -fuse-ld=gold -Wl,-plugin-opt=save-temps"

Otherwise, you can't compile the libav with AFLGo.

If you want to know other configure flags, try ./confiure --help

But after I have tried, the libav can't be compiled with the LTO (i.e. -flto), becasue the libav contains some asm instructions, and LLVM (clang) can not convert it to IR, which will cause error such as undefined reference at link time.

CC      libavutil/xtea.o
AR      libavfilter/libavfilter.a
AR      libavcodec/libavcodec.a
AR      libavresample/libavresample.a
AR      libswscale/libswscale.a
AR      libavutil/libavutil.a
LD      avconv
avconv.o:ld-temp.o:function decode_nal_unit: error: undefined reference to 'ff_h264_cabac_tables'
avconv.o:ld-temp.o:function decode_nal_unit: error: undefined reference to 'ff_h264_cabac_tables'
avconv.o:ld-temp.o:function decode_nal_unit: error: undefined reference to 'ff_h264_cabac_tables'
avconv.o:ld-temp.o:function decode_nal_unit: error: undefined reference to 'ff_h264_cabac_tables'
avconv.o:ld-temp.o:function put_vc1_mspel_mc01_mmx: error: undefined reference to 'ff_pw_53'
avconv.o:ld-temp.o:function put_vc1_mspel_mc03_mmx: error: undefined reference to 'ff_pw_53'
avconv.o:ld-temp.o:function put_vc1_mspel_mc10_mmx: error: undefined reference to 'ff_pw_53'
avconv.o:ld-temp.o:function put_vc1_mspel_mc11_mmx: error: undefined reference to 'ff_pw_53'
avconv.o:ld-temp.o:function put_vc1_mspel_mc11_mmx: error: undefined reference to 'ff_pw_128'
avconv.o:ld-temp.o:function put_vc1_mspel_mc11_mmx: error: undefined reference to 'ff_pw_128'
avconv.o:ld-temp.o:function put_vc1_mspel_mc12_mmx: error: undefined reference to 'ff_pw_128'
avconv.o:ld-temp.o:function put_vc1_mspel_mc12_mmx: error: undefined reference to 'ff_pw_128'
avconv.o:ld-temp.o:function put_cavs_qpel16_mc01_3dnow: error: undefined reference to 'ff_pw_42'
avconv.o:ld-temp.o:function put_cavs_qpel16_mc01_3dnow: error: undefined reference to 'ff_pw_42'
avconv.o:ld-temp.o:function put_cavs_qpel16_mc01_3dnow: error: undefined reference to 'ff_pw_42'
avconv.o:ld-temp.o:function put_cavs_qpel16_mc01_3dnow: error: undefined reference to 'ff_pw_42'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2YCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_w1111'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2YOffset'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2YCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_w1111'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2YOffset'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_w1111'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2UCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2VCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2VCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_w1111'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2VCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2VCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_mmx: error: undefined reference to 'ff_bgr2UVOffset'
avconv.o:ld-temp.o:function rgb24toyv12_3dnow: error: undefined reference to 'ff_bgr2YCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_3dnow: error: undefined reference to 'ff_bgr2YOffset'
avconv.o:ld-temp.o:function rgb24toyv12_3dnow: error: undefined reference to 'ff_bgr2YCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_3dnow: error: undefined reference to 'ff_bgr2YOffset'
avconv.o:ld-temp.o:function rgb24toyv12_3dnow: error: undefined reference to 'ff_bgr2UCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_3dnow: error: undefined reference to 'ff_bgr2UVOffset'
avconv.o:ld-temp.o:function rgb24toyv12_mmxext: error: undefined reference to 'ff_bgr2UCoeff'
avconv.o:ld-temp.o:function rgb24toyv12_mmxext: error: undefined reference to 'ff_bgr2UVOffset'
avconv.o:ld-temp.o:function yuv2bgr24_X_ar_mmxext: error: undefined reference to 'ff_M24A'
avconv.o:ld-temp.o:function yuv2bgr24_X_ar_mmxext: error: undefined reference to 'ff_M24C'
avconv.o:ld-temp.o:function yuv2bgr24_X_ar_mmxext: error: undefined reference to 'ff_M24B'
avconv.o:ld-temp.o:function yuv2bgr24_X_ar_mmxext: error: undefined reference to 'ff_M24B'
avconv.o:ld-temp.o:function yuv2bgr24_X_mmxext: error: undefined reference to 'ff_M24A'
avconv.o:ld-temp.o:function yuv2bgr24_X_mmxext: error: undefined reference to 'ff_M24C'
avconv.o:ld-temp.o:function yuv2bgr24_X_mmxext: error: undefined reference to 'ff_M24B'
avconv.o:ld-temp.o:function yuv2bgr24_X_mmxext: error: undefined reference to 'ff_M24B'
avconv.o:ld-temp.o:function yuv2bgr24_1_mmxext: error: undefined reference to 'ff_M24A'
avconv.o:ld-temp.o:function yuv2bgr24_1_mmxext: error: undefined reference to 'ff_M24C'
avconv.o:ld-temp.o:function yuv2bgr24_1_mmxext: error: undefined reference to 'ff_M24A'
avconv.o:ld-temp.o:function yuv2bgr24_1_mmxext: error: undefined reference to 'ff_M24C'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

So you can't build libav with AFLGo because of the flaw of LLVM (clang)