Blog: How-Tos
Double-Free RCE in VLC. A honggfuzz how-to
Introduction
I spent three months working on VLC using Honggfuzz, tweaking it to suit the target. In the process, I found five vulnerabilities, one of which was a high-risk double-free issue and merited CVE-2019-12874.
Here’s the VLC advisory https://www.videolan.org/security/sa1901.html.
Here’s how I found it. I hope you find the how-to useful and it inspires you to get fuzzing.
Background
VLC
VLC is a free media player which is open-source, portable, cross-platform and streaming media server, developed by the VideoLAN project. Media players such as VLC usually have a very complex codebase, including parsing and supporting a large number of file media file formats, math calculations, codecs, demux, text renderers and more complex code.
Figure 1: Loaded modules within the VLC binary.
honggfuzz
For this project I used honggfuzz, a modern, feedback-driven fuzzer based on code coverage, developed by Robert Swiecki.
Why honggfuzz?
It provides an easy way to instrument a target (unfortunately it did not work for this target but we will see how I was able to overcome those issues), it has some very powerful commands such as mutate only X amount of bytes (using the -F parameter), an easy to use command line and it uses AddressSanitzer instrumentation for software coverage saving all the unique crashes as well as the coverage files which hit new block codes.
It would be very difficult to discover those bugs without using code coverage, as given the complexity of the code, I would probably never be able to hit those paths!
Getting VLC and building it
VLC depends on many libraries and external projects. On an Ubuntu system, this is easily solved by just getting all the dependencies via apt:
$ apt-get build-dep vlc
(If you’re on ubuntu make sure to also install libxcb-xkb-dev package.) Now I’ll grab the source code – remember you want to be running the very latest version!
While you’re there, let’s run bootstrap which will generate the makefiles for our platform.
$ git clone https://github.com/videolan/vlc $ ./bootstrap
Once that’s done, I also want to add support for AddressSanitizer. Unfortunately, passing –with-sanitizer=address when issuing the configure command is not enough as it will give errors just before compilation finishes due to missing compilation flags. As such, I need to revert the following commit, so I can compile VLC successfully and add AddressSanitizer instrumentation.
$ git revert e85682585ab27a3c0593c403b892190c52009960
Getting samples
First things first, I need to start by getting decent samples. FAs luck would have it the FFmpeg test suite has already a massive decent samples (that may or may not crash FFmpeg) which will help me get started. For this iteration I tried to fuzz the .mkv format, so the following command quickly gave decent initial seed files:
$ wget -r https://samples.ffmpeg.org/ -A mkv
Figure 2: Getting samples with wget.
Once there are a decent number of samples, the next step it to limit to relative small samples only such as 5mb:
$ find . -name "*.mkv" -type f -size -5M -exec mv -f {} ~/Desktop/mkv_samples/ \;
Code Coverage (using GCC)
Once we have our samples, we need to verify whether our initial seed does indeed give us decent coverage – the more code lines/blocks we hit, the better chances we might have to find a bug.
Let’s compile VLC using GCC’s coverage flags:
$ CC=gcc CXX=g++ ./configure --enable-debug --enable-coverage $ make -j8
Once compilation is successful, we can confirm if we have the gcno files:
$ find . -name *.gcno
At this phase, we are ready to run one by one our seed files and get some nice graphs. Depending on the samples and movies length, we need to figure out a way to play X seconds and exit cleanly VLC otherwise we’re going to be here all night!
Luckily, VLC has already the following two parameters: –play-and-exit and –run-time=n (where n a number in seconds). Let’s quickly navigate back to our samples folder and run this little bash script:
#!/bin/bash FILES=/home/symeon/Desktop/vlc_samples-master/honggfuzz_coverage/*.mkv for f in $FILES do echo "[*] Processing $f file..." ASAN_OPTIONS=detect_leaks=0 timeout 5 ./vlc-static --play-and-exit --run-time=5 "$f" done
Once executed, you should be seeing VLC playing 5 seconds, exiting and then looping over the videos one by one. Continuing, we are going to use @ea_foundation‘s covnavi little tool, which gets all the coverage information and does all the heavy lifting for you.
Figure 3: Generating coverage using gcov.
Notice that a new folder web has been created, if you open index.html with your favourite browser, navigate to demux/mkv and take a look at the initial coverage. With our basic sample set, we managed to hit 45.1% lines and 33.9% functions!
Figure 4: Initial coverage after running the inital seed files.
Excellent, we can confirm that we have a decent amount of coverage and we are ready to move on to fuzzing!
The harness
While searching the documentation, it turns out that VLC has provided a sample API code which can be used to play a media file for a few seconds and then shut down the player.
That’s exactly what we are looking for! They also have a provided an extensive list with all the modules which can be found here.
#include <stdio.h>
#include <stdlib.h>
#include <vlc/vlc.h>
int main(int argc, char* argv[])
{
libvlc_instance_t * inst;
libvlc_media_player_t *mp;
libvlc_media_t *m;
if(argc < 2)
{
printf("usage: %s <input>\n", argv[0]);
return 0;
}
/* Load the VLC engine */
inst = libvlc_new (0, NULL);
/* Create a new item */
m = libvlc_media_new_path (inst, argv[1]);
/* Create a media player playing environement */
mp = libvlc_media_player_new_from_media (m);
/* No need to keep the media now */
libvlc_media_release (m);
/* play the media_player */
libvlc_media_player_play (mp);
sleep (2); /* Let it play a bit */
/* Stop playing */
libvlc_media_player_stop (mp);
/* Free the media_player */
libvlc_media_player_release (mp);
libvlc_release (inst);
return 0;
}
In order to compile the above harness, we need to link against our fresh compiled library. Navigate to /etc/ld.so.conf.d and create a new file libvlc.conf and include the path of the liblvc:
/home/symeon/vlc-coverage/lib/.libs
Make sure to execute
ldconfig
to update the ldconfig. Now let’s compile the harness using our fresh libraries and link it against ASAN.
hfuzz-clang harness.c -I/home/symeon/Desktop/vlc/include -L/home/symeon/Desktop/vlc/lib/.libs -o harness -lasan -lvlc
After compiling our harness and executing it, unfortunately this would lead to the following crash making it impossible to use it for our fuzzing purposes:
Figure 5: VLC harness crashing on config_getPsz function.
Interestingly enough,by installing the libvlc-dev library (from the ubuntu repository) and linking against this library, the harness would successfully get executed, however this is not that useful for us as we would not have any coverage at all, something that we don’t want to. For our next step, let’s try to instrument the whole VLC binary using clang!
Instrumenting VLC with honggfuzz (clang coverage)
Since our previous method did not quite work, let’s try to compile VLC and use honggfuzz’s instrumentation. For this one, I will be using the latest clang as well the compiler-rt runtime libraries which adds support for code coverage.
$:~/vlc-coverage/bin$ clang --version clang version 9.0.0 (https://github.com/llvm/llvm-project.git 281a5beefa81d1e5390516e831c6a08d69749791) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/symeon/Desktop/llvm-project/build/bin
Following honggfuzz’s feedback-driven instructions we need to run the following commands and will enable AddressSanitizer as well:
$ export CC=/home/symeon/Desktop/honggfuzz/hfuzz_cc/hfuzz-clang $ export CXX=/home/symeon/Desktop/honggfuzz/hfuzz_cc/hfuzz-clang++ $ ./configure --enable-debug --with-sanitizer=address
Once the configuration succeeds, now let’s try to compile it:
$ make -j4
After a while however, compilation fails:
<scratch space>:231:1: note: expanded from here
VLC_COMPILER
^
../config.h:785:34: note: expanded from macro 'VLC_COMPILER'
#define VLC_COMPILER " "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.o...
^
3 errors generated.
make[3]: *** [Makefile:3166: version.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory '/home/symeon/vlc-cov/covnavi/vlc/src'
make[2]: *** [Makefile:2160: all] Error 2
make[2]: Leaving directory '/home/symeon/vlc-cov/covnavi/vlc/src'
make[1]: *** [Makefile:1567: all-recursive] Error 1
make[1]: Leaving directory '/home/symeon/vlc-cov/covnavi/vlc'
make: *** [Makefile:1452: all] Error 2
Looking at the config.log, we can see the following:
#define VLC_COMPILE_BY "symeon"
#define VLC_COMPILE_HOST "ubuntu"
#define VLC_COMPILER " "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/8/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/8 -L/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/8/../../.. -L/home/symeon/Desktop/llvm-project/build/bin/../lib -L/lib -L/usr/lib --whole-archive /home/symeon/Desktop/llvm-project/build/lib/clang/9.0.0/lib/linux/libclang_rt.ubsan_standalone-x86_64.a --no-whole-archive --dynamic-list=/home/symeon/Desktop/llvm-project/build/lib/clang/9.0.0/lib/linux/libclang_rt.ubsan_standalone-x86_64.a.syms --wrap=strcmp --wrap=strcasecmp --wrap=strncmp --wrap=strncasecmp --wrap=strstr --wrap=strcasestr --wrap=memcmp --wrap=bcmp --wrap=memmem --wrap=strcpy --wrap=ap_cstr_casecmp --wrap=ap_cstr_casecmpn --wrap=ap_strcasestr --wrap=apr_cstr_casecmp --wrap=apr_cstr_casecmpn --wrap=CRYPTO_memcmp --wrap=OPENSSL_memcmp --wrap=OPENSSL_strcasecmp --wrap=OPENSSL_strncasecmp --wrap=memcmpct --wrap=xmlStrncmp --wrap=xmlStrcmp --wrap=xmlStrEqual --wrap=xmlStrcasecmp --wrap=xmlStrncasecmp --wrap=xmlStrstr --wrap=xmlStrcasestr --wrap=memcmp_const_time --wrap=strcsequal -u HonggfuzzNetDriver_main -u LIBHFUZZ_module_instrument -u LIBHFUZZ_module_memorycmp /tmp/libhfnetdriver.1000.419f7f6c4058b450.a /tmp/libhfuzz.1000.746a32a18d2c8f8a.a /tmp/libhfnetdriver.1000.419f7f6c4058b450.a --no-as-needed -lpthread -lrt -lm -ldl -lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/8/crtend.o /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o"
Apparently, something breaks the VLC_COMPILER variable and thus instrumentation fails. Let’s not give up, and proceed with the compilation using the following command:
$ make clean $ CC=clang CXX=clang++ CFLAGS="-fsanitize-coverage=trace-pc-guard,indirect-calls,trace-cmp" CXXFLAGS="-fsanitize-coverage=trace-pc-guard,indirect-calls,trace-cmp" ./configure --enable-debug --with-sanitizer=address $ ASAN_OPTIONS=detect_leaks=0 make -j8
will give us the following output:
GEN ../modules/plugins.dat make[2]: Leaving directory '/home/symeon/vlc-cov/covnavi/vlc/bin' Making all in test make[2]: Entering directory '/home/symeon/vlc-cov/covnavi/vlc/test' make[2]: Nothing to be done for 'all'. make[2]: Leaving directory '/home/symeon/vlc-cov/covnavi/vlc/test' make[2]: Entering directory '/home/symeon/vlc-cov/covnavi/vlc' GEN cvlc GEN rvlc GEN nvlc GEN vlc make[2]: Leaving directory '/home/symeon/vlc-cov/covnavi/vlc' make[1]: Leaving directory '/home/symeon/vlc-cov/covnavi/vlc'
Now although the compilation is successful, the binaries are missing honggfuzz’s instrumentation. As such, we need to remove the existing vlc_static binary, and manually link it with libhfuzz library. To do that, we need to figure out where linkage of the binary occurs. Let’s remove the vlc-static binary:
$ cd bin $ rm vlc-static
And run strace while compiling/linking the vlc-binary:
$ ASAN_OPTIONS=detect_leaks=0 strace -s 1024 -f -o compilation_flags.log make CCLD vlc-static GEN ../modules/plugins.dat
The above command, will specify the maximum string size to 1024 characters (default is 32), and will save all the output to specified file. Opening the log file and looking for “-o vlc-static” gives us the following result:
-- snip --
103391 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 103392
103391 rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
103391 vfork( <unfinished ...>
103393 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
103393 prlimit64(0, RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}, NULL) = 0
103393 execve("/bin/bash", ["/bin/bash", "-c", "echo \" CCLD \" vlc-static;../doltlibtool --silent --tag=CC --mode=link clang - DTOP_BUILDDIR=\\\"$(cd \"..\"; pwd)\\\" -DTOP_SRCDIR=\\\"$(cd \"..\"; pwd)\\\" -fsanitize-coverage=trace-pc-guard,indirect-calls,trace-cmp -Werror=unknown-warning-option -Werror=invalid-command-line-argument -pthread -Wall -Wextra -Wsign-compare -Wundef -Wpointer-arith -Wvolatile-register-var -Wformat -Wformat-security -Wbad-function-cast -Wwrite-strings -Wmissing-prototypes -Werror-implicit-function-declaration -Winit-self -pipe -fvisibility=hidden -fsanitize=address -g -fsanitize-address-use-after-scope -fno-omit-frame-pointer -fno-math-errno -funsafe-math-optimizations -funroll-loops -fstack-protector-strong -no-install -static -fsanitize=address -o vlc-static vlc_static-vlc.o vlc_static-override.o ../lib/libvlc.la "], 0x5565fb56ae10 /* 59 vars */ <unfinished ...>
103391 <... vfork resumed> ) = 103393
103391 rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT TERM XCPU XFSZ], NULL, 8) = 0
103391 wait4(-1, <unfinished ...>
103393 <... execve resumed> ) = 0
103393 brk(NULL) = 0x557581212000
Bingo! We managed to find the compilation flags and libraries that vlc-static requires. The final step is to link against libhfuzz.a’s library by issuing the following command:
$ ~/vlc-coverage/bin$ ../doltlibtool --tag=CC --mode=link clang -DTOP_BUILDDIR="/home/symeon/vlc-coverage" -DTOP_SRCDIR="/home/symeon/vlc-coverage" -fsanitize-coverage=trace-pc-guard,trace-cmp -Werror=unknown-warning-option -Werror=invalid-command-line-argument -pthread -Wall -Wextra -Wsign-compare -Wundef -Wpointer-arith -Wvolatile-register-var -Wformat -Wformat-security -Wbad-function-cast -Wwrite-strings -Wmissing-prototypes -Werror-implicit-function-declaration -Winit-self -pipe -fvisibility=hidden -fsanitize=address -g -fsanitize-address-use-after-scope -fno-omit-frame-pointer -fno-math-errno -funsafe-math-optimizations -funroll-loops -fstack-protector-strong -no-install -static -o vlc-static vlc_static-vlc.o vlc_static-override.o ../lib/libvlc.la -Wl,--whole-archive -L/home/symeon/Desktop/honggfuzz/libhfuzz/ -lhfuzz -u,LIBHFUZZ_module_instrument -u,LIBHFUZZ_module_memorycmp -Wl,--no-whole-archive
As a last step, let’s confirm that the vlc_static binary includes libhfuzz’s symbols:
$ ~/vlc-coverage/bin$ nm vlc-static | grep LIBHFUZZ
Figure 6: Examining the symbols and linkage of Libhfuzz.a library.
Fuzzing it!
For this part, we will be using one VM with 100GB RAM and 8 cores!
Figure 7: Monster VM ready to fuzz VLC.
With the instrumented binary, let’s copy it over to our ramdisk (/run/shm), copy over the samples and start fuzzing it!
$ cp ./vlc-static /run/shm $ cp -r ./mkv_samples /run/shm
Now fire it up as below, -f is the folder with our samples, and -F will limit to maximum 16kB.
$ honggfuzz -f mkv_samples -t 5 -F 16536 -- ./vlc-static --play-and-exit --run-time=4 ___FILE___
If everything succeeds, you should be getting massive coverage for both edge and pc similar to the screenshot below:
Figure 8: Honggfuzz fuzzing the instrumented binary and getting coverage information.
Hopefully within few hours you should get your first crash, which can be found in the same directory where honggfuzz was executed (unless modified) along with a text file HONGGFUZZ.REPORT.TXT, with information such as honggfuzz arguments, date of the crash, fault instruction as well as well the stack trace.
Figure 9: honggfuzz displaying information regarding the crash.
Crash results/triaging
After three days of fuzzing, honggfuzz discovered a few interesting crashes such as SIGSEV, SIGABRT, and SIGPFPE.
Despite the name SIGABRT, running the crashers under AddressSanitizer (we have already instrumented VLC) revealed that these bugs were in fact mostly heap-based out-of-bounds read vulnerabilities. We can simply loop through the crashers with the previously instrumented binary using this simple bash script:
$ cat asan_triage.sh #!/bin/bash FILES=/home/symeon/Desktop/crashers/* OUTPUT=asan.txt for f in $FILES do echo "[*] Processing $f file..." >> $OUTPUT 2>&1 ASAN_OPTIONS=detect_leaks=0,verbosity=1 timeout 12 ./vlc-static --play-and-exit --run-time=10 "$f" >>$OUTPUT 2>&1 done
Figure 10: Quickly triaging the crashes honggfuzz found.
Once you run it, you shouldn’t see any output since we redirecting all the output/errors to a file asan.txt. Quickly opening this file, it reveals the root cause of the crash, as well as symbolised stack traces where the crash occurred.
$ cat asan.txt | grep AddressSanitizer -A 5 ==59237==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x02d000000000 in thread T5 #0 0x4ac420 in __interceptor_free /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cc:123:3 #1 0x7f674f0e8610 in es_format_Clean /home/symeon/Desktop/vlc/src/misc/es_format.c:496:9 #2 0x7f672fde9dac in mkv::mkv_track_t::~mkv_track_t() /home/symeon/Desktop/vlc/modules/demux/mkv/mkv.cpp:892:5 #3 0x7f672fc3f494 in std::default_delete<mkv::mkv_track_t>::operator()(mkv::mkv_track_t*) const /usr/lib/gcc/x86_64-linux- gnu/8/../../../../include/c++/8/bits/unique_ptr.h:81:2 #4 0x7f672fc3f2d0 in std::unique_ptr<mkv::mkv_track_t, std::default_delete<mkv::mkv_track_t> >::~unique_ptr() /usr/lib/gcc/x86_64-linux- gnu/8/../../../../include/c++/8/bits/unique_ptr.h:274:4 -- SUMMARY: AddressSanitizer: bad-free /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cc:123:3 in __interceptor_free Thread T5 created by T4 here: #0 0x444dd0 in __interceptor_pthread_create /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_interceptors.cc:209:3 #1 0x7f674f15b6a1 in vlc_clone_attr /home/symeon/Desktop/vlc/src/posix/thread.c:421:11 #2 0x7f674f15b0ca in vlc_clone /home/symeon/Desktop/vlc/src/posix/thread.c:433:12 #3 0x7f674ef6e141 in input_Start /home/symeon/Desktop/vlc/src/input/input.c:200:25 -- ==59286==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000041fa0 at pc 0x7fabef0d00a7 bp 0x7fabf7074bd0 sp 0x7fabf7074bc8 READ of size 8 at 0x602000041fa0 thread T5 #0 0x7fabef0d00a6 in mkv::demux_sys_t::FreeUnused() /home/symeon/Desktop/vlc/modules/demux/mkv/demux.cpp:267:34 #1 0x7fabef186e6e in mkv::Open(vlc_object_t*) /home/symeon/Desktop/vlc/modules/demux/mkv/mkv.cpp:257:12 #2 0x7fac0e2b01b4 in demux_Probe /home/symeon/Desktop/vlc/src/input/demux.c:180:15 #3 0x7fac0e1f82b7 in module_load /home/symeon/Desktop/vlc/src/modules/modules.c:122:15 -- SUMMARY: AddressSanitizer: heap-buffer-overflow /home/symeon/Desktop/vlc/modules/demux/mkv/demux.cpp:267:34 in mkv::demux_sys_t::FreeUnused() Shadow bytes around the buggy address: 0x0c04800003a0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd 0x0c04800003b0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd 0x0c04800003c0: fa fa fd fd fa fa 00 02 fa fa fd fd fa fa fd fd 0x0c04800003d0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa 00 00 -- ==59343==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6250016a7a4f at pc 0x0000004ab6da bp 0x7f75e5457b10 sp 0x7f75e54572c0 READ of size 128 at 0x6250016a7a4f thread T15 #0 0x4ab6d9 in __asan_memcpy /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:22:3 #1 0x7f75ece563c2 in lavc_CopyPicture /home/symeon/Desktop/vlc/modules/codec/avcodec/video.c:435:13 #2 0x7f75ece52ac3 in DecodeBlock /home/symeon/Desktop/vlc/modules/codec/avcodec/video.c:1257:17 #3 0x7f75ece4d587 in DecodeVideo /home/symeon/Desktop/vlc/modules/codec/avcodec/video.c:1354:12 -- SUMMARY: AddressSanitizer: heap-buffer-overflow /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:22:3 in __asan_memcpy Shadow bytes around the buggy address: 0x0c4a802ccef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c4a802ccf00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c4a802ccf10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c4a802ccf20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 -- ==59411==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6040000716b4 at pc 0x0000004ab6da bp 0x7f97b4ea8290 sp 0x7f97b4ea7a40 READ of size 13104 at 0x6040000716b4 thread T5 #0 0x4ab6d9 in __asan_memcpy /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:22:3 #1 0x7f97aceb2858 in mkv::matroska_segment_c::TrackInit(mkv::mkv_track_t*)::TrackCodecHandlers::StringProcessor_1783_handler(char const*&, mkv::matroska_segment_c::TrackInit(mkv::mkv_track_t*)::HandlerPayload&) /home/symeon/Desktop/vlc/modules/demux/mkv/matroska_segment_parse.cpp:1807:25 #2 0x7f97aceb1e7b in mkv::matroska_segment_c::TrackInit(mkv::mkv_track_t*)::TrackCodecHandlers::StringProcessor_1783_callback(char const*, void*) /home/symeon/Desktop/vlc/modules/demux/mkv/matroska_segment_parse.cpp:1783:9 #3 0x7f97ace4ed16 in (anonymous namespace)::StringDispatcher::send(char const* const&, void* const&) const /home/symeon/Desktop/vlc/modules/demux/mkv/string_dispatcher.hpp:128:13 -- SUMMARY: AddressSanitizer: heap-buffer-overflow /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:22:3 in __asan_memcpy Shadow bytes around the buggy address: 0x0c0880006280: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd 0x0c0880006290: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd 0x0c08800062a0: fa fa fd fd fd fd fd fd fa fa 00 00 00 00 00 02 0x0c08800062b0: fa fa 00 00 00 00 04 fa fa fa 00 00 00 00 00 04
Fantastic! We have our PoCs, along with decent symbolised traces revealing the lines where the crash occurred!
Reviewing/Increasing coverage
Honggfuzz by default will save all the new samples that produce new coverage to the same folder with our samples (can be modified via the –covdir_all parameter). This is where things get interesting. Although we manged to discover a few vulnerabilities in our initial fuzzing, it’s time to run all the produced coverage again, and see which comparisons honggfuzz could not find (a.k.a magic values, maybe crc checksums or string comparisons).
For this example, I will be re-running again the previously bash script, feeding all the *.cov file set.
Figure 11: A total of 86254 files were saved during three days of fuzzing.
As you can see above, a total of 86254 files were saved as they produced new paths.
Now it’s time to iterate over those files again:
Figure 12: Iterating over the .cov files honggfuzz produced to measure new coverage.
Let’s re-run the coverage and see how much code we hit!
Figure 13: Our improved coverage after iterating over the cov files honggfuzz produced.
So after three days of fuzzing, we slightly bumped our overall coverage from 45.1% to 50%! Notice how the Ebml_parser.cpp from the initial 71.5% increased to 96.8% and in fact we were able to find some bugs on EBML parsing functionality while fuzzing .mkv files!
What would be our next steps? How can we improve our coverage? After manually reviewing the coverage, it turns out functions such as void matroska_segment_c::ParseAttachments( KaxAttachments *attachments ) were never hit!
Figure 14: Coverage showing no execution of the ParseAttachment code base.
After a bit of research, it turns out that a tool named mkvpropedit can be used to add attachments to our sample files. Let’s try that:
$ mkvpropedit SampleVideo_720x480_5mb.mkv --add-attachment '/home/symeon/Pictures/Screenshot from 2019-03-24 11-47-04.png'
Figure 15: Adding attachments to an existing mkv file.
Brilliant, this looks like it worked! Finally let’s confirm it by setting up a breakpoint on the relevant code, and run VLC with the new sample:
Figure 16: Hitting our breakpoint and expanding our coverage!
Excellent! We’ve managed to successfully create a new attachment and hit new functions within the mkv::matroska_segment codebase! Our next step would be similar to this technique, adjust the samples and freshly fuzz our target!
Discovered vulnerabilities
After running our fuzzing project for two weeks, as you can see from the following screenshot we performed a total 1 million executions (!), resulting in 1547 crashes of which 36 were unique.
Figure 17: A unique 36 crashes after fuzzing VLC for 15 days!
Figure 18: A double free vulnerability while parsing a malformed mkv file.
Many crashes were divisions by zero, and null pointer dereferences. A few heap based out-of-bounds write were also discovered which were not able to reproduce reliably.
However, the following five vulnerabilities were disclosed to the security team of VLC.
1. Double Free in mkv::mkv_track_t::~mkv_track_t()
==79009==ERROR: AddressSanitizer: attempting double-free on 0x602000048e50 in thread T5: mkv demux error: Couldn't allocate buffer to inflate data, ignore track 4 [000061100006a080] mkv demux error: Couldn't handle the track 4 compression #0 0x4ac420 in __interceptor_free /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cc:123:3 #1 0x7fb7722c3b6f in mkv::mkv_track_t::~mkv_track_t() /home/symeon/Desktop/vlc/modules/demux/mkv/mkv.cpp:895:5 #2 0x7fb77214ad4b in mkv::matroska_segment_c::ParseTrackEntry(libmatroska::KaxTrackEntry const*) /home/symeon/Desktop/vlc/modules/demux/mkv/matroska_segment_parse.cpp:992:13
The above vulnerability was fixed with the release of VLC 3.0.7 based on the following commit: http://git.videolan.org/?p=vlc.git;a=commit;h=81023659c7de5ac2637b4a879195efef50846102.
2. Freeing on address which was not malloced in es_format_Clean.
[000061100005ff40] mkv demux error: cannot load some cues/chapters/tags etc. (broken seekhead or file) [000061100005ff40] ================================================================= ==92463==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x02d000000000 in thread T5 mkv demux error: cannot use the segment #0 0x4ac420 in __interceptor_free /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cc:123:3 #1 0x7f7470232230 in es_format_Clean /home/symeon/Desktop/vlc/src/misc/es_format.c:496:9 #2 0x7f7452f82a6c in mkv::mkv_track_t::~mkv_track_t() /home/symeon/Desktop/vlc/modules/demux/mkv/mkv.cpp:892:5 #3 0x7f7452dd78e4 in std::default_delete<mkv::mkv_track_t>::operator()(mkv::mkv_track_t*) const /usr/lib/gcc/x86_64-linux-gnu/8/../../../../include/c++/8/bits/unique_ptr.h:81:2
3. Heap Out Of Bounds Read in mkv::demux_sys_t::FreeUnused()
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null) [00006060001d1860] decdev_vaapi_drm generic error: vaInitialize: unknown libva error [h264 @ 0x6190000a4680] top block unavailable for requested intra mode -1 [h264 @ 0x6190000a4680] error while decoding MB 10 0, bytestream 71 ================================================================= ==104180==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62500082c24f at pc 0x0000004ab6da bp 0x7f6f5ac3faf0 sp 0x7f6f5ac3f2a0 READ of size 128 at 0x62500082c24f thread T8 #0 0x4ab6d9 in __asan_memcpy /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:22:3 #1 0x7f6f5ad1748f in lavc_CopyPicture /home/symeon/Desktop/vlc/modules/codec/avcodec/video.c:435:13 #2 0x7f6f5ad13b89 in DecodeBlock /home/symeon/Desktop/vlc/modules/codec/avcodec/video.c:1259:17 #3 0x7f6f5ad0e537 in DecodeVideo /home/symeon/Desktop/vlc/modules/codec/avcodec/video.c:1356:12
4. Heap Out Of Bounds Read in mkv::demux_sys_t::FreeUnused()
[0000611000069f40] mkv demux error: No tracks supported ================================================================= ==81972==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000482c0 at pc 0x7f0a692c7a37 bp 0x7f0a6bf2ac10 sp 0x7f0a6bf2ac08 READ of size 8 at 0x6020000482c0 thread T7 #0 0x7f0a692c7a36 in mkv::demux_sys_t::FreeUnused() /home/symeon/Desktop/vlc/modules/demux/mkv/demux.cpp:267:34 #1 0x7f0a6937eaf1 in mkv::Open(vlc_object_t*) /home/symeon/Desktop/vlc/modules/demux/mkv/mkv.cpp:257:12 #2 0x7f0a86792691 in demux_Probe /home/symeon/Desktop/vlc/src/input/demux.c:180:15 #3 0x7f0a866d8d17 in module_load /home/symeon/Desktop/vlc/src/modules/modules.c:122:15
5. Heap Out Of Bounds Read in mkv::matroska_segment_c::TrackInit
================================================================= ==83326==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6040000b3134 at pc 0x0000004ab6da bp 0x7ffb4f076250 sp 0x7ffb4f075a00 READ of size 13104 at 0x6040000b3134 thread T7 #0 0x4ab6d9 in __asan_memcpy /home/symeon/Desktop/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cc:22:3 #1 0x7ffb4d84008d in mkv::matroska_segment_c::TrackInit(mkv::mkv_track_t*)::TrackCodecHandlers::StringProcessor_1783_handler(char const*&, mkv::matroska_segment_c::TrackInit(mkv::mkv_track_t*)::HandlerPayload&) /home/symeon/Desktop/vlc/modules/demux/mkv/matroska_segment_parse.cpp:1807:25 #2 0x7ffb4d83f6ab in mkv::matroska_segment_c::TrackInit(mkv::mkv_track_t*)::TrackCodecHandlers::StringProcessor_1783_callback(char const*, void*) /home/symeon/Desktop/vlc/modules/demux/mkv/matroska_segment_parse.cpp:1783:9 #3 0x7ffb4d7dc486 in (anonymous namespace)::StringDispatcher::send(char const* const&, void* const&) const /home/symeon/Desktop/vlc/modules/demux/mkv/string_dispatcher.hpp:128:13
Note: Some of those bugs were also previously discovered and disclosed via the HackerOne Bug Bounty, and the rest of the bugs have not been addressed as of now.
Advanced fuzzing with libFuzz
While searching for previous techniques, I stumbled upon this blog post, where one of the VLC developers used libFuzz to get deeper coverage. The developer used for example the vlc_stream_MemoryNew() (https://www.videolan.org/developers/vlc/doc/doxygen/html/stream__memory_8c.html)which reads data from the byte stream and fuzzed the demux process. As expected, he manged to find a few interesting vulnerabilities. This proves that the more effort you do to write your own harness and research your target, the better results (and bugs) you will get!
Is there any way to scan/check for malformed MKV or AVI files?
TL;DR We didn’t turn the crash into a fully working exploit, but someone with the time and effort could create one. The best mitigation is to update VLC and use the latest version. The moment that someone creates an exploit then Yes, AVs will jump in and start producing signatures and analysing the file.
Yes, it is possible for AV’s using a file signature or heuristic analysis to detect them. E.g. if the malicious files use heap spraying techniques in order to allocate fake objects and takes advantage of the double-free vulnerability. BUT an attacker requires a fully working exploit which will be also publicly available in order for AVs to start picking up the malicious activity.
Takeaways
We started with zero knowledge of how VLC works, and we’ve learnt how to create a very simple harness based on the documentation (which was unsuccessful).
We nevertheless continued with instrumenting VLC with honggfuzz, and although the standard process of instrumenting the binary didn’t work (hfuzz-clang), we were able to tweak a bit the parameters and successfully instrument the binary.
We continued by gathering samples, compiled and linked VLC against the libhfuzz library adding coverage support, started the fuzzing process, got crashes and triaged our crashes!
We were then able to measure our initial coverage and improve our samples increasing the overall coverage. Targeting only the .mkv format we saw that we were able to get a total of 50% coverage of the mkv file format. Remember that VLC supports a number of different video and audio file formats – sure enough there is still a lot of code that can be fuzzed!
Finally, although we used a relatively fast VM for this project, it should be noted that even a slow 4GB VM can be used and give you bugs!
Acknowledgements
This blog post would not be possible without guidance from @robertswiecki, helping with the compilation and linkage process, as well as all giving me the tips and tricks described above in this guide. Finally, thanks @AlanMonie and @Yekki_1 for helping me with the fuzzing VM.
VideoLAN have issued an advisory.
