NAME

docs/dev/fuzzing.pod - Fuzzing Parrot with afl - the american fuzzy lop

DESCRIPTION

This document describes how to find parrot crashes on invalid input with the american fuzzy lop.

This will not work on MS Windows. It is tested on x86_64 linux, but should also work on darwin and other POSIX intel systems.

Setup afl

Download afl from http://lcamtuf.coredump.cx/afl/

Make and install afl.

    cd ~/

    wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz
    tar xfz afl-latest.tgz

    cd afl-<TAB>

    # be sure to unset ASAN_OPTIONS
    grep ASAN | env
    unset ASAN_OPTIONS
    export ASAN_OPTIONS

    make
    sudo make install

Compile parrot with the instrumenting afl-gcc:

    cd <your-parrot>
    perl Configure.pl --cc=afl-gcc --ld=afl-gcc --link=afl-gcc
    make -s -j4

Read the afl documentation

See http://lcamtuf.coredump.cx/afl/ and under afl-VERSION/docs. And maybe also https://en.wikipedia.org/wiki/Fuzz_testing

Some notes:

afl can work with clang, but gcc is preferred.

afl can work with ASAN or MSAN, but since parrot is too slow it is advised to run it without. Our coverage with the normal testsuite is pretty good, so you better smoke the normal testsuite with ASAN or MSAN or VALGRIND=valgrind instead.

afl tries to create bad input files, to crash or hang the process. Bad and uncontrolled input may impact the stability of your testmachine. E.g. it might randomly overwrite files or memory or fill up your logs or harddisc space. Use a fast kvm VM or docker. Only one core is needed if you don't fuzz in parallel.

parrot_old has faster startup, but does not play nice with the afl forkserver. You can use it with -C, the crash mode though. Get a testcase which crashes and find more crashes. This way I found https://github.com/parrot/parrot/issues/1168

With perl5 or potion I got exec speeds of 1500/sec, with parrot_old 150/sec, with parrot 100/sec. This is very slow. So strip your input testcases to the absolute minimum. An afl-fuzz run usually lasts a day or more, but at least a few hours. I did not dare yet to fuzz nqp or perl6 yet, as it would be even slower.

Set your terminal screen to black background, seriously. You hardly see anything on white.

imcc Parser

You can fuzz .pir and .pasm input to check the imcc parser.

    mkdir afl-out afl-testcase
    cp t/op/literal.t testcase/

    afl-fuzz -m4000 -o afl-out -i afl-testcase -- parrot -r @@

With keywords

Adding our regular pir and pasm keywords helps creating more meaningful permutations. One file in a new directory per keyword.

    mkdir -p afl-out2 afl-testcase2/keywords

    perl -e'`echo $_ > afl-testcase2/keywords/$_` for qw(.sym .arg prototyped
      non_prototyped .class
      .endclass .param inc dec new defined global clone .call .result .return
      .local .const .globalconst
      end goto if unless call branch jump jsr ret invoke invokecc throw
      rethrow die_hard .emit .eom .sub .end .begin_call
      .end_call .pcc_sub .begin_return .end_return .begin_yield
      .end_yield .loadlib .namespace .endnamespace .macro .include int float
      string pmc ne eq le lt ge gt == =head1 =end =pod)'

    afl-fuzz -m4000 -i afl-testcase2 -x afl-testcase2/keywords -o afl-out2 -- parrot -r @@

pbc packfiles

Fuzz binary input to check the PackFile reader. Binary files are the typical usecase for afl. See https://github.com/parrot/parrot/issues/1169

    mkdir -p afl-out3 afl-testcase3
    cp t/native_pbc/*_8_le.pbc afl-testcase3/
    afl-fuzz -m4000 -i afl-testcase3 -o afl-out3 -- parrot -r @@

During fuzzing

You can examine the results while running the fuzzer, and you can even fix the found error, recompile, Ctrl-C the fuzzer and resume it with -i-, i.e. afl-fuzz -m4000 -i- -o afl-out3 -- parrot -r @@

It will reinstrument the existing paths and cases, and save away the old results.

Crosscheck the results for false positives

afl-fuzz does sometimes categorize all exit codes > 0 as crash, while it should do only do it with > 128. So you need to verify the found crashes, like this:

    for c in afl-out/crashes/*; do ./parrot $c >/dev/null 2>&1 || echo $? $c; done

and check only the exit codes > 128.

Further investigations - crash explorer

If you found a new crash, try out the crash explorer mode -C by copying this crash result as new testcase and start a new afl-fuzz with -C. This will find other variations of the crash faster than in the normal mode. See http://lcamtuf.blogspot.de/2014/11/afl-fuzz-crash-exploration-mode.html

Other tools

There exist also less intelligent blackbox fuzzers, like csmith.

And more intelligent whitebox fuzzers, which do taint symbolic input and use constraint solvers to track fishy source paths back to the input, e.g. mayhem, cbmc or the mythical smart Microsoft fuzzer (https://en.wikipedia.org/wiki/Fuzz_testing#cite_ref-AutoDO-14_16-0).

However afl is usually the best option for now. It is fast, simple and good enough.

AUTHOR

Reini Urban <rurban@cpan.org>

Copyright © 2002-2014, Parrot Foundation.