Fuzzing with AFL | Part 2: Trying Smarter(Apache)

Date: March 12, 2022

This document(‘Trying Smarter’) summarises my experience with Apache httpd and Advanced Fuzzing.

Re-cap from Part 1: Trying Harder

Analyzed the final build of redis-server in order to find interesting entry points.
Hot-patched the server code in a way that allows us to fuzz a very ‘strategic’ function (processInputBuffer): we added an exit() call, which avoids the server from hanging and continue to the next fuzzing attempt.
We used preeny’s desock.so in order to redirect incoming network traffic via stdin and make the final build ‘fuzzable’.

This setup was quite expensive in terms of performance, since it requires launching the server just to execute one fuzzing attempt each time. This can decrease the fuzzing speed, and get us a very slow coverage. To overcome this ‘slowness’ cost, I ‘tried Harder’ and purchased a (relatively) strong ec2 instance in order to launch lots of AFL instances to multiply(x32) the fuzzing speed.

Intro to Part 2: Trying Smarter

In this post, we will ‘Try Smarter‘: We’ll implement some of the ideas from the previous post on the Apache httpd server.

But this time, I will also share insights about my obstacles/how I managed to:

Improve the fuzzing speed by introducing a concept called ‘Persistent Fuzzing’
Improve the fuzzer’s stabillty
Get a better idea on how we can avoid self-DoS when fuzzing Apache (I killed my ec2 multiple times during this experiment lmao)

Initial setup Overview

Unlike the previous blogpost , I didn’t use preeny’s desock.so. Instead, I used n30m1nd approach. Essentially, what he was implementing is launching a new thread inside Apache that will create a connection to fetch the fuzzing input from stdin and send it internally in order to get the instrumentation working.

At first, I tried to apply his .patch file, but then I realized that the line-numbers are changed from one Apache version to another. So I decided I need a new file-modification approach that will be a bit more reliable/support more than one specific Apache version. To do this, I had to create my own setup and re-factor very small parts of the implementation. Now, instead of a .patch command, the substitution/hot-patching is done with python.

In order to get your own setup, clone this repo: https://github.com/0xbigshaq/apache-afl and just run the afl-toolchain.sh file. I made sure to automate the whole process so it will be easier to follow along.

The rest of the blogpost(below) will describe the proccess I went through in order to create/understand better the setup components.

Tips & Tricks

The `__AFL_LOOP` macro / Persistent Fuzzing

To speed up the fuzzing proccess, we’ll get to know a new concept called Persistent Fuzzing.

Quick intro for what is Persistent Fuzzing can be found in the AFL docs:

‘In persistent mode, AFL++ fuzzes a target multiple times in a single forked process, instead of forking a new process for each fuzz execution. This is the most effective way to fuzz, as the speed can easily be x10 or x20 times faster without any disadvantages. All professional fuzzing uses this mode’

__AFL_LOOP(1000) is a useful macro that detects if the program is running under AFL. If it is, the loop will run 1000 times with 1000 different inputs. If the program runs on its own(==not using AFL) the loop runs only once. This is quite useful since it enables us to use the same build to fuzz AND triage findings.

before:

Enabling persistent fuzzing: https://github.com/0xbigshaq/apache-afl/commit/eb6de675

after:

x132 faster, noice! now we can leverage that ec2 instance power to the max.

Note/Reminder: In the previous blogpost, we ran 32 instances of AFL just to get to 1300~ execs per second(each one was performing 40~ attempts per sec, which is very similar to what we had here), and when I did that, all the CPU cores were screaming. In this setup, we managed to achieve 4k~ execs per second with only one AFL instance and less than 50% cpu consumption. Now that’s what I call fuzzing smarter :D

Increasing Stabillity

After fixing the fuzzing speed, I encountered a different issue, the fuzzer’s stabillity was not as high as it should be. This can lead to false-positive crashes & bad coverage.

After reading about it more, turns out that this is possibly a side-effect of calls to PRNG functions(random and friends), I found a workaround in Antonio Morale’s research on Apache(scroll down to see referenecs/urls). What we need to do is to reduce entropy in order for the server to take a similar(or, not too different) execution paths when being fuzzed, he did it with a very clever technique, by just replacing the seed to a constant seed:

https://github.com/antonio-morales/Apache_2.5.1/commit/e0be82bc

    // rv = apr_generate_random_bytes(seed, sizeof(seed));
    char constant_seed[] = {0x78,0xAB,0xF5,0xDB,0xE2,0x7F,0xD2,0x8A};
    memcpy(seed, constant_seed, sizeof(seed));
    rv = APR_SUCCESS;

After applying this patch to my build and running make again, things were looking better:

We’re back in business, happy fuzzing :^)

Other Useful Tips

When fuzzing Apache, don’t forget to:

Redirect access logs to /dev/null: If you won’t do this, your server will DoS itself & your storage space will run out very quickly.
Redirect error logs to /dev/stdout: Much more useful for debugging purposes / getting errors feedback quickly when you’re trying out new configurations on the httpd server.
exit properly, because if you won’t: the fuzzer will start more and more Apache instances without closing the previous ones until your machine will exhaust.

Going Forward / Areas for Improvement

There are additional stuff in the repo that can be found/will be in the unstable branch, which is currently WIP(work-in-progress), among them are:

Adding LLVM + Compiling compiler-rt from source for better/verbose reports w/ symbols information.
Custom ASan interceptors for the APR allocator (instead of the default ones currently in-use)
Custom mutators w/ some HTTP Grammar
Sample configs for various Apache modueles

I think the most important guideline/part about this project is: The build process needs to be automated and as universal as possible(in order to support as much as Apache versions possible)

Once the work on the unstable branch is finished & merged to main, the whole setup will be automated/fully weaponized and it’s gonna blast. So stay tuned, star the repo and feel free to contribute.

io::pewpew()

Fuzzing with AFL | Part 2: Trying Smarter(Apache)

Re-cap from Part 1: Trying Harder

Intro to Part 2: Trying Smarter

Initial setup Overview

Tips & Tricks

The `__AFL_LOOP` macro / Persistent Fuzzing

Increasing Stabillity

Other Useful Tips

Going Forward / Areas for Improvement

Further Reading

Fuzzing Material

Analysis:

Setup:

Discussions:

Fuzzing with AFL | Part 2: Trying Smarter(Apache)

Re-cap from Part 1: Trying Harder

Intro to Part 2: Trying Smarter

Initial setup Overview

Tips & Tricks

The __AFL_LOOP macro / Persistent Fuzzing

Increasing Stabillity

Other Useful Tips

Going Forward / Areas for Improvement

Further Reading

Fuzzing Material

Analysis:

Setup:

Discussions:

The `__AFL_LOOP` macro / Persistent Fuzzing