This document(‘Trying Smarter’) summarises my experience with Apache httpd and Advanced Fuzzing.
Re-cap from Part 1: Trying Harder
In the previous post, we:
- Analyzed the final build of
redis-server
in order to find interesting entry points. - Hot-patched the server code in a way that allows us to fuzz a very ‘strategic’ function (
processInputBuffer
): we added anexit()
call, which avoids the server from hanging and continue to the next fuzzing attempt. - We used preeny’s
desock.so
in order to redirect incoming network traffic viastdin
and make the final build ‘fuzzable’.
This setup was quite expensive in terms of performance, since it requires launching the server just to execute one fuzzing attempt each time. This can decrease the fuzzing speed, and get us a very slow coverage. To overcome this ‘slowness’ cost, I ‘tried Harder’ and purchased a (relatively) strong ec2 instance in order to launch lots of AFL instances to multiply(x32) the fuzzing speed.
Intro to Part 2: Trying Smarter
In this post, we will ‘Try Smarter’: We’ll implement some of the ideas from the previous post on the Apache httpd server.
But this time, I will also share insights about my obstacles/how I managed to:
- Improve the fuzzing speed by introducing a concept called ‘Persistent Fuzzing’
- Improve the fuzzer’s stabillty
- Get a better idea on how we can avoid self-DoS when fuzzing Apache (I killed my ec2 multiple times during this experiment lmao)
Initial setup Overview
Unlike the previous blogpost , I didn’t use preeny’s desock.so
. Instead, I used n30m1nd approach. Essentially, what he was implementing is launching a new thread inside Apache that will create a connection to fetch the fuzzing input from stdin
and send it internally in order to get the instrumentation working.
At first, I tried to apply his .patch
file, but then I realized that the line-numbers are changed from one Apache version to another. So I decided I need a new file-modification approach that will be a bit more reliable/support more than one specific Apache version. To do this, I had to create my own setup and re-factor very small parts of the implementation. Now, instead of a .patch
command, the substitution/hot-patching is done with python.
In order to get your own setup, clone this repo: https://github.com/0xbigshaq/apache-afl and just run the afl-toolchain.sh
file. I made sure to automate the whole process so it will be easier to follow along.
The rest of the blogpost(below) will describe the proccess I went through in order to create/understand better the setup components.
Tips & Tricks
The __AFL_LOOP
macro / Persistent Fuzzing
To speed up the fuzzing proccess, we’ll get to know a new concept called Persistent Fuzzing.
Quick intro for what is Persistent Fuzzing can be found in the AFL docs:
‘In persistent mode, AFL++ fuzzes a target multiple times in a single forked process, instead of forking a new process for each fuzz execution. This is the most effective way to fuzz, as the speed can easily be x10 or x20 times faster without any disadvantages. All professional fuzzing uses this mode’
__AFL_LOOP(1000)
is a useful macro that detects if the program is running under AFL. If it is, the loop will run 1000 times with 1000 different inputs. If the program runs on its own(==not using AFL) the loop runs only once. This is quite useful since it enables us to use the same build to fuzz AND triage findings.
before:
Enabling persistent fuzzing: https://github.com/0xbigshaq/apache-afl/commit/eb6de675
after:
x132 faster, noice! now we can leverage that ec2 instance power to the max.
Note/Reminder: In the previous blogpost, we ran 32 instances of AFL just to get to 1300~ execs per second(each one was performing 40~ attempts per sec, which is very similar to what we had here), and when I did that, all the CPU cores were screaming. In this setup, we managed to achieve 4k~ execs per second with only one AFL instance and less than 50% cpu consumption. Now that’s what I call fuzzing smarter :D
Increasing Stabillity
After fixing the fuzzing speed, I encountered a different issue, the fuzzer’s stabillity was not as high as it should be. This can lead to false-positive crashes & bad coverage.
After reading about it more, turns out that this is possibly a side-effect of calls to PRNG functions(random
and friends), I found a workaround in Antonio Morale’s research on Apache(scroll down to see referenecs/urls). What we need to do is to reduce entropy in order for the server to take a similar(or, not too different) execution paths when being fuzzed, he did it with a very clever technique, by just replacing the seed to a constant seed:
https://github.com/antonio-morales/Apache_2.5.1/commit/e0be82bc
After applying this patch to my build and running make
again, things were looking better:
We’re back in business, happy fuzzing :^)
Other Useful Tips
When fuzzing Apache, don’t forget to:
- Redirect access logs to
/dev/null
: If you won’t do this, your server will DoS itself & your storage space will run out very quickly. - Redirect error logs to
/dev/stdout
: Much more useful for debugging purposes / getting errors feedback quickly when you’re trying out new configurations on the httpd server. exit
properly, because if you won’t: the fuzzer will start more and more Apache instances without closing the previous ones until your machine will exhaust.
Going Forward / Areas for Improvement
There are additional stuff in the repo that can be found/will be in the unstable
branch, which is currently WIP(work-in-progress), among them are:
- Adding LLVM + Compiling
compiler-rt
from source for better/verbose reports w/ symbols information. - Custom ASan interceptors for the APR allocator (instead of the default ones currently in-use)
- Custom mutators w/ some HTTP Grammar
- Sample configs for various Apache modueles
I think the most important guideline/part about this project is: The build process needs to be automated and as universal as possible(in order to support as much as Apache versions possible)
Once the work on the unstable
branch is finished & merged to main
, the whole setup will be automated/fully weaponized and it’s gonna blast. So stay tuned, star the repo and feel free to contribute.
Further Reading
To achieve this, I had to read a lot of sh*t, on many different sources. And intuitively connecting the dots. In this section, I will share all the useful stuff I found so you won’t have to go thorugh the same proccess, hopefully it will help someone reading this on the internet:
Fuzzing Material
- Persistent fuzzing &
__AFL_LOOP
: https://toastedcornflakes.github.io/articles/fuzzing_capstone_with_afl.html - 2017, Javier Jimenez: Fuzzing Apache httpd server with American Fuzzy Lop + persistent mode
- 2021, Antonio Morales amzing series:
- More of Antonio Moarles: Fuzzing101
- Apple Conference Talk(WWDC-2015): Advanced Debugging and the Address Sanitizer – Mike Swingler, Anna Zaks
- dayzerosec’s podcast/stream: https://youtu.be/crWjsXvVZxg?t=2102
- Book: Fuzzing: Brute Force Vulnerability Discovery
Analysis:
- Automated setup for Apache + gdb-gui
- rr-debugger tool: for better root-cause analysis.
Setup:
- To overcome the painful
./configure
and apache’s DSOs/dynamic modules: http://victoriastation.it/manual/en/programs/configure.html - To understand how to deal with APR and APR-util: https://publib.boulder.ibm.com/httpserv/manual24/install.html
- To compile everything combined(with deps included) from sources: https://arjang.ac.ir/downloads/Landings/Hooshmandi/compile-apache-2.4-from-source.pdf
- To run the fuzzing container with full capabilities(tmpfs, cores, etc.): https://docs.docker.com/reference/commandline/run/#full-capabilities
Discussions:
- afl-users Google Group: https://groups.google.com/g/afl-users
- address-sanitizer Google Group: https://groups.google.com/g/address-sanitizer
- LLVM Disclosure: https://discourse.llvm.org/
- ‘Awesome Fuzzing’ Discord channel: https://discord.gg/HYReYDtgx2
- twitter: @fuzzinglabs, @fuzztheworld