On Linux, most---if not all---debuggers use the ptrace debugging API to control their target processes. However, ptrace proves unsatisfactory for many malware analysis and reverse engineering tasks: So-called split-personality malware often adapts its behavior in the presence of a debugger, yet ptrace makes no attempt to hide from a target process. Furthermore, ptrace enforces a strict one-to-many relation meaning that while each tracer can trace many tracees, each tracee can only be controlled by at most one tracer. Simultaneously, the complex API and signal-based communications provide opportunities for erroneous usage.
Previous works have identified the newer uprobes tracing API as a candidate for building a replacement for ptrace, but ultimately rejected it due to lack of practical use and documentation. Building upon uprobes, we introduce plutonium-dbg, a Linux kernel module providing debugging facilities independent of the limitations of ptrace alongside a GDB-compatible interface. Our approach aims to mitigate some of the design flaws of ptrace that make it both hard to use and easy to detect by malicious software.
We show how plutonium-dbg's design and implementation remove many of the most frequently named issues with ptrace, and that our method improves on traditional ptrace-based debuggers (GDB and LLDB) when evaluated on software samples that attempt to detect the presence of a debugger.
Tobias Holl is a computer science student at TUM with a passion for reverse engineering and IT security. By day, he develops high-performance parallel software in C++, with a focus on computer vision and machine learning.
Philipp Klocke is a hacker, nerd and tech-enthusiast. He occasionally plays CTF and pursues a B.Sc. at the Technical University of Munich.
Since 2018 Fabian Franzen is a PhD student and researcher at the Chair of IT-Security of the Technical University of Munich (TUM). When he is not trying to teach his students the foundations of IT security, he is interested in various research topics. More specifically, these are reverse engineering, binary exploitation, Android security and improving systems security by introducing additional features to the Linux Kernel.