Linux Syscall Tracer

Inspiration

When an analyst gets a suspicious binary, the first question is what it actually does. Source code is rarely available, and reading disassembly for every sample is slow. Almost everything a program does to a system goes through system calls: opening files, starting processes, writing data, opening sockets. If you can watch those calls, you can see the behavior without the source.

The standard tool for this is strace. I wanted to understand how a tracer like that works by building one from scratch, and to shape its output for forensics: the trace is kept separate from the program's own output, so it can be saved as evidence.

What it does

The tracer runs any Linux program and prints every system call it makes, with arguments and results:

• Names each call (about 65 common x86_64 syscalls) and prints unknown ones as syscall_N(...).
• Decodes path arguments, such as the files a program opens, as strings read from the program's own memory.
• Decodes execve completely, including the launch of the traced program and its argv array.
• Decodes the contents of write buffers, with escapes for binary data.
• Prints failures with the error text, for example = -1 (No such file or directory).
• Writes the trace to stderr, so 2> trace.log captures a clean trace.

How I built it

The tracer is written in C and uses ptrace, with a Makefile and no other dependencies.

1. The tracer forks. The child calls PTRACE_TRACEME and stops itself, and the parent attaches its options and resumes it with PTRACE_SYSCALL. The child then stops at every syscall boundary.
2. Stops come in pairs, an entry and an exit. At entry I read the registers: the syscall number is in orig_rax and the arguments are in rdi, rsi, rdx, r10, r8, r9. The return value is read from rax at exit.
3. A pointer argument is only meaningful inside the traced program, so I read the data it points to word by word with PTRACE_PEEKDATA. For example, the $i$-th entry of an argv array lives at

$$\text{addr}_i = \text{base} + i \cdot \texttt{sizeof(long)}$$

and each entry is itself a pointer to a string, which takes a second read.

1. The kernel reports a failed call by returning a negative errno in the range

$$-4095 \le \texttt{rax} \le -1$$

so a result in that range is printed as an error, and anything else as a normal value.

I built it one function at a time: the loop first, then names and return values, then raw arguments, then string decoding, then execve handling, then the write buffer.

What I learned

• How a debugger-style tracer controls another process, and why ptrace calls only work while the traced process is stopped.
• Why the syscall number is read from orig_rax and not rax, and why arguments must be read at the entry stop and the result at the exit stop.
• That reading another process's memory is a different problem from reading your own: a pointer is only an address in someone else's address space.
• How execve replaces a process image, and how much of the tracer has to be written around that fact.
• How signals, ptrace events, and syscall stops are all reported through the same waitpid call and have to be told apart.

Challenges

• A successful execve killed the traced program. The kernel sends a SIGTRAP after an exec. My loop treated every stop that wasn't a syscall as a real signal and passed it back to the child, which terminates on SIGTRAP. The fix was to enable PTRACE_O_TRACEEXEC, so an exec shows up as a distinguishable event stop, and to resume event stops with no signal.
• The launch execve was invisible. The program's own execve ran before the tracer had started watching syscalls. The child now stops itself before the exec, so the tracer is attached first.
• argv is a pointer to pointers. Printing it takes two levels of reads, a cap on how many entries to show, and handling for unreadable memory, with the NULL terminator as the only signal of where the array ends.
• The last syscall has no exit. exit_group never returns, so the entry and exit pairing is left open at the end, and that case has to be handled explicitly.
• Mixed output. The trace and the program's output were interleaved in one stream. Moving the trace to stderr made the two separable with shell redirection.

Limitations and what's next

The tracer follows only the original process, so child processes and threads are not traced. It supports x86_64 only, and arguments other than paths, execve argv, and write buffers are shown as raw hex values. Next I would add child following, decoding of socket addresses, and a viewer for saved traces.

Built with

C, ptrace, GNU Make, Linux (WSL2)

Built With

• c
• git
• linux
• makefile
• markdown
• ptrace