What I’ve Been Doing On My Summer Vacation or, “It has to work; Otherwise gdb wouldn’t”

tduehr | July 17th, 2008 | Filed Under: Apple, Uncategorized

An intern expects to be given simple projects, like coffee retrieval, or “Hello, World.” So I’ve been sorely disappointed by Matasano. I have been offered coffee retrieval services by senior engineers and my latest project has been anything but “Hello, World.”

In fact, it’s been more like, “Hello, OS X. Tell me your secrets”.

This is the story of one trial-by-fire project handed to an intern that turned out to be more complicated than anyone expected.

1.

It started with Thomas, innocently enough, handing me some debugger code. It was both C and Ruby, and for Solaris and Win32. He said, “I would like you to port this Win32 Ruby code to OS X.”

“Um, okay.”

At that point I’d just finished learning the basics of Ruby via my previous Matasano project, a database backed HTTP proxy. I knew nothing about debuggers, let alone the low level C library calls I’d need and Ruby bindings to make them work. I know, fun, right?

I started simply and dusted the C off in my head so I could begin to read and understand the code Thomas dumped on me, and perhaps learn how a debugger works and gets used. It took a day or two just to read it. I’d ask the office some fairly basic question about debuggers, and receive in return a much longer response than I’d anticipated. Like a tutorial on the workings of x86 assembly. Eventually, I got to a point where I was almost comfortable with how the C debugger worked.

When staring at C code stopped doing me any good, and writing Ruby code started seeming feasible, I moved on to porting the Ruby code. “How hard could it be?”.

2.

Thomas gave me a starting point. Our Ruby code called directly into C libraries using Win32API and Ruby/DL. We have wrapper libraries that make those C calls look like Ruby library functions. So, for instance, in our Wrap32 library, we have:

# just grab some local memory
    def malloc(sz)
      r = CALLS["msvcrt!malloc:L=L"].call(sz)
      raise WinX.new(:malloc) if r == 0
      return r
    end

We had a small piece of this written for OS X as well. I had to build it out. I started with getpid(), a simple system call I could make sure worked before I moved on to something harder. It worked right away. My confidence was high. I was feeling cocky.

Here I should mention that I’d never worked on a decently large coding project before. This was my first.

Throughout this entire project I’ve been trying to write the entire thing far before I actually write even a single function. So, I had many questions:

  • What was the script implementing the debugger to look like?

  • Was it to be event driven?

  • Did we want objects to represent each process, threads, or to make his lunch for him?

I was overzealous. The team was patient. Thomas said simply, “There is no spoon. You’ll need ptrace() and wait() for the breakpoint insertion and signal catching. Just copy the functionality from the Win32 version.”

3.

An brief word from the team about how debuggers work.

The thing you most want to do with a debugger is set and handle breakpoints. On X86, there are two kinds of breakpoints: hardware and software. You mostly use software breakpoints. They way software breakpoints work is, you pick the place in the program you want to break at, and you replace the instruction at that point with “INT 3″ (conveniently enough, this is just the byte “0xCC”). When the program hits the INT instruction, it generates an interrupt. The OS catches the interrupt and kills the program.

Unless you have a debugger attached. If you have a debugger attached, instead of killing the program, the OS tells the debugger. The debugger then swaps the original instruction back in, “rewinds” the prograam back to it, and resumes execution.

Every OS has debugging features. They boil down to the following four capabilities:

  • Reading and writing the memory of another process (that’s how you swap INT in for instructions to set breakpoints).

  • Catching events from other processes, like breakpoint interrupts.

  • Starting, stopping, and pausing threads inside other processes.

  • Changing the register state in other processes, for instance by moving the EIP register back 1 byte to rewind the INT 3 instruction that just fired.

The best known Unix debugger interface is ptrace(), and it basically does all four of those things for you, along with the wait() call for detecting events. On Win32, any program can read or write from a process it has the right permissions for, even if it isn’t a debugger; the debugger mostly exists to catch interrupts.

4.

Coding the wrappers for ptrace(), wait(), and waitpid() didn’t take too long. Each just takes a few integers and returns an integer. But ptrace works with request codes, like “PEEK” to read memory or “STEP” to single-step the process. I couldn’t test without knowin all the request codes. So, I started reading man pages, poking at code and trying to get my OS X functions to work.

“To the headers!” I cried. But which one and where are they? As I mentioned, I’m a little new to real — as in non-academic — programming. Google worked OK to get the man pages, but didn’t include the request code numeric values, just the names and what they did. Frustrated, I asked for help.

“find /usr/include | xargs grep ptrace | less” was the response I got from Thomas. You didn’t know he speaks *nix? He does. Hexadecimal too, from what I’ve heard.

A little reading and some copying later I had the constants I needed, and began to test my ptrace and wait functions. The code wasn’t pretty but it seemed to work. I could attach to a process by PID and wait() for it. Now I just needed to get its registers and I’d be almost done.

It didn’t take long to sketch my code based on the Win32 debugger I was given to start with. Soon I had what I thought was the start of a functional debugger in Ruby, along with a handy explanation of the Ruby way of doing things. Up until that point I’d been trying to do things the C way, passing variables by reference, trying to make the Ruby function call an exact match to the C call, and other things I’d picked up from the C/C++/JAVA I learned in college.

I thought I was doing well. Then I tried to find the OSX equivalent of PTRACE_GETREGS to read the registers from other processes, which is kind of important for debuggers.

5.

Here everything starts to get more complicated.

It turns out Apple, in their infinite wisdom, had gutted ptrace(). The OS X man page lists the following request codes:

  • PT_ATTACH — to pick a process to debug

  • PT_DENY_ATTACH — so processes can stop themselves from being debugged

  • PT_TRACE_ME — so debuggers can launch processes that start debugged

  • PT_CONTINUE — to restart a program after it’s been stopped

  • PT_STEP — to execute just one instruction in the process

  • PT_KILL — to kill the process

  • PT_DETACH — to release the process

No mention of reading or writing memory or registers. Which would have been discouraging if the man page had not also mentioned PT_GETREGS, PT_SETREGS, PT_GETFPREGS, and PT_SETFPREGS in the error codes section. So, I checked ptrace.h. There I found:

  • PT_READ_I — to read instruction words

  • PT_READ_D — to read data words

  • PT_READ_U — to read U area data if you’re old enough to remember what the U area is

  • PT_WRITE_I — and write instructions

  • PT_WRITE_D — and data

  • PT_WRITE_U — and U

  • PT_SIGEXC — and EXC SIGs

  • PT_THUPDATE — and update THs

  • PT_ATTACHEXC — and attach EXCs

There’s one problem solved. I can read and write memory for breakpoints. But I still can’t get access to registers, and I need to be able to mess with EIP.

That’s when I start hearing “It has to work, otherwise gdb wouldn’t”, rather frequently, from more than one person.

Well, ptrace() won’t work for retrieving registers in OS X.

Matasano Secret Intern X referred me to Nemo’s article at uninformed.org. In it, Nemo lays out the Mach kernel calls that replace some of the lost ptrace() functionality. So, I wrote wrappers for:

  • task_for_pid — to find the Mach task of an OS X process

  • mach_task_self — to get my debugger’s task

  • task_threads — to walk the threads inside a task

  • thread_get_state — to get the registers for one of those threads

  • thread_set_state — to change those registers

Since I wasn’t using them natively in C I needed to know more about the usage of each function.

“No problem,” I thought, “I’ll just fire up terminal and… Oh, bloit!” No man pages.

I pored over Nemo’s work, what I could find in the headers, and figured out how to call the functions. Now another problem. The Mach functions take pointers to raw C memory.

The way I was told to handle this was, pack the data I needed into Ruby strings or native numeric types with Ruby/DL. After a long, dark period of messing with calls to “strdup” and “DL.malloc”, I found “String#to_ptr”, and at last managed to get the Mach functions working.

I had also found the correct way to get errno through Ruby/DL: DL.last_error. This appears to be documented nowhere in English.

Except for an odd bus error I ran into now and then (but couldn’t duplicate), my Ruby debugger was working and could read and write registers. I’d even checked to make sure they were coming back to me in the correct sequence.

Then, running my get_registers() function repeatedly, I found the registers of a stopped process changing on every call. When I printed them without marshalling they contained the names of some of the functions I’d written occasionally.

“Oh, bloit! I’m really chakked now. I’ve been calling a bloitting buffer overflow a register lookup,” I said to myself. I despaired of my project and my future.

6.

On the train home and all weekend I looked through Apple’s documentation. Google. The header files “It has to work; Otherwise gdb wouldn’t,” another friend said. But he wasn’t able to find the documentation I was looking for. He did find fxr.watson.org and some better explanations of the functions at web.mit.edu/darwin/src/modules/xnu/osfmk/man/. Those turned out to be gold later.

During week one of coding:

  • several necessary functions wrapped and working

  • DL.txt is really the only Ruby/DL documentation that exists

  • Ruby/DL is great for simple C function wrapping but rough around the edges when it comes to more interesting calls.

  • Avergage familiarity with Ruby

  • Basic understanding of how a debugger works

  • A Ruby object that can attach to a process, continue it, detach from it and wait() for it.

  • One really convoluted method to read/write random locations in memory

  • Average familiarity with system calls in C (now rust free)

7.

Starting the following week, things went a little smoother.

I had my coding flow going. I had better documentation than just header files. I started reading the Mach kernel code.

I wrote a small program in C to test the sequence of system calls I was using in Ruby. If It worked in C, why didn’t it work in Ruby? Then, I found it. I was calling task_threads() wrong, passing an pointer where it expected a pointer-to-pointer. Whee! I vetted the results with gdb’s output.

My code said:

"regs = ["c0003", "32390", "bffff74c", "90e441ba", "0", "0", "bffff768", "bffff74c", "1f", "286", "90e441ba", "7", "1f", "1f", "0", "37"]"

gdb replied:

eax            0xc0003786435
ecx            0xbffff74c-1073744052
edx            0×90e441ba-1864089158
ebx            0×32390205712
esp            0xbffff74c0xbffff74c
ebp            0xbffff7680xbffff768
esi            0×00
edi            0×00
eip            0×90e441b50×90e441b5
eflags         0×286646
cs             0×77
ss             0×1f31
ds             0×1f31
es             0×1f31
fs             0×00
gs             0×3755

They agreed! I went home for the day.

8.

Now for wait(), to catch debugger events. wait() was hanging the debugger if I called it more than once. I set it up to use the NOHANG option. I fixed an return value error.

Then, I tested single-stepping with ptrace. Kernel panic.

I put that on the list of broken parts of ptrace to be replaced by a Mach call.

Next up was setting breakpoints. They seemed to install themselves without error but the child wasn’t stopping when ran the command that would hit the breakpoint I’d set. Upon inspection, the breakpoint was replacing an instruction of -1. Which gdb told me was actually 0x55.

I started researching the problem, finding only hints. Did I mention ptrace was gutted in OS X? I read the source for Apple’s version of gdb. Thomas gave me a copy of a DTrace truss and said, “Just do whatever gdb does.”

It took me a while to get the script working. It seems iTunes causes errors in truss (also dtruss) whenever it’s running. I closed iTunes and started using watching gdb for ptrace calls. Rather quickly I noticed an extreme lack of call to ptrace.

Was gdb even using ptrace for reading the process’ memory?

(gdb) PID/LWP   SYSCALL(args)  = return
break *0×420f
Breakpoint 1 at 0×420f
(gdb) run
Starting program: /usr/bin/ftp
Reading symbols for shared libraries ++++. done
ftp>   939/94968960:  ptrace(0×0, 0×0, 0×0, 0×0) = 0 0
939/94968960:  ptrace(0xC, 0×0, 0×0, 0×0) = 0 0
930/66961480:  ptrace(0xD, 0×3AB, 0×2C1B, 0×0) = 0 0
930/66961480:  ptrace(0xD, 0×3AB, 0×2C1B, 0×0) = 0 0
930/66961480:  ptrace(0xD, 0×3AB, 0×2C1B, 0×0) = 0 0

It became apparent ptrace was only really used by gdb to:

  • prevent the process from exiting on signals

  • passing signals to the child after it processed them.

I then remembered that uninformed.org article. A quick read reminded me that Mach vm_read and vm_write were needed to replace PT_READ and PT_WRITE.

The next day, Thomas was in the office to check on my progress. To move things along he implemented vm_read and vm_write for me while I confirmed a few things with truss and looked for vm_read calls in gdb. I didn’t find any. When he finished the functions, I used them in my breakpoint setting routines. No errors.

No stopping at breakpoints either.

Again the instructions were -1. When I mentioned this Thomas informed me I’d probably need vm_protect as well. Why hadn’t I thought of that? Not too long after that I was able to set and remove breakpoints correctly! I went home for the long weekend.

During week two of coding:

  • wrapped and implemented all necessary system calls

  • added thread state and breakpoint manipulation to Debuggerx

  • gained some knowledge of OS X internals

  • found a repeatable kernel panic

  • learned basic usage of dtrace and gdb

  • learned I tend to overthink my code before writing it

  • began to use irb as a scratch pad for testing functions

9.

Now another problem. You can set a breakpoint with the debugger. You can catch the breakpoint. You can resume the process. But you can’t reset the breakpoint without single stepping: to resume the process, you have to clear the breakpoint.

But PT_STEP was panicking the kernel!

I settled on setting the TRAP flag in the EFLAGS register to simulate single-stepping with ptrace. This seemed to work. But now I’m getting bus errors when I resume the process. I verified with Thomas how they were supposed to work. I tried watching gdb for vm_write from truss again, nothing. After some debugging I discovered waitpid() was clearing the trap flag, which Thomas informed me was correct behavior. Some more monkeying around trying to get it working ate up the rest of the day.

The next day, I was able to pass through a breakpoint and reset it. Only problem was, the breakpoint wasn’t being reset fast enough, it wasn’t done immediately one step after it was hit. After clearing some confusion on my part with Thomas, I decided to try PT_STEP again. It worked and didn’t panic the kernel this time. Finally, I had a debugging tool that was complete!

All that remained was to clean up some debug tracing prints and implement a better method to view the registers. Both fairly simple things completed early the next day.

10.

There it is, the story of the birth of DebuggerX. A “simple” porting task handed to an intern to better his understanding of debuggers and Ruby. During the project I’d become quite familiar with Ruby, learned some OS X internals, found a kernel panic in ptrace, and learned better programming technics. I still tend to overthink my code and “have a hard time believing that you’re supposed to ask programs to do the things it looks like they need to do,” according to Thomas, but I have learned it’s quite a bit easier to try something in code than in your head. Since completion of the project as originally stated, I’ve added calls to get information about a thread and began looking into retrieving a list of function symbols from the process’ file. I’ll make another post about that in the future.

Viewing 42 Comments

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus