Mangling win32 executables with a hex editor

This is a short note about how to make small manipulations in executables or DLLs in order to get rid of malware behaviour. For example, if some application pops up a dialog box which I’d like to eliminate. It can also be the final step in cracking (which is very recommended as an educational experience).

Keep in mind that getting something useful done with this technique requires a very good understanding of assembly language, and how high-level languages are translated into machine code.

The general idea is to hook up a debugger (Microsoft Visual Studio’s will do, using Tools > Debug Processes), and try to get a breakpoint exactly where the bad things happens. Then, after verifying that there is a clear relation between the certain point in the code to the undesired behavior, use the debugger to skip it a few times, in order to be sure that it’s the fix. Tracing the API calls can be very helpful in finding the crucial point, as I’ve explained in another post. But if the offending behavior involves some message box (even if the popup only announces the issue), odds are that the critical point can be found by looking at the call stack, when attaching the debugger to the process, with the popup still open. Look for where the caller is the application itself (or the related DLL/OCX).

Lastly, the code must be changed in the original file. Fortunately, this can be done with a hex editor, since no CRC check is performed on executables.

One thing to bear in mind is that the code is possibly relocated when loaded into memory. During this relocation, absolute addresses in the machine code are mangled to point at the right place. This is why any opcode, which contains an address can’t be just changed to NOPs: The linker will do bad things there.

A crucial step is to match the memory viewed in the debugger with a position in the file. First we need to know where the EXE, DLL or OCX are mapped in memory. The Dumper application from the WinAPIOverride32 suite gives the mapped addresses for each component of a process, for example. The file is typically mapped linearly into memory, with the first byte going to the first address in memory. An application such as the PE Viewer (sources can be downloaded from here) can be helpful in getting a closer look on the Portable Executable data structure, but this is usually not necessary.

Once the hex data in the file matches what we see in the debugger, we’re left with pinpointing the position in the hex editor, and make the little change. There are a few classic simple tricks for manipulating machine code:

• Put a RET opcode in the beginning of the subroutine which does the bad thing. RET’s opcode is 0xC3. Eliminating the call itself is problematic, since the linker may fiddle with the addresses.
• Put NOPs where some offending operation takes place. NOP’s opcode is 0x90. Override only code which contains no absolute adresses.
• Insert JMPs (opcode 0xEB). This is cool in particular when there is some kind of JNE or JEQ branching between desired and undesired behavior. Or just to skip some piece of code. This is a two-byte instruction, in which the second byte is a signed offset for how far to jump. Offset zero means NOP (go to the next instruction).

When the mangling is done, I suggest opening the application with the debugger again, and see that the disassembly makes sense, and that the change is correct. Retaining the break point once is good to catch the critical event again, and see that the program flows correctly from there on.