Believe not all that you see

Last week there was very interesting story that can be useful for debugger users. During the application debugging (using WinDbg) we encountered with stable repeated crash. The source of the crash was found easily, but it confused us completely. The problem was in one processor instruction. The instruction add eax,10h was ended up with subtraction of 34h from eax register.

We had checked it several times but the result was absolutely identical. After that, we made several experiments with the following results:

1.    Whatever we put into the place of 10h, the result was subtraction of 34h from eax register.

2.    After we moved this instruction three bytes above or bellow the current adress (just noping the gap where instruction was) the effect disappeared immediately. But when we returned it back in its palce, the effect returned as well.

After some time of puzzling over I looked into the breakpoints list and noticed that there was a breakpoint set at faulting instruction. But instruction wasn’t highlighted in the debugger. When I checked the address of the breakpoint, I found out that the breakpoint was set not inside the first instruction byte (like it should be), but inside the third byte – that is in the place of 10h operand. Of course this “breakpoint” didn’t act like an actual breakpoint, processor just executed add eax,CCh instruction instead. And because of debugger hides all its breakpoints before it’s pass the control to user, we always saw add eax,10h instruction instead of real one.

So, the obvious question is: Why the debugger set this breakpoint?

And the answer is simple. The unresolved breakpoint was set during another debugging session of this process, and since the process binaries and symbols set were changed, the debugger figured out wrong address for the breakpoint (in fact, completely wrong :)). And because of debugger still considered this breakpoint as unresolved, we were encountering with this crash over and over again after restart of the process.