Saturday, June 26, 2010

The bug is on the table

A few months ago there was an interesting story in the company where I work. One guy from a neighboring department asked me to help him. He encountered one crash after another over the past few days. Unusual in these crashes was that they occurred in different processes and under different circumstances, but within one computer on the test bench (consisted of 5 machines) and with identical symptoms. In all cases it was an access violation caused by illegal instruction pointer.

Here is an example of one of the call stacks:

0:000> knL
 # ChildEBP RetAddr 
00 0012d7f8 7c90e9ab ntdll!KiFastSystemCallRet
01 0012d7fc 7c8094f2 ntdll!ZwWaitForMultipleObjects+0xc
02 0012d898 7c809c86 kernel32!WaitForMultipleObjectsEx+0x12c
03 0012d8b4 6945763c kernel32!WaitForMultipleObjects+0x18
04 0012e248 694582b1 faultrep!StartDWException+0x5df
05 0012f2bc 7c863059 faultrep!ReportFault+0x533
06 0012f530 7c36e289 kernel32!UnhandledExceptionFilter+0x4cf
07 0012f54c 0041b84f msvcr71!_XcptFilter+0x15f
08 0012f558 7c363943 radar!WinMainCRTStartup+0x1d7
09 0012f580 7c9037bf msvcr71!_except_handler3+0x61
0a 0012f5a4 7c90378b ntdll!ExecuteHandler2+0x26
0b 0012f654 7c90eafa ntdll!ExecuteHandler+0x24
0c 0012f654 fc910d93 ntdll!KiUserExceptionDispatcher+0xe
WARNING: Frame IP not in any known module. Following frames may be wrong.
0d 0012f950 1f4d5318 0xfc910d93
0e 0012fa10 7c363593 0x1f4d5318
0f 0012fa58 087e6827 msvcr71!free+0xc3
10 0012fa8c 087e6f41 NaviRadarRendererPlain!std::vector,std::allocator > >::_Insert_n+0x147
11 0012fafc 087e46f1 NaviRadarRendererPlain!TScanlines::calculate+0x5d1
12 0012fc2c 087e418e NaviRadarRendererPlain!TRadarToScreen_::RenderBlock+0x521
13 0012fc64 087e2378 NaviRadarRendererPlain!TRadarToScreen_::Render+0x16e
14 0012fc88 087c4cf0 NaviRadarRendererPlain!n_d3d_render::renderer_t::render+0x68
15 0012fce0 03d5c5e7 NaviRadarLayer!TRadarLayerImpl::Draw+0x360
16 0012fd1c 03d55c8e TotUser!TRadarManager_::Update+0x77
17 0012fd4c 087c821a TotUser!TCrtPanel_::InvalidateRadar+0xbe
18 0012fd54 10ac8959 NaviRadarLayer!TRadarLayerImpl::OnVideoBlockReceived+0x7a
19 0012fd7c 10ac8ab6 TkRadar20!IAdviseHostImpl::TypedForEach+0x49
1a 0012fd90 10ac8e88 TkRadar20!RadarConnector::VideoBlockReceived+0x26
1b 0012fda0 77d48709 TkRadar20!detail::WndProc+0x48
1c 0012fdcc 77d487eb user32!InternalCallWinProc+0x28
1d 0012fe34 77d489a5 user32!UserCallWinProcCheckWow+0x150
1e 0012fe94 77d4bccc user32!DispatchMessageWorker+0x306
1f 0012fea4 7c1b1645 user32!DispatchMessageA+0xf
20 0012feb4 7c1b1357 mfc71!AfxInternalPumpMessage+0x3e
21 0012fed0 0040acc0 mfc71!CWinThread::Run+0x54
22 0012ff08 7c1ae5f1 radar!CECSNTApp::Run+0x30
23 0012ff18 0041b7fd mfc71!AfxWinMain+0x68
24 0012ffc0 7c816d4f radar!WinMainCRTStartup+0x185
25 0012fff0 00000000 kernel32!BaseProcessStart+0x23