Can lightning strike the same place twice? (Or can a bomb to hit an elephant?)

In the old soviet movie about World War II was mentioned the story about the old Professor of Mathematics. When Leningrad has been bombing he is never descended to bombshelter along with other people. He told that he calculated the probability of bomb hitting his house and the probability was too small to hiding from bombs. But one day his neighbors saw him in a bombshelter and asked why is he hiding with others. He replied that on that day a bomb hit the only elephant in the cities zoo, and probability of hitting the elephant was absolutely equal with probability of been hit himself.

Why did I remember this story? Because during the last week I encountered twice with the failures which was the result of circumstances those probability was very low. Like people say: “If gun is hanging on the wall it will shoot one day or another”.

Why it's not crashing? (conclusion)

Well, today I received the final message from MS where they told me once again that there is nothing can be done on their side and advised me to catch exception in user code and call MiniDumpWriteDump after that. I think the story can be considered finished.

Se la vie, mes amis. Se la vie.

In search of the Dr. Watson

A week ago I returned from the business trip to Italy. It was a trip to the yacht where was the real problems with the navigation system stability. One or two times a day all navigation data from sensors freeze and the navigation equipment was inoperative until full bridge (5 stations) rebooting.

Just before the trip I received some information concerning the problem from our service engineer.
There were several observations:

1.    All navigation data freeze simultaneously but the navigation system (radar, cartographic system and over subsystems) continued to work without any other noticeable problems.

2.    Sometimes the message box with “Pure virtual function call” notification was observed after data freezing.

3.    First time the problem was observed just after the last update of the product.

4.    In the end I even received HD images from all stations of the bridge (they was created after the last update and after the problem appeared for the first time).


Description of the problem was clearly pointed to the fact that the cause of the problem is crashing (or freezing) of the navigation server processes at all stations simultaneously (each computer has its own navigation server executed and described problem is only possible if there is no servers alive). The only thing that confused me was mentioned earlier “Pure virtual function call” error. This error is not very common and occurs rare enough in its pure form. Most common cause of the error is improper sequence of objects chain deinitialization and destruction as well as undigested strategy of objects ownership (of course I’m not take into account errors with calling pure virtual method of child class inside a parent class destructor or deleting an object twice – I believe that our code better :). One of the classic situations caused by assertion message box triggered during object destruction (usually in debug mode). Message box message loop can provoke a call (through a pure virtual interface) to a partially destroyed object. As a result we will have clear pure virtual function call.

Why it's not crashing?

Several months ago, my colleagues encountered a very strange behavior of one of our processes after an exception was thrown. They told me that they know exactly that the process throws an exception which is not expected to be handled by their code, that is an exception must be unhandled and the process must be terminated immediately. But nothing like that happens. The process just continues its execution. At first, I thought that there must be some code down the stack which registers an exception handler that handles any exception indiscriminately. Therefore, the first thing I did was search for the try{}catch(…){} pattern down the stack. When I found nothing, I looked at the exception handlers list at the point where an exception was thrown.

It looked as follows:

0012fd14: USER32!_except_handler3+0 (7e440457)   CRT scope  0, func:   USER32!UserCallWinProc+10a (7e44aa1c) 0012fd6c: USER32!_except_handler3+0 (7e440457)   CRT scope  0, filter: USER32!DispatchMessageWorker+113 (7e440712)                 func:   USER32!DispatchMessageWorker+126 (7e44072a) 0012ffb0: VanishExcept!ILT+375(__except_handler3)+0 (0041117c) 0012ffe0: kernel32!_except_handler3+0 (7c839af0)   CRT scope  0, filter: kernel32!BaseProcessStart+29 (7c84377a)                 func:   kernel32!BaseProcessStart+3a (7c843790) Invalid exception stack at ffffffff

There were no user exception handlers at all, but instead was an exception handler I have not seen before:

0012fd14: USER32!_except_handler3+0 (7e440457)   CRT scope  0, func:   USER32!UserCallWinProc+10a (7e44aa1c) 0012fd6c: USER32!_except_handler3+0 (7e440457)   CRT scope  0, filter: USER32!DispatchMessageWorker+113 (7e440712)                 func:   USER32!DispatchMessageWorker+126 (7e44072a) 0012ffb0: VanishExcept!ILT+375(__except_handler3)+0 (0041117c) 0012ffe0: kernel32!_except_handler3+0 (7c839af0)   CRT scope  0, filter: kernel32!BaseProcessStart+29 (7c84377a)                 func:   kernel32!BaseProcessStart+3a (7c843790) Invalid exception stack at ffffffff

The bug is on the table

A few months ago there was an interesting story in the company where I work. One guy from a neighboring department asked me to help him. He encountered one crash after another over the past few days. Unusual in these crashes was that they occurred in different processes and under different circumstances, but within one computer on the test bench (consisted of 5 machines) and with identical symptoms. In all cases it was an access violation caused by illegal instruction pointer.

Here is an example of one of the call stacks:

0:000> knL  # ChildEBP RetAddr  00 0012d7f8 7c90e9ab ntdll!KiFastSystemCallRet 01 0012d7fc 7c8094f2 ntdll!ZwWaitForMultipleObjects+0xc 02 0012d898 7c809c86 kernel32!WaitForMultipleObjectsEx+0x12c 03 0012d8b4 6945763c kernel32!WaitForMultipleObjects+0x18 04 0012e248 694582b1 faultrep!StartDWException+0x5df 05 0012f2bc 7c863059 faultrep!ReportFault+0x533 06 0012f530 7c36e289 kernel32!UnhandledExceptionFilter+0x4cf 07 0012f54c 0041b84f msvcr71!_XcptFilter+0x15f 08 0012f558 7c363943 radar!WinMainCRTStartup+0x1d7 09 0012f580 7c9037bf msvcr71!_except_handler3+0x61 0a 0012f5a4 7c90378b ntdll!ExecuteHandler2+0x26 0b 0012f654 7c90eafa ntdll!ExecuteHandler+0x24 0c 0012f654 fc910d93 ntdll!KiUserExceptionDispatcher+0xe WARNING: Frame IP not in any known module. Following frames may be wrong. 0d 0012f950 1f4d5318 0xfc910d93 0e 0012fa10 7c363593 0x1f4d5318 0f 0012fa58 087e6827 msvcr71!free+0xc3 10 0012fa8c 087e6f41 NaviRadarRendererPlain!std::vector,std::allocator > >::_Insert_n+0x147 11 0012fafc 087e46f1 NaviRadarRendererPlain!TScanlines::calculate+0x5d1 12 0012fc2c 087e418e NaviRadarRendererPlain!TRadarToScreen_::RenderBlock+0x521 13 0012fc64 087e2378 NaviRadarRendererPlain!TRadarToScreen_::Render+0x16e 14 0012fc88 087c4cf0 NaviRadarRendererPlain!n_d3d_render::renderer_t::render+0x68 15 0012fce0 03d5c5e7 NaviRadarLayer!TRadarLayerImpl::Draw+0x360 16 0012fd1c 03d55c8e TotUser!TRadarManager_::Update+0x77 17 0012fd4c 087c821a TotUser!TCrtPanel_::InvalidateRadar+0xbe 18 0012fd54 10ac8959 NaviRadarLayer!TRadarLayerImpl::OnVideoBlockReceived+0x7a 19 0012fd7c 10ac8ab6 TkRadar20!IAdviseHostImpl::TypedForEach+0x49 1a 0012fd90 10ac8e88 TkRadar20!RadarConnector::VideoBlockReceived+0x26 1b 0012fda0 77d48709 TkRadar20!detail::WndProc+0x48 1c 0012fdcc 77d487eb user32!InternalCallWinProc+0x28 1d 0012fe34 77d489a5 user32!UserCallWinProcCheckWow+0x150 1e 0012fe94 77d4bccc user32!DispatchMessageWorker+0x306 1f 0012fea4 7c1b1645 user32!DispatchMessageA+0xf 20 0012feb4 7c1b1357 mfc71!AfxInternalPumpMessage+0x3e 21 0012fed0 0040acc0 mfc71!CWinThread::Run+0x54 22 0012ff08 7c1ae5f1 radar!CECSNTApp::Run+0x30 23 0012ff18 0041b7fd mfc71!AfxWinMain+0x68 24 0012ffc0 7c816d4f radar!WinMainCRTStartup+0x185 25 0012fff0 00000000 kernel32!BaseProcessStart+0x23