The way to find a hotspot

Today I want to describe the way to find a hotspot without profilers. I often use this approach and find it effective and simple enough. The only tools you need are any Windows debugger (I prefer WinDbg, but you can use your favorite one) which should always be in your hands if you decided to investigate some performance issue (as well as any other problem with your software) and tool like Process Explorer which available free from Windows Sysinternals. You can use this method without profiler at all or in conjunction with profiler to clarify the result of profilers work.

Before explaining the approach, I want to define the term “hotspot”. There are different definitions which can be applied in different contexts. In this article we consider a hotspot as a limited set of instructions on which execution spends significant CPU time or more CPU time than you expect from it.

The process is divided into the three steps. If you used a profiler and have a call graph, you can miss the first two steps and jump to step number three to verify the profilers result and find out the hotspot most precisely. Nevertheless, I recommend you to go through all steps because the results returned by profilers sometimes can be very obscure and you will lose time rolling around a call graph or other related information. Even if your profiler was absolutely right, you’ll have one more confirmation of its result. Sometimes it’s better to check twice :)

The bug in MFC tool tip control

Yesterday, I encountered the bug in MFC’s tool tip control implementation.
The bug dozes in the following code:

LRESULT CToolTipCtrl::OnAddTool(WPARAM wParam, LPARAM lParam) {     TOOLINFO ti = *(LPTOOLINFO)lParam;     <----- HERE     if ((ti.hinst == NULL) && (ti.lpszText != LPSTR_TEXTCALLBACK)         && (ti.lpszText != NULL))     {         void* pv;         if (!m_mapString.Lookup(ti.lpszText, pv))             m_mapString.SetAt(ti.lpszText, NULL);         // set lpszText to point to the permanent memory associated         // with the CString         VERIFY(m_mapString.LookupKey(ti.lpszText, ti.lpszText));     }     return DefWindowProc(TTM_ADDTOOL, wParam, (LPARAM)&ti); }

To untangle a rope

Last weekend I was launching a kite with my daughter and her friend. It was great fun, but when I was veering the kite down its string entangled. I spent about 30 minutes (maybe even more – time flies fast :)) to untangle the string. I followed the string loops, analyzed the types of knots, imagined what will happen to string if I pass it through one loop or another. I went through a lot of different combinations inside my mind and devised more and more new techniques. When I finished finally, I felt satisfaction. I felt no tiredness or irritation, furthermore I had the feeling like I untangle the strings all my life. Then I thought about it I realized that it’s not far from the truth. The process of untangling string (as well as rope, chain and so on) is very similar with the process of software defects researching (the term was first introduced by Dmitry Vostokov). So if you in doubt to be software researcher or not – try to untangle a rope first :)

Can lightning strike the same place twice? (Or can a bomb to hit an elephant?)

In the old soviet movie about World War II was mentioned the story about the old Professor of Mathematics. When Leningrad has been bombing he is never descended to bombshelter along with other people. He told that he calculated the probability of bomb hitting his house and the probability was too small to hiding from bombs. But one day his neighbors saw him in a bombshelter and asked why is he hiding with others. He replied that on that day a bomb hit the only elephant in the cities zoo, and probability of hitting the elephant was absolutely equal with probability of been hit himself.

Why did I remember this story? Because during the last week I encountered twice with the failures which was the result of circumstances those probability was very low. Like people say: “If gun is hanging on the wall it will shoot one day or another”.

Why it's not crashing? (conclusion)

Well, today I received the final message from MS where they told me once again that there is nothing can be done on their side and advised me to catch exception in user code and call MiniDumpWriteDump after that. I think the story can be considered finished.

Se la vie, mes amis. Se la vie.

In search of the Dr. Watson

A week ago I returned from the business trip to Italy. It was a trip to the yacht where was the real problems with the navigation system stability. One or two times a day all navigation data from sensors freeze and the navigation equipment was inoperative until full bridge (5 stations) rebooting.

Just before the trip I received some information concerning the problem from our service engineer.
There were several observations:

1.    All navigation data freeze simultaneously but the navigation system (radar, cartographic system and over subsystems) continued to work without any other noticeable problems.

2.    Sometimes the message box with “Pure virtual function call” notification was observed after data freezing.

3.    First time the problem was observed just after the last update of the product.

4.    In the end I even received HD images from all stations of the bridge (they was created after the last update and after the problem appeared for the first time).


Description of the problem was clearly pointed to the fact that the cause of the problem is crashing (or freezing) of the navigation server processes at all stations simultaneously (each computer has its own navigation server executed and described problem is only possible if there is no servers alive). The only thing that confused me was mentioned earlier “Pure virtual function call” error. This error is not very common and occurs rare enough in its pure form. Most common cause of the error is improper sequence of objects chain deinitialization and destruction as well as undigested strategy of objects ownership (of course I’m not take into account errors with calling pure virtual method of child class inside a parent class destructor or deleting an object twice – I believe that our code better :). One of the classic situations caused by assertion message box triggered during object destruction (usually in debug mode). Message box message loop can provoke a call (through a pure virtual interface) to a partially destroyed object. As a result we will have clear pure virtual function call.

Why it's not crashing?

Several months ago, my colleagues encountered a very strange behavior of one of our processes after an exception was thrown. They told me that they know exactly that the process throws an exception which is not expected to be handled by their code, that is an exception must be unhandled and the process must be terminated immediately. But nothing like that happens. The process just continues its execution. At first, I thought that there must be some code down the stack which registers an exception handler that handles any exception indiscriminately. Therefore, the first thing I did was search for the try{}catch(…){} pattern down the stack. When I found nothing, I looked at the exception handlers list at the point where an exception was thrown.

It looked as follows:

0012fd14: USER32!_except_handler3+0 (7e440457)   CRT scope  0, func:   USER32!UserCallWinProc+10a (7e44aa1c) 0012fd6c: USER32!_except_handler3+0 (7e440457)   CRT scope  0, filter: USER32!DispatchMessageWorker+113 (7e440712)                 func:   USER32!DispatchMessageWorker+126 (7e44072a) 0012ffb0: VanishExcept!ILT+375(__except_handler3)+0 (0041117c) 0012ffe0: kernel32!_except_handler3+0 (7c839af0)   CRT scope  0, filter: kernel32!BaseProcessStart+29 (7c84377a)                 func:   kernel32!BaseProcessStart+3a (7c843790) Invalid exception stack at ffffffff

There were no user exception handlers at all, but instead was an exception handler I have not seen before:

0012fd14: USER32!_except_handler3+0 (7e440457)   CRT scope  0, func:   USER32!UserCallWinProc+10a (7e44aa1c) 0012fd6c: USER32!_except_handler3+0 (7e440457)   CRT scope  0, filter: USER32!DispatchMessageWorker+113 (7e440712)                 func:   USER32!DispatchMessageWorker+126 (7e44072a) 0012ffb0: VanishExcept!ILT+375(__except_handler3)+0 (0041117c) 0012ffe0: kernel32!_except_handler3+0 (7c839af0)   CRT scope  0, filter: kernel32!BaseProcessStart+29 (7c84377a)                 func:   kernel32!BaseProcessStart+3a (7c843790) Invalid exception stack at ffffffff