Hi everyone,

This is the final part in my blog series on Hopper testing. The blog is targeted at QA engineers familiar with the Microsoft Hopper test.

As we have discussed, for certification, the Hopper test needs to be able to randomly click on features at a rapid pace for 25 hours without causing the device to fail. When there is a failure, the limited log files and random nature of the test can make it difficult to pinpoint the cause. It’s like trying to find the one final snowflake that triggered an avalanche.

Here are a few tips for spotting patterns in seemingly random events.

Instead of trying to create a failed Hopper session, it’s better to run many failed Hopper sessions and determine what they have in common. To start looking for the common thread, you need a good set of data. I recommend having at least 200 – and maybe as many as 1,000 – data points to work from.

I create a spreadsheet to track the programs and DLLs running at the time of each failure. Keep in mind that if you have 200 failed Hopper sessions, they didn’t necessarily fail because of the same bug. You may have two bugs, or two dozen, or 200. Sort your spreadsheet so that like conditions are next to each other and try to group the failures into as few buckets as possible.

Once you have identified a potential problem application, you can use the FocusApp application (http://blogs.msdn.com/hopperx/archive/2005/11/30/the-cat-parade.aspx) to force Hopper to test it. FocusApp regularly checks to see if that particular application is on top of the stack and changes the focus to it if it is not. By forcing Hopper to focus on your suspected problem application, you raise the likelihood it will find a flaw.

It may also help to measure the probability of a failure for a given set of conditions. Use the data in our earlier spreadsheet to generate a probability map for a Hopper failure. For example, you may find a failure rate of 65 percent when applications A and B are running, but of only 10 percent when applications A and C are running. This information can be very helpful in pinpointing the actual problem.

Finally, don’t save the Hopper test for the end of your development cycle. I start recording Hopper scores as soon as we have the software platform running. Your Hopper scores will naturally drop as you add components, including third-party drivers and applications software, and will rise as you fix bugs. Tracking those changes over time, however, will make it easier to identify and fix potential problems – and help you pass the crucial certification test on the first try.

. . . . . . . . . . . . . . . . . . . . . . .
Mauricio Soto
Senior Software Engineer
Professional Engineering Services
BSQUARE Corporation | Contact Me!