Is there test data available for this project?

Feb 13, 2009 at 11:15 AM

I was glad to learn of this project. The company I work for is deeply committed to Microsoft technology in developing a vast suite of tax and accounting applications. We are currently developing testing automation for this ambitious project.

The number of "generic actions" that we can call from the automation framework is limited. Certain desirable functionality has been deemed out of scope for the foreseeable future, especially those functionalities that are commonly deemed problematic. That is why the Visual Verification API is particularly interesting to me.

Maybe I have missed it but I haven't found any statistical analysis of any of these libraries that can demonstrate their efficacy. This would be particularly helpful in deciding to promote this API for inclusion in our test suite. If there is some data such as this, could you direct me? If not, wouldn't it be a good thing to include in this project site?

Thanks,

David Smith

Coordinator
Feb 16, 2009 at 11:25 PM
Hi David,

This is a very interesting comment. Could you elaborate on what you mean by statistical analysis? If you mean things such as ANOVA or regression analysis -- we have not really done anything like that. Visual verification is a tricky area, where one has to consider various factors (such as graphics cards, drivers, os-es, etc.) to come up with a robust VV strategy, and we can definitely share empirical experience and best practices to help you find the optimum compromise between test stability and test sensitivity, if that's what you are asking for.

If not, could you give us a rough idea of your scenarios and what you are looking for and we will try to help.

Thanks,
Ivo
Feb 17, 2009 at 11:49 AM
I'm sorry I wasn't more specific. I think you have the right idea though. I am looking for metrics generated by using this API (particularly the VisualVerification API) in testing actual software projects. I am interested in numbers like bugs found, bugs missed, and false bugs.

I am trying to come up with a good enducement to get my company to even try this API in our automated testing.

Thanks,

David Smith
Feb 18, 2009 at 1:53 AM
Hi Ivo,

I thought of a better way to clarify my need.

The decision on automated visual verification has already been made by those higher in the company hierarchy than me. I believe that automated visual verification can be very helpful to our bottom line and to quality if it can be properly implemented. In order to change minds on this, I need empirical evidence from a reputable source.

Please, take it as a compliment.

Thanks,

David Smith
Coordinator
Mar 27, 2009 at 11:33 PM
Hi David,

In our team at Microsoft (we are the Windows Presentation Foundation team) we have about 87,000 tests, covering the various components of WPF. About 38,000 of these are graphics-related tests (2d, 3d, Animations, Effects, Imaging, and Media). The vast majority of these tests do not use master image validation, because the maintenance of master images (across various different OS-es, GPUs, DPI settings, theme settings, etc, etc) is expensive. We prefer to either do analytical visual verification (where we can predict the outcome of a drawing operation and then in the test confirm that the reality matches our predictions -- we use this technique for validation of drawing primitives) or do master-based visual verification of complex integrated scenarios (e.g. we take a snapshot of a window containing a large number of different widgets and use that single snapshot for master based verification).

As I mentioned in a previous post, visual verification is a technique that should be used carefully, because the test maintenance costs tend to explode very quickly. I always advise folks to look at alternatives before deploying visual verification tests.

Having said all that, in TestApi v0.2 we will soon be shipping updates to the visual verification APIs (ability to use masks, etc.) that will make it slightly easier to deploy robust visual verification tests with lower maintenance costs.

Ivo