At Zecter, we have a couple of products (ZumoDrive, and ZumoCast) that have installation wizards.  These wizards are very important to us because this is how most customers first interact with us, so we spend a lot of time tweaking them.  We've implemented the wizard in HTML, and we render it in a web view built into the native client software.  

The Curse

The wizard runs on Windows, Mac, and Linux.  It is internationalized in seven languages.  There are 5 steps in the wizard.  This means that there are up to 21 different possible views of each of the 5 steps.

21 x 5 = 105 possible views

So there are 105 possible views that can be affected by CSS, HTML, or string changes to the layout.  This is not fun to test manually, and we haven't done a great job of making sure our wizard looks great in all supported locales.

Integration Tests Are Not Enough

Our server runs on Rails, which has support for integration testing.  This is great for testing whether or not sign up and login work through the wizard.  It's not great, however, for testing whether or not the wizard looks right in the native client on each platform in each language.  Sure, we can verify that various bits of information and scripts are displayed, but we can't see how they're displayed and if they're aligned properly.  So to do regression testing you have to run through each screen of the wizard on each platform in each language.  If you find a bug, you have to do it again.  And again.  And so on...  

The Solution

The Wizard's Curse from Adam Thorsen on Vimeo.

The solution we've come up with is to write a test helper that takes a snapshot in png format of the wizard screens as the integration tests are executing.  In order to do this we use wkhtmltopdf.  We can set up the test environment to render for each platform and be able to browse through the snapshots using the cover flow view in Finder ( we do web dev on macs.)  This way we don't have to boot up the Windows VM and configure it to connect to the development webserver until I can see that all the screens are rendering reasonably well and are platform appropriate.  This drastically reduces the amount of manual testing required.

Other Solutions?

I'd be interested in hearing other people's solutions to these kinds of problems.  Please comment.