Why QA is Important

This is a diagram of most of the code bases within Copilot. Of the sixteen nodes displayed, we actively maintain fifteen. Of those, four integrate with other Fog Creek products, like FogBugz. In our most recent product release, Copilot OneClick, we heavily modified nine components and added two. It was the most complex release in Copilot history, yet it was the smoothest.

Often when we've released new software, it happens in starts and fits. We would push something out only to realize that we'd forgotten a crucial config setting or had left in an extra bit of logging. Sometimes two new components would conflict with each other. To fix it, we would roll back to the previous working version, make our fixes, test them out on the staging server, and roll it back out.

It sounds like we were being irresponsible, like we were pushing new code with many substantial bugs. The reality was that it was always something small and easy to overlook, which got amplified into a much larger problem by the complexity of the many interacting systems. Forget to put a default on a column in a new table and five different components could simultaneously crash.

The problem was that Ben and I were too close too the code. With only two of us working on the project, there was never any way for us to test code without having worked on it, and that made us bad testers. We knew too much about how the application was supposed to work, which made it easy to overlook small but important details. But a couple of those small mistakes would quickly turn into an outright crisis.

We had an outside QA analyst at the time who helped test Copilot's user experience. Her input spurred changes that made Copilot much more intuitive and sleek. But she was usually testing individual components on internal servers, not the end-to-end system on production servers, except for the largest releases.

Then in March our QA superstar, Alison, came on to the Copilot project. For the first time in three years we had a full-time, dedicated tester with enough time to do both targeted component testing as well as end-to-end system tests.

At first I felt a little internal resistance to her feedback. I knew I wasn't writing perfect code, but I didn't really like someone else pointing it out to me. It took a little while to understand that she was not trying to prove me wrong, that our goals were actually identical. Soon, I began enjoying getting assigned cases from her. They were detailed reproductions of the bugs, making them easy – and fun – to fix.

Soon we got into a nice pattern: I would typically work a couple hours later and kick off a build before I went home. Then, in the morning before I got in, she would start testing the new build. By the time I got in, I would have a list of things to fix, which helped me focus my time, making me a more efficient coder. As we got closer to releasing, we would push builds to the staging servers, and in a matter of hours before we knew, for sure, whether the build was ready to go to production.

So did she find all of the bugs before we pushed to production? Not quite. We released OneClick with one significant bug: the installer could not be downloaded from Internet Explorer because IIS injects extra HTTP headers into HTTPS connections, which IE then has trouble understanding. She caught it on the production servers a little after we rolled out. The funny thing is, it's a bug we've fixed before, just one line fix in ASP.Net:

Response.ClearHeaders();

So why did we miss it before going live? Because our staging server doesn't use HTTPS; it's not close enough to the production server. It wasn't possible for her to find that given the testing environment. But that was it. Really. Nine components significantly modified, two new, and only one line needed to be changed in all of it. We didn't even need to cut a new build, we just pushed out one patched file and all was fixed.

Instead of the panic and worry that we'd experienced before, now pushing new code is a calm, steady process. We can be confident in the quality of the code. They key was to get another pair of eyes on the product, someone who could ask the questions that we didn't think of while coding.

Cross-posted at http://blog.copilot.com/

Posted November 24, 2008