I want to say something about how we currently do QA for releases.
The need to explain has arisen because there was a problem with 188.8.131.52, where the Notes button and some label drop downs were missing in a conversation.
It only affected a small number of users – it was in fact an edge case in Tom’s new keyboard filtering code that only occurred if you had no labels in a given category (e.g. Projects). But the problem’s skomocale isn’t really the point. Any problem is a problem.
We’ve now reached a fairly mature release process. New functionality gets developed and internally tested on 3 of our accounts. It’s then released to the alpha group for testing. And, if it’s a major change, it’s released to people who read the blog. Finally, once well worn, it’s released to the mainstream via auto update.
But we still have one Achilles’ heel: Gmail changes.
And the damage is two fold…
First, the actual Gmail change disrupts ActiveInbox. Normally, we find out about this within hours of it happening (timezone permitting), diagnose it and get it fixed.
Second, because a Gmail change is instantly worldwide, it forces us to roll out the fix with the bare minimum of time for testing. This is where problems can get overlooked.
I’m trying to think if there’s a way we can improve our process for releasing a Gmail-change fix. I’m thinking that, instead of the standard QA process, we think up unique tests specific to whatever small thing we’ve just ‘fixed’. A more creative process. I’ll start doing this from the next time something goes wrong.
This post was written by andymitchell9496