Hi all,
I want to say something about how we currently do QA for releases.
The need to explain has arisen because there was a problem with 4.0.4.38, where the Notes button and some label drop downs were missing in a conversation.
It only affected a small number of users – it was in fact an edge case in Tom’s new keyboard filtering code that only occurred if you had no labels in a given category (e.g. Projects). But the problem’s skomocale isn’t really the point. Any problem is a problem.
We’ve now reached a fairly mature release process. New functionality gets developed and internally tested on 3 of our accounts. It’s then released to the alpha group for testing. And, if it’s a major change, it’s released to people who read the blog. Finally, once well worn, it’s released to the mainstream via auto update.
But we still have one Achilles’ heel: Gmail changes.
And the damage is two fold…
First, the actual Gmail change disrupts ActiveInbox. Normally, we find out about this within hours of it happening (timezone permitting), diagnose it and get it fixed.
Second, because a Gmail change is instantly worldwide, it forces us to roll out the fix with the bare minimum of time for testing. This is where problems can get overlooked.
I’m trying to think if there’s a way we can improve our process for releasing a Gmail-change fix. I’m thinking that, instead of the standard QA process, we think up unique tests specific to whatever small thing we’ve just ‘fixed’. A more creative process. I’ll start doing this from the next time something goes wrong.
This was written by Andy Mitchell
8 Comments
Please don’t beat yourself up. If you have no notice from Gmail, there’s not much you can do. When I see a change in Gmail, I kind of expect any plug in to be broken. What’s different about Active InBox is how quickly you get out a fix and it just starts working again, if that fix isn’t quite right for all cases then you care enough to get another out in very quick time. I’m impressed and I thank you for your dedication and professionalism. On the (slightly perverse) plus side, when AIB is broken, I really, really miss it!
+1, definitely!!
Is your team currently using any automated browser testing tools like Selenium? If no, I would highly encourage looking into it. It may be the magic bullet that lowers your time commitment to identify issues and allow you more time for addressing the bug once it is identified. If you are already using an automated testing tool, disregard.
I second DaveK’s feedback, your team is doing cutting edge amazing work and such, you are going to have a steeper challenge, but that is an acceptable risk given the benefit of your product.
BTW – I have been pushing within our company to get adoption of your product. So far we have several people utilizing AIB and all of the response has been amazing. Keep up the good work!
I think given the volatility of gmail the users of your product SHOULD anticipate that there’s going to be issues any time there’s a major gmail update. In cases where gmail has broken things, I have just turned ActiveInbox off for a few days and tried again. In most cases, the glitches are relatively minor and we can live with them for a couple of days until a patch is applied.
Using a tool like Selenium may help; however, given that the underlying gmail app changes you’ll get a lot false positives on automated testing.
You’ve got a huge liability that your product is built on top of someone else’s product, and that particular developer doesn’t do predictable or even consistent updates across all of its user base. As long as you’re proactive on things and keep us up to date then I’ll continue to pay for my premium membership.
Have you thought about developing your own webmail? Perhaps this solution is overkill but it’s an idea…
I suggest that you put a bar at the top of the page indicating to users that gmail changes have temporarily broken the product, and then another when the fix rolls out indicating that a patch has been applied, with a link on it to click and file a report if the patch is not working for them (maybe with the option of rolling back). I really don’t like it when I get the “we’re turning this off” window I saw once came up, because it prevented me from getting at a rather key piece of info I needed behind it, and I had to be creative with Firebugs and CSS to make it “go away” so I could get my job done (I wasn’t having any actual problem with activeinbox oddly enough).
A ribbon/bar at the top showing the status as broken by gmail, or quick patched for a gmail fix with the option of clicking to file a report or roll back the fix. NOT the giant turn off activeinbox thing I have seen once before (my inbox was working just fine and I was depending on it late at night to get the job done – if you are curious I used firebugs to snuff your dialoge and get back to work.
Oh, sorry about the double entry. I didn’t see that my logging in using openId made my comment save as anonymous, and future comments be properly attributed to me. That’s a bug (not yours though is it?).
You could mitigate some of the gmail interface changes issues with a test account on a google apps domain in the ‘Rapid Release’ track.