This is slightly premature, as we’re not 100% complete yet, but I felt it might help clear up some of the weirder side effects that have been occurring to explain what I’ve been bashing my keyboard (and occasionally head) about since October last year.
The super-short version is that I’ve nearly completely re-written ActiveInbox (again, after 2018’s rewrite to work with the new Gmail… I’m starting to feel a great affinity with Punxsutawney Phil the Groundhog), to make security the biggest influence on every decision we make.
We’ve started with a ‘stopped’… namely we’ve stopped your Gmail and Calendar data from ever touching our server. Now everything is either stored on your machine, or on Google’s servers.
There’s two reasons for this:
- All browser extensions have privileged access to your sensitive data (and as a Gmail extension, that data includes your emails, aka your identity). We carry a huge responsibility not to leak your data.
- Data hacks around the internet have been dramatically intensifying in recent years (including successfully against Google and LinkedIn). It’s getting much harder to keep your data safe.
Essentially, this matters because I’m keen that we all get to sleep well. You knowing your identity and data is perfectly safe, me knowing I’ve not willingly destroyed anyone’s life!
The Risks of Gmail extensions
ActiveInbox has always had ‘good’ security, not least because we made the decision early on to only store what we believed was the absolute bare minimum of your Gmail data on our servers (the most we put into our database was your subject lines for emails you stored Notes on, as a visual nicety for you to be able to know what the Note was for if you ever exported your data). So if we were compromised, there would be minimal data leaked. And no one in our office would be tempted to read your emails as a substitute for ‘People’ magazine (I am joking, we’ve always had policies around what data access is allowed: primarily that it’s only allowed if you contact us to help with a customer support issue).
But, the biggest risk, and one that most Gmail add-ons do, is to store your OAuth “Refresh tokens” on their server. They seem innocent, as they’re just a bit of text consisting of about 100 characters, and so they’re absolutely not your emails, BUT an OAuth token is what you grant to a 3rd party (like us) to say “it’s ok for them to access my Gmail”. To make this approval last a long time, “Refresh tokens” are granted, that can generate new “Access tokens” for up to 6 months.
They’re just as serious as having a new key cut for your front door.
It’s considered best practice to keep those Refresh tokens somewhere secure, like a server, because in the hands of a malicious person, they can be used to read all your emails. Or even to impersonate you by sending emails from your account. The worst-case I’ve considered is a hacker who has access to your email can issue Password Resets for your other critical services.
The problem I have with keeping them on a server is two-fold:
- Even world-class security teams in major organisations (e.g. LinkedIn) have suffered major breaches. If that happens to us, or any service you’ve granted an OAuth token too, it potentially compromises your Gmail account.
- Assuming we succeed with bullet-proof server security, our own staff – including me – still have access to the database containing those OAuth tokens, and thus we become a security risk from your perspective. Like an ancient philosophical debate, I can never fully persuade you I’m trustworthy.
Google is thinking the same way. Partially in reaction to their own Google+ being hacked, they launched Project Strobe, to lock down all their services. For Gmail, that means:
- 3rd Party vendors like us, if we store Gmail data on our servers, are required to submit to a server security assessment once a year.
- They track unusual activity on your account (and let us track it too on your behalf), to warn you of any suspicious activity.
But still, we figured we could go further…
How We Made Dramatic Security Improvements
The biggest thing we did was stop any of your Gmail or Calendar data, including those extremely sensitive OAuth Refresh tokens, from ever touching our server. In fact they now exist entirely within the protected areas of Chrome within your computer, and on Google’s own servers. This diagram might help at https://www.activeinboxhq.com/security.
Moving the OAuth tokens to your computer isn’t without risk: as a rule, personal computers are less secure than servers. But, we couldn’t overcome two problems:
- The moment that your OAuth token touches our server, even if we devote all our energy to extreme security, there’s still potential for theft. E.g. we (as developers) would still have theoretical access to your OAuth tokens. And Google’s Security Assessment is only once a year, leaving 364 days where a security weakness might occur.
- If our server holds 1000s of people’s OAuth tokens, it represents a hugely appealing target to hackers. Whereas when the OAuth token is just on your machine, a hacker would only succeed in getting 1 OAuth token per target. It’s much less attractive to even try.
We actually ended up going further to mitigate any problems with storing it on your machine, primarily against viruses/malware accessing your hard drive, by breaking the OAuth token into fragments, encrypting them, and only then storing them on your hard drive.
Finally, we took extra steps to also secure ourselves as a general browser extension (this is possibly only interesting to geeks):
- Any data that is sent to our server (e.g. your Preferences) must be validated by a schema. For example, a schema dictates that “the value named A can only contain data that is a 1 or a 0”. This stops either a theoretically rogue developer working for us, or any other form of hacker, from sneaking Gmail data into uploads to our – or any other – server.
- We have a tightest possible defaults for our Content Security Policy, which is a software contract enforced by Chrome itself, so our extension code has the scantest permission to do anything, so if we’re compromised, the attacker can barely do anything either. We also locked down which servers the extension can talk too, so it’s impossible for malicious actors to send your data to their server.
The Difficulties
As a major re-write, it’s not been without bursts of turbulence. Even though I tested each release on our multiple test accounts, and beta tested it with 300 very patient testers, there’s still enough legacy preferences and different environments that people have that caused problems – which are still ultimately my fault, and for which I’m sorry. And one whole day of downtime for everyone, which was absolutely my fault. Let’s go into that.
The August 21st Day Of Silence
I made what I thought was a minor change to make the server record certain errors to its hard drive for easier analysis: which is sane. And I did it prior to boarding a 12 hour flight: which is insane. Because I didn’t anticipate that those hard drive records would overwhelm the server, causing it to crash. And to make me almost straight-jacket-worthy, I’d made the decision as part of this whole Security Overhaul to be the only person trusted with low-level server permissions… so I’d created an operational bottleneck where no-one else could fix it.
Within 10 minutes of landing, I’d got the laptop out, rebooted the server and fixed the log issue – and been sternly harried by gate staff who wanted me to board my connecting flight… but it had still caused a whole day of frustration to you all. I’m really sorry – and in no small part embarrassed – about this. I’ve now made it possible for others on the team to restart the server (but still without low-level access to it, honouring our security goals). We’ve also extended everyone’s account by a day, by way of humble apology.
The ‘Unauthorised App’ screen when you add an email to Google Calendar
Having so majority rewritten ActiveInbox, and especially because the calendar feature was a 100% rewrite, we’ve had to re-submit ourselves for verification with Google. Most of ActiveInbox has pre-existing trust, but the calendar permissions did not.
It takes a long time to get through Google’s verification process, but we’re hopeful it’ll only be a few more weeks at most.
(Just to reiterate the key info: the rewrite was specifically to stop any of your Calendar data even transiting through our server, so it’s purely within your machine and Google’s servers – and significantly more secure than it used to be, despite the warning).
Sunsetting ‘Send Later’
As this feature is now built into Gmail itself, this was an easy decision to make. But fear not: while it’s been removed from ActiveInbox’s interface, you’re not forgotten as we are still sending the emails you had scheduled to go in the future.
We do have a neat new interface for Send Later – dubbed ‘Perfect Delivery’ – that means we may reintroduce it again later, but only if it proves to be substantially better than Gmail’s own offering.
Sunsetting ActiveInbox Mobile
This was by far the hardest decision.
When we built our Mobile app, as 2016’s big project, I took a bank loan to accelerate what we could do, sought out 2 amazing new developers and one superb designer, and ultimately released a product we were ridiculously proud of. But throughout 2018 and 2019, humbled by the onslaught of necessary rewrites to ActiveInbox as Gmail changed, we couldn’t invest in it as we needed, and it fell behind Gmail’s own mobile app to such an extent that I could no longer recommend it to you with integrity.
Prior to shutting it down, I did a survey to 1,200 of you (and many of you were kind enough to email me personally as well), to understand what the impact would be. It confirmed my hunch that while you did want a mobile app, ours now had enough jagged edges and weird spots that it wasn’t good enough to keep going.
It was going to take a lot of investment just to make it competitive with Gmail’s own app.
And to add to that cost, if we kept it going, we’d have to submit to Google’s new Security Assessment (because push notifications go through our server), which would cost $15,000-$75,000 per year, payable immediately.
This made it a hard conundrum as to how we allocate our small-team resources most effectively.
Ultimately, I decided it best to take the painful decision to shut it down, and efficiently allocate our development time by tackling things linearly, and doing so with 100% focus (rather than spreading ourselves thinly and not really perfecting anything). That means the next half a year will see:
- Finish the security overhaul of ActiveInbox itself
- Release some of the features we’ve been prototyping for a long time (possibly as a sibling product – more on that soon!)
- Polish the mobile app, and tackle the things you said were causing frustration (support multiple accounts, pre-cache emails so they load faster, fix weird bits of the interface, and fix the bugs that affected Android).
- Set aside cashflow to pay for the ongoing Security Assessments needed for the mobile app (as Gmail data would transmit through our server to provide lock-screen notifications).
Our Next Steps
It’s going to take a few more weeks to finish the security work to make ActiveInbox truly protect your Gmail data, and to let the dust settle (which is to say, lots of little bug fixings for individuals over Skype calls!).
After that, it’s onto exciting new things! We’ve been prototyping new features for both ActiveInbox, and a potential sibling product, with various groups of beta testers who have been very generous with their time, and it’s finally time to start rolling those out.
This was written by Andy Mitchell