ProtonBlog(new window)

Missing Emails Restored

Share this page

As we reported earlier on our blog, we had an incident(new window) which caused some emails from over a 20 hour period to disappear.

Immediately afterwards, we initiated data recovery steps and within a day, we were able to recover the data and begin restoring emails into users accounts. On Saturday, we finally finished restoring emails to the last impacted accounts.

Our goal is to maintain 100% data availability and we apologize to those users who weren’t able to access some emails for a couple days while we worked on the recovery. Needless to say, a number of steps have been taken to avoid a repeat of this problem and we have strengthened our standard operating procedures (SOP) to include even more safeguards.

Technical Details

The root cause was found to be a Linux service called monit which automatically restarts services when it detects them to be crashed or is otherwise not running for some reason.

In our SOP, the first step for most procedures is to shut down monit. However, when one of our new engineers went to perform some changes on Monday, this was not done. The database changes we were doing on Monday required the database server to be shut down for a period of time, and the commands to do this were indeed issued. However, since monit was still running, the database server was automatically turned back on unbeknownst to the engineer. As a result, changes were made on a running database leading to data corruption.

While it is easy to lay blame on an individual engineer for not following the SOP, there are also organizational deficiencies that allowed this lapse to occur. The team as a whole is under immense time pressure to work quickly and support more users, so shortcuts were tolerated. This was generally OK because the core developers understood the system very well and knew with certainty which steps could be skipped without risk. However, we also inadvertently created an environment for new employees where the SOP was treated a guideline and not rules that had to be followed to the letter.

To remedy this situation, we have now enacted new regulations where changes on the production systems can only be made with the approval of ALL core developers. Furthermore, SOP shortcuts will no longer be tolerated, regardless of who is making the change.

These changes will inevitably slightly slow down our development and scaling process, but as a group, our core priorities are security and reliability and these must come before all other considerations. We would like to thank everybody (especially those still on the waiting list) for their understanding.

As a side note, we are also actively looking to grow our team so we can develop Proton Mail faster, if you or somebody you know is interested, please check out our current job openings(new window).

Secure your emails, protect your privacy
Get Proton Mail free

Share this page

Andy Yen(new window)

Andy is the founder and CEO of Proton. He is a long-time advocate for privacy rights and has spoken at TED, Web Summit, and the United Nations about online privacy issues. Previously, Andy was a research scientist at CERN and has a PhD in particle physics from Harvard University.

Related articles

Can you password-protect a folder in Google Drive?
Protecting a folder with a password is a simple yet effective way of securing files. You may wonder whether you can password-protect a folder in Google Drive. We explain what access controls Google Drive offers and what you can do to improve your sec
Proton Pass now supports passkeys on all devices and plans
We’re excited to announce that Proton Pass supports passkeys for everyone, allowing you to manage and use passkeys across all devices seamlessly. Passkeys are an easy and secure alternative to traditional passwords that can help prevent phishing atta
what is a passkey?
Passkeys are a new way to secure your online accounts using cryptographic keys instead of passwords. They offer a high level of convenience and security, and are a real game-changer in the way we access and secure sites. What is a passkey, though, an
Apple’s marketing team has built a powerful association between the iPhone and privacy. The company’s ad campaigns claim that “what happens on your iPhone, stays on your iPhone.” And, “Privacy. That’s iPhone.” But Apple’s lawyers are telling a diffe
A cyberattack on national public employment service France Travail has exposed the personal data of as many as 43 million people.  The latest breach is the second major cybersecurity attack to happen in France in the past month, raising concerns abo
If I share a folder in Google Drive, can anybody see my other folders
Google Drive makes it easy to share files and folders, but you may have wondered at some point whether the people you’ve shared a folder with can see your other folders. We answer this question below and also share some tips for truly secure link sha
In 2014, Proton Mail was introduced as a web app, revolutionizing how we think about email privacy. Today, we’re excited to broaden the horizons of secure communication by launching the Proton Mail desktop app. Anyone can now use the new Proton Mail