Infrastructure Upgrades

ProtonMail's server infrastructure, 100% owned and operated by ProtonMail.

Many of you might have noticed that ProtonMail had a brief scheduled downtime last week. That was actually the first step of a major infrastructure upgrade that we have just completed.  Thanks to the support from our crowdfunding contributors and around-the-clock work of our team, ProtonMail today is more secure and reliable than it has ever been, even with the huge number of additional users we have recently invited from the waiting list.

For those users who have been on our waiting list for several months, the wait will soon be over as our new infrastructure will allow us to support almost everybody. We will be inviting nearly everybody over the next month!

The reason it has taken us so long to get to this point is because building an email architecture that is secure, scalable, and also reliable is no easy task. In this post, we will be describing some of the work the ProtonMail team has been doing in the past couple months to keep your data safe.

Hardware and Network

ProtonMail’s infrastructure scaling is complicated by the fact that we run our own servers which means we also need to build in redundancy on the hardware and network level which greatly increases the required effort. Fortunately, our team has worked on building and managing large scale systems at CERN and are able to draw from that experience.

Because ProtonMail’s encryption is zero access and we do not have the ability to read our user’s encrypted data, in some ways, it does not matter where we store encrypted data. However, as we have seen in the past, third parties simply cannot be trusted to safeguard online privacy and freedom. The ONLY way to ensure the highest level of data security and uptime is to have full control over the server hardware and network. This is why despite the added difficulty and complexity, we go a step beyond and only use hardware that we physically own and control within Switzerland to host ProtonMail.

All of our servers feature fully encrypted disks and we use RAID arrays with high redundancy for our storage. The redundancy even extends to the way we power our servers. Within each datacenter, only half of our servers are connected to a single power unit so a failure of an upstream power unit cannot take all servers offline.

Distribution of ProtonMail datacenters in Switzerland.
Distribution of ProtonMail datacenters in Switzerland.

Datacenter Redundancy

While we have excellent redundancy within our main datacenter, to ensure even higher reliability, ProtonMail began to build out in a second datacenter this summer. Today, ProtonMail’s hardware infrastructure is spread out across two datacenters in Switzerland to ensure that a catastrophic disaster at one datacenter will not lead to data loss. In a follow up post, we will talk more about ProtonMail’s datacenters.

Infrastructure Architecture

The diagram below gives a high level overview of ProtonMail’s latest architecture after last week’s upgrade. The overarching design philosophy is to eliminate as many single points of failure as possible in order to make ProtonMail the most reliable encrypted email service ever built.

ProtonMail's server infrastructure, 100% owned and operated by ProtonMail.
ProtonMail’s server infrastructure, with all servers owned and controlled by ProtonMail, running 100% open source software.

Load Balancing

As ProtonMail’s userbase grew, we rapidly exceeded the capacity of a single server which made it necessary to load balance across multiple servers. Our load balancing system splits the load among multiple web and mail servers and also provide instantant failover in the event of a web or mail server crash.

Web Servers

All ProtonMail servers (web servers included) exclusively run open source software and are Linux based. Our architecture allows additional web servers to be added without downtime. Furthermore, any individual web servers can be taken offline without impacting users. This gives full redundancy in the event of a web server failure, and also allows us to take machines offline at any time to perform security updates.

Mail Servers

ProtonMail’s mail infrastructure is also fully redundant and any mail server can fail without impacting inbound or outbound mail deliverability. Our mail software architecture also allows us to buffer mail on the mail servers. This means in the event of a database failure, mail servers can save incoming messages until the database servers come back online so a database failure will not lead to the loss of incoming messages.

Database Layer

We use a cluster of database servers to store encrypted user messages. We have multiple SQL servers with automatic failover which allows us to lose SQL servers without system downtime. The data servers are clusterized so that individual data servers can be lost without leading to data loss or downtime.

As an additional layer of security, we have a backup data cluster which replicates from the master cluster in real time so in the event of a catastrophic failure of the primary cluster, we can switch to the backup with minimal data loss.

DNS

For added security against DNS attacks and better control over our domain, ProtonMail also runs our own DNS infrastructure which is distributed between our two datacenters for redundancy. Our DNS root zone is managed by SWITCH which administers .ch domain names on behalf of the Swiss Federal Office of Communications (OFCOM).

Monitoring

ProtonMail utilizes a sophisticated monitoring system that is also distributed between two datacenters in order to monitor the health of our hardware and also detect for potential network intrusions or abnormalities.

Looking Forward

When ProtonMail was first opened to the public back in May, our architecture at that time was run on just two servers (a primary and a backup) and was rapidly overloaded by users from around the world. Our current architecture is a huge advancement from that and would not have been possible without many months of hard work from our team and the support of our crowdfunding contributors.

There is still much infrastructure work to be done and we will continue to add improvements on two main fronts. First, we will keep pushing to eliminate single points of failure to reduce the risk of downtime. Secondly, we will work on bringing more components of the internet infrastructure needed to run ProtonMail under our direct control to improve privacy and reliability. We recently took a step in this direction by joining Réseaux IP Européens NCC and becoming a Local Internet Registry which serves ProtonMail exclusively. As you can see, we are far from done and 2015 will certainly be a busy year!

About the Author

Andy Yen

Andy is the Co-Founder of ProtonMail. He is a long time advocate of privacy rights and has spoken around the world about online privacy issues. Previously, Andy was a research scientist at CERN and has a PhD in Particle Physics from Harvard University. You can watch his TED talk online to learn more about ProtonMail's mission.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

26 comments on “Infrastructure Upgrades

  • I’m using ProtonMail in about 1 month and so far so good, it’s being a awesome experience and I’m not using as main mail service because is in beta.
    I’m glad to say, PortonMail team is making a great work, the support is simple awesome and I’m excited to see new versions with new features.

    Keep the excellent work.

    Reply
  • Andy,

    I signed up when you were just in beta and have been waiting for this day ever since. It doesn’t make sense for just one person in a community of users to be signed up on Proton. Now I will be able to send out notifications to everyone in my address book that they too can get fully secure email. Will you have a webpage that I can point users to that will point out how and why you do what you do?

    Yours,
    Paul Thomann

    Reply
    • Hi Paul, we are working on a new version of our website which will have a page dedicated to that. Coming soon also is an invite system that will allow existing users to invite people so they can get accounts without having to wait on the waiting list. We hope to roll out most of these changes in January so stay tuned. Thanks for your support!

      Reply
      • Mobile app will be awesome! That is probably the only thing keeping me from completely making the switch.

        Love the two step password login
        Ability to encrypt to an outside email
        And for them to be able to encrypt a response
        Awesome features… Thanks so much!!!

        Reply
  • Thanks for sharing info. Will be nice to send out all invites over next month. Looking forward to it.
    All the best for 2015.

    Reply
  • Really glad to be finally on it, and cannot wait to extend invite to my contacts. A quick question: is it enabled for email clients? (useful if there are messages you want to store permanently, for instance)
    Thanks!

    Reply
    • We don’t support 3rd party clients yet, but we’re working on our own mobile app and will try to build client support in the future.

      Reply
  • Hello Andy,

    I was wondering if there is a way to “recover” the username submitted when I first requested an invite!!

    Thank you guys for all that you’re doing. Can’t wait for the invite next month.

    Best Regards,

    Wafa

    Reply
  • Im really glad things went as well as they did for you! Had my main ProtonMail account since August, and Im loving it! Never going back to creepy mail providers again, ever.

    Reply
  • I am very happy to be among the early adopters. Actually I had forgotten that I registered my interest until receiving your invitation to create an account today here in Tasmania.

    In regards to secure password management may I suggest Clef and Waltz – I have no affiliation with them however I have just started using their platform and it is far superior to the traditional password managers that you have mentioned. They can be found here https://getclef.com/

    Thank you once again for all of your teams dedicated hard work that has bought encrypted email to the world
    Ian

    Reply
  • I appreciate your intention very much to protect privacy.
    It is my experience that spam not only knocks but enter comfortably into your inbox as soon as you open a new e mail account with some service provider.
    A suggestion for your consideration
    Please start writing protonmail.ch as “ProtonMail.ch” i.e. p and c in capital.
    This will give very easy identification of a new word and it will also emphasize that it is a “Mail” service.

    Reply
  • Please prepare and publish a guide in pdf format so that it can be downloaded and read on other machines later on.
    It will create the awareness about privacy and popularize ProtonMail.ch amongst those who have not yet using it.
    The guide can be updated from time to time to enable the user to use the service efficiently.

    Reply
  • I am using protonmail for last two month.Awesome experience. I hope soon I use protonmail account as primary mail account. Best of Luck.

    Deep Jyoti

    Reply