Coinbase Wallet Extension

Coinbase Wallet Extension: Your go-to solution for smart crypto management. From market insights to trade execution, enhance your digital currency experience with ease and efficiency.

Three and a half years ago, Coinbase launched using a simple hosting platform: Heroku.

It was the right solution at the time. With just two technical founders building the product (neither with any serious dev-ops experience) we knew that Heroku would be more battle tested than any solution we could hack together on our own.

But we also knew this wouldn’t work forever. Early in our company’s history, we started to contemplate the next version of our infrastructure that would run inside AWS. It had to be built from the ground up with security in mind (the most common ways that bitcoin companies die is due to theft and hacking) but we didn’t want to compromise on engineering happiness and productivity.

After about a year, we finally completed the transition and we’ve been running inside AWS for quite some time now. This post outlines some of what we learned during the transition. It can be used as a starting point to building paranoid and productive infrastructure in the cloud.

Today, Coinbase securely stores about 10% of all bitcoin in circulation.

Disclaimer: Though we discuss some of our security measures below, our security measures are continually evolving. These are just a few measures that have existed at one point in our growth. For more on our approach to security, see this YouTube talk.

Layered Security & No Single Point Of Failure

Two of the most important principles we followed when designing our infrastructure are layered security and eliminating single points of failure. Both of these concepts encourage you to not put all your eggs in one basket. Instead, strive for redundancy and consensus amongst multiple parties. These concepts are used heavily in bank security, nuclear launches, certificate authorities, corporate governance, and even human resources.

Separate passwords and second factor tokens amongst different people.

A simple example of this in practice is securing your administrator account on AWS with a two factor token that is controlled by a second person. If you have one person who controls the password to the account, give the second factor token to another party. Store the second factor in a vault or safe deposit box off site for some physical (in addition to crypto) based security. It can prevent a single person maliciously (or accidentally) ending the company.

Lock Down Production Access

The developers on your team should not have (or need) production SSH access to do their regular work (deploying code, spinning up new services, debugging, etc).

However, it is difficult to entirely eliminate the need for SSH access. Some people in the company will always need a way to debug obscure problems. When people do need SSH access, here is how you can lock it down:

1. Add two factor to your SSH Every SSH should require a second factor. You can use Duo two factor authentication for SSH which pushes an approval request to your phone, or a FIDO U2F key which is like a small hardware security module on a USB stick. As mentioned above, if you don’t want anyone to be able to unilaterally SSH into production, you can even separate these keys and require all SSH to be “pair programmed”.

2. Use special laptops for SSH access You may want to avoid being able to SSH into production using your regular laptop. Getting malware on a laptop due to spear phishing has been responsible for many (if not most) of the high profile breaches we’ve seen in recent years. People often assume that hacks are caused by 0-day vulnerabilities or other sophisticated techniques. But in reality, simple spear phishing (clicking on spoofed links in emails) is far more likely to get you. We’ve seen attackers dedicate 6 months or more to establishing relationships, all with the intent of spear phishing. Set aside some special machines in the office that are in a locked room, that you only use for SSH access. Throw a Dropcam in the room to record who enters and leaves. Don’t use the machines in the room to open email, or browse the internet (you can use your regular laptop for this). You should probably wipe these special SSH machines on a regular basis as well.

3. Heavily Audit SSH Access Set up bastion hosts that all SSH requests must run through. Restrict who has access to these least-privilege hosts and let the team know when they’re accessed (via Slack notifications). You can also wake people up (PagerDuty) when certain commands are issued. To avoid an untraceable action after getting inside, durably log every action and keystroke which goes through the bastion host. Coinbase wrote a custom piece of software to handle this portion, and it may be something we open source in the future. The storage of the SSH logs is just as important, since they often contain sensitive info. We run a separate disaster recovery environment that guarantees storage of every action in our environment for at least 10 years. Immutable logging is important because it gives you an audit trail if a breach ever happens to find the root cause.

4. Limit SSH access to people who are less likely to steal Set up special rules around who gets production access. For each employee who gets access, run a background check on them to check for criminal records, make copies of their driver’s license and passport, and you may even want to collect a copy of their fingerprints. Make sure you have everything you need to issue an arrest warrant if something ever goes wrong. This one can be controversial, but you may want to only grant production access to people who are citizens of the country where you operate, especially if they have family ties there. Most people would be unwilling to steal $1M if it meant never being able to see their friends and family ever again. You want to create a culture where production access is taken very seriously. It should come with a great deal of responsibility and oversight.

However, it is difficult to entirely eliminate the need for SSH access. Some people in the company will always need a way to debug obscure problems. When people do need SSH access, here is how you can lock it down:

The developers on your team should not have (or need) production SSH access to do their regular work (deploying code, spinning up new services, debugging, etc).