Learning from the Dropbox data breach

TL;DR

• In 2022, Dropbox experienced a significant data breach due to a successful phishing attack.
• The breach highlighted the vulnerability of API keys and the risks associated with committing them to repositories.
• MFA (Multi-Factor Authentication) does not provide complete protection against phishing attacks.
• Strategies to mitigate such attacks include mandating U2F keys, avoiding the commitment of API keys to repositories, and implementing limits and restrictions on API key usage.
• Companies should invest in advanced authentication methods and continuously work to limit damage in case of a breach.

In 2022, a Dropbox employee was successfully phished by an email posing to be from CircleCI. The damage was significant—attackers gained access to 130 GitHub repositories containing hard-coded secret API keys that gated customer PII. There has been no evidence of any further damage, but it’s entirely possible that Dropbox accounts, Dropbox user data, and payment information were sold to bad actors.

The 2022 Dropbox exploit is interesting because nontrivial authentication measures were already implemented at the company. The employee, for instance, used a hardware key to authenticate their credentials—a technique incorrectly advertised as immune from phishing. At the same time, Dropbox’s security protocols for limiting a successful phishing attack were severely lacking, given the stolen API keys.

There are some lessons to be learned as we break down the trends, the exploits, and the vulnerabilities that made the 2022 Dropbox attack not only an improbable event but a likely one. Additionally, several low-cost techniques could’ve been used to safeguard against this type of cyber threat, as well as additional measures that would’ve limited the damage of the breach.

Learning from the Dropbox data breach: Understanding the contributing factors

Several contributing factors made this Dropbox attack a semi-predictable event. Of course, this is partially speculation; there is no way to know the attacker’s original intent should they have been successful (phishing attacks could be used for various purposes, including ransomware or malware installations, blackmail, etc.). The point, however, is that Dropbox’s attack isn’t an anomaly, and it’s likely that other companies will face similar data breaches without the proper precautions.

The Dropbox data breach: How APIs became fancy backdoors for hackers

If you ask a CTO about their top-of-mind security priorities, it’ll likely have to do with safeguarding customer data. And, to the credit of many CTOs, companies often use IAM or even just-in-time (JIT) access to protect client data in cloud databases (e.g., RDS).

Unfortunately, that same attitude rarely carries over to third-party APIs. APIs are essentially backdoors into customer data hosted on third-party platforms (like Stripe or Datadog). That data might not be created or maintained by the organization’s core apps, but it still contains the PII of customers. Given that APIs are not gated with the same authentication protocols as user accounts—and data harvesting can be easily automated—they are a significant attack vector.

The best way to highlight this disparity is to compare logging into the same SaaS app with a traditional user account versus with the API. When a Dropbox finance employee logs into Stripe, they are asked for additional verification steps. We all know of text SMS, authenticator apps, hardware authentication keys, etc. However, if a Dropbox developer needs to access their Stripe prospect data programmatically, they typically use a lengthy Stripe secret API key.

Accordingly, API keys are gold to hackers because they’re a super-password to a backdoor. The added advantage sweetens the pie for phishing since employees might have open access to API keys. Of course, there are underutilized techniques to protect API keys and limit the damage of API key leaks, and we’ll discuss those in the solutions section.

The risks of SMTP-linked API key leaks and their impact on phishing scams

Not only are phishing attacks becoming more potent, but they’re also growing increasingly easy to kick off. Recently, CloudSEK disclosed that over 54M recipients are vulnerable due to leaked API keys to SaaS services with SMTP access. These are keys provisioned for marketing software such as Mailchimp or SendGrid, tools used by many dev-facing organizations. While it’s likely that most of these breached keys have been recycled by now, threat actors could exploit any remaining live ones to very convincingly phish users.

Many employees are under the impression that a phishing email has to come from an untrustworthy domain. However, with stolen API credentials to SMTP-granted apps, hackers can send emails from trusted domains (which is precisely what happened to Namecheap customers).

Instead, users must pay attention to the websites that phishing emails direct them to. Sometimes, phishing websites look very real, not just in design but in imitating domains by mixing Unicode characters that look nearly identical (like the Cyrillic e). For Dropbox, it was an email from a fake CircleCI domain (circle-ci[dot].com).

The limitations of MFA in protecting against cybersecurity breaches

If you’re thinking, “Hey, MFA fixes this,” you’re unfortunately wrong. And, to your credit, this is a widespread misconception about MFA.

MFA was designed to protect users from password leaks/socially engineered password guessing. And while MFA might’ve curbed some more amateur phishing attacks (given that MFA codes are rolling and ephemeral), it doesn’t protect users from robust phishing campaigns known as real-time phishing proxies. Powered by programs like Evilginx and Modlishka, hackers can create multi-step authentication flows that request the SMS or authenticator app codes. If a user was duped into entering their password, they might also be tricked into entering their MFA code.

In the background, the real-time phishing proxy logs in on the user's behalf, accomplishing what is practically a man-in-the-middle (MITM) attack. If successful, the RTPP will score a bearer token, which could be used to impersonate the user further (which is precisely what happened to Dropbox).

Evilginx 2.0, RTPP proxy software

One of the maintainers of the popular RTPP tool, Modlishka, argues that companies can combat RTPPs by using hardware keys that employ the Universal 2nd Factor (U2F) protocol. Hardware keys are often advertised as the silver arrow to kill all cyber-attacks because they use a public/private key pair strategy similar to the TLS protocol. The approach considers the message’s origin when authenticating a user, preventing a phishing proxy from completing the request (or decrypting the bearer token).

Google is evidence of U2F’s success when U2F is mandated; despite being an obvious target, Google has not suffered a phishing attack since the implementation of U2F in 2017. However, Dropbox implemented hardware keys, too. So, how did it still get breached?

Analyzing Dropbox's mistakes with security measures

The role of a non-U2F, OTP key in the Dropbox data breach

Literature online often conflates different hardware key protocols. Unfortunately, a significant difference exists between a U2F and an OTP hardware key. U2F keys require a browser with WebAuthn support and use a real-time challenge to authenticate a user. They can also completely replace the username and password step (not strictly advised if the physical U2F key is stolen).

OTP hardware keys, meanwhile, can be used with any browser. Still, they are fundamentally not too different from SMS two-factor authentication one-time password codes or authenticator apps like Authy. The often advertised difference is that OTP hardware keys cannot be compromised because they aren’t an online third-party service vulnerable to traditional attack vectors. This is fair; however, OTP keys can be phished because they produce a code, not solve a challenge.

After entering their GitHub credentials while logging in to a fake CircleCI login page, Dropbox's employee used an OTP hardware key. In the background, an RTPP harvested a bearer token, which was used to access the real GitHub. There was little sign of suspicious activity.

Examining the implications of open access to code repositories

After gaining access, the hacker was able to access a lot of repositories.

Dropbox did not disclose the seniority of the compromised employee (technically speaking, it could’ve been the CTO) or their architecture. Often, engineering organizations have to grant every employee, even interns, access to all repositories because of microservice architecture.

There are often valid reasons why so many repositories may be accessible, so this is not a security mistake. It is, however, an example that illustrates how damaging a successful phishing attack can potentially be.

The dangers of plaintext API keys and their vulnerability to phishing attacks

The more significant mistake was that Dropbox was committing secret keys to repositories. This is a mistake for a few reasons. First, it causes those keys to become vulnerable to phishing exploits, which is what happened with Dropbox. Second, it enables hackers to have a single attack vector for accessing all keys (a compromised GitHub account means all keys are vulnerable). It also makes those keys available to CI/CD systems such as CircleCI, which can (and have) been compromised.

API keys should never be committed directly to repositories.

Strategies for mitigating the Dropbox data breach

Enhancing security with exclusive use and mandate of U2F keys

Had the Dropbox employee used a U2F hardware key instead of an OTP hardware key, the bearer token wouldn’t have been compromised in a man-in-the-middle attack. So, an excellent start to mitigating the attack is to mandate that employees switch to U2F keys and browsers that support U2F via WebAuthn. And, to Dropbox’s credit, that’s what they did.

Some have argued that phishing exploits are impossible with U2F keys. I wouldn’t go as far as saying phishing is impossible under U2F. It isn’t only that U2F might have an undiscovered exploit; U2F also relies on a cryptographic challenge that could be compromised by quantum computing should it ever succeed.

A downgrade attack is a more relevant U2F exploit for adopters to be aware of. In a downgrade attack, an RTPP requests the victim for a hardware key but only pretends to accept it. In the background, the RTPP demands a different authentication method from the service. Then, it requests an additional authentication method (e.g., an SMS 2FA code or an OTP code) from the user, posing it as a third authentication method. If the user doesn’t catch on, they might enter this “third” verification method, thinking it is “extra” security when it bypasses the U2F step altogether.

Organizations would need to make U2F the only acceptable authentication technique to prevent downgrade attacks. Unfortunately, that’s not realistic. U2F can be required on some tools (such as Google or Rippling); for others, U2F isn’t accepted or cannot be mandated as the sole 2FA technique. Additionally, U2F support is sometimes only available on prohibitively expensive tiers. For these reasons, employees must remain wary of phishing attacks and mindful of downgrade attacks.

Best practices: Avoid committing API keys to repositories

Companies should avoid committing real API keys to repositories (e.g., using .env files). Some could argue that managing secrets externally creates more exposure—as employees might revert to insecure communication channels to share API secrets—but that’s a poor man’s outlook on the problem.

Instead of committing secrets to repositories, companies should employ a single source of truth secrets manager to manage, govern, and orchestrate secrets. As a disclosure, this is related to what we do at Doppler—so take this recommendation with a grain of salt. We built Doppler because storing secrets in general engineering tools is less secure than keeping them in limited service with proper authentication techniques to safeguard and credibly orchestrate secrets.

The developer still has access to the secrets with Doppler or any other secrets manager. Accordingly, an employee could still be phished for their secrets manager credentials. The difference is that fewer employees need access to the secrets manager; employees only need access to the secrets pertinent to their roles, and any actions taken on the secrets manager are logged for auditing. While a secrets manager doesn’t eliminate the risk of phishing, it dramatically minimizes the exposure if an employee does get phished.

Implementing limits on API keys for enhanced security

If API keys are stolen, other techniques can limit the actual damage. Of course, these techniques should never be considered a substitute for suitable authentication protocols, just a limiting factor.

Unfortunately, these techniques do rely on the API vendor supporting them—many do not.

The first technique is to use scoped API keys. Scoped keys are API keys with strict access to only relevant API functions. For instance, a scoped key could allow a user to return an individual account using an identifier but not a list of all accounts. A scoped key can also minimize the risk of damaged data by restricting write permissions.

The second technique is actually to limit who can use an API key. Ideally, an API key cannot be used by any arbitrary device. A naive method to accomplish this is IP allowlisting, wherein only specific preconfigured IPs can use the key. This is an excellent technique in theory, but it is incompatible with many deployment workflows where IPs are routinely recycled. The need to constantly update an allowlist can risk a service outage.

A more robust technique is to replace API keys (or supplement them) with identity-based authentication, in which a server is verified as a company—owned server using the same methods discussed above. This is supported by most enterprise-friendly cloud providers and is connected to the cloud provider’s authentication suite (e.g., IAM or Active Directory). However, it requires the API to integrate with that cloud provider, which often isn’t the case.

The third technique involves automated secret rotation, where the secret manager can programmatically request and deploy a new secret. This is ideal; in the case of a breach, new keys can quickly be issued.

Unfortunately, many APIs don’t support any of these techniques. For those, using a secrets manager and practicing least privileged access to minimize the odds of an API key breach is especially necessary.

How to prevent a security breach: Key measures to take

Dropbox’s 2022 phishing exploit is unfortunate but not surprising. Advanced authentication methods such as U2F are still in their early days, and many companies conflate U2F with lesser hardware-backed protocols like OTP keys.

After building a solid security policy to protect against phishing, companies should continue to invest in limiting damage in case their systems are still breached. After all, someone could breach a system by physically stealing keys (harder, but possible). Companies can avoid the fate of Dropbox by not committing secrets to source code, relying on more stringent API authentication protocols, and limiting API access.