The Current Secrets Rotation Process Is Broken

On January 4th, CircleCI posted a security notice alluding to a breach of customer API tokens, and consequently, secrets. Further details, such as what exactly happened, aren’t public yet. But, the notice mentions multiple times that secrets should be immediately rotated. If it were only that easy.

☣️ Responding to breaches and leaks will continue to be hard until services support programmatically managing secrets

The Elephant In The Room

Secrets (and its sibling passwords) aren’t perfect; they are, however, ubiquitous. The majority of organizations, at one time or another, will leak some or all of their secrets. Without an effective mitigation and response strategy, you’re stuck using a spreadsheet to wade through what needs to be rotated, hoping you don’t miss anything. The entire time your CTO is breathing down your neck because the CEO is breathing down theirs. Most importantly, your customers’ lives may be materially impacted.

Why Rotation Sucks

Secrets rotation has always been encouraged but remained a TODO. Reasons for not performing rotation often included potential downtime and fear of the unknown. The root of these issues - orchestration and observability - have largely been addressed.

There is one reason, however, that still persists and is the single biggest inhibitor to performing rotation - programmatic support for managing secrets in third-party tools.

☣️ Requiring users to login to a dashboard to manage secrets is not scalable. There are few things users would rather be doing less when responding to an incident

Keep Your Users Secure

It may seem intimidating to make managing secrets more accessible, but it’s actually more secure.

• Reduces secret sharing
• Reduces secret lifetime
• Facilitates least privilege adoption
• Allows for automated and push-button rotation

There are only a handful of endpoints needed to support managing secrets programmatically (preferably via OAuth):

• Create a secret
• Delete a secret
• Bonus points: duplicate and roll

Once these endpoints are implemented you'll just need to ensure you support more than one active secret instance at a time. That’s it.

A Better Way

Doppler set out to help organizations become more secure when using secrets. We’ve spent a lot of time thinking about rotation. A lot. We’re building towards a world where secrets are regularly rotated - automatically; where leaks can be responded to with the click of a button; where rotation doesn’t involve downtime and deployments are triggered automatically.

If you are a CircleCI customer leveraging Doppler secrets rotation, you can rotate five of the most commonly used secrets - instantly. And we’ve got a lot more on the way.

We’re excited about making companies more secure through rotation. You can check out a detailed write-up here. We think engineers will appreciate the decisions we’ve made and the outcomes we support.

Closing Thoughts

CircleCI’s notice triggered this blog post, but it’s not a dig at them. The intent is to highlight how close we actually are to making our industry fundamentally more secure. One may belittle secrets all they’d like, but we prefer to acknowledge things as they are and then make them better.

Stay Secure,
Thomas Piccirello
Doppler Co-Founder & CTO

Appendix

• There are a tremendous number of other things a service can do to keep their users safe; those are outside the scope of this blog post.
• Using secrets is getting better all the time thanks to protocols and patterns like OIDC, dynamic secrets, and OAuth.