From Chaos to Control: Transform Your Secrets Workflow with SecretOps

Every application has three components: code, compute, and secrets. As an industry, we're getting pretty good at managing the first two. For code, we have automation around git, like pull requests and CI. For compute, we have automation around our resources, like Terraform and Kubernetes. But the automation around secrets really hasn't matured like the other components.

Take a minute to think about how your team is managing secrets as they flow from development to production. If I were a developer on your team, how would I go about adding a new secret? Suppose I'm tasked with integrating with a new service, like SendGrid. Where do I put that API key during my local development? How do I share that key with the team when my changes are merged? What do I need to do to make sure that the secret exists in production before my feature lands? Does your team’s process for secrets feel as mature as your processes for code and compute?

SecretOps is the framework for automating secrets workflows in your team. My goal isn't to sell you on Doppler; it's to sell you on SecretOps.

There are a few layers to SecretOps and they aren't really features or boxes to be checked; they're more like workflow automations that build on each other. Identifying these layers lets us talk about them more abstractly and figure out which parts of our workflow need improvement.

Let's dive in.

Secrets Storage

I worked on a small team some years ago with some really talented engineers. When I joined, I cloned the repo I was supposed to be working on and tried to fire up the server. It crashed, of course, because I was missing an environment variable called GOOGLE_MAPS_API_KEY.

Naturally, I pinged my manager and he directed me to...let's call him Steve. Every team has a Steve, right? This is the engineer who has been at the company for ages and they know all the things. "Ah yep," he said, "I'll Slack you my environment file".

Lucky for me, Steve wasn't on PTO.

What happens when I need to go add that SendGrid integration? Do I just add SENDGRID_API_KEY to my env file? How do I make sure all of the other devs get it? How do I get it into staging? Production?

Steve can tell me, of course. And I really hope the answer isn't "file a ticket with the ops team." Regardless, we can already tell that there are too many human steps in this process. Steps that we're quite likely to forget.

This is the secrets storage layer. It's more than keeping secrets safe, it's about knowing exactly where to look to find a secret, in any environment, across time. In a SecretOps platform, the answer to the question "where is secret X?" should always be obvious.

When secrets are centralized, it becomes a lot easier to reason about how and where they're used.

Secrets Governance

The natural next question, though, becomes: "Who can access these secrets?"

My product engineers need access to development environments in almost all projects but they really don't need to see production secrets. What happens when we add new projects?

These are all questions around secret governance. Teams ultimately need to decide for themselves who needs access to what to do their job. However, a good SecretOps tool can give you the flexibility and granularity to achieve the access control that you want.

The name of the game is, of course, automation. A SecretOps tool can give you abstractions like groups and roles to grant access automatically. Paired with SAML and SCIM, you can stop thinking about getting developers access to secrets when they join the team.

The most important piece of governance, though, is audit logs. Every change should be tracked and traceable back to the user who made it. Secrets are living values that can change over time and their histories should be managed, just like we do with code.

Secrets Orchestration

It's great that we have all of our secrets secured, organized, and accessible to the right users. But how do we actually use them in our workloads?

When I worked with Steve, we kept our secrets in a 1Password vault. Honestly, this does a pretty good job of covering the secrets storage and governance requirements for most teams. Where it really starts to fall short is at the orchestration layer. When a new secret needed to be added to production, we just needed to make sure to copy it into 1Password and copy it to the environment variable store for our compute instances.

Manual processes like these take down production apps. It's the reason that we're scared to touch secrets and risk aversion slows us down.

The orchestration layer is all about getting secrets to where they need to be consumed. It's like CI/CD for secrets.

A good SecretOps platform should "meet you where you are" when it comes to orchestration:

• When you’re working in development, you’ll want a CLI or similar tool to inject dev secrets into your running app
• If you're working in Kubernetes, maybe native Kubernetes secrets would be the easiest to work with. Your SecretOps platform could actively sync secrets from the "source of truth" into native Kubernetes secrets. Or perhaps a sidecar which serves secrets to your app would make more sense.
• If you're shipping a Lambda, maybe it would be the most convenient for your app to pull secrets from AWS Secrets Manager. Your SecretOps platform could sync secrets there for you instead.
• If you're shipping to managed platform like Netlify or Fly.io, their native store will probably be the easiest to work with. Sync your secrets there.

The key is that the process is automatic and that your team isn't worrying about if their workloads will have access to the secrets they've added.

A very powerful extension of orchestration is automatic restarts. What if your workloads redeployed, or your website was automatically rebuilt, as soon as your secrets were changed?

Secrets Lifecycle

The problem with secrets is that they live forever. The SendGrid API key that I generated years ago will still work today if I don't rotate it. Any former teammate who saw that value could have taken it with them when they left.

I don't have a story about rotation. When I worked with Steve, we just didn't rotate secrets, plain and simple. We knew that we should but rotating almost felt more risky than leaving them be. If we did rotate, who would do it? How often? Which secrets? We must have dozens of secrets per project, this has to be done by hand?

Rotation seemed so daunting because we hadn't built up our earlier layers. We weren't confident in which services were using which secrets and we didn't have a centralized place to update them. Heck, if we had storage, governance, and orchestration figured out, manual rotation wouldn't have even been that bad.

We can do even better though. If you're confident in your earlier layers, a SecretOps platform can perform this rotation for you automatically. Imagine setting up a system where every 30 days, a new SendGrid API key would be issued and saved to your secrets store, delivering the secret to where it needs to be, and automatically redeploying your apps -- while still keeping the old API key around for a while to bleed off until it's eventually revoked.

Secrets Observability

Every time a secret comes across the wire and is delivered to a user or a workload, it increases the potential for a leak. When viewed this way, a secret read is effectively as important as a secret write. Why wouldn't we want to track that event as well?

A SecretOps platform can not only track these events but help you use them to answer questions like:

• Someone just left the team, I need to know every secret they've seen
• Are any services still using this legacy API key?
• A secret has been leaked. Which users or services had access to it?

This layer was so far away from me and Steve that I don't really think we even considered it possible.

Wrap-Up

I'm happy to say that Steve and the team have put a ton of work into their SecretOps layers and I know that they're reaping the benefits.

Every application has three components: code, compute, and secrets. Hopefully I've sold you on the idea that you can get a massive amount of value from automating your secrets, just as you have from automating your code and compute.