Taming secrets sprawl with SecretOps

Passwords, tokens, encryption keys, and API keys start to pile up as we spin up new cloud resources and release new apps. Because different systems and people need to access this authentication information, these secrets suddenly become duplicated across various services — even in some places they shouldn’t be!

A typical IT environment may use container-related secrets managers, such as Kubernetes Secrets or Docker secrets. Or, they might use platform-specific secrets managers such as AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager. They may even use key-value stores, such as Hashicorp Vault and CyberArk Conjur.

Organizations frequently use environment variables in individual virtual machines (VMs) and serverless functions to configure applications and provide secrets such as API keys. Unfortunately, some developers are still hard-code sensitive data into their application’s source code, or using plain text unencrypted files such as a .env file. Of course, this is never a good idea, and most teams realize this, but a viable alternative that won’t impact productivity is often not apparent.

It becomes increasingly challenging for companies to distribute, organize, and secure these secrets as IT infrastructure grows. This haphazard storage of secrets is called secrets sprawl.

Secret sprawl is such a common problem because secrets hold together these different application components. Most modern applications consist of smaller building blocks and use a growing number of technologies.

As developers ensure each application has everything it needs, it’s easy for our secrets to get out of control. Everything seems fine until we need to make a change or — even worse — there’s a security breach. Suddenly, tracking down those credentials becomes a time-sucking, trying task that we never expected.

Let’s look at some of the pain points secrets sprawl causes and how we can get our secrets back under control.

Secrets sprawl: Pain points

When our secrets are sprawled all over our infrastructure, the development and operations teams (and, really, the whole organization) face some pain points. First, it’s challenging to keep track of where all those secrets are. When we need to find specific secrets, we might end up digging through various secrets managers, code, or even someone’s desk for that elusive USB stick.

Did our credentials just change? Now we have to track down every duplicate copy of that secret across our entire infrastructure.

Second, security is a notable secrets sprawl pain point. When our secrets spread across a wide area, it’s challenging to track who has access — let alone manage that access and enforce access rules. We should protect and encrypt all these secrets locations, but there are just so many. And they’re growing every day!

We know we should frequently change our secrets to reduce the chance of compromise (especially when an employee leaves), but secrets sprawl makes this a ton of work. Maybe we really wanted to work on that new feature instead, so we decided to just change our credentials next month.

Third, secrets sprawl makes it more challenging to share secrets. It’s great that our credentials are nice and secure in Kubernetes secrets. But that doesn’t help when we need to use that secret for an Elastic Compute Cloud (EC2) virtual machine or in our continuous integration and continuous deployment (CI/CD) pipelines.

Also, if secrets are in multiple locations, maybe only a few employees know where they all are. As a result, the organization becomes dependent on those specific employees, making it even harder to get new team members up to speed on the secrets architecture. There’s also the potential that the keepers of the secret might feel a sense of ownership and be reluctant to share. When they do share the secrets, doing it in plain text isn’t ideal.

Most organizations plan to scale their development team and architecture. Without transparency in secrets management, secret sprawling becomes a major scaling bottleneck. Growth also means more secrets, and more secrets means more locations in which secrets are stored.

Now that we know some of the pain points secrets sprawl causes, let’s look at some solutions.

Managing secrets sprawl with a secrets manager

Centralization helps keep secrets sprawl in check. When you keep all your secrets in one central location, you know where to find them and can easily make changes and monitor who and what machines and services have access. Secrets managers help centralize your secrets by storing them in one place where all your services connect.

If your company is small enough, you might decide to deploy all applications as Kubernetes containers and store all secrets in Kubernetes Secrets. This centralization tightens access control and makes it easier to audit secrets. Maintaining a centralized and encrypted secrets store ensures the organization applies consistent threat protection. It reduces unauthorized access and boosts security posture.

In a large company, one administrator might manage multiple Kubernetes clusters as you scale. In this situation, individual users can’t enforce application-wide policies. This means secrets are better managed.

Sticking to a single cloud provider and using its built-in secrets manager helps. But, this really limits what you can do with your infrastructure. A single provider might not have all the services you need for dynamically growing infrastructures. Many organizations are pivoting to the multiple cloud computing services approach for better disaster recovery and better servicing users distributed over a larger area (or even internationally) with edge computing.

Choosing a multi-cloud secrets manager

Multi-cloud capable secret managers such as Doppler and HashiCorp Vault reduce secrets sprawl by centralizing secrets storage with integrations for a wide range of platforms.

Vault’s Key-Value store, while flexible, wasn't designed for organizing secrets by application or microservice. Developers must determine how secrets will be stored and extracted from Vault and injected into their application. Some organizations will need and appreciate Vault’s flexibility but it could be prohibitively complex for others, making them avoid using it and defeating its purpose. Its storage and maintenance costs can also add up. Above all, secrets management is not just about key-value storage.

But Doppler offers a better solution. It was built for today's microservice world, organizing secrets by application with a customizable list of environments. It enables you to control all your secrets centrally, and you can set it up across every environment, from development to production.

Doppler delivers an integration experience for every cloud provider and platform. While Doppler provides centralized secrets storage management, you can still use the built-in secrets managers your favorite cloud services provide, such as AWS Lambda, Azure Key Vault, Heroku, Vercel, and a growing list of others. Secrets automatically sync to external secrets stores, saving developers time so they can focus on building products and features.

With Doppler, it’s a completely managed service, saving teams time and money otherwise spent on deploying, updating, and supporting a self-hosted secrets manager.

Conclusion

Because of the growing trend to deploy microservices to multiple clouds and platforms, every organization faces secrets sprawl, even if they don’t realize it yet. It’s a natural consequence of growth. As developers, we can be so immersed in our secrets sprawl, or only focussed on our particular application that we aren’t even aware of the bigger picture and impact secret sprawl is having across teams. It becomes the status quo despite hindering productivity.

Secrets managers reduce secrets sprawl by securing all these credentials in a central location. Change your credentials everywhere with just one click and see exactly who and what can access any given secret.

‍

Whether you’re part of a fresh startup or a growing organization, it’s never too early or too late to wrangle your secrets into one manageable dashboard. Doppler is fast becoming the preferred secrets manager for taming secrets sprawl that both developers and security folks love using and the free Community plan is a great way to evaluate the platform and get started.