Secrets Management Is NOT Just a Key-Value Store

For those who’ve managed secrets via .env files or have briefly looked at secrets managers such as HashiCorp Vault , it’s easy to get the impression that “secrets management” is simply the storage and retrieval of secrets from a Key-Value data store.

But recurring tasks such as instantly comparing secret values between environments when troubleshooting, or being alerted when a new secret is introduced are crucial features every secrets manager should provide. Without them, development teams are forced to cobble together their own solutions, often in silos on a per-application basis.

The reality is that .env files and traditional secrets managers were never designed to meet the needs of modern application development teams, where microservices and multi-cloud deployments are the new normal.

Essentially, we want the flexibility of a Key-Value secret storage solution but with an operating model and feature set that decreases complexity, increases developer productivity, and standardizes how secrets are managed for applications in every part of the business.

In this post, we’ll explore the most important aspects of secrets management and why Key-Value storage is only the beginning.

Access Control and Permissions

Who needs access to secrets? What should their level of permissions be and which applications and environments should they have access to? Is access provided according to org structure or application ownership?

If access controls aren’t fine-grained enough, you risk violating the principle of least privilege but if too rigid, can impede the ability of teams to embrace a culture of DevOps and “shifting left” to bring security earlier into the application development process.

The following is a starting point for secret and access permission requirements:

• Ability to segment parts of the business into separate workplaces so for example, an acquired start-up can onboard to using the secrets manager while remaining entirely separate from the organization’s existing secrets.
• Consider how access will be managed across cloud and platform boundaries, e.g. AWS, GCP, and Vercel?
• Ability to grant access to secrets for specific applications and environments within each application.
• In most cases, developers should only have access to secrets belonging to their applications and only for the appropriate environment (e.g., dev and test). DevSecOps will have greater permission levels so that they can work cross-functionally in any environment.
• Temporary secrets access to external entities (e.g. contractors) should be scoped to a subset of application secrets via an auth/service token.
• One-off sharing of secrets to external entities (e.g. TLS Certificates to an external firm) should be captured in an activity log and sent via encrypted means using a service such as Doppler Share.
• Mandated MFA for secrets manager dashboard access.
• Privilege elevation (e.g. developer viewing production secrets) be time-limited and captured in an activity log.
• Seamless integration with existing Single Sign-on solutions such as SAML and SCIM to allow default access permission levels to be assigned based on role and/or group membership.

A further consideration is restricting machine access to a CIDR IP address range, preventing leaked service tokens from use outside of trusted corporate and cloud networks.

There’s a lot to consider for access control and permissions and your secrets manager should make it straightforward to meet these requirements.

Versioning and Rollback

Versioning and the ability to roll back changes is a must and shouldn’t come at the cost of reduced performance or additional storage fees.

While most secret managers support versioning and rollback, it’s critical to evaluate the actual process.

Being able to roll back a misconfigured change in a single click with support for triggering an automatic application reload or deploy is something teams should come to expect from a secrets manager, enabling a misconfiguration fix to be applied in seconds.

Secret Documentation

There are wildly different perspectives on documenting code, but the one thing all developers can agree upon is that useful code comments provide context that explains the “why”, not just what the code does.

Such context and comments can be especially valuable for secrets, but where should this be documented?

A reasonable option is a README or inline code comments where the secret is in use, however, the context may be lost if a secret’s value is changed in your secrets manager by someone who hasn’t viewed the code.

Being able to notate secrets directly in the secrets manager reduces the chance of misconfiguration errors that could’ve been easily avoided, had the appropriate comments (and context) be provided.

Local Development

The closer your development environment is to production, the less likely unexpected problems are to arise during deployment, which is why secrets should be accessed the same way in development, as they are in production.

Your secrets manager should treat Development as a proper environment in its own right and be responsible for supplying secrets to developers locally. This saves each developer from manually updating and patching their own environment variables list in their IDE or .env file.

Development environments also present a unique challenge where secret values will often be specific to each developer, e.g. when working on a feature branch that uses a new secret.

Developers need the flexibility to override secrets during development without affecting other developers on the team. This is an essential feature a secrets manager should provide to ensure secrets access remains consistent for every environment.

A Single Unified View for Application Configuration

One of the reasons developers love .env files is that it gives them a single unified view of how an application is configured as they tend to hold both secrets and config data such as API keys, port bindings, and feature flags.

When assessing the transition from .env files to a secrets manager, teams are often confused as to whether secrets and config should be separated. For example, should secrets be migrated to the secrets manager while config continues to exist in .env files?

While secret managers are designed to store sensitive data, it makes sense to store both config and secrets together, as there is no logistical or financial advantage gained by separating them.

If secrets and config are simply displayed as Key-Value pairs in the secrets manager’s dashboard, it’s up to the viewer to define a way to visually organize secrets by application, then further segment into separate environments.

Your secrets manager should provide a simple built-in mechanism to view secrets for a specific application and environment, and storing config as well as secrets in your secret manager ensures you’ll still get a single unified view of how your application is configured.

Reduce Friction to Increase Adoption

While implementing a secrets manager is often a security endeavor, it's better viewed as a productivity one, as every project team can appreciate time and effort saved. And if security posture is improved, well even better!

But if the focus is on security at the cost of productivity (even if short term), gaining widespread adoption will be significantly more difficult, as product teams are incentivized to ship new features, not improve security (unless mandated to do so).

Therefore, the secrets manager that is likely to have the biggest impact in improving security, is the one that provides a nearly frictionless experience for teams to implement and use on a daily basis.

Preventing Misconfiguration Issues

Misconfiguration issues are much easier to deal with if caught before changes are applied in production. To achieve this, developers and DevOps Engineers need access to a secrets activity log that pushes secret events to where they naturally communicate such as a Slack or Microsoft Teams channel.

The secrets management dashboard should also provide indicators of potential problems such as if a secret was added to the staging environment but not production, as well as troubleshooting features such as the ability to instantly compare a secret value across every environment.

A secrets manager must therefore help to prevent known issues associated with application configuration and deployment scenarios to increase production stability and reduce time spent firefighting issues caused by misconfiguration.

Automate Application Reload and Redeployment

The Infrastructure as Code movement put automation at the heart of deployment workflows and an essential feature of any secrets manager should be the option to automatically trigger an application to reload or redeploy if its config or secrets change.

This again highlights the need for secret managers to go beyond simply Key-Value storage, as, as they must know which application to trigger a redeployment for, based on the application and environment the changed secret belongs to.

Secrets Management Goes Beyond Secret Storage

By now, it should be clear that secrets management extends beyond the secure storage of secrets as Key-Value pairs. We’ve touched on processes, best practices, as well as various risks, challenges, and issues that can arise and what a secrets manager should reasonably do to mitigate them.

These relate to key elements of storage, access, visibility, integrations, workflows, and troubleshooting processes for not just secrets management, but application configuration generally.

Doppler is simply the result of building the SecretOps platform we wish had but simply didn’t exist.

Doppler enables you to stop using old ways to store, share, and access secrets via .env files.

It’s your centralized source of truth for managing secrets across multiple clouds, platforms, and packaging formats, from cloud-native build packs to containers, serverless, and more.