Revolutionizing secrets management: How AgentSync is cutting AWS costs by 10% and preventing 1000% growth in secrets spend

Customer background

Founded in 2018 in Denver, Colorado, AgentSync is reshaping the insurance SaaS landscape. The company develops advanced infrastructure solutions that optimize the onboarding, licensing, and appointing processes. Focused on customer-centric design, APIs, and robust automation, AgentSync ensures that compliance and growth go hand-in-hand. Its solutions meet a critical need in the insurance industry for effective and efficient distribution, establishing flexible connections between carriers, agencies, MGAs, and producers that enhance the delivery of insurance products. Essential for hundreds of insurance companies, AgentSync's products help scale distribution and reduce costs, aligning with the needs of tech-savvy consumers and the evolving regulatory landscape.

The challenge

As AgentSync grew, its operational and technological frameworks began to strain under the increasing complexity and costs associated with managing secrets across their expanding suite of services and development environments. These challenges were impacting the engineering team’s development speed, security posture, and financial efficiency.

Secrets sprawl

The rapid growth of AgentSync’s software offerings led to an exponential increase in the number of secrets—such as credentials, tokens, and API keys—that needed to be managed. This secrets sprawl resulted in the duplication of secrets across various services, often to maintain separation of concerns (SoC). "We were dealing with secret sprawl where you have secrets absolutely everywhere,” recalled Dallas Slaughter, a Staff DevOps engineer on the team. To ensure that processes were running smoothly, it was crucial to coordinate changes across environments and keep secrets synchronized. Any discrepancy or outdated secret could lead to a misconfiguration and a potential outage.

"As you start to develop more products and services, you can end up with just secrets absolutely everywhere.”

Escalating AWS Secrets Manager costs

AWS Secrets Manager’s cost was skyrocketing and becoming financially unsustainable. As the number of secrets and associated API calls grew, service usage grew significantly as well. Due to this surge, coupled with the complexities of AWS's consumption-based pricing model, AgentSync was on track to quadruple their expenses on AWS Secrets Manager within the next six to twelve months.

Moreover, as development expanded, these costs were projected to rise even further, potentially increasing the AWS spend by as much as 30%. This escalation in costs meant not only higher direct expenses from AWS but also greater operational complexity. Managing these costs became critical as they threatened to consume a larger portion of the budget.

”The projected costs for AWS Secrets Manager were set to reach 20 to 30% of our AWS budget within the next year without plateauing and would only worsen with scale. After reviewing the escalating number of secrets and API calls, we realized the costs would be substantial and it would be difficult to roll back.”

Operational overhead

The existing practices of managing secrets became burdensome. Managing AWS resources, assigning permissions, and ensuring the availability of secrets across different services added layers of complexity. "We were tasked with creating and managing resources. Handling permissions was particularly cumbersome when different services require access to secrets,” Dallas detailed. Rollbacks were notably problematic, which Dallas described as "annoying, difficult, and rarely done because they can be so frustrating."

Impact on developer productivity

The lack of standardized methods for secrets retrieval across languages, SDKs, and tools hindered developer productivity. As projects multiplied and the team grew, both new and seasoned engineers struggled with inconsistent practices when switching between projects. Dallas noted, "In a large codebase, you might find six different ways to retrieve secrets because developers either aren't aware of existing helper methods or the methods are specific to certain classes. This leads to widespread inconsistency and significant overhead."

Moreover, developers had to navigate a patchwork of tools requiring detailed configurations, such as SOPs and Terraform. Managing KMS keys and ensuring correct IAM roles added further complexity and overhead.

Challenges in onboarding and knowledge management

Onboarding new developers was inefficient, as it required them to learn varied and often project-specific methods for managing secrets. Much of the knowledge about specific projects was not documented but instead resided with existing team members. Relying on institutional knowledge meant that new developers often had to seek out specific individuals for guidance, which delayed their ability to contribute. This complexity not only delayed project ramp-ups but also heightened the risk of errors. Dallas explained,

"Secrets were often stored in insecure locations, such as directly in code or in .env files, which is not the correct approach. The lack of a central authority on secrets management created significant confusion.”

The DevOps team spent considerable time each month troubleshooting access issues and configuring secrets within AWS environments. This frequent need for support not only diverted valuable DevOps resources from strategic projects but also required continual training and retraining of developers on proper secrets management procedures.

Security and compliance risks

Distributing secrets across multiple environments introduced significant risks. Each new secret and its associated permissions expanded the potential attack surface.

"With 4,000 secrets dispersed across various regions and applications, we're constantly concerned about security. It's incredibly difficult to track what's occurring across such varied environments precisely.”

Accidental changes to credentials had led to several service outages, impacting service level objectives (SLOs) and causing customer dissatisfaction. Compliance became more challenging as the team had to account for thousands of secrets, verifying that each was properly managed, accessed, and secured according to SOC 2 requirements.

The team recognized the need for a more mature, scalable, and efficient secrets management strategy.

The solution

Dallas's prior experience with Doppler significantly influenced the decision-making process. He recalled, "Having been on the Team plan, I’ve done a lot with Doppler and was excited about using it again”. After a successful Enterprise trial, he and the team decided to roll out Doppler organization-wide.

Unified secrets management

Doppler centralized the management of all secrets, significantly reducing complexity and improving security. It automatically synced secrets across all environments, ensuring that each service was up-to-date without manual intervention. Features like secrets referencing drastically cut down on duplication by allowing secrets to be shared across multiple environments and applications. “With secrets referencing, we maintain one source of truth for each secret, minimizing the number of secrets to manage and enhancing security,” Dallas noted.

"The real value of Doppler lies in its consistency across all environments. Whether you're working locally, in production, or within the CI/CD pipeline, the process is the same. You simply run Doppler, and it works seamlessly."

Cloud cost management

By consolidating secret management tools and migrating secrets to Doppler, Dallas and his team significantly reduced their reliance on AWS Secrets Manager. With fewer secrets to manage and less reliance on API calls, Doppler lowered operational expenses and prevented vendor lock-in. “With Doppler, our AWS Secrets Manager spend is on track to drop significantly, saving us a considerable amount every month,” Dallas highlighted.

Operational efficiency and reduced overhead

Doppler minimized the time and resources needed to manage permissions and configurations. Its Terraform provider automated permission changes through code, facilitating easier audits, version control, and rollbacks, thus reducing time spent on repetitive tasks.

Doppler's dynamic secrets management, which injects secrets directly into the runtime environment, reduced the exposure of sensitive information and cuts down on the overhead associated with manual secrets management. Dallas summarized, "Doppler has removed all the aspects I disliked about our previous tools, massively simplifying our operations.”

Improving developer productivity and onboarding

One of the most significant benefits of implementing Doppler was in developer productivity and onboarding. By standardizing secret retrieval and configuration across all development stages, Doppler enabled new developers to become productive without having to navigate a myriad of different secrets management systems. Whether working locally or in production, developers use the same commands and interfaces to interact with secrets. This consistency reduces the learning curve and minimizes the risk for errors. Dallas highlighted, "The way you work on something locally is precisely how it operates in production. There’s no need to adjust settings or configure differently. You simply run Doppler."

Enhanced security and compliance

Doppler has bolstered AgentSync's compliance with SOC 2 and enhanced their overall security posture. Role-based access features like User Groups and Custom Roles enabled fine-grained control over who can access specific secrets, tailored by role and environment. Doppler’s logging and observability features such as activity /access logs and log forwarding to Slack and DataDog facilitated easier monitoring and compliance checks, providing clear, accessible change logs. "Doppler’s audit trails and secure storage have significantly boosted our security and compliance posture and allowed our security team to easily track every change made to a secret.”

The impact

10% reduction in cloud spend and 1000% prevented growth in secrets spend

By migrating secrets from AWS to Doppler, Dallas and his team achieved a 75% reduction in AWS Secrets Manager usage. This strategic shift not only reduced their immediate expenditures but also averted a forecasted increase in costs. Moreover, it eliminated additional expenses related to other services like AWS KMS and CloudTrail. As a result, Dallas and his team immediately cut their AWS spend by 10% and prevented a projected 1000% growth in spend related to secrets.

“By adopting Doppler early, we sidestepped significant projected costs on our AWS cloud spend. Reducing just 100 secrets in AWS Secrets Manager—excluding the savings from API calls—can fund one Doppler enterprise seat or even two-plus Team seats. The savings are substantial.”

Operational expense (OPEX) savings

Doppler’s implementation has drastically reduced the time and effort required to manage secrets. DevOps support requests related to secrets plummeted from an average of 23 hours per month to just 2-3 hours. Similarly, the engineering team reduced the time they spend in managing secrets by the same amount.

The team consolidated vendors and eliminated at least four different tools previously required for application secrets management (SOPS, Terraform, Chamber, 1Password). Dallas reflected, “No one enjoyed working with all these tools, and I'm probably forgetting some from edge cases. The reduction in operational effort has been staggering."

Boost in developer productivity and experience

Doppler’s adoption has also boosted developer team productivity and morale. The migration to Doppler resulted in the elimination of approximately 300 lines of code related to AWS SM API calls per service, simplifying the development process. Additionally, removing multiple outdated tools has streamlined the developers' toolkit, leading to a more efficient development environment and happier teams.

Shift left security & streamlined compliance

With Doppler, AgentSync has significantly enhanced its security posture.

“Doppler has enabled us to shift left, so far left that issues are addressed before we even encounter them. It comes out of the box with things that would be really big undertakings.”

Compliance tasks that previously took 2-4 hours now require no more than 5 minutes to produce necessary evidence.

The partnership ahead

Looking ahead, Dallas is looking forward to deepening AgentSync’s partnership with Doppler. By migrating most secrets to Doppler, he realized a considerable 10% reduction and completely prevented a skyrocketing increase in AWS spending. Dallas anticipates seeing potentially up to a 30% reduction in future AWS spending. This substantial cost saving highlights the strategic value of Doppler for improving both financial and operational performance. "Switching to Doppler didn't just save money—it made our developers happier and our platforms more secure,” concluded Dallas.