If your team tried to rotate every credential today, something would break.
Not because your engineers are careless. Because your secrets are everywhere.
Hardcoded in the app. In Parameter Store. In Secrets Manager. In Vault. In a Kubernetes Secret nobody rotated. In a .env file that “isn’t production.”
Most platforms end up here. Not by design. By accumulation.
How Secrets Sprawl Happens
Each decision made sense at the time. Ship pressure. Familiar tools. Inherited systems. The progression usually looks like this:
Year one. The application is small. Credentials go in environment variables, maybe AWS Parameter Store. One or two services, a handful of secrets. Everyone knows where everything is.
Year two. The platform grows. Somebody introduces HashiCorp Vault for dynamic secrets. Some teams adopt it. Others keep using Parameter Store because it’s what they know. A third team uses Kubernetes Secrets directly because it’s simpler for their workflow.
Year three. A compliance audit flags that credentials aren’t being rotated. The policy says 90 days. The reality is 18 months - because nobody’s sure what will break if they rotate. A .env file in a legacy service turns out to contain a production database password. Nobody knows who put it there.
Year four. You have five systems holding credentials, no single inventory, and a rotation policy that exists only on paper.
This isn’t a developer problem. Developers will use whatever path works. If the platform offers five patterns, you’ll get five more.
The issue is architectural.
Why This Is an Operational Problem, Not Just a Security One
The security risk of secrets sprawl is well understood. But the operational impact is often worse - and it’s what actually prevents teams from improving their security posture.
Rotation becomes risky instead of routine
The most common reason credentials don’t get rotated is that the last time someone tried, something broke. A service went down. A cron job failed. A Lambda broke. A connection string hidden in a repo took production with it.
Everyone remembers.
So the 90-day rotation policy becomes a line in a document. And the credential sits unchanged for 18 months. Not as a policy decision - as a survival strategy.
Blast radius is unknown during incidents
When a credential is compromised, the first question is: “What has access to this, and what depends on it?” If your secrets are spread across five systems with no dependency mapping, that question takes hours to answer instead of minutes. Hours you don’t have during an active incident.
Audit trails are fragmented
If an auditor asks “who accessed this credential and when?”, can you answer that from a single system? Or do you need to correlate logs from Vault, CloudTrail, Kubernetes audit logs, and whatever else is in the mix?
Offboarding depends on hope
When an engineer leaves, can you confidently revoke every credential they had access to? Or does it depend on someone remembering which systems they touched?
Onboarding means teaching multiple patterns
New engineers need to learn not just one way to manage secrets, but several - and understand which pattern applies to which service. This slows onboarding and increases the chance of mistakes.
What a Mature Secrets Model Looks Like
The goal isn’t to standardise on a single tool at all costs. It’s to establish a clear model that covers three things:
1. A Single Supported Pattern
Engineers should have one well-documented, well-supported path for managing secrets in each context. That doesn’t mean one tool for everything - it means clear guidance:
- Application secrets: Use the External Secrets Operator to sync secrets from your chosen backend (Vault, AWS Secrets Manager, etc.) into Kubernetes Secrets. Applications consume Kubernetes Secrets. One pattern, regardless of the backend.
- Infrastructure secrets: Managed through your IaC pipeline with secrets stored in a backend that supports access auditing and rotation.
- CI/CD secrets: Managed through your pipeline platform’s native secret management (GitHub Actions secrets, GitLab CI variables), with short-lived credentials where possible.
The key principle: developers should never need to decide where a secret goes. The platform should make the right path the obvious path.
2. A Clear Ownership Model
Every secret should have an identifiable owner - the team or service responsible for its lifecycle. This means:
- Secrets are tagged or annotated with ownership metadata
- Rotation responsibility is assigned, not assumed
- Expiry policies are enforced, not suggested
- Orphaned secrets (no identifiable owner) are flagged and investigated
3. A Complete Inventory
You can’t manage what you haven’t mapped. A secrets inventory should capture:
- What the secret is (database credential, API key, TLS certificate, service account token)
- Where it’s stored (which system, which path)
- What depends on it (which services, which environments)
- When it was last rotated
- Who owns it
- How it gets rotated (manual process, automated, or not at all)
This inventory doesn’t need to be a separate system. It can be a combination of your secrets backend’s metadata, your configuration management, and your service catalogue. But it needs to exist, and it needs to be maintained.
A Practical Path Forward
If you’re starting from a sprawled state, here’s a realistic approach to improving it:
Phase 1 - Audit and inventory (2-4 weeks)
Map every secret across every system. Identify owners. Flag anything that hasn’t been rotated in 12+ months. Flag anything with no clear owner. This is unglamorous work, but it’s the foundation for everything else.
Phase 2 - Standardise the pattern (4-8 weeks)
Choose your target architecture. For most Kubernetes-native platforms, that’s:
- A secrets backend (Vault or cloud-native like AWS Secrets Manager)
- External Secrets Operator to sync into Kubernetes
- Clear documentation and templates for each use case
New services should use the standard pattern from day one. Don’t migrate everything at once.
Phase 3 - Migrate incrementally (ongoing)
Work through the inventory, migrating services from legacy patterns to the standard. Prioritise by risk: production credentials with no rotation, shared credentials, anything hardcoded.
Phase 4 - Automate rotation (ongoing)
Once secrets are in a system that supports it, enable automated rotation. Start with the easiest candidates (database passwords, API keys with provider support) and expand from there.
Phase 5 - Enforce and monitor
- Block new secrets from being created outside the standard pattern (admission controllers, pipeline checks)
- Alert on secrets approaching their rotation deadline
- Regularly audit the inventory for drift
The Takeaway
Secrets sprawl isn’t something that gets fixed in a sprint. It’s the result of years of accumulated decisions, and unwinding it takes sustained effort.
But the cost of not fixing it isn’t just a theoretical security risk. It’s the rotation that doesn’t happen because nobody’s sure what will break. It’s the incident response that takes hours instead of minutes. It’s the audit finding that could have been avoided.
Until there is a single supported pattern, a clear ownership model, and a complete inventory - you don’t have secrets management. You have secrets sprawl.
If you had to rotate everything today - what would break first? If you don’t know the answer, that’s the place to start.