Terraform State: The Part Everyone Gets Wrong First

Every Terraform horror story I've been called in to clean up (and consulting means I've seen a few) traces back to state. Not to modules, not to provider bugs, not to HCL's quirks. State. It's the least glamorous part of the tool and the one that decides whether your infrastructure-as-code setup is an asset or a time bomb.

Here's the mental model I wish someone had handed me at my first DevOps job: the state file is the blast radius. Everything else follows from that.

Remote state is table stakes, not an optimization

If your state lives on someone's laptop, you don't have infrastructure as code; you have infrastructure as that person. The first client engagement where I found terraform.tfstate committed to git (yes, with RDS passwords in it, more on that later), the team had three engineers each holding a slightly different local state for the same AWS account. Every apply was Russian roulette.

Minimum viable backend on AWS:

terraform {
  backend "s3" {
    bucket       = "acme-terraform-state"
    key          = "prod/networking/terraform.tfstate"
    region       = "eu-central-1"
    encrypt      = true
    use_lockfile = true
  }
}

Two notes. First, use_lockfile = true is the S3-native locking that landed in Terraform 1.10. You no longer need the DynamoDB table that every tutorial from 2016 onward tells you to build. If you're on an older version, keep the dynamodb_table argument; either way, locking is not optional. Two concurrent applies against the same state can corrupt it, and "we're a small team, it won't happen" lasts exactly until your CI pipeline and a human apply at the same moment.

Second, turn on bucket versioning. State corruption happens; versioning turns a disaster into a five-minute rollback.

State as blast-radius boundary

Every resource in a state file is exposed to every operation on that state file. One state file for your whole company means:

• A typo in a dev-environment security group triggers a plan that touches production databases.
• terraform plan takes eleven minutes because it refreshes 2,400 resources.
• A state corruption event takes out your ability to manage everything at once.

I inherited exactly this at a client: one state, ~3,000 resources, plans so slow the team had stopped running them locally and just YOLO'd applies through CI. We spent six weeks carving it up. The carving is painful (hello, moved blocks); living with the monolith is worse.

Split by lifecycle, not by team

The common instinct is to split state along org-chart lines: platform team gets a state, app teams get states. That's better than nothing, but the split that actually works is by rate of change and blast tolerance:

Things that change together and break together belong in the same state. Things with wildly different lifecycles should never share one. You don't want your VPC's fate coupled to a Lambda someone deploys ten times a day. Cross-state references go through terraform_remote_state data sources or, better, plain data sources looking things up by tag/name, which keeps the coupling looser.

Workspaces vs directories: pick directories

Opinion time, and I'll die on this hill: use directories per environment with shared modules, not workspaces.

infra/
├── modules/
│   ├── network/
│   └── eks-cluster/
├── prod/
│   ├── main.tf        # instantiates modules with prod values
│   └── backend.tf     # its own state key
└── staging/
    ├── main.tf
    └── backend.tf

Workspaces look appealing: one directory, terraform workspace select prod, done. In practice:

• Which workspace you're in is invisible in the code and lives in a dotfile. Every team using workspaces eventually applies staging config to prod. I've watched it happen twice.
• Real environments are never identical. Prod has deletion protection, bigger instances, extra alarms. With workspaces you end up with count = terraform.workspace == "prod" ? 1 : 0 sprinkled everywhere, which is conditional spaghetti pretending to be DRY.
• The backend key is shared-ish and magic. With directories, the state layout is explicit in backend.tf, greppable, reviewable.

Directories cost you a little duplication in the root modules. That duplication is documentation: a diff of prod/main.tf against staging/main.tf tells you exactly how the environments differ. Workspaces hide that.

(HCP Terraform's "workspaces" are a different, saner concept, closer to what I'm calling directories. Naming is hard.)

import and moved: refactoring without downtime

Two features that make state surgery civilized, both criminally underused:

# Adopt an existing resource without touching the CLI
import {
  to = aws_s3_bucket.assets
  id = "acme-legacy-assets"
}

# Rename/move without destroy-and-recreate
moved {
  from = aws_instance.web
  to   = module.web.aws_instance.this
}

Declarative import blocks (1.5+) mean brownfield adoption is now plannable and reviewable. You see in the plan output exactly what will be absorbed. moved blocks are how you restructure modules without Terraform deciding your production database needs to be destroyed and recreated because its address changed. Before these existed we did terraform state mv by hand at 2 a.m.; you don't have to live like that.

Drift: run plans even when nothing changed

Drift is the gap between state and reality: someone clicked something in the console, an autoscaler did its job, a provider default changed. The state file doesn't know until you refresh.

The cheap fix: a scheduled CI job running terraform plan -detailed-exitcode nightly against every state, alerting on exit code 2. It costs nothing and it converts "we discovered the manual firewall change during an outage" into "we got a Slack message Tuesday morning." Every client I set this up for finds real drift within the first week.

The secrets-in-state problem

Uncomfortable truth: Terraform state stores every attribute of every resource in plaintext, including ones marked sensitive. RDS master passwords, IAM secret keys created via aws_iam_access_key, TLS private keys from the tls provider, all sitting in that S3 object.

What that means in practice:

• Treat state storage with production-secrets rigor: encryption at rest, tight bucket policy, access logging. Read access to state is access to your secrets.
• Better: keep secrets out of Terraform's hands. Use aws_rds_cluster with manage_master_user_password = true so Secrets Manager generates the password and Terraform never sees it. Reference secrets by ARN, don't create their values in HCL.
• Ephemeral resources and write-only arguments (Terraform 1.10/1.11) finally let providers handle secret values without persisting them to state. Adopt them as your providers support them.

The short version

Remote backend with locking and versioning, day one. Split state by lifecycle, size each file to the blast radius you can stomach. Directories over workspaces. moved and import blocks for surgery. Nightly drift plans. And assume anyone who can read your state can read your secrets, because they can.

None of this is clever. It's plumbing. But it's the plumbing that determines whether Terraform scales with your team or becomes the thing everyone's afraid to touch.