Migrate the Terraform S3 backends off dynamodb_table onto use_lockfile and delete the lock tables #218

Overview

We need to replace the deprecated dynamodb_table backend parameter with use_lockfile = true in hackforla/incubator and hackforla/devops-security, then delete the two now-unused DynamoDB lock tables, because HashiCorp deprecated DynamoDB-based state locking in Terraform 1.11 and will remove it in a future minor version. Both repos already run 1.16.1, so every plan and apply in both is printing the deprecation warning today.

Action Items

Facts verified 2026-09-07, so you do not have to re-derive them:

• The warning fires 3x per plan run and 4x per apply, in every run sampled in both repos back to 2026-08-08. It reads Warning: Deprecated Parameter / The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile" instead. and points at backend.tf line 7, which is misleading — that line is an empty backend "s3" {} block. The parameter is in terraform/prod.backend.tfvars, which the workflows pass as backend_config_file.
• use_lockfile went generally available in Terraform 1.11, in the same release that deprecated the DynamoDB arguments. Both repos resolve required_version = "~> 1.16.0" to 1.16.1, so there is no version work.
• No new IAM permission is needed. S3-native locking needs s3:GetObject, s3:PutObject and s3:DeleteObject on <key>.tflock. The apply roles (incubator-tf-apply, devops-security-tf-apply) hold AdministratorAccess. The plan roles never take a lock at all — dflook/terraform-plan passes -lock=false unconditionally (image/entrypoints/plan.sh:15, checked at both the pinned v1.49.0 and at main). That is also why the plan roles work today holding only ReadOnlyAccess over DynamoDB.
• Both state buckets (hfla-incubator-terraform-state, hfla-ops-terraform-state) have versioning Enabled.
• Both lock tables hold exactly one item and it is not a lock — it is the …/terraform.tfstate-md5 digest row. No lock is currently held in either.
• Both tables have DeletionProtectionEnabled: true, so deleting them is a two-call operation. Neither has point-in-time recovery.
• These are the only two DynamoDB tables in account 035866691871. After step 3 the account has none.

1. hackforla/incubator — 1 PR, terraform/prod.backend.tfvars:

• Delete the dynamodb_table = "hfla_incubator_terraform_table" line and add use_lockfile = true.
• Add a no-op comment line to any .tf file in the same PR. Both workflows filter on paths: ['**/*.tf', '**/*.yaml'], and prod.backend.tfvars matches neither pattern — so a PR that changes only the tfvars triggers no workflow at all, on the PR or on the merge. This is not optional: without it there is nothing to verify against and the merge produces no apply run.
• Treat a non-empty plan as a hard gate. A backend change produces zero resource changes. If the plan shows any, stop and raise it rather than merging.
• After the PR merges, confirm the Terraform apply (OIDC) run succeeds and that no Deprecated Parameter warning appears anywhere in its log.

2. hackforla/devops-security — 1 PR. This one is bigger than incubator's, because three other things in this repo reference the lock table:

• terraform/prod.backend.tfvars — delete dynamodb_table = "hfla_ops_terraform_table", add use_lockfile = true.
• terraform/aws-custom-policies/tf-plan-scoped.json — delete the whole AllowDynamoDBLockTableAccessForTerraformPlan statement. This file is live: terraform/aws-custom-policies.tf feeds it to modules/aws-policies, which deploys it as the IAM policy IncubatorTfPlanSecretsRead, currently attached to the incubator-tf-plan role. Leaving the statement would leave a live policy granting DynamoDB actions on a table that no longer exists.
  • Do not add .tflock permissions to replace it. The statement immediately above it already grants s3:PutObject/s3:DeleteObject on arn:aws:s3:::hfla-ops-terraform-state/*, which covers the lock file — and the plan role does not lock anyway.
  • Do not "fix" the fact that a policy named IncubatorTfPlanSecretsRead, attached to incubator's plan role, grants access to devops-security's backend. That cross-wiring predates this work. If it looks wrong to you, raise it as its own issue.
• CONTRIBUTING.md — delete the #### Set up DynamoDB to store the backend state section (lines 157–174, through the *** after Back to Top), and in the Creating Local tfvars file example (~line 251) swap dynamodb_table = "hfla_ops_terraform_table" for use_lockfile = true. This is the section that tells every new member to create a lock table by hand.
• .github/ISSUE_TEMPLATE/pre-work-template-devops-security.md — delete the - [ ] Create the DynamoDB table sub-item (line 40) and reword its parent (line 38) so it no longer promises a DynamoDB step.
• Same no-op .tf comment requirement, and it is worse here: this repo's filter is paths: ['**/*.tf'] only, so .tfvars, .json, .md and the issue template all fail to trigger it.
• The plan gate is different in this repo. Expect exactly one change: an in-place update of module.aws_custom_policies.aws_iam_policy.custom_policy["IncubatorTfPlanSecretsRead"], from the JSON edit. Anything else — any replacement, any second resource — is a stop-and-raise.
• After the PR merges, confirm Apply Terraform changes on merge succeeds with no Deprecated Parameter warning, and that the policy's new default version no longer contains the DynamoDB statement.
• After the merge, re-open /issues/new/choose and confirm the pre-work template renders without the DynamoDB step. GitHub renders issue templates from the default branch only, so this genuinely cannot be checked from the branch.

3. Delete both tables — only after both PRs have merged and both apply runs are green:

• Confirm no lock is held in either table. aws dynamodb scan --table-name <table> --region us-west-2 --query 'Items[].LockID.S' must return only the -md5 row.

• Record both table definitions in a comment on this issue before deleting, so the change is reversible without a PR trail to read. Capture aws dynamodb describe-table and aws dynamodb list-tags-of-resource for each.

• Disable deletion protection, then delete. Both calls need admin in 035866691871 — if you do not have it, hand this step off rather than working around it.

  for t in hfla_incubator_terraform_table hfla_ops_terraform_table; do
    aws dynamodb update-table --table-name "$t" --region us-west-2 --no-deletion-protection-enabled
    aws dynamodb delete-table --table-name "$t" --region us-west-2
  done

• Verify: aws dynamodb list-tables --region us-west-2 returns an empty list, and the next apply in each repo still succeeds.

• To reverse, recreate the table and re-enable protection. Terraform rewrites the -md5 digest row itself on the next apply, so the row does not need restoring. hfla_ops_terraform_table additionally carried tags managed-by=exempt and project=devops-security; hfla_incubator_terraform_table carried none.

  aws dynamodb create-table --table-name <table> --region us-west-2 \
      --attribute-definitions AttributeName=LockID,AttributeType=S \
      --key-schema AttributeName=LockID,KeyType=HASH \
      --billing-mode PAY_PER_REQUEST
  aws dynamodb update-table --table-name <table> --region us-west-2 --deletion-protection-enabled

Do the backend swap in one step, not a two-phase migration — but know why:

• Terraform documents that dynamodb_table and use_lockfile may be set simultaneously, and that is the official migration path. Its only purpose is to keep two clients that disagree about the locking mechanism from both acquiring a lock. Since holding both keeps the deprecation warning firing, a phased migration means four PRs to clear a warning that two will clear.
• The risk it covers is real but small here: while this is in flight, a terraform apply from a local checkout that still has dynamodb_table would not see a CI lock taken through S3, or vice versa. Mitigate by not running a local apply against either repo on the day the PRs merge, and by the lock check in step 3.
• Note for anyone with an existing local checkout: CI initialises from a clean container every run, so there is no cached backend config and no migration prompt. A local .terraform/ directory will prompt Backend configuration changed and needs terraform init -reconfigure.

Out of scope — do not do these here:

• Do not touch the action pins in these repos' workflows. actions/checkout, aws-actions/configure-aws-credentials and dflook/terraform-* belong to Bump actions/checkout to v5 across devops, devops-security and incubator #183, Bump aws-actions/configure-aws-credentials from v3 to v6 in incubator's Terraform workflows incubator#158, Bump dflook/terraform-plan and terraform-apply from v1 to v3 in incubator's Terraform workflows incubator#159 and Bump configure-aws-credentials to v6 and dflook/terraform-* to v3 in the Terraform workflows devops-security#170.
• Do not tag or otherwise adopt the two state buckets. hfla-incubator-terraform-state is still untagged and still shows as unmanaged in the Terraform coverage report; that is separate work.

Resources/Instructions

• Files, all on main in both repos. Line numbers were accurate 2026-09-07 and may drift — locate each by its content rather than by position.
  • hackforla/incubator: terraform/prod.backend.tfvars
  • hackforla/devops-security: terraform/prod.backend.tfvars, terraform/aws-custom-policies/tf-plan-scoped.json, CONTRIBUTING.md, .github/ISSUE_TEMPLATE/pre-work-template-devops-security.md
• S3 backend documentation — the use_lockfile argument, the statement that DynamoDB locking "is deprecated and will be removed in a future minor version", the simultaneous-configuration note, and the .tflock permission list.
• Terraform 1.11 CHANGELOG — "S3 native state locking is now generally available", the release that introduced the deprecation.
• dflook/terraform-plan's -lock=false behaviour: image/entrypoints/plan.sh line 15.
• Runs showing the warning, for comparison after the fix: incubator apply 34061802778, devops-security apply 34009820416.