Local state cannot be unlocked by another process on terraform

Question

My terraform remote states and lockers are configured on s3 and dynamodb under aws account, On gitlab runner some plan task has been crashed and on the next execution plan the following error pops up:

Error: Error locking state: Error acquiring the state lock: ConditionalCheckFailedException:
The conditional request failed

Lock Info:
  ID:        <some-hash>
  Path:      remote-terrform-states/app/terraform.tfstate
  Operation: OperationTypePlan
  Who:       root@runner-abc-project-123-concurrent-0
  Version:   0.14.10
  Created:   2022-01-01 00:00:00 +0000 UTC
  Info:  some really nice info

While trying to unlock this locker in order to perform additional execution plan again - I get the following error:

  terraform force-unlock <some-hash-abc-123>

  #output:
  Local state cannot be unlocked by another process

How do we release this terraform locker?

Answer 1

According to reference of terraform command: force-unlock

Manually unlock the state for the defined configuration.

This will not modify your infrastructure. This command removes the lock on the state for the current configuration. The behavior of this lock is dependent on the backend being used. Local state files cannot be unlocked by another process.

Explanation: apparently the execution plan is processing the plan output file locally and being apply on the second phase of terraform steps, like the following example:

phase 1: terraform plan -out execution-plan.out

phase 2: terraform apply -input=false execution-plan.out

Make sure that filename is same in phase 1 and 2

However - if phase 1 is being terminated or accidentally crashing, the locker will be assigned to the local state file and therefore must be removed on the dynamodb itself and not with the terraform force-unlock command.

Solution: Locate this specific item under the dynamodb terraform lockers table and explicitly remove the locked item, you can do either with aws console or through the api. For example:

aws dynamodb delete-item \
    --table-name terraform-locker-bucket \
    --key file://key.json

Contents of key.json:

{
 "LockID": "remote-terrform-states/app/terraform.tfstate",
 "Info": {
   "ID":"<some-hash>",
   "Operation":"OperationTypePlan",
   "Who":"root@runner-abc-project-123-concurrent-0",
   "Version":"0.14.10",
   "Created":"2022-01-01 00:00:00 +0000 UTC",
   "Info":"some really nice info"
   }
 }

Answer 2

terraform force-unlock <lock id>

For terragrunt, in <terragruntfile>.hcl directory, run terragrunt force-unlock <lock id>. If didn't work, remove terragrunt.lock.hcl and .terragrunt-cache/ and try again.

Also

Answer 3

If you are using Terragrunt and can see that the lock is for a specific module, you can do the following:

1. Navigate to the relevant terragrunt directory for that module
2. run terragrunt force-unlock
3. type "yes" to confirm

Lock should now be unlocked locally and on remote.

Answer 4

Here is my case. I use Circleci and I have Terraform-plan as a job in my workflows. The job was failing because Terraform state was locked. I tried to add terraform force-unlock as a command under terraform-plan, but it faced the error Local state files cannot be unlocked by another process.

So what I did. I have an infrastructure directory in my application root that holds Terraform configurations. Down in the directory there are two files .terraform.lock.hcl and main.tf that contain some configs, inclusing Terraform backend, that in my case it is "s3". So I had to call the unlock command in this directory:

cd infrastructure/dev
terraform force-unlock '<lock_id>' 

The lock_id can be found in the error message, when it says the state is locked.

Note that you should have this directory even if you are not using Circleci.

I hope it helps someone.