Juntiger

Cleaning Up Terraform State and Remaining Google Cloud Costs

Published Updated Views —

Removing empty state objects and the backend bucket, then checking the billable resources and IAM left in the project

Even when terraform state list is empty in both the server and recovery roots, state objects and the backend bucket remain in GCS. Confirm the exact object paths before deleting those records. Then search the whole project for disks, static IPs, Storage, and function-related billing items outside Terraform. Deleting state does not delete the real resources.

Prerequisites

terraform state list must print nothing in the server root and in the recovery root you applied. If even one entry remains, do not run the deletion commands below.

Destructive commands

Once you delete the state objects, Terraform’s management record is hard to recover. The local verification did not run the deletion commands below. Do not put a wildcard or ** in them.

The read-only listing commands that find soft-deleted objects are the exception: they use ** following the official Google Cloud syntax. Do not copy that value into gcloud storage rm.

Empty Listings in Both Terraform States

Start with the server root.

Check

cd "${HOME}/minecraft-one-root"
terraform state list

If you applied part 09, check the recovery root as well.

cd "${HOME}/minecraft-spot-recovery"
terraform state list

If there is any output, go back to part 10 and check the real resources and the destroy errors. Do not force the listing empty with state rm.

Backend Bucket and Prefix

Read the bucket and the prefix from the backend.hcl of both roots and save them as variables for the commands that follow.

Check

The block below reads the bucket and prefix values from the two backend.hcl files, puts them into four variables, and prints them at the end. sed extracts only the quoted values and does not change a file or touch a resource. Copy the whole block, run it, and check the last output.

cd "${HOME}/minecraft-one-root"
export STATE_BUCKET="$(
  sed -n \
    's/^[[:space:]]*bucket[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' \
    "${HOME}/minecraft-one-root/backend.hcl"
)"
export SERVER_PREFIX="$(
  sed -n \
    's/^[[:space:]]*prefix[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' \
    "${HOME}/minecraft-one-root/backend.hcl"
)"

export RECOVERY_BUCKET=""
export RECOVERY_PREFIX=""
if [[ -f "${HOME}/minecraft-spot-recovery/backend.hcl" ]]; then
  export RECOVERY_BUCKET="$(
    sed -n \
      's/^[[:space:]]*bucket[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' \
      "${HOME}/minecraft-spot-recovery/backend.hcl"
  )"
  export RECOVERY_PREFIX="$(
    sed -n \
      's/^[[:space:]]*prefix[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' \
      "${HOME}/minecraft-spot-recovery/backend.hcl"
  )"
fi

printf 'Server: %s / %s\nRecovery: %s / %s\n' \
  "${STATE_BUCKET}" \
  "${SERVER_PREFIX}" \
  "${RECOVERY_BUCKET}" \
  "${RECOVERY_PREFIX}"

The GCS backend object of the default workspace is <prefix>/default.tfstate. The public example uses minecraft/one-root/default.tfstate for the server and minecraft/spot-recovery/default.tfstate for the recovery root. Do not continue if STATE_BUCKET or SERVER_PREFIX is empty. If the recovery backend.hcl exists, RECOVERY_BUCKET and RECOVERY_PREFIX must also have values and the two bucket names must match. Empty recovery values are normal only when the recovery root was never created.

Every Version of the State Objects

Object Versioning is on, so read the live object and its earlier generations first.

Check

gcloud storage ls \
  --recursive \
  --all-versions \
  "gs://${STATE_BUCKET}/${SERVER_PREFIX}/default.tfstate"

if [[ -n "${RECOVERY_PREFIX}" ]]; then
  gcloud storage ls \
    --recursive \
    --all-versions \
    "gs://${STATE_BUCKET}/${RECOVERY_PREFIX}/default.tfstate"
fi

If you did not apply the recovery root, a missing second path is normal. Do not delete anything if you see another project’s state or a prefix you did not expect.

The ordinary listing does not show soft-deleted objects. Check the deletion record separately.

Check: soft-deleted state objects

gcloud storage ls \
  "gs://${STATE_BUCKET}/${SERVER_PREFIX}/**" \
  --soft-deleted \
  --full

if [[ -n "${RECOVERY_PREFIX}" ]]; then
  gcloud storage ls \
    "gs://${STATE_BUCKET}/${RECOVERY_PREFIX}/**" \
    --soft-deleted \
    --full
fi

Delete the live and noncurrent state objects only when both Terraform states are empty and the real resource deletion in part 10 is finished.

Deletes data: every live and noncurrent version of the server state

gcloud storage rm \
  --all-versions \
  "gs://${STATE_BUCKET}/${SERVER_PREFIX}/default.tfstate"

Deletes data: when the recovery state exists

if [[ -z "${RECOVERY_PREFIX}" ]]; then
  printf 'No recovery prefix, so nothing is deleted.\n'
else
  gcloud storage rm \
    --all-versions \
    "gs://${STATE_BUCKET}/${RECOVERY_PREFIX}/default.tfstate"
fi

When the deletion commands finish, run the --soft-deleted listing again on both prefixes. Record the hard delete time of the state generations in the output. Even when the ordinary object listing is empty, the cost of these retained copies can remain until that time.

Deleting the Empty Backend Bucket

Read the whole bucket recursively.

Check

gcloud storage ls \
  --recursive \
  --all-versions \
  "gs://${STATE_BUCKET}/"

Keep the backend bucket if another Terraform state is in it. Delete the empty bucket only when it belongs to this project alone and the output is completely empty.

Deletes data: when the live and noncurrent object listing is empty

gcloud storage buckets delete "gs://${STATE_BUCKET}"

The command fails if the bucket is not empty. This guide does not use the command that force-deletes the contents recursively. If a soft delete policy was on, the deleted bucket stays soft-deleted for the retention period.

Check: soft-deleted buckets in the project

export PROJECT_ID="$(gcloud config get-value project)"

gcloud storage ls \
  --buckets \
  --soft-deleted \
  --full \
  --project="${PROJECT_ID}"

If the state bucket you deleted appears in the output, record the hard delete time. You cannot permanently delete the bucket before that time, and storage cost can remain.

Billable Resources Left in the Project

Check

gcloud compute instances list \
  --project="${PROJECT_ID}"

gcloud compute disks list \
  --project="${PROJECT_ID}"

gcloud compute addresses list \
  --project="${PROJECT_ID}"

gcloud functions list \
  --v2 \
  --project="${PROJECT_ID}" \
  --regions="asia-northeast3"

gcloud storage buckets list \
  --project="${PROJECT_ID}"

gcloud artifacts repositories list \
  --project="${PROJECT_ID}" \
  --location="asia-northeast3"

For a project dedicated to this guide, the VM, disk, and static IP listings must be empty. In a shared project, other workloads may remain, so confirm that the server names and addresses from the destroy plan are gone. The function listing must not contain the recovery function. A backup bucket you kept in part 10 remains, which is normal, and you keep checking its Storage cost.

A Cloud Functions deployment may have left an Artifact Registry repository or image. Check first whether another function uses that repository. Do not delete it without evidence that it belongs to the Minecraft server alone.

Cleaning Up the OS Login Project IAM

Remove the roles/compute.osAdminLogin you added to your own account in part 05 if another VM does not need it.

Conditional run: when you do not manage another VM in this project

export ADMIN_ACCOUNT="$(
  gcloud auth list --filter=status:ACTIVE \
    --format="value(account)"
)"

gcloud projects remove-iam-policy-binding "${PROJECT_ID}" \
  --member="user:${ADMIN_ACCOUNT}" \
  --role="roles/compute.osAdminLogin"

Keep the role if another VM uses it too. Do not guess at other roles an organization administrator granted and remove them. Check on the IAM screen whether the serviceAccountUser binding of the VM service account was deleted along with the service account.

The last billing report

Open Billing → Reports in the Google Cloud Console and narrow it to the project and the dates. The report can update later than the usage. Look at it once right after the deletion, again a few hours later, and again the next day.

If the cost keeps growing, look in this order.

  1. Check the stored volume and the soft delete retained volume of the backup bucket you kept.
  2. Read the Compute Engine disk and external IP listings again for the whole project.
  3. Check the stored volume in Cloud Functions and Artifact Registry.
  4. Open the per-SKU cost in the billing report and find which product is charging.

Other workloads in the same project may use enabled APIs. During this cleanup, do not turn an API off without identifying every workload that uses it.

Completion Check

  • You deleted only the exact GCS objects, with the server and recovery states empty.
  • You read every object version in the backend bucket and deleted the bucket only when it was empty.
  • You recorded the hard delete time of the soft-deleted state objects and buckets.
  • The VM, disk, static IP, and recovery function are not in the listings.
  • You recorded the purpose and deletion owner of the backup bucket and Artifact Registry repository you kept.
  • You removed only the OS Login project IAM you do not need.
  • You checked the billing report right after the deletion and again after the delayed update.

References

Comments

Comments

    Image preview