Secrets Spread Across HCL, State, Metadata, and a ZIP

Putting a default on a Terraform variable is convenient. You do not have to attach -var every time, and you do not have to maintain a separate terraform.tfvars. The 2025 configuration applied that convenience to secrets as well.

variable "discord_bot_token" {
  description = "Discord bot token"
  type        = string
  default     = "<DISCORD_BOT_TOKEN>"
}

That file held one bot token, three webhook URLs, and the RCON password the same way. I wrote the values in one place, but the values did not stay there. I compared the repository code and state to trace every location where they were copied.

Everything quoted below is a placeholder. No real value appears anywhere in this post, and I did not use masking that leaves a few characters at either end, because length and prefix alone give away what a value is.

The Six Places One Value Reached

flowchart TB
  V["variables.tf default"] --> V2["Variable file in another root"]
  V --> S["Fallback constants in three scripts"]
  V --> Z["The .env inside discord-v2.zip"]
  Z --> M
  V --> P["plan"]
  P --> ST["terraform.tfstate"]
  ST --> B["State backup file"]
  B --> G["Git commit"]
  V --> M["VM metadata"]
  M --> E["The .env on the VM disk"]

Walking through where the value arrived, in order. First, the variable file in the preemption recovery directory carried the same webhook URL as a default. Change one place and the other stays as it was.

It was in the script bodies too. startup.sh, backup.sh, and shutdown.sh each held a different webhook URL as a constant, and all three matched the variable defaults.

WEBHOOK_URL=$(get_md "instance/attributes/discord-webhook-url")
if [ -z "$WEBHOOK_URL" ]; then
  warn "discord-webhook-url metadata missing—using fallback"
  WEBHOOK_URL="<DISCORD_WEBHOOK_URL>"
fi

Delete the value from the variable and this line stays. On top of that, backup.sh and shutdown.sh look the metadata key up with underscores while Terraform sent it with hyphens. Those two scripts always got an empty value and used nothing but this fallback constant. Removing only the variable default leaves this constant in the code.

It went up to the VM as well. The token, RCON password, and three webhooks were passed as instance metadata keys. Metadata is not a secret store. Principals allowed to read the instance configuration and processes inside the VM can access these values.

The startup script used the values read from metadata to build an environment file on disk. Permissions were restricted to 0600, but counting only this far, the value sits in three places: the code, the metadata, and the disk.

The same value went into the application bundle. discord-v2.zip, which packages the Discord bot source, contains a .env file, and the bot token in it is the same value as the variable default. On every deploy a batch file rebuilds this ZIP, and Terraform reads it with filebase64 and loads it into metadata.

The same values also entered state in plaintext.

State records resource attributes as they are. In this configuration, sensitive = true only hid output while the value inside the file remained plaintext. It used no remote backend either, so state was a file in the working directory. Terraform can leave backup files when it changes local state. This repository contains both terraform.tfstate.backup and timestamped snapshots, but with no execution logs I cannot attribute each file to a particular command.

Five State Backups Left Behind .gitignore

The repository’s .gitignore has the rules it needs.

*.tfstate
*.tfstate.backup

Going by the rules alone, state does not go into Git. When I checked while writing this in August 2026, five state backups were still tracked.

restart-preempted-instance/terraform.tfstate.<epoch>.backup
restart-preempted-instance/terraform.tfstate.<epoch>.backup
restart-preempted-instance/terraform.tfstate.<epoch>.backup
restart-preempted-instance/terraform.tfstate.<epoch>.backup
start-minecraft-server/terraform.tfstate.<epoch>.backup

.gitignore applies only to files that are not tracked yet. A file that has been committed once stays tracked even if the rule is added later. These five files kept being tracked after the rule went in.

One tracked server state backup is over 220,000 bytes. Counting patterns without reading values shows what is inside.

Pattern	Count
Discord webhook URL	4
Bot-token-shaped string	1
RCON password key	6
SSH public key	1

This table counts only what is visible in plaintext. The same file holds the discord-bot-zip metadata value as a base64 string, and decoding it produces the .env seen earlier, intact. That is one more copy of the token, and a pattern search does not catch it. Hunting for secrets without decoding encoded fields as well means missing them.

A value written into a variable default passed through the plan and entered state in plaintext. When a backup of that state was committed, the value stayed in the repository with it. Further back in the history, the state file itself, not a .backup, was committed at one point.

The Range a History Rewrite Undoes

The repository still carries traces of a git filter-repo run. That means a history rewrite happened once.

The five files above are still tracked. Whether they fell outside the rewrite’s scope or were committed again after it, the repository alone cannot tell. What is certain is that fixing a history does not finish in one command.

The following remains after a history rewrite.

• Copies already cloned or forked do not change
• The memory of anyone who saw the value before the rewrite cannot be undone
• Objects cached by the remote host may remain
• Above all, the value itself is still valid

Rewriting history removes the values from clones made afterward. It does not revoke values that were already copied.

Invalidation Comes Before Cleaning Up the Code

When you find that a secret has spread, the order matters.

Invalidate the value. Reissue the token, retire the webhook, change the password. Once a value is scattered across six places, there is no way to prove which copy you missed. With no proof available, the only route left is making the value itself useless.

Then clean up the code and the history. Delete the variable defaults, remove the fallback constants from the scripts, and take tracked files out with git rm --cached. We saw above that fixing only .gitignore leaves already-tracked files in place. Rewrite the history too if needed.

Last, make sure it does not come back. Move to a structure that does not keep values in code.

Reverse the order and the old token and webhook stay alive the whole time you are cleaning up the code.

What the Current Configuration Does Differently

The public example in the build track excludes RCON and Discord integration and separates real input files from the repository and state from the working directory.

I made the game server need no secrets. RCON is never opened, so there is no RCON password. No Discord integration goes in, so there is no token and no webhook. Administrative access runs through OS Login, so no key is planted on the server. This configuration has no value to reissue or retire in the first place.

Inputs split the example file from the real file. The repository holds only terraform.tfvars.example, and every required value is a placeholder starting with replace-. terraform.tfvars and backend.hcl, which hold the real values, are excluded from Git from the start. The exclusion rule goes in before the file with real values is ever created.

I moved state to a remote backend. The state bucket controls access with IAM and has versioning on. With no state in the working directory, there is nothing to commit by accident. The fact that sensitive attributes land in state does not change, so it is handled through where the state is stored and who can reach it.

For a configuration that needs secrets, keep the value in Secret Manager and let the VM fetch it at runtime with its own service account. Pass only the secret resource identifier through Terraform. Reading a secret version through a Terraform data source and passing its value to another resource can put the value back into plan and state, so avoid that path. Give the VM service account permission to read only the required secret version.

Once a value enters code, it can be copied into plan, state, metadata, and disk. Cleanup then requires invalidating the value and checking whether every copy was removed, not merely editing the code.

References

• Sensitive data in Terraform state
• Terraform guidance on managing credentials
• Secret Manager overview
• Compute Engine metadata security considerations
• git filter-repo