Juntiger

VM Restart and Minecraft Readiness

Published

How to connect separate stages, from duplicate recovery events through Minecraft readiness, into one verification path.

The recovery function’s HTTP response, VM RUNNING, systemd active, and a connectable Minecraft server are different completion conditions. The original code polled for TERMINATED and RUNNING, but Compute operation errors, duplicate Pub/Sub delivery, and Minecraft protocol readiness were not connected into one end-to-end result.

The analysis also needed a trustworthy artifact. Terraform deployed a prebuilt ZIP rather than the source directory, and main.py inside that ZIP differed from the Python source beside it in the repository. Correct source does not explain production behavior when the deployed artifact is different.

From Source to Executable Artifact

The first-generation function embedded the hash of an existing ZIP in the object name and uploaded that archive to Cloud Storage.

resource "google_storage_bucket_object" "function_zip" {
  name = "restart-${filesha256("function.zip")}.zip"
  bucket = google_storage_bucket.function_source.name
  source = "function.zip"
}

resource "google_cloudfunctions_function" "restart_vm" {
  name = "restart-minecraft-vm"
  runtime="python310"
  source_archive_bucket = google_storage_bucket.function_source.name
  source_archive_object = google_storage_bucket_object.function_zip.name
  entry_point = "restart_vm"
}

filesha256 tells Terraform when the ZIP changes; it does not verify that the ZIP matches the Python source. Unless someone rebuilds the archive, a source-only change never appears in the deployment plan.

flowchart LR
  accTitle: Connection between recovery function source and deployment output
  accDescr: Python sources and dependencies become a deterministic ZIP. After its contents and hash are checked, Terraform uploads that exact artifact as a function revision.
  SRC[main.py] --> BUILD[clean build directory]
  REQ[locked requirements] --> BUILD
  BUILD --> ZIP[deterministic function.zip]
  ZIP --> CHECK[Contents/SHA-256 check]
  CHECK --> TF[Terraform plan]
  TF --> GCS[Cloud Storage object]
  GCS --> FN[function revision]
  FN --> RECORD[Revision and hash record]

Reproducible packaging starts from a clean directory instead of modifying an old ZIP. Copy only the required source and dependencies, normalize timestamps and file order, and then create the archive. Before Terraform receives the path, verify the file list, entry point, dependency versions, and SHA-256. After deployment, record the function revision and hash in structured logs or release metadata.

Terraform’s archive_file data source allows you to tie source changes to your plan.

data "archive_file" "restart_function" {
  type = "zip"
  source_dir = "${path.module}/function"
  output_path = "${path.module}/build/restart-function.zip"
}

resource "google_storage_bucket_object" "function_zip" {
  name = "restart-${data.archive_file.restart_function.output_sha256}.zip"
  bucket = google_storage_bucket.function_source.name
  source = data.archive_file.restart_function.output_path
}

When dependencies are vendored into the ZIP, package installation must be a separate, reproducible build step; archive_file does not install Python packages. If the runtime installs requirements.txt instead, the source archive should include locked versions.

Designing for Duplicate Delivery

Pub/Sub provides at-least-once delivery by default. If the function exits before recording its result, or if the acknowledgement is lost, the same message can be processed again. Separate Monitoring incidents can also refer to the same VM termination.

The most dangerous section is check-then-act.

Run A: Check VM status TERMINATED
Run B: Check VM status TERMINATED
Run A: Call instances.start
Run B: Call instances.start

The second start may only produce a harmless error, but logs and notifications will still report a failure. Other APIs may have real duplicate side effects. Deriving a stable API request ID from the incident ID lets retries represent the same operation.

from uuid import UUID

def request_id_for(incident_id: str) -> str:
    # In actual implementation, incident ID is reliably converted to namespace UUID.
    return str(UUID(incident_id))

operation = compute.instances().start(
    project=project,
    zone=zone,
    instance=instance,
    requestId=request_id_for(incident_id),
).execute()

Because a Monitoring incident ID may not already be a UUID, this is only a conceptual example. A real implementation can derive the same UUID from the same string, for example with UUID v5. Generating a random UUID on each attempt provides no deduplication.

The actions for each state are also made idempotent.

Current statusAction
RUNNINGSuccessful no-op; check the existing operation and readiness
PROVISIONING / STAGINGObserve the existing transition without calling start again
STOPPINGWait for TERMINATED for timeout
TERMINATEDStart with a stable requestId
Status query failedRecord the error class; abort immediately on permission errors

A database or distributed lock becomes worth considering when duplicate side effects are observed or recovery covers several targets concurrently. For one VM, state-aware no-ops, stable request IDs, and structured logs address the more immediate risks.

Waiting for the Compute Operation

The actual function’s startup code discarded the start response and only polled the VM status.

def start_vm_and_wait(compute, project, zone, instance):
    compute.instances().start(
        project=project, zone=zone, instance=instance
    ).execute()

    for _ in range(0, 120, 5):
        vm = compute.instances().get(
            project=project, zone=zone, instance=instance
        ).execute()
        if vm.get("status") == "RUNNING":
            return True
        time.sleep(5)
    return False

Compute Engine returns a zonal operation for a start request. Even when its status is DONE, the operation may contain an error. Waiting for the operation first separates an API-operation failure from a slow VM state transition.

def wait_for_zone_operation(compute, project, zone, operation_name):
    while True:
        operation = compute.zoneOperations().get(
            project=project,
            zone=zone,
            operation=operation_name,
        ).execute()
        if operation["status"] == "DONE":
            if "error" in operation:
                raise RuntimeError(operation["error"])
            return operation
        time.sleep(2)

Checking the instance after the operation completes is not redundant. The operation reports whether the API action finished and failed; the instance reports the target resource’s current state. A timeout record should include the operation name, last operation status, last instance status, and elapsed time.

stateDiagram-v2
  accTitle: Compute Engine and Minecraft Readiness
  accDescr: The VM changes from STOPPING to TERMINATED to PROVISIONING, STAGING, and RUNNING, and then becomes READY after preparing the OS, systemd, and JVM.
  [*] --> STOPPING: termination event
  STOPPING --> TERMINATED
  TERMINATED --> PROVISIONING: instances.start
  PROVISIONING --> STAGING
  STAGING --> RUNNING
  RUNNING --> OS_READY: boot complete
  OS_READY --> SERVICE_ACTIVE: systemd active
  SERVICE_ACTIVE --> LISTENING: TCP or UDP bound
  LISTENING --> READY: external status succeeds
  SERVICE_ACTIVE --> FAILED: restart loop or initialization error

Readiness after VM RUNNING

The startup script runs after the OS boots. Compute Engine can mark the VM as RUNNING even if package installation, an external download, backup restoration, permission setup, or unit creation later fails. Likewise, systemd active proves that the Java process is alive, not that world and plugin initialization is complete.

sequenceDiagram
  accTitle: Layered Minecraft readiness checks
  accDescr: After the function confirms the VM is running, readiness proceeds through the guest OS, systemd, local listener, server initialization, and an external Minecraft protocol probe.
  participant Fn as recovery function
  participant API as Compute API
  participant OS as guest OS
  participant SD as systemd
  participant MC as Minecraft
  participant Ext as external probe
  Fn->>API: Check operation DONE and VM RUNNING
  API-->>Fn: RUNNING
  OS->>SD: Start minecraft.service
  SD->>MC: Run Java
  MC->>MC: Load world and plugins
  MC-->>OS: Done message and local listener
  Ext->>MC: Request status through public IP
  MC-->>Ext: protocol response
  Ext-->>Fn: READY recording

Inspections inside and outside the VM detect different failures.

systemctl is-active --quiet minecraft.service
journalctl -u minecraft.service -b --no-pager | tail -n 100
ss -lnt '( sport = :25565 )'

A local TCP listener proves that the JVM opened the port, but says nothing about the public IP, VPC firewall, or route. Geyser’s Bedrock UDP listener can be checked separately with ss -lnu. The final probe should run outside the VM: send a Java server-list ping and a Bedrock status request, then verify the returned server identity or version.

Matching only the Done (...) line couples readiness to a particular version and log format. Where possible, use the Minecraft protocol probe as the final condition and keep logs for diagnosis. Success notifications should distinguish VM_RUNNING from MINECRAFT_READY so players are not given a false expectation that they can already connect.

Migrating to Cloud Run Functions

Second-generation functions are now Cloud Run functions. Terraform can declare build and service settings separately in google_cloudfunctions2_function, along with a Pub/Sub Eventarc trigger.

resource "google_cloudfunctions2_function" "restart_vm" {
  name = "restart-minecraft-vm"
  location = var.region

  build_config {
    runtime="python312"
    entry_point = "restart_vm"
    source {
      storage_source {
        bucket = google_storage_bucket.function_source.name
        object = google_storage_bucket_object.function_zip.name
      }
    }
  }

  service_config {
    available_memory = "256M"
    timeout_seconds = 300
    service_account_email = google_service_account.restart.email
    max_instance_count = 1
    max_instance_request_concurrency = 1
  }

  event_trigger {
    trigger_region = var.region
    event_type = "google.cloud.pubsub.topic.v1.messagePublished"
    pubsub_topic = google_pubsub_topic.vm_restart.id
    retry_policy = "RETRY_POLICY_RETRY"
  }
}

max_instance_count = 1 does not by itself serialize handler execution because one Cloud Run instance can process concurrent requests. The example also limits per-instance concurrency to one, but still cannot guarantee exactly-once execution. Pub/Sub redelivery and overlap between revisions mean the handler must remain idempotent. A 300-second timeout also cannot guarantee that Minecraft will be ready in time. For long recovery windows, it is clearer to persist recovery state and let a separate probe update readiness than to keep one function request open.

Permissions and Structured Logs

The production function used the default service account with project-wide roles/compute.instanceAdmin.v1 and roles/logging.viewer. A replacement should use a dedicated service account whose permissions are derived from the API calls it actually makes:

  • Read instance status
  • Start the target instance
  • Read the zonal operation
  • Write function logs
  • Access a specific Secret Manager version if a webhook is required

Exclude Logs Viewer unless the function queries activity logs directly. Permission tests should include rejection cases: starting the target VM should succeed, while starting another VM, deleting an instance, changing a disk, or modifying a firewall should fail.

The following fields are common to all logs.

{
  "recovery_id": "example-incident-id",
  "event_state": "open",
  "instance": "minecraft-server",
  "phase": "WAITING_FOR_READY",
  "operation": "operation-name",
  "elapsed_ms": 84231,
  "result": "pending"
}

The event timeline should include the deployed revision and source hash. That connects observed production behavior to the exact code that produced it.

timeline
  accTitle: Correlation time base of one recovery event
  accDescr: One incident ID links message receipt, the Compute operation, VM and systemd state, protocol readiness, and the final notification.
  title Records linked by recovery_id
  Incident : alert opened
  Function : message received : source hash recorded
  Compute : start operation created : operation DONE
  Instance : RUNNING
  Guest : startup script exited : systemd active
  Minecraft : protocol READY
  Notify : recovery-complete notification

References

Share

Share

Image preview