Automatic Recovery for a Preemptible VM on GCP

The server used Compute Engine with preemptible = true and automatic_restart = false. Legacy preemptible VMs could be reclaimed by the platform and never ran for more than 24 hours. Restarting the same VM required both the shutdown observer and the authority to call the Compute API to live outside that VM.

The recovery control path was six steps.

sequenceDiagram
  accTitle: Recovery sequence from preemption log to Minecraft restart
  accDescr: Compute Engine shutdown activity log passes through Logging, Monitoring, and Pub/Sub to reach the function, and after the function starts the VM, systemd runs Minecraft.
  participant GCE as Compute Engine
  participant Log as Cloud Logging
  participant Mon as Cloud Monitoring
  participant PS as Pub/Sub
  participant Fn as Cloud Function
  participant API as Compute API
  participant SD as systemd
  GCE->>Log: preempted activity log
  Log->>Mon: counter metric point
  Mon->>PS: incident open
  PS->>Fn: alert payload
  Fn->>API: instances.get
  loop until TERMINATED
    Fn->>API: instances.get
  end
  Fn->>API: instances.start
  loop until RUNNING
    Fn->>API: instances.get
  end
  GCE->>SD: OS boot
  SD->>SD: Start Minecraft unit

The Cloud Function did not handle world files or the JVM. It only inspected VM state and requested a start. After boot, systemd owned the process lifecycle. This separation meant the function needed neither SSH keys nor access to the Minecraft directory.

From shutdown log to recovery signal

The log-based metric used in production matched both preemption and ordinary stop methods. The example below preserves that structure while replacing the original identifiers.

resource "google_logging_metric" "vm_stopped" {
  name = "minecraft-vm-stopped"
  description = "Detect preemption or stop activity"
  filter = <<-EOT
    resource.type="gce_instance"
    AND (
      protoPayload.methodName="compute.instances.preempted"
      OR protoPayload.methodName="v1.compute.instances.stop"
    )
  EOT

  metric_descriptor {
    metric_kind = "DELTA"
    value_type = "INT64"
  }
}

This filter has two problems. First, a deliberate maintenance stop by an operator also becomes a recovery event. Second, the filter is not scoped to a specific instance ID or zone. If another VM is added to the project, stopping it could also trigger the Minecraft restart function.

Detection rules should instead be derived from the actions the system is allowed to take.

flowchart TD
  accTitle: Recovery decision by termination cause
  accDescr: Only a platform preemption starts the VM automatically. Operator stops, deletion, and unknown causes require a different response.
  E[Termination activity log] --> T{Expected instance and zone?}
  T -- No --> IGNORE[Record and ignore]
  T -- Yes --> C{Termination cause}
  C -- Platform preemption --> START[Start automatically]
  C -- Operator stop --> HOLD[Leave stopped]
  C -- Delete or Terraform replacement --> BLOCK[Do not start]
  C -- Unknown --> REVIEW[Alert and review]

Because an instance name can be reused, the numeric instance ID from the log is the safer primary identifier; the project and zone should be checked as well. If Terraform replaces the VM, the metric filter must receive the new ID. Filtering by name avoids that dependency, but then operational policy must define what a newly created resource with the same name is allowed to mean.

What the Two 60-Second Settings Mean

Monitoring conditions included alignment_period = "60s" and duration = "60s".

condition_threshold {
  filter = (
    "resource.type=\"gce_instance\" AND "
    "metric.type=\"logging.googleapis.com/user/minecraft-vm-stopped\""
  )
  comparison = "COMPARISON_GT"
  threshold_value = 0
  duration = "60s"

  aggregates {
    alignment_period = "60s"
    per_series_aligner = "ALIGN_COUNT"
  }
}

The alignment period groups the raw time series into 60-second windows. The ALIGN_COUNT used at the time counts the number of points in a window; it does not add the values of a log-based counter. If the metric records a point whose value is 0 in a window with no events, ALIGN_COUNT can return 1—one point—instead of 0. Combined with threshold > 0, that creates a false-positive path in which the condition is true even though no shutdown occurred.

If the question is “How many matching shutdown logs occurred in 60 seconds?”, the point values need to be added with ALIGN_SUM.

aggregates {
  alignment_period = "60s"
  per_series_aligner = "ALIGN_SUM"
}

The duration controls how long the aligned condition must remain true before an incident opens. Alignment period and duration cannot simply be added—or treated as overlapping—to mean “run the function exactly 60 seconds later.” After changing the policy, synthetic logs should cover a zero-event window, one real termination, and multiple terminations, with the incident result checked for each case.

timeline
  accTitle: Time accumulated between shutdown and automatic recovery
  accDescr: Recovery latency includes log ingestion, metric alignment, alert evaluation, Pub/Sub delivery, function polling, and VM boot.
  title Stages that make up recovery latency
  Termination : Compute activity log created
  Logging : Log ingestion and metric point creation
  Monitoring : 60-second alignment : condition-duration evaluation
  Pub/Sub : Incident message publication and delivery
  Function : termination polling : start request : running-state polling
  VM : OS boot : systemd : Minecraft readiness

Measuring real recovery time requires recording the following timestamps under the same recovery_id.

time	Measurement location	Noticeable Delay
`event_time`	Compute activity log	Preemption baseline
`incident_opened_at`	Monitoring payload	Detection/evaluation delay
`function_received_at`	First function log	Pub/Sub delivery and cold-start delay
`start_requested_at`	Compute operation	Function decision delay
`vm_running_at`	Compute status query	Provisioning delay
`minecraft_ready_at`	External protocol probe	OS, systemd, and JVM startup delay

Configuration values alone cannot reveal which stage was slow. Discord timestamps are useful to players, but they are a poor source of operational truth because webhook failures and delivery delays do not change the recovery state itself.

Delivery path from Monitoring to Pub/Sub

The Monitoring notification channel points to the Pub/Sub topic, and the Monitoring service agent Topic publish permission was granted.

resource "google_monitoring_notification_channel" "restart" {
  display_name = "Minecraft VM restart"
  type="pubsub"
  labels = {
    topic = google_pubsub_topic.vm_restart.id
  }
}

resource "google_pubsub_topic_iam_member" "monitoring_publish" {
  topic = google_pubsub_topic.vm_restart.name
  role = "roles/pubsub.publisher"
  member = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-monitoring-notification.iam.gserviceaccount.com"
}

Permission to publish to the topic and permission for the function to process its messages are separate. The notification channel lets Monitoring publish, while the function trigger delivers the Pub/Sub message to the runtime. One service account does not need to own both roles.

Pub/Sub provides at-least-once delivery. An open notification for the same incident can arrive more than once, and function execution can be retried. Closing an incident also creates a separate message. The production function even attempted a start for state != "open" messages when the VM was TERMINATED, so a close message was not merely a success notification.

def handle_alert(payload):
    incident = payload.get("incident", {})
    state = incident.get("state", "unknown")

    if state != "open":
        status = get_instance_status()
        if status == "TERMINATED":
            start_vm_and_wait()
        return

Restarting on close can compensate for a missed open event, but it also creates another way to undo an intentional manual stop. A clearer policy checks the incident_id, state transition, termination cause, and target VM together, and uses close messages only to close existing recovery records.

State Polling Inside the Function

The VM could still be STOPPING when the function received the shutdown event. It polled every five seconds for up to 45 seconds for TERMINATED, called start, and then waited up to 120 seconds for RUNNING.

def wait_until_terminated(compute, project, zone, instance,
                          timeout=45, interval=5):
    for _ in range(0, timeout, interval):
        vm = compute.instances().get(
            project=project, zone=zone, instance=instance
        ).execute()
        if vm.get("status") == "TERMINATED":
            return True
        time.sleep(interval)
    return False

The 45- and 120-second values are limits in the operational code, not guarantees from Compute Engine. Because the loop logged API errors and kept waiting, permission failures, transient errors, and an ordinary slow transition could all collapse into the same timeout. Retryable HTTP errors should be separated from authentication or authorization failures that can stop immediately, and the last known status should be recorded with the error code.

instances.start returns a long-running operation. An HTTP-successful request does not mean that the VM is running. Persisting the operation ID, checking DONE and error, and only then continuing to instance-status polling preserves the reason when startup fails.

Applying the Design to Current Spot VMs

Current Spot VMs do not have the old 24-hour limit. Compute Engine can still preempt them at any time, however, and provides no SLA, automatic restart, or live migration. After a best-effort termination notice, the VM is stopped or deleted. To reuse the same VM and disk, set the termination action explicitly to STOP.

scheduling {
  provisioning_model = "SPOT"
  instance_termination_action = "STOP"
  automatic_restart = false
  on_host_maintenance = "TERMINATE"
}

For a single Minecraft server, restarting the same VM makes it easier to preserve the static IP and disk state. A managed instance group or a regular VM is a better fit when:

The interruption time is short and must be restored immediately even during spot capacity shortage.
The same server can be recreated from an immutable image.
The world state is separated into separate storage and restoration procedures.
The complexity of automatic healing and zone distribution can be tolerated.

Even a correct start request can keep failing when the zone has no Spot capacity. A separate policy must decide whether to retry the same machine type, create a VM in another zone, or switch to a regular VM, including whether the world disk and static IP can move with it.

Least Privilege and Failure Paths

The production function received project-wide roles/compute.instanceAdmin.v1 and roles/logging.viewer. If it only reads the Pub/Sub payload and never queries logs, Logging Viewer is unnecessary. Instance Admin also permits far more than starting a VM.

The required operations are reading the target instance, requesting a start, and reading the zone operation. A dedicated service account and custom role can cover those permissions, while a fixed instance-and-zone allowlist in the code provides a second boundary when IAM must be granted above the individual VM.

The automatic path should also have a separate manual recovery procedure.

Find recovery_id in incident and function execution logs.
Check the current status and recent operation errors of the target VM.
If the event was a preemption and the VM is TERMINATED, start it explicitly.
Check the serial console, startup-script, and systemd logs.
Record recovery as complete only after an external Minecraft status probe succeeds.