GCP ACE Drill: GKE Pod Scheduling - The Preemptible Node Pool Dynamics

While preparing for the Google Cloud Associate Cloud Engineer (ACE) exam, many candidates struggle with GKE troubleshooting scenarios, especially when Pods remain in Pending status. In the real world, this is fundamentally about understanding Kubernetes scheduler behavior, resource allocation, and the trade-offs of using Preemptible nodes. Let’s drill into a simulated scenario.

The Scenario

#

You work for StreamWave Gaming, a fast-growing mobile game studio that recently migrated their backend services to Google Kubernetes Engine (GKE). To optimize costs during development, the DevOps team configured a single-node pool using Preemptible VMs for their non-production workloads.

You’ve just deployed a new microservice using a Kubernetes Deployment with 2 replicas. After waiting a few minutes, you run kubectl get pods and notice one Pod is Running, but the other remains stuck in Pending status.

Key Requirements

#

Identify the most likely root cause for why the second Pod cannot be scheduled, considering the cluster is using a single Preemptible node pool.

The Options

#

• A) The pending Pod’s resource requests are too large to fit on a single node of the cluster.
• B) Too many Pods are already running in the cluster, and there are not enough resources left to schedule the pending Pod.
• C) The node pool is configured with a service account that does not have permission to pull the container image used by the pending Pod.
• D) The pending Pod was originally scheduled on a node that has been preempted between the creation of the Deployment and your verification of the Pods’ status. It is currently being rescheduled on a new node.

Correct Answer

#

Option B.

The Architect’s Analysis

#

Correct Answer

#

Option B – Too many Pods are already running in the cluster, and there are not enough resources left to schedule the pending Pod.

Step-by-Step Winning Logic

#

The Kubernetes scheduler places Pods based on available CPU and memory resources. In a cluster with a single Preemptible node pool, if existing Pods consume most of the node’s capacity, the scheduler cannot place additional Pods—they remain in Pending status.

Why this matters for ACE:

• The exam tests your ability to diagnose cluster capacity issues using kubectl describe pod and understand resource requests/limits.
• Key indicator: When you see “Insufficient CPU” or “Insufficient memory” in Pod events, it’s a scheduling resource constraint, not a configuration error.

Google Cloud Best Practice:

• Use GKE Cluster Autoscaler to automatically add nodes when Pods are pending due to resource shortages.
• For production, combine On-Demand nodes (for critical workloads) with Preemptible nodes (for batch jobs).

The Traps (Distractor Analysis)

#

Why not Option A?

#

“The pending Pod’s resource requests are too large to fit on a single node.”

• Trap Logic: This would cause the Pod to remain permanently Pending with an event like “0/1 nodes available: pod exceeds node capacity.”
• Reality Check: If one Pod is already Running, it means the Pod spec does fit on the node. The issue is cumulative resource usage, not individual Pod size.
• Exam Tip: Always check if any Pod from the Deployment succeeded. If yes, the Pod spec is valid.

Why not Option C?

#

“The node pool’s service account lacks permission to pull the container image.”

• Trap Logic: Image pull errors manifest as ImagePullBackOff or ErrImagePull status, not Pending.
• ACE Fundamentals:
  • Pending = Scheduler cannot find a node.
  • ImagePullBackOff = Kubelet cannot fetch the container image.
• IAM Reality: By default, GKE nodes use the Compute Engine default service account, which has roles/storage.objectViewer for GCR images. Image pull failures are rare unless using private registries with misconfigured Image Pull Secrets.

Why not Option D?

#

“The pending Pod was originally scheduled on a node that has been preempted and is being rescheduled.”

• Trap Logic: This is a plausible real-world scenario but not the most likely cause given the timing.
• Technical Reality:
  • Preemptible VMs give a 30-second termination notice.
  • When a node is preempted, all Pods enter Terminating status, then become Pending for rescheduling.
  • However, the question states “after a few minutes”—if preemption had occurred, you’d likely see both Pods in Pending (not just one), or events showing node NotReady.
• Exam Strategy: Google’s ACE exam prefers the simpler explanation (Occam’s Razor). Resource exhaustion is more common than mid-deployment preemption.

The Architect Blueprint

#

Diagnostic Workflow for Pending Pods

#

Diagram Note: This flowchart shows the systematic approach to diagnose why a Pod remains Pending—start with kubectl describe pod, analyze Events, then investigate nodes.

Real-World Practitioner Insight

#

Exam Rule

#

For the ACE exam, remember:

• Pending Pod + Single node pool + “a few minutes” = Most likely resource exhaustion (Option B).
• Always use kubectl describe pod <pod-name> to see the Events section—it will explicitly state the scheduling failure reason.

Key Command:

kubectl describe pod <pod-name> | grep -A 10 Events

Look for:

Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/1 nodes are available: 1 Insufficient cpu.

Real World

#

In production at StreamWave Gaming, we’d take this approach:

1. Immediate Fix:

   • Manually scale the node pool: gcloud container clusters resize <cluster> --node-pool <pool-name> --num-nodes 2
   • Or delete non-critical Pods to free resources.

2. Long-Term Solution:

   • Enable GKE Cluster Autoscaler:

     gcloud container clusters update <cluster> \
       --enable-autoscaling \
       --min-nodes 1 \
       --max-nodes 5 \
       --node-pool <pool-name>

   • Use Pod Disruption Budgets (PDBs) to ensure at least one replica stays running during node disruptions.
   • Migrate critical services to a separate node pool with On-Demand VMs (not Preemptible).

3. Monitoring:

   • Set up Cloud Monitoring alerts for:
     • kubernetes.io/pod/pending metric
     • Node CPU/memory utilization > 80%
   • Use GKE Workload Metrics to right-size resource requests.

Cost Trade-off:

• Preemptible nodes: ~$0.01/hour per vCPU
• On-Demand nodes: ~$0.033/hour per vCPU
• Autoscaler overhead: ~5% cost increase but prevents over-provisioning.

Key Takeaways for ACE Candidates

#