⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 85 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #74 (L2) · Advanced Scenarios

Q: NodePort service not reachable from outside.

Check firewall/security group rules allow the NodePort (30000-32767) range.

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"A <strong>NodePort service not reachable from outside</strong> almost always fails in one of four places: the Service has no endpoints, kube-proxy has not programmed the node port, a firewall or cloud security group blocks the port, or you are using an address that is not routable from your client. Test from the inside out: pod, ClusterIP, node itself, then external client. Interviewers ask this because it tests whether you understand that a NodePort is only a port opened on every node, forwarded by kube-proxy rules to pod endpoints."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: A <strong>NodePort service not reachable from outside</strong> almost always fails in one of four places: the Service has no endpoints, kube-proxy has not programmed the node port, a firewall or cloud security group blocks the port, or you are using an address that is not routable from your client. Test from the inside out: pod, ClusterIP, node itself, then external client. Interviewers ask this because it tests whether you understand that a NodePort is only a port opened on every node, forwarded by kube-proxy rules to pod endpoints.

1️⃣

Symptom and the Four Layers to Check

Reachability is a chain; find the first broken link instead of changing things at random.

  • Layer 1, Service and endpoints: does the Service select ready pods, and does targetPort match the container port?
  • Layer 2, node dataplane: has kube-proxy (iptables, IPVS or nftables mode) created rules for the node port on that node?
  • Layer 3, network path: does a host firewall, cloud security group, NACL or on-prem firewall allow the port from your client to the node IP?
  • Layer 4, addressing and policy: is the node IP reachable from the client, and does externalTrafficPolicy or a NetworkPolicy change which pods answer?
  • What the rule does: kube-proxy matches traffic to any node IP on the node port and rewrites it to a pod endpoint, which may be on another node; the Service object itself runs no process.
2️⃣

Hands-on: NodePort Service Not Reachable From Outside

Run these in order. The first one that fails is your root cause.

kubectl get svc my-app -n prod -o wide          # TYPE NodePort, PORT(S) 80:30080/TCP
kubectl get endpointslices -n prod -l kubernetes.io/service-name=my-app
kubectl get pods -n prod -l app=my-app -o wide   # note node names and IPs

# From inside the cluster
kubectl run tmp --rm -it --image=curlimages/curl --restart=Never -- \
  curl -sv http://my-app.prod.svc.cluster.local:80

# On a node (SSH or SSM)
curl -sv http://127.0.0.1:30080
sudo iptables -t nat -S KUBE-NODEPORTS | grep 30080    # iptables mode
# From your laptop
curl -sv --max-time 5 http://<node-external-ip>:30080
  • Endpoints: an empty ENDPOINTS list means a selector or readiness problem, not a network problem.
  • Inside the cluster: if the ClusterIP works from a debug pod but the NodePort does not, the problem is outside the Service object.
  • On the node: curl to localhost:<nodePort> on the node isolates host rules from the external network path.
Advertisement
3️⃣

Root Causes and Fixes

Each failure has a signature that points to the fix.

  • Empty endpoints: selector does not match pod labels, or pods fail readiness. Fix labels or the probe; the NodePort cannot work with no backends.
  • Wrong targetPort: the Service forwards to a port the container does not listen on, giving connection refused or reset. Align targetPort with the container port or its name.
  • Firewall or security group: a timeout (no response) points here. On AWS, the node security group must allow inbound TCP on the node port; the default range is 30000-32767 unless changed with --service-node-port-range. a timeout points to a firewall or routing problem, while connection refused points to a reachable host with nothing listening on the target, which is typically a targetPort or pod issue.
  • Private node IP: using an internal address from the internet times out. Use the node's public address, a bastion, or put a LoadBalancer or Ingress in front.
  • externalTrafficPolicy: Local: only nodes that run a ready pod answer; other nodes drop the traffic. Query a node that hosts a pod, or use Cluster and accept source IP masquerading.
4️⃣

Verification and Special Environments

Confirm the fix from the same place that failed.

# Which mode is kube-proxy running, and are there rules for the port?
kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep -E 'mode|nodePortAddresses'
sudo ipvsadm -Ln | grep 30080          # ipvs mode
# AWS: is the port open on the node security group?
aws ec2 describe-security-groups --group-ids sg-0123456789abcdef0 \
  --query 'SecurityGroups[].IpPermissions[?FromPort<=`30080` && ToPort>=`30080`]'
  • Retest externally: a successful curl -sv from the original client is the only real proof; node-local success is not enough.
  • Local clusters: on minikube, kind or Docker Desktop the node IP is inside a VM or Docker network. Use minikube service my-app --url, extra port mappings in kind, or kubectl port-forward.
  • kube-proxy health: kubectl -n kube-system get pods -l k8s-app=kube-proxy and its logs reveal a crashed or misconfigured proxy; some CNIs replace kube-proxy entirely, so rules look different.
  • NetworkPolicy: a policy selecting the pods can block the traffic; remember NodePort traffic may arrive with the node's IP after SNAT.
5️⃣

Production Practice: Prefer Load Balancer or Ingress

NodePort is a building block, not an internet-facing design.

  • Front with a load balancer: use a LoadBalancer Service or Ingress/Gateway so clients do not depend on individual node IPs that change with autoscaling.
  • Restrict exposure: limit the source CIDR in security groups and use nodePortAddresses on kube-proxy if only specific interfaces should listen.
  • Pin the port deliberately: set nodePort explicitly only when an external system needs it, and document it to avoid collisions.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Test the NodePort with curl on the node itself first; success there with failure from your laptop proves the problem is the network path, and you can stop debugging Kubernetes."
⚡ 60-Second Elevator Pitch Talking Points
  • A NodePort opens one port on every node and kube-proxy forwards it to the Service endpoints.
  • I check endpoints, ClusterIP from a debug pod, curl on the node, then the external path and security groups.
  • In production I avoid raw NodePort for public traffic and use a load balancer or Ingress with restricted source ranges.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Debugging firewalls before checking endpoints A Service with no ready endpoints fails no matter how open the network is.
  2. Confusing port, targetPort and nodePort Traffic is forwarded to targetPort on the pod, so a mismatch gives refused connections that look like network issues.
  3. Using the node's private IP from outside the VPC That address is not routable from the internet, so every test times out regardless of cluster health.
  4. Ignoring externalTrafficPolicy: Local Nodes without a local pod drop the traffic, so the Service works on some nodes and not others.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →