⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All FinOps & System Design Interview Questions Scenario 8 of 44 in FinOps & System Design
Staff Platform Architect System Design API Gateway & Edge Infrastructure System Design
🎯 Target Role / Context: Principal Platform Architect / Staff SRE System Design Interview

Q: Your enterprise serves 400 microservices through a fragmented collection of individual cloud load balancers and bespoke authentication libraries. Security teams lack visibility, and developers spend 20% of their time implementing boilerplate auth, CORS, and logging. How do you design and deploy a centralized, highly available Global API Gateway architecture?

Architectural design for a resilient global API Gateway tier handling millions of requests per second with distributed authentication (OAuth2/OIDC), Layer 7 routing, rate limiting, and mTLS mesh termination.

#System Design #API Gateway #Envoy #Kong #WAF #OAuth2 #Traffic Management
🎙️ Candidate Opening & Architectural Context
"Scattered ingress points introduce security vulnerabilities and inconsistent policy enforcement. We designed an enterprise API Gateway platform powered by Envoy Proxy and Kong Gateway, establishing a unified edge perimeter for authentication, traffic shaping, and observability."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's FinOps Certified Practitioner (FOCP) Program covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Scattered ingress points introduce security vulnerabilities and inconsistent policy enforcement. We designed an enterprise API Gateway platform powered by Envoy Proxy and Kong Gateway, establishing a unified edge perimeter for authentication, traffic shaping, and observability.

1️⃣

Architect Edge Ingress Topology: Global Anycast to Regional Gateway Clusters

Route global client traffic to the closest regional API Gateway instance:

  • Global Anycast DNS / CDN: Incoming traffic hits Anycast edge Points of Presence (Cloudflare / CloudFront) for DDoS mitigation, TLS termination, and static asset caching.
  • Regional Gateway Fleet: Traffic forwards over private cloud backbones to regional Kubernetes clusters hosting autoscaled API Gateway pods (Envoy/Kong) across 3 AZs.
Pro Tip: Terminating TLS at edge PoPs reduces TCP handshake latency by up to 70% for international clients.
2️⃣

Implement High-Speed Stateless Authentication & Token Validation

Offload authentication completely from downstream microservices:

  • Local JWT Verification: Gateway downloads public JSON Web Key Sets (JWKS) from corporate IdP (Okta/Entra ID) and caches them in memory.
  • Stateless Token Validation: Cryptographically validates RSA256 signature, expiry, and scopes locally on the Gateway in <0.2ms without making network calls to the IdP.
  • Internal Identity Injection: Gateway strips external auth headers and injects trusted internal headers (X-User-Id, X-Tenant-Id, X-User-Roles) into downstream requests.
Pro Tip: In-memory JWKS validation allows the gateway to verify hundreds of thousands of requests per second without overwhelming the central Identity Provider.
Advertisement
3️⃣

Configure Dynamic Layer 7 Routing & Canary Traffic Splitting

Enable progressive delivery and fine-grained request steering:

  • Declarative CRDs: Developers define routes using Kubernetes Gateway API (HTTPRoute), configuring path prefix matching, header rewrites, and timeout limits.
  • Canary Traffic Shifts: Supports percentage-based traffic splits (e.g., 95% v1, 5% v2) and header-based routing (e.g., routing X-Beta-Tester: true directly to experimental staging pods).
Pro Tip: Kubernetes Gateway API standardizes route declarations, preventing vendor lock-in to proprietary gateway ingress annotations.
4️⃣

Centralize Observability, Distributed Tracing, and Audit Logging

Provide complete visibility across all incoming API traffic:

  • W3C Trace Injection: Automatically injects W3C traceparent headers on untracked requests and exports OpenTelemetry spans to Grafana Tempo.
  • Access Log Streaming: Streams structured JSON access logs containing status codes, response latencies, and client geo-data directly to ClickHouse for real-time security analysis.
Pro Tip: A single centralized access log pipeline enables instant cross-organizational threat detection and latency debugging.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"An enterprise API Gateway centralizes edge concerns—stateless JWT authentication, Layer 7 canary routing, rate limiting, and telemetry—relieving hundreds of microservices from maintaining boilerplate logic."
⚡ 60-Second Elevator Pitch Talking Points
  • Terminate edge TLS at global Anycast PoPs to maximize client connection speeds.
  • Validate JWT tokens locally using cached JWKS keys in <0.2ms with zero IdP network calls.
  • Empower developers to declare routes and canary splits using Kubernetes Gateway API CRDs.
  • Inject W3C trace context and stream real-time access logs to ClickHouse for complete observability.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring System Design Enterprise 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
  2. Setting target deregistration delay to zero instead of allowing connection draining when configuring System Design Enterprise Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
  3. Failing to query CloudWatch ELB metrics (HTTPCode_ELB_5XX vs HTTPCode_Target_5XX) during load balancer outages when configuring System Design Enterprise Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime.
  4. Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring System Design Enterprise Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.
Advertisement
Want more FinOps & System Design scenarios?
Explore our complete collection of scenario-based FinOps & System Design interview runbooks.
Browse All FinOps & System Design Questions →