Q: Netflix runs multi-cloud. Describe your approach to cross-cloud routing, IAM, and secret syncing.
Engineering blueprint for designing an enterprise multi-cloud substrate across AWS and GCP covering private BGP interconnects, federated OIDC workload identities, and continuous secret synchronization.
#Multi-Cloud #AWS #GCP #BGP #Workload Identity #SPIFFE #Vault #System Design
🎙️ Candidate Opening & Architectural Context
"True enterprise multi-cloud is not about running the exact same Kubernetes YAML everywhere. It is about building a unified abstraction across three foundational pillars: low-latency private network routing without public internet traversal, cryptographic workload identity federation without static API keys, and centralized secret lifecycle reconciliation."
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Private Cross-Cloud Routing (AWS Direct Connect + GCP Cloud Interconnect)
Never route internal multi-cloud traffic over the public internet with IPsec tunnels due to MTU fragmentation and jitter:
- Colocation Interconnect Fabric: Utilize Equinix Fabric or Megaport to cross-connect AWS Direct Connect (DX) and GCP Dedicated Interconnect with redundant 10Gbps cross-connects.
- BGP Routing over Transit Gateway: Establish eBGP sessions between AWS Transit Gateway (TGW) and GCP Cloud Router with non-overlapping RFC 1918 CIDRs (e.g., AWS `10.100.0.0/16`, GCP `10.200.0.0/16`).
- Bidirectional Forwarding Detection (BFD): Configure BFD with 300ms transmit/receive intervals to detect link failures and trigger sub-second route convergence.
2️⃣
Zero-Static-Key IAM Federation (SPIFFE/SPIRE & Workload Identity)
Eliminate long-lived cloud credentials across providers using OIDC Workload Identity Federation:
# AWS Trust Policy allowing GCP Service Account to assume AWS IAM Role:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Federated": "accounts.google.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"accounts.google.com:aud": "https://iam.googleapis.com/projects/12345/locations/global/workloadIdentityPools/gcp-pool/providers/gcp-provider"
}
}
}]
}- A workload running on Google Cloud GKE acquires an ephemeral short-lived Google OIDC token, exchanges it at AWS STS for temporary IAM credentials, and accesses Amazon S3 without a single hardcoded secret.
3️⃣
Centralized Secret Syncing Engine (HashiCorp Vault + External Secrets Operator)
Treat secrets as a single control plane replicated securely across clouds:
- Vault Primary-Replica Cluster: Multi-region Vault cluster with KMS auto-unseal (AWS KMS in us-east-1, GCP Cloud KMS in us-central1).
- Kubernetes External Secrets Operator (ESO): Deployed in both AWS EKS and GCP GKE clusters, fetching credentials locally from Vault and instantiating native Kubernetes `v1/Secret` objects.
- Automated Rotation: Database credentials (RDS and Cloud SQL) are dynamically generated with 1-hour TTLs via Vault database secrets engines.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Multi-cloud operational resilience requires eliminating static trust: private layer-3 BGP fabrics replace public tunnels, OIDC workload identity replaces static API keys, and declarative secret operators replace manual syncing."
⚡ 60-Second Elevator Pitch Talking Points
- We build a private layer-3 backbone using AWS Direct Connect and GCP Cloud Interconnect over an Equinix Cloud Exchange fabric with redundant eBGP sessions and BFD for sub-second failover.
- For IAM, we forbid static IAM access keys. We deploy SPIFFE/SPIRE and OIDC Workload Identity Federation so workloads exchange cryptographic JWTs across clouds to assume native temporary IAM roles.
- For secrets, we maintain a federated HashiCorp Vault cluster with cloud-native KMS auto-unseal, synced via External Secrets Operator into local Kubernetes clusters.
- All operational credentials feature automated rotation with 1-hour dynamic TTLs to minimize credential blast radius.
Advertisement