⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 135 of 179 in AWS & Cloud Architecture
Lead Platform Engineer GCP & Cloud Multi-Cluster GitOps GitOps Architecture
🎯 Target Role / Context: Lead Platform Engineer & GitOps Governance Scenario

Q: How do you maintain identical security configurations, RBAC roles, and network policies across 40 GKE clusters spread across Google Cloud and on-premises environments without manual drift? Walk me through Anthos Config Sync.

Deploying declarative multi-cluster governance across hybrid GKE and on-premises clusters using Anthos Config Sync (RootSync/RepoSync) and Gatekeeper Policy Controller.

#GCP #Anthos #Config Sync #GitOps #Policy Controller #Kubernetes
🎙️ Candidate Opening & Architectural Context
"Our financial institution managed 40 GKE clusters across three GCP regions and two private datacenters. Manual kubectl applies and divergent cluster configs created compliance audit failures."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Our financial institution managed 40 GKE clusters across three GCP regions and two private datacenters. Manual kubectl applies and divergent cluster configs created compliance audit failures.

1️⃣

Register Clusters to Anthos Fleet (GKE Hub)

Establish a unified administrative plane across all distributed clusters:

  • Fleet Membership: Registered all cloud and on-prem clusters to the central Google Cloud Project Fleet using gcloud container fleet memberships register.
  • Connect Gateway: Enabled Google Connect Gateway to provide unified IAM-based administrative access to all clusters through Google Cloud.
Pro Tip: Fleets establish a shared identity domain (sameness) across multiple clusters for namespaces, services, and workloads.
2️⃣

Configure Config Sync with RootSync and RepoSync CRDs

Implement hierarchical multi-tenant GitOps synchronization:

  • RootSync (Platform Team): Configured central RootSync pointing to the cluster-admin git repository enforcing cluster-wide CRDs, NetworkPolicies, and RBAC.
  • RepoSync (Product Squads): Configured namespace-scoped RepoSync allowing individual product teams to manage their own application deployments in isolated subdirectories.
  • Authentication: Authenticated Config Sync to GitHub Enterprise via Workload Identity and GitHub App tokens.
Pro Tip: RepoSync enforces strict namespace boundaries: product squads cannot declare cluster-scoped resources or alter other teams' configs.
Advertisement
3️⃣

Enforce Declarative Policy Controller (OPA Gatekeeper)

Block non-compliant resources at the admission level:

  • ConstraintTemplates: Deployed Google-curated policy templates (e.g. K8sRequiredLabels, K8sDenyPrivileged, K8sRestrictedPorts).
  • Enforcement Actions: Configured enforcementAction: deny on production clusters and dryrun on development clusters for progressive rollout.
  • Audit Dashboard: Monitored policy compliance metrics directly inside the Google Cloud Console Anthos dashboard.
Pro Tip: Policy Controller intercepts admission requests before etcd persistence, stopping bad manifests even if applied outside GitOps.
4️⃣

Validate Automated Drift Correction & Self-Healing

Verify that unauthorized changes are automatically overwritten:

  • Drift Simulation: Manually deleted a required NetworkPolicy using kubectl delete networkpolicy deny-egress.
  • Self-Healing: Within 15 seconds, Config Sync's in-cluster reconciler detected the divergence and recreated the resource.
  • CLI Status: Checked synchronization state with nomos status.
Pro Tip: Nomos status provides instant cluster-by-cluster visibility into commit SHAs, errors, and sync status.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Anthos Config Sync and Policy Controller deliver scalable multi-cluster governance by decoupling platform-level security (RootSync) from developer applications (RepoSync) while enforcing automated drift reconciliation."
⚡ 60-Second Elevator Pitch Talking Points
  • Registered distributed GKE clusters into an Anthos Fleet with Connect Gateway.
  • Configured hierarchical GitOps using RootSync for cluster-admin rules and RepoSync for tenant squads.
  • Enforced OPA Gatekeeper Policy Controller constraints to block non-compliant resources at admission time.
  • Verified automated 15-second self-healing drift correction using nomos status.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring Enterprise Multi Cluster Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring Enterprise Multi Cluster Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring Enterprise Multi Cluster Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring Enterprise Multi Cluster Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →