Q: Engineers in your team run 'terraform apply' locally from their laptops, leading to concurrent state file collisions, unreviewed infrastructure changes, and lost drift history. How do you design and deploy Atlantis on Kubernetes to automate Terraform / Terragrunt planning and applying directly via GitHub PR comments with strict approval gates?
Engineering a collaborative, secure Infrastructure as Code (IaC) deployment pipeline using Atlantis for Terraform & Terragrunt, with automated PR comment planning, state locking, and apply controls.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Running Terraform locally from engineer workstations is an operational anti-pattern that leads to unreviewed production changes and leaked state files. We deployed Atlantis on Kubernetes to turn pull requests into the sole control plane for infrastructure execution.
Deploy Atlantis StatefulSet on Kubernetes with Cloud IAM Roles
Establish a secure, persistent Atlantis execution engine:
- StatefulSet Installation: Deployed Atlantis StatefulSet with persistent volume storage for checked-out Git repositories and Terraform workspaces.
- Cloud Authentication: Authenticated Atlantis pod via AWS IAM Roles for Service Accounts (IRSA) / Azure Workload Identity, eliminating static cloud credentials.
Configure atlantis.yaml Project Directory Layout for Terragrunt
Define automated plan and apply execution workflows:
- atlantis.yaml Spec: Configured project declarations mapping directories (e.g.
live/prod/vpc,live/prod/eks). - Terragrunt Workflow: Defined custom workflow steps:
plan: { steps: [{ run: 'terragrunt plan -out $PLANFILE' }] }andapply: { steps: [{ run: 'terragrunt apply $PLANFILE' }] }.
Execute Automated PR Planning & Distributed State Locking
Provide transparency and prevent concurrent modifications to the same infrastructure:
- Automated PR Plan: Opening a PR triggers Atlantis to automatically run
terragrunt plan, posting the formatted diff directly as a comment on the GitHub PR. - Atlantis Lock: Atlantis locks the modified directory; if another engineer opens a PR touching the same module, Atlantis rejects the plan until the first PR is merged or unlocked.
Enforce Mandatory Peer Approvals & In-PR Apply Execution
Ensure changes are reviewed by senior engineers before hitting production:
- Apply Requirements: Configured
apply_requirements: [approved, mergeable]; Atlantis refuses to executeatlantis applyunless the PR has at least 2 peer approvals and passes CI checks. - In-PR Apply: Approved engineer comments
atlantis apply; Atlantis executes the apply and automatically merges the PR and closes the branch upon success.
- Deploy Atlantis on Kubernetes with AWS IRSA / Azure Workload Identity.
- Configure atlantis.yaml with custom Terragrunt plan and apply workflows.
- Lock modified project directories automatically to prevent concurrent state collisions.
- Enforce mandatory peer approvals before unlocking the 'atlantis apply' PR comment command.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Promoting application releases to production without an automated, canary-validated rollback mechanism when configuring GitOps Atlantis Pull Pipelines that lack automated rollback mechanisms force engineering teams to scramble manual hotfixes during severe production regressions.
- Running concurrent deployment pipeline jobs without stage mutex locking on shared target environments when configuring GitOps Atlantis Pull Concurrent pipeline runs on the same environment race to apply state changes, causing out-of-order commits and unpredictable deployment state.
- Omitting dependency lockfiles (package-lock.json, go.sum, Cargo.lock) from continuous integration builds when configuring GitOps Atlantis Pull Omitting lockfiles allows upstream package updates to break builds unpredictably without any changes to application codebase.
- Failing to define dynamic cache invalidation keys for CI package managers and layer caches when configuring GitOps Atlantis Pull Stale build caches mask underlying dependency incompatibilities or pull outdated libraries, while un-cached runs slow feedback loops.