Q: Tell me a real scenario where YOU personally introduced a failure in production infrastructure. What happened, what did you learn, and what architectural safeguards did you implement afterward?

During a routine Terraform refactoring of our AWS networking layer, I was renaming a security group module to follow new company naming standards. While `terraform plan` showed changes to security group descriptions and tags, I missed that changing the resource identifier in HCL caused Terraform to plan a destructive replacement (`destroy and then create`) of the primary Redis ElastiCache security group instead of an in-place update. Upon running apply, Terraform dropped the existing security group before the replacement provisioned, instantly severing 40,000 active web client connections and causing 500 Internal Server Errors across checkout.

I did not panic or attempt a convoluted rollback. I immediately joined the incident channel, announced: 'I just ran a Terraform apply on the Redis security group that destroyed the ingress rule; I am manually attaching the prior security group right now.' Using AWS CLI, I re-attached the legacy security group to the ElastiCache cluster within 90 seconds, restoring customer checkout. Total outage duration: 2 minutes 15 seconds.

The failure was not merely 'human error.' The systemic weaknesses were: 1. Terraform lacked `create_before_destroy = true` lifecycle rules on networking dependencies. 2. CI pipeline did not enforce automated plan checking for destructive actions (`-` or `- / +`). 3. Applying Terraform changes to production was permitted from local engineer workstations rather than strictly through an automated GitOps pipeline with approval gates.

I authored a blameless post-mortem and implemented three permanent platform safeguards: 1. **Policy as Code (OPA Conftest / Sentinel)**: Blocked any PR from merging if the plan contains destructive replacements of stateful resources (`aws_security_group`, `aws_db_instance`, `aws_elasticache_cluster`). 2. **Mandatory State Move Procedures**: Established runbooks requiring `terraform state mv` when renaming resources to guarantee in-place refactoring. 3. **Atlantis GitOps Gateways**: Revoked local AWS apply permissions; all production Terraform changes now run strictly via Atlantis with two peer approvals.