Staff Cloud Architect AWS AWS Large-Scale Cloud Data Migration Staff Architecture

Q: You have around 500 TB of data in one AWS account and need to migrate it to another account. How would you design the secure, automated migration solution?

Enterprise architectural blueprint for migrating 500 Terabytes of live object data between two distinct AWS accounts securely, with automated checksum verification, KMS key grant delegation, and minimal cost.

#AWS #S3 #Data Migration #DataSync #Architecture #FinOps

🎙️ Candidate Opening & Architectural Context

"Migrating 500 TB of data cross-account cannot be done with a naive 'aws s3 sync' CLI script. It requires high-throughput managed services: S3 Cross-Account Replication paired with S3 Batch Replication, or AWS DataSync, with cross-account IAM delegation and KMS CMK key grants."

🛠️ Production Runbook & Step-by-Step Resolution

Select Architecture: S3 Batch Replication vs. AWS DataSync

For pure S3-to-S3 object migration while preserving metadata and versioning, S3 Batch Replication is ideal. For scheduled differential syncs with bandwidth rate-limiting, AWS DataSync provides automated verification.

Configure Cross-Account IAM Role & Destination Bucket Policy

Create an IAM role in Account A with permission to read source objects and replicate to Account B. Configure Account B's bucket policy to allow Account A's replication role to write objects with BucketOwnerEnforced ownership.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"AWS": "arn:aws:iam::111111111111:role/s3-replication-role"},
    "Action": ["s3:ReplicateObject", "s3:ReplicateDelete", "s3:ObjectOwnerOverrideToBucketOwner"],
    "Resource": "arn:aws:s3:::destination-bucket-account-b/*"
  }]
}

Delegate KMS CMK Encryption Permissions

If source objects use KMS CMK encryption, the KMS key policy in Account A must grant kms:Decrypt to the replication role, and Account B's KMS key must grant kms:Encrypt.

Execute S3 Batch Replication for Historical Data

Enable S3 Cross-Region/Cross-Account Replication for all new incoming objects. Generate an S3 Inventory report on the source bucket and initiate an S3 Batch Replication job to backfill the existing 500 TB.

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Use S3 Replication for ongoing writes combined with S3 Batch Replication on an S3 Inventory report for the 500 TB backfill. Enforce BucketOwnerEnforced to guarantee destination ownership."

⚡ 60-Second Elevator Pitch Talking Points

Enable S3 Cross-Account Replication to replicate all new incoming objects in real-time.
Generate an S3 Inventory list of the source bucket and launch an S3 Batch Replication job for the 500 TB.
Configure cross-account KMS CMK key grants for seamless decryption and re-encryption.
Set Object Ownership to BucketOwnerEnforced in Account B so destination account owns all migrated objects.