Q: You have around 500 TB of data in one AWS account and need to migrate it to another account. How would you design the secure, automated migration solution?
Enterprise architectural blueprint for migrating 500 Terabytes of live object data between two distinct AWS accounts securely, with automated checksum verification, KMS key grant delegation, and minimal cost.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Migrating 500 TB of data cross-account cannot be done with a naive 'aws s3 sync' CLI script. It requires high-throughput managed services: S3 Cross-Account Replication paired with S3 Batch Replication, or AWS DataSync, with cross-account IAM delegation and KMS CMK key grants.
Select Architecture: S3 Batch Replication vs. AWS DataSync
For pure S3-to-S3 object migration while preserving metadata and versioning, S3 Batch Replication is ideal. For scheduled differential syncs with bandwidth rate-limiting, AWS DataSync provides automated verification.
Configure Cross-Account IAM Role & Destination Bucket Policy
Create an IAM role in Account A with permission to read source objects and replicate to Account B. Configure Account B's bucket policy to allow Account A's replication role to write objects with BucketOwnerEnforced ownership.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::111111111111:role/s3-replication-role"},
"Action": ["s3:ReplicateObject", "s3:ReplicateDelete", "s3:ObjectOwnerOverrideToBucketOwner"],
"Resource": "arn:aws:s3:::destination-bucket-account-b/*"
}]
}
Delegate KMS CMK Encryption Permissions
If source objects use KMS CMK encryption, the KMS key policy in Account A must grant kms:Decrypt to the replication role, and Account B's KMS key must grant kms:Encrypt.
Execute S3 Batch Replication for Historical Data
Enable S3 Cross-Region/Cross-Account Replication for all new incoming objects. Generate an S3 Inventory report on the source bucket and initiate an S3 Batch Replication job to backfill the existing 500 TB.
- Enable S3 Cross-Account Replication to replicate all new incoming objects in real-time.
- Generate an S3 Inventory list of the source bucket and launch an S3 Batch Replication job for the 500 TB.
- Configure cross-account KMS CMK key grants for seamless decryption and re-encryption.
- Set Object Ownership to BucketOwnerEnforced in Account B so destination account owns all migrated objects.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Relying on legacy bucket ACLs instead of modern S3 Block Public Access and IAM bucket policies when configuring Migrating Data Across AWS recommends disabling ACLs (Bucket Owner Enforced) in favor of centralized declarative bucket policies and account-level Block Public Access controls.
-
Confusing SSE-S3 encryption with KMS customer-managed key (SSE-KMS) authorization policies when configuring Migrating Data Across
Accessing SSE-KMS encrypted objects requires both
s3:GetObjectandkms:Decryptpermissions; neglecting KMS key policies causes silent 403 Forbidden errors. - Using sequential timestamps or predictable prefixes for high-throughput S3 read and write keys when configuring Migrating Data Across Sequential key prefixes limit S3 horizontal partitioning; randomized hash prefixes distribute workloads across partitions to achieve 3,500 PUT and 5,500 GET requests/sec.
- Failing to audit S3 object delete markers when troubleshooting missing versioned objects when configuring Migrating Data Across Issuing a DELETE on a versioned bucket only adds a delete marker; engineers must retrieve previous versions or purge markers to restore data.