Q: Your team is running stateful database containers (PostgreSQL, MongoDB) on standalone AWS EC2 instances managed by Docker Compose. If an EC2 host suffers a hardware failure, an operator must provision a new instance, detach the AWS EBS volume from the dead host, reattach it to the new host, and mount it before starting the container. You need to implement an automated Docker storage plugin (using RexRay or Docker CSI) that dynamically attaches and mounts AWS EBS and EFS volumes on container startup.
Architect persistent cloud block and file storage for Docker containers. Configure Docker volume plugins (RexRay / CSI plugins) to dynamically attach, format, and mount AWS EBS and EFS volumes.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Architect persistent cloud block and file storage for Docker containers. Configure Docker volume plugins (RexRay / CSI plugins) to dynamically attach, format, and mount AWS EBS and EFS volumes.
Understand Docker Volume Plugin Architecture
Docker supports storage extensibility through managed volume plugins (V2 plugin architecture). When a container requests a named volume backed by a plugin, the plugin communicates with cloud provider APIs (e.g., AWS EC2/EBS APIs) to dynamically attach the cloud disk to the host and mount it to /var/lib/docker/plugins/....
<!-- Volume Plugin Flow -->
docker run -v my-ebs-vol:/data db:latest
└── Docker Volume Plugin (RexRay / AWS EBS Driver)
├── Calls AWS API: DescribeVolumes & AttachVolume (vol-0a1b2c -> i-09876)
├── Waits for /dev/xvdf to appear on Linux host
├── Formats (mkfs.ext4 if new) & mounts to host
└── Passes mountpoint into container /data
Install and Configure Docker Cloud Volume Plugin
Install the managed storage plugin using docker plugin install, granting necessary IAM permissions for AWS EBS and EC2 volume management.
# Install RexRay AWS EBS volume plugin
docker plugin install rexray/ebs \
EBS_REGION=us-east-1 \
EBS_MAXRETRIES=10 \
--grant-all-permissions
# Verify active plugins
docker plugin ls
Provision and Mount Persistent EBS Volume Declaratively
Create a persistent named volume backed by the cloud driver specifying volume size, IOPS, and volume type (gp3). Mount the volume into the database container.
# Create 100GB GP3 volume via Docker CLI
docker volume create -d rexray/ebs \
--name prod-postgres-ebs \
-o size=100 \
-o volumeType=gp3 \
-o iops=3000
# Run stateful container using the persistent cloud volume
docker run -d \
--name db-primary \
-v prod-postgres-ebs:/var/lib/postgresql/data \
postgres:16
Handle Volume Failover Across Hosts
Simulate host failure. Terminate the primary EC2 host and launch the database container on a secondary EC2 host using the same volume name. The plugin preemptively detaches the volume from the dead node and attaches it to the new host.
# On Host B:
docker run -d \
--name db-primary \
-v prod-postgres-ebs:/var/lib/postgresql/data \
postgres:16
# RexRay detects existing EBS volume, detaches from dead node, attaches to Host B!
- W
- e
- e
- n
- a
- b
- l
- e
- d
- a
- u
- t
- o
- m
- a
- t
- e
- d
- d
- i
- s
- a
- s
- t
- e
- r
- r
- e
- c
- o
- v
- e
- r
- y
- f
- o
- r
- s
- t
- a
- t
- e
- f
- u
- l
- D
- o
- c
- k
- e
- r
- w
- o
- r
- k
- l
- o
- a
- d
- s
- b
- y
- i
- n
- t
- e
- g
- r
- a
- t
- i
- n
- g
- m
- a
- n
- a
- g
- e
- d
- c
- l
- o
- u
- d
- v
- o
- l
- u
- m
- e
- p
- l
- u
- g
- i
- n
- s
- .
- P
- e
- r
- s
- i
- s
- t
- e
- n
- t
- d
- a
- t
- a
- b
- a
- s
- e
- s
- m
- o
- u
- n
- t
- n
- a
- m
- e
- d
- E
- B
- S
- v
- o
- l
- u
- m
- e
- s
- b
- a
- c
- k
- e
- d
- b
- y
- t
- h
- e
- s
- t
- o
- r
- a
- g
- e
- d
- r
- i
- v
- e
- r
- ;
- i
- f
- a
- n
- u
- n
- d
- e
- r
- l
- y
- i
- n
- g
- h
- o
- s
- t
- f
- a
- i
- l
- s
- ,
- l
- a
- u
- n
- c
- h
- i
- n
- g
- t
- h
- e
- c
- o
- n
- t
- a
- i
- n
- e
- r
- o
- n
- a
- n
- e
- w
- i
- n
- s
- t
- a
- n
- c
- e
- a
- u
- t
- o
- m
- a
- t
- i
- c
- a
- l
- l
- y
- d
- e
- t
- a
- c
- h
- e
- s
- t
- h
- e
- E
- B
- S
- v
- o
- l
- u
- m
- e
- a
- n
- d
- r
- e
- -
- m
- o
- u
- n
- t
- s
- i
- t
- i
- n
- s
- e
- c
- o
- n
- d
- s
- w
- i
- t
- h
- z
- e
- r
- o
- d
- a
- t
- a
- l
- o
- s
- s
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring Integrating Persistent Storage
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring Integrating Persistent Storage
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring Integrating Persistent Storage Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring Integrating Persistent Storage Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.