Q: A mission-critical payment service caused a host-wide outage when an unhandled application loop spewed 80GB of debug logs over the weekend. Because Docker's default logging driver was unconfigured, the `*-json.log` file in `/var/lib/docker/containers/` filled the host root filesystem to 100%, causing the Docker daemon, kubelet, and etcd to crash. You must configure global log rotation in `daemon.json`, benchmark log drivers (`json-file` vs `local`), and implement non-blocking delivery to prevent slow log writers from freezing application I/O.
Configure production-grade Docker daemon logging drivers in `daemon.json`. Prevent host node filesystem exhaustion using `local` and `json-file` log rotation with non-blocking buffer modes.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Analyze the Mechanics of Docker Default Logging Exhaustion
By default, Docker uses the `json-file` driver with no size limit (`max-size: unlimited`) and blocking I/O mode. A verbose container will continuously write to disk until host storage is exhausted, crashing host systemd daemons.
# Inspect the runaway container log on the host filesystem
ls -lh /var/lib/docker/containers/<CONTAINER_ID>/<CONTAINER_ID>-json.log
# Check filesystem usage
df -h /var/lib/dockerConfigure Enterprise Log Rotation in daemon.json
Update `/etc/docker/daemon.json` with global log rotation parameters. Limit each log file to 50MB with a maximum of 3 rotated files, ensuring a maximum disk footprint of 150MB per container.
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "3",
"mode": "non-blocking",
"max-buffer-size": "4m",
"compress": "true"
}
}
EOFEvaluate the High-Performance 'local' Log Driver
For performance-critical environments, configure Docker's internal `local` log driver. Unlike `json-file`, the `local` driver uses an efficient binary format, applies automatic log compression, and enforces default 20MB file limits without manual configuration.
{
"log-driver": "local",
"log-opts": {
"max-size": "20m",
"max-file": "5",
"compress": "true"
}
}Apply Configuration Safely via SIGHUP Signal
Reload the Docker daemon configuration without stopping active containers by sending a `SIGHUP` signal to `dockerd` or executing `systemctl reload docker`.
# Reload daemon configuration dynamically
sudo systemctl reload docker
# Verify active daemon settings
docker info --format '{{.LoggingDriver}}'
# Verify log rotation on newly launched containers
docker run -d --name log-test alpine sh -c 'while true; do echo "test log $(date)"; sleep 0.1; done'
docker inspect log-test --format '{{json .HostConfig.LogConfig}}' | jq .- W
- e
- e
- l
- i
- m
- i
- n
- a
- t
- e
- d
- d
- i
- s
- k
- e
- x
- h
- a
- u
- s
- t
- i
- o
- n
- o
- u
- t
- a
- g
- e
- s
- a
- c
- r
- o
- s
- s
- o
- u
- r
- c
- o
- n
- t
- a
- i
- n
- e
- r
- f
- l
- e
- e
- t
- b
- y
- h
- a
- r
- d
- e
- n
- i
- n
- g
- `
- d
- a
- e
- m
- o
- n
- .
- j
- s
- o
- n
- `
- .
- W
- e
- c
- o
- n
- f
- i
- g
- u
- r
- e
- d
- s
- t
- r
- i
- c
- t
- 5
- 0
- M
- B
- l
- o
- g
- f
- i
- l
- e
- r
- o
- t
- a
- t
- i
- o
- n
- w
- i
- t
- h
- c
- o
- m
- p
- r
- e
- s
- s
- i
- o
- n
- a
- n
- d
- s
- e
- t
- `
- m
- o
- d
- e
- :
- n
- o
- n
- -
- b
- l
- o
- c
- k
- i
- n
- g
- `
- w
- i
- t
- h
- a
- 4
- M
- B
- b
- u
- f
- f
- e
- r
- .
- T
- h
- i
- s
- g
- u
- a
- r
- a
- n
- t
- e
- e
- s
- t
- h
- a
- t
- c
- o
- n
- t
- a
- i
- n
- e
- r
- l
- o
- g
- g
- i
- n
- g
- c
- a
- n
- n
- o
- t
- m
- o
- n
- o
- p
- o
- l
- i
- z
- e
- n
- o
- d
- e
- d
- i
- s
- k
- s
- p
- a
- c
- e
- o
- r
- d
- e
- g
- r
- a
- d
- e
- a
- p
- p
- l
- i
- c
- a
- t
- i
- o
- n
- t
- h
- r
- o
- u
- g
- h
- p
- u
- t
- d
- u
- r
- i
- n
- g
- h
- i
- g
- h
- -
- f
- r
- e
- q
- u
- e
- n
- c
- y
- l
- o
- g
- s
- u
- r
- g
- e
- s
- .