Q: During a flash sale, your frontend API containers experienced high latency spikes. Docker healthchecks configured with an aggressive 2-second timeout (`--timeout=2s`) and 2 retries began failing because the busy application took 2.5 seconds to reply to the health check probe. Docker marked the containers `unhealthy`, triggering orchestrator restart loops that killed healthy instances and amplified the traffic load on remaining nodes, causing a complete cascading system collapse. You must properly calibrate healthcheck parameters.

A container healthcheck has three lifecycle states: `starting` (during `start-period`), `healthy` (probe exit code 0), and `unhealthy` (probe exit code 1 or timeout). Crucially, during `start-period`, failing health checks do not count towards the retry limit.

<!-- Healthcheck Lifecycle -->
Container Starts ───> State: STARTING (start-period: 30s)
                          │ (Failures do not trigger unhealthy!)
                          └─── Probe succeeds -> State: HEALTHY
                                  │ (Normal monitoring begins: interval=10s)
                                  └── Failures >= retries -> State: UNHEALTHY

Avoid hair-trigger thresholds. Set `interval` to 10-15 seconds, `timeout` to 5 seconds, and `retries` to 3 or 4. Ensure `start-period` is long enough to accommodate slow application warm-ups and schema checks.

# Dockerfile
HEALTHCHECK --interval=15s \
            --timeout=5s \
            --start-period=40s \
            --retries=3 \
  CMD curl -f http://localhost:8080/health/liveness || exit 1

A common mistake is designing health checks that test heavy downstream dependencies (e.g., executing a database query or external API call). If the database slows down, all API containers fail their healthchecks and reboot simultaneously. Healthchecks should test internal container viability (liveness), not transitive dependencies.

// Healthcheck endpoint implementation (Node.js/Express)
app.get('/health/liveness', (req, res) => {
  // Check ONLY local process viability, memory pressure, and event loop lag
  if (isEventLoopBlocked() || isOutOfMemory()) {
    return res.status(503).send('Unhealthy');
  }
  return res.status(200).send('OK'); // Do NOT query external DB here!
});

Use `docker inspect` to view the timestamp, exit code, and stdout/stderr output of the last 5 health check executions.

# View recent healthcheck execution logs
docker inspect --format '{{json .State.Health}}' my-app | jq .