Q: A high-throughput microservice in production experiences periodic 200ms latency spikes. Traditional application APM agents and CPU profilers show normal CPU usage and no garbage collection pauses. Profiling with `strace` in production caused a 400% performance degradation and crashed client connections. You must use eBPF and `bpftrace` to trace disk I/O latency and network packet drops directly in the Linux kernel filtered by the container's cgroup ID with less than 1% overhead.
Achieve kernel-level observability into Docker containers without injecting code or restarting processes. Use eBPF and bpftrace to trace container filesystem latency, dropped packets, and DNS queries by cgroup.
bpftrace to trace container filesystem latency, dropped packets, and DNS queries by cgroup."
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Achieve kernel-level observability into Docker containers without injecting code or restarting processes. Use eBPF and bpftrace to trace container filesystem latency, dropped packets, and DNS queries by cgroup.
Understand Why eBPF Outperforms strace for Container Observability
strace relies on the ptrace system call, which intercepts every system call, pauses the target process twice (on entry and exit), and forces context switches to user space. eBPF executes sandboxed bytecode directly inside the Linux kernel on kprobes and tracepoints with sub-microsecond overhead.
<!-- Performance Impact -->
strace: Pauses container process on EVERY syscall! Overhead: 50% - 400% (Never run in Prod!)
eBPF: JIT-compiled in-kernel verification. Copies only aggregated histograms. Overhead: < 1%
Identify Container Cgroup ID for Scoped Filtering
Obtain the container's 64-bit cgroup ID or namespace inode number to filter eBPF probes specifically to that container, avoiding host-wide trace noise.
# Get container cgroup v2 ID
CONTAINER_ID=$(docker inspect my-app --format '{{.Id}}')
CGROUP_PATH="/sys/fs/cgroup/docker/${CONTAINER_ID}"
CGROUP_ID=$(python3 -c "import os; print(os.stat('$CGROUP_PATH').st_ino)")
echo "Filtering eBPF for cgroup ID: $CGROUP_ID"
Trace Container Disk I/O Latency using bpftrace One-Liner
Attach a bpftrace program to the kernel's block I/O tracepoints, generating a power-of-two histogram of disk latency specifically for the container.
# Trace block I/O latency histogram (in microseconds) for container cgroup
sudo bpftrace -e '
tracepoint:block:block_bio_issue {
if (cgroup == '$CGROUP_ID') {
@start[args->dev, args->sector] = nsecs;
}
}
tracepoint:block:block_bio_complete {
if (@start[args->dev, args->sector]) {
@lat_us = hist((nsecs - @start[args->dev, args->sector]) / 1000);
delete(@start[args->dev, args->sector]);
}
}
'
Trace Dropped TCP Packets and TCP Retransmissions
Monitor kernel TCP retransmissions and dropped packets per container using the BCC tcpretrans tool or bpftrace kernel tracepoints.
# Monitor TCP retransmits across container processes
sudo /usr/share/bcc/tools/tcpretrans -c
strace. Filtering by container cgroup IDs allows SREs to diagnose disk I/O latency and TCP drops in live production environments safely."
- W
- e
- d
- i
- a
- g
- n
- o
- s
- e
- d
- m
- i
- c
- r
- o
- s
- e
- r
- v
- i
- c
- e
- l
- a
- t
- e
- n
- c
- y
- s
- p
- i
- k
- e
- s
- i
- n
- p
- r
- o
- d
- u
- c
- t
- i
- o
- n
- w
- i
- t
- h
- o
- u
- t
- a
- d
- d
- i
- n
- g
- A
- P
- M
- o
- v
- e
- r
- h
- e
- a
- d
- b
- y
- u
- s
- i
- n
- g
- e
- B
- P
- F
- a
- n
- d
- `
- b
- p
- f
- t
- r
- a
- c
- e
- `
- .
- W
- h
- i
- l
- e
- `
- s
- t
- r
- a
- c
- e
- `
- w
- o
- u
- l
- d
- h
- a
- v
- e
- d
- e
- g
- r
- a
- d
- e
- d
- p
- e
- r
- f
- o
- r
- m
- a
- n
- c
- e
- ,
- o
- u
- r
- e
- B
- P
- F
- t
- r
- a
- c
- e
- p
- o
- i
- n
- t
- f
- i
- l
- t
- e
- r
- e
- d
- b
- y
- t
- h
- e
- c
- o
- n
- t
- a
- i
- n
- e
- r
- '
- s
- c
- g
- r
- o
- u
- p
- I
- D
- i
- d
- e
- n
- t
- i
- f
- i
- e
- d
- t
- h
- a
- t
- s
- l
- o
- w
- N
- V
- M
- e
- b
- l
- o
- c
- k
- w
- r
- i
- t
- e
- s
- w
- e
- r
- e
- b
- l
- o
- c
- k
- i
- n
- g
- a
- p
- p
- l
- i
- c
- a
- t
- i
- o
- n
- w
- o
- r
- k
- e
- r
- t
- h
- r
- e
- a
- d
- s
- ,
- l
- e
- a
- d
- i
- n
- g
- t
- o
- a
- n
- i
- m
- m
- e
- d
- i
- a
- t
- e
- E
- B
- S
- v
- o
- l
- u
- m
- e
- c
- l
- a
- s
- s
- u
- p
- g
- r
- a
- d
- e
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring Overhead Container Observability
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring Overhead Container Observability
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring Overhead Container Observability Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring Overhead Container Observability Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.