Q: Service is running but not listening on the expected port — how would you troubleshoot?
Structured diagnostic sequence for services running in systemd but failing port checks: 127.0.0.1 vs 0.0.0.0 binding, privileged ports, firewall drops, and SELinux blocks.
systemctl Active but Port Is Dead? 5 Checks to Find Why | Linux Interview
Watch Naveed break down the 5-point triage sequence when systemctl reports active (running) but the listening port answers nothing—from ss -tulpn socket inspection and 127.0.0.1 IP binding traps to CAP_NET_BIND_SERVICE, iptables, and SELinux port policies.
- Verify open sockets: ss -tulpn | grep <process> or lsof -i :<port> to check listening state
- Check IP binding address: Ensure 0.0.0.0 (all interfaces) rather than 127.0.0.1 (localhost only)
- Audit privileged ports (<1024): Grant AmbientCapabilities=CAP_NET_BIND_SERVICE in systemd
- Inspect host firewall & SELinux: Check iptables/nftables drops and semanage port rules
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When systemctl reports 'active (running)' but you cannot connect to the expected port, I trace through 5 discrete checkpoints: daemon listening sockets, IP binding address, privileged port permissions, firewall rules, and security modules.
Verify Open Sockets (ss / lsof / netstat)
What ports is the process actually listening on?
ss -tulpn | grep <process-name-or-PID>: (Modern replacement for netstat). Shows all TCP/UDP listening sockets with process names.lsof -i :<port>: Check if another service has already bound to that port (e.g. Apache already listening on 80 when starting Nginx).- If the process is NOT listed in
ss -tulpn: The service is running a worker/watcher process that never opened a server socket, or socket binding threw an exception during initialization.
The IP Binding Address Trap (127.0.0.1 vs 0.0.0.0)
The #1 configuration trap in service setup:
- Look at the Local Address in
ss -tulpn: - Bound to
127.0.0.1:<port>(orlocalhost): The service will ONLY accept connections originating locally from the same host! External network connections will be dropped or refused. - Must be bound to
0.0.0.0:<port>(or:::<port>for IPv6): Listens on all network interfaces. - Fix: Update configuration file (e.g.
server.host = '0.0.0.0'in app config).
Privileged Port & Permission Issues (Ports < 1024)
Linux security restrictions on well-known ports:
- Ports below 1024 (e.g. 80, 443, 53) require root privileges to bind by default.
- If a non-root systemd service tries to bind to port 80: It will silently fail or log
Permission denied. - Fix without running as root: Grant the binary socket capability:
sudo setcap 'cap_net_bind_service=+ep' /path/to/binary, or configureAmbientCapabilities=CAP_NET_BIND_SERVICEin the systemd unit file.
Firewall (iptables/nftables) & SELinux
Host-level security layers blocking incoming packets:
- Host Firewall: Check
sudo iptables -L -n -v,sudo ufw status, orsudo nft list ruleset. Verify incoming traffic on the port is not REJECTED or DROPPED. - SELinux / AppArmor: On RHEL/CentOS, SELinux prevents services from binding to non-standard ports (e.g. Nginx binding to port 8088).
- Check SELinux:
getenforceandsudo ausearch -m avc -ts recent. - Fix SELinux port policy:
sudo semanage port -a -t http_port_t -p tcp 8088.
- Check open sockets: 'ss -tulpn | grep <process>' or 'lsof -i :<port>' to verify if socket exists.
- Check binding address: Verify service is bound to 0.0.0.0 (all interfaces), NOT 127.0.0.1 (localhost only).
- Check logs: 'journalctl -u <service> -e' for bind errors, port conflicts, or permission denied.
- Check privileged ports (<1024): Non-root users cannot bind to 80/443 without 'AmbientCapabilities=CAP_NET_BIND_SERVICE' in systemd.
- Check firewall: Inspect 'iptables -L -n -v' or 'ufw status' for incoming port drops.
- Check SELinux/AppArmor: Look for AVC denials in 'audit.log'; assign port context via 'semanage port'.
Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Issuing immediate kill -9 (SIGKILL) commands rather than allowing graceful termination via kill -15 (SIGTERM) when configuring Service Running Listening SIGKILL immediately halts execution without allowing processes to flush memory buffers, commit database transactions, or remove stale PID locks.
-
Overlooking file system inode exhaustion (df -i) when diagnosing 'No space left on device' errors when configuring Service Running Listening
Millions of small files or unrotated session tokens can exhaust available file system inodes even when standard disk storage (
df -h) shows ample free space. - Exhausting open file descriptor allocations (ulimit -n) under intensive concurrent socket connections when configuring Service Running Listening Reaching file descriptor exhaustion throws 'Too many open files' errors, preventing services from accepting new TCP client socket connections.
-
Permitting defunct zombie processes to accumulate without fixing parent process SIGCHLD waitpid reaping when configuring Service Running Listening
Zombie processes consume process table entries; accumulating zombies can exhaust the system maximum PID limit (
pid_max).