Q: Service is running but not listening on the expected port — how would you troubleshoot?
Structured diagnostic sequence for services running in systemd but failing port checks: 127.0.0.1 vs 0.0.0.0 binding, privileged ports, firewall drops, and SELinux blocks.
#Linux #ss #netstat #systemd #SELinux #Firewall #IP Binding
🎙️ Candidate Opening & Architectural Context
"When systemctl reports 'active (running)' but you cannot connect to the expected port, I trace through 5 discrete checkpoints: daemon listening sockets, IP binding address, privileged port permissions, firewall rules, and security modules."
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Verify Open Sockets (ss / lsof / netstat)
What ports is the process actually listening on?
ss -tulpn | grep <process-name-or-PID>: (Modern replacement for netstat). Shows all TCP/UDP listening sockets with process names.lsof -i :<port>: Check if another service has already bound to that port (e.g. Apache already listening on 80 when starting Nginx).- If the process is NOT listed in
ss -tulpn: The service is running a worker/watcher process that never opened a server socket, or socket binding threw an exception during initialization.
2️⃣
The IP Binding Address Trap (127.0.0.1 vs 0.0.0.0)
The #1 configuration trap in service setup:
- Look at the Local Address in
ss -tulpn: - Bound to
127.0.0.1:<port>(orlocalhost): The service will ONLY accept connections originating locally from the same host! External network connections will be dropped or refused. - Must be bound to
0.0.0.0:<port>(or:::<port>for IPv6): Listens on all network interfaces. - Fix: Update configuration file (e.g.
server.host = '0.0.0.0'in app config).
3️⃣
Privileged Port & Permission Issues (Ports < 1024)
Linux security restrictions on well-known ports:
- Ports below 1024 (e.g. 80, 443, 53) require root privileges to bind by default.
- If a non-root systemd service tries to bind to port 80: It will silently fail or log
Permission denied. - Fix without running as root: Grant the binary socket capability:
sudo setcap 'cap_net_bind_service=+ep' /path/to/binary, or configureAmbientCapabilities=CAP_NET_BIND_SERVICEin the systemd unit file.
4️⃣
Firewall (iptables/nftables) & SELinux
Host-level security layers blocking incoming packets:
- Host Firewall: Check
sudo iptables -L -n -v,sudo ufw status, orsudo nft list ruleset. Verify incoming traffic on the port is not REJECTED or DROPPED. - SELinux / AppArmor: On RHEL/CentOS, SELinux prevents services from binding to non-standard ports (e.g. Nginx binding to port 8088).
- Check SELinux:
getenforceandsudo ausearch -m avc -ts recent. - Fix SELinux port policy:
sudo semanage port -a -t http_port_t -p tcp 8088.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Check 'ss -tulpn | grep <PID>' first. If listening on 127.0.0.1, change binding to 0.0.0.0. If port < 1024, check CAP_NET_BIND_SERVICE. If bound correctly, check iptables and SELinux port policies."
⚡ 60-Second Elevator Pitch Talking Points
- Check open sockets: 'ss -tulpn | grep <process>' or 'lsof -i :<port>' to verify if socket exists.
- Check binding address: Verify service is bound to 0.0.0.0 (all interfaces), NOT 127.0.0.1 (localhost only).
- Check logs: 'journalctl -u <service> -e' for bind errors, port conflicts, or permission denied.
- Check privileged ports (<1024): Non-root users cannot bind to 80/443 without 'AmbientCapabilities=CAP_NET_BIND_SERVICE' in systemd.
- Check firewall: Inspect 'iptables -L -n -v' or 'ufw status' for incoming port drops.
- Check SELinux/AppArmor: Look for AVC denials in 'audit.log'; assign port context via 'semanage port'.
Advertisement