⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Linux & SRE Interview Questions Scenario 4 of 135 in Linux & SRE
Senior DevOps / SRE Linux Networking & Daemons Core Linux Networking
🎯 Target Role / Context: Linux System Administration & Networking

Q: Service is running but not listening on the expected port — how would you troubleshoot?

Structured diagnostic sequence for services running in systemd but failing port checks: 127.0.0.1 vs 0.0.0.0 binding, privileged ports, firewall drops, and SELinux blocks.

#Linux #ss #netstat #systemd #SELinux #Firewall #IP Binding
🎙️ Candidate Opening & Architectural Context
"When systemctl reports 'active (running)' but you cannot connect to the expected port, I trace through 5 discrete checkpoints: daemon listening sockets, IP binding address, privileged port permissions, firewall rules, and security modules."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When systemctl reports 'active (running)' but you cannot connect to the expected port, I trace through 5 discrete checkpoints: daemon listening sockets, IP binding address, privileged port permissions, firewall rules, and security modules.

1️⃣

Verify Open Sockets (ss / lsof / netstat)

What ports is the process actually listening on?

  • ss -tulpn | grep <process-name-or-PID>: (Modern replacement for netstat). Shows all TCP/UDP listening sockets with process names.
  • lsof -i :<port>: Check if another service has already bound to that port (e.g. Apache already listening on 80 when starting Nginx).
  • If the process is NOT listed in ss -tulpn: The service is running a worker/watcher process that never opened a server socket, or socket binding threw an exception during initialization.
2️⃣

The IP Binding Address Trap (127.0.0.1 vs 0.0.0.0)

The #1 configuration trap in service setup:

  • Look at the Local Address in ss -tulpn:
  • Bound to 127.0.0.1:<port> (or localhost): The service will ONLY accept connections originating locally from the same host! External network connections will be dropped or refused.
  • Must be bound to 0.0.0.0:<port> (or :::<port> for IPv6): Listens on all network interfaces.
  • Fix: Update configuration file (e.g. server.host = '0.0.0.0' in app config).
Advertisement
3️⃣

Privileged Port & Permission Issues (Ports < 1024)

Linux security restrictions on well-known ports:

  • Ports below 1024 (e.g. 80, 443, 53) require root privileges to bind by default.
  • If a non-root systemd service tries to bind to port 80: It will silently fail or log Permission denied.
  • Fix without running as root: Grant the binary socket capability:
    sudo setcap 'cap_net_bind_service=+ep' /path/to/binary, or configure AmbientCapabilities=CAP_NET_BIND_SERVICE in the systemd unit file.
4️⃣

Firewall (iptables/nftables) & SELinux

Host-level security layers blocking incoming packets:

  • Host Firewall: Check sudo iptables -L -n -v, sudo ufw status, or sudo nft list ruleset. Verify incoming traffic on the port is not REJECTED or DROPPED.
  • SELinux / AppArmor: On RHEL/CentOS, SELinux prevents services from binding to non-standard ports (e.g. Nginx binding to port 8088).
  • Check SELinux: getenforce and sudo ausearch -m avc -ts recent.
  • Fix SELinux port policy: sudo semanage port -a -t http_port_t -p tcp 8088.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Check 'ss -tulpn | grep <PID>' first. If listening on 127.0.0.1, change binding to 0.0.0.0. If port < 1024, check CAP_NET_BIND_SERVICE. If bound correctly, check iptables and SELinux port policies."
⚡ 60-Second Elevator Pitch Talking Points
  • Check open sockets: 'ss -tulpn | grep <process>' or 'lsof -i :<port>' to verify if socket exists.
  • Check binding address: Verify service is bound to 0.0.0.0 (all interfaces), NOT 127.0.0.1 (localhost only).
  • Check logs: 'journalctl -u <service> -e' for bind errors, port conflicts, or permission denied.
  • Check privileged ports (<1024): Non-root users cannot bind to 80/443 without 'AmbientCapabilities=CAP_NET_BIND_SERVICE' in systemd.
  • Check firewall: Inspect 'iptables -L -n -v' or 'ufw status' for incoming port drops.
  • Check SELinux/AppArmor: Look for AVC denials in 'audit.log'; assign port context via 'semanage port'.
🎬 60-Second Video Walkthrough
Watch the Animated Breakdown & Practice Live

Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.

🔔 Subscribe on YouTube (Free) ▶ Watch Short (60s) ↗
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Issuing immediate kill -9 (SIGKILL) commands rather than allowing graceful termination via kill -15 (SIGTERM) when configuring Service Running Listening SIGKILL immediately halts execution without allowing processes to flush memory buffers, commit database transactions, or remove stale PID locks.
  2. Overlooking file system inode exhaustion (df -i) when diagnosing 'No space left on device' errors when configuring Service Running Listening Millions of small files or unrotated session tokens can exhaust available file system inodes even when standard disk storage (df -h) shows ample free space.
  3. Exhausting open file descriptor allocations (ulimit -n) under intensive concurrent socket connections when configuring Service Running Listening Reaching file descriptor exhaustion throws 'Too many open files' errors, preventing services from accepting new TCP client socket connections.
  4. Permitting defunct zombie processes to accumulate without fixing parent process SIGCHLD waitpid reaping when configuring Service Running Listening Zombie processes consume process table entries; accumulating zombies can exhaust the system maximum PID limit (pid_max).
Advertisement
Want more Linux & SRE scenarios?
Explore our complete collection of scenario-based Linux & SRE interview runbooks.
Browse All Linux & SRE Questions →