Linux Processes Without the Headache

I spent years killing processes with kill -9 like some kind of barbarian. Then I actually read the man pages. Turns out Linux gives you a whole toolbox for managing processes, and most of it is way simpler than I thought ( if you ignore the weird signal names ).

Here is what I actually use day to day, on servers that I care about and on my own machines.

ps Is Not Your Only Option

Everyone starts with ps aux. Fine. It works. But after a while you want something that actually updates in real time.

My goto stack for checking what is running:

# Quick snapshot
ps aux --sort=-%mem | head -20

# Real-time view ( way better than top )
htop

# Process tree - shows parent/child relationships
pstree -p -s

# Find who is eating your memory
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head

# Specific process by name
pgrep -af nginx

The thing I wish someone told me earlier: pgrep and pkill exist. Stop piping ps through grep and then cutting the PID with awk. Just use the right tool.

Signals Are Not Just Kill -9

The signal system in Linux is simple but most people only know SIGKILL ( 9 ) and SIGTERM ( 15 ). There are more, and some of them matter.

# Graceful stop ( let the process clean up )
kill -TERM <pid>
# or just
kill <pid>  # SIGTERM is the default

# Force kill ( no cleanup, last resort )
kill -9 <pid>

# Reload config without stopping
kill -HUP <pid>

# Pause a process
kill -STOP <pid>

# Resume a paused process
kill -CONT <pid>

# Kill by name ( all nginx workers, not just one )
pkill -f nginx

# Kill everything in a process group
kill -- -<pgid>

Rule I follow: always try SIGTERM first. Wait 5 seconds. If the process is still there, then SIGKILL. Running kill -9 on everything is how you get corrupted databases and half-written files. I learned that the expensive way.

Background, Foreground, and Keeping Processes Alive

Running something in the background is not just adding & to the command. That works until your SSH session drops and everything dies.

# Run in background ( dies when terminal closes )
./script.sh &

# Keep running after terminal closes
nohup ./script.sh &

# Detach a job you already started
./long-task
# press Ctrl+Z to suspend
bg          # resume in background
disown -h   # separate from terminal

# List background jobs
jobs -l

# Bring job 2 to foreground
fg %2

If you are running long tasks on servers, use tmux or screen. Way more flexible and you can reattach later. I wrote about my tmux setup before, it is still my most used tool.

Nice and Renice ( Because Some Processes Are Rude )

When a backup job is eating all your CPU and the web server is crawling, nice and renice are the answer. Not terminating it. Just making it polite.

# Start a process with low priority ( nice value 19 = lowest )
nice -n 19 ./backup.sh

# Change priority of running process
renice -n 10 -p 12345

# Make your database higher priority
renice -n -5 -p 

# Check current nice values
ps -eo pid,ni,cmd | head

Nice values go from -20 ( highest priority ) to 19 ( lowest ). Default is 0. Most things run at 0 and that is fine until it is not. I set my backup scripts to 15 and let the important stuff run at default.

When Processes Go Rogue: OOM and Beyond

Linux has an OOM ( Out Of Memory ) process terminator. When memory runs out, the kernel picks a process and stops it. Usually the one using the most memory. Sometimes that is your database. That is bad.

# Check OOM scores ( higher = more likely to be stopped )
cat /proc//oom_score

# Protect important processes ( never terminate )
echo -1000 > /proc//oom_score_adj

# Make a process first to be stopped
echo 1000 > /proc//oom_score_adj

# See recent OOM terminations
dmesg | grep -i oom
journalctl -k | grep -i oom

On every production server I set up, I protect the database process with oom_score_adj = -1000. Do not skip this. One OOM event on a Sunday and you will remember.

Process Limits That Actually Matter

ulimit is one of those things nobody sets until something breaks. Here are the limits I configure on every server:

# Check current limits for a process
cat /proc//limits

# Set limits in /etc/security/limits.conf
# username  soft/hard  resource  value

www-data  soft  nofile  65535
www-data  hard  nofile  65535
postgres  soft  nofile  65535
postgres  hard  nofile  65535

# For systemd services, use LimitNOFILE instead
# /etc/systemd/system/nginx.service.d/override.conf
[Service]
LimitNOFILE=65535
LimitNPROC=4096

The default open file limit is 1024 on most distros. If you run a busy web server or database, you will hit that ceiling fast. Set it to 65535 and stop worrying about it.

Conclusion

Process management on Linux is not complicated. The tools are there. pgrep instead of piping ps. -TERM before -9. Set your OOM protection. Raise your file limits. Use htop instead of top.

That is it. No magic, no 500-line scripts. Just the stuff that keeps my servers running without 3 AM alerts.