Processes, Signals, and Job Control

How Linux processes work and how to control them from Bash: ps, pgrep, top, signals, job control, surviving logout, wait, /proc, and lsof.

By The Dark DebuggerUpdated 38 min read

This is the first article in the System Administration section. It explains what a process is, how to find one, how to stop one properly, how to juggle several from one shell, and how to answer the two questions every on-call engineer asks sooner or later: "what is holding this port?" and "why is this disk still full?". You should be comfortable writing small scripts (your first scripts) and know what a trap is (error handling and traps).

Why this matters

Picture a few ordinary mornings on a Linux server:

  • The nightly import that normally takes 20 minutes is still running at 9 a.m. You need to find it, see how long it has been running, and stop it without corrupting the half-written output.
  • A deploy fails with Address already in use. Something is sitting on port 8080 and nobody knows what.
  • A two-hour backup you started over SSH died when your laptop went to sleep.
  • df says the disk is 100% full, you deleted a 40 GB log, and df still says 100%.

Every one of these is a process problem, and every one has a precise answer that takes a single command once you know the model. That model is what this article builds.

What a process is

A process is a running instance of a program. The kernel gives each one a PID (process ID), remembers which process started it (the PPID, parent process ID), and tracks its state. Bash exposes its own PID as $$ and its parent's as $PPID:

echo "my PID is $$, my parent is $PPID"
bash -c 'echo "child PID is $$, its parent is $PPID"'
my PID is 12927, my parent is 12924
child PID is 13008, its parent is 12927

The second line is the important one: running bash -c created a new process whose parent is the shell we typed into. Every command you run that is not a builtin works this way: the shell forks a child, the child runs the program, and the shell waits for it (or doesn't, if you added &).

Let's start a few background processes and look at the family tree:

sleep 300 &
(sleep 301 & sleep 302 & wait) &
ps -o pid,ppid,stat,etime,cmd --forest
    PID    PPID STAT     ELAPSED CMD
  12927   12924 Ss         00:03 bash
  13039   12927 S          00:01  \_ sleep 300
  13047   12927 S          00:00  \_ bash
  13048   13047 S          00:00  |   \_ sleep 301
  13049   13047 S          00:00  |   \_ sleep 302
  13061   12927 R+         00:00  \_ ps -o pid,ppid,stat,etime,cmd --forest

The parentheses created a subshell (PID 13047, which shows up as another bash), and the two sleep commands are its children, not ours. pstree draws the same thing more compactly:

pstree -p $$
bash(12927)─┬─bash(13047)─┬─sleep(13048)
            │             └─sleep(13049)
            ├─pstree(13129)
            └─sleep(13039)

When a parent exits before its children, the children are not killed automatically. They are re-parented ("adopted") by PID 1 or by the nearest process that registered as a subreaper (on systemd machines that is often your systemd --user instance). That is why a background job can keep running after the script that started it has finished.

Process states

The STAT column is a letter for the state plus optional flags. These are the ones you will actually see:

CodeMeaningTypical cause
RRunning or ready to runUsing CPU right now
SSleeping, interruptibleWaiting for input, a timer, a network reply
DSleeping, uninterruptibleWaiting on disk or NFS I/O; cannot be killed until the I/O returns
TStoppedSuspended with Ctrl-Z or SIGSTOP
ZZombieExited, but the parent has not collected its exit status yet
sSession leaderUsually a login shell or a daemon
+In the foreground process groupThe thing currently attached to your terminal
lMulti-threadedJava, Python servers, most daemons
< / NHigh / low priorityNegative / positive nice value

You can produce T and Z on purpose to see them. A stopped process:

sleep 100 & p=$!
kill -STOP $p; ps -o pid,stat,cmd -p $p
kill -CONT $p; ps -o pid,stat,cmd -p $p
    PID STAT CMD
  16219 T+   sleep 100
    PID STAT CMD
  16219 S+   sleep 100

A zombie needs a parent that never calls wait. exec sleep 4 replaces the shell with sleep, which has no idea it has a child:

bash -c 'sleep 1 & exec sleep 4' &
sleep 2
ps -o pid,ppid,stat,cmd --ppid $!
    PID    PPID STAT CMD
  16086   16085 Z+   [sleep] <defunct>

A zombie uses no memory or CPU; it is just an entry in the process table holding an exit code. You cannot kill a zombie (it is already dead). It disappears when its parent calls wait or exits. A handful is harmless; thousands mean a buggy parent that never reaps its children, and the fix is restarting the parent.

Listing processes with ps

ps prints a snapshot. It accepts two historical option styles, and you will see both in the wild:

ps aux | head -4
ps -ef | head -4
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.1  21736 13180 ?        Ss   21:13   0:00 /sbin/init
root           2  0.0  0.0   3804  2508 hvc0     Sl+  21:13   0:00 /init
root           3  0.0  0.0   3664  2140 hvc0     S+   21:13   0:00 init-watcher
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 21:13 ?        00:00:00 /sbin/init
root           2       1  0 21:13 hvc0     00:00:00 /init
root           3       2  0 21:13 hvc0     00:00:00 init-watcher

(The /init and init-watcher lines are WSL plumbing; on a normal server PID 2 is kthreadd.)

ps aux (BSD style)ps -ef (System V style)
SelectsAll processes (a + x)All processes (-e)
Shows memory%MEM, VSZ, RSSno
Shows parentnoPPID
Good for"what is eating memory?""who started this?"

VSZ is the virtual address space (including shared libraries and memory that was never touched); RSS is the resident set, the RAM actually in use, in KiB. When you ask "how much memory does it use", you mean RSS.

Pick your own columns with -o

For anything you read more than once, choose the columns yourself. -o takes a list, -p takes a comma-separated PID list, and pgrep -d, produces exactly that:

ps -o pid,ppid,stat,etimes,rss,args -p "$(pgrep -d, -f 'sleep 30[0-2]')"
    PID    PPID STAT ELAPSED   RSS COMMAND
  13039   12927 S          4  1964 sleep 300
  13048   13047 S          3  1964 sleep 301
  13049   13047 S          3  1964 sleep 302

Useful column names: pid, ppid, user, stat, etime (elapsed as [[dd-]hh:]mm:ss), etimes (elapsed in seconds, ideal for scripts), lstart (full start time), %cpu, %mem, rss, comm (program name, max 15 characters), args (full command line).

--sort takes column names, with - for descending. Here are the biggest memory users on the lab machine while I was writing this:

ps -eo pid,user,rss,%mem,comm --sort=-rss | head -6
    PID USER       RSS %MEM COMMAND
   1443 sid      695700  8.6 java
   9039 sid      101592  1.2 pwsh
   9090 sid      85516  1.0 terraform
   8578 sid      79116  0.9 apt-get
    202 root     23264  0.2 unattended-upgr

In scripts, put = after a column name to suppress the header, so the output is just the value:

ps -o etimes= -p $!
      4

That is how you write "alert if the import has been running longer than two hours": (( $(ps -o etimes= -p "$pid") > 7200 )).

Finding processes by name: pgrep and pkill

The habit everyone starts with is ps aux | grep something. It has a well-known flaw:

ps aux | grep 'sleep 301'
sid        13048  0.0  0.0   3136  1964 pts/8    S    21:28   0:00 sleep 301
sid        13291  0.0  0.0   4104  2164 pts/8    S+   21:28   0:00 grep sleep 301

The grep process itself matches, because its own command line contains the pattern. The classic workaround is a bracket expression that matches the text but not itself ('[s]leep 301'). The better fix is pgrep, which never matches itself and prints PIDs you can use directly:

pgrep -f 'sleep 30[0-2]'
pgrep -af 'sleep 30[0-2]'
pgrep -c -f 'sleep 30[12]'
pgrep -n -af 'sleep 30[0-2]'
pgrep -a -P $$
13039
13048
13049
13039 sleep 300
13048 sleep 301
13049 sleep 302
2
13049 sleep 302
13039 sleep 300
13047 bash
OptionEffect
(none)Match the pattern (an extended regex) against the program name only
-fMatch against the full command line instead
-aPrint the full command line next to each PID
-lPrint the program name next to each PID
-xRequire the whole name to match, not a substring
-u userOnly that user's processes
-P ppidOnly children of that parent
-n / -oOnly the newest / oldest match
-cPrint the count instead of PIDs
-d,Use , as the separator (handy for ps -p)

pgrep exits with status 0 if anything matched and 1 if nothing did, so it works in conditions:

pgrep -f 'sleep 30[12]' > /dev/null && echo "backup sleeps still running"
backup sleeps still running

pkill takes exactly the same selection options and sends a signal (SIGTERM by default) to every match. Always run the pgrep -a version first and read the list; pkill does not ask for confirmation.

Reading top

top is the live view. Its header is the fastest summary of a machine's health, so it is worth learning to read line by line. In batch mode (-b, one iteration with -n1) you can capture it:

top -bn1 -w 100 | head -12
top - 21:27:04 up 16 min,  1 user,  load average: 0.80, 0.63, 0.31
Tasks:  49 total,   1 running,  48 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  1.2 sy,  0.0 ni, 86.9 id, 11.9 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   7849.7 total,   3235.4 free,   1531.6 used,   3271.0 buff/cache     
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   6318.1 avail Mem 
 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0   21736  13180   9764 S   0.0   0.2   0:00.65 systemd
      2 root      20   0    3804   2508   2376 S   0.0   0.0   0:00.09 init-systemd(Ub
      3 root      20   0    3664   2140   2040 S   0.0   0.0   0:00.00 init-watcher
      7 root      20   0    3820   2420   2300 S   0.0   0.0   0:00.00 init
     53 root      19  -1   66848  19784  18580 S   0.0   0.2   0:01.25 systemd-journal
  • Line 1: current time, uptime, logged-in users, and the 1-, 5- and 15-minute load averages. Load is explained properly in system monitoring; the short version is "compare it with the number of CPUs (nproc)".
  • Tasks: watch the zombie count. Non-zero and growing means a parent is not reaping children.
  • %Cpu(s): us user code, sy kernel, ni niced processes, id idle, wa waiting for I/O, st stolen by the hypervisor. The 11.9 wa above was real: another process was downloading packages and the CPUs were waiting on the disk. High wa with low us means "the disk is the bottleneck, not the CPU".
  • Mem / Swap: read avail Mem, not free. Linux uses idle RAM as disk cache (buff/cache) and gives it back on demand, so free is always misleadingly small.

Interactively, the keys worth remembering are P (sort by CPU), M (sort by memory), c (show full command lines), 1 (one line per CPU), k (send a signal to a PID), u (filter by user) and q (quit). htop is a friendlier version if it is installed, but top is on every machine you will ever SSH into.

Signals

A signal is a small numbered message the kernel delivers to a process. Most signals can be caught (the process runs a handler), ignored, or left at their default action (usually "terminate"). kill -l lists them all:

kill -l
 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP
 6) SIGABRT	 7) SIGBUS	 8) SIGFPE	 9) SIGKILL	10) SIGUSR1
11) SIGSEGV	12) SIGUSR2	13) SIGPIPE	14) SIGALRM	15) SIGTERM
16) SIGSTKFLT	17) SIGCHLD	18) SIGCONT	19) SIGSTOP	20) SIGTSTP
21) SIGTTIN	22) SIGTTOU	23) SIGURG	24) SIGXCPU	25) SIGXFSZ
[…]

You only need a handful day to day:

SignalNo.Sent byDefaultCan be caught?Use it for
SIGTERM15kill PID (the default)terminateyes"Please shut down cleanly." Always try this first.
SIGINT2Ctrl-CterminateyesInterrupting a foreground command
SIGHUP1Terminal closing, kill -HUPterminateyesDaemons use it as "reload your config"
SIGKILL9kill -9terminatenoLast resort when TERM was ignored
SIGSTOP19kill -STOPstopnoFreezing a process
SIGTSTP20Ctrl-ZstopyesSuspending a foreground job
SIGCONT18fg, bg, kill -CONTcontinueyesResuming a stopped process
SIGUSR1/210/12kill -USR1terminateyesApplication-defined ("dump stats", "reopen logs")

Signal numbers above 15 differ between architectures, so use names in scripts: kill -TERM, kill -HUP, never kill -15.

A script that handles signals properly

Here is a long-running worker that reloads its config on HUP and cleans up its scratch file on the way out:

#!/usr/bin/env bash
# worker.sh - a long-running job that reacts to signals
set -u
 
scratch=$(mktemp /tmp/worker.XXXXXX)
batch_size=0
 
load_config() {
  source ./worker.conf
  echo "$(date +%T) loaded worker.conf: batch_size=$batch_size"
}
cleanup() {
  rm -f "$scratch"
  echo "$(date +%T) removed $scratch"
}
 
trap load_config HUP
trap 'echo "$(date +%T) SIGTERM: stopping"; exit 143' TERM
trap 'echo "$(date +%T) SIGINT: interrupted"; exit 130' INT
trap cleanup EXIT
 
load_config
echo "$(date +%T) worker $$ started, scratch file $scratch"
while true; do
  sleep 1 &
  wait $!
done

Start it in the background, change the config, send HUP, then TERM:

cat worker.conf
./worker.sh > worker.log 2>&1 &
echo 'batch_size=500' > worker.conf
kill -HUP $!
kill $!
wait $!; echo "exit status: $?"
cat worker.log
ls /tmp/worker.*
batch_size=100
[1] 14917
[1]+  Exit 143                ./worker.sh > worker.log 2>&1
exit status: 143
21:29:08 loaded worker.conf: batch_size=100
21:29:08 worker 14917 started, scratch file /tmp/worker.8NnvQh
21:29:11 loaded worker.conf: batch_size=500
21:29:12 SIGTERM: stopping
21:29:12 removed /tmp/worker.8NnvQh
ls: cannot access '/tmp/worker.*': No such file or directory

The HUP reloaded the config without restarting, the TERM ran the handler, and the EXIT trap removed the scratch file. Now the same thing with kill -9:

./worker.sh > worker.log 2>&1 &
kill -9 $!
wait $!; echo "exit status: $?"
cat worker.log
ls /tmp/worker.*
[1] 15391
[1]+  Killed                  ./worker.sh > worker.log 2>&1
exit status: 137
21:29:18 loaded worker.conf: batch_size=100
21:29:18 worker 15391 started, scratch file /tmp/worker.X00kKM
/tmp/worker.X00kKM

No message, no cleanup, and the temp file is left behind. That is the whole argument against reaching for kill -9 first: SIGKILL is handled by the kernel, the process never gets a chance to flush buffers, release locks, remove PID files or finish writing the file it had open. With a database or anything holding a lock file, "left behind" turns into "won't start again until someone cleans up by hand". The rule is: TERM, wait a few seconds, check, and only then KILL.

Exit codes tell you which signal it was

A process killed by signal N exits with status 128 + N. kill -l translates back:

kill -l 143 137 130
TERM
KILL
INT

When a cron job or CI step "failed with 137", that is SIGKILL, very often from the kernel's out-of-memory killer or a container runtime enforcing a memory limit. 143 is a polite TERM (a timeout or a systemctl stop), 130 is someone pressing Ctrl-C.

Bash only runs a trap between commands

This one confused me the first time. Bash does not interrupt a foreground command to run a trap; it waits for the command to finish:

#!/usr/bin/env bash
# slow-trap.sh
trap 'echo "$(date +%T) caught SIGTERM"; exit 143' TERM
echo "$(date +%T) started, PID $$"
sleep 5
bash slow-trap.sh & sleep 0.5; echo "$(date +%T) sending SIGTERM"; kill $!; wait $!
21:29:33 started, PID 15980
21:29:34 sending SIGTERM
21:29:38 caught SIGTERM

Four seconds of delay, because the trap waited for sleep 5. With a long rsync in the foreground, that could be an hour. That is why worker.sh uses sleep 1 & wait $!: the wait builtin is interrupted by signals, so the handler runs immediately.

Escalating automatically with timeout

timeout runs a command with a time limit. -k adds the escalation step: send TERM at the limit, then KILL if the process is still alive after the grace period.

#!/usr/bin/env bash
# stubborn.sh - ignores SIGTERM, like a process wedged in a bad state
trap 'echo "$(date +%T) got SIGTERM, ignoring it"' TERM
while true; do sleep 1; done
time timeout -k 3 2 ./stubborn.sh; echo "exit: $?"
Terminated
21:40:32 got SIGTERM, ignoring it
Killed
 
real	0m5.008s
user	0m0.003s
sys	0m0.000s
exit: 137

At 2 seconds timeout sent SIGTERM. The Terminated line is the inner sleep 1 dying (timeout signals the whole process group), then the script's trap printed its defiant message and kept looping. Three seconds later came SIGKILL, which cannot be trapped, and the total was 5 seconds with exit status 137. timeout -k is the simplest way to put a hard ceiling on a cron job or CI step that sometimes hangs.

Job control: &, jobs, fg, bg and Ctrl-Z

An interactive shell can run one foreground job (attached to your keyboard) and any number of background jobs. Job control is easiest to understand as a terminal session, so here is one, typed exactly like this (^Z and ^C are Ctrl-Z and Ctrl-C):

$ sleep 600
^Z
[1]+  Stopped                 sleep 600
$ jobs
[1]+  Stopped                 sleep 600
$ bg
[1]+ sleep 600 &
$ jobs -l
[1]+ 13624 Running                 sleep 600 &
$ sleep 700 &
[2] 13845
$ jobs
[1]-  Running                 sleep 600 &
[2]+  Running                 sleep 700 &
$ fg %1
sleep 600
^C
$ jobs
[2]+  Running                 sleep 700 &
$ kill %2
$ jobs
[2]+  Terminated              sleep 700
ActionWhat it does
cmd &Start in the background; prints [job] PID
Ctrl-ZSend SIGTSTP to the foreground job: it stops (state T)
jobs / jobs -lList this shell's jobs (-l adds PIDs)
bg [%n]Resume a stopped job in the background (SIGCONT)
fg [%n]Bring a job to the foreground, resuming it if stopped
kill %nSignal a job by job number instead of PID
%+ / %-The current (+) and previous (-) job; fg with no argument uses %+

The everyday use is the "oops" move: you start something that takes longer than expected (find / or a big tar), press Ctrl-Z, type bg, and get your prompt back without losing the work.

Job numbers (%1) belong to one interactive shell. Scripts run without job control, so in a script you use PIDs ($!) and wait, covered below.

Keeping processes alive after logout

When your terminal goes away (you close the window, SSH drops, the laptop sleeps), the shell receives SIGHUP. Bash then forwards SIGHUP to every job in its job table, and the default action of SIGHUP is to terminate. I tested the four common ways to protect a job by starting one of each and then sending the shell the same SIGHUP a dropped SSH connection would:

$ sleep 1001 &
[1] 14064
$ nohup sleep 1002 &
[2] 14072
$ nohup: ignoring input and appending output to 'nohup.out'
 
$ sleep 1003 &
[3] 14077
$ disown %3
$ setsid -f sleep 1004
$ jobs -l
[1]- 14064 Running                 sleep 1001 &
[2]+ 14072 Running                 nohup sleep 1002 &
$ pgrep -af 'sleep 100'
14064 sleep 1001
14072 sleep 1002
14077 sleep 1003
14159 sleep 1004
$ echo "shell is $$"; kill -HUP $$
shell is 14024

Then, from another shell:

pgrep -af 'sleep 100'
14072 sleep 1002
14077 sleep 1003
14159 sleep 1004

The plain background job died; the other three survived. What each one does:

  • nohup cmd & makes the command ignore SIGHUP and, if stdout is a terminal, appends output to nohup.out. Redirect it yourself instead: nohup ./import.sh > import.log 2>&1 &.
  • disown %n removes an already-running job from the shell's job table, so the shell never forwards SIGHUP to it. Use it when you forgot nohup. (disown -h keeps the job listed but still skips the SIGHUP.)
  • setsid -f cmd starts the command in a brand-new session with no controlling terminal at all. It is the most thorough of the three.
  • tmux or screen keep a whole terminal session alive on the server. For interactive work you want to reattach to later (a long migration you are watching), this is the most practical answer.

For anything that must run reliably, none of these is the real answer: they give you no restart on failure, no logs beyond a file, and nothing survives a reboot. The modern answer is to let systemd own the process. systemd-run does that for a one-off command without writing a unit file:

systemd-run --user --unit=report-build sleep 600
systemctl --user status report-build --no-pager --lines=0
Running as unit: report-build.service; invocation ID: 27c36475b161402680cf63bba865096d
● report-build.service - /usr/bin/sleep 600
     Loaded: loaded (/run/user/1000/systemd/transient/report-build.service; transient)
  Transient: yes
     Active: active (running) since Mon 2026-11-09 21:45:52 EST; 1s ago
   Main PID: 16209 (sleep)
      Tasks: 1 (limit: 9408)
     Memory: 612.0K (peak: 1.7M)
        CPU: 2.334s
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/report-build.service
             └─16209 /usr/bin/sleep 600

The process now belongs to systemd, not to your shell, so closing the terminal does not touch it. Its output goes to the journal (journalctl --user -u report-build), and you stop it with systemctl --user stop report-build (systemd sends SIGTERM, waits, then SIGKILL, exactly the escalation described above). A transient unit disappears once it stops. One catch: a user's systemd instance normally stops when that user's last session ends. To keep --user units running after logout, an administrator enables lingering with loginctl enable-linger username. For services that need to run as a system account, write a proper unit; timers and units are covered in system monitoring and scheduling.

Waiting for background work: $! and wait

In a script, $! holds the PID of the most recent background command and wait PID blocks until that process exits, then returns its exit status. That combination is how you run things in parallel without losing track of failures:

#!/usr/bin/env bash
# parallel-archive.sh - archive several directories at once, report each result
dirs=(invoices images reports contracts)
pids=()
 
for dir in "${dirs[@]}"; do
  tar -czf "$dir.tar.gz" "$dir" 2> "$dir.err" &
  pids+=("$!")
  echo "started $dir as PID $!"
done
 
failed=0
for i in "${!dirs[@]}"; do
  if wait "${pids[i]}"; then
    echo "ok    ${dirs[i]}"
  else
    rc=$?
    echo "FAIL  ${dirs[i]} (exit $rc): $(head -1 "${dirs[i]}.err")"
    failed=1
  fi
done
rm -f ./*.err
exit "$failed"
bash parallel-archive.sh; echo "script exit: $?"
ls -l *.tar.gz
started invoices as PID 10411
started images as PID 10412
started reports as PID 10414
started contracts as PID 10416
ok    invoices
ok    images
ok    reports
FAIL  contracts (exit 2): tar: contracts: Cannot stat: No such file or directory
script exit: 1
-rw-r--r-- 1 sid sid     45 Nov  9 21:42 contracts.tar.gz
-rw-r--r-- 1 sid sid   3198 Nov  9 21:42 images.tar.gz
-rw-r--r-- 1 sid sid   1674 Nov  9 21:42 invoices.tar.gz
-rw-r--r-- 1 sid sid 428702 Nov  9 21:42 reports.tar.gz

Two lessons in that output. First, a bare wait with no arguments always returns 0, so you must wait on each PID to see failures. Second, look at contracts.tar.gz: tar created a 45-byte archive before discovering the directory did not exist. A failed job can leave partial output behind, so a real script deletes the output on failure (the backup script writes to a temporary name and only renames on success).

Limiting parallelism with wait -n

Starting 500 jobs at once will flatten a machine. wait -n waits for whichever background job finishes next, which makes a simple job pool:

#!/usr/bin/env bash
# throttle.sh - process files with at most 3 jobs running at once
max_jobs=3
for n in 1 2 3 4 5 6 7; do
  while (( $(jobs -rp | wc -l) >= max_jobs )); do
    wait -n
  done
  ( sleep "0.$((RANDOM % 9 + 1))"; echo "$(date +%T.%3N) finished task $n" ) &
done
wait
echo "all tasks done"
bash throttle.sh
21:25:26.756 finished task 3
21:25:26.859 finished task 1
21:25:27.053 finished task 2
21:25:27.168 finished task 4
21:25:27.762 finished task 6
21:25:27.770 finished task 5
21:25:27.776 finished task 7
all tasks done

jobs -rp lists the PIDs of running jobs, so the inner loop blocks while three are active. Tasks finish out of order, which is the point. wait -n returns the exit status of the job that finished, and since Bash 5.1 wait -n -p var also stores that job's PID in var, so a pool can report exactly which task failed.

/proc: every process as a directory

On Linux, /proc/PID/ is a live view of a process, generated by the kernel when you read it. Tools like ps and top are mostly readers of these files, and you can read them directly when you need one specific fact:

./worker.sh > worker.log 2>&1 &
pid=$!
grep -E '^(Name|State|PPid|VmRSS|Threads)' /proc/$pid/status
tr '\0' ' ' < /proc/$pid/cmdline; echo
ls -l /proc/$pid/cwd /proc/$pid/exe
ls -l /proc/$pid/fd
tr '\0' '\n' < /proc/$pid/environ | grep -E '^(HOME|LANG)='
cat /proc/loadavg
[1] 15653
Name:	bash
State:	S (sleeping)
PPid:	15615
VmRSS:	    4036 kB
Threads:	1
bash ./worker.sh 
lrwxrwxrwx 1 sid sid 0 Nov  9 21:29 /proc/15653/cwd -> /home/sid/proc/signals
lrwxrwxrwx 1 sid sid 0 Nov  9 21:29 /proc/15653/exe -> /usr/bin/bash
total 0
lrwx------ 1 sid sid 64 Nov  9 21:29 0 -> /dev/pts/9
l-wx------ 1 sid sid 64 Nov  9 21:29 1 -> /home/sid/proc/signals/worker.log
l-wx------ 1 sid sid 64 Nov  9 21:29 2 -> /home/sid/proc/signals/worker.log
lr-x------ 1 sid sid 64 Nov  9 21:29 255 -> /home/sid/proc/signals/worker.sh
[…]
HOME=/home/sid
LANG=C.UTF-8
5.10 2.31 0.99 4/470 15958
PathWhat it tells you
/proc/PID/statusName, state, parent, user IDs, memory (VmRSS), thread count
/proc/PID/cmdlineThe exact argument list, separated by NUL bytes (hence the tr)
/proc/PID/environThe environment it started with (also NUL-separated)
/proc/PID/cwdSymlink to its current directory
/proc/PID/exeSymlink to the binary actually running
/proc/PID/fd/One symlink per open file descriptor: files, sockets, pipes
/proc/loadavgLoad averages, running/total tasks, last PID used

Two practical notes. environ is a snapshot from process start, so it answers "what PATH did cron give this job?", not "what is it set to now". And you can only read the fd, environ and cwd of your own processes unless you are root.

What is holding this port or this file?

Start a small web server, then ask the system who owns its port:

python3 -m http.server 18081 --bind 127.0.0.1 --directory site > http.log 2>&1 &
ss -ltnp 'sport = :18081'
lsof -nP -iTCP:18081 -sTCP:LISTEN
lsof -t -iTCP:18081 -sTCP:LISTEN
fuser -v -n tcp 18081
[1] 8983
State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
LISTEN 0      5          127.0.0.1:18081      0.0.0.0:*    users:(("python3",pid=8983,fd=3))
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python3 8983  sid    3u  IPv4  35812      0t0  TCP 127.0.0.1:18081 (LISTEN)
8983
                     USER        PID ACCESS COMMAND
18081/tcp:           sid        8983 F.... python3
  • ss -ltnp: listening, tCP, numeric, show process. The filter 'sport = :18081' limits it to one port.
  • lsof -nP -iTCP:PORT -sTCP:LISTEN lists the process with the listening socket. Always pass -nP: without them lsof resolves addresses and port numbers to names. When I first ran this demo on port 8081, lsof printed localhost:tproxy because 8081 is registered as "tproxy" in /etc/services.
  • lsof -t prints only PIDs, made for scripts: kill $(lsof -t -iTCP:18081 -sTCP:LISTEN).
  • fuser is the older tool; fuser -k would kill the owner, which is too blunt to recommend.

Without root, ss -p and lsof only show processes you own. On a server, put sudo in front to see everyone's.

The deleted file that still fills the disk

On Linux, deleting a file removes its name. The data is only freed when the last process holding it open closes it. Here is a stand-in for a service that keeps its log open:

#!/usr/bin/env bash
# chatty-app.sh - stand-in for a service that keeps its log file open
exec 3>>logs/app.log
head -c 20M /dev/zero >&3          # pretend weeks of logging already happened
while true; do
  echo "$(date +%T) heartbeat" >&3
  sleep 1
done
./chatty-app.sh &
ls -lh logs/
rm logs/app.log
ls -A logs/
lsof -a +L1 -p $!
[1] 9914
total 21M
-rw-r--r-- 1 sid sid 21M Nov  9 21:42 app.log
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NLINK  NODE NAME
bash    9914  sid    3w   REG   8,48 20971634     0 56374 /home/sid/proc/logs/app.log (deleted)

The directory is empty, but the 20 MB are still allocated: lsof +L1 lists open files with a link count below 1, i.e. deleted but open. That is the answer to "I deleted the log and df didn't change". The proper fix is to restart or signal the process so it reopens its log. If you cannot restart it, truncate the file through the process's own descriptor:

: > /proc/$!/fd/3
lsof -a +L1 -p $!
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NLINK  NODE NAME
bash    9914  sid    3w   REG   8,48       19     0 56374 /home/sid/proc/logs/app.log (deleted)

From 20 MB to 19 bytes (one heartbeat line). For the future, truncate big logs in place (: > app.log) instead of deleting them, or use logrotate's copytruncate for programs that never reopen their logs.

Where you'll use this

The most common real-world use of everything above is a small control script for a process that has no systemd unit yet: a vendor tool, an internal API, a dev server on a shared box. It needs to start the process detached, remember its PID, report status honestly, and stop it without kill -9 unless it has to. Here is a complete one for a Python web server:

#!/usr/bin/env bash
# appctl - start, stop and check a background web server through a PID file
set -u
 
APP_DIR=${APP_DIR:-$PWD/site}
PORT=${PORT:-18081}
RUN_DIR=${RUN_DIR:-$PWD/run}
PIDFILE=$RUN_DIR/app.pid
LOG=$RUN_DIR/app.log
 
is_running() {
  local pid
  [[ -f $PIDFILE ]] || return 1
  pid=$(<"$PIDFILE")
  kill -0 "$pid" 2>/dev/null || return 1
  # PIDs are recycled: confirm this PID is still our server, not a stranger
  tr '\0' ' ' < "/proc/$pid/cmdline" | grep -q "http.server $PORT"
}
 
start() {
  local pid i
  if is_running; then
    echo "already running (PID $(<"$PIDFILE"))"
    return 0
  fi
  mkdir -p "$RUN_DIR"
  nohup python3 -m http.server "$PORT" --bind 127.0.0.1 \
    --directory "$APP_DIR" >> "$LOG" 2>&1 &
  pid=$!
  echo "$pid" > "$PIDFILE"
  # Wait up to 5s until our PID owns the listening socket (or dies trying)
  for ((i = 0; i < 50; i++)); do
    kill -0 "$pid" 2>/dev/null || break
    if [[ $(lsof -t -nP -iTCP:"$PORT" -sTCP:LISTEN 2>/dev/null) == "$pid" ]]; then
      echo "started (PID $pid) on port $PORT"
      return 0
    fi
    sleep 0.1
  done
  echo "failed to start; last log line:" >&2
  tail -n 1 "$LOG" >&2
  kill "$pid" 2>/dev/null
  rm -f "$PIDFILE"
  return 1
}
 
stop() {
  local timeout=${1:-10} pid i
  if ! is_running; then
    echo "not running"
    rm -f "$PIDFILE"
    return 0
  fi
  pid=$(<"$PIDFILE")
  kill -TERM "$pid"
  for ((i = 0; i < timeout * 10; i++)); do
    if ! kill -0 "$pid" 2>/dev/null; then
      echo "stopped PID $pid with SIGTERM"
      rm -f "$PIDFILE"
      return 0
    fi
    sleep 0.1
  done
  echo "PID $pid ignored SIGTERM for ${timeout}s, sending SIGKILL" >&2
  kill -KILL "$pid"
  rm -f "$PIDFILE"
}
 
status() {
  if is_running; then
    echo "running (PID $(<"$PIDFILE"))"
  else
    echo "stopped"
    return 3
  fi
}
 
case ${1:-} in
  start)   start ;;
  stop)    stop "${2:-10}" ;;
  restart) stop && start ;;
  status)  status ;;
  *)       echo "usage: appctl {start|stop [timeout]|restart|status}" >&2; exit 2 ;;
esac

A full run, including the two situations that break naive scripts: a stale PID file pointing at a process that no longer exists, and the port already being taken.

./appctl status
./appctl start
./appctl start
curl -s http://127.0.0.1:18081/
./appctl status; echo "exit: $?"
./appctl restart
./appctl stop
./appctl status; echo "exit: $?"
stopped
started (PID 11214) on port 18081
already running (PID 11214)
<h1>Order status</h1>
running (PID 11214)
exit: 0
stopped PID 11214 with SIGTERM
started (PID 11345) on port 18081
stopped PID 11345 with SIGTERM
stopped
exit: 3
echo 99999 > run/app.pid
./appctl status
python3 -m http.server 18081 --bind 127.0.0.1 > /dev/null 2>&1 &
./appctl start
./appctl bogus; echo "exit: $?"
stopped
[1] 11473
failed to start; last log line:
OSError: [Errno 98] Address already in use
usage: appctl {start|stop [timeout]|restart|status}
exit: 2

My first version of start did the obvious thing: sleep 0.5, then check that the PID was alive. During one test run of the port-conflict case it printed started (PID 9641) on port 18081 for a server that was about to die with Address already in use. Python had simply taken longer than half a second to start and fail. "The process exists" is not the same as "the service works", which is why the version above waits until its own PID owns the listening socket. The capstone goes one step further and checks an HTTP health endpoint.

The design decisions worth copying:

  • kill -0 checks existence without sending anything. It returns 0 if the process exists and you may signal it.
  • The PID file is verified, not trusted. PIDs are reused, so after a crash and a reboot your PID file may point at someone else's process. Checking /proc/PID/cmdline prevents stop from killing a stranger.
  • stop escalates. TERM, poll for up to timeout seconds, then KILL, and it says which one happened.
  • status uses exit code 3 for "not running", the convention init scripts use, so other scripts can test it.

The capstone deployment toolkit reuses this pattern to restart an app after each release.

Common mistakes

pkill -f kills the shell that ran it

You want a cleanup command to stop a job, so you run pkill -f from inside bash -c (or over ssh host '…', which is the same thing):

#!/usr/bin/env bash
# pkill-self.sh
sleep 120 &
bash -c 'pkill -f "sleep 120"; echo "cleanup finished"'
echo "exit status: $?"
bash pkill-self.sh
Terminated
exit status: 143

"cleanup finished" never printed. pkill excludes itself, but not its parent: the bash -c process's own command line contains the text sleep 120, so pkill -f killed the shell that was running it (exit 143, SIGTERM). Anchor the pattern so it can only match the real target. With the middle line changed to

bash -c 'pkill -f "^sleep 120$"; echo "cleanup finished"'

the script prints:

cleanup finished
exit status: 0

The same trap catches ssh server "pkill -f myapp" and cron lines that mention the process they are trying to kill. Run pgrep -af with the exact pattern first; if the list includes your own shell, fix the pattern.

pgrep finds nothing for a name you can see

pgrep systemd-timesyncd
pgrep -l systemd-timesyn
pgrep: pattern that searches for process name longer than 15 characters will result in zero matches
Try `pgrep -f' option to match against the complete command line.
159 systemd-timesyn

The kernel stores only the first 15 characters of a program's name (comm). Recent pgrep versions warn you; older ones silently return nothing. Scripts have the same problem in a different form: ./nightly-invoice-export.sh with a #!/usr/bin/env bash line runs as a process named bash, so pgrep nightly finds nothing. For scripts and long names, match the full command line with pgrep -f.

Address already in use

    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use

Every server prints its own version of this (EADDRINUSE, bind() failed). It means another process is listening on that address and port, very often an old copy of the same app that a previous deploy failed to stop. Don't guess and don't restart random things: run ss -ltnp 'sport = :PORT' or lsof -nP -iTCP:PORT -sTCP:LISTEN, read the command line of the owner, and stop it gracefully.

Summary

  • Every process has a PID, a parent (PPID) and a state; $$, $PPID and $! give you the shell's, its parent's and the last background job's PIDs.
  • ps -o pid,ppid,stat,etimes,args with -p, -C or --sort beats ps aux | grep; ps -o col= drops the header for scripts. pgrep -af finds by full command line and never matches itself.
  • In top, read load against nproc, wa for I/O waits and avail Mem instead of free.
  • SIGTERM asks, SIGKILL forces. Killed processes exit with 128 + signal number (143 = TERM, 137 = KILL, 130 = INT). Bash runs traps only between foreground commands, so use cmd & wait $! in long-running scripts.
  • Ctrl-Z stops a foreground job; bg, fg and jobs manage it. On logout, jobs get SIGHUP unless started with nohup/setsid or disowned; for real services use systemd.
  • wait PID returns that job's exit status; wait -n builds a simple job pool.
  • /proc/PID/ exposes a process's command line, environment, cwd and open files; ss -ltnp and lsof -nP -i answer "who owns this port"; lsof +L1 finds deleted files that still occupy disk space.

Exercises

Exercise 1: Top three memory users in MiBeasy

Print the three processes using the most resident memory as PID RSS_MiB COMMAND, with RSS converted from KiB to MiB with one decimal place.

Hint

ps -eo pid,rss,comm --sort=-rss does the sorting; awk can skip the header and divide.

Show Solution
ps -eo pid,rss,comm --sort=-rss | awk 'NR == 1 { print "PID RSS_MiB COMMAND"; next }
  NR <= 4 { printf "%s %.1f %s\n", $1, $2 / 1024, $3 }'
PID RSS_MiB COMMAND
1656 517.5 java
650 114.9 pwsh
5940 112.6 pwsh

Line 1 of ps output is the header, so awk replaces it and then prints lines 2 to 4. ps reports rss in KiB, hence the division by 1024.

Exercise 2: Rescue a slow foreground commandeasy

Start sleep 500 in the foreground. Without pressing Ctrl-C, get your prompt back while it keeps running, confirm with jobs -l that it is running in the background, then terminate it using its job number.

Show Solution
$ sleep 500
^Z
[1]+  Stopped                 sleep 500
$ bg %1
[1]+ sleep 500 &
$ jobs -l
[1]+  5991 Running                 sleep 500 &
$ kill %1
$ jobs
[1]+  Terminated              sleep 500

Ctrl-Z stops the job (SIGTSTP), bg resumes it in the background with SIGCONT, and kill %1 sends SIGTERM by job number. The final jobs reports the termination once and then forgets the job.

Exercise 3: A status signal for worker.shmedium

Extend worker.sh so that kill -USR1 PID makes it print how many loop iterations it has completed so far, without stopping. Test it by sending USR1 twice and then TERM.

Show Solution

Add a counter, a report function and a USR1 trap; the loop increments the counter:

iterations=0
 
report() {
  echo "$(date +%T) status: $iterations iterations, batch_size=$batch_size"
}
 
trap report USR1
 
# … the other traps and load_config call stay the same …
 
while true; do
  sleep 1 &
  wait $!
  iterations=$((iterations + 1))
done

Test script:

./worker.sh &
pid=$!
sleep 3.5; kill -USR1 "$pid"
sleep 2;   kill -USR1 "$pid"
sleep 0.5; kill -TERM "$pid"
wait "$pid"; echo "exit status: $?"
21:41:15 loaded worker.conf: batch_size=100
21:41:15 worker 6082 started, scratch file /tmp/worker.QImFwA
21:41:19 status: 3 iterations, batch_size=100
21:41:20 status: 5 iterations, batch_size=100
21:41:20 SIGTERM: stopping
21:41:20 removed /tmp/worker.QImFwA
exit status: 143

Because the loop waits with wait $!, the USR1 handler runs immediately rather than at the end of the current sleep. This "report status on USR1" pattern is common in real daemons (dd prints its progress on USR1, for example).

Exercise 4: Who owns this port?medium

Write whoport.sh PORT that prints the PID, user and full command line of the process listening on that TCP port, or nothing is listening on PORT with exit status 1.

Show Solution
#!/usr/bin/env bash
# whoport.sh PORT - show the process listening on a TCP port
set -u
port=${1:?usage: whoport.sh PORT}
 
pids=$(lsof -t -nP -iTCP:"$port" -sTCP:LISTEN 2>/dev/null) || true
if [[ -z $pids ]]; then
  echo "nothing is listening on $port" >&2
  exit 1
fi
for pid in $pids; do
  ps -o pid=,user=,args= -p "$pid"
done

With a test server on port 18082:

./whoport.sh 18082; echo "exit: $?"
./whoport.sh 18083; echo "exit: $?"
   6206 sid      python3 -m http.server 18082 --bind 127.0.0.1
exit: 0
nothing is listening on 18083
exit: 1

lsof exits with 1 when it finds nothing, which would trip set -e style scripts, hence || true and the explicit empty check. The loop handles servers with several listening processes (pre-forked workers). On a real server, run it with sudo to see other users' processes.

Exercise 5: stop_gracefullyhard

Write a function stop_gracefully PID [TIMEOUT] that sends SIGTERM, waits up to TIMEOUT seconds (default 5) for the process to exit, escalates to SIGKILL if needed, and returns 0 if TERM was enough, 2 if it had to use KILL, and 1 if the PID does not exist. Test it against worker.sh and stubborn.sh running detached.

Show Solution
#!/usr/bin/env bash
# stop-gracefully.sh - TERM first, KILL only when the process will not exit
 
# stop_gracefully PID [TIMEOUT]
# returns 0 if SIGTERM was enough, 2 if SIGKILL was needed, 1 if no such process
stop_gracefully() {
  local pid=$1 timeout=${2:-5} waited=0
  if ! kill -0 "$pid" 2>/dev/null; then
    echo "PID $pid is not running" >&2
    return 1
  fi
  kill -TERM "$pid"
  while kill -0 "$pid" 2>/dev/null; do
    if (( waited >= timeout * 10 )); then
      echo "PID $pid still alive after ${timeout}s, sending SIGKILL" >&2
      kill -KILL "$pid"
      return 2
    fi
    sleep 0.1
    waited=$((waited + 1))
  done
  echo "PID $pid exited $((waited / 10)).$((waited % 10))s after SIGTERM"
}
 
stop_gracefully "$@"

Test:

setsid -f ./worker.sh > worker.log 2>&1
setsid -f ./stubborn.sh > stubborn.log 2>&1
sleep 1
./stop-gracefully.sh "$(pgrep -f 'bash ./worker.sh')"; echo "returned $?"
./stop-gracefully.sh "$(pgrep -f 'bash ./stubborn.sh')" 3; echo "returned $?"
./stop-gracefully.sh 99999; echo "returned $?"
PID 6294 exited 0.1s after SIGTERM
returned 0
PID 6296 still alive after 3s, sending SIGKILL
returned 2
PID 99999 is not running
returned 1

The loop counts tenths of a second so the timeout is accurate without bc. Returning distinct codes lets a caller log "had to force-kill", which is worth knowing: a service that regularly ignores SIGTERM has a shutdown bug someone should fix.


Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), procps-ng 4.0.4, lsof 4.95, November 2026. Every command was run in the lab as shown; the job-control and logout transcripts were typed into a real interactive Bash on a pseudo-terminal, with Ctrl-Z and Ctrl-C sent as keystrokes. The machine was shared with other workloads, which is why java, pwsh and other processes appear in the system-wide listings. loginctl enable-linger needs administrator rights and was not run. I/O-blocked D state processes are described but not demonstrated, because producing one on purpose needs a hung disk or NFS mount. Paths from the test machine's scratch directories are shown shortened to /home/sid/….

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.