Processes, Signals, and Job Control
How Linux processes work and how to control them from Bash: ps, pgrep, top, signals, job control, surviving logout, wait, /proc, and lsof.
By The Dark DebuggerUpdated 38 min read
This is the first article in the System Administration section. It explains what a
process is, how to find one, how to stop one properly, how to juggle several from one
shell, and how to answer the two questions every on-call engineer asks sooner or later:
"what is holding this port?" and "why is this disk still full?". You should be
comfortable writing small scripts (your first scripts)
and know what a trap is (error handling and traps).
Why this matters
Picture a few ordinary mornings on a Linux server:
- The nightly import that normally takes 20 minutes is still running at 9 a.m. You need to find it, see how long it has been running, and stop it without corrupting the half-written output.
- A deploy fails with
Address already in use. Something is sitting on port 8080 and nobody knows what. - A two-hour backup you started over SSH died when your laptop went to sleep.
dfsays the disk is 100% full, you deleted a 40 GB log, anddfstill says 100%.
Every one of these is a process problem, and every one has a precise answer that takes a single command once you know the model. That model is what this article builds.
What a process is
A process is a running instance of a program. The kernel gives each one a PID
(process ID), remembers which process started it (the PPID, parent process ID), and
tracks its state. Bash exposes its own PID as $$ and its parent's as $PPID:
echo "my PID is $$, my parent is $PPID"
bash -c 'echo "child PID is $$, its parent is $PPID"'my PID is 12927, my parent is 12924
child PID is 13008, its parent is 12927The second line is the important one: running bash -c created a new process whose
parent is the shell we typed into. Every command you run that is not a builtin works
this way: the shell forks a child, the child runs the program, and the shell waits for
it (or doesn't, if you added &).
Let's start a few background processes and look at the family tree:
sleep 300 &
(sleep 301 & sleep 302 & wait) &
ps -o pid,ppid,stat,etime,cmd --forest PID PPID STAT ELAPSED CMD
12927 12924 Ss 00:03 bash
13039 12927 S 00:01 \_ sleep 300
13047 12927 S 00:00 \_ bash
13048 13047 S 00:00 | \_ sleep 301
13049 13047 S 00:00 | \_ sleep 302
13061 12927 R+ 00:00 \_ ps -o pid,ppid,stat,etime,cmd --forestThe parentheses created a subshell (PID 13047, which shows up as another bash), and
the two sleep commands are its children, not ours. pstree draws the same thing more
compactly:
pstree -p $$bash(12927)─┬─bash(13047)─┬─sleep(13048)
│ └─sleep(13049)
├─pstree(13129)
└─sleep(13039)When a parent exits before its children, the children are not killed automatically.
They are re-parented ("adopted") by PID 1 or by the nearest process that registered as
a subreaper (on systemd machines that is often your systemd --user instance). That is
why a background job can keep running after the script that started it has finished.
Process states
The STAT column is a letter for the state plus optional flags. These are the ones you
will actually see:
| Code | Meaning | Typical cause |
|---|---|---|
R | Running or ready to run | Using CPU right now |
S | Sleeping, interruptible | Waiting for input, a timer, a network reply |
D | Sleeping, uninterruptible | Waiting on disk or NFS I/O; cannot be killed until the I/O returns |
T | Stopped | Suspended with Ctrl-Z or SIGSTOP |
Z | Zombie | Exited, but the parent has not collected its exit status yet |
s | Session leader | Usually a login shell or a daemon |
+ | In the foreground process group | The thing currently attached to your terminal |
l | Multi-threaded | Java, Python servers, most daemons |
< / N | High / low priority | Negative / positive nice value |
You can produce T and Z on purpose to see them. A stopped process:
sleep 100 & p=$!
kill -STOP $p; ps -o pid,stat,cmd -p $p
kill -CONT $p; ps -o pid,stat,cmd -p $p PID STAT CMD
16219 T+ sleep 100
PID STAT CMD
16219 S+ sleep 100A zombie needs a parent that never calls wait. exec sleep 4 replaces the shell with
sleep, which has no idea it has a child:
bash -c 'sleep 1 & exec sleep 4' &
sleep 2
ps -o pid,ppid,stat,cmd --ppid $! PID PPID STAT CMD
16086 16085 Z+ [sleep] <defunct>A zombie uses no memory or CPU; it is just an entry in the process table holding an
exit code. You cannot kill a zombie (it is already dead). It disappears when its parent
calls wait or exits. A handful is harmless; thousands mean a buggy parent that never
reaps its children, and the fix is restarting the parent.
Listing processes with ps
ps prints a snapshot. It accepts two historical option styles, and you will see both
in the wild:
ps aux | head -4
ps -ef | head -4USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 21736 13180 ? Ss 21:13 0:00 /sbin/init
root 2 0.0 0.0 3804 2508 hvc0 Sl+ 21:13 0:00 /init
root 3 0.0 0.0 3664 2140 hvc0 S+ 21:13 0:00 init-watcher
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 21:13 ? 00:00:00 /sbin/init
root 2 1 0 21:13 hvc0 00:00:00 /init
root 3 2 0 21:13 hvc0 00:00:00 init-watcher(The /init and init-watcher lines are WSL plumbing; on a normal server PID 2 is
kthreadd.)
ps aux (BSD style) | ps -ef (System V style) | |
|---|---|---|
| Selects | All processes (a + x) | All processes (-e) |
| Shows memory | %MEM, VSZ, RSS | no |
| Shows parent | no | PPID |
| Good for | "what is eating memory?" | "who started this?" |
VSZ is the virtual address space (including shared libraries and memory that was
never touched); RSS is the resident set, the RAM actually in use, in KiB. When you
ask "how much memory does it use", you mean RSS.
Pick your own columns with -o
For anything you read more than once, choose the columns yourself. -o takes a list,
-p takes a comma-separated PID list, and pgrep -d, produces exactly that:
ps -o pid,ppid,stat,etimes,rss,args -p "$(pgrep -d, -f 'sleep 30[0-2]')" PID PPID STAT ELAPSED RSS COMMAND
13039 12927 S 4 1964 sleep 300
13048 13047 S 3 1964 sleep 301
13049 13047 S 3 1964 sleep 302Useful column names: pid, ppid, user, stat, etime (elapsed as [[dd-]hh:]mm:ss),
etimes (elapsed in seconds, ideal for scripts), lstart (full start time), %cpu,
%mem, rss, comm (program name, max 15 characters), args (full command line).
--sort takes column names, with - for descending. Here are the biggest memory users
on the lab machine while I was writing this:
ps -eo pid,user,rss,%mem,comm --sort=-rss | head -6 PID USER RSS %MEM COMMAND
1443 sid 695700 8.6 java
9039 sid 101592 1.2 pwsh
9090 sid 85516 1.0 terraform
8578 sid 79116 0.9 apt-get
202 root 23264 0.2 unattended-upgrIn scripts, put = after a column name to suppress the header, so the output is just
the value:
ps -o etimes= -p $! 4That is how you write "alert if the import has been running longer than two hours":
(( $(ps -o etimes= -p "$pid") > 7200 )).
Finding processes by name: pgrep and pkill
The habit everyone starts with is ps aux | grep something. It has a well-known flaw:
ps aux | grep 'sleep 301'sid 13048 0.0 0.0 3136 1964 pts/8 S 21:28 0:00 sleep 301
sid 13291 0.0 0.0 4104 2164 pts/8 S+ 21:28 0:00 grep sleep 301The grep process itself matches, because its own command line contains the pattern.
The classic workaround is a bracket expression that matches the text but not itself
('[s]leep 301'). The better fix is pgrep, which never matches itself and prints
PIDs you can use directly:
pgrep -f 'sleep 30[0-2]'
pgrep -af 'sleep 30[0-2]'
pgrep -c -f 'sleep 30[12]'
pgrep -n -af 'sleep 30[0-2]'
pgrep -a -P $$13039
13048
13049
13039 sleep 300
13048 sleep 301
13049 sleep 302
2
13049 sleep 302
13039 sleep 300
13047 bash| Option | Effect |
|---|---|
| (none) | Match the pattern (an extended regex) against the program name only |
-f | Match against the full command line instead |
-a | Print the full command line next to each PID |
-l | Print the program name next to each PID |
-x | Require the whole name to match, not a substring |
-u user | Only that user's processes |
-P ppid | Only children of that parent |
-n / -o | Only the newest / oldest match |
-c | Print the count instead of PIDs |
-d, | Use , as the separator (handy for ps -p) |
pgrep exits with status 0 if anything matched and 1 if nothing did, so it works in
conditions:
pgrep -f 'sleep 30[12]' > /dev/null && echo "backup sleeps still running"backup sleeps still runningpkill takes exactly the same selection options and sends a signal (SIGTERM by default)
to every match. Always run the pgrep -a version first and read the list; pkill
does not ask for confirmation.
Reading top
top is the live view. Its header is the fastest summary of a machine's health, so it
is worth learning to read line by line. In batch mode (-b, one iteration with -n1)
you can capture it:
top -bn1 -w 100 | head -12top - 21:27:04 up 16 min, 1 user, load average: 0.80, 0.63, 0.31
Tasks: 49 total, 1 running, 48 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 1.2 sy, 0.0 ni, 86.9 id, 11.9 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 7849.7 total, 3235.4 free, 1531.6 used, 3271.0 buff/cache
MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 6318.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 21736 13180 9764 S 0.0 0.2 0:00.65 systemd
2 root 20 0 3804 2508 2376 S 0.0 0.0 0:00.09 init-systemd(Ub
3 root 20 0 3664 2140 2040 S 0.0 0.0 0:00.00 init-watcher
7 root 20 0 3820 2420 2300 S 0.0 0.0 0:00.00 init
53 root 19 -1 66848 19784 18580 S 0.0 0.2 0:01.25 systemd-journal- Line 1: current time, uptime, logged-in users, and the 1-, 5- and 15-minute load
averages. Load is explained properly in
system monitoring; the short version is
"compare it with the number of CPUs (
nproc)". - Tasks: watch the
zombiecount. Non-zero and growing means a parent is not reaping children. - %Cpu(s):
ususer code,sykernel,niniced processes,ididle,wawaiting for I/O,ststolen by the hypervisor. The 11.9waabove was real: another process was downloading packages and the CPUs were waiting on the disk. Highwawith lowusmeans "the disk is the bottleneck, not the CPU". - Mem / Swap: read
avail Mem, notfree. Linux uses idle RAM as disk cache (buff/cache) and gives it back on demand, sofreeis always misleadingly small.
Interactively, the keys worth remembering are P (sort by CPU), M (sort by memory),
c (show full command lines), 1 (one line per CPU), k (send a signal to a PID),
u (filter by user) and q (quit). htop is a friendlier version if it is installed,
but top is on every machine you will ever SSH into.
Signals
A signal is a small numbered message the kernel delivers to a process. Most signals
can be caught (the process runs a handler), ignored, or left at their default
action (usually "terminate"). kill -l lists them all:
kill -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
[…]You only need a handful day to day:
| Signal | No. | Sent by | Default | Can be caught? | Use it for |
|---|---|---|---|---|---|
SIGTERM | 15 | kill PID (the default) | terminate | yes | "Please shut down cleanly." Always try this first. |
SIGINT | 2 | Ctrl-C | terminate | yes | Interrupting a foreground command |
SIGHUP | 1 | Terminal closing, kill -HUP | terminate | yes | Daemons use it as "reload your config" |
SIGKILL | 9 | kill -9 | terminate | no | Last resort when TERM was ignored |
SIGSTOP | 19 | kill -STOP | stop | no | Freezing a process |
SIGTSTP | 20 | Ctrl-Z | stop | yes | Suspending a foreground job |
SIGCONT | 18 | fg, bg, kill -CONT | continue | yes | Resuming a stopped process |
SIGUSR1/2 | 10/12 | kill -USR1 | terminate | yes | Application-defined ("dump stats", "reopen logs") |
Signal numbers above 15 differ between architectures, so use names in scripts:
kill -TERM, kill -HUP, never kill -15.
A script that handles signals properly
Here is a long-running worker that reloads its config on HUP and cleans up its scratch file on the way out:
#!/usr/bin/env bash
# worker.sh - a long-running job that reacts to signals
set -u
scratch=$(mktemp /tmp/worker.XXXXXX)
batch_size=0
load_config() {
source ./worker.conf
echo "$(date +%T) loaded worker.conf: batch_size=$batch_size"
}
cleanup() {
rm -f "$scratch"
echo "$(date +%T) removed $scratch"
}
trap load_config HUP
trap 'echo "$(date +%T) SIGTERM: stopping"; exit 143' TERM
trap 'echo "$(date +%T) SIGINT: interrupted"; exit 130' INT
trap cleanup EXIT
load_config
echo "$(date +%T) worker $$ started, scratch file $scratch"
while true; do
sleep 1 &
wait $!
doneStart it in the background, change the config, send HUP, then TERM:
cat worker.conf
./worker.sh > worker.log 2>&1 &
echo 'batch_size=500' > worker.conf
kill -HUP $!
kill $!
wait $!; echo "exit status: $?"
cat worker.log
ls /tmp/worker.*batch_size=100
[1] 14917
[1]+ Exit 143 ./worker.sh > worker.log 2>&1
exit status: 143
21:29:08 loaded worker.conf: batch_size=100
21:29:08 worker 14917 started, scratch file /tmp/worker.8NnvQh
21:29:11 loaded worker.conf: batch_size=500
21:29:12 SIGTERM: stopping
21:29:12 removed /tmp/worker.8NnvQh
ls: cannot access '/tmp/worker.*': No such file or directoryThe HUP reloaded the config without restarting, the TERM ran the handler, and the EXIT
trap removed the scratch file. Now the same thing with kill -9:
./worker.sh > worker.log 2>&1 &
kill -9 $!
wait $!; echo "exit status: $?"
cat worker.log
ls /tmp/worker.*[1] 15391
[1]+ Killed ./worker.sh > worker.log 2>&1
exit status: 137
21:29:18 loaded worker.conf: batch_size=100
21:29:18 worker 15391 started, scratch file /tmp/worker.X00kKM
/tmp/worker.X00kKMNo message, no cleanup, and the temp file is left behind. That is the whole argument
against reaching for kill -9 first: SIGKILL is handled by the kernel, the process never
gets a chance to flush buffers, release locks, remove PID files or finish writing the
file it had open. With a database or anything holding a lock file, "left behind" turns
into "won't start again until someone cleans up by hand". The rule is: TERM, wait a few
seconds, check, and only then KILL.
Exit codes tell you which signal it was
A process killed by signal N exits with status 128 + N. kill -l translates back:
kill -l 143 137 130TERM
KILL
INTWhen a cron job or CI step "failed with 137", that is SIGKILL, very often from the
kernel's out-of-memory killer or a container runtime enforcing a memory limit. 143 is a
polite TERM (a timeout or a systemctl stop), 130 is someone pressing Ctrl-C.
Bash only runs a trap between commands
This one confused me the first time. Bash does not interrupt a foreground command to run a trap; it waits for the command to finish:
#!/usr/bin/env bash
# slow-trap.sh
trap 'echo "$(date +%T) caught SIGTERM"; exit 143' TERM
echo "$(date +%T) started, PID $$"
sleep 5bash slow-trap.sh & sleep 0.5; echo "$(date +%T) sending SIGTERM"; kill $!; wait $!21:29:33 started, PID 15980
21:29:34 sending SIGTERM
21:29:38 caught SIGTERMFour seconds of delay, because the trap waited for sleep 5. With a long rsync in the
foreground, that could be an hour. That is why worker.sh uses sleep 1 & wait $!:
the wait builtin is interrupted by signals, so the handler runs immediately.
Escalating automatically with timeout
timeout runs a command with a time limit. -k adds the escalation step: send TERM at
the limit, then KILL if the process is still alive after the grace period.
#!/usr/bin/env bash
# stubborn.sh - ignores SIGTERM, like a process wedged in a bad state
trap 'echo "$(date +%T) got SIGTERM, ignoring it"' TERM
while true; do sleep 1; donetime timeout -k 3 2 ./stubborn.sh; echo "exit: $?"Terminated
21:40:32 got SIGTERM, ignoring it
Killed
real 0m5.008s
user 0m0.003s
sys 0m0.000s
exit: 137At 2 seconds timeout sent SIGTERM. The Terminated line is the inner sleep 1 dying
(timeout signals the whole process group), then the script's trap printed its defiant
message and kept looping. Three seconds later came SIGKILL, which cannot be trapped, and
the total was 5 seconds with exit status 137. timeout -k is the simplest way to put a
hard ceiling on a cron job or CI step that sometimes hangs.
Job control: &, jobs, fg, bg and Ctrl-Z
An interactive shell can run one foreground job (attached to your keyboard) and any
number of background jobs. Job control is easiest to understand as a terminal
session, so here is one, typed exactly like this (^Z and ^C are Ctrl-Z and Ctrl-C):
$ sleep 600
^Z
[1]+ Stopped sleep 600
$ jobs
[1]+ Stopped sleep 600
$ bg
[1]+ sleep 600 &
$ jobs -l
[1]+ 13624 Running sleep 600 &
$ sleep 700 &
[2] 13845
$ jobs
[1]- Running sleep 600 &
[2]+ Running sleep 700 &
$ fg %1
sleep 600
^C
$ jobs
[2]+ Running sleep 700 &
$ kill %2
$ jobs
[2]+ Terminated sleep 700| Action | What it does |
|---|---|
cmd & | Start in the background; prints [job] PID |
| Ctrl-Z | Send SIGTSTP to the foreground job: it stops (state T) |
jobs / jobs -l | List this shell's jobs (-l adds PIDs) |
bg [%n] | Resume a stopped job in the background (SIGCONT) |
fg [%n] | Bring a job to the foreground, resuming it if stopped |
kill %n | Signal a job by job number instead of PID |
%+ / %- | The current (+) and previous (-) job; fg with no argument uses %+ |
The everyday use is the "oops" move: you start something that takes longer than
expected (find / or a big tar), press Ctrl-Z, type bg, and get your prompt back
without losing the work.
Job numbers (%1) belong to one interactive shell. Scripts run without job control,
so in a script you use PIDs ($!) and wait, covered below.
Keeping processes alive after logout
When your terminal goes away (you close the window, SSH drops, the laptop sleeps), the shell receives SIGHUP. Bash then forwards SIGHUP to every job in its job table, and the default action of SIGHUP is to terminate. I tested the four common ways to protect a job by starting one of each and then sending the shell the same SIGHUP a dropped SSH connection would:
$ sleep 1001 &
[1] 14064
$ nohup sleep 1002 &
[2] 14072
$ nohup: ignoring input and appending output to 'nohup.out'
$ sleep 1003 &
[3] 14077
$ disown %3
$ setsid -f sleep 1004
$ jobs -l
[1]- 14064 Running sleep 1001 &
[2]+ 14072 Running nohup sleep 1002 &
$ pgrep -af 'sleep 100'
14064 sleep 1001
14072 sleep 1002
14077 sleep 1003
14159 sleep 1004
$ echo "shell is $$"; kill -HUP $$
shell is 14024Then, from another shell:
pgrep -af 'sleep 100'14072 sleep 1002
14077 sleep 1003
14159 sleep 1004The plain background job died; the other three survived. What each one does:
nohup cmd &makes the command ignore SIGHUP and, if stdout is a terminal, appends output tonohup.out. Redirect it yourself instead:nohup ./import.sh > import.log 2>&1 &.disown %nremoves an already-running job from the shell's job table, so the shell never forwards SIGHUP to it. Use it when you forgotnohup. (disown -hkeeps the job listed but still skips the SIGHUP.)setsid -f cmdstarts the command in a brand-new session with no controlling terminal at all. It is the most thorough of the three.tmuxorscreenkeep a whole terminal session alive on the server. For interactive work you want to reattach to later (a long migration you are watching), this is the most practical answer.
For anything that must run reliably, none of these is the real answer: they give you no
restart on failure, no logs beyond a file, and nothing survives a reboot. The modern
answer is to let systemd own the process. systemd-run does that for a one-off command
without writing a unit file:
systemd-run --user --unit=report-build sleep 600
systemctl --user status report-build --no-pager --lines=0Running as unit: report-build.service; invocation ID: 27c36475b161402680cf63bba865096d
● report-build.service - /usr/bin/sleep 600
Loaded: loaded (/run/user/1000/systemd/transient/report-build.service; transient)
Transient: yes
Active: active (running) since Mon 2026-11-09 21:45:52 EST; 1s ago
Main PID: 16209 (sleep)
Tasks: 1 (limit: 9408)
Memory: 612.0K (peak: 1.7M)
CPU: 2.334s
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/report-build.service
└─16209 /usr/bin/sleep 600The process now belongs to systemd, not to your shell, so closing the terminal does not
touch it. Its output goes to the journal (journalctl --user -u report-build), and you
stop it with systemctl --user stop report-build (systemd sends SIGTERM, waits, then
SIGKILL, exactly the escalation described above). A transient unit disappears once it
stops. One catch: a user's systemd instance normally stops
when that user's last session ends. To keep --user units running after logout, an
administrator enables lingering with loginctl enable-linger username. For services
that need to run as a system account, write a proper unit; timers and units are covered in
system monitoring and scheduling.
Waiting for background work: $! and wait
In a script, $! holds the PID of the most recent background command and wait PID
blocks until that process exits, then returns its exit status. That combination is
how you run things in parallel without losing track of failures:
#!/usr/bin/env bash
# parallel-archive.sh - archive several directories at once, report each result
dirs=(invoices images reports contracts)
pids=()
for dir in "${dirs[@]}"; do
tar -czf "$dir.tar.gz" "$dir" 2> "$dir.err" &
pids+=("$!")
echo "started $dir as PID $!"
done
failed=0
for i in "${!dirs[@]}"; do
if wait "${pids[i]}"; then
echo "ok ${dirs[i]}"
else
rc=$?
echo "FAIL ${dirs[i]} (exit $rc): $(head -1 "${dirs[i]}.err")"
failed=1
fi
done
rm -f ./*.err
exit "$failed"bash parallel-archive.sh; echo "script exit: $?"
ls -l *.tar.gzstarted invoices as PID 10411
started images as PID 10412
started reports as PID 10414
started contracts as PID 10416
ok invoices
ok images
ok reports
FAIL contracts (exit 2): tar: contracts: Cannot stat: No such file or directory
script exit: 1
-rw-r--r-- 1 sid sid 45 Nov 9 21:42 contracts.tar.gz
-rw-r--r-- 1 sid sid 3198 Nov 9 21:42 images.tar.gz
-rw-r--r-- 1 sid sid 1674 Nov 9 21:42 invoices.tar.gz
-rw-r--r-- 1 sid sid 428702 Nov 9 21:42 reports.tar.gzTwo lessons in that output. First, a bare wait with no arguments always returns 0, so
you must wait on each PID to see failures. Second, look at contracts.tar.gz: tar
created a 45-byte archive before discovering the directory did not exist. A failed job
can leave partial output behind, so a real script deletes the output on failure (the
backup script writes to a temporary name and only
renames on success).
Limiting parallelism with wait -n
Starting 500 jobs at once will flatten a machine. wait -n waits for whichever
background job finishes next, which makes a simple job pool:
#!/usr/bin/env bash
# throttle.sh - process files with at most 3 jobs running at once
max_jobs=3
for n in 1 2 3 4 5 6 7; do
while (( $(jobs -rp | wc -l) >= max_jobs )); do
wait -n
done
( sleep "0.$((RANDOM % 9 + 1))"; echo "$(date +%T.%3N) finished task $n" ) &
done
wait
echo "all tasks done"bash throttle.sh21:25:26.756 finished task 3
21:25:26.859 finished task 1
21:25:27.053 finished task 2
21:25:27.168 finished task 4
21:25:27.762 finished task 6
21:25:27.770 finished task 5
21:25:27.776 finished task 7
all tasks donejobs -rp lists the PIDs of running jobs, so the inner loop blocks while three are
active. Tasks finish out of order, which is the point. wait -n returns the exit status
of the job that finished, and since Bash 5.1 wait -n -p var also stores that job's PID
in var, so a pool can report exactly which task failed.
/proc: every process as a directory
On Linux, /proc/PID/ is a live view of a process, generated by the kernel when you
read it. Tools like ps and top are mostly readers of these files, and you can read
them directly when you need one specific fact:
./worker.sh > worker.log 2>&1 &
pid=$!
grep -E '^(Name|State|PPid|VmRSS|Threads)' /proc/$pid/status
tr '\0' ' ' < /proc/$pid/cmdline; echo
ls -l /proc/$pid/cwd /proc/$pid/exe
ls -l /proc/$pid/fd
tr '\0' '\n' < /proc/$pid/environ | grep -E '^(HOME|LANG)='
cat /proc/loadavg[1] 15653
Name: bash
State: S (sleeping)
PPid: 15615
VmRSS: 4036 kB
Threads: 1
bash ./worker.sh
lrwxrwxrwx 1 sid sid 0 Nov 9 21:29 /proc/15653/cwd -> /home/sid/proc/signals
lrwxrwxrwx 1 sid sid 0 Nov 9 21:29 /proc/15653/exe -> /usr/bin/bash
total 0
lrwx------ 1 sid sid 64 Nov 9 21:29 0 -> /dev/pts/9
l-wx------ 1 sid sid 64 Nov 9 21:29 1 -> /home/sid/proc/signals/worker.log
l-wx------ 1 sid sid 64 Nov 9 21:29 2 -> /home/sid/proc/signals/worker.log
lr-x------ 1 sid sid 64 Nov 9 21:29 255 -> /home/sid/proc/signals/worker.sh
[…]
HOME=/home/sid
LANG=C.UTF-8
5.10 2.31 0.99 4/470 15958| Path | What it tells you |
|---|---|
/proc/PID/status | Name, state, parent, user IDs, memory (VmRSS), thread count |
/proc/PID/cmdline | The exact argument list, separated by NUL bytes (hence the tr) |
/proc/PID/environ | The environment it started with (also NUL-separated) |
/proc/PID/cwd | Symlink to its current directory |
/proc/PID/exe | Symlink to the binary actually running |
/proc/PID/fd/ | One symlink per open file descriptor: files, sockets, pipes |
/proc/loadavg | Load averages, running/total tasks, last PID used |
Two practical notes. environ is a snapshot from process start, so it answers "what
PATH did cron give this job?", not "what is it set to now". And you can only read the
fd, environ and cwd of your own processes unless you are root.
What is holding this port or this file?
Start a small web server, then ask the system who owns its port:
python3 -m http.server 18081 --bind 127.0.0.1 --directory site > http.log 2>&1 &
ss -ltnp 'sport = :18081'
lsof -nP -iTCP:18081 -sTCP:LISTEN
lsof -t -iTCP:18081 -sTCP:LISTEN
fuser -v -n tcp 18081[1] 8983
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 5 127.0.0.1:18081 0.0.0.0:* users:(("python3",pid=8983,fd=3))
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python3 8983 sid 3u IPv4 35812 0t0 TCP 127.0.0.1:18081 (LISTEN)
8983
USER PID ACCESS COMMAND
18081/tcp: sid 8983 F.... python3ss -ltnp: listening, tCP, numeric, show process. The filter'sport = :18081'limits it to one port.lsof -nP -iTCP:PORT -sTCP:LISTENlists the process with the listening socket. Always pass-nP: without them lsof resolves addresses and port numbers to names. When I first ran this demo on port 8081, lsof printedlocalhost:tproxybecause 8081 is registered as "tproxy" in/etc/services.lsof -tprints only PIDs, made for scripts:kill $(lsof -t -iTCP:18081 -sTCP:LISTEN).fuseris the older tool;fuser -kwould kill the owner, which is too blunt to recommend.
Without root, ss -p and lsof only show processes you own. On a server, put sudo
in front to see everyone's.
The deleted file that still fills the disk
On Linux, deleting a file removes its name. The data is only freed when the last process holding it open closes it. Here is a stand-in for a service that keeps its log open:
#!/usr/bin/env bash
# chatty-app.sh - stand-in for a service that keeps its log file open
exec 3>>logs/app.log
head -c 20M /dev/zero >&3 # pretend weeks of logging already happened
while true; do
echo "$(date +%T) heartbeat" >&3
sleep 1
done./chatty-app.sh &
ls -lh logs/
rm logs/app.log
ls -A logs/
lsof -a +L1 -p $![1] 9914
total 21M
-rw-r--r-- 1 sid sid 21M Nov 9 21:42 app.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
bash 9914 sid 3w REG 8,48 20971634 0 56374 /home/sid/proc/logs/app.log (deleted)The directory is empty, but the 20 MB are still allocated: lsof +L1 lists open files
with a link count below 1, i.e. deleted but open. That is the answer to "I deleted the
log and df didn't change". The proper fix is to restart or signal the process so it
reopens its log. If you cannot restart it, truncate the file through the process's own
descriptor:
: > /proc/$!/fd/3
lsof -a +L1 -p $!COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
bash 9914 sid 3w REG 8,48 19 0 56374 /home/sid/proc/logs/app.log (deleted)From 20 MB to 19 bytes (one heartbeat line). For the future, truncate big logs in place
(: > app.log) instead of deleting them, or use logrotate's copytruncate for programs
that never reopen their logs.
Where you'll use this
The most common real-world use of everything above is a small control script for a
process that has no systemd unit yet: a vendor tool, an internal API, a dev server on a
shared box. It needs to start the process detached, remember its PID, report status
honestly, and stop it without kill -9 unless it has to. Here is a complete one for a
Python web server:
#!/usr/bin/env bash
# appctl - start, stop and check a background web server through a PID file
set -u
APP_DIR=${APP_DIR:-$PWD/site}
PORT=${PORT:-18081}
RUN_DIR=${RUN_DIR:-$PWD/run}
PIDFILE=$RUN_DIR/app.pid
LOG=$RUN_DIR/app.log
is_running() {
local pid
[[ -f $PIDFILE ]] || return 1
pid=$(<"$PIDFILE")
kill -0 "$pid" 2>/dev/null || return 1
# PIDs are recycled: confirm this PID is still our server, not a stranger
tr '\0' ' ' < "/proc/$pid/cmdline" | grep -q "http.server $PORT"
}
start() {
local pid i
if is_running; then
echo "already running (PID $(<"$PIDFILE"))"
return 0
fi
mkdir -p "$RUN_DIR"
nohup python3 -m http.server "$PORT" --bind 127.0.0.1 \
--directory "$APP_DIR" >> "$LOG" 2>&1 &
pid=$!
echo "$pid" > "$PIDFILE"
# Wait up to 5s until our PID owns the listening socket (or dies trying)
for ((i = 0; i < 50; i++)); do
kill -0 "$pid" 2>/dev/null || break
if [[ $(lsof -t -nP -iTCP:"$PORT" -sTCP:LISTEN 2>/dev/null) == "$pid" ]]; then
echo "started (PID $pid) on port $PORT"
return 0
fi
sleep 0.1
done
echo "failed to start; last log line:" >&2
tail -n 1 "$LOG" >&2
kill "$pid" 2>/dev/null
rm -f "$PIDFILE"
return 1
}
stop() {
local timeout=${1:-10} pid i
if ! is_running; then
echo "not running"
rm -f "$PIDFILE"
return 0
fi
pid=$(<"$PIDFILE")
kill -TERM "$pid"
for ((i = 0; i < timeout * 10; i++)); do
if ! kill -0 "$pid" 2>/dev/null; then
echo "stopped PID $pid with SIGTERM"
rm -f "$PIDFILE"
return 0
fi
sleep 0.1
done
echo "PID $pid ignored SIGTERM for ${timeout}s, sending SIGKILL" >&2
kill -KILL "$pid"
rm -f "$PIDFILE"
}
status() {
if is_running; then
echo "running (PID $(<"$PIDFILE"))"
else
echo "stopped"
return 3
fi
}
case ${1:-} in
start) start ;;
stop) stop "${2:-10}" ;;
restart) stop && start ;;
status) status ;;
*) echo "usage: appctl {start|stop [timeout]|restart|status}" >&2; exit 2 ;;
esacA full run, including the two situations that break naive scripts: a stale PID file pointing at a process that no longer exists, and the port already being taken.
./appctl status
./appctl start
./appctl start
curl -s http://127.0.0.1:18081/
./appctl status; echo "exit: $?"
./appctl restart
./appctl stop
./appctl status; echo "exit: $?"stopped
started (PID 11214) on port 18081
already running (PID 11214)
<h1>Order status</h1>
running (PID 11214)
exit: 0
stopped PID 11214 with SIGTERM
started (PID 11345) on port 18081
stopped PID 11345 with SIGTERM
stopped
exit: 3echo 99999 > run/app.pid
./appctl status
python3 -m http.server 18081 --bind 127.0.0.1 > /dev/null 2>&1 &
./appctl start
./appctl bogus; echo "exit: $?"stopped
[1] 11473
failed to start; last log line:
OSError: [Errno 98] Address already in use
usage: appctl {start|stop [timeout]|restart|status}
exit: 2My first version of start did the obvious thing: sleep 0.5, then check that the PID
was alive. During one test run of the port-conflict case it printed
started (PID 9641) on port 18081 for a server that was about to die with
Address already in use. Python had simply taken longer than half a second to start and
fail. "The process exists" is not the same as "the service works", which is why the
version above waits until its own PID owns the listening socket. The
capstone goes one step further and checks
an HTTP health endpoint.
The design decisions worth copying:
kill -0checks existence without sending anything. It returns 0 if the process exists and you may signal it.- The PID file is verified, not trusted. PIDs are reused, so after a crash and a
reboot your PID file may point at someone else's process. Checking
/proc/PID/cmdlinepreventsstopfrom killing a stranger. stopescalates. TERM, poll for up totimeoutseconds, then KILL, and it says which one happened.statususes exit code 3 for "not running", the convention init scripts use, so other scripts can test it.
The capstone deployment toolkit reuses this pattern to restart an app after each release.
Common mistakes
pkill -f kills the shell that ran it
You want a cleanup command to stop a job, so you run pkill -f from inside bash -c
(or over ssh host '…', which is the same thing):
#!/usr/bin/env bash
# pkill-self.sh
sleep 120 &
bash -c 'pkill -f "sleep 120"; echo "cleanup finished"'
echo "exit status: $?"bash pkill-self.shTerminated
exit status: 143"cleanup finished" never printed. pkill excludes itself, but not its parent: the
bash -c process's own command line contains the text sleep 120, so pkill -f killed
the shell that was running it (exit 143, SIGTERM). Anchor the pattern so it can only
match the real target. With the middle line changed to
bash -c 'pkill -f "^sleep 120$"; echo "cleanup finished"'the script prints:
cleanup finished
exit status: 0The same trap catches ssh server "pkill -f myapp" and cron lines that mention the
process they are trying to kill. Run pgrep -af with the exact pattern first; if the
list includes your own shell, fix the pattern.
pgrep finds nothing for a name you can see
pgrep systemd-timesyncd
pgrep -l systemd-timesynpgrep: pattern that searches for process name longer than 15 characters will result in zero matches
Try `pgrep -f' option to match against the complete command line.
159 systemd-timesynThe kernel stores only the first 15 characters of a program's name (comm). Recent
pgrep versions warn you; older ones silently return nothing. Scripts have the same
problem in a different form: ./nightly-invoice-export.sh with a #!/usr/bin/env bash
line runs as a process named bash, so pgrep nightly finds nothing. For scripts and
long names, match the full command line with pgrep -f.
Address already in use
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in useEvery server prints its own version of this (EADDRINUSE, bind() failed). It means
another process is listening on that address and port, very often an old copy of the
same app that a previous deploy failed to stop. Don't guess and don't restart random
things: run ss -ltnp 'sport = :PORT' or lsof -nP -iTCP:PORT -sTCP:LISTEN, read the
command line of the owner, and stop it gracefully.
Summary
- Every process has a PID, a parent (PPID) and a state;
$$,$PPIDand$!give you the shell's, its parent's and the last background job's PIDs. ps -o pid,ppid,stat,etimes,argswith-p,-Cor--sortbeatsps aux | grep;ps -o col=drops the header for scripts.pgrep -affinds by full command line and never matches itself.- In
top, read load againstnproc,wafor I/O waits andavail Meminstead offree. - SIGTERM asks, SIGKILL forces. Killed processes exit with 128 + signal number (143 =
TERM, 137 = KILL, 130 = INT). Bash runs traps only between foreground commands, so use
cmd & wait $!in long-running scripts. - Ctrl-Z stops a foreground job;
bg,fgandjobsmanage it. On logout, jobs get SIGHUP unless started withnohup/setsidordisowned; for real services use systemd. wait PIDreturns that job's exit status;wait -nbuilds a simple job pool./proc/PID/exposes a process's command line, environment, cwd and open files;ss -ltnpandlsof -nP -ianswer "who owns this port";lsof +L1finds deleted files that still occupy disk space.
Exercises
Print the three processes using the most resident memory as PID RSS_MiB COMMAND,
with RSS converted from KiB to MiB with one decimal place.
Hint
ps -eo pid,rss,comm --sort=-rss does the sorting; awk can skip the header and divide.
Show Solution
ps -eo pid,rss,comm --sort=-rss | awk 'NR == 1 { print "PID RSS_MiB COMMAND"; next }
NR <= 4 { printf "%s %.1f %s\n", $1, $2 / 1024, $3 }'PID RSS_MiB COMMAND
1656 517.5 java
650 114.9 pwsh
5940 112.6 pwshLine 1 of ps output is the header, so awk replaces it and then prints lines 2 to 4.
ps reports rss in KiB, hence the division by 1024.
Start sleep 500 in the foreground. Without pressing Ctrl-C, get your prompt back while
it keeps running, confirm with jobs -l that it is running in the background, then
terminate it using its job number.
Show Solution
$ sleep 500
^Z
[1]+ Stopped sleep 500
$ bg %1
[1]+ sleep 500 &
$ jobs -l
[1]+ 5991 Running sleep 500 &
$ kill %1
$ jobs
[1]+ Terminated sleep 500Ctrl-Z stops the job (SIGTSTP), bg resumes it in the background with SIGCONT, and
kill %1 sends SIGTERM by job number. The final jobs reports the termination once and
then forgets the job.
Extend worker.sh so that kill -USR1 PID makes it print how many loop iterations it
has completed so far, without stopping. Test it by sending USR1 twice and then TERM.
Show Solution
Add a counter, a report function and a USR1 trap; the loop increments the counter:
iterations=0
report() {
echo "$(date +%T) status: $iterations iterations, batch_size=$batch_size"
}
trap report USR1
# … the other traps and load_config call stay the same …
while true; do
sleep 1 &
wait $!
iterations=$((iterations + 1))
doneTest script:
./worker.sh &
pid=$!
sleep 3.5; kill -USR1 "$pid"
sleep 2; kill -USR1 "$pid"
sleep 0.5; kill -TERM "$pid"
wait "$pid"; echo "exit status: $?"21:41:15 loaded worker.conf: batch_size=100
21:41:15 worker 6082 started, scratch file /tmp/worker.QImFwA
21:41:19 status: 3 iterations, batch_size=100
21:41:20 status: 5 iterations, batch_size=100
21:41:20 SIGTERM: stopping
21:41:20 removed /tmp/worker.QImFwA
exit status: 143Because the loop waits with wait $!, the USR1 handler runs immediately rather than at
the end of the current sleep. This "report status on USR1" pattern is common in real
daemons (dd prints its progress on USR1, for example).
Write whoport.sh PORT that prints the PID, user and full command line of the process
listening on that TCP port, or nothing is listening on PORT with exit status 1.
Show Solution
#!/usr/bin/env bash
# whoport.sh PORT - show the process listening on a TCP port
set -u
port=${1:?usage: whoport.sh PORT}
pids=$(lsof -t -nP -iTCP:"$port" -sTCP:LISTEN 2>/dev/null) || true
if [[ -z $pids ]]; then
echo "nothing is listening on $port" >&2
exit 1
fi
for pid in $pids; do
ps -o pid=,user=,args= -p "$pid"
doneWith a test server on port 18082:
./whoport.sh 18082; echo "exit: $?"
./whoport.sh 18083; echo "exit: $?" 6206 sid python3 -m http.server 18082 --bind 127.0.0.1
exit: 0
nothing is listening on 18083
exit: 1lsof exits with 1 when it finds nothing, which would trip set -e style scripts, hence
|| true and the explicit empty check. The loop handles servers with several listening
processes (pre-forked workers). On a real server, run it with sudo to see other users'
processes.
Write a function stop_gracefully PID [TIMEOUT] that sends SIGTERM, waits up to TIMEOUT
seconds (default 5) for the process to exit, escalates to SIGKILL if needed, and returns
0 if TERM was enough, 2 if it had to use KILL, and 1 if the PID does not exist. Test it
against worker.sh and stubborn.sh running detached.
Show Solution
#!/usr/bin/env bash
# stop-gracefully.sh - TERM first, KILL only when the process will not exit
# stop_gracefully PID [TIMEOUT]
# returns 0 if SIGTERM was enough, 2 if SIGKILL was needed, 1 if no such process
stop_gracefully() {
local pid=$1 timeout=${2:-5} waited=0
if ! kill -0 "$pid" 2>/dev/null; then
echo "PID $pid is not running" >&2
return 1
fi
kill -TERM "$pid"
while kill -0 "$pid" 2>/dev/null; do
if (( waited >= timeout * 10 )); then
echo "PID $pid still alive after ${timeout}s, sending SIGKILL" >&2
kill -KILL "$pid"
return 2
fi
sleep 0.1
waited=$((waited + 1))
done
echo "PID $pid exited $((waited / 10)).$((waited % 10))s after SIGTERM"
}
stop_gracefully "$@"Test:
setsid -f ./worker.sh > worker.log 2>&1
setsid -f ./stubborn.sh > stubborn.log 2>&1
sleep 1
./stop-gracefully.sh "$(pgrep -f 'bash ./worker.sh')"; echo "returned $?"
./stop-gracefully.sh "$(pgrep -f 'bash ./stubborn.sh')" 3; echo "returned $?"
./stop-gracefully.sh 99999; echo "returned $?"PID 6294 exited 0.1s after SIGTERM
returned 0
PID 6296 still alive after 3s, sending SIGKILL
returned 2
PID 99999 is not running
returned 1The loop counts tenths of a second so the timeout is accurate without bc. Returning
distinct codes lets a caller log "had to force-kill", which is worth knowing: a service
that regularly ignores SIGTERM has a shutdown bug someone should fix.
Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), procps-ng 4.0.4, lsof 4.95,
November 2026. Every command was run in the lab as shown; the job-control and logout
transcripts were typed into a real interactive Bash on a pseudo-terminal, with Ctrl-Z and
Ctrl-C sent as keystrokes. The machine was shared with other workloads, which is why
java, pwsh and other processes appear in the system-wide listings. loginctl enable-linger needs administrator rights and was not run. I/O-blocked D state
processes are described but not demonstrated, because producing one on purpose needs a
hung disk or NFS mount. Paths from the test machine's scratch directories are shown shortened to /home/sid/….
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.