Error Handling, Traps, and Debugging

Make Bash scripts fail safely with set -euo pipefail and its real caveats, clean up with traps, handle Ctrl-C and SIGTERM, and find bugs with bash -x, PS4 and shellcheck.

By The Dark DebuggerUpdated 38 min read

This article is about what happens when things go wrong: how to make a script stop instead of ploughing on after a failure, how to clean up temporary files no matter how the script ends, how to react to Ctrl-C and kill, and how to find the bug once something has broken. It assumes everything from the three previous articles, especially exit codes (Your First Scripts), functions (Loops, Functions, and Reading Input) and arithmetic (Arrays, Associative Arrays, and Arithmetic). Option parsing with getopts, logging and lock files build on this and are covered in Writing Production-Ready Scripts.

Why error handling matters

By default, Bash runs every line of a script whether or not the line before it worked. That default is what makes this script dangerous:

#!/usr/bin/env bash
# cleanup-build.sh DIR - empty a build directory before a fresh build
build_dir=$1
cd "$build_dir"
rm -rf -- *
echo "cleaned $build_dir"

I ran it in a scratch directory with a typo in the argument:

ls -F
./cleanup-build.sh buidl
ls -F
NOTES.md  build/  cleanup-build-safe.sh*  cleanup-build.sh*  deploy.sh
./cleanup-build.sh: line 4: cd: buidl: No such file or directory
cleaned buidl

The final ls -F printed nothing. cd failed, the script carried on in the current directory, deleted everything in it (including the build tree, the notes and the script itself), and then reported success. Run from a home directory or a project root, the same typo destroys real work.

The rest of this article is the set of tools that prevent this: options that stop on errors, traps that clean up, defensive checks, and debugging aids for when it still goes wrong.

set -e: exit when a command fails

set -e (also written set -o errexit) makes the script exit as soon as a command fails. The same cleanup script with set -euo pipefail added as line 3 (we'll cover -u and pipefail next):

./cleanup-build-safe.sh buidl; echo "exit $?"
ls -F
./cleanup-build-safe.sh: line 5: cd: buidl: No such file or directory
exit 1
NOTES.md  build/  cleanup-build-safe.sh*  deploy.sh

cd failed, the script stopped with cd's status, and nothing was deleted. In a straightforward sequence of commands it does exactly what you'd hope:

#!/usr/bin/env bash
set -e
echo "backing up config"
cp /etc/myapp/app.conf /tmp/app.conf.bak
echo "restarting service"
./e-basic.sh; echo "exit $?"
backing up config
cp: cannot stat '/etc/myapp/app.conf': No such file or directory
exit 1

Where set -e does not trigger

The rules for when set -e applies are subtle, and most of the bugs blamed on Bash error handling live here. Each of these was run with set -e at the top.

Commands that are being tested. A failure in an if or while condition, in any part of an && or || list except the last, or after !, is treated as an answer, not an error:

#!/usr/bin/env bash
set -e
if grep -q nosuchuser /etc/passwd; then echo "found"; fi
grep -q nosuchuser /etc/passwd || echo "not found, carrying on"
false && echo "never printed"
! true
echo "still running: failures in if, ||, && and ! don't trigger set -e"
./e-conditions.sh; echo "exit $?"
not found, carrying on
still running: failures in if, ||, && and ! don't trigger set -e
exit 0

That's by design: otherwise you couldn't test anything. But it has a consequence that surprises almost everyone.

Functions called as a condition. When a function is called from an if (or with ||, &&, !), set -e is switched off for everything inside that function:

#!/usr/bin/env bash
set -e
deploy() {
  cp /srv/releases/app-2.14.3.tar.gz /tmp/     # fails...
  echo "deploy: extracting and restarting"      # ...and this still runs
}
 
if deploy; then
  echo "deploy reported success"
fi
 
deploy
echo "not reached"
./e-function.sh; echo "exit $?"
cp: cannot stat '/srv/releases/app-2.14.3.tar.gz': No such file or directory
deploy: extracting and restarting
deploy reported success
cp: cannot stat '/srv/releases/app-2.14.3.tar.gz': No such file or directory
exit 1

Called from if, deploy ignored the failed cp, carried on, and returned the status of its last echo: success. Called on its own line, the same function stopped at cp. A function that has to work in both situations must check its own commands (cp … || return 1), which is what Exercise 2 practises.

Command substitutions and local. A failure inside $( ) used as an argument is ignored, and local hides the status of the substitution it wraps (the same trap shown in the functions article). Only a plain assignment v=$(cmd) passes the failure on:

#!/usr/bin/env bash
set -e
get_version() { cat /srv/app/VERSION; }
 
show() {
  local v=$(get_version)
  echo "local hid the failure, v='$v'"
}
show
 
echo "version: $(get_version) <- a failure inside an argument is ignored"
 
v=$(get_version)
echo "not reached: a plain assignment does trigger set -e"
./e-subst.sh; echo "exit $?"
cat: /srv/app/VERSION: No such file or directory
local hid the failure, v=''
cat: /srv/app/VERSION: No such file or directory
version:  <- a failure inside an argument is ignored
cat: /srv/app/VERSION: No such file or directory
exit 1

Arithmetic that evaluates to zero. (( count++ )) when count is 0 "fails" and kills the script, as shown in the arrays article. Use count=$(( count + 1 )).

Pipelines. Only the last command's status counts, unless you also set pipefail (next section but one).

SituationDoes set -e stop the script?
A plain command failsyes
v=$(failing_cmd)yes
Failure in an if/while/until conditionno
Failure before the last && or || in a listno
Failure inside a function called from if, &&, || or !no, for the whole function
local v=$(failing_cmd) or echo "$(failing_cmd)"no
(( expr )) that evaluates to 0yes, often by surprise
First command of a pipeline failsonly with pipefail

My opinion: use set -e as a safety net that catches the commands you forgot to check, but don't rely on it as your error handling. For every command whose failure matters, check it explicitly with || die "…" or an if, which works the same way no matter how the function was called.

set -u: treat unset variables as errors

A typo in a variable name normally expands to an empty string, silently. This is how one-character mistakes turn into rm -rf /:

#!/usr/bin/env bash
backup_root=/srv/backups
echo "would run: rm -rf $backup_rot/daily"
./u-off.sh
would run: rm -rf /daily

With set -u (or set -o nounset), using an unset variable is a fatal error. The same script with set -u added as line 2:

./u-on.sh; echo "exit $?"
./u-on.sh: line 4: backup_rot: unbound variable
exit 1

That also applies to positional parameters, so a script that takes an optional argument needs a default expansion. "$@", $* and empty arrays are fine in Bash 5.2:

#!/usr/bin/env bash
set -u
empty=()
echo "no-arg forms are fine: '$*' ${#empty[@]} '${empty[*]}'"
echo "optional first argument: '${1:-}'"
echo "first argument: $1"
./u-args.sh; echo "exit $?"
./u-args.sh web01
no-arg forms are fine: '' 0 ''
optional first argument: ''
./u-args.sh: line 6: $1: unbound variable
exit 1
no-arg forms are fine: 'web01' 0 ''
optional first argument: 'web01'
first argument: web01

${1:-} means "$1, or empty if unset", which tells set -u the emptiness is intentional. For variables that must be set, such as configuration from the environment, ${VAR:?message} stops the script with your message. The : command does nothing, so : "${VAR:?…}" is a one-line requirement check:

#!/usr/bin/env bash
set -euo pipefail
: "${DEPLOY_ENV:?set DEPLOY_ENV to staging or production}"
: "${API_TOKEN:?is required}"
echo "deploying to $DEPLOY_ENV"
./require-env.sh; echo "exit $?"
DEPLOY_ENV=staging ./require-env.sh; echo "exit $?"
DEPLOY_ENV=staging API_TOKEN=abc123 ./require-env.sh
./require-env.sh: line 3: DEPLOY_ENV: set DEPLOY_ENV to staging or production
exit 1
./require-env.sh: line 4: API_TOKEN: is required
exit 1
deploying to staging

set -o pipefail: don't hide failures in pipelines

A pipeline's exit status is the status of its last command. When the first command fails, the pipeline still looks successful:

#!/usr/bin/env bash
set -e
grep ERROR /var/log/myapp/app.log | sort | head -n 5
echo "still running: only the last command of a pipeline counts"
./e-pipe.sh; echo "exit $?"
grep: /var/log/myapp/app.log: No such file or directory
still running: only the last command of a pipeline counts
exit 0

With set -o pipefail, the pipeline's status is that of the last command that failed, so set -e can see it. The same script with set -eo pipefail on line 2, saved as e-pipefail.sh:

./e-pipefail.sh; echo "exit $?"
grep: /var/log/myapp/app.log: No such file or directory
exit 2

To see every status in a pipeline, read the PIPESTATUS array immediately afterwards:

grep ERROR /var/log/myapp/app.log | sort | wc -l
echo "${PIPESTATUS[@]}"
grep: /var/log/myapp/app.log: No such file or directory
0
2 0 0

pipefail has one well-known false alarm. When a command such as head exits early, the command feeding it gets SIGPIPE on its next write and dies with status 141 (128 + 13):

set -o pipefail
yes | head -n 1; echo "status $?"
set +o pipefail
yes | head -n 1; echo "status $?"
y
status 141
y
status 0

Under set -eo pipefail, a pipeline like some_command | head -n 1 can therefore kill the script even though nothing went wrong. When you hit it, restructure (grep -m 1 instead of grep | head -n 1), or accept the status on that one line with || true and a comment saying why.

The strict-mode header

Most well-written Bash scripts start with:

#!/usr/bin/env bash
set -euo pipefail

Treat it as a seatbelt. It turns many silent failures into loud, early exits, and it makes typos in variable names fatal. It doesn't make a script correct: the exceptions above still apply, and the ERR trap and explicit checks below are what give you useful error messages. You'll also see IFS=$'\n\t' added to the header in some codebases; I prefer quoting variables properly, which makes it unnecessary.

Cleaning up with trap EXIT and mktemp

Scripts that create temporary files should create them with mktemp, which makes a unique, private file (or directory with -d), and remove them with a trap. trap 'commands' EXIT runs the commands whenever the script exits: normally, through exit, through set -e, or through Ctrl-C:

#!/usr/bin/env bash
# report.sh LOG - build a small report, using a temporary directory for scratch files
set -euo pipefail
 
workdir=$(mktemp -d)
trap 'rm -rf -- "$workdir"; echo "cleanup: removed $workdir" >&2' EXIT
 
echo "working in $workdir"
grep -o '" [0-9][0-9][0-9] ' "$1" | tr -d '" ' > "$workdir/codes"
sort "$workdir/codes" | uniq -c | sort -rn > "$workdir/counts"
head -n 3 "$workdir/counts"
echo "report done"
./report.sh access.log
./report.sh missing.log; echo "exit $?"
ls -d /tmp/tmp.* 2>/dev/null | wc -l
working in /tmp/tmp.JqdWbmLaaY
    249 200
     45 404
     40 304
report done
cleanup: removed /tmp/tmp.JqdWbmLaaY
working in /tmp/tmp.TEEoPcgKXl
grep: missing.log: No such file or directory
cleanup: removed /tmp/tmp.TEEoPcgKXl
exit 2
0

On the failed run, set -e stopped the script at grep, and the trap still removed the directory. Nothing was left in /tmp.

Three rules for traps:

  1. Use single quotes. The trap string is evaluated when the trap fires. In double quotes, variables are expanded when the trap is set, so a later change is missed. See Common mistakes for the demonstration.
  2. Set the trap right after creating the thing it cleans up, before anything else can fail.
  3. One trap per signal. A second trap … EXIT replaces the first; it doesn't add to it:
#!/usr/bin/env bash
trap 'echo "removing temp files"' EXIT
trap 'echo "releasing lock"' EXIT
trap -p EXIT
./trap-replace.sh
trap -- 'echo "releasing lock"' EXIT
releasing lock

The temp files were never removed. If a script needs several cleanup actions, put them all in one cleanup function and trap that, as the backup script at the end of this article does. trap -p shows what is currently set.

trap ERR: report where a script failed

set -e stops the script, but the only message you get is whatever the failing command printed. An ERR trap runs whenever a command fails (under the same rules as set -e), and it can report the line number and the command:

#!/usr/bin/env bash
set -eEuo pipefail    # -E: functions inherit the ERR trap
trap 'echo "error: line $LINENO: \"$BASH_COMMAND\" exited with status $?" >&2' ERR
 
fetch_release() {
  local version=$1
  cp "/srv/releases/app-$version.tar.gz" /tmp/
}
 
echo "deploying"
fetch_release 2.14.3
echo "not reached"
./err-trap.sh; echo "exit $?"
deploying
cp: cannot stat '/srv/releases/app-2.14.3.tar.gz': No such file or directory
error: line 7: "cp "/srv/releases/app-$version.tar.gz" /tmp/" exited with status 1
exit 1

$LINENO is the line that failed, $BASH_COMMAND the command text, and $? its status. The -E in set -eEuo pipefail (also set -o errtrace) matters. Without it, functions don't inherit the ERR trap, and the same script with plain set -euo pipefail, failing inside fetch_release, prints no report at all:

./err-trap-noE.sh; echo "exit $?"
deploying
cp: cannot stat '/srv/releases/app-2.14.3.tar.gz': No such file or directory
exit 1

For bigger scripts, a trap function can walk the call stack. FUNCNAME lists the active functions (innermost first) and BASH_LINENO the line each was called from:

#!/usr/bin/env bash
set -eEuo pipefail
 
on_error() {
  local status=$? i
  echo "error: '$BASH_COMMAND' failed with status $status" >&2
  for ((i = 1; i < ${#FUNCNAME[@]} - 1; i++)); do
    echo "  in ${FUNCNAME[i]}() at line ${BASH_LINENO[i-1]}" >&2
  done
  echo "  in the script body at line ${BASH_LINENO[i-1]}" >&2
}
trap on_error ERR
 
fetch_release() {
  cp "/srv/releases/app-$1.tar.gz" /tmp/
}
 
deploy() {
  echo "deploying $1"
  fetch_release "$1"
}
 
main() {
  deploy 2.14.3
}
 
main "$@"
./stacktrace.sh; echo "exit $?"
deploying 2.14.3
cp: cannot stat '/srv/releases/app-2.14.3.tar.gz': No such file or directory
error: 'cp "/srv/releases/app-$1.tar.gz" /tmp/' failed with status 1
  in fetch_release() at line 15
  in deploy() at line 20
  in main() at line 24
  in the script body at line 27
exit 1

local status=$? must be the first line of the handler; any command before it would overwrite $?. The loop stops one short of the end because the last entry in FUNCNAME is Bash's name for the script's top level (also main, which is confusing when your own function is called main too).

Signals: Ctrl-C and SIGTERM

When you press Ctrl-C, the terminal sends SIGINT to every process in the foreground job. When systemd stops a service, a container runtime stops a container, or someone runs kill PID, the process gets SIGTERM. A script that holds temporary files or half-written output needs to handle both. Processes, Signals, and Job Control covers signals in general; here we look at what they do to a script.

The good news first. In Bash, the EXIT trap also runs when the script is killed by SIGINT or SIGTERM, so the mktemp pattern above is already safe. This script just sleeps with a temp file:

#!/usr/bin/env bash
tmp=$(mktemp)
trap 'rm -f -- "$tmp"; echo "EXIT trap: removed $tmp"' EXIT
echo "working, temp file is $tmp"
sleep 30
echo "finished normally"

Pressing Ctrl-C after a second, then checking the status:

./exit-only.sh
echo "exit $?"
working, temp file is /tmp/tmp.lIVUKlKgGI
^CEXIT trap: removed /tmp/tmp.lIVUKlKgGI
 
exit 130

To send SIGTERM the way a service manager would, from another process, I used a small driver script. It starts the target in the background, waits two seconds, sends the signal and reports the exit status:

#!/usr/bin/env bash
# signal-driver.sh SCRIPT - start SCRIPT in the background, send SIGTERM after 2s
./"$1" &
pid=$!
sleep 2
echo "$(date +%T) driver: kill -TERM $pid"
kill -TERM "$pid"
wait "$pid"
status=$?
echo "$(date +%T) driver: $1 exited with status $status"
./signal-driver.sh exit-only.sh
working, temp file is /tmp/tmp.tZQxh3sBuU
21:59:58 driver: kill -TERM 63255
EXIT trap: removed /tmp/tmp.tZQxh3sBuU
21:59:58 driver: exit-only.sh exited with status 143

Both cleaned up, and the exit statuses report the signal: 130 is 128 + 2 (SIGINT) and 143 is 128 + 15 (SIGTERM). Nothing can catch SIGKILL (kill -9), so no trap runs then; that is one reason to send SIGTERM first.

Handling a signal yourself, and why your trap might be late

When you want to do something on a signal (log it, finish the current item, exit with a specific status), trap it by name. Here's a long-running worker:

#!/usr/bin/env bash
# worker.sh - process batches until told to stop
workdir=$(mktemp -d)
trap 'rm -rf -- "$workdir"; echo "$(date +%T) worker: removed $workdir"' EXIT
trap 'echo "$(date +%T) worker: caught SIGTERM, stopping"; exit 143' TERM
trap 'echo "$(date +%T) worker: caught SIGINT, stopping"; exit 130' INT
 
echo "$(date +%T) worker: started as PID $$"
while true; do
  echo "$(date +%T) worker: processing a batch"
  sleep 10
done

Sending it SIGTERM two seconds in:

./signal-driver.sh worker.sh
22:00:00 worker: started as PID 63433
22:00:00 worker: processing a batch
22:00:02 driver: kill -TERM 63433
22:00:10 worker: caught SIGTERM, stopping
22:00:10 worker: removed /tmp/tmp.TDL96cZKmH
22:00:10 driver: worker.sh exited with status 143

Look at the timestamps: the signal arrived at :02 and the trap ran at :10. Bash doesn't run a trap while it is waiting for a foreground command to finish; it runs it when sleep 10 returns. For a service manager that gives a process 10 seconds to stop before killing it outright, that delay matters.

The fix is to run the slow command in the background and wait for it. wait is interrupted by trapped signals:

  sleep 10 &
  wait $!

With only those two lines changed, saved as worker-responsive.sh:

./signal-driver.sh worker-responsive.sh
22:00:12 worker: started as PID 63819
22:00:12 worker: processing a batch
22:00:14 driver: kill -TERM 63819
22:00:14 worker: caught SIGTERM, stopping
22:00:14 worker: removed /tmp/tmp.DBM1CJzeXl
22:00:14 driver: worker-responsive.sh exited with status 143

Ctrl-C behaves differently, which confuses people testing at a terminal. Running the original worker.sh (foreground sleep) at a terminal and pressing Ctrl-C gave an immediate reaction:

./worker.sh
21:56:52 worker: started as PID 52258
21:56:52 worker: processing a batch
^C21:56:55 worker: caught SIGINT, stopping
21:56:55 worker: removed /tmp/tmp.FCIMwRuvWe

Ctrl-C signals the whole foreground process group, so sleep died at once and Bash ran the trap straight away. kill PID signals only the script. So "it stops fine when I press Ctrl-C" doesn't prove it will stop promptly under systemd or Docker; test with kill -TERM. One more detail: in the responsive version, the background sleep is left running for the rest of its 10 seconds after the script exits. A real worker should kill its child in the cleanup function, which Exercise 4 does.

Defensive patterns

A handful of small habits prevent most script disasters.

A die function for fatal errors, used with || on every command whose failure matters. Pair it with a need check so the script fails on line 1 with a clear message, not on line 80 with a confusing one:

#!/usr/bin/env bash
# defensive.sh - the checks a deploy script makes before touching anything
set -euo pipefail
 
die() {
  echo "${0##*/}: $*" >&2
  exit 1
}
 
need() {
  command -v "$1" >/dev/null 2>&1 || die "required command '$1' not found"
}
 
need tar
need curl
need aws
 
echo "all tools present"
./defensive.sh; echo "exit $?"
defensive.sh: required command 'aws' not found
exit 1

Guard destructive commands against empty variables. ${var:?message} aborts the command (and, in a script, the script) before it runs if var is empty or unset. I demonstrated it with ls rather than rm, for obvious reasons; the guard works identically in front of rm -rf:

#!/usr/bin/env bash
cache_dir=""          # imagine this came from a config file that failed to load
ls -d -- "${cache_dir:?cache_dir is empty, refusing to continue}"/*
echo "not reached"
./guard.sh; echo "exit $?"
./guard.sh: line 3: cache_dir: cache_dir is empty, refusing to continue
exit 1

Without the guard, "$cache_dir"/* becomes /*. Writing rm -rf -- "${dir:?}"/* costs nothing and makes that impossible.

Other habits worth having:

  • cd "$dir" || die "cannot cd to $dir" on every cd, even with set -e (a cd inside a function called from if won't stop the script otherwise).
  • -- before filename arguments (rm -- "$f", cp -- "$src" "$dst") so a file named -rf is treated as a file.
  • Validate every input before making any change, as the scripts in Your First Scripts do.
  • Write to a temporary file and mv it into place, so readers never see a half-written file. The backup script below does this.

Debugging: bash -n, set -x and PS4

bash -n: check syntax without running

bash -n script.sh parses the script and reports syntax errors without executing anything. Here a for loop is missing its done:

#!/usr/bin/env bash
for host in web01 web02; do
  if [[ $host == web01 ]]; then
    echo "primary: $host"
  fi
echo "checked all hosts"
bash -n broken-syntax.sh; echo "exit $?"
bash -n e-function.sh; echo "exit $?"
broken-syntax.sh: line 7: syntax error: unexpected end of file
exit 2
exit 0

It's fast and safe enough to run in an editor hook or CI step, but it only catches syntax. e-function.sh, with its very real logic bug, passes cleanly.

bash -x: watch the script run

bash -x script.sh (or set -x inside the script) prints each command, after expansion, before running it. That "after expansion" is what makes it so useful: you see what the command actually received. This backup script has a quoting bug:

#!/usr/bin/env bash
# copy-config.sh FILE - back up a config file (has a bug)
src=$1
dest="backup/$(basename "$src").bak"
mkdir -p backup
cp $src $dest
echo "saved $dest"
./copy-config.sh "nginx prod.conf"
bash -x copy-config.sh "nginx prod.conf"
cp: target 'prod.conf.bak': No such file or directory
saved backup/nginx prod.conf.bak
+ src='nginx prod.conf'
++ basename 'nginx prod.conf'
+ dest='backup/nginx prod.conf.bak'
+ mkdir -p backup
+ cp nginx prod.conf backup/nginx prod.conf.bak
cp: target 'prod.conf.bak': No such file or directory
+ echo 'saved backup/nginx prod.conf.bak'
saved backup/nginx prod.conf.bak

The error alone is baffling: where did prod.conf.bak come from? The trace line + cp nginx prod.conf backup/nginx prod.conf.bak answers it: the unquoted variables split into four arguments, so cp was asked to copy three files into a "directory" called prod.conf.bak. Quoting "$src" and "$dest" fixes it. Lines starting ++ come from a nested level, here the command substitution.

A more useful PS4

The + prefix comes from the PS4 variable, and you can put the file name, line number and function in it. Use single quotes so it's evaluated on every line:

export PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '
bash -x stacktrace.sh 2>&1 | head -n 12
+ stacktrace.sh:2:main: set -eEuo pipefail
+ stacktrace.sh:12:main: trap on_error ERR
+ stacktrace.sh:27:main: main
+ stacktrace.sh:24:main: deploy 2.14.3
+ stacktrace.sh:19:deploy: echo 'deploying 2.14.3'
deploying 2.14.3
+ stacktrace.sh:20:deploy: fetch_release 2.14.3
+ stacktrace.sh:15:fetch_release: cp /srv/releases/app-2.14.3.tar.gz /tmp/
cp: cannot stat '/srv/releases/app-2.14.3.tar.gz': No such file or directory
++ stacktrace.sh:15:fetch_release: on_error
++ stacktrace.sh:5:on_error: local status=1 i
++ stacktrace.sh:6:on_error: echo 'error: '\''cp "/srv/releases/app-$1.tar.gz" /tmp/'\'' failed with status 1'

In a long script, trace just the part you're investigating with set -x … set +x, and send the trace to a file with BASH_XTRACEFD so it doesn't mix with the script's own output:

#!/usr/bin/env bash
# trace-region.sh - trace only the part of the script you're debugging
exec 5>> trace.log            # open trace.log on file descriptor 5
BASH_XTRACEFD=5               # send xtrace output there instead of stderr
 
echo "checking config"
set -x
port=$(grep -o '[0-9]*' nginx.conf)
[[ $port -gt 1024 ]] && echo "port $port is unprivileged"
set +x
echo "done"
./trace-region.sh
cat trace.log
checking config
port 8443 is unprivileged
done
++ grep -o '[0-9]*' nginx.conf
+ port=8443
+ [[ 8443 -gt 1024 ]]
+ echo 'port 8443 is unprivileged'
+ set +x

Be careful with set -x in scripts that handle secrets: the trace prints expanded values, including passwords and tokens, wherever the trace goes.

shellcheck: find bugs before they run

ShellCheck is a static analyser for shell scripts. It knows every trap in this article series and points at the exact characters responsible. If you adopt one tool from this article, make it this one, and run it in your editor and in CI. Here's a short deploy script containing several classic bugs:

#!/usr/bin/env bash
# deploy-buggy.sh RELEASE_DIR - copy a release into the web root
release_dir=$1
web_root=/srv/www
 
for f in $(ls $release_dir); do
  cp $release_dir/$f $web_root/
done
 
cd $web_root
version=$(cat VERSION)
if [ $? -ne 0 ]; then
  echo "no VERSION file"
fi
 
read -p "Restart nginx? " answer
echo "Deployed $version"
shellcheck deploy-buggy.sh
echo "exit $?"
 
In deploy-buggy.sh line 6:
for f in $(ls $release_dir); do
         ^----------------^ SC2045 (error): Iterating over ls output is fragile. Use globs.
              ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
 
Did you mean: 
for f in $(ls "$release_dir"); do
 
 
In deploy-buggy.sh line 7:
  cp $release_dir/$f $web_root/
     ^----------^ SC2086 (info): Double quote to prevent globbing and word splitting.
                  ^-- SC2086 (info): Double quote to prevent globbing and word splitting.
 
Did you mean: 
  cp "$release_dir"/"$f" $web_root/
 
 
In deploy-buggy.sh line 10:
cd $web_root
^----------^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
 
Did you mean: 
cd $web_root || exit
 
 
In deploy-buggy.sh line 12:
if [ $? -ne 0 ]; then
     ^-- SC2181 (style): Check exit code directly with e.g. 'if ! mycmd;', not indirectly with $?.
 
 
In deploy-buggy.sh line 16:
read -p "Restart nginx? " answer
^--^ SC2162 (info): read without -r will mangle backslashes.
                          ^----^ SC2034 (warning): answer appears unused. Verify use (or export if used externally).
 
For more information:
  https://www.shellcheck.net/wiki/SC2045 -- Iterating over ls output is fragi...
  https://www.shellcheck.net/wiki/SC2034 -- answer appears unused. Verify use...
  https://www.shellcheck.net/wiki/SC2164 -- Use 'cd ... || exit' or 'cd ... |...
exit 1

Every finding is a bug we've met in this series:

CodeProblemFix
SC2045looping over $(ls) breaks on spacesfor f in "$release_dir"/*
SC2086unquoted variable is split and globbed"$release_dir", "$f"
SC2164cd can fail and the script continues elsewherecd "$web_root" || exit 1
SC2181testing $? separately is fragileif ! version=$(cat VERSION); then
SC2162read without -r mangles backslashesread -r
SC2034variable assigned but never useduse it, or remove it

Notice that SC2086, the most common finding of all, is only info level. Running shellcheck -S warning hides info and style findings, which is handy for a first pass over an old codebase, but those "info" items are the word-splitting bugs, so don't stop there. The fixed script passes cleanly, and shellcheck exits 0, which is what makes it easy to enforce in CI:

#!/usr/bin/env bash
# deploy-fixed.sh RELEASE_DIR - copy a release into the web root
set -euo pipefail
release_dir=$1
web_root=/srv/www
 
for f in "$release_dir"/*; do
  cp -- "$f" "$web_root/"
done
 
cd "$web_root" || exit 1
if ! version=$(cat VERSION); then
  echo "no VERSION file" >&2
  exit 1
fi
 
read -rp "Restart nginx? " answer
echo "Deployed $version (restart: $answer)"
shellcheck deploy-fixed.sh; echo "exit $?"
exit 0

When a warning is wrong for your situation (a variable that is meant to split into several words, say), silence that one line with a directive comment and say why, rather than turning the check off globally:

#!/usr/bin/env bash
# RSYNC_OPTS deliberately holds several options, so it must word-split.
RSYNC_OPTS="-a --delete"
# shellcheck disable=SC2086
rsync $RSYNC_OPTS src/ dest/

(An array, as shown in the arrays article, is usually the better fix.)

Where you'll use this

Every script that runs unattended (backups, deploys, data exports, CI steps) needs this article's toolkit, because nobody is watching when it fails. The usual combination is: strict mode, an ERR trap for a useful message, an EXIT trap for cleanup, die for clear validation errors, and a write-to-temp-then-rename step so that a failure never leaves half a result where something else will read it.

Here's a complete backup script built that way. It makes a timestamped tarball of a directory, verifies the archive can be read back, moves it into place atomically, and prunes old backups:

#!/usr/bin/env bash
# backup-dir.sh SOURCE_DIR BACKUP_DIR [KEEP]
# Write a verified, timestamped tarball of SOURCE_DIR into BACKUP_DIR and keep
# only the newest KEEP backups (default 7).
set -Eeuo pipefail
 
readonly prog=${0##*/}
tmpdir=""
 
die() {
  echo "$prog: $*" >&2
  exit 1
}
 
on_error() {
  local status=$?
  echo "$prog: line $1: '$BASH_COMMAND' failed with status $status" >&2
}
 
cleanup() {
  if [[ -n $tmpdir ]]; then
    rm -rf -- "$tmpdir"
  fi
}
 
trap 'on_error $LINENO' ERR
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
 
main() {
  (( $# >= 2 && $# <= 3 )) || die "usage: $prog SOURCE_DIR BACKUP_DIR [KEEP]"
  local src=$1 dest=$2 keep=${3:-7}
 
  [[ -d $src ]] || die "source '$src' is not a directory"
  [[ $keep =~ ^[1-9][0-9]*$ ]] || die "KEEP must be a positive whole number"
  command -v tar >/dev/null || die "tar is not installed"
  mkdir -p -- "$dest"
 
  local base name
  base=$(basename -- "$src")
  name="$base-$(date +%Y%m%d-%H%M%S).tar.gz"
 
  # Build in a temp dir inside $dest so the final mv is an atomic rename.
  tmpdir=$(mktemp -d "$dest/.incoming.XXXXXX")
  tar -czf "$tmpdir/$name" -C "$(dirname -- "$src")" "$base"
  tar -tzf "$tmpdir/$name" > /dev/null          # verify the archive reads back
  mv -- "$tmpdir/$name" "$dest/$name"
  echo "$prog: wrote $dest/$name"
 
  # Timestamps in the names sort chronologically, so the oldest come first.
  local backups=("$dest/$base"-*.tar.gz)
  local excess=$(( ${#backups[@]} - keep ))
  if (( excess > 0 )); then
    rm -f -- "${backups[@]:0:excess}"
    echo "$prog: pruned $excess old backup(s), keeping $keep"
  fi
}
 
main "$@"

Four runs with KEEP=3, a second or so apart, then the result:

./backup-dir.sh data backups 3
./backup-dir.sh data backups 3
./backup-dir.sh data backups 3
./backup-dir.sh data backups 3
ls -A backups
backup-dir.sh: wrote backups/data-20261109-220128.tar.gz
backup-dir.sh: wrote backups/data-20261109-220130.tar.gz
backup-dir.sh: wrote backups/data-20261109-220131.tar.gz
backup-dir.sh: wrote backups/data-20261109-220133.tar.gz
backup-dir.sh: pruned 1 old backup(s), keeping 3
data-20261109-220130.tar.gz  data-20261109-220131.tar.gz  data-20261109-220133.tar.gz

Now the failures. Bad input is caught by die before anything happens. A failure in the middle (I made one file unreadable) is reported by the ERR trap with its line number, and the EXIT trap removes the half-built archive, so the backup directory still contains only complete, verified backups:

./backup-dir.sh; echo "exit $?"
./backup-dir.sh missing backups; echo "exit $?"
./backup-dir.sh data backups 0; echo "exit $?"
chmod 000 data/settings.conf
./backup-dir.sh data backups 3; echo "exit $?"
chmod 644 data/settings.conf
ls -A backups
backup-dir.sh: usage: backup-dir.sh SOURCE_DIR BACKUP_DIR [KEEP]
exit 1
backup-dir.sh: source 'missing' is not a directory
exit 1
backup-dir.sh: KEEP must be a positive whole number
exit 1
tar: data/settings.conf: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors
backup-dir.sh: line 46: 'tar -czf "$tmpdir/$name" -C "$(dirname -- "$src")" "$base"' failed with status 2
exit 2
data-20261109-220130.tar.gz  data-20261109-220131.tar.gz  data-20261109-220133.tar.gz

No .incoming.* directory was left behind, and shellcheck backup-dir.sh reports nothing. Things worth copying from this script into your own:

  • tmpdir="" is set before the traps, so cleanup never touches an unset variable, even if the script fails before mktemp runs.
  • The INT and TERM traps just call exit, which runs the EXIT trap. All the cleanup logic lives in one place.
  • The temp directory is inside the destination, so the final mv is a rename on the same filesystem: other processes see either no file or a complete one, never a partial archive.
  • Pruning happens only after a new backup has been written and verified, so a broken run can never reduce the number of good backups.

Scheduling this from cron, adding a lock so two runs can't overlap, and logging to a file are the subject of Writing Production-Ready Scripts and Real-World Scripts: Backups, Log Reports, and Cleanup.

Common mistakes

A trap in double quotes uses stale values

#!/usr/bin/env bash
file=/tmp/first.tmp
trap "echo double quotes: would remove $file" EXIT
file=/tmp/second.tmp
./trap-quoting.sh
double quotes: would remove /tmp/first.tmp

With single quotes (trap 'echo single quotes: would remove $file' EXIT, saved as trap-quoting2.sh), the variable is read when the trap runs:

./trap-quoting2.sh
single quotes: would remove /tmp/second.tmp

Double quotes expanded $file when the trap was set. Sometimes that's what you want, but for cleanup it almost never is.

die inside $( ) only exits the subshell

#!/usr/bin/env bash
die() { echo "error: $*" >&2; exit 1; }
 
get_db_host() {
  [[ -f db.conf ]] || die "db.conf is missing"
  cat db.conf
}
 
host=$(get_db_host)
echo "connecting to '$host'"
./die-subshell.sh; echo "exit $?"
error: db.conf is missing
connecting to ''
exit 0

$( ) runs in a subshell, so exit ended the subshell, not the script. The script then continued with an empty host. Without set -e nothing stops it; check the assignment instead: host=$(get_db_host) || exit 1. With that one line changed:

./die-subshell-fixed.sh; echo "exit $?"
error: db.conf is missing
exit 1

$? clobbered by a command substitution in the same line

This one caught me while testing the signal examples: my driver printed exited with status 0 for scripts that had clearly exited with 143. The cause:

#!/usr/bin/env bash
grep -q nosuchuser /etc/passwd
echo "$(date +%T) grep returned $?"
 
grep -q nosuchuser /etc/passwd
status=$?
echo "$(date +%T) grep returned $status"
./status-clobber.sh
22:02:16 grep returned 0
22:02:16 grep returned 1

Expansions happen left to right, so $(date +%T) ran first and reset $? to 0 before $? was expanded. Save $? into a variable on the very next line, before doing anything else.

Trusting set -e inside functions

If a function is called from if, &&, || or with !, set -e is off for its whole body, so a failed step in the middle is ignored. The deploy example above shows it. Add explicit || return 1 (or || die) to the commands that matter inside any function whose result is tested.

Summary

  • set -e exits on failing commands, but not in if/while conditions, &&/|| lists, !, local v=$(…), arguments containing $( ), or anywhere inside a function called as a condition. Treat it as a safety net and check important commands explicitly.
  • set -u makes unset variables fatal (use ${var:-} for optional values and : "${VAR:?message}" for required ones). set -o pipefail makes a pipeline fail when any part fails; watch for status 141 from cmd | head.
  • Create temporary files with mktemp and remove them with a single-quoted trap '…' EXIT, set right after creation. A second trap on the same signal replaces the first, so put all cleanup in one function.
  • trap … ERR with $LINENO, $BASH_COMMAND and $? reports where a script failed. Use set -E so functions inherit it; FUNCNAME and BASH_LINENO give a call stack.
  • In Bash the EXIT trap also runs on SIGINT and SIGTERM (exit statuses 130 and 143). Traps for a signal wait until the foreground command finishes, so run long commands as cmd & wait $! when a script must react promptly. Ctrl-C signals the whole foreground group; kill PID signals only the script.
  • Use die with || on commands that must succeed, command -v to check prerequisites, ${var:?} in front of destructive paths, and -- before filenames. exit inside $( ) only leaves the subshell.
  • Debug with bash -n (syntax only), bash -x or set -x (commands after expansion), a PS4 that shows file, line and function, and BASH_XTRACEFD to send the trace to a file.
  • Run shellcheck on every script; it catches unquoted variables, $(ls) loops, unchecked cd and the rest, and exits non-zero so CI can enforce it.

Exercises

Exercise 1: Sort a file in place, safelyeasy

Write sort-unique.sh FILE that sorts FILE and removes duplicate lines, writing the result back to FILE. Use a temporary file created with mktemp and make sure it is removed whether the script succeeds or fails. Test it with a file that has duplicates and with a file that doesn't exist, and confirm no temp files are left.

Hint

sort -u FILE > FILE empties the file before sort reads it, which is why you need the temporary file.

Show Solution
#!/usr/bin/env bash
# sort-unique.sh FILE - sort FILE in place, removing duplicate lines
set -euo pipefail
 
tmp=$(mktemp)
trap 'rm -f -- "$tmp"' EXIT
 
sort -u -- "$1" > "$tmp"
cat -- "$tmp" > "$1"
echo "sorted $1: $(wc -l < "$1") unique lines"
cat hosts.txt
./sort-unique.sh hosts.txt
cat hosts.txt
./sort-unique.sh nope.txt; echo "exit $?"
ls /tmp/tmp.* 2>/dev/null | wc -l
web02
web01
db01
web02
web01
sorted hosts.txt: 3 unique lines
db01
web01
web02
sort: cannot read: nope.txt: No such file or directory
exit 2
0

cat "$tmp" > "$1" rewrites the original file in place, which keeps its permissions and ownership; mv "$tmp" "$1" would be atomic but would give the file mktemp's private 600 permissions. Pick based on whether anything reads the file while you write it.

Exercise 2: Why did it say published?medium

This release script uses set -e, there is no dist/app-2.15.0.tar.gz, and yet it prints published 2.15.0, exits 0 and writes releases/LATEST. Explain why, then fix it so a failed copy makes it print publish failed and exit 1, without removing the if.

#!/usr/bin/env bash
set -e
 
publish() {
  cp "dist/app-$1.tar.gz" releases/
  echo "$1" > releases/LATEST
}
 
if publish 2.15.0; then
  echo "published 2.15.0"
else
  echo "publish failed" >&2
  exit 1
fi
./release-buggy.sh; echo "exit $?"
cat releases/LATEST
cp: cannot stat 'dist/app-2.15.0.tar.gz': No such file or directory
published 2.15.0
exit 0
2.15.0
Show Solution

Because publish is called as the condition of an if, set -e is disabled for everything inside it. The failed cp is ignored, the echo into LATEST succeeds, and the function returns the status of that last command: 0. The fix is to make the function check its own steps:

#!/usr/bin/env bash
set -e
 
publish() {
  cp "dist/app-$1.tar.gz" releases/ || return 1
  echo "$1" > releases/LATEST || return 1
}
 
if publish 2.15.0; then
  echo "published 2.15.0"
else
  echo "publish failed" >&2
  exit 1
fi
./release-fixed.sh; echo "exit $?"
ls releases
cp: cannot stat 'dist/app-2.15.0.tar.gz': No such file or directory
publish failed
exit 1

ls releases printed nothing: no LATEST file pointing at a release that doesn't exist. Explicit || return 1 works the same whether the function is called from if or on its own line, which is exactly why it's more reliable than set -e.

Exercise 3: Make shellcheck happymedium

Run shellcheck on this script, then rewrite it so that shellcheck reports nothing, it handles log names containing spaces, and it takes the directory as an optional second argument. Test it on a directory with a 2 MB, a 900 KB and a 120 KB log and a 500 KB limit.

#!/bin/bash
# rotate-check.sh - warn about log files larger than a limit
limit=$1
for log in $(find /var/log/myapp -name *.log); do
  size=$(du -k --apparent-size $log | cut -f1)
  if [ $size -gt $limit ]; then
    echo "$log is ${size}K"
  fi
done
Hint

truncate -s 2M "biglogs/batch run.log" creates a sparse test file without using real disk space; that's also why the script asks du for --apparent-size. To read find results safely, use -print0 with while IFS= read -r -d '' log.

Show Solution
shellcheck rotate-check.sh
 
In rotate-check.sh line 4:
for log in $(find /var/log/myapp -name *.log); do
           ^-- SC2044 (warning): For loops over find output are fragile. Use find -exec or a while read loop.
                                       ^---^ SC2061 (warning): Quote the parameter to -name so the shell won't interpret it.
                                       ^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.
 
 
In rotate-check.sh line 5:
  size=$(du -k --apparent-size $log | cut -f1)
                               ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
 
Did you mean: 
  size=$(du -k --apparent-size "$log" | cut -f1)
 
 
In rotate-check.sh line 6:
  if [ $size -gt $limit ]; then
       ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.
                 ^----^ SC2086 (info): Double quote to prevent globbing and word splitting.
 
Did you mean: 
  if [ "$size" -gt "$limit" ]; then
 
For more information:
  https://www.shellcheck.net/wiki/SC2044 -- For loops over find output are fr...
  https://www.shellcheck.net/wiki/SC2061 -- Quote the parameter to -name so t...
  https://www.shellcheck.net/wiki/SC2035 -- Use ./*glob* or -- *glob* so name...

The rewrite:

#!/usr/bin/env bash
# rotate-check-fixed.sh LIMIT_KB [DIR] - warn about log files larger than a limit
set -euo pipefail
limit=$1
dir=${2:-/var/log/myapp}
 
while IFS= read -r -d '' log; do
  size=$(du -k --apparent-size -- "$log" | cut -f1)
  if (( size > limit )); then
    echo "$log is ${size}K"
  fi
done < <(find "$dir" -name '*.log' -print0)
shellcheck rotate-check-fixed.sh; echo "exit $?"
ls -ls biglogs
./rotate-check-fixed.sh 500 biglogs
exit 0
total 0
0 -rw-r--r-- 1 sid sid  921600 Nov  9 21:56  api.log
0 -rw-r--r-- 1 sid sid 2097152 Nov  9 21:56 'batch run.log'
0 -rw-r--r-- 1 sid sid  122880 Nov  9 21:56  worker.log
biglogs/batch run.log is 2048K
biglogs/api.log is 900K

The unquoted *.log was the nastiest bug: if the current directory happened to contain a .log file, the shell would expand the pattern before find saw it, and find would search for that one name. The 0 in the first column of ls -ls shows the test files use no disk blocks, because they're sparse.

Exercise 4: A worker that stops promptly and cleanlyhard

Write worker-ex.sh: it creates a temporary state directory, then loops forever printing batch N and sleeping 10 seconds per batch. On SIGTERM or Ctrl-C it must react within a second, kill its sleep, remove the state directory, and exit with 143 for SIGTERM or 130 for SIGINT. Test both signals.

Hint

Let the signal traps only set a variable, and check it in the loop condition. Run sleep in the background and wait for it so the trap can run immediately, and remember $! so the cleanup can kill it.

Show Solution
#!/usr/bin/env bash
# worker-ex.sh - a worker that stops promptly and cleanly on SIGTERM or Ctrl-C
set -euo pipefail
 
state_dir=$(mktemp -d)
stop_signal=""
sleep_pid=""
 
cleanup() {
  if [[ -n $sleep_pid ]]; then
    kill "$sleep_pid" 2>/dev/null || true
  fi
  rm -rf -- "$state_dir"
  echo "$(date +%T) cleaned up $state_dir"
}
trap cleanup EXIT
trap 'stop_signal=TERM' TERM
trap 'stop_signal=INT' INT
 
echo "$(date +%T) worker $$ started"
batch=0
while [[ -z $stop_signal ]]; do
  batch=$(( batch + 1 ))
  echo "$(date +%T) batch $batch"
  sleep 10 &
  sleep_pid=$!
  wait "$sleep_pid" || true    # a trapped signal makes wait return early (status > 128)
done
 
echo "$(date +%T) got SIG$stop_signal after batch $batch, exiting"
if [[ $stop_signal == TERM ]]; then
  exit 143
fi
exit 130

With SIGTERM sent by the driver from the signals section, and then run directly with Ctrl-C pressed at the terminal:

./signal-driver.sh worker-ex.sh
22:00:58 worker 65865 started
22:00:58 batch 1
22:01:00 driver: kill -TERM 65865
22:01:00 got SIGTERM after batch 1, exiting
22:01:00 cleaned up /tmp/tmp.Fv3nERfzQg
22:01:00 driver: worker-ex.sh exited with status 143
./worker-ex.sh
echo "exit $?"
22:01:02 worker 66076 started
22:01:02 batch 1
^C22:01:05 got SIGINT after batch 1, exiting
22:01:05 cleaned up /tmp/tmp.vAX2SwhQXc
exit 130

Keeping the trap handlers to a single assignment is a good habit: the real work happens in normal code, at a point where the script is in a known state. || true after wait is needed because of set -e: an interrupted wait returns a status above 128, which would otherwise end the script before it reaches its own exit code.


Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2) with ShellCheck 0.9.0, November 2026. Every script was run as shown; Ctrl-C was sent through a real terminal, and SIGTERM by a small driver script that starts the target in the background, waits two seconds and runs kill -TERM. The two surprises while testing were that Bash 5.2 runs the EXIT trap on SIGTERM even without a TERM trap, and the $?-after-$(date) bug in my own driver, which is now in Common mistakes. The ${var:?} guard was demonstrated with ls instead of rm -rf.

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.