Writing Production-Ready Scripts
Build a reusable Bash script template step by step: strict mode, getopts, logging, dry run, safe config files, flock locking, trap cleanup, idempotency and exit codes.
By The Dark DebuggerUpdated 37 min read
This article turns a typical ten-line "works on my machine" script into one you can hand to cron, a CI job or a colleague without worrying. We build a reusable template piece by piece, test every piece including the failure paths, and finish with the complete template. It pulls together error handling and traps, loops and functions and the scheduling and locking from system monitoring and cron. The next article uses the template for three complete tools.
Why this exists: the first draft
The task: a reporting system drops CSV exports into a directory. Finished files should be
compressed into archive/YYYY/MM/ by the month they were produced, and removed from the
inbox. Here is the version almost everyone writes first:
#!/bin/bash
# archive-draft.sh - first draft: works on my machine
SRC=$1
DEST=$2
cd $SRC
for f in $(ls *.csv); do
month=$(date -r $f +%Y/%m)
mkdir -p $DEST/$month
gzip -c $f > $DEST/$month/$f.gz
rm $f
done
echo "done"It works on the author's test files. Then the finance team exports a file called
Q3 revenue report.csv:
./archive-draft.sh exports $PWD/archive
find exports archive -type f | sortdate: Q3: No such file or directory
gzip: Q3: No such file or directory
rm: cannot remove 'Q3': No such file or directory
date: revenue: No such file or directory
gzip: revenue: No such file or directory
rm: cannot remove 'revenue': No such file or directory
date: report.csv: No such file or directory
gzip: report.csv: No such file or directory
rm: cannot remove 'report.csv': No such file or directory
done
archive/2026/10/orders-2026-10-30.csv.gz
archive/Q3.gz
archive/report.csv.gz
archive/revenue.gz
exports/Q3 revenue report.csvWord splitting turned one file into three, the script created three empty .gz files,
and it still printed done. That is the mild failure. Here is the bad one, a typo in the
source directory, run from a directory that happens to contain a CSV:
ls
../archive-draft.sh exprots /tmp/archive
lsimportant-ledger.csv
../archive-draft.sh: line 5: cd: exprots: No such file or directory
doneThe second ls printed nothing. cd failed, the script carried on in the current
directory, archived important-ledger.csv somewhere nobody would look, and deleted it.
In cron, the current directory is your home directory.
shellcheck would have flagged both problems before the script ever ran (output trimmed):
shellcheck archive-draft.shIn archive-draft.sh line 5:
cd $SRC
^-----^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
Did you mean:
cd "$SRC" || exit
In archive-draft.sh line 6:
for f in $(ls *.csv); do
^---------^ SC2045 (error): Iterating over ls output is fragile. Use globs.
^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.
[…]The rest of this article fixes this script properly, one concern at a time. The finished
tool is called archive-exports.
Strict mode: set -Eeuo pipefail
The first line after the header in every production script:
set -Eeuo pipefailEach letter closes a specific hole:
bash -c 'cd /no/such/dir; echo "still running in $PWD"'
bash -c 'set -e; cd /no/such/dir; echo "still running"'; echo "exit: $?"bash: line 1: cd: /no/such/dir: No such file or directory
still running in /home/sid/prod
bash: line 1: cd: /no/such/dir: No such file or directory
exit: 1-e (errexit) stops the script when a command fails, instead of continuing in a
broken state. That alone would have saved important-ledger.csv.
bash -c 'set -u; rm -rf "$BUILD_DIR/"* ; echo after'; echo "exit: $?"bash: line 1: BUILD_DIR: unbound variable
exit: 127-u (nounset) makes an unset variable an error. Without it, "$BUILD_DIR/"* with an
unset variable expands to /*, and you can guess the rest.
bash -c 'grep -c ERROR missing.log | sort; echo "status: $?"'
bash -c 'set -o pipefail; grep -c ERROR missing.log | sort; echo "status: $?"'grep: missing.log: No such file or directory
status: 0
grep: missing.log: No such file or directory
status: 2-o pipefail makes a pipeline fail if any command in it fails. Without it, a
pipeline's status is that of the last command (sort, which happily sorted nothing).
-E (errtrace) makes the ERR trap, which we add below for error reporting, fire
inside functions too. Without it, a failure inside a function exits silently.
Strict mode is a safety net, not a guarantee: set -e is ignored in some contexts, which
is the first of the common mistakes below. The
error handling article covers the rules in depth.
Header, settings and precedence
The top of the script says what it does, how to call it and what its exit codes mean, because the next person to read it will be someone debugging it at 3 a.m. Settings get defaults in one place:
readonly SCRIPT_NAME=${0##*/}
readonly VERSION="1.0.0"
# Settings. Precedence: these defaults < config file (-c) < command-line options.
SOURCE_DIR=""
ARCHIVE_DIR=""
MIN_AGE=10
LOG_FILE=""
LOCK_FILE="${TMPDIR:-/tmp}/$SCRIPT_NAME.lock"${0##*/} strips the directory from the script's path, so messages say
archive-exports, not /usr/local/bin/archive-exports. The precedence rule (defaults,
then config file, then command-line options) is the one every well-behaved Unix tool
follows: a config file sets the normal values, and a one-off -m 5 on the command line
overrides it for one run.
usage() and getopts
usage() is documentation that cannot drift away from the script:
./archive-exports -hUsage: archive-exports [options] -s SOURCE_DIR -a ARCHIVE_DIR
Compress finished CSV exports from SOURCE_DIR into ARCHIVE_DIR/YYYY/MM/.
Options:
-s DIR directory containing *.csv exports (required)
-a DIR archive directory (required)
-m MIN only archive files unchanged for MIN minutes (default: 10)
-c FILE read settings from FILE (KEY=value lines)
-l FILE also append log lines to FILE
-n dry run: show what would happen, change nothing
-v verbose (debug) logging
-h show this help and exit
-V show the version and exit
Exit codes: 0 ok, 1 failure, 2 usage error, 3 config error,
4 missing dependency, 5 another run in progressOptions are parsed with the getopts builtin:
parse_args() {
local opt
while getopts ':s:a:m:c:l:nvhV' opt; do
case $opt in
s) opt_source=$OPTARG ;;
a) opt_archive=$OPTARG ;;
m) opt_min_age=$OPTARG ;;
c) CONFIG_FILE=$OPTARG ;;
l) opt_log=$OPTARG ;;
n) DRY_RUN=1 ;;
v) VERBOSE=1 ;;
h) usage; exit 0 ;;
V) echo "$SCRIPT_NAME $VERSION"; exit 0 ;;
:) usage_error "option -$OPTARG requires an argument" ;;
\?) usage_error "unknown option: -$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
if (( $# > 0 )); then
usage_error "unexpected argument: $1"
fi
}How to read the option string ':s:a:m:c:l:nvhV':
- A letter followed by
:takes an argument (-s DIR), available in$OPTARG. - A letter alone is a flag (
-n). - The leading
:switches getopts to silent mode: instead of printing its own message, it returns:for a missing argument and?for an unknown option, with the option letter in$OPTARG, so the script can report errors its own way.
getopts handles combined flags (-nv), arguments attached or separate (-m5 or
-m 5) and stops at --. It only does short options; if you need --source=DIR, you
parse with a while/case loop over "$@" yourself.
Parsed values go into opt_* variables rather than straight into the settings, so that
they can be applied after the config file is read. Required options and value formats
are validated in one place:
apply_settings() {
if [[ -n $CONFIG_FILE ]]; then load_config "$CONFIG_FILE"; fi
if [[ -n $opt_source ]]; then SOURCE_DIR=$opt_source; fi
if [[ -n $opt_archive ]]; then ARCHIVE_DIR=$opt_archive; fi
if [[ -n $opt_min_age ]]; then MIN_AGE=$opt_min_age; fi
if [[ -n $opt_log ]]; then LOG_FILE=$opt_log; fi
[[ -n $SOURCE_DIR ]] || usage_error "missing required option -s SOURCE_DIR"
[[ -n $ARCHIVE_DIR ]] || usage_error "missing required option -a ARCHIVE_DIR"
[[ $MIN_AGE =~ ^[0-9]+$ ]] || usage_error "-m needs a whole number of minutes, got '$MIN_AGE'"
[[ -d $SOURCE_DIR ]] || die 1 "source directory does not exist: $SOURCE_DIR"
}Every way of calling it wrong produces a clear message and a distinct exit status:
./archive-exports -s exports; echo "exit: $?"
./archive-exports -s exports -a archive -m; echo "exit: $?"
./archive-exports -s exports -a archive -m ten; echo "exit: $?"
./archive-exports -x -s exports -a archive; echo "exit: $?"
./archive-exports -s exprots -a archive; echo "exit: $?"2026-11-10T01:19:16-0500 archive-exports[4217] ERROR: missing required option -a ARCHIVE_DIR
Try 'archive-exports -h' for more information.
exit: 2
2026-11-10T01:19:16-0500 archive-exports[4220] ERROR: option -m requires an argument
Try 'archive-exports -h' for more information.
exit: 2
2026-11-10T01:19:16-0500 archive-exports[4223] ERROR: -m needs a whole number of minutes, got 'ten'
Try 'archive-exports -h' for more information.
exit: 2
2026-11-10T01:19:16-0500 archive-exports[4226] ERROR: unknown option: -x
Try 'archive-exports -h' for more information.
exit: 2
2026-11-10T01:19:16-0500 archive-exports[4229] ERROR: source directory does not exist: exprots
exit: 1Compare the last one with the first draft: the same typo now stops the script before it touches anything.
Logging with timestamps and levels
A script that runs unattended needs a record of what it did. The rules the template follows:
- Log to stderr, not stdout. Stdout is for the script's real output (data another program might consume); diagnostics go to stderr, which cron and systemd capture separately.
- Every line has a timestamp, the script name, the PID and a level, so lines from several runs or several scripts in one log file can be told apart.
- Optionally also append to a file (
-l FILE), for cron jobs that need their own log.
log() {
local level=$1 line
shift
line="$(date '+%Y-%m-%dT%H:%M:%S%z') ${SCRIPT_NAME}[$$] $level: $*"
printf '%s\n' "$line" >&2
if [[ -n $LOG_FILE ]]; then
printf '%s\n' "$line" >> "$LOG_FILE"
fi
}
debug() { if (( VERBOSE )); then log DEBUG "$@"; fi; }
info() { log INFO "$@"; }
warn() { log WARN "$@"; }
error() { log ERROR "$@"; }
die() { local code=$1; shift; error "$@"; exit "$code"; }
usage_error() {
error "$*"
echo "Try '$SCRIPT_NAME -h' for more information." >&2
exit 2
}Two details. debug uses an if rather than (( VERBOSE )) && log …: written the short
way, the function would return status 1 whenever verbose mode is off, and under set -e
a function returning 1 stops the script. And the braces in ${SCRIPT_NAME}[$$] are
required. My first version had $SCRIPT_NAME[$$], which shellcheck rejected with
SC1087 (error): Use braces when expanding arrays, because $name[...] looks like a
mistyped array access.
A real run with a log file:
./archive-exports -s exports -a archive -l archive-exports.log; echo "exit: $?"2026-11-10T01:19:16-0500 archive-exports[4267] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: archived Q3 revenue report.csv -> 2026/11/Q3 revenue report.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: archived customers-2026-10-31.csv -> 2026/10/customers-2026-10-31.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: archived orders-2026-10-30.csv -> 2026/10/orders-2026-10-30.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: done: 3 archived, 0 already archived, 0 conflicts
exit: 0The file with spaces in its name went where it should. The ISO 8601 timestamp with a UTC
offset sorts correctly and stays unambiguous across daylight-saving changes. If the script
runs under systemd, you can drop the timestamp (the journal adds its own) or send lines to
logger -t "$SCRIPT_NAME" instead of a file.
A dry-run mode
For anything that deletes or moves files, -n should show exactly what would happen and
change nothing. The trick is to route every state-changing command through one function:
# run CMD [ARGS...]: execute, or just log it when -n was given
run() {
if (( DRY_RUN )); then
info "dry-run: $*"
else
debug "run: $*"
"$@"
fi
}./archive-exports -n -s exports -a archive; echo "exit: $?"
find archive -type f | wc -l2026-11-10T01:19:16-0500 archive-exports[4232] INFO: start: source=exports archive=archive min_age=10m dry_run=1
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: mkdir -p -- archive/2026/11
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: compress_into exports/Q3 revenue report.csv archive/2026/11/Q3 revenue report.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: rm -f -- exports/Q3 revenue report.csv
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: mkdir -p -- archive/2026/10
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: compress_into exports/customers-2026-10-31.csv archive/2026/10/customers-2026-10-31.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: rm -f -- exports/customers-2026-10-31.csv
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: mkdir -p -- archive/2026/10
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: compress_into exports/orders-2026-10-30.csv archive/2026/10/orders-2026-10-30.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: rm -f -- exports/orders-2026-10-30.csv
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: done (dry run, nothing changed): 3 archived, 0 already archived, 0 conflicts
exit: 0
0run "$@" executes its arguments as a command, so it cannot handle a redirection like
gzip -c f > out; the redirection would apply to run itself. The fix is to put that
step in a small function (compress_into, below) and call run compress_into …. Notice
also that Q3 revenue report.csv is ambiguous in the dry-run output; exercise 2 fixes
that.
Config files without source
The tempting way to read a config file is source archive-exports.conf. Don't: source
executes the file as Bash code, with the script's privileges. This is a config file a
colleague (or an attacker with write access to /etc) could edit:
SOURCE_DIR=$(touch /tmp/pwned)/exports
ARCHIVE_DIR=archivebash -c 'source evil.conf; echo "SOURCE_DIR=$SOURCE_DIR"'; ls /tmp/pwnedSOURCE_DIR=/exports
/tmp/pwnedThe command substitution ran. Instead, the template parses KEY=value lines itself and
accepts only keys it knows:
# Read KEY=value lines. Only known keys are accepted; nothing is executed.
load_config() {
local file=$1 line key value n=0
local re_skip='^[[:space:]]*(#.*)?$'
local re_pair='^[[:space:]]*([A-Z_]+)[[:space:]]*=[[:space:]]*(.*)$'
[[ -r $file ]] || die 3 "cannot read config file: $file"
while IFS= read -r line || [[ -n $line ]]; do
n=$((n + 1))
[[ $line =~ $re_skip ]] && continue
[[ $line =~ $re_pair ]] || die 3 "$file:$n: expected KEY=value: $line"
key=${BASH_REMATCH[1]}
value=${BASH_REMATCH[2]}
value=${value%"${value##*[![:space:]]}"} # trim trailing spaces
value=${value#\"} value=${value%\"} # strip optional quotes
case $key in
SOURCE_DIR | ARCHIVE_DIR | MIN_AGE | LOG_FILE | LOCK_FILE)
printf -v "$key" '%s' "$value" ;;
*)
die 3 "$file:$n: unknown setting '$key'" ;;
esac
done < "$file"
debug "loaded settings from $file"
}- The regexes live in variables; writing them inline in
[[ =~ ]]invites quoting bugs. || [[ -n $line ]]handles a last line without a trailing newline, which editors on other systems often produce.printf -v "$key"assigns to the variable named by$key, safely, withouteval.- Unknown keys are an error, not ignored: a typo like
ARCHVE_DIRshould fail loudly.
The same evil file, read by the template, is just a strange directory name:
./archive-exports -c evil.conf; echo "exit: $?"; ls /tmp/pwned2026-11-10T01:19:16-0500 archive-exports[4368] ERROR: source directory does not exist: $(touch /tmp/pwned)/exports
exit: 1
ls: cannot access '/tmp/pwned': No such file or directoryA normal config file, with a command-line override and -v to see what happened:
# archive-exports settings
SOURCE_DIR=/home/sid/prod/exports
ARCHIVE_DIR = "/home/sid/prod/archive"
MIN_AGE=60
LOG_FILE=/home/sid/prod/archive-exports.log./archive-exports -v -c archive-exports.conf -m 5; echo "exit: $?"
cat bad.conf; ./archive-exports -c bad.conf; echo "exit: $?"2026-11-10T01:19:16-0500 archive-exports[4349] DEBUG: loaded settings from archive-exports.conf
2026-11-10T01:19:16-0500 archive-exports[4349] DEBUG: holding lock /tmp/archive-exports.lock
2026-11-10T01:19:16-0500 archive-exports[4349] INFO: start: source=/home/sid/prod/exports archive=/home/sid/prod/archive min_age=5m dry_run=0
2026-11-10T01:19:16-0500 archive-exports[4349] INFO: done: 0 archived, 0 already archived, 0 conflicts
exit: 0
SOURCE_DIR=/srv/exports
RETENTION_DAYS=30
2026-11-10T01:19:16-0500 archive-exports[4363] ERROR: bad.conf:2: unknown setting 'RETENTION_DAYS'
exit: 3The config said 60 minutes, -m 5 won. Values are taken literally, so ~ is not
expanded; use absolute paths in config files.
Checking dependencies up front
A script that fails halfway because flock is not installed has already done half its
work. Check everything it needs before doing anything:
require_commands() {
local cmd missing=()
for cmd in "$@"; do
command -v "$cmd" > /dev/null 2>&1 || missing+=("$cmd")
done
if (( ${#missing[@]} > 0 )); then
die 4 "missing required commands: ${missing[*]}"
fi
}command -v is the portable, builtin way to ask "is this on PATH?" (which is an
external program with inconsistent behavior across systems). Collecting all missing
commands before failing saves the "install one, rerun, install the next" loop. To see the
failure path, I added two tools the lab machine does not have to the
require_commands gzip find flock date mktemp cmp line:
./archive-exports-zstd -s exports -a archive; echo "exit: $?"2026-11-10T01:19:16-0500 archive-exports-zstd[4378] ERROR: missing required commands: zstd pigz
exit: 4Locking with flock
If cron starts the next run while the previous one is still compressing a big file, two copies will race on the same files. The lock pattern from the cron article, inside the script:
acquire_lock() {
exec {lock_fd}> "$LOCK_FILE"
if ! flock -n "$lock_fd"; then
die 5 "another run is in progress (lock: $LOCK_FILE)"
fi
debug "holding lock $LOCK_FILE"
}exec {lock_fd}> file asks Bash to open the file on a free descriptor number and store
the number in lock_fd, so you never collide with a descriptor the script already uses.
The lock lives as long as the process; the kernel releases it on exit, even after
kill -9. To test it, hold the lock from another process:
flock /tmp/archive-exports.lock sleep 3 & sleep 0.5; ./archive-exports -s exports -a archive; echo "exit: $?"; wait2026-11-10T01:19:17-0500 archive-exports[4429] ERROR: another run is in progress (lock: /tmp/archive-exports.lock)
exit: 5Put the lock file somewhere every possible caller can write and nobody cleans
mid-run: /tmp or $XDG_RUNTIME_DIR for a single user, /run/lock or /var/lock for
system jobs. The lock file itself is left behind after a run; that is harmless, because
the lock is the open descriptor, not the file's existence.
Temp files, traps and atomic writes
Three pieces work together so that a failure at any moment leaves no half-written archive behind.
Write to a temp file, then rename. mv within one filesystem is atomic: the archive
either exists completely or not at all. The temp file goes in the destination directory,
because a rename across filesystems is a copy and no longer atomic:
# compress_into SRC DEST: write DEST atomically via a temp file in the same directory
compress_into() {
local src=$1 dest=$2
TMP_FILE=$(mktemp "${dest%/*}/.${dest##*/}.XXXXXX")
gzip -c -- "$src" > "$TMP_FILE"
chmod --reference="$src" -- "$TMP_FILE" # mktemp creates files as 0600
mv -f -- "$TMP_FILE" "$dest"
TMP_FILE=""
}The chmod line was added after testing. My first version produced archives with mode
-rw-------, readable only by the user who ran the script, because mktemp creates
files with mode 0600 for safety, and mv keeps the temp file's permissions. Nobody else
could read the archives. --reference copies the source file's mode instead.
Clean up on every exit. The EXIT trap runs however the script ends:
cleanup() {
local rc=$?
if [[ -n $TMP_FILE && -e $TMP_FILE ]]; then
rm -f -- "$TMP_FILE"
debug "removed temporary file $TMP_FILE"
fi
exit "$rc"
}
on_error() {
local rc=$? line=$1
error "line $line: '$BASH_COMMAND' exited with status $rc"
}
trap cleanup EXIT
trap 'on_error $LINENO' ERR
trap 'die 130 "interrupted"' INT
trap 'die 143 "terminated"' TERMReport where it failed. The ERR trap fires just before set -e exits, and
$BASH_COMMAND holds the command that failed. Here a CSV exported by another user is not
readable:
./archive-exports -s exports -a archive; echo "exit: $?"
ls -A archive/2026/11/
grep -n 'gzip -c -- "$src"' archive-exports2026-11-10T01:19:19-0500 archive-exports[4769] INFO: start: source=exports archive=archive min_age=10m dry_run=0
gzip: exports/invoices-2026-11-01.csv: Permission denied
2026-11-10T01:19:19-0500 archive-exports[4769] ERROR: line 195: 'gzip -c -- "$src" > "$TMP_FILE"' exited with status 1
exit: 1
Q3 revenue report.csv.gz
195: gzip -c -- "$src" > "$TMP_FILE"The error names the exact line and command, and ls -A (which shows dot files) proves the
temp file mktemp had already created was removed.
Now an interruption. A 1 GB export takes several seconds to compress; we stop the run
in the middle the way systemctl stop does, by sending SIGTERM to the whole process
group (setsid gives the script its own group so the kill -- -PGID form can target it):
setsid ./archive-exports -v -s exports -a archive & sleep 1; ls -A archive/2026/11/; kill -TERM -- -$!; wait $!; echo "exit: $?"; ls -A archive/2026/11/2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: holding lock /tmp/archive-exports.lock
2026-11-10T01:19:19-0500 archive-exports[4793] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: mkdir -p -- archive/2026/11
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: compress_into exports/invoices-2026-11-01.csv archive/2026/11/invoices-2026-11-01.csv.gz
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: rm -f -- exports/invoices-2026-11-01.csv
2026-11-10T01:19:19-0500 archive-exports[4793] INFO: archived invoices-2026-11-01.csv -> 2026/11/invoices-2026-11-01.csv.gz
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: mkdir -p -- archive/2026/11
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: compress_into exports/warehouse-snapshot-2026-11-03.csv archive/2026/11/warehouse-snapshot-2026-11-03.csv.gz
.warehouse-snapshot-2026-11-03.csv.gz.1pLGq7
Q3 revenue report.csv.gz
invoices-2026-11-01.csv.gz
Terminated
2026-11-10T01:19:20-0500 archive-exports[4793] ERROR: terminated
2026-11-10T01:19:20-0500 archive-exports[4793] DEBUG: removed temporary file archive/2026/11/.warehouse-snapshot-2026-11-03.csv.gz.1pLGq7
exit: 143
Q3 revenue report.csv.gz
invoices-2026-11-01.csv.gz(I had restored the invoice file's permissions, so it was archived this time.) While the
big file compressed, the half-written .warehouse-…XXXXXX temp file was visible. After
SIGTERM, the gzip child died (Terminated), the TERM trap logged and exited with 143,
the EXIT trap removed the temp file, and the source CSV was untouched for the next run.
If only the script's own PID gets the signal, the behavior from
process management shows up: Bash runs the trap
only after the foreground gzip finishes.
./archive-exports -s exports -a archive & sleep 1; kill -TERM $!; wait $!; echo "exit: $?"; ls -A archive/2026/11/2026-11-10T01:19:20-0500 archive-exports[4845] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:26-0500 archive-exports[4845] ERROR: terminated
exit: 143
Q3 revenue report.csv.gz
invoices-2026-11-01.csv.gzSix seconds of delay, but the same clean result. systemd, Ctrl-C and timeout all signal
the whole group, so in practice you get the fast path.
Idempotency: safe to run twice
An idempotent script can be run again after any failure, or by accident, and ends in the same correct state. That is what makes "just rerun it" a safe answer at 3 a.m. Running the archiver a second time finds nothing to do:
./archive-exports -s exports -a archive; echo "exit: $?"2026-11-10T01:19:16-0500 archive-exports[4312] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:16-0500 archive-exports[4312] INFO: done: 0 archived, 0 already archived, 0 conflicts
exit: 0The harder cases are the in-between states. If a run dies after writing the archive but
before deleting the source, the next run finds both. And sometimes a file is re-exported
with corrected data under the same name. archive_file checks before acting:
archive_file() {
local src=$1 name month_dir dest
name=${src##*/}
month_dir="$ARCHIVE_DIR/$(date -r "$src" +%Y/%m)"
dest="$month_dir/$name.gz"
if [[ -e $dest ]]; then
if gzip -dc -- "$dest" | cmp -s - "$src"; then
info "already archived, removing source: $name"
run rm -f -- "$src"
already=$((already + 1))
else
warn "conflict: ${dest#"$ARCHIVE_DIR"/} exists with different content, leaving $name"
conflicts=$((conflicts + 1))
fi
return 0
fi
run mkdir -p -- "$month_dir"
run compress_into "$src" "$dest"
run rm -f -- "$src"
if (( ! DRY_RUN )); then
info "archived $name -> ${dest#"$ARCHIVE_DIR"/}"
fi
archived=$((archived + 1))
}I recreated both situations: an already-archived file put back into the inbox, and a corrected re-export of the customers file.
./archive-exports -s exports -a archive; echo "exit: $?"
ls exports2026-11-10T01:19:16-0500 archive-exports[4324] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:16-0500 archive-exports[4324] WARN: conflict: 2026/10/customers-2026-10-31.csv.gz exists with different content, leaving customers-2026-10-31.csv
2026-11-10T01:19:16-0500 archive-exports[4324] INFO: already archived, removing source: orders-2026-10-30.csv
2026-11-10T01:19:16-0500 archive-exports[4324] INFO: done: 0 archived, 1 already archived, 1 conflicts
exit: 1customers-2026-10-31.csv
notes.txt
orders-2026-11-10.csvThe duplicate was cleaned up because the content was provably identical. The conflict was
not resolved automatically: overwriting an archive is a decision for a human, so the
script leaves the file, warns, and exits 1 so monitoring notices. Other idempotency habits
used here: mkdir -p instead of mkdir, rm -f instead of rm, and the -mmin +10
filter that skips files still being written.
Exit codes
Exit codes are the script's API for cron, systemd, CI and other scripts. Document them in
usage() and keep them stable.
| Code | Meaning in this template | Notes |
|---|---|---|
| 0 | Success | Including "nothing to do" |
| 1 | Runtime failure | Something went wrong while working |
| 2 | Usage error | Bad or missing options; same convention as Bash builtins and GNU tools |
| 3 | Config error | Unreadable file, unknown key, bad syntax |
| 4 | Missing dependency | A required command is not installed |
| 5 | Already running | The lock is held; often not worth an alert |
| 126 / 127 | Reserved by the shell | Command not executable / not found |
| 128 + N | Reserved by the shell | Killed by signal N (130 INT, 143 TERM, 137 KILL) |
Some teams use the BSD sysexits.h values instead (64 usage, 69 unavailable, 75
temporary failure, 78 config). Either is fine; what matters is that "already running"
and "broken" are different numbers, so a scheduler can retry the first and page someone
for the second.
shellcheck clean
Run shellcheck on every change. It knows hundreds of Bash pitfalls, and the template
passes with no output:
shellcheck archive-exports test-archive-exports.sh && echo cleancleanWhen shellcheck is wrong about a specific line (it occasionally is), disable that one
check on that one line with a reason, rather than turning it off globally. For example,
this help text really does want a literal $HOME:
echo 'Use $HOME in the config file, it is expanded later'In sc1.sh line 4:
echo 'Use $HOME in the config file, it is expanded later'
^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.# shellcheck disable=SC2016 # literal text for the user, expanded later by the app
echo 'Use $HOME in the config file, it is expanded later'With the directive in place, shellcheck is clean again, and the comment tells the next reader why.
The complete template
Here is the whole script. Copy it, rename it, replace archive_file and the find in
main with your own work, and keep everything else.
#!/usr/bin/env bash
#
# archive-exports - compress finished CSV exports into dated archive folders
#
# Moves *.csv files that have not changed for MIN_AGE minutes from SOURCE_DIR
# to ARCHIVE_DIR/YYYY/MM/NAME.csv.gz. Safe to re-run and safe to run from cron.
#
# Exit codes: 0 success, 1 runtime failure, 2 usage error, 3 config error,
# 4 missing dependency, 5 another run in progress
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
readonly VERSION="1.0.0"
# Settings. Precedence: these defaults < config file (-c) < command-line options.
SOURCE_DIR=""
ARCHIVE_DIR=""
MIN_AGE=10
LOG_FILE=""
LOCK_FILE="${TMPDIR:-/tmp}/$SCRIPT_NAME.lock"
CONFIG_FILE=""
DRY_RUN=0
VERBOSE=0
opt_source="" opt_archive="" opt_min_age="" opt_log=""
TMP_FILE=""
archived=0 already=0 conflicts=0
usage() {
cat <<EOF
Usage: $SCRIPT_NAME [options] -s SOURCE_DIR -a ARCHIVE_DIR
Compress finished CSV exports from SOURCE_DIR into ARCHIVE_DIR/YYYY/MM/.
Options:
-s DIR directory containing *.csv exports (required)
-a DIR archive directory (required)
-m MIN only archive files unchanged for MIN minutes (default: $MIN_AGE)
-c FILE read settings from FILE (KEY=value lines)
-l FILE also append log lines to FILE
-n dry run: show what would happen, change nothing
-v verbose (debug) logging
-h show this help and exit
-V show the version and exit
Exit codes: 0 ok, 1 failure, 2 usage error, 3 config error,
4 missing dependency, 5 another run in progress
EOF
}
# ---- logging: everything goes to stderr, and to LOG_FILE if set -------------
log() {
local level=$1 line
shift
line="$(date '+%Y-%m-%dT%H:%M:%S%z') ${SCRIPT_NAME}[$$] $level: $*"
printf '%s\n' "$line" >&2
if [[ -n $LOG_FILE ]]; then
printf '%s\n' "$line" >> "$LOG_FILE"
fi
}
debug() { if (( VERBOSE )); then log DEBUG "$@"; fi; }
info() { log INFO "$@"; }
warn() { log WARN "$@"; }
error() { log ERROR "$@"; }
die() { local code=$1; shift; error "$@"; exit "$code"; }
usage_error() {
error "$*"
echo "Try '$SCRIPT_NAME -h' for more information." >&2
exit 2
}
# ---- cleanup and error reporting --------------------------------------------
cleanup() {
local rc=$?
if [[ -n $TMP_FILE && -e $TMP_FILE ]]; then
rm -f -- "$TMP_FILE"
debug "removed temporary file $TMP_FILE"
fi
exit "$rc"
}
on_error() {
local rc=$? line=$1
error "line $line: '$BASH_COMMAND' exited with status $rc"
}
trap cleanup EXIT
trap 'on_error $LINENO' ERR
trap 'die 130 "interrupted"' INT
trap 'die 143 "terminated"' TERM
# ---- helpers ------------------------------------------------------------------
# run CMD [ARGS...]: execute, or just log it when -n was given
run() {
if (( DRY_RUN )); then
info "dry-run: $*"
else
debug "run: $*"
"$@"
fi
}
require_commands() {
local cmd missing=()
for cmd in "$@"; do
command -v "$cmd" > /dev/null 2>&1 || missing+=("$cmd")
done
if (( ${#missing[@]} > 0 )); then
die 4 "missing required commands: ${missing[*]}"
fi
}
# Read KEY=value lines. Only known keys are accepted; nothing is executed.
load_config() {
local file=$1 line key value n=0
local re_skip='^[[:space:]]*(#.*)?$'
local re_pair='^[[:space:]]*([A-Z_]+)[[:space:]]*=[[:space:]]*(.*)$'
[[ -r $file ]] || die 3 "cannot read config file: $file"
while IFS= read -r line || [[ -n $line ]]; do
n=$((n + 1))
[[ $line =~ $re_skip ]] && continue
[[ $line =~ $re_pair ]] || die 3 "$file:$n: expected KEY=value: $line"
key=${BASH_REMATCH[1]}
value=${BASH_REMATCH[2]}
value=${value%"${value##*[![:space:]]}"} # trim trailing spaces
value=${value#\"} value=${value%\"} # strip optional quotes
case $key in
SOURCE_DIR | ARCHIVE_DIR | MIN_AGE | LOG_FILE | LOCK_FILE)
printf -v "$key" '%s' "$value" ;;
*)
die 3 "$file:$n: unknown setting '$key'" ;;
esac
done < "$file"
debug "loaded settings from $file"
}
acquire_lock() {
exec {lock_fd}> "$LOCK_FILE"
if ! flock -n "$lock_fd"; then
die 5 "another run is in progress (lock: $LOCK_FILE)"
fi
debug "holding lock $LOCK_FILE"
}
# ---- argument handling ----------------------------------------------------------
parse_args() {
local opt
while getopts ':s:a:m:c:l:nvhV' opt; do
case $opt in
s) opt_source=$OPTARG ;;
a) opt_archive=$OPTARG ;;
m) opt_min_age=$OPTARG ;;
c) CONFIG_FILE=$OPTARG ;;
l) opt_log=$OPTARG ;;
n) DRY_RUN=1 ;;
v) VERBOSE=1 ;;
h) usage; exit 0 ;;
V) echo "$SCRIPT_NAME $VERSION"; exit 0 ;;
:) usage_error "option -$OPTARG requires an argument" ;;
\?) usage_error "unknown option: -$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
if (( $# > 0 )); then
usage_error "unexpected argument: $1"
fi
}
apply_settings() {
if [[ -n $CONFIG_FILE ]]; then load_config "$CONFIG_FILE"; fi
if [[ -n $opt_source ]]; then SOURCE_DIR=$opt_source; fi
if [[ -n $opt_archive ]]; then ARCHIVE_DIR=$opt_archive; fi
if [[ -n $opt_min_age ]]; then MIN_AGE=$opt_min_age; fi
if [[ -n $opt_log ]]; then LOG_FILE=$opt_log; fi
[[ -n $SOURCE_DIR ]] || usage_error "missing required option -s SOURCE_DIR"
[[ -n $ARCHIVE_DIR ]] || usage_error "missing required option -a ARCHIVE_DIR"
[[ $MIN_AGE =~ ^[0-9]+$ ]] || usage_error "-m needs a whole number of minutes, got '$MIN_AGE'"
[[ -d $SOURCE_DIR ]] || die 1 "source directory does not exist: $SOURCE_DIR"
}
# ---- the actual work ------------------------------------------------------------
# compress_into SRC DEST: write DEST atomically via a temp file in the same directory
compress_into() {
local src=$1 dest=$2
TMP_FILE=$(mktemp "${dest%/*}/.${dest##*/}.XXXXXX")
gzip -c -- "$src" > "$TMP_FILE"
chmod --reference="$src" -- "$TMP_FILE" # mktemp creates files as 0600
mv -f -- "$TMP_FILE" "$dest"
TMP_FILE=""
}
archive_file() {
local src=$1 name month_dir dest
name=${src##*/}
month_dir="$ARCHIVE_DIR/$(date -r "$src" +%Y/%m)"
dest="$month_dir/$name.gz"
if [[ -e $dest ]]; then
if gzip -dc -- "$dest" | cmp -s - "$src"; then
info "already archived, removing source: $name"
run rm -f -- "$src"
already=$((already + 1))
else
warn "conflict: ${dest#"$ARCHIVE_DIR"/} exists with different content, leaving $name"
conflicts=$((conflicts + 1))
fi
return 0
fi
run mkdir -p -- "$month_dir"
run compress_into "$src" "$dest"
run rm -f -- "$src"
if (( ! DRY_RUN )); then
info "archived $name -> ${dest#"$ARCHIVE_DIR"/}"
fi
archived=$((archived + 1))
}
main() {
parse_args "$@"
apply_settings
require_commands gzip find flock date mktemp cmp
acquire_lock
info "start: source=$SOURCE_DIR archive=$ARCHIVE_DIR min_age=${MIN_AGE}m dry_run=$DRY_RUN"
local file
while IFS= read -r -d '' file; do
archive_file "$file"
done < <(find "$SOURCE_DIR" -maxdepth 1 -type f -name '*.csv' -mmin +"$MIN_AGE" -print0 | sort -z)
local note=""
if (( DRY_RUN )); then note=" (dry run, nothing changed)"; fi
info "done${note}: $archived archived, $already already archived, $conflicts conflicts"
if (( conflicts > 0 )); then
exit 1
fi
}
main "$@"Putting everything in functions and calling main "$@" on the last line has a practical
benefit: Bash reads the whole file before running anything, so editing the script while a
long run is in progress cannot corrupt the running copy. (Bash reads scripts
incrementally; without the main wrapper, an edit mid-run can make the running process
execute half of the old line and half of the new one.)
Where you'll use this
Deploying it as a scheduled job
On a server, the script goes to /usr/local/bin/archive-exports, its settings to
/etc/archive-exports.conf, and cron or a systemd timer calls it with only -c:
# /etc/cron.d/archive-exports
MAILTO=""
*/15 * * * * reports /usr/local/bin/archive-exports -c /etc/archive-exports.conf 2>/dev/nullStderr is discarded there because the config sets LOG_FILE, so every line also lands in
the log file. Before trusting it, run it the way cron will, with an almost empty
environment:
env -i PATH=/usr/bin:/bin HOME=$HOME ./archive-exports -c archive-exports.conf 2>/dev/null; echo "exit: $?"
cat archive-exports.logexit: 0
2026-11-10T01:19:27-0500 archive-exports[5023] INFO: start: source=/home/sid/prod/exports archive=/home/sid/prod/archive min_age=60m dry_run=0
2026-11-10T01:19:27-0500 archive-exports[5023] INFO: archived stock-2026-11-05.csv -> 2026/11/stock-2026-11-05.csv.gz
2026-11-10T01:19:27-0500 archive-exports[5023] INFO: done: 1 archived, 0 already archived, 0 conflictsWith a systemd timer instead, drop LOG_FILE and let the journal collect stderr.
Testing it
A script with clear exit codes is easy to test with another script. This one builds a throwaway sandbox, runs each scenario and compares exit statuses:
#!/usr/bin/env bash
# test-archive-exports.sh - scenario tests for archive-exports
set -uo pipefail
script=$(realpath ./archive-exports)
pass=0 fail=0
check() { # check DESCRIPTION EXPECTED_STATUS ACTUAL_STATUS
if [[ $2 == "$3" ]]; then
echo "PASS $1"
pass=$((pass + 1))
else
echo "FAIL $1 (expected status $2, got $3)"
fail=$((fail + 1))
fi
}
sandbox=$(mktemp -d)
trap 'rm -rf "$sandbox"' EXIT
cd "$sandbox" || exit 1
export TMPDIR=$sandbox # keeps the lock file inside the sandbox
mkdir exports archive
printf 'order_id,total\n70001,19.99\n' > exports/orders.csv
touch -d '2 hours ago' exports/orders.csv
month=$(date -d '2 hours ago' +%Y/%m)
"$script" > /dev/null 2>&1
check "no arguments is a usage error" 2 $?
"$script" -s exports -a archive -m soon > /dev/null 2>&1
check "non-numeric -m is a usage error" 2 $?
"$script" -n -s exports -a archive > /dev/null 2>&1
[[ -f exports/orders.csv && ! -e archive/$month ]]
check "dry run changes nothing" 0 $?
"$script" -s exports -a archive > /dev/null 2>&1
check "first run succeeds" 0 $?
cmp -s <(gzip -dc "archive/$month/orders.csv.gz") <(printf 'order_id,total\n70001,19.99\n')
check "archive has the original content" 0 $?
"$script" -s exports -a archive > /dev/null 2>&1
check "second run succeeds and does nothing" 0 $?
flock "$TMPDIR/archive-exports.lock" sleep 2 &
sleep 0.3
"$script" -s exports -a archive > /dev/null 2>&1
check "a concurrent run is refused" 5 $?
wait
echo "$pass passed, $fail failed"
(( fail == 0 ))./test-archive-exports.sh; echo "exit: $?"PASS no arguments is a usage error
PASS non-numeric -m is a usage error
PASS dry run changes nothing
PASS first run succeeds
PASS archive has the original content
PASS second run succeeds and does nothing
PASS a concurrent run is refused
7 passed, 0 failed
exit: 0The test script deliberately uses set -uo pipefail without -e, because it expects
commands to fail and checks their statuses. Run it in CI next to shellcheck, and every
change to the script is checked in seconds.
Common mistakes
set -e does nothing inside a function called by if
#!/usr/bin/env bash
set -e
backup() {
cp /srv/db/nightly.dump /backups/
echo "backup: copied, now rotating old dumps"
}
if backup; then
echo "backup succeeded"
fibash set-e-if.shcp: cannot stat '/srv/db/nightly.dump': No such file or directory
backup: copied, now rotating old dumps
backup succeededset -e is suspended for everything that runs as part of an if condition, a while
condition, or the left side of && / ||, including whole functions called there.
The cp failed, the function kept going, and its last command succeeded, so the backup
"succeeded". Inside functions that may be used as conditions, check critical commands
explicitly: cp … || return 1.
local var=$(cmd) hides the failure
#!/usr/bin/env bash
set -e
get_version() {
local version=$(cat /opt/shop/VERSION)
echo "deploying version '$version'"
}
get_versionbash local-mask.sh
shellcheck local-mask.shcat: /opt/shop/VERSION: No such file or directory
deploying version ''In local-mask.sh line 4:
local version=$(cat /opt/shop/VERSION)
^-----^ SC2155 (warning): Declare and assign separately to avoid masking return values.The exit status of that line is the status of local, which succeeded. Declare first,
assign second (local version then version=$(cat …)), which is why the template's
functions always do it in two steps.
getopts without the leading colon
#!/usr/bin/env bash
set -u
while getopts 'nv' opt; do
case $opt in
n) echo "dry run" ;;
v) echo "verbose" ;;
?) echo "unknown option: -$OPTARG" >&2; exit 2 ;;
esac
donebash getopts-nocolon.sh -x; echo "exit: $?"getopts-nocolon.sh: illegal option -- x
getopts-nocolon.sh: line 7: OPTARG: unbound variable
exit: 1Without the leading :, getopts prints its own message and leaves OPTARG unset, which
set -u then turns into a second, confusing error and the wrong exit code. With
':nv', getopts stays quiet and puts the bad letter in OPTARG.
Summary
- Start every script with
set -Eeuo pipefail, a header that documents usage and exit codes, and settings with defaults in one place; let a config file override the defaults and command-line options override both. - Parse options with
getopts ':…'(leading colon), validate everything before doing any work, and exit 2 on usage errors. - Log to stderr with timestamp, script name, PID and level; optionally tee to a file.
- Route every state-changing command through a
runfunction so-ncan show the plan without executing it. - Never
sourcea config file you do not fully control; parseKEY=valuewith an allow-list andprintf -v. - Check dependencies with
command -vup front; take aflocklock on a dedicated file descriptor; write outputs to a temp file in the destination directory andmvthem into place; clean up in anEXITtrap and report failures from anERRtrap. - Make reruns safe: check for existing results, compare content, refuse to overwrite on conflict. Use distinct exit codes for "failed", "misused" and "already running".
- Keep it
shellcheck-clean and test it with a script that asserts exit codes.
Exercises
Add a -q option: INFO and DEBUG lines are no longer printed to stderr, but WARN and
ERROR still are, and the log file (if -l is given) still receives every line. Update
usage(). Make sure shellcheck stays clean.
Show Solution
The changes, as a diff against the template:
+QUIET=0
[…]
+ -q quiet: print only warnings and errors (the log file gets everything)
[…]
- printf '%s\n' "$line" >&2
+ if (( ! QUIET )) || [[ $level == WARN || $level == ERROR ]]; then
+ printf '%s\n' "$line" >&2
+ fi
[…]
- while getopts ':s:a:m:c:l:nvhV' opt; do
+ while getopts ':s:a:m:c:l:nqvhV' opt; do
[…]
+ q) QUIET=1 ;;With one new file to archive and one conflict:
./archive-exports-q -q -s exports -a archive -l quiet.log; echo "exit: $?"
cat quiet.log2026-11-10T01:18:46-0500 archive-exports-q[956] WARN: conflict: 2026/10/customers-2026-10-03.csv.gz exists with different content, leaving customers-2026-10-03.csv
exit: 1
2026-11-10T01:18:46-0500 archive-exports-q[956] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:18:46-0500 archive-exports-q[956] WARN: conflict: 2026/10/customers-2026-10-03.csv.gz exists with different content, leaving customers-2026-10-03.csv
2026-11-10T01:18:46-0500 archive-exports-q[956] INFO: archived orders-2026-10-02.csv -> 2026/10/orders-2026-10-02.csv.gz
2026-11-10T01:18:46-0500 archive-exports-q[956] INFO: done: 1 archived, 0 already archived, 1 conflictsThe terminal (or cron's mail) shows only the line that needs attention; the log keeps the
full story. When I first added only the getopts and case lines, shellcheck caught
the missing piece: SC2034 (warning): QUIET appears unused.
In the dry-run output, compress_into exports/Q3 revenue report.csv … is ambiguous: you
cannot tell where one argument ends. Change run so each argument is shell-quoted, and the
logged command could be pasted into a terminal as is.
Hint
printf '%q' prints a string quoted for reuse as shell input.
Show Solution
run() {
if (( DRY_RUN )); then
info "dry-run:$(printf ' %q' "$@")"
else
debug "run: $*"
"$@"
fi
}./ex2 -n -s exports -a archive 2>&1 | grep dry-run2026-11-09T22:04:55-0500 ex2[79637] INFO: dry-run: mkdir -p -- archive/2026/11
2026-11-09T22:04:55-0500 ex2[79637] INFO: dry-run: compress_into exports/Q3\ revenue\ report.csv archive/2026/11/Q3\ revenue\ report.csv.gz
2026-11-09T22:04:55-0500 ex2[79637] INFO: dry-run: rm -f -- exports/Q3\ revenue\ report.csvprintf ' %q' "$@" applies the format once per argument, so each gets a leading space
and its own quoting. Spaces became \ , and the argument boundaries are unambiguous.
Add -k MONTHS: after archiving, delete ARCHIVE_DIR/YYYY/MM folders older than the
current month minus MONTHS (so -k 3 in November keeps August to November). It must
honor -n, touch nothing that is not a YYYY/MM folder, and reject -k 0 or non-numbers
as a usage error.
Show Solution
New pieces (plus KEEP_MONTHS="" with the other settings and k: in the getopts string
with k) KEEP_MONTHS=$OPTARG ;;):
# in apply_settings
[[ -z $KEEP_MONTHS || $KEEP_MONTHS =~ ^[1-9][0-9]*$ ]] ||
usage_error "-k needs a positive number of months, got '$KEEP_MONTHS'"
# prune_archive N: delete ARCHIVE_DIR/YYYY/MM folders older than the current
# month minus N months (N=3 in November keeps Aug, Sep, Oct and Nov)
prune_archive() {
local keep=$1 cutoff dir ym
cutoff=$(date -d "$(date +%Y-%m-01) -$keep months" +%Y%m)
while IFS= read -r -d '' dir; do
ym=${dir#"$ARCHIVE_DIR"/}
ym=${ym//\//} # 2026/07 -> 202607
if (( ym < cutoff )); then
info "pruning ${dir#"$ARCHIVE_DIR"/} (older than $keep months)"
run rm -rf -- "$dir"
fi
done < <(find "$ARCHIVE_DIR" -mindepth 2 -maxdepth 2 -type d \
-regextype posix-extended -regex '.*/[0-9]{4}/[0-9]{2}' -print0 | sort -z)
}
# in main, after the archiving loop
if [[ -n $KEEP_MONTHS ]]; then
prune_archive "$KEEP_MONTHS"
fiTest archive with months from December 2025 to November 2026 and an unrelated
tmp-restore folder:
./archive-exports-k -n -k 3 -s exports -a archive; echo "exit: $?"
./archive-exports-k -k 3 -s exports -a archive; echo "exit: $?"
ls archive archive/2026
./archive-exports-k -k 0 -s exports -a archive; echo "exit: $?"2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: start: source=exports archive=archive min_age=10m dry_run=1
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: pruning 2025/12 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: dry-run: rm -rf -- archive/2025/12
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: pruning 2026/05 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: dry-run: rm -rf -- archive/2026/05
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: pruning 2026/06 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: dry-run: rm -rf -- archive/2026/06
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: pruning 2026/07 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: dry-run: rm -rf -- archive/2026/07
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: done (dry run, nothing changed): 0 archived, 0 already archived, 0 conflicts
exit: 0
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: pruning 2025/12 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: pruning 2026/05 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: pruning 2026/06 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: pruning 2026/07 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: done: 0 archived, 0 already archived, 0 conflicts
exit: 0
archive:
2025
2026
tmp-restore
archive/2026:
08
09
10
11
2026-11-10T01:18:46-0500 archive-exports-k[1075] ERROR: -k needs a positive number of months, got '0'
Try 'archive-exports-k -h' for more information.
exit: 2Comparing YYYYMM as integers avoids date arithmetic on folder names. Computing the
cutoff from the first of the month matters: date -d "2026-03-31 -1 month" lands
in March again (there is no 31 February), while the first of a month always exists. The
-regex filter is the safety rail: tmp-restore was never considered. The now-empty
2025 folder stays; find "$ARCHIVE_DIR" -mindepth 1 -type d -empty -delete would tidy
it if you want that.
Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), ShellCheck 0.9.0, util-linux
2.39.3, November 2026. Every listing was run as shown in a scratch directory with
generated CSV exports; the 1 GB file used for the interruption test was a sparse file, so
it compressed in seconds without using real disk space. The /etc/cron.d entry was not
installed (it needs root); the same command was run under env -i to reproduce cron's
environment instead. Two details came straight out of testing: the 0600 permissions that
mktemp gave the first archives, and shellcheck's SC1087 on the log line. Paths from the test machine's scratch directories are shown shortened to /home/sid/….
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.