Writing Production-Ready Scripts

Build a reusable Bash script template step by step: strict mode, getopts, logging, dry run, safe config files, flock locking, trap cleanup, idempotency and exit codes.

By The Dark DebuggerUpdated 37 min read

This article turns a typical ten-line "works on my machine" script into one you can hand to cron, a CI job or a colleague without worrying. We build a reusable template piece by piece, test every piece including the failure paths, and finish with the complete template. It pulls together error handling and traps, loops and functions and the scheduling and locking from system monitoring and cron. The next article uses the template for three complete tools.

Why this exists: the first draft

The task: a reporting system drops CSV exports into a directory. Finished files should be compressed into archive/YYYY/MM/ by the month they were produced, and removed from the inbox. Here is the version almost everyone writes first:

#!/bin/bash
# archive-draft.sh - first draft: works on my machine
SRC=$1
DEST=$2
cd $SRC
for f in $(ls *.csv); do
  month=$(date -r $f +%Y/%m)
  mkdir -p $DEST/$month
  gzip -c $f > $DEST/$month/$f.gz
  rm $f
done
echo "done"

It works on the author's test files. Then the finance team exports a file called Q3 revenue report.csv:

./archive-draft.sh exports $PWD/archive
find exports archive -type f | sort
date: Q3: No such file or directory
gzip: Q3: No such file or directory
rm: cannot remove 'Q3': No such file or directory
date: revenue: No such file or directory
gzip: revenue: No such file or directory
rm: cannot remove 'revenue': No such file or directory
date: report.csv: No such file or directory
gzip: report.csv: No such file or directory
rm: cannot remove 'report.csv': No such file or directory
done
archive/2026/10/orders-2026-10-30.csv.gz
archive/Q3.gz
archive/report.csv.gz
archive/revenue.gz
exports/Q3 revenue report.csv

Word splitting turned one file into three, the script created three empty .gz files, and it still printed done. That is the mild failure. Here is the bad one, a typo in the source directory, run from a directory that happens to contain a CSV:

ls
../archive-draft.sh exprots /tmp/archive
ls
important-ledger.csv
../archive-draft.sh: line 5: cd: exprots: No such file or directory
done

The second ls printed nothing. cd failed, the script carried on in the current directory, archived important-ledger.csv somewhere nobody would look, and deleted it. In cron, the current directory is your home directory.

shellcheck would have flagged both problems before the script ever ran (output trimmed):

shellcheck archive-draft.sh
In archive-draft.sh line 5:
cd $SRC
^-----^ SC2164 (warning): Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
   ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
 
Did you mean: 
cd "$SRC" || exit
 
 
In archive-draft.sh line 6:
for f in $(ls *.csv); do
         ^---------^ SC2045 (error): Iterating over ls output is fragile. Use globs.
              ^-- SC2035 (info): Use ./*glob* or -- *glob* so names with dashes won't become options.
[…]

The rest of this article fixes this script properly, one concern at a time. The finished tool is called archive-exports.

Strict mode: set -Eeuo pipefail

The first line after the header in every production script:

set -Eeuo pipefail

Each letter closes a specific hole:

bash -c 'cd /no/such/dir; echo "still running in $PWD"'
bash -c 'set -e; cd /no/such/dir; echo "still running"'; echo "exit: $?"
bash: line 1: cd: /no/such/dir: No such file or directory
still running in /home/sid/prod
bash: line 1: cd: /no/such/dir: No such file or directory
exit: 1

-e (errexit) stops the script when a command fails, instead of continuing in a broken state. That alone would have saved important-ledger.csv.

bash -c 'set -u; rm -rf "$BUILD_DIR/"* ; echo after'; echo "exit: $?"
bash: line 1: BUILD_DIR: unbound variable
exit: 127

-u (nounset) makes an unset variable an error. Without it, "$BUILD_DIR/"* with an unset variable expands to /*, and you can guess the rest.

bash -c 'grep -c ERROR missing.log | sort; echo "status: $?"'
bash -c 'set -o pipefail; grep -c ERROR missing.log | sort; echo "status: $?"'
grep: missing.log: No such file or directory
status: 0
grep: missing.log: No such file or directory
status: 2

-o pipefail makes a pipeline fail if any command in it fails. Without it, a pipeline's status is that of the last command (sort, which happily sorted nothing).

-E (errtrace) makes the ERR trap, which we add below for error reporting, fire inside functions too. Without it, a failure inside a function exits silently.

Strict mode is a safety net, not a guarantee: set -e is ignored in some contexts, which is the first of the common mistakes below. The error handling article covers the rules in depth.

Header, settings and precedence

The top of the script says what it does, how to call it and what its exit codes mean, because the next person to read it will be someone debugging it at 3 a.m. Settings get defaults in one place:

readonly SCRIPT_NAME=${0##*/}
readonly VERSION="1.0.0"
 
# Settings. Precedence: these defaults < config file (-c) < command-line options.
SOURCE_DIR=""
ARCHIVE_DIR=""
MIN_AGE=10
LOG_FILE=""
LOCK_FILE="${TMPDIR:-/tmp}/$SCRIPT_NAME.lock"

${0##*/} strips the directory from the script's path, so messages say archive-exports, not /usr/local/bin/archive-exports. The precedence rule (defaults, then config file, then command-line options) is the one every well-behaved Unix tool follows: a config file sets the normal values, and a one-off -m 5 on the command line overrides it for one run.

usage() and getopts

usage() is documentation that cannot drift away from the script:

./archive-exports -h
Usage: archive-exports [options] -s SOURCE_DIR -a ARCHIVE_DIR
 
Compress finished CSV exports from SOURCE_DIR into ARCHIVE_DIR/YYYY/MM/.
 
Options:
  -s DIR    directory containing *.csv exports (required)
  -a DIR    archive directory (required)
  -m MIN    only archive files unchanged for MIN minutes (default: 10)
  -c FILE   read settings from FILE (KEY=value lines)
  -l FILE   also append log lines to FILE
  -n        dry run: show what would happen, change nothing
  -v        verbose (debug) logging
  -h        show this help and exit
  -V        show the version and exit
 
Exit codes: 0 ok, 1 failure, 2 usage error, 3 config error,
            4 missing dependency, 5 another run in progress

Options are parsed with the getopts builtin:

parse_args() {
  local opt
  while getopts ':s:a:m:c:l:nvhV' opt; do
    case $opt in
      s) opt_source=$OPTARG ;;
      a) opt_archive=$OPTARG ;;
      m) opt_min_age=$OPTARG ;;
      c) CONFIG_FILE=$OPTARG ;;
      l) opt_log=$OPTARG ;;
      n) DRY_RUN=1 ;;
      v) VERBOSE=1 ;;
      h) usage; exit 0 ;;
      V) echo "$SCRIPT_NAME $VERSION"; exit 0 ;;
      :) usage_error "option -$OPTARG requires an argument" ;;
      \?) usage_error "unknown option: -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))
  if (( $# > 0 )); then
    usage_error "unexpected argument: $1"
  fi
}

How to read the option string ':s:a:m:c:l:nvhV':

  • A letter followed by : takes an argument (-s DIR), available in $OPTARG.
  • A letter alone is a flag (-n).
  • The leading : switches getopts to silent mode: instead of printing its own message, it returns : for a missing argument and ? for an unknown option, with the option letter in $OPTARG, so the script can report errors its own way.

getopts handles combined flags (-nv), arguments attached or separate (-m5 or -m 5) and stops at --. It only does short options; if you need --source=DIR, you parse with a while/case loop over "$@" yourself.

Parsed values go into opt_* variables rather than straight into the settings, so that they can be applied after the config file is read. Required options and value formats are validated in one place:

apply_settings() {
  if [[ -n $CONFIG_FILE ]]; then load_config "$CONFIG_FILE"; fi
  if [[ -n $opt_source ]]; then SOURCE_DIR=$opt_source; fi
  if [[ -n $opt_archive ]]; then ARCHIVE_DIR=$opt_archive; fi
  if [[ -n $opt_min_age ]]; then MIN_AGE=$opt_min_age; fi
  if [[ -n $opt_log ]]; then LOG_FILE=$opt_log; fi
 
  [[ -n $SOURCE_DIR ]] || usage_error "missing required option -s SOURCE_DIR"
  [[ -n $ARCHIVE_DIR ]] || usage_error "missing required option -a ARCHIVE_DIR"
  [[ $MIN_AGE =~ ^[0-9]+$ ]] || usage_error "-m needs a whole number of minutes, got '$MIN_AGE'"
  [[ -d $SOURCE_DIR ]] || die 1 "source directory does not exist: $SOURCE_DIR"
}

Every way of calling it wrong produces a clear message and a distinct exit status:

./archive-exports -s exports; echo "exit: $?"
./archive-exports -s exports -a archive -m; echo "exit: $?"
./archive-exports -s exports -a archive -m ten; echo "exit: $?"
./archive-exports -x -s exports -a archive; echo "exit: $?"
./archive-exports -s exprots -a archive; echo "exit: $?"
2026-11-10T01:19:16-0500 archive-exports[4217] ERROR: missing required option -a ARCHIVE_DIR
Try 'archive-exports -h' for more information.
exit: 2
2026-11-10T01:19:16-0500 archive-exports[4220] ERROR: option -m requires an argument
Try 'archive-exports -h' for more information.
exit: 2
2026-11-10T01:19:16-0500 archive-exports[4223] ERROR: -m needs a whole number of minutes, got 'ten'
Try 'archive-exports -h' for more information.
exit: 2
2026-11-10T01:19:16-0500 archive-exports[4226] ERROR: unknown option: -x
Try 'archive-exports -h' for more information.
exit: 2
2026-11-10T01:19:16-0500 archive-exports[4229] ERROR: source directory does not exist: exprots
exit: 1

Compare the last one with the first draft: the same typo now stops the script before it touches anything.

Logging with timestamps and levels

A script that runs unattended needs a record of what it did. The rules the template follows:

  • Log to stderr, not stdout. Stdout is for the script's real output (data another program might consume); diagnostics go to stderr, which cron and systemd capture separately.
  • Every line has a timestamp, the script name, the PID and a level, so lines from several runs or several scripts in one log file can be told apart.
  • Optionally also append to a file (-l FILE), for cron jobs that need their own log.
log() {
  local level=$1 line
  shift
  line="$(date '+%Y-%m-%dT%H:%M:%S%z') ${SCRIPT_NAME}[$$] $level: $*"
  printf '%s\n' "$line" >&2
  if [[ -n $LOG_FILE ]]; then
    printf '%s\n' "$line" >> "$LOG_FILE"
  fi
}
debug() { if (( VERBOSE )); then log DEBUG "$@"; fi; }
info()  { log INFO "$@"; }
warn()  { log WARN "$@"; }
error() { log ERROR "$@"; }
die()   { local code=$1; shift; error "$@"; exit "$code"; }
 
usage_error() {
  error "$*"
  echo "Try '$SCRIPT_NAME -h' for more information." >&2
  exit 2
}

Two details. debug uses an if rather than (( VERBOSE )) && log …: written the short way, the function would return status 1 whenever verbose mode is off, and under set -e a function returning 1 stops the script. And the braces in ${SCRIPT_NAME}[$$] are required. My first version had $SCRIPT_NAME[$$], which shellcheck rejected with SC1087 (error): Use braces when expanding arrays, because $name[...] looks like a mistyped array access.

A real run with a log file:

./archive-exports -s exports -a archive -l archive-exports.log; echo "exit: $?"
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: archived Q3 revenue report.csv -> 2026/11/Q3 revenue report.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: archived customers-2026-10-31.csv -> 2026/10/customers-2026-10-31.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: archived orders-2026-10-30.csv -> 2026/10/orders-2026-10-30.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4267] INFO: done: 3 archived, 0 already archived, 0 conflicts
exit: 0

The file with spaces in its name went where it should. The ISO 8601 timestamp with a UTC offset sorts correctly and stays unambiguous across daylight-saving changes. If the script runs under systemd, you can drop the timestamp (the journal adds its own) or send lines to logger -t "$SCRIPT_NAME" instead of a file.

A dry-run mode

For anything that deletes or moves files, -n should show exactly what would happen and change nothing. The trick is to route every state-changing command through one function:

# run CMD [ARGS...]: execute, or just log it when -n was given
run() {
  if (( DRY_RUN )); then
    info "dry-run: $*"
  else
    debug "run: $*"
    "$@"
  fi
}
./archive-exports -n -s exports -a archive; echo "exit: $?"
find archive -type f | wc -l
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: start: source=exports archive=archive min_age=10m dry_run=1
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: mkdir -p -- archive/2026/11
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: compress_into exports/Q3 revenue report.csv archive/2026/11/Q3 revenue report.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: rm -f -- exports/Q3 revenue report.csv
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: mkdir -p -- archive/2026/10
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: compress_into exports/customers-2026-10-31.csv archive/2026/10/customers-2026-10-31.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: rm -f -- exports/customers-2026-10-31.csv
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: mkdir -p -- archive/2026/10
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: compress_into exports/orders-2026-10-30.csv archive/2026/10/orders-2026-10-30.csv.gz
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: dry-run: rm -f -- exports/orders-2026-10-30.csv
2026-11-10T01:19:16-0500 archive-exports[4232] INFO: done (dry run, nothing changed): 3 archived, 0 already archived, 0 conflicts
exit: 0
0

run "$@" executes its arguments as a command, so it cannot handle a redirection like gzip -c f > out; the redirection would apply to run itself. The fix is to put that step in a small function (compress_into, below) and call run compress_into …. Notice also that Q3 revenue report.csv is ambiguous in the dry-run output; exercise 2 fixes that.

Config files without source

The tempting way to read a config file is source archive-exports.conf. Don't: source executes the file as Bash code, with the script's privileges. This is a config file a colleague (or an attacker with write access to /etc) could edit:

SOURCE_DIR=$(touch /tmp/pwned)/exports
ARCHIVE_DIR=archive
bash -c 'source evil.conf; echo "SOURCE_DIR=$SOURCE_DIR"'; ls /tmp/pwned
SOURCE_DIR=/exports
/tmp/pwned

The command substitution ran. Instead, the template parses KEY=value lines itself and accepts only keys it knows:

# Read KEY=value lines. Only known keys are accepted; nothing is executed.
load_config() {
  local file=$1 line key value n=0
  local re_skip='^[[:space:]]*(#.*)?$'
  local re_pair='^[[:space:]]*([A-Z_]+)[[:space:]]*=[[:space:]]*(.*)$'
 
  [[ -r $file ]] || die 3 "cannot read config file: $file"
  while IFS= read -r line || [[ -n $line ]]; do
    n=$((n + 1))
    [[ $line =~ $re_skip ]] && continue
    [[ $line =~ $re_pair ]] || die 3 "$file:$n: expected KEY=value: $line"
    key=${BASH_REMATCH[1]}
    value=${BASH_REMATCH[2]}
    value=${value%"${value##*[![:space:]]}"}   # trim trailing spaces
    value=${value#\"} value=${value%\"}         # strip optional quotes
    case $key in
      SOURCE_DIR | ARCHIVE_DIR | MIN_AGE | LOG_FILE | LOCK_FILE)
        printf -v "$key" '%s' "$value" ;;
      *)
        die 3 "$file:$n: unknown setting '$key'" ;;
    esac
  done < "$file"
  debug "loaded settings from $file"
}
  • The regexes live in variables; writing them inline in [[ =~ ]] invites quoting bugs.
  • || [[ -n $line ]] handles a last line without a trailing newline, which editors on other systems often produce.
  • printf -v "$key" assigns to the variable named by $key, safely, without eval.
  • Unknown keys are an error, not ignored: a typo like ARCHVE_DIR should fail loudly.

The same evil file, read by the template, is just a strange directory name:

./archive-exports -c evil.conf; echo "exit: $?"; ls /tmp/pwned
2026-11-10T01:19:16-0500 archive-exports[4368] ERROR: source directory does not exist: $(touch /tmp/pwned)/exports
exit: 1
ls: cannot access '/tmp/pwned': No such file or directory

A normal config file, with a command-line override and -v to see what happened:

# archive-exports settings
SOURCE_DIR=/home/sid/prod/exports
ARCHIVE_DIR = "/home/sid/prod/archive"
MIN_AGE=60
LOG_FILE=/home/sid/prod/archive-exports.log
./archive-exports -v -c archive-exports.conf -m 5; echo "exit: $?"
cat bad.conf; ./archive-exports -c bad.conf; echo "exit: $?"
2026-11-10T01:19:16-0500 archive-exports[4349] DEBUG: loaded settings from archive-exports.conf
2026-11-10T01:19:16-0500 archive-exports[4349] DEBUG: holding lock /tmp/archive-exports.lock
2026-11-10T01:19:16-0500 archive-exports[4349] INFO: start: source=/home/sid/prod/exports archive=/home/sid/prod/archive min_age=5m dry_run=0
2026-11-10T01:19:16-0500 archive-exports[4349] INFO: done: 0 archived, 0 already archived, 0 conflicts
exit: 0
SOURCE_DIR=/srv/exports
RETENTION_DAYS=30
2026-11-10T01:19:16-0500 archive-exports[4363] ERROR: bad.conf:2: unknown setting 'RETENTION_DAYS'
exit: 3

The config said 60 minutes, -m 5 won. Values are taken literally, so ~ is not expanded; use absolute paths in config files.

Checking dependencies up front

A script that fails halfway because flock is not installed has already done half its work. Check everything it needs before doing anything:

require_commands() {
  local cmd missing=()
  for cmd in "$@"; do
    command -v "$cmd" > /dev/null 2>&1 || missing+=("$cmd")
  done
  if (( ${#missing[@]} > 0 )); then
    die 4 "missing required commands: ${missing[*]}"
  fi
}

command -v is the portable, builtin way to ask "is this on PATH?" (which is an external program with inconsistent behavior across systems). Collecting all missing commands before failing saves the "install one, rerun, install the next" loop. To see the failure path, I added two tools the lab machine does not have to the require_commands gzip find flock date mktemp cmp line:

./archive-exports-zstd -s exports -a archive; echo "exit: $?"
2026-11-10T01:19:16-0500 archive-exports-zstd[4378] ERROR: missing required commands: zstd pigz
exit: 4

Locking with flock

If cron starts the next run while the previous one is still compressing a big file, two copies will race on the same files. The lock pattern from the cron article, inside the script:

acquire_lock() {
  exec {lock_fd}> "$LOCK_FILE"
  if ! flock -n "$lock_fd"; then
    die 5 "another run is in progress (lock: $LOCK_FILE)"
  fi
  debug "holding lock $LOCK_FILE"
}

exec {lock_fd}> file asks Bash to open the file on a free descriptor number and store the number in lock_fd, so you never collide with a descriptor the script already uses. The lock lives as long as the process; the kernel releases it on exit, even after kill -9. To test it, hold the lock from another process:

flock /tmp/archive-exports.lock sleep 3 & sleep 0.5; ./archive-exports -s exports -a archive; echo "exit: $?"; wait
2026-11-10T01:19:17-0500 archive-exports[4429] ERROR: another run is in progress (lock: /tmp/archive-exports.lock)
exit: 5

Put the lock file somewhere every possible caller can write and nobody cleans mid-run: /tmp or $XDG_RUNTIME_DIR for a single user, /run/lock or /var/lock for system jobs. The lock file itself is left behind after a run; that is harmless, because the lock is the open descriptor, not the file's existence.

Temp files, traps and atomic writes

Three pieces work together so that a failure at any moment leaves no half-written archive behind.

Write to a temp file, then rename. mv within one filesystem is atomic: the archive either exists completely or not at all. The temp file goes in the destination directory, because a rename across filesystems is a copy and no longer atomic:

# compress_into SRC DEST: write DEST atomically via a temp file in the same directory
compress_into() {
  local src=$1 dest=$2
  TMP_FILE=$(mktemp "${dest%/*}/.${dest##*/}.XXXXXX")
  gzip -c -- "$src" > "$TMP_FILE"
  chmod --reference="$src" -- "$TMP_FILE"    # mktemp creates files as 0600
  mv -f -- "$TMP_FILE" "$dest"
  TMP_FILE=""
}

The chmod line was added after testing. My first version produced archives with mode -rw-------, readable only by the user who ran the script, because mktemp creates files with mode 0600 for safety, and mv keeps the temp file's permissions. Nobody else could read the archives. --reference copies the source file's mode instead.

Clean up on every exit. The EXIT trap runs however the script ends:

cleanup() {
  local rc=$?
  if [[ -n $TMP_FILE && -e $TMP_FILE ]]; then
    rm -f -- "$TMP_FILE"
    debug "removed temporary file $TMP_FILE"
  fi
  exit "$rc"
}
 
on_error() {
  local rc=$? line=$1
  error "line $line: '$BASH_COMMAND' exited with status $rc"
}
 
trap cleanup EXIT
trap 'on_error $LINENO' ERR
trap 'die 130 "interrupted"' INT
trap 'die 143 "terminated"' TERM

Report where it failed. The ERR trap fires just before set -e exits, and $BASH_COMMAND holds the command that failed. Here a CSV exported by another user is not readable:

./archive-exports -s exports -a archive; echo "exit: $?"
ls -A archive/2026/11/
grep -n 'gzip -c -- "$src"' archive-exports
2026-11-10T01:19:19-0500 archive-exports[4769] INFO: start: source=exports archive=archive min_age=10m dry_run=0
gzip: exports/invoices-2026-11-01.csv: Permission denied
2026-11-10T01:19:19-0500 archive-exports[4769] ERROR: line 195: 'gzip -c -- "$src" > "$TMP_FILE"' exited with status 1
exit: 1
Q3 revenue report.csv.gz
195:  gzip -c -- "$src" > "$TMP_FILE"

The error names the exact line and command, and ls -A (which shows dot files) proves the temp file mktemp had already created was removed.

Now an interruption. A 1 GB export takes several seconds to compress; we stop the run in the middle the way systemctl stop does, by sending SIGTERM to the whole process group (setsid gives the script its own group so the kill -- -PGID form can target it):

setsid ./archive-exports -v -s exports -a archive & sleep 1; ls -A archive/2026/11/; kill -TERM -- -$!; wait $!; echo "exit: $?"; ls -A archive/2026/11/
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: holding lock /tmp/archive-exports.lock
2026-11-10T01:19:19-0500 archive-exports[4793] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: mkdir -p -- archive/2026/11
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: compress_into exports/invoices-2026-11-01.csv archive/2026/11/invoices-2026-11-01.csv.gz
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: rm -f -- exports/invoices-2026-11-01.csv
2026-11-10T01:19:19-0500 archive-exports[4793] INFO: archived invoices-2026-11-01.csv -> 2026/11/invoices-2026-11-01.csv.gz
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: mkdir -p -- archive/2026/11
2026-11-10T01:19:19-0500 archive-exports[4793] DEBUG: run: compress_into exports/warehouse-snapshot-2026-11-03.csv archive/2026/11/warehouse-snapshot-2026-11-03.csv.gz
.warehouse-snapshot-2026-11-03.csv.gz.1pLGq7
Q3 revenue report.csv.gz
invoices-2026-11-01.csv.gz
Terminated
2026-11-10T01:19:20-0500 archive-exports[4793] ERROR: terminated
2026-11-10T01:19:20-0500 archive-exports[4793] DEBUG: removed temporary file archive/2026/11/.warehouse-snapshot-2026-11-03.csv.gz.1pLGq7
exit: 143
Q3 revenue report.csv.gz
invoices-2026-11-01.csv.gz

(I had restored the invoice file's permissions, so it was archived this time.) While the big file compressed, the half-written .warehouse-…XXXXXX temp file was visible. After SIGTERM, the gzip child died (Terminated), the TERM trap logged and exited with 143, the EXIT trap removed the temp file, and the source CSV was untouched for the next run.

If only the script's own PID gets the signal, the behavior from process management shows up: Bash runs the trap only after the foreground gzip finishes.

./archive-exports -s exports -a archive & sleep 1; kill -TERM $!; wait $!; echo "exit: $?"; ls -A archive/2026/11/
2026-11-10T01:19:20-0500 archive-exports[4845] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:26-0500 archive-exports[4845] ERROR: terminated
exit: 143
Q3 revenue report.csv.gz
invoices-2026-11-01.csv.gz

Six seconds of delay, but the same clean result. systemd, Ctrl-C and timeout all signal the whole group, so in practice you get the fast path.

Idempotency: safe to run twice

An idempotent script can be run again after any failure, or by accident, and ends in the same correct state. That is what makes "just rerun it" a safe answer at 3 a.m. Running the archiver a second time finds nothing to do:

./archive-exports -s exports -a archive; echo "exit: $?"
2026-11-10T01:19:16-0500 archive-exports[4312] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:16-0500 archive-exports[4312] INFO: done: 0 archived, 0 already archived, 0 conflicts
exit: 0

The harder cases are the in-between states. If a run dies after writing the archive but before deleting the source, the next run finds both. And sometimes a file is re-exported with corrected data under the same name. archive_file checks before acting:

archive_file() {
  local src=$1 name month_dir dest
  name=${src##*/}
  month_dir="$ARCHIVE_DIR/$(date -r "$src" +%Y/%m)"
  dest="$month_dir/$name.gz"
 
  if [[ -e $dest ]]; then
    if gzip -dc -- "$dest" | cmp -s - "$src"; then
      info "already archived, removing source: $name"
      run rm -f -- "$src"
      already=$((already + 1))
    else
      warn "conflict: ${dest#"$ARCHIVE_DIR"/} exists with different content, leaving $name"
      conflicts=$((conflicts + 1))
    fi
    return 0
  fi
 
  run mkdir -p -- "$month_dir"
  run compress_into "$src" "$dest"
  run rm -f -- "$src"
  if (( ! DRY_RUN )); then
    info "archived $name -> ${dest#"$ARCHIVE_DIR"/}"
  fi
  archived=$((archived + 1))
}

I recreated both situations: an already-archived file put back into the inbox, and a corrected re-export of the customers file.

./archive-exports -s exports -a archive; echo "exit: $?"
ls exports
2026-11-10T01:19:16-0500 archive-exports[4324] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:19:16-0500 archive-exports[4324] WARN: conflict: 2026/10/customers-2026-10-31.csv.gz exists with different content, leaving customers-2026-10-31.csv
2026-11-10T01:19:16-0500 archive-exports[4324] INFO: already archived, removing source: orders-2026-10-30.csv
2026-11-10T01:19:16-0500 archive-exports[4324] INFO: done: 0 archived, 1 already archived, 1 conflicts
exit: 1
customers-2026-10-31.csv
notes.txt
orders-2026-11-10.csv

The duplicate was cleaned up because the content was provably identical. The conflict was not resolved automatically: overwriting an archive is a decision for a human, so the script leaves the file, warns, and exits 1 so monitoring notices. Other idempotency habits used here: mkdir -p instead of mkdir, rm -f instead of rm, and the -mmin +10 filter that skips files still being written.

Exit codes

Exit codes are the script's API for cron, systemd, CI and other scripts. Document them in usage() and keep them stable.

CodeMeaning in this templateNotes
0SuccessIncluding "nothing to do"
1Runtime failureSomething went wrong while working
2Usage errorBad or missing options; same convention as Bash builtins and GNU tools
3Config errorUnreadable file, unknown key, bad syntax
4Missing dependencyA required command is not installed
5Already runningThe lock is held; often not worth an alert
126 / 127Reserved by the shellCommand not executable / not found
128 + NReserved by the shellKilled by signal N (130 INT, 143 TERM, 137 KILL)

Some teams use the BSD sysexits.h values instead (64 usage, 69 unavailable, 75 temporary failure, 78 config). Either is fine; what matters is that "already running" and "broken" are different numbers, so a scheduler can retry the first and page someone for the second.

shellcheck clean

Run shellcheck on every change. It knows hundreds of Bash pitfalls, and the template passes with no output:

shellcheck archive-exports test-archive-exports.sh && echo clean
clean

When shellcheck is wrong about a specific line (it occasionally is), disable that one check on that one line with a reason, rather than turning it off globally. For example, this help text really does want a literal $HOME:

echo 'Use $HOME in the config file, it is expanded later'
In sc1.sh line 4:
echo 'Use $HOME in the config file, it is expanded later'
     ^-- SC2016 (info): Expressions don't expand in single quotes, use double quotes for that.
# shellcheck disable=SC2016  # literal text for the user, expanded later by the app
echo 'Use $HOME in the config file, it is expanded later'

With the directive in place, shellcheck is clean again, and the comment tells the next reader why.

The complete template

Here is the whole script. Copy it, rename it, replace archive_file and the find in main with your own work, and keep everything else.

#!/usr/bin/env bash
#
# archive-exports - compress finished CSV exports into dated archive folders
#
# Moves *.csv files that have not changed for MIN_AGE minutes from SOURCE_DIR
# to ARCHIVE_DIR/YYYY/MM/NAME.csv.gz. Safe to re-run and safe to run from cron.
#
# Exit codes: 0 success, 1 runtime failure, 2 usage error, 3 config error,
#             4 missing dependency, 5 another run in progress
 
set -Eeuo pipefail
 
readonly SCRIPT_NAME=${0##*/}
readonly VERSION="1.0.0"
 
# Settings. Precedence: these defaults < config file (-c) < command-line options.
SOURCE_DIR=""
ARCHIVE_DIR=""
MIN_AGE=10
LOG_FILE=""
LOCK_FILE="${TMPDIR:-/tmp}/$SCRIPT_NAME.lock"
 
CONFIG_FILE=""
DRY_RUN=0
VERBOSE=0
opt_source="" opt_archive="" opt_min_age="" opt_log=""
TMP_FILE=""
archived=0 already=0 conflicts=0
 
usage() {
  cat <<EOF
Usage: $SCRIPT_NAME [options] -s SOURCE_DIR -a ARCHIVE_DIR
 
Compress finished CSV exports from SOURCE_DIR into ARCHIVE_DIR/YYYY/MM/.
 
Options:
  -s DIR    directory containing *.csv exports (required)
  -a DIR    archive directory (required)
  -m MIN    only archive files unchanged for MIN minutes (default: $MIN_AGE)
  -c FILE   read settings from FILE (KEY=value lines)
  -l FILE   also append log lines to FILE
  -n        dry run: show what would happen, change nothing
  -v        verbose (debug) logging
  -h        show this help and exit
  -V        show the version and exit
 
Exit codes: 0 ok, 1 failure, 2 usage error, 3 config error,
            4 missing dependency, 5 another run in progress
EOF
}
 
# ---- logging: everything goes to stderr, and to LOG_FILE if set -------------
 
log() {
  local level=$1 line
  shift
  line="$(date '+%Y-%m-%dT%H:%M:%S%z') ${SCRIPT_NAME}[$$] $level: $*"
  printf '%s\n' "$line" >&2
  if [[ -n $LOG_FILE ]]; then
    printf '%s\n' "$line" >> "$LOG_FILE"
  fi
}
debug() { if (( VERBOSE )); then log DEBUG "$@"; fi; }
info()  { log INFO "$@"; }
warn()  { log WARN "$@"; }
error() { log ERROR "$@"; }
die()   { local code=$1; shift; error "$@"; exit "$code"; }
 
usage_error() {
  error "$*"
  echo "Try '$SCRIPT_NAME -h' for more information." >&2
  exit 2
}
 
# ---- cleanup and error reporting --------------------------------------------
 
cleanup() {
  local rc=$?
  if [[ -n $TMP_FILE && -e $TMP_FILE ]]; then
    rm -f -- "$TMP_FILE"
    debug "removed temporary file $TMP_FILE"
  fi
  exit "$rc"
}
 
on_error() {
  local rc=$? line=$1
  error "line $line: '$BASH_COMMAND' exited with status $rc"
}
 
trap cleanup EXIT
trap 'on_error $LINENO' ERR
trap 'die 130 "interrupted"' INT
trap 'die 143 "terminated"' TERM
 
# ---- helpers ------------------------------------------------------------------
 
# run CMD [ARGS...]: execute, or just log it when -n was given
run() {
  if (( DRY_RUN )); then
    info "dry-run: $*"
  else
    debug "run: $*"
    "$@"
  fi
}
 
require_commands() {
  local cmd missing=()
  for cmd in "$@"; do
    command -v "$cmd" > /dev/null 2>&1 || missing+=("$cmd")
  done
  if (( ${#missing[@]} > 0 )); then
    die 4 "missing required commands: ${missing[*]}"
  fi
}
 
# Read KEY=value lines. Only known keys are accepted; nothing is executed.
load_config() {
  local file=$1 line key value n=0
  local re_skip='^[[:space:]]*(#.*)?$'
  local re_pair='^[[:space:]]*([A-Z_]+)[[:space:]]*=[[:space:]]*(.*)$'
 
  [[ -r $file ]] || die 3 "cannot read config file: $file"
  while IFS= read -r line || [[ -n $line ]]; do
    n=$((n + 1))
    [[ $line =~ $re_skip ]] && continue
    [[ $line =~ $re_pair ]] || die 3 "$file:$n: expected KEY=value: $line"
    key=${BASH_REMATCH[1]}
    value=${BASH_REMATCH[2]}
    value=${value%"${value##*[![:space:]]}"}   # trim trailing spaces
    value=${value#\"} value=${value%\"}         # strip optional quotes
    case $key in
      SOURCE_DIR | ARCHIVE_DIR | MIN_AGE | LOG_FILE | LOCK_FILE)
        printf -v "$key" '%s' "$value" ;;
      *)
        die 3 "$file:$n: unknown setting '$key'" ;;
    esac
  done < "$file"
  debug "loaded settings from $file"
}
 
acquire_lock() {
  exec {lock_fd}> "$LOCK_FILE"
  if ! flock -n "$lock_fd"; then
    die 5 "another run is in progress (lock: $LOCK_FILE)"
  fi
  debug "holding lock $LOCK_FILE"
}
 
# ---- argument handling ----------------------------------------------------------
 
parse_args() {
  local opt
  while getopts ':s:a:m:c:l:nvhV' opt; do
    case $opt in
      s) opt_source=$OPTARG ;;
      a) opt_archive=$OPTARG ;;
      m) opt_min_age=$OPTARG ;;
      c) CONFIG_FILE=$OPTARG ;;
      l) opt_log=$OPTARG ;;
      n) DRY_RUN=1 ;;
      v) VERBOSE=1 ;;
      h) usage; exit 0 ;;
      V) echo "$SCRIPT_NAME $VERSION"; exit 0 ;;
      :) usage_error "option -$OPTARG requires an argument" ;;
      \?) usage_error "unknown option: -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))
  if (( $# > 0 )); then
    usage_error "unexpected argument: $1"
  fi
}
 
apply_settings() {
  if [[ -n $CONFIG_FILE ]]; then load_config "$CONFIG_FILE"; fi
  if [[ -n $opt_source ]]; then SOURCE_DIR=$opt_source; fi
  if [[ -n $opt_archive ]]; then ARCHIVE_DIR=$opt_archive; fi
  if [[ -n $opt_min_age ]]; then MIN_AGE=$opt_min_age; fi
  if [[ -n $opt_log ]]; then LOG_FILE=$opt_log; fi
 
  [[ -n $SOURCE_DIR ]] || usage_error "missing required option -s SOURCE_DIR"
  [[ -n $ARCHIVE_DIR ]] || usage_error "missing required option -a ARCHIVE_DIR"
  [[ $MIN_AGE =~ ^[0-9]+$ ]] || usage_error "-m needs a whole number of minutes, got '$MIN_AGE'"
  [[ -d $SOURCE_DIR ]] || die 1 "source directory does not exist: $SOURCE_DIR"
}
 
# ---- the actual work ------------------------------------------------------------
 
# compress_into SRC DEST: write DEST atomically via a temp file in the same directory
compress_into() {
  local src=$1 dest=$2
  TMP_FILE=$(mktemp "${dest%/*}/.${dest##*/}.XXXXXX")
  gzip -c -- "$src" > "$TMP_FILE"
  chmod --reference="$src" -- "$TMP_FILE"    # mktemp creates files as 0600
  mv -f -- "$TMP_FILE" "$dest"
  TMP_FILE=""
}
 
archive_file() {
  local src=$1 name month_dir dest
  name=${src##*/}
  month_dir="$ARCHIVE_DIR/$(date -r "$src" +%Y/%m)"
  dest="$month_dir/$name.gz"
 
  if [[ -e $dest ]]; then
    if gzip -dc -- "$dest" | cmp -s - "$src"; then
      info "already archived, removing source: $name"
      run rm -f -- "$src"
      already=$((already + 1))
    else
      warn "conflict: ${dest#"$ARCHIVE_DIR"/} exists with different content, leaving $name"
      conflicts=$((conflicts + 1))
    fi
    return 0
  fi
 
  run mkdir -p -- "$month_dir"
  run compress_into "$src" "$dest"
  run rm -f -- "$src"
  if (( ! DRY_RUN )); then
    info "archived $name -> ${dest#"$ARCHIVE_DIR"/}"
  fi
  archived=$((archived + 1))
}
 
main() {
  parse_args "$@"
  apply_settings
  require_commands gzip find flock date mktemp cmp
  acquire_lock
 
  info "start: source=$SOURCE_DIR archive=$ARCHIVE_DIR min_age=${MIN_AGE}m dry_run=$DRY_RUN"
  local file
  while IFS= read -r -d '' file; do
    archive_file "$file"
  done < <(find "$SOURCE_DIR" -maxdepth 1 -type f -name '*.csv' -mmin +"$MIN_AGE" -print0 | sort -z)
 
  local note=""
  if (( DRY_RUN )); then note=" (dry run, nothing changed)"; fi
  info "done${note}: $archived archived, $already already archived, $conflicts conflicts"
  if (( conflicts > 0 )); then
    exit 1
  fi
}
 
main "$@"

Putting everything in functions and calling main "$@" on the last line has a practical benefit: Bash reads the whole file before running anything, so editing the script while a long run is in progress cannot corrupt the running copy. (Bash reads scripts incrementally; without the main wrapper, an edit mid-run can make the running process execute half of the old line and half of the new one.)

Where you'll use this

Deploying it as a scheduled job

On a server, the script goes to /usr/local/bin/archive-exports, its settings to /etc/archive-exports.conf, and cron or a systemd timer calls it with only -c:

# /etc/cron.d/archive-exports
MAILTO=""
*/15 * * * *  reports  /usr/local/bin/archive-exports -c /etc/archive-exports.conf 2>/dev/null

Stderr is discarded there because the config sets LOG_FILE, so every line also lands in the log file. Before trusting it, run it the way cron will, with an almost empty environment:

env -i PATH=/usr/bin:/bin HOME=$HOME ./archive-exports -c archive-exports.conf 2>/dev/null; echo "exit: $?"
cat archive-exports.log
exit: 0
2026-11-10T01:19:27-0500 archive-exports[5023] INFO: start: source=/home/sid/prod/exports archive=/home/sid/prod/archive min_age=60m dry_run=0
2026-11-10T01:19:27-0500 archive-exports[5023] INFO: archived stock-2026-11-05.csv -> 2026/11/stock-2026-11-05.csv.gz
2026-11-10T01:19:27-0500 archive-exports[5023] INFO: done: 1 archived, 0 already archived, 0 conflicts

With a systemd timer instead, drop LOG_FILE and let the journal collect stderr.

Testing it

A script with clear exit codes is easy to test with another script. This one builds a throwaway sandbox, runs each scenario and compares exit statuses:

#!/usr/bin/env bash
# test-archive-exports.sh - scenario tests for archive-exports
set -uo pipefail
 
script=$(realpath ./archive-exports)
pass=0 fail=0
 
check() {   # check DESCRIPTION EXPECTED_STATUS ACTUAL_STATUS
  if [[ $2 == "$3" ]]; then
    echo "PASS  $1"
    pass=$((pass + 1))
  else
    echo "FAIL  $1 (expected status $2, got $3)"
    fail=$((fail + 1))
  fi
}
 
sandbox=$(mktemp -d)
trap 'rm -rf "$sandbox"' EXIT
cd "$sandbox" || exit 1
export TMPDIR=$sandbox          # keeps the lock file inside the sandbox
mkdir exports archive
printf 'order_id,total\n70001,19.99\n' > exports/orders.csv
touch -d '2 hours ago' exports/orders.csv
month=$(date -d '2 hours ago' +%Y/%m)
 
"$script" > /dev/null 2>&1
check "no arguments is a usage error" 2 $?
"$script" -s exports -a archive -m soon > /dev/null 2>&1
check "non-numeric -m is a usage error" 2 $?
"$script" -n -s exports -a archive > /dev/null 2>&1
[[ -f exports/orders.csv && ! -e archive/$month ]]
check "dry run changes nothing" 0 $?
"$script" -s exports -a archive > /dev/null 2>&1
check "first run succeeds" 0 $?
cmp -s <(gzip -dc "archive/$month/orders.csv.gz") <(printf 'order_id,total\n70001,19.99\n')
check "archive has the original content" 0 $?
"$script" -s exports -a archive > /dev/null 2>&1
check "second run succeeds and does nothing" 0 $?
flock "$TMPDIR/archive-exports.lock" sleep 2 &
sleep 0.3
"$script" -s exports -a archive > /dev/null 2>&1
check "a concurrent run is refused" 5 $?
wait
 
echo "$pass passed, $fail failed"
(( fail == 0 ))
./test-archive-exports.sh; echo "exit: $?"
PASS  no arguments is a usage error
PASS  non-numeric -m is a usage error
PASS  dry run changes nothing
PASS  first run succeeds
PASS  archive has the original content
PASS  second run succeeds and does nothing
PASS  a concurrent run is refused
7 passed, 0 failed
exit: 0

The test script deliberately uses set -uo pipefail without -e, because it expects commands to fail and checks their statuses. Run it in CI next to shellcheck, and every change to the script is checked in seconds.

Common mistakes

set -e does nothing inside a function called by if

#!/usr/bin/env bash
set -e
backup() {
  cp /srv/db/nightly.dump /backups/
  echo "backup: copied, now rotating old dumps"
}
if backup; then
  echo "backup succeeded"
fi
bash set-e-if.sh
cp: cannot stat '/srv/db/nightly.dump': No such file or directory
backup: copied, now rotating old dumps
backup succeeded

set -e is suspended for everything that runs as part of an if condition, a while condition, or the left side of && / ||, including whole functions called there. The cp failed, the function kept going, and its last command succeeded, so the backup "succeeded". Inside functions that may be used as conditions, check critical commands explicitly: cp … || return 1.

local var=$(cmd) hides the failure

#!/usr/bin/env bash
set -e
get_version() {
  local version=$(cat /opt/shop/VERSION)
  echo "deploying version '$version'"
}
get_version
bash local-mask.sh
shellcheck local-mask.sh
cat: /opt/shop/VERSION: No such file or directory
deploying version ''
In local-mask.sh line 4:
  local version=$(cat /opt/shop/VERSION)
        ^-----^ SC2155 (warning): Declare and assign separately to avoid masking return values.

The exit status of that line is the status of local, which succeeded. Declare first, assign second (local version then version=$(cat …)), which is why the template's functions always do it in two steps.

getopts without the leading colon

#!/usr/bin/env bash
set -u
while getopts 'nv' opt; do
  case $opt in
    n) echo "dry run" ;;
    v) echo "verbose" ;;
    ?) echo "unknown option: -$OPTARG" >&2; exit 2 ;;
  esac
done
bash getopts-nocolon.sh -x; echo "exit: $?"
getopts-nocolon.sh: illegal option -- x
getopts-nocolon.sh: line 7: OPTARG: unbound variable
exit: 1

Without the leading :, getopts prints its own message and leaves OPTARG unset, which set -u then turns into a second, confusing error and the wrong exit code. With ':nv', getopts stays quiet and puts the bad letter in OPTARG.

Summary

  • Start every script with set -Eeuo pipefail, a header that documents usage and exit codes, and settings with defaults in one place; let a config file override the defaults and command-line options override both.
  • Parse options with getopts ':…' (leading colon), validate everything before doing any work, and exit 2 on usage errors.
  • Log to stderr with timestamp, script name, PID and level; optionally tee to a file.
  • Route every state-changing command through a run function so -n can show the plan without executing it.
  • Never source a config file you do not fully control; parse KEY=value with an allow-list and printf -v.
  • Check dependencies with command -v up front; take a flock lock on a dedicated file descriptor; write outputs to a temp file in the destination directory and mv them into place; clean up in an EXIT trap and report failures from an ERR trap.
  • Make reruns safe: check for existing results, compare content, refuse to overwrite on conflict. Use distinct exit codes for "failed", "misused" and "already running".
  • Keep it shellcheck-clean and test it with a script that asserts exit codes.

Exercises

Exercise 1: A quiet mode for croneasy

Add a -q option: INFO and DEBUG lines are no longer printed to stderr, but WARN and ERROR still are, and the log file (if -l is given) still receives every line. Update usage(). Make sure shellcheck stays clean.

Show Solution

The changes, as a diff against the template:

+QUIET=0
[…]
+  -q        quiet: print only warnings and errors (the log file gets everything)
[…]
-  printf '%s\n' "$line" >&2
+  if (( ! QUIET )) || [[ $level == WARN || $level == ERROR ]]; then
+    printf '%s\n' "$line" >&2
+  fi
[…]
-  while getopts ':s:a:m:c:l:nvhV' opt; do
+  while getopts ':s:a:m:c:l:nqvhV' opt; do
[…]
+      q) QUIET=1 ;;

With one new file to archive and one conflict:

./archive-exports-q -q -s exports -a archive -l quiet.log; echo "exit: $?"
cat quiet.log
2026-11-10T01:18:46-0500 archive-exports-q[956] WARN: conflict: 2026/10/customers-2026-10-03.csv.gz exists with different content, leaving customers-2026-10-03.csv
exit: 1
2026-11-10T01:18:46-0500 archive-exports-q[956] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:18:46-0500 archive-exports-q[956] WARN: conflict: 2026/10/customers-2026-10-03.csv.gz exists with different content, leaving customers-2026-10-03.csv
2026-11-10T01:18:46-0500 archive-exports-q[956] INFO: archived orders-2026-10-02.csv -> 2026/10/orders-2026-10-02.csv.gz
2026-11-10T01:18:46-0500 archive-exports-q[956] INFO: done: 1 archived, 0 already archived, 1 conflicts

The terminal (or cron's mail) shows only the line that needs attention; the log keeps the full story. When I first added only the getopts and case lines, shellcheck caught the missing piece: SC2034 (warning): QUIET appears unused.

Exercise 2: Copy-pasteable dry-run outputmedium

In the dry-run output, compress_into exports/Q3 revenue report.csv … is ambiguous: you cannot tell where one argument ends. Change run so each argument is shell-quoted, and the logged command could be pasted into a terminal as is.

Hint

printf '%q' prints a string quoted for reuse as shell input.

Show Solution
run() {
  if (( DRY_RUN )); then
    info "dry-run:$(printf ' %q' "$@")"
  else
    debug "run: $*"
    "$@"
  fi
}
./ex2 -n -s exports -a archive 2>&1 | grep dry-run
2026-11-09T22:04:55-0500 ex2[79637] INFO: dry-run: mkdir -p -- archive/2026/11
2026-11-09T22:04:55-0500 ex2[79637] INFO: dry-run: compress_into exports/Q3\ revenue\ report.csv archive/2026/11/Q3\ revenue\ report.csv.gz
2026-11-09T22:04:55-0500 ex2[79637] INFO: dry-run: rm -f -- exports/Q3\ revenue\ report.csv

printf ' %q' "$@" applies the format once per argument, so each gets a leading space and its own quoting. Spaces became \ , and the argument boundaries are unambiguous.

Exercise 3: Retention for the archivehard

Add -k MONTHS: after archiving, delete ARCHIVE_DIR/YYYY/MM folders older than the current month minus MONTHS (so -k 3 in November keeps August to November). It must honor -n, touch nothing that is not a YYYY/MM folder, and reject -k 0 or non-numbers as a usage error.

Show Solution

New pieces (plus KEEP_MONTHS="" with the other settings and k: in the getopts string with k) KEEP_MONTHS=$OPTARG ;;):

# in apply_settings
  [[ -z $KEEP_MONTHS || $KEEP_MONTHS =~ ^[1-9][0-9]*$ ]] ||
    usage_error "-k needs a positive number of months, got '$KEEP_MONTHS'"
 
# prune_archive N: delete ARCHIVE_DIR/YYYY/MM folders older than the current
# month minus N months (N=3 in November keeps Aug, Sep, Oct and Nov)
prune_archive() {
  local keep=$1 cutoff dir ym
  cutoff=$(date -d "$(date +%Y-%m-01) -$keep months" +%Y%m)
  while IFS= read -r -d '' dir; do
    ym=${dir#"$ARCHIVE_DIR"/}
    ym=${ym//\//}                      # 2026/07 -> 202607
    if (( ym < cutoff )); then
      info "pruning ${dir#"$ARCHIVE_DIR"/} (older than $keep months)"
      run rm -rf -- "$dir"
    fi
  done < <(find "$ARCHIVE_DIR" -mindepth 2 -maxdepth 2 -type d \
             -regextype posix-extended -regex '.*/[0-9]{4}/[0-9]{2}' -print0 | sort -z)
}
 
# in main, after the archiving loop
  if [[ -n $KEEP_MONTHS ]]; then
    prune_archive "$KEEP_MONTHS"
  fi

Test archive with months from December 2025 to November 2026 and an unrelated tmp-restore folder:

./archive-exports-k -n -k 3 -s exports -a archive; echo "exit: $?"
./archive-exports-k -k 3 -s exports -a archive; echo "exit: $?"
ls archive archive/2026
./archive-exports-k -k 0 -s exports -a archive; echo "exit: $?"
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: start: source=exports archive=archive min_age=10m dry_run=1
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: pruning 2025/12 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: dry-run: rm -rf -- archive/2025/12
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: pruning 2026/05 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: dry-run: rm -rf -- archive/2026/05
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: pruning 2026/06 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: dry-run: rm -rf -- archive/2026/06
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: pruning 2026/07 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: dry-run: rm -rf -- archive/2026/07
2026-11-10T01:18:46-0500 archive-exports-k[1014] INFO: done (dry run, nothing changed): 0 archived, 0 already archived, 0 conflicts
exit: 0
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: start: source=exports archive=archive min_age=10m dry_run=0
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: pruning 2025/12 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: pruning 2026/05 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: pruning 2026/06 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: pruning 2026/07 (older than 3 months)
2026-11-10T01:18:46-0500 archive-exports-k[1046] INFO: done: 0 archived, 0 already archived, 0 conflicts
exit: 0
archive:
2025
2026
tmp-restore
 
archive/2026:
08
09
10
11
2026-11-10T01:18:46-0500 archive-exports-k[1075] ERROR: -k needs a positive number of months, got '0'
Try 'archive-exports-k -h' for more information.
exit: 2

Comparing YYYYMM as integers avoids date arithmetic on folder names. Computing the cutoff from the first of the month matters: date -d "2026-03-31 -1 month" lands in March again (there is no 31 February), while the first of a month always exists. The -regex filter is the safety rail: tmp-restore was never considered. The now-empty 2025 folder stays; find "$ARCHIVE_DIR" -mindepth 1 -type d -empty -delete would tidy it if you want that.


Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), ShellCheck 0.9.0, util-linux 2.39.3, November 2026. Every listing was run as shown in a scratch directory with generated CSV exports; the 1 GB file used for the interruption test was a sparse file, so it compressed in seconds without using real disk space. The /etc/cron.d entry was not installed (it needs root); the same command was run under env -i to reproduce cron's environment instead. Two details came straight out of testing: the 0600 permissions that mktemp gave the first archives, and shellcheck's SC1087 on the log line. Paths from the test machine's scratch directories are shown shortened to /home/sid/….

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.