Real-World Scripts: Backups, Log Reports, and Cleanup

Three complete, tested Bash tools built on a shared library: rotating verified tar backups, a daily nginx traffic report, and a disk cleanup with safety rails.

By The Dark DebuggerUpdated 38 min read

This article builds three tools that almost every Linux server ends up needing: a backup script with rotation and verification, a daily report from web server access logs, and a cleanup job for directories that fill up with old files. Each one follows the template from writing production-ready scripts, so read that first. For each tool you get the requirements, the complete code, a test run with real output, and how to schedule it (cron and systemd timers).

Why these three

They cover the three things that most often go wrong on a server nobody is watching: data that cannot be restored, problems nobody notices until customers complain, and disks that fill up. They also share a property that makes them good practice: each one is simple to write badly and needs real care to write well. A backup that is never verified, a report that silently counts the wrong day, or a cleanup that follows a symlink into /etc are all one missing line away.

A shared library

The template from the previous article is one self-contained file, which is right for a single tool. With three tools, copying the same 80 lines of logging, locking and cleanup into each one means fixing every bug three times. So the helpers move into lib/common.sh, and each script sources it:

# lib/common.sh - shared helpers for the ops scripts. Source it; do not run it.
#
# The caller sets SCRIPT_NAME and may set LOG_FILE, DRY_RUN and VERBOSE.
# Provides logging, a dry-run aware run(), dependency checks, flock locking,
# and cleanup of registered temporary paths on any exit.
# Exit codes used here: 2 usage, 4 missing dependency, 5 already running,
# 130/143 interrupted/terminated.
 
LOG_FILE=${LOG_FILE:-}
DRY_RUN=${DRY_RUN:-0}
VERBOSE=${VERBOSE:-0}
TMP_PATHS=()
 
log() {
  local level=$1 line
  shift
  line="$(date '+%Y-%m-%dT%H:%M:%S%z') ${SCRIPT_NAME}[$$] $level: $*"
  printf '%s\n' "$line" >&2
  if [[ -n $LOG_FILE ]]; then
    printf '%s\n' "$line" >> "$LOG_FILE"
  fi
}
debug() { if (( VERBOSE )); then log DEBUG "$@"; fi; }
info()  { log INFO "$@"; }
warn()  { log WARN "$@"; }
error() { log ERROR "$@"; }
die()   { local code=$1; shift; error "$@"; exit "$code"; }
 
usage_error() {
  error "$*"
  echo "Try '$SCRIPT_NAME -h' for more information." >&2
  exit 2
}
 
# run CMD [ARGS...]: execute, or only log the command (shell-quoted) in dry-run mode
run() {
  if (( DRY_RUN )); then
    info "dry-run:$(printf ' %q' "$@")"
  else
    debug "run:$(printf ' %q' "$@")"
    "$@"
  fi
}
 
require_commands() {
  local cmd missing=()
  for cmd in "$@"; do
    command -v "$cmd" > /dev/null 2>&1 || missing+=("$cmd")
  done
  if (( ${#missing[@]} > 0 )); then
    die 4 "missing required commands: ${missing[*]}"
  fi
}
 
# acquire_lock FILE: exit 5 if another process holds the lock
acquire_lock() {
  exec {LOCK_FD}> "$1"
  if ! flock -n "$LOCK_FD"; then
    die 5 "another run is in progress (lock: $1)"
  fi
}
 
# register_tmp PATH: delete PATH (file or directory) when the script exits
register_tmp() { TMP_PATHS+=("$1"); }
 
_cleanup() {
  local rc=$? path
  for path in "${TMP_PATHS[@]}"; do
    if [[ -e $path ]]; then
      rm -rf -- "$path"
      debug "removed temporary path $path"
    fi
  done
  exit "$rc"
}
 
_on_error() {
  local rc=$? line=$1
  # first line only: a multi-line awk program would flood the log
  error "line $line: '${BASH_COMMAND%%$'\n'*}' exited with status $rc"
  exit 1    # keep the documented exit codes: tar's 2 is not a usage error
}
 
trap _cleanup EXIT
trap '_on_error $LINENO' ERR
trap 'die 130 "interrupted"' INT
trap 'die 143 "terminated"' TERM

Three differences from the template, each added because testing showed it was needed:

  • run logs commands with printf ' %q', so paths with spaces are unambiguous (this was exercise 2 of the previous article).
  • register_tmp lets a script register any number of temporary files or directories, and _cleanup removes them all.
  • _on_error prints only the first line of the failed command, and exits with 1. Both came out of the testing below: a failing gawk program dumped 60 lines into the log, and a failing tar made the backup script exit with tar's own status 2, which collided with "usage error".

Each script starts the same way. realpath "$0" finds the library next to the script even when it is called through a symlink or from another directory, and the # shellcheck source= comment tells shellcheck where to look when run with -x:

set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"

All three scripts below pass shellcheck -x with no warnings:

shellcheck -x backup.sh log-report.sh cleanup.sh && echo clean
clean

Tool 1: rotating, verified backups

Requirements

  • Back up one or more directories into a single NAME-YYYYmmdd-HHMMSS.tar.gz.
  • Exclude junk that should not be backed up (logs, caches) by pattern.
  • Never leave a half-written archive that looks like a real backup.
  • Verify the archive before it counts, and store a SHA-256 checksum next to it.
  • Keep the newest N backups and delete older ones, but only after the new one is good.
  • Refuse to start if the destination does not have room.
  • Locking, dry run, logging and clear exit codes from the template.

The code

#!/usr/bin/env bash
#
# backup.sh - rotating, verified tar.gz backups of one or more directories
#
# Usage: backup.sh [-n] [-v] [-k KEEP] [-x PATTERN]... [-l LOG] -b NAME -d DEST DIR...
#
# Creates DEST/NAME-YYYYmmdd-HHMMSS.tar.gz plus a .sha256 file, verifies it,
# then deletes the oldest backups so only KEEP remain.
# Exit codes: 0 ok, 1 failure, 2 usage error, 4 missing dependency, 5 already running
 
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
 
NAME="" DEST="" KEEP=7
EXCLUDES=()
 
usage() {
  cat <<EOF
Usage: $SCRIPT_NAME [options] -b NAME -d DEST DIR...
 
Back up DIR... into DEST/NAME-YYYYmmdd-HHMMSS.tar.gz, verify it, keep the newest KEEP.
 
Options:
  -b NAME     backup name, used as the file prefix (required)
  -d DEST     directory to write backups to (required)
  -k KEEP     number of backups to keep (default: $KEEP)
  -x PATTERN  exclude files matching PATTERN (tar --exclude syntax, repeatable)
  -l FILE     also append log lines to FILE
  -n          dry run
  -v          verbose
  -h          show this help
EOF
}
 
parse_args() {
  local opt
  while getopts ':b:d:k:x:l:nvh' opt; do
    case $opt in
      b) NAME=$OPTARG ;;
      d) DEST=$OPTARG ;;
      k) KEEP=$OPTARG ;;
      x) EXCLUDES+=("--exclude=$OPTARG") ;;
      l) LOG_FILE=$OPTARG ;;
      n) DRY_RUN=1 ;;
      v) VERBOSE=1 ;;
      h) usage; exit 0 ;;
      :) usage_error "option -$OPTARG requires an argument" ;;
      \?) usage_error "unknown option: -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))
  SOURCES=("$@")
 
  [[ -n $NAME ]] || usage_error "missing required option -b NAME"
  [[ $NAME =~ ^[A-Za-z0-9._-]+$ ]] || usage_error "NAME may only contain letters, digits, . _ -"
  [[ -n $DEST ]] || usage_error "missing required option -d DEST"
  [[ $KEEP =~ ^[1-9][0-9]*$ ]] || usage_error "-k needs a number >= 1, got '$KEEP'"
  (( ${#SOURCES[@]} > 0 )) || usage_error "no directories to back up"
  local src
  for src in "${SOURCES[@]}"; do
    [[ -d $src ]] || die 1 "not a directory: $src"
  done
}
 
# check_space: refuse to start if DEST has less free space than the sources use
check_space() {
  local need avail
  need=$(du -sk -- "${SOURCES[@]}" | awk '{ s += $1 } END { print s }')
  avail=$(df -Pk -- "$DEST" | awk 'NR == 2 { print $4 }')
  debug "sources use ${need} KiB, ${avail} KiB free in $DEST"
  if (( avail < need )); then
    die 1 "not enough space in $DEST: need ${need} KiB, have ${avail} KiB"
  fi
}
 
create_archive() {
  local archive=$1 partial=$1.partial src members=()
  for src in "${SOURCES[@]}"; do
    src=$(realpath -- "$src")
    members+=(-C "$(dirname -- "$src")" "$(basename -- "$src")")
  done
  register_tmp "$partial"
  tar -czf "$partial" "${EXCLUDES[@]}" "${members[@]}"
 
  # verify before the backup is allowed to count
  gzip -t -- "$partial"
  local entries
  entries=$(tar -tzf "$partial" | wc -l)
  (( entries > 0 )) || die 1 "archive is empty: $partial"
  mv -- "$partial" "$archive"
  (cd -- "$DEST" && sha256sum -- "${archive##*/}" > "${archive##*/}.sha256")
  info "created ${archive##*/}: $entries entries, $(du -h -- "$archive" | cut -f1)"
}
 
# rotate: keep the newest KEEP backups of NAME, delete the rest
rotate() {
  local backups=() old
  shopt -s nullglob
  backups=("$DEST/$NAME"-[0-9]*-[0-9]*.tar.gz)   # glob results are sorted, oldest first
  shopt -u nullglob
  if (( ${#backups[@]} <= KEEP )); then
    debug "${#backups[@]} backups present, keeping all"
    return 0
  fi
  for old in "${backups[@]:0:${#backups[@]}-KEEP}"; do
    info "rotating out ${old##*/}"
    run rm -f -- "$old" "$old.sha256"
  done
}
 
main() {
  parse_args "$@"
  require_commands tar gzip sha256sum flock du df realpath
  run mkdir -p -- "$DEST"
  acquire_lock "${TMPDIR:-/tmp}/backup-$NAME.lock"
  if (( ! DRY_RUN )); then check_space; fi
 
  local started archive
  started=$(date +%s)
  archive="$DEST/$NAME-$(date +%Y%m%d-%H%M%S).tar.gz"
  [[ ! -e $archive ]] || die 1 "$archive already exists"
 
  info "backing up ${SOURCES[*]} to ${archive##*/} (keep $KEEP)"
  run create_archive "$archive"
  rotate
  info "done in $(( $(date +%s) - started ))s"
}
 
main "$@"

The decisions that matter:

  • -C dir basename for each source. tar stores paths relative to each -C directory, so the archive contains shop/… and etc-nginx/… rather than the full absolute path (see the common mistakes).
  • Write .partial, verify, then rename. The rotation glob only matches *.tar.gz, so a crashed run can never be mistaken for a backup or push a good one out of the rotation. The partial file is registered for cleanup.
  • Rotation runs after the new backup is verified. If today's backup fails, the old ones are all still there.
  • The timestamp in the name sorts chronologically, so a plain glob lists backups oldest first and "${backups[@]:0:count-KEEP}" is exactly the ones to delete.
  • The checksum file uses a bare file name (that is why sha256sum runs inside DEST), so sha256sum -c still works after the backups are copied elsewhere.

Test run

A small shop application: code, config, uploads, plus logs and a session cache that should not be backed up.

du -sh srv/shop/*
12K	srv/shop/app
108K	srv/shop/cache
8.0K	srv/shop/config
2.3M	srv/shop/logs
252K	srv/shop/uploads

A dry run first, then four real runs a second apart with -k 3:

./backup.sh -n -b shop -d backups -k 3 -x '*.log' -x 'shop/cache' srv/shop etc-nginx
./backup.sh -b shop -d backups -k 3 -x '*.log' -x 'shop/cache' srv/shop etc-nginx
2026-11-10T01:30:33-0500 backup.sh[9585] INFO: dry-run: mkdir -p -- backups
2026-11-10T01:30:33-0500 backup.sh[9585] INFO: backing up srv/shop etc-nginx to shop-20261110-013033.tar.gz (keep 3)
2026-11-10T01:30:33-0500 backup.sh[9585] INFO: dry-run: create_archive backups/shop-20261110-013033.tar.gz
2026-11-10T01:30:33-0500 backup.sh[9585] INFO: done in 0s
2026-11-10T01:30:33-0500 backup.sh[9605] INFO: backing up srv/shop etc-nginx to shop-20261110-013033.tar.gz (keep 3)
2026-11-10T01:30:33-0500 backup.sh[9605] INFO: created shop-20261110-013033.tar.gz: 18 entries, 4.0K
2026-11-10T01:30:33-0500 backup.sh[9605] INFO: done in 0s

The fourth run is the first one that has something to rotate:

2026-11-10T01:30:37-0500 backup.sh[9802] INFO: backing up srv/shop etc-nginx to shop-20261110-013037.tar.gz (keep 3)
2026-11-10T01:30:37-0500 backup.sh[9802] INFO: created shop-20261110-013037.tar.gz: 18 entries, 4.0K
2026-11-10T01:30:37-0500 backup.sh[9802] INFO: rotating out shop-20261110-013033.tar.gz
2026-11-10T01:30:37-0500 backup.sh[9802] INFO: done in 0s
ls -l backups
(cd backups && sha256sum -c shop-20261110-013037.tar.gz.sha256)
tar -tzf backups/shop-20261110-013037.tar.gz
total 24
-rw-r--r-- 1 sid sid 1277 Nov 10 01:30 shop-20261110-013034.tar.gz
-rw-r--r-- 1 sid sid   94 Nov 10 01:30 shop-20261110-013034.tar.gz.sha256
-rw-r--r-- 1 sid sid 1277 Nov 10 01:30 shop-20261110-013036.tar.gz
-rw-r--r-- 1 sid sid   94 Nov 10 01:30 shop-20261110-013036.tar.gz.sha256
-rw-r--r-- 1 sid sid 1277 Nov 10 01:30 shop-20261110-013037.tar.gz
-rw-r--r-- 1 sid sid   94 Nov 10 01:30 shop-20261110-013037.tar.gz.sha256
shop-20261110-013037.tar.gz: OK
shop/
shop/config/
shop/config/app.env
shop/logs/
shop/uploads/
shop/uploads/2026/
shop/uploads/2026/11/
shop/uploads/2026/11/invoice-104.pdf
shop/uploads/2026/11/invoice-105.pdf
shop/uploads/2026/11/invoice-103.pdf
shop/uploads/2026/11/invoice-101.pdf
shop/uploads/2026/11/invoice-102.pdf
shop/uploads/2026/11/invoice-106.pdf
shop/app/
shop/app/server.py
shop/app/requirements.txt
etc-nginx/
etc-nginx/shop.conf

The logs/ directory is there but empty and cache/ is gone, as the exclusions asked. (The archives are tiny because the test uploads are zero-filled.) A backup you have never restored is a hope, not a backup, so restore one and compare:

mkdir -p restore-test && tar -xzf backups/shop-20261110-013037.tar.gz -C restore-test && diff -r --exclude=cache --exclude='*.log' srv/shop restore-test/shop && diff -r etc-nginx restore-test/etc-nginx && echo 'restore matches source'
restore matches source

Now the failure path. A config file that the backup user cannot read is a classic: in a naive script, tar prints a warning, carries on, and you have a "successful" backup without the one file you needed.

./backup.sh -b shop -d backups -k 3 -x '*.log' -x 'shop/cache' srv/shop etc-nginx; echo "exit: $?"
ls -A backups | grep -c partial
2026-11-10T01:30:38-0500 backup.sh[9887] INFO: backing up srv/shop etc-nginx to shop-20261110-013038.tar.gz (keep 3)
tar: shop/config/app.env: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors
2026-11-10T01:30:38-0500 backup.sh[9887] ERROR: line 85: 'tar -czf "$partial" "${EXCLUDES[@]}" "${members[@]}"' exited with status 2
exit: 1
0

The run failed loudly, the partial archive was removed, and rotation never ran, so the three good backups are untouched. And two runs at once:

flock /tmp/backup-shop.lock sleep 2 & sleep 0.3; ./backup.sh -b shop -d backups srv/shop; echo "exit: $?"; wait
2026-11-10T01:30:38-0500 backup.sh[9931] ERROR: another run is in progress (lock: /tmp/backup-shop.lock)
exit: 5

Scheduling it

Nightly at 02:30, keeping two weeks:

30 2 * * *  /opt/ops/backup.sh -b shop -d /backups/shop -k 14 -x '*.log' -x 'shop/cache' -l /var/log/ops/backup.log /srv/shop /etc/nginx 2>/dev/null

or as a systemd service and timer (OnCalendar=*-*-* 02:30:00, Persistent=true, so a backup missed while the server was down runs at boot). Two more things a real setup needs: a copy of the backups on another machine (rsync or object storage; a backup on the same disk dies with the disk), and the exercise below that verifies every backup's checksum on a schedule.

Tool 2: a daily access-log report

Requirements

  • Read nginx access logs in the standard combined format with $request_time (seconds) appended, plain or gzipped (rotated logs are usually .gz).
  • Report one day: totals, status classes and the 5xx error rate, top client IPs, endpoints producing 5xx errors, the slowest endpoints by average and 95th percentile, and requests per hour.
  • Group /products/1234 and /products/1301 as /products/:id, and ignore query strings, or every URL is unique and the "slowest endpoint" list is useless.
  • Write to stdout, or atomically to a file for publishing.

Sample data

To test against something realistic, I generated a day of traffic for a small shop with gawk: about 6,400 requests across 15 client IPs, a crawler, a vulnerability scanner probing /.env and /wp-login.php, a slow search API that sometimes times out with 502/504, an occasional 500 on checkout, and two malformed requests. The generator:

# gen-access-log.awk - write a realistic nginx access log for one day (plus a
# little of the day before), in "combined" format with $request_time appended.
# Usage: gawk -v day=08 -f gen-access-log.awk > access.log
function pick(list,   a, n) { n = split(list, a, " "); return a[int(rand() * n) + 1] }
function between(lo, hi) { return lo + rand() * (hi - lo) }
function line(d, h, m, s, ip, method, path, status, bytes, rt,   ua) {
  ua = (ip ~ /^66\.249\./) ? "Googlebot/2.1 (+http://www.google.com/bot.html)" \
     : (ip ~ /^185\.220\./) ? "python-requests/2.31.0" \
     : "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36"
  printf "%s - - [%02d/Nov/2026:%02d:%02d:%02d -0500] \"%s %s HTTP/1.1\" %d %d \"-\" \"%s\" %.3f\n",
         ip, d, h, m, s, method, path, status, bytes, ua, rt
}
BEGIN {
  srand(42)
  ips = "203.0.113.42 203.0.113.42 203.0.113.42 198.51.100.7 198.51.100.7 192.0.2.15 " \
        "192.0.2.77 192.0.2.80 198.51.100.23 203.0.113.9 203.0.113.118 192.0.2.201 " \
        "66.249.66.1 66.249.66.1 198.51.100.64 203.0.113.250 192.0.2.33 198.51.100.140"
  hours = "0 1 2 3 4 5 6 7 7 8 8 9 9 9 10 10 10 11 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19 19 19 20 20 20 21 21 22 22 23"
  for (i = 0; i < 6400; i++) {
    d = (i < 250) ? day - 1 : day
    h = pick(hours); m = int(rand() * 60); s = int(rand() * 60)
    ip = pick(ips); r = rand(); status = 200; method = "GET"
    if (r < 0.12)      { path = "/"; rt = between(0.010, 0.060); bytes = 18200 }
    else if (r < 0.25) { path = "/products"; rt = between(0.040, 0.150); bytes = 42100 }
    else if (r < 0.47) { path = "/products/" int(between(1000, 1400)); rt = between(0.030, 0.180); bytes = 23800 }
    else if (r < 0.57) { path = "/api/cart"; method = "POST"; rt = between(0.020, 0.110); bytes = 610 }
    else if (r < 0.69) { path = "/api/search?q=" pick("boots jacket socks gloves scarf"); bytes = 9100
                         rt = (rand() < 0.15) ? between(1.5, 4.8) : between(0.200, 0.900)
                         if (rand() < 0.07) { status = pick("502 504"); rt = between(5.0, 6.0); bytes = 166 } }
    else if (r < 0.75) { path = "/api/orders/" int(between(70000, 79999)); rt = between(0.080, 0.350); bytes = 1400 }
    else if (r < 0.79) { path = "/checkout"; method = "POST"; rt = between(0.250, 1.200); bytes = 2300
                         if (rand() < 0.04) { status = 500; bytes = 312 } }
    else if (r < 0.93) { path = "/static/app.js"; rt = between(0.001, 0.004); bytes = 184000
                         if (rand() < 0.4) { status = 304; bytes = 0 } }
    else if (r < 0.96) { path = "/login"; rt = between(0.050, 0.200); bytes = 5200 }
    else               { ip = "185.220.101.4"; path = pick("/wp-login.php /.env /xmlrpc.php /admin.php")
                         status = 404; rt = between(0.001, 0.003); bytes = 153 }
    line(d, h, m, s, ip, method, path, status, bytes, rt)
  }
  # two malformed requests, as every real server receives
  printf "185.220.101.4 - - [%02d/Nov/2026:03:12:44 -0500] \"\\x16\\x03\\x01\\x02\\x00\" 400 157 \"-\" \"-\" 0.000\n", day
  printf "185.220.101.4 - - [%02d/Nov/2026:03:12:45 -0500] \"-\" 400 0 \"-\" \"-\" 0.000\n", day
}
gawk -v day=08 -f gen-access-log.awk | sort -s -k4,4 > logs/access.log
wc -l logs/access.log
sed -n '1p;3000p;$p' logs/access.log
6402 logs/access.log
203.0.113.42 - - [07/Nov/2026:00:05:37 -0500] "GET /api/search?q=boots HTTP/1.1" 200 9100 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36" 0.561
198.51.100.23 - - [08/Nov/2026:12:01:31 -0500] "GET /static/app.js HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36" 0.001
192.0.2.77 - - [08/Nov/2026:23:59:54 -0500] "GET /products/1042 HTTP/1.1" 200 23800 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36" 0.082

Like a real log file just after midnight rotation, it starts with the tail end of the previous day, which the report must not count.

The code

The heavy lifting is one gawk program (see awk for the language). With space as the field separator, the IP is $1, the timestamp $4, the method $6 (with its opening quote), the path $7, status $9, bytes $10, and the request time is the last field, $NF, because the user agent in between contains spaces.

#!/usr/bin/env bash
#
# log-report.sh - daily traffic report from nginx access logs
#
# Usage: log-report.sh [-v] [-d YYYY-MM-DD] [-t TOP] [-o FILE] LOGFILE...
#
# Expects the "combined" log format with $request_time as the last field.
# Reads plain or gzipped logs. Writes the report to stdout, or atomically to FILE.
# Exit codes: 0 ok, 1 failure (e.g. no lines for that day), 2 usage error,
#             4 missing dependency
 
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
 
DAY=$(date -d yesterday +%F)
TOP=10
OUTPUT=""
 
usage() {
  cat <<EOF
Usage: $SCRIPT_NAME [options] LOGFILE...
 
Options:
  -d DATE   day to report on, YYYY-MM-DD (default: yesterday, $DAY)
  -t N      length of the top-N lists (default: $TOP)
  -o FILE   write the report to FILE instead of stdout
  -v        verbose
  -h        show this help
EOF
}
 
parse_args() {
  local opt
  while getopts ':d:t:o:vh' opt; do
    case $opt in
      d) DAY=$OPTARG ;;
      t) TOP=$OPTARG ;;
      o) OUTPUT=$OPTARG ;;
      v) VERBOSE=1 ;;
      h) usage; exit 0 ;;
      :) usage_error "option -$OPTARG requires an argument" ;;
      \?) usage_error "unknown option: -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))
  LOGS=("$@")
  (( ${#LOGS[@]} > 0 )) || usage_error "no log files given"
  [[ $TOP =~ ^[1-9][0-9]*$ ]] || usage_error "-t needs a positive number, got '$TOP'"
  date -d "$DAY" +%F > /dev/null 2>&1 || usage_error "not a valid date: $DAY"
  local f
  for f in "${LOGS[@]}"; do
    [[ -r $f ]] || die 1 "cannot read $f"
  done
}
 
# report LOGDAY LABEL TOP: read log lines on stdin, print the report on stdout
report() {
  gawk -v day="$1" -v label="$2" -v top="$3" '
    # nginx writes [08/Nov/2026:13:55:36 -0500]; keep only the wanted day
    index($4, "[" day ":") != 1 { next }
    $6 !~ /^"(GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS)$/ { malformed++; next }
    {
      total++
      ip = $1; path = $7; status = $9; bytes = $10; rt = $NF
      sub(/\?.*/, "", path)                   # drop query strings
      gsub(/\/[0-9]+(\/|$)/, "/:id/", path)   # /products/1234 -> /products/:id
      sub(/\/$/, "", path); if (path == "") path = "/"
      hits[ip]++
      class[substr(status, 1, 1) "xx"]++
      if (status >= 500) { errors[path]++; err5++ }
      sent += bytes
      n[path]++; sum[path] += rt; times[path, n[path]] = rt
      hour[substr($4, 14, 2)]++
    }
    function topn(arr, label, k,   i, cmd) {
      printf "\n%s\n", label
      cmd = "sort -k2,2nr -k1,1 | head -n " k
      for (i in arr) printf "  %-22s %6d\n", i, arr[i] | cmd
      close(cmd)
    }
    END {
      if (total == 0) exit 3
      printf "Traffic report for %s\n", label
      printf "requests: %d   unique IPs: %d   sent: %.1f MiB   malformed: %d\n",
             total, length(hits), sent / 1048576, malformed
 
      printf "\nstatus classes\n"
      for (c in class) printf "  %s %6d  %5.1f%%\n", c, class[c], 100 * class[c] / total | "sort"
      close("sort")
      printf "  5xx error rate: %.2f%%\n", 100 * err5 / total
 
      topn(hits, "top client IPs", top)
      if (err5) topn(errors, "endpoints returning 5xx", top)
 
      printf "\nslowest endpoints (min. 20 requests)\n"
      printf "  %-22s %6s %8s %8s %8s\n", "endpoint", "reqs", "avg s", "p95 s", "max s"
      cmd = "sort -k4,4nr | head -n " top
      for (p in n) {
        if (n[p] < 20) continue
        delete v
        for (i = 1; i <= n[p]; i++) v[i] = times[p, i]
        asort(v)
        printf "  %-22s %6d %8.3f %8.3f %8.3f\n", p, n[p], sum[p] / n[p], v[int(0.95 * n[p] + 0.999)], v[n[p]] | cmd
      }
      close(cmd)
 
      printf "\nrequests per hour\n"
      for (h = 0; h < 24; h++) {
        hh = sprintf("%02d", h)
        printf "  %s %5d %s\n", hh, hour[hh], substr("##################################################", 1, int(hour[hh] / 10))
      }
    }'
}
 
main() {
  parse_args "$@"
  require_commands gawk zcat sort date
  local logday
  logday=$(LC_ALL=C date -d "$DAY" +%d/%b/%Y)
  debug "reading ${LOGS[*]} for $logday"
 
  local tmp="" rc=0
  if [[ -n $OUTPUT ]]; then
    tmp=$(mktemp "$OUTPUT.XXXXXX")
    register_tmp "$tmp"
    exec 3> "$tmp"
  else
    exec 3>&1
  fi
  zcat -f -- "${LOGS[@]}" | report "$logday" "$DAY" "$TOP" >&3 || rc=$?
  exec 3>&-
  case $rc in
    0 | 141) ;;   # 141 = SIGPIPE: the reader stopped early (| head), not an error
    3) die 1 "no requests for $DAY in ${LOGS[*]}" ;;
    *) die 1 "report failed with status $rc" ;;
  esac
 
  if [[ -n $OUTPUT ]]; then
    chmod 644 "$tmp"
    mv -f -- "$tmp" "$OUTPUT"
    info "report written to $OUTPUT"
  fi
}
 
main "$@"

Notes on the parts that are easy to get wrong:

  • Month names depend on the locale. nginx always writes Nov; date +%b writes the month in the current locale, which on a German server is Nov. and on a French one nov.. LC_ALL=C pins it.
  • Filtering by day with index($4, "[08/Nov/2026:") == 1 is an exact string test. A regex built from a date would need escaping for the / characters.
  • The 95th percentile needs all values per endpoint, sorted. asort is a gawk extension; mawk, the default awk on stock Debian and Ubuntu, does not have it, which is why the script calls gawk by name and checks for it in require_commands.
  • zcat -f passes uncompressed files through unchanged, so one command reads both access.log and access.log.2.gz.
  • The case $rc block turns gawk's "no lines for that day" (exit 3) into a clean error, and treats 141 as success; the common mistakes explain why.

Test run

./log-report.sh -d 2026-11-08 logs/access.log
Traffic report for 2026-11-08
requests: 6150   unique IPs: 15   sent: 171.5 MiB   malformed: 2
 
status classes
  2xx   5492   89.3%
  3xx    357    5.8%
  4xx    247    4.0%
  5xx     54    0.9%
  5xx error rate: 0.88%
 
top client IPs
  203.0.113.42             1035
  198.51.100.7              632
  66.249.66.1               629
  192.0.2.201               353
  192.0.2.15                345
  192.0.2.33                336
  192.0.2.80                334
  203.0.113.9               331
  198.51.100.23             330
  203.0.113.118             330
 
endpoints returning 5xx
  /api/search                47
  /checkout                   7
 
slowest endpoints (min. 20 requests)
  endpoint                 reqs    avg s    p95 s    max s
  /api/search               735    1.261    5.291    5.976
  /checkout                 250    0.688    1.144    1.197
  /api/orders/:id           399    0.214    0.338    0.348
  /login                    162    0.127    0.191    0.200
  /products/:id            1321    0.103    0.172    0.180
  /products                 846    0.096    0.145    0.150
  /api/cart                 622    0.066    0.106    0.110
  /                         716    0.035    0.057    0.060
  /static/app.js            852    0.003    0.004    0.004
  /.env                      66    0.002    0.003    0.003
 
requests per hour
  00   143 ##############
  01   124 ############
  02   140 ##############
  03   132 #############
  04   162 ################
  05   136 #############
  06   149 ##############
  07   271 ###########################
  08   248 ########################
  09   422 ##########################################
  10   400 ########################################
  11   410 #########################################
  12   280 ############################
  13   267 ##########################
  14   277 ###########################
  15   258 #########################
  16   282 ############################
  17   253 #########################
  18   286 ############################
  19   408 ########################################
  20   415 #########################################
  21   287 ############################
  22   265 ##########################
  23   135 #############

Reading it the way you would on a Monday morning: the error rate is under 1%, but 47 of the 54 server errors come from /api/search, which is also the slowest endpoint, with a 95th percentile of 5.3 s. That looks like an upstream timeout of about 5 seconds (the 502/504 responses all took 5 to 6 s), so the search backend is the thing to investigate. The average alone (1.26 s) would have hidden how bad the tail is. And /.env with 66 hits is the scanner; fail2ban or a firewall rule is the fix for that, not the report.

The previous day, from a rotated gzipped log, piped into head, then a day with no traffic, then output to a file:

gzip logs/access.log && ls logs
./log-report.sh -d 2026-11-07 -t 3 logs/access.log.gz | head -5; echo "exit: ${PIPESTATUS[0]}"
./log-report.sh -d 2026-11-06 logs/access.log.gz; echo "exit: $?"
./log-report.sh -d 2026-11-08 -o reports/2026-11-08.txt logs/access.log.gz; echo "exit: $?"
ls -l reports
access.log.gz
Traffic report for 2026-11-07
requests: 250   unique IPs: 15   sent: 5.8 MiB   malformed: 0
 
status classes
  2xx    225   90.0%
exit: 0
2026-11-10T01:30:40-0500 log-report.sh[9982] ERROR: no requests for 2026-11-06 in logs/access.log.gz
exit: 1
2026-11-10T01:30:40-0500 log-report.sh[9995] INFO: report written to reports/2026-11-08.txt
exit: 0
total 4
-rw-r--r-- 1 sid sid 2243 Nov 10 01:30 2026-11-08.txt

The empty day is an error on purpose: a report that says "0 requests" usually means the log path or rotation changed, and someone should find out.

The whole report took 0.07 s for 6,400 lines when I timed it. gawk handles a few million lines a minute on ordinary hardware; beyond that, a log pipeline (Loki, Elasticsearch, ClickHouse) is the better tool.

Scheduling it

nginx rotates access.log to access.log.1 around midnight (via logrotate), so at 06:00 yesterday's traffic is in access.log.1 (plus a few minutes in access.log):

0 6 * * *  /opt/ops/log-report.sh -o /srv/reports/access-$(date +\%F -d yesterday).txt /var/log/nginx/access.log.1 /var/log/nginx/access.log 2>> /var/log/ops/log-report.log

Note the \% from the cron article. Mail the file, post it to chat, or publish it on an internal page.

Tool 3: disk cleanup with safety rails

Requirements

  • Delete regular files older than N days from scratch directories (upload temp dirs, render caches, export spools), then remove directories left empty.
  • Default to a dry run. Deleting requires -y. For a tool whose only job is rm, the safe mode must be the one you get by accident.
  • Refuse dangerous targets: /, system directories, the home directory, symlinks, and anything that resolves to them (/etc/../usr, ~/project/..).
  • Never follow symlinks or cross into other filesystems.
  • Stop if far more files match than expected: a wrong -d value should not be able to wipe a directory.
  • Handle any file name, including spaces and newlines; report what was freed.

The code

#!/usr/bin/env bash
#
# cleanup.sh - delete old files from scratch directories, carefully
#
# Usage: cleanup.sh [-y] [-f] [-v] [-d DAYS] [-x PATTERN]... [-m MAX] [-l LOG] DIR...
#
# Dry run is the DEFAULT: nothing is deleted unless -y is given.
# Exit codes: 0 ok, 1 failure or refused, 2 usage error, 4 missing dependency,
#             5 already running
 
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
 
DAYS=14
MAX_FILES=500
FORCE=0
DRY_RUN=1                 # the safe default for a deleting tool
EXCLUDES=()
# Never clean these, or anything that resolves to them
readonly PROTECTED=(/ /bin /boot /dev /etc /home /lib /lib64 /opt /proc /root
                    /run /sbin /srv /sys /usr /var /var/lib /var/log "$HOME")
 
usage() {
  cat <<EOF
Usage: $SCRIPT_NAME [options] DIR...
 
Delete regular files older than DAYS days under each DIR, then remove
directories left empty. Without -y it only reports what it would delete.
 
Options:
  -d DAYS     minimum age in days (default: $DAYS, at least 1)
  -x PATTERN  never delete files whose name matches PATTERN (repeatable)
  -m MAX      refuse to delete more than MAX files in one run (default: $MAX_FILES)
  -f          allow more than MAX files
  -y          really delete (default is a dry run)
  -l FILE     also append log lines to FILE
  -v          verbose
  -h          show this help
EOF
}
 
parse_args() {
  local opt
  while getopts ':d:x:m:l:fyvh' opt; do
    case $opt in
      d) DAYS=$OPTARG ;;
      x) EXCLUDES+=(! -name "$OPTARG") ;;
      m) MAX_FILES=$OPTARG ;;
      f) FORCE=1 ;;
      y) DRY_RUN=0 ;;
      l) LOG_FILE=$OPTARG ;;
      v) VERBOSE=1 ;;
      h) usage; exit 0 ;;
      :) usage_error "option -$OPTARG requires an argument" ;;
      \?) usage_error "unknown option: -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))
  DIRS=("$@")
  (( ${#DIRS[@]} > 0 )) || usage_error "no directories given"
  [[ $DAYS =~ ^[1-9][0-9]*$ ]] || usage_error "-d needs a whole number of days >= 1, got '$DAYS'"
  [[ $MAX_FILES =~ ^[1-9][0-9]*$ ]] || usage_error "-m needs a positive number, got '$MAX_FILES'"
}
 
# check_dir DIR: set REAL_DIR to the resolved path if DIR is safe to clean, else die
# (it must not run inside $(...): die would only exit the subshell)
check_dir() {
  local dir=$1 real p
  [[ -d $dir ]] || die 1 "not a directory: $dir"
  [[ ! -L $dir ]] || die 1 "refusing to clean a symlink: $dir"
  real=$(realpath -e -- "$dir")
  for p in "${PROTECTED[@]}"; do
    [[ $real != "$p" ]] || die 1 "refusing to clean protected directory: $dir ($real)"
  done
  [[ $real == /*/* ]] || die 1 "refusing to clean a top-level directory: $real"
  REAL_DIR=$real
}
 
# list_old DIR: NUL-separated "SIZE PATH" records for files that would be deleted
list_old() {
  find -P "$1" -xdev -type f -mtime +"$((DAYS - 1))" "${EXCLUDES[@]}" -printf '%s %p\0'
}
 
main() {
  parse_args "$@"
  require_commands find realpath numfmt flock
  acquire_lock "${TMPDIR:-/tmp}/cleanup.lock"
 
  local dir real count bytes size path total_count=0 total_bytes=0
  for dir in "${DIRS[@]}"; do
    check_dir "$dir"
    real=$REAL_DIR
 
    count=$(list_old "$real" | tr -cd '\0' | wc -c)
    if (( count > MAX_FILES && ! FORCE )); then
      die 1 "$real: $count files match, more than the limit of $MAX_FILES; check -d, or use -f"
    fi
 
    bytes=0
    while IFS=' ' read -r -d '' size path; do
      debug "old: $path"
      run rm -f -- "$path"
      bytes=$((bytes + size))
    done < <(list_old "$real")
 
    if (( ! DRY_RUN )); then
      find -P "$real" -xdev -mindepth 1 -type d -empty -delete
    fi
    info "$real: $count files, $(numfmt --to=iec "$bytes")B older than $DAYS days"
    total_count=$((total_count + count))
    total_bytes=$((total_bytes + bytes))
  done
 
  if (( DRY_RUN )); then
    info "dry run: would delete $total_count files ($(numfmt --to=iec "$total_bytes")B); rerun with -y to delete"
  else
    info "deleted $total_count files, freed $(numfmt --to=iec "$total_bytes")B"
  fi
}
 
main "$@"

The safety rails, one by one:

  • realpath -e before any check. Comparing the string the user typed is useless: /etc/../usr is not equal to /usr, but it is /usr. Every check runs on the resolved path.
  • A deny-list plus a depth rule. System directories and $HOME are refused by name, and /*/* rejects anything at the top level (/tmp, /data), so the cleanup always targets a specific subdirectory.
  • find -P … -xdev -type f: never follow symlinks (-P is the default, written out so nobody "fixes" it to -L), never descend into another mounted filesystem, and only ever delete regular files.
  • The count guard. The number of matches is computed first; if it exceeds -m, the run stops before deleting anything.
  • -mtime +$((DAYS - 1)). find -mtime +14 means "more than 14 full days", i.e. 15 or more, because the age is rounded down to whole days first. Subtracting one gives "at least DAYS days old", which is what people expect from -d 14.
  • NUL-separated records (-printf '%s %p\0', read -d '') survive any file name; IFS=' ' read -r size path splits only at the first space, so a path containing spaces stays intact.

Test run

A scratch area with old exports, a render cache whose name contains a space, old and new thumbnails, a .keep marker file, and a symlink pointing at a directory of precious config that is also old:

find scratch precious -printf '%TY-%Tm-%Td %7s %p\n' | sort -k3
2026-11-10    4096 precious
2026-08-12      19 precious/app.env
2026-11-10    4096 scratch
2026-11-10      11 scratch/config-link
2026-11-10    4096 scratch/exports
2026-10-01       0 scratch/exports/.keep
2026-10-01      11 scratch/exports/orders-2026-10-01.csv
2026-10-01      11 scratch/exports/orders-2026-10-02.csv
2026-10-01      11 scratch/exports/orders-2026-10-03.csv
2026-11-08      11 scratch/exports/orders-2026-11-09.csv
2026-11-10    4096 scratch/render cache
2026-11-10    4096 scratch/render cache/2026-10
2026-10-20  122880 scratch/render cache/2026-10/page 1.html
2026-10-20  122880 scratch/render cache/2026-10/page 2.html
2026-10-20  122880 scratch/render cache/2026-10/page 3.html
2026-10-20  122880 scratch/render cache/2026-10/page 4.html
2026-11-10    4096 scratch/thumbs
2026-11-10    4096 scratch/thumbs/new
2026-11-08   30720 scratch/thumbs/new/p1.jpg
2026-11-08   30720 scratch/thumbs/new/p2.jpg
2026-11-10    4096 scratch/thumbs/old
2026-10-11   30720 scratch/thumbs/old/p1.jpg
2026-10-11   30720 scratch/thumbs/old/p2.jpg
2026-10-11   30720 scratch/thumbs/old/p3.jpg
2026-10-11   30720 scratch/thumbs/old/p4.jpg
2026-10-11   30720 scratch/thumbs/old/p5.jpg
2026-10-11   30720 scratch/thumbs/old/p6.jpg

Running without -y only reports (output trimmed to a few of the 13 lines):

./cleanup.sh -x '.keep' scratch; echo "exit: $?"
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: dry-run: rm -f -- /home/sid/ops/scratch/exports/orders-2026-10-01.csv
[…]
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: dry-run: rm -f -- /home/sid/ops/scratch/thumbs/old/p6.jpg
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: dry-run: rm -f -- /home/sid/ops/scratch/render\ cache/2026-10/page\ 3.html
[…]
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: /home/sid/ops/scratch: 13 files, 661KB older than 14 days
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: dry run: would delete 13 files (661KB); rerun with -y to delete
exit: 0

With a deliberately low limit, the guard stops the run:

./cleanup.sh -m 5 -x '.keep' scratch; echo "exit: $?"
2026-11-10T01:30:40-0500 cleanup.sh[10106] ERROR: /home/sid/ops/scratch: 13 files match, more than the limit of 5; check -d, or use -f
exit: 1

For real, then what is left, then a second run:

./cleanup.sh -y -x '.keep' -l cleanup.log scratch; echo "exit: $?"
find scratch precious | sort
./cleanup.sh -y -x '.keep' scratch; echo "exit: $?"
2026-11-10T01:30:40-0500 cleanup.sh[10119] INFO: /home/sid/ops/scratch: 13 files, 661KB older than 14 days
2026-11-10T01:30:40-0500 cleanup.sh[10119] INFO: deleted 13 files, freed 661KB
exit: 0
precious
precious/app.env
scratch
scratch/config-link
scratch/exports
scratch/exports/.keep
scratch/exports/orders-2026-11-09.csv
scratch/thumbs
scratch/thumbs/new
scratch/thumbs/new/p1.jpg
scratch/thumbs/new/p2.jpg
2026-11-10T01:30:40-0500 cleanup.sh[10169] INFO: /home/sid/ops/scratch: 0 files, 0B older than 14 days
2026-11-10T01:30:40-0500 cleanup.sh[10169] INFO: deleted 0 files, freed 0B
exit: 0

Old files are gone, the emptied render cache and thumbs/old directories are gone, the recent files and the .keep marker stayed, and precious/app.env (90 days old, reachable through the symlink) was never touched. The second run is a no-op. Now the inputs that must be refused:

./cleanup.sh -d 0 scratch; echo "exit: $?"
./cleanup.sh /; echo "exit: $?"
./cleanup.sh ~/projects/..; echo "exit: $?"
./cleanup.sh /etc/../usr; echo "exit: $?"
./cleanup.sh /tmp; echo "exit: $?"
./cleanup.sh scratch/config-link; echo "exit: $?"
2026-11-10T01:30:40-0500 cleanup.sh[10191] ERROR: -d needs a whole number of days >= 1, got '0'
Try 'cleanup.sh -h' for more information.
exit: 2
2026-11-10T01:30:40-0500 cleanup.sh[10196] ERROR: refusing to clean protected directory: / (/)
exit: 1
2026-11-10T01:30:40-0500 cleanup.sh[10204] ERROR: refusing to clean protected directory: /home/sid/projects/.. (/home/sid)
exit: 1
2026-11-10T01:30:40-0500 cleanup.sh[10212] ERROR: refusing to clean protected directory: /etc/../usr (/usr)
exit: 1
2026-11-10T01:30:40-0500 cleanup.sh[10220] ERROR: refusing to clean a top-level directory: /tmp
exit: 1
2026-11-10T01:30:40-0500 cleanup.sh[10228] ERROR: refusing to clean a symlink: scratch/config-link
exit: 1

Scheduling it

Weekly, Sunday 04:00, after the backup has run:

0 4 * * 0  /opt/ops/cleanup.sh -y -d 14 -x '.keep' -l /var/log/ops/cleanup.log /srv/shop/tmp /var/cache/shop/thumbs 2>/dev/null

The -y is the one place it is written out: a human running the script by hand gets the dry run, and the scheduled job is the only thing that deletes.

Where you'll use this

These three are the core of a small ops toolkit, typically installed together in /opt/ops/ (with lib/ next to them) and scheduled from one crontab or a set of timers. Here is the full crontab I installed on the lab machine to check that cron accepts every line, before removing it again:

MAILTO=""
# m  h  dom mon dow  command
30   2  *   *   *    /opt/ops/backup.sh -b shop -d /backups/shop -k 14 -x '*.log' -x 'shop/cache' -l /var/log/ops/backup.log /srv/shop /etc/nginx 2>/dev/null
0    6  *   *   *    /opt/ops/log-report.sh -o /srv/reports/access-$(date +\%F -d yesterday).txt /var/log/nginx/access.log.1 /var/log/nginx/access.log 2>> /var/log/ops/log-report.log
0    4  *   *   0    /opt/ops/cleanup.sh -y -d 14 -x '.keep' -l /var/log/ops/cleanup.log /srv/shop/tmp /var/cache/shop/thumbs 2>/dev/null
crontab ops.crontab && crontab -l | grep -c ops && crontab -r
systemd-analyze calendar '*-*-* 02:30:00' '*-*-* 06:00:00' 'Sun *-*-* 04:00:00' | grep -E 'Normalized|Next'
3
Normalized form: *-*-* 02:30:00
    Next elapse: Tue 2026-11-10 02:30:00 EST
Normalized form: *-*-* 06:00:00
    Next elapse: Tue 2026-11-10 06:00:00 EST
Normalized form: Sun *-*-* 04:00:00
    Next elapse: Sun 2026-11-15 04:00:00 EST

The second command shows the matching OnCalendar= values if you prefer timers. Before trusting any of them in cron, run each once the way cron will, with env -i PATH=/usr/bin:/bin HOME=$HOME ./backup.sh …; I did that for the backup script and it behaved identically. Beyond this shop, the same patterns cover database dumps (pg_dump | gzip instead of tar, same verify/rotate logic), reports from any line-oriented log (application logs, mail logs, firewall logs), and cleanup of CI build directories, Docker build caches or upload spools.

Common mistakes

exit inside $(...) only exits the subshell

My first version of cleanup.sh returned the resolved path from check_dir by printing it, and called real=$(check_dir "$dir"). Refusing / then produced this:

2026-11-10T01:28:17-0500 cleanup.sh[3598] ERROR: refusing to clean protected directory: / (/)
2026-11-10T01:28:17-0500 cleanup.sh[3598] ERROR: line 92: 'real=$(check_dir "$dir")' exited with status 1
exit: 1

Command substitution runs in a subshell, so die exited only the subshell. The parent script survived, saw a failed assignment, and its ERR trap logged a second, confusing error. It still stopped (thanks to set -e), but without set -e it would have carried on with an empty $real. Functions that may call exit should return results through a global variable (REAL_DIR here) or a printf -v name, not through $(...).

Exit status 141 when you pipe into head

With set -o pipefail, ./log-report.sh … | head -5 at first produced a screenful of errors (trimmed here):

  203.0.113.42               38
2026-11-10T01:27:00-0500 log-report.sh[2710] ERROR: line 60: 'gawk -v day="$1" -v label="$2" -v top="$3" '
    # nginx writes [08/Nov/2026:13:55:36 -0500]; keep only the wanted day
    index($4, "[" day ":") != 1 { next }
[…]
    }'' exited with status 141
2026-11-10T01:27:00-0500 log-report.sh[2710] ERROR: line 125: 'report "$logday" "$DAY" "$TOP"' exited with status 141

head exits after five lines, the next write by gawk gets SIGPIPE, and 141 is 128 + 13 (SIGPIPE). pipefail reports that as a failure. For a command whose output a reader may legitimately cut short, treat 141 as success, as log-report.sh now does. The same output is also why the library's ERR handler prints only the first line of $BASH_COMMAND.

Absolute paths in tar

tar -czf /tmp/abs.tar.gz /home/sid/ops/etc-shop; tar -tzf /tmp/abs.tar.gz | head -2
tar: Removing leading `/' from member names
home/sid/ops/etc-shop/
home/sid/ops/etc-shop/nginx-shop.conf

GNU tar strips the leading / so that extracting cannot overwrite system paths, warns on stderr every time (which pollutes cron logs), and stores the whole directory chain. Restoring then means digging through home/sid/…. Using -C parent name for each source, as backup.sh does, gives short, relocatable member names and no warning.

Summary

  • Shared helpers belong in one sourced library once you have more than one tool; locate it with $(dirname "$(realpath "$0")") and tell shellcheck with # shellcheck source=.
  • Backups: archive to a .partial name, verify (gzip -t, tar -t), rename, write a checksum, and only then rotate. Use tar -C dir name. Test a restore with diff -r.
  • Log reports: normalize URLs (:id, no query strings), use percentiles not just averages, pin the locale for month names, read rotated logs with zcat -f, and treat an empty day as an error.
  • Cleanup: dry run by default, resolve paths with realpath -e before checking them, refuse protected and top-level directories and symlinks, find -P -xdev -type f, a match-count guard, and NUL-separated file lists.
  • exit inside $(...) exits only the subshell; return values through variables.
  • Under pipefail, a pipe into head can yield 141 (SIGPIPE); decide explicitly whether that counts as failure.
  • Make the ERR trap exit with your own documented failure code, not the failed command's status.

Exercises

Exercise 1: Verify every backupeasy

Write verify-backups.sh DEST that checks every *.tar.gz in DEST against its .sha256 file and with gzip -t, prints OK or FAILED per archive, and exits 1 if any archive fails (or none were found). Test it by truncating a copy of a backup.

Show Solution
#!/usr/bin/env bash
# verify-backups.sh DEST - check every backup in DEST against its checksum
set -uo pipefail
dest=${1:?usage: verify-backups.sh DEST}
failed=0 checked=0
 
shopt -s nullglob
for sum in "$dest"/*.tar.gz.sha256; do
  archive=${sum%.sha256}
  checked=$((checked + 1))
  if (cd "$dest" && sha256sum --quiet -c "${sum##*/}") && gzip -t "$archive"; then
    echo "OK      ${archive##*/}"
  else
    echo "FAILED  ${archive##*/}"
    failed=$((failed + 1))
  fi
done
 
echo "$checked checked, $failed failed"
(( checked > 0 && failed == 0 ))
./verify-backups.sh backups; echo "exit: $?"
truncate -s -100 backups/shop-20261110-013037.tar.gz
./verify-backups.sh backups; echo "exit: $?"
OK      shop-20261110-013034.tar.gz
OK      shop-20261110-013036.tar.gz
OK      shop-20261110-013037.tar.gz
3 checked, 0 failed
exit: 0
OK      shop-20261110-013034.tar.gz
OK      shop-20261110-013036.tar.gz
shop-20261110-013037.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
FAILED  shop-20261110-013037.tar.gz
3 checked, 1 failed
exit: 1

Bit rot, a disk problem or an interrupted copy to remote storage all show up here. Schedule it daily on the machine that holds the copies, and alert on exit status 1.

Exercise 2: Busiest minutesmedium

Traffic spikes hide inside hourly totals. Add a "busiest minutes" section to log-report.sh that lists the five minutes of the day with the most requests.

Hint

The timestamp field $4 holds [08/Nov/2026:13:55:36; characters 14 to 18 are HH:MM. The script already has a topn function.

Show Solution

Two lines: count in the main rule, print in END.

       hour[substr($4, 14, 2)]++
+      minute[substr($4, 14, 5)]++
[…]
       if (err5) topn(errors, "endpoints returning 5xx", top)
+      topn(minute, "busiest minutes", 5)
./ex2-log-report.sh -d 2026-11-08 logs/access.log.gz | sed -n '/busiest minutes/,/^$/p'
busiest minutes
  11:26                      17
  10:15                      14
  19:16                      14
  19:51                      14
  09:24                      13
 

topn sorts numerically on the count and breaks ties alphabetically, which is why 10:15 comes before 19:16. On real traffic, compare the busiest minute with the hourly average: 17 requests in a minute against roughly 7 per minute in that hour is a burst worth knowing about when you size the server.

Exercise 3: Don't delete files that are still openhard

As the process management article showed, deleting a file that a process still has open frees no space and can break the process. Make cleanup.sh skip (with a warning) any old file that some process has open, and make the summary count only files actually deleted. Test it with a background process holding one of three old files open.

Show Solution

The loop gets a check before run rm, and lsof joins require_commands:

    while IFS=' ' read -r -d '' size path; do
      # deleting a file a process still has open frees no space (see lsof +L1)
      if lsof -t -- "$path" > /dev/null 2>&1; then
        warn "skipping open file: $path"
        count=$((count - 1))
        continue
      fi
      debug "old: $path"
      run rm -f -- "$path"
      bytes=$((bytes + size))
    done < <(list_old "$real")

The test: three 20-day-old temp files, one held open by a background subshell with exec 3< scratch2/worker-tmp/job-2.tmp; sleep 5.

./ex3-cleanup.sh -y scratch2; echo "exit: $?"
ls scratch2/worker-tmp
2026-11-10T01:34:34-0500 ex3-cleanup.sh[15879] WARN: skipping open file: /home/sid/ops/scratch2/worker-tmp/job-2.tmp
2026-11-10T01:34:34-0500 ex3-cleanup.sh[15879] INFO: /home/sid/ops/scratch2: 2 files, 12B older than 14 days
2026-11-10T01:34:34-0500 ex3-cleanup.sh[15879] INFO: deleted 2 files, freed 12B
exit: 0
job-2.tmp

lsof -t FILE exits 0 when at least one process has the file open. Two caveats: without root, lsof only sees your own processes, so run the cleanup as the user that owns the files (or as root); and calling lsof once per file is slow for tens of thousands of files. For large runs, collect the open paths once with lsof -Fn +D DIR and look them up in an associative array.


Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), GNU tar 1.35, GNU Awk 5.2.1, GNU coreutils 9.4, ShellCheck 0.9.0, November 2026. All three scripts and the exercises were run as shown against generated test data (zero-filled stand-in files and a deterministic synthetic access log). The crontab was installed only to confirm cron accepts it and removed straight away; nothing ran against real /srv, /backups or /var/log paths. Testing produced three of the fixes described above: the ERR handler's first-line trimming and exit code, the SIGPIPE handling in the report, and check_dir no longer running inside $(...). Paths from the test machine's scratch directories are shown shortened to /home/sid/….

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.