Real-World Scripts: Backups, Log Reports, and Cleanup
Three complete, tested Bash tools built on a shared library: rotating verified tar backups, a daily nginx traffic report, and a disk cleanup with safety rails.
By The Dark DebuggerUpdated 38 min read
This article builds three tools that almost every Linux server ends up needing: a backup script with rotation and verification, a daily report from web server access logs, and a cleanup job for directories that fill up with old files. Each one follows the template from writing production-ready scripts, so read that first. For each tool you get the requirements, the complete code, a test run with real output, and how to schedule it (cron and systemd timers).
Why these three
They cover the three things that most often go wrong on a server nobody is watching:
data that cannot be restored, problems nobody notices until customers complain, and disks
that fill up. They also share a property that makes them good practice: each one is
simple to write badly and needs real care to write well. A backup that is never verified,
a report that silently counts the wrong day, or a cleanup that follows a symlink into
/etc are all one missing line away.
A shared library
The template from the previous article is one self-contained file, which is right for a
single tool. With three tools, copying the same 80 lines of logging, locking and cleanup
into each one means fixing every bug three times. So the helpers move into
lib/common.sh, and each script sources it:
# lib/common.sh - shared helpers for the ops scripts. Source it; do not run it.
#
# The caller sets SCRIPT_NAME and may set LOG_FILE, DRY_RUN and VERBOSE.
# Provides logging, a dry-run aware run(), dependency checks, flock locking,
# and cleanup of registered temporary paths on any exit.
# Exit codes used here: 2 usage, 4 missing dependency, 5 already running,
# 130/143 interrupted/terminated.
LOG_FILE=${LOG_FILE:-}
DRY_RUN=${DRY_RUN:-0}
VERBOSE=${VERBOSE:-0}
TMP_PATHS=()
log() {
local level=$1 line
shift
line="$(date '+%Y-%m-%dT%H:%M:%S%z') ${SCRIPT_NAME}[$$] $level: $*"
printf '%s\n' "$line" >&2
if [[ -n $LOG_FILE ]]; then
printf '%s\n' "$line" >> "$LOG_FILE"
fi
}
debug() { if (( VERBOSE )); then log DEBUG "$@"; fi; }
info() { log INFO "$@"; }
warn() { log WARN "$@"; }
error() { log ERROR "$@"; }
die() { local code=$1; shift; error "$@"; exit "$code"; }
usage_error() {
error "$*"
echo "Try '$SCRIPT_NAME -h' for more information." >&2
exit 2
}
# run CMD [ARGS...]: execute, or only log the command (shell-quoted) in dry-run mode
run() {
if (( DRY_RUN )); then
info "dry-run:$(printf ' %q' "$@")"
else
debug "run:$(printf ' %q' "$@")"
"$@"
fi
}
require_commands() {
local cmd missing=()
for cmd in "$@"; do
command -v "$cmd" > /dev/null 2>&1 || missing+=("$cmd")
done
if (( ${#missing[@]} > 0 )); then
die 4 "missing required commands: ${missing[*]}"
fi
}
# acquire_lock FILE: exit 5 if another process holds the lock
acquire_lock() {
exec {LOCK_FD}> "$1"
if ! flock -n "$LOCK_FD"; then
die 5 "another run is in progress (lock: $1)"
fi
}
# register_tmp PATH: delete PATH (file or directory) when the script exits
register_tmp() { TMP_PATHS+=("$1"); }
_cleanup() {
local rc=$? path
for path in "${TMP_PATHS[@]}"; do
if [[ -e $path ]]; then
rm -rf -- "$path"
debug "removed temporary path $path"
fi
done
exit "$rc"
}
_on_error() {
local rc=$? line=$1
# first line only: a multi-line awk program would flood the log
error "line $line: '${BASH_COMMAND%%$'\n'*}' exited with status $rc"
exit 1 # keep the documented exit codes: tar's 2 is not a usage error
}
trap _cleanup EXIT
trap '_on_error $LINENO' ERR
trap 'die 130 "interrupted"' INT
trap 'die 143 "terminated"' TERMThree differences from the template, each added because testing showed it was needed:
runlogs commands withprintf ' %q', so paths with spaces are unambiguous (this was exercise 2 of the previous article).register_tmplets a script register any number of temporary files or directories, and_cleanupremoves them all._on_errorprints only the first line of the failed command, and exits with 1. Both came out of the testing below: a failinggawkprogram dumped 60 lines into the log, and a failingtarmade the backup script exit with tar's own status 2, which collided with "usage error".
Each script starts the same way. realpath "$0" finds the library next to the script
even when it is called through a symlink or from another directory, and the
# shellcheck source= comment tells shellcheck where to look when run with -x:
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"All three scripts below pass shellcheck -x with no warnings:
shellcheck -x backup.sh log-report.sh cleanup.sh && echo cleancleanTool 1: rotating, verified backups
Requirements
- Back up one or more directories into a single
NAME-YYYYmmdd-HHMMSS.tar.gz. - Exclude junk that should not be backed up (logs, caches) by pattern.
- Never leave a half-written archive that looks like a real backup.
- Verify the archive before it counts, and store a SHA-256 checksum next to it.
- Keep the newest N backups and delete older ones, but only after the new one is good.
- Refuse to start if the destination does not have room.
- Locking, dry run, logging and clear exit codes from the template.
The code
#!/usr/bin/env bash
#
# backup.sh - rotating, verified tar.gz backups of one or more directories
#
# Usage: backup.sh [-n] [-v] [-k KEEP] [-x PATTERN]... [-l LOG] -b NAME -d DEST DIR...
#
# Creates DEST/NAME-YYYYmmdd-HHMMSS.tar.gz plus a .sha256 file, verifies it,
# then deletes the oldest backups so only KEEP remain.
# Exit codes: 0 ok, 1 failure, 2 usage error, 4 missing dependency, 5 already running
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
NAME="" DEST="" KEEP=7
EXCLUDES=()
usage() {
cat <<EOF
Usage: $SCRIPT_NAME [options] -b NAME -d DEST DIR...
Back up DIR... into DEST/NAME-YYYYmmdd-HHMMSS.tar.gz, verify it, keep the newest KEEP.
Options:
-b NAME backup name, used as the file prefix (required)
-d DEST directory to write backups to (required)
-k KEEP number of backups to keep (default: $KEEP)
-x PATTERN exclude files matching PATTERN (tar --exclude syntax, repeatable)
-l FILE also append log lines to FILE
-n dry run
-v verbose
-h show this help
EOF
}
parse_args() {
local opt
while getopts ':b:d:k:x:l:nvh' opt; do
case $opt in
b) NAME=$OPTARG ;;
d) DEST=$OPTARG ;;
k) KEEP=$OPTARG ;;
x) EXCLUDES+=("--exclude=$OPTARG") ;;
l) LOG_FILE=$OPTARG ;;
n) DRY_RUN=1 ;;
v) VERBOSE=1 ;;
h) usage; exit 0 ;;
:) usage_error "option -$OPTARG requires an argument" ;;
\?) usage_error "unknown option: -$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
SOURCES=("$@")
[[ -n $NAME ]] || usage_error "missing required option -b NAME"
[[ $NAME =~ ^[A-Za-z0-9._-]+$ ]] || usage_error "NAME may only contain letters, digits, . _ -"
[[ -n $DEST ]] || usage_error "missing required option -d DEST"
[[ $KEEP =~ ^[1-9][0-9]*$ ]] || usage_error "-k needs a number >= 1, got '$KEEP'"
(( ${#SOURCES[@]} > 0 )) || usage_error "no directories to back up"
local src
for src in "${SOURCES[@]}"; do
[[ -d $src ]] || die 1 "not a directory: $src"
done
}
# check_space: refuse to start if DEST has less free space than the sources use
check_space() {
local need avail
need=$(du -sk -- "${SOURCES[@]}" | awk '{ s += $1 } END { print s }')
avail=$(df -Pk -- "$DEST" | awk 'NR == 2 { print $4 }')
debug "sources use ${need} KiB, ${avail} KiB free in $DEST"
if (( avail < need )); then
die 1 "not enough space in $DEST: need ${need} KiB, have ${avail} KiB"
fi
}
create_archive() {
local archive=$1 partial=$1.partial src members=()
for src in "${SOURCES[@]}"; do
src=$(realpath -- "$src")
members+=(-C "$(dirname -- "$src")" "$(basename -- "$src")")
done
register_tmp "$partial"
tar -czf "$partial" "${EXCLUDES[@]}" "${members[@]}"
# verify before the backup is allowed to count
gzip -t -- "$partial"
local entries
entries=$(tar -tzf "$partial" | wc -l)
(( entries > 0 )) || die 1 "archive is empty: $partial"
mv -- "$partial" "$archive"
(cd -- "$DEST" && sha256sum -- "${archive##*/}" > "${archive##*/}.sha256")
info "created ${archive##*/}: $entries entries, $(du -h -- "$archive" | cut -f1)"
}
# rotate: keep the newest KEEP backups of NAME, delete the rest
rotate() {
local backups=() old
shopt -s nullglob
backups=("$DEST/$NAME"-[0-9]*-[0-9]*.tar.gz) # glob results are sorted, oldest first
shopt -u nullglob
if (( ${#backups[@]} <= KEEP )); then
debug "${#backups[@]} backups present, keeping all"
return 0
fi
for old in "${backups[@]:0:${#backups[@]}-KEEP}"; do
info "rotating out ${old##*/}"
run rm -f -- "$old" "$old.sha256"
done
}
main() {
parse_args "$@"
require_commands tar gzip sha256sum flock du df realpath
run mkdir -p -- "$DEST"
acquire_lock "${TMPDIR:-/tmp}/backup-$NAME.lock"
if (( ! DRY_RUN )); then check_space; fi
local started archive
started=$(date +%s)
archive="$DEST/$NAME-$(date +%Y%m%d-%H%M%S).tar.gz"
[[ ! -e $archive ]] || die 1 "$archive already exists"
info "backing up ${SOURCES[*]} to ${archive##*/} (keep $KEEP)"
run create_archive "$archive"
rotate
info "done in $(( $(date +%s) - started ))s"
}
main "$@"The decisions that matter:
-C dir basenamefor each source. tar stores paths relative to each-Cdirectory, so the archive containsshop/…andetc-nginx/…rather than the full absolute path (see the common mistakes).- Write
.partial, verify, then rename. The rotation glob only matches*.tar.gz, so a crashed run can never be mistaken for a backup or push a good one out of the rotation. The partial file is registered for cleanup. - Rotation runs after the new backup is verified. If today's backup fails, the old ones are all still there.
- The timestamp in the name sorts chronologically, so a plain glob lists backups
oldest first and
"${backups[@]:0:count-KEEP}"is exactly the ones to delete. - The checksum file uses a bare file name (that is why
sha256sumruns insideDEST), sosha256sum -cstill works after the backups are copied elsewhere.
Test run
A small shop application: code, config, uploads, plus logs and a session cache that should not be backed up.
du -sh srv/shop/*12K srv/shop/app
108K srv/shop/cache
8.0K srv/shop/config
2.3M srv/shop/logs
252K srv/shop/uploadsA dry run first, then four real runs a second apart with -k 3:
./backup.sh -n -b shop -d backups -k 3 -x '*.log' -x 'shop/cache' srv/shop etc-nginx
./backup.sh -b shop -d backups -k 3 -x '*.log' -x 'shop/cache' srv/shop etc-nginx2026-11-10T01:30:33-0500 backup.sh[9585] INFO: dry-run: mkdir -p -- backups
2026-11-10T01:30:33-0500 backup.sh[9585] INFO: backing up srv/shop etc-nginx to shop-20261110-013033.tar.gz (keep 3)
2026-11-10T01:30:33-0500 backup.sh[9585] INFO: dry-run: create_archive backups/shop-20261110-013033.tar.gz
2026-11-10T01:30:33-0500 backup.sh[9585] INFO: done in 0s
2026-11-10T01:30:33-0500 backup.sh[9605] INFO: backing up srv/shop etc-nginx to shop-20261110-013033.tar.gz (keep 3)
2026-11-10T01:30:33-0500 backup.sh[9605] INFO: created shop-20261110-013033.tar.gz: 18 entries, 4.0K
2026-11-10T01:30:33-0500 backup.sh[9605] INFO: done in 0sThe fourth run is the first one that has something to rotate:
2026-11-10T01:30:37-0500 backup.sh[9802] INFO: backing up srv/shop etc-nginx to shop-20261110-013037.tar.gz (keep 3)
2026-11-10T01:30:37-0500 backup.sh[9802] INFO: created shop-20261110-013037.tar.gz: 18 entries, 4.0K
2026-11-10T01:30:37-0500 backup.sh[9802] INFO: rotating out shop-20261110-013033.tar.gz
2026-11-10T01:30:37-0500 backup.sh[9802] INFO: done in 0sls -l backups
(cd backups && sha256sum -c shop-20261110-013037.tar.gz.sha256)
tar -tzf backups/shop-20261110-013037.tar.gztotal 24
-rw-r--r-- 1 sid sid 1277 Nov 10 01:30 shop-20261110-013034.tar.gz
-rw-r--r-- 1 sid sid 94 Nov 10 01:30 shop-20261110-013034.tar.gz.sha256
-rw-r--r-- 1 sid sid 1277 Nov 10 01:30 shop-20261110-013036.tar.gz
-rw-r--r-- 1 sid sid 94 Nov 10 01:30 shop-20261110-013036.tar.gz.sha256
-rw-r--r-- 1 sid sid 1277 Nov 10 01:30 shop-20261110-013037.tar.gz
-rw-r--r-- 1 sid sid 94 Nov 10 01:30 shop-20261110-013037.tar.gz.sha256
shop-20261110-013037.tar.gz: OK
shop/
shop/config/
shop/config/app.env
shop/logs/
shop/uploads/
shop/uploads/2026/
shop/uploads/2026/11/
shop/uploads/2026/11/invoice-104.pdf
shop/uploads/2026/11/invoice-105.pdf
shop/uploads/2026/11/invoice-103.pdf
shop/uploads/2026/11/invoice-101.pdf
shop/uploads/2026/11/invoice-102.pdf
shop/uploads/2026/11/invoice-106.pdf
shop/app/
shop/app/server.py
shop/app/requirements.txt
etc-nginx/
etc-nginx/shop.confThe logs/ directory is there but empty and cache/ is gone, as the exclusions asked.
(The archives are tiny because the test uploads are zero-filled.) A backup you have never
restored is a hope, not a backup, so restore one and compare:
mkdir -p restore-test && tar -xzf backups/shop-20261110-013037.tar.gz -C restore-test && diff -r --exclude=cache --exclude='*.log' srv/shop restore-test/shop && diff -r etc-nginx restore-test/etc-nginx && echo 'restore matches source'restore matches sourceNow the failure path. A config file that the backup user cannot read is a classic: in a naive script, tar prints a warning, carries on, and you have a "successful" backup without the one file you needed.
./backup.sh -b shop -d backups -k 3 -x '*.log' -x 'shop/cache' srv/shop etc-nginx; echo "exit: $?"
ls -A backups | grep -c partial2026-11-10T01:30:38-0500 backup.sh[9887] INFO: backing up srv/shop etc-nginx to shop-20261110-013038.tar.gz (keep 3)
tar: shop/config/app.env: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors
2026-11-10T01:30:38-0500 backup.sh[9887] ERROR: line 85: 'tar -czf "$partial" "${EXCLUDES[@]}" "${members[@]}"' exited with status 2
exit: 1
0The run failed loudly, the partial archive was removed, and rotation never ran, so the three good backups are untouched. And two runs at once:
flock /tmp/backup-shop.lock sleep 2 & sleep 0.3; ./backup.sh -b shop -d backups srv/shop; echo "exit: $?"; wait2026-11-10T01:30:38-0500 backup.sh[9931] ERROR: another run is in progress (lock: /tmp/backup-shop.lock)
exit: 5Scheduling it
Nightly at 02:30, keeping two weeks:
30 2 * * * /opt/ops/backup.sh -b shop -d /backups/shop -k 14 -x '*.log' -x 'shop/cache' -l /var/log/ops/backup.log /srv/shop /etc/nginx 2>/dev/nullor as a systemd service and timer (OnCalendar=*-*-* 02:30:00, Persistent=true, so a
backup missed while the server was down runs at boot). Two more things a real setup
needs: a copy of the backups on another machine (rsync or object storage; a backup
on the same disk dies with the disk), and the exercise below that verifies every backup's
checksum on a schedule.
Tool 2: a daily access-log report
Requirements
- Read nginx access logs in the standard
combinedformat with$request_time(seconds) appended, plain or gzipped (rotated logs are usually.gz). - Report one day: totals, status classes and the 5xx error rate, top client IPs, endpoints producing 5xx errors, the slowest endpoints by average and 95th percentile, and requests per hour.
- Group
/products/1234and/products/1301as/products/:id, and ignore query strings, or every URL is unique and the "slowest endpoint" list is useless. - Write to stdout, or atomically to a file for publishing.
Sample data
To test against something realistic, I generated a day of traffic for a small shop with
gawk: about 6,400 requests across 15 client IPs, a crawler, a vulnerability scanner
probing /.env and /wp-login.php, a slow search API that sometimes times out with
502/504, an occasional 500 on checkout, and two malformed requests. The generator:
# gen-access-log.awk - write a realistic nginx access log for one day (plus a
# little of the day before), in "combined" format with $request_time appended.
# Usage: gawk -v day=08 -f gen-access-log.awk > access.log
function pick(list, a, n) { n = split(list, a, " "); return a[int(rand() * n) + 1] }
function between(lo, hi) { return lo + rand() * (hi - lo) }
function line(d, h, m, s, ip, method, path, status, bytes, rt, ua) {
ua = (ip ~ /^66\.249\./) ? "Googlebot/2.1 (+http://www.google.com/bot.html)" \
: (ip ~ /^185\.220\./) ? "python-requests/2.31.0" \
: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36"
printf "%s - - [%02d/Nov/2026:%02d:%02d:%02d -0500] \"%s %s HTTP/1.1\" %d %d \"-\" \"%s\" %.3f\n",
ip, d, h, m, s, method, path, status, bytes, ua, rt
}
BEGIN {
srand(42)
ips = "203.0.113.42 203.0.113.42 203.0.113.42 198.51.100.7 198.51.100.7 192.0.2.15 " \
"192.0.2.77 192.0.2.80 198.51.100.23 203.0.113.9 203.0.113.118 192.0.2.201 " \
"66.249.66.1 66.249.66.1 198.51.100.64 203.0.113.250 192.0.2.33 198.51.100.140"
hours = "0 1 2 3 4 5 6 7 7 8 8 9 9 9 10 10 10 11 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19 19 19 20 20 20 21 21 22 22 23"
for (i = 0; i < 6400; i++) {
d = (i < 250) ? day - 1 : day
h = pick(hours); m = int(rand() * 60); s = int(rand() * 60)
ip = pick(ips); r = rand(); status = 200; method = "GET"
if (r < 0.12) { path = "/"; rt = between(0.010, 0.060); bytes = 18200 }
else if (r < 0.25) { path = "/products"; rt = between(0.040, 0.150); bytes = 42100 }
else if (r < 0.47) { path = "/products/" int(between(1000, 1400)); rt = between(0.030, 0.180); bytes = 23800 }
else if (r < 0.57) { path = "/api/cart"; method = "POST"; rt = between(0.020, 0.110); bytes = 610 }
else if (r < 0.69) { path = "/api/search?q=" pick("boots jacket socks gloves scarf"); bytes = 9100
rt = (rand() < 0.15) ? between(1.5, 4.8) : between(0.200, 0.900)
if (rand() < 0.07) { status = pick("502 504"); rt = between(5.0, 6.0); bytes = 166 } }
else if (r < 0.75) { path = "/api/orders/" int(between(70000, 79999)); rt = between(0.080, 0.350); bytes = 1400 }
else if (r < 0.79) { path = "/checkout"; method = "POST"; rt = between(0.250, 1.200); bytes = 2300
if (rand() < 0.04) { status = 500; bytes = 312 } }
else if (r < 0.93) { path = "/static/app.js"; rt = between(0.001, 0.004); bytes = 184000
if (rand() < 0.4) { status = 304; bytes = 0 } }
else if (r < 0.96) { path = "/login"; rt = between(0.050, 0.200); bytes = 5200 }
else { ip = "185.220.101.4"; path = pick("/wp-login.php /.env /xmlrpc.php /admin.php")
status = 404; rt = between(0.001, 0.003); bytes = 153 }
line(d, h, m, s, ip, method, path, status, bytes, rt)
}
# two malformed requests, as every real server receives
printf "185.220.101.4 - - [%02d/Nov/2026:03:12:44 -0500] \"\\x16\\x03\\x01\\x02\\x00\" 400 157 \"-\" \"-\" 0.000\n", day
printf "185.220.101.4 - - [%02d/Nov/2026:03:12:45 -0500] \"-\" 400 0 \"-\" \"-\" 0.000\n", day
}gawk -v day=08 -f gen-access-log.awk | sort -s -k4,4 > logs/access.log
wc -l logs/access.log
sed -n '1p;3000p;$p' logs/access.log6402 logs/access.log
203.0.113.42 - - [07/Nov/2026:00:05:37 -0500] "GET /api/search?q=boots HTTP/1.1" 200 9100 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36" 0.561
198.51.100.23 - - [08/Nov/2026:12:01:31 -0500] "GET /static/app.js HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36" 0.001
192.0.2.77 - - [08/Nov/2026:23:59:54 -0500] "GET /products/1042 HTTP/1.1" 200 23800 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36" 0.082Like a real log file just after midnight rotation, it starts with the tail end of the previous day, which the report must not count.
The code
The heavy lifting is one gawk program (see awk for
the language). With space as the field separator, the IP is $1, the timestamp $4, the
method $6 (with its opening quote), the path $7, status $9, bytes $10, and the
request time is the last field, $NF, because the user agent in between contains spaces.
#!/usr/bin/env bash
#
# log-report.sh - daily traffic report from nginx access logs
#
# Usage: log-report.sh [-v] [-d YYYY-MM-DD] [-t TOP] [-o FILE] LOGFILE...
#
# Expects the "combined" log format with $request_time as the last field.
# Reads plain or gzipped logs. Writes the report to stdout, or atomically to FILE.
# Exit codes: 0 ok, 1 failure (e.g. no lines for that day), 2 usage error,
# 4 missing dependency
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
DAY=$(date -d yesterday +%F)
TOP=10
OUTPUT=""
usage() {
cat <<EOF
Usage: $SCRIPT_NAME [options] LOGFILE...
Options:
-d DATE day to report on, YYYY-MM-DD (default: yesterday, $DAY)
-t N length of the top-N lists (default: $TOP)
-o FILE write the report to FILE instead of stdout
-v verbose
-h show this help
EOF
}
parse_args() {
local opt
while getopts ':d:t:o:vh' opt; do
case $opt in
d) DAY=$OPTARG ;;
t) TOP=$OPTARG ;;
o) OUTPUT=$OPTARG ;;
v) VERBOSE=1 ;;
h) usage; exit 0 ;;
:) usage_error "option -$OPTARG requires an argument" ;;
\?) usage_error "unknown option: -$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
LOGS=("$@")
(( ${#LOGS[@]} > 0 )) || usage_error "no log files given"
[[ $TOP =~ ^[1-9][0-9]*$ ]] || usage_error "-t needs a positive number, got '$TOP'"
date -d "$DAY" +%F > /dev/null 2>&1 || usage_error "not a valid date: $DAY"
local f
for f in "${LOGS[@]}"; do
[[ -r $f ]] || die 1 "cannot read $f"
done
}
# report LOGDAY LABEL TOP: read log lines on stdin, print the report on stdout
report() {
gawk -v day="$1" -v label="$2" -v top="$3" '
# nginx writes [08/Nov/2026:13:55:36 -0500]; keep only the wanted day
index($4, "[" day ":") != 1 { next }
$6 !~ /^"(GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS)$/ { malformed++; next }
{
total++
ip = $1; path = $7; status = $9; bytes = $10; rt = $NF
sub(/\?.*/, "", path) # drop query strings
gsub(/\/[0-9]+(\/|$)/, "/:id/", path) # /products/1234 -> /products/:id
sub(/\/$/, "", path); if (path == "") path = "/"
hits[ip]++
class[substr(status, 1, 1) "xx"]++
if (status >= 500) { errors[path]++; err5++ }
sent += bytes
n[path]++; sum[path] += rt; times[path, n[path]] = rt
hour[substr($4, 14, 2)]++
}
function topn(arr, label, k, i, cmd) {
printf "\n%s\n", label
cmd = "sort -k2,2nr -k1,1 | head -n " k
for (i in arr) printf " %-22s %6d\n", i, arr[i] | cmd
close(cmd)
}
END {
if (total == 0) exit 3
printf "Traffic report for %s\n", label
printf "requests: %d unique IPs: %d sent: %.1f MiB malformed: %d\n",
total, length(hits), sent / 1048576, malformed
printf "\nstatus classes\n"
for (c in class) printf " %s %6d %5.1f%%\n", c, class[c], 100 * class[c] / total | "sort"
close("sort")
printf " 5xx error rate: %.2f%%\n", 100 * err5 / total
topn(hits, "top client IPs", top)
if (err5) topn(errors, "endpoints returning 5xx", top)
printf "\nslowest endpoints (min. 20 requests)\n"
printf " %-22s %6s %8s %8s %8s\n", "endpoint", "reqs", "avg s", "p95 s", "max s"
cmd = "sort -k4,4nr | head -n " top
for (p in n) {
if (n[p] < 20) continue
delete v
for (i = 1; i <= n[p]; i++) v[i] = times[p, i]
asort(v)
printf " %-22s %6d %8.3f %8.3f %8.3f\n", p, n[p], sum[p] / n[p], v[int(0.95 * n[p] + 0.999)], v[n[p]] | cmd
}
close(cmd)
printf "\nrequests per hour\n"
for (h = 0; h < 24; h++) {
hh = sprintf("%02d", h)
printf " %s %5d %s\n", hh, hour[hh], substr("##################################################", 1, int(hour[hh] / 10))
}
}'
}
main() {
parse_args "$@"
require_commands gawk zcat sort date
local logday
logday=$(LC_ALL=C date -d "$DAY" +%d/%b/%Y)
debug "reading ${LOGS[*]} for $logday"
local tmp="" rc=0
if [[ -n $OUTPUT ]]; then
tmp=$(mktemp "$OUTPUT.XXXXXX")
register_tmp "$tmp"
exec 3> "$tmp"
else
exec 3>&1
fi
zcat -f -- "${LOGS[@]}" | report "$logday" "$DAY" "$TOP" >&3 || rc=$?
exec 3>&-
case $rc in
0 | 141) ;; # 141 = SIGPIPE: the reader stopped early (| head), not an error
3) die 1 "no requests for $DAY in ${LOGS[*]}" ;;
*) die 1 "report failed with status $rc" ;;
esac
if [[ -n $OUTPUT ]]; then
chmod 644 "$tmp"
mv -f -- "$tmp" "$OUTPUT"
info "report written to $OUTPUT"
fi
}
main "$@"Notes on the parts that are easy to get wrong:
- Month names depend on the locale. nginx always writes
Nov;date +%bwrites the month in the current locale, which on a German server isNov.and on a French onenov..LC_ALL=Cpins it. - Filtering by day with
index($4, "[08/Nov/2026:") == 1is an exact string test. A regex built from a date would need escaping for the/characters. - The 95th percentile needs all values per endpoint, sorted.
asortis a gawk extension;mawk, the defaultawkon stock Debian and Ubuntu, does not have it, which is why the script callsgawkby name and checks for it inrequire_commands. zcat -fpasses uncompressed files through unchanged, so one command reads bothaccess.logandaccess.log.2.gz.- The
case $rcblock turns gawk's "no lines for that day" (exit 3) into a clean error, and treats 141 as success; the common mistakes explain why.
Test run
./log-report.sh -d 2026-11-08 logs/access.logTraffic report for 2026-11-08
requests: 6150 unique IPs: 15 sent: 171.5 MiB malformed: 2
status classes
2xx 5492 89.3%
3xx 357 5.8%
4xx 247 4.0%
5xx 54 0.9%
5xx error rate: 0.88%
top client IPs
203.0.113.42 1035
198.51.100.7 632
66.249.66.1 629
192.0.2.201 353
192.0.2.15 345
192.0.2.33 336
192.0.2.80 334
203.0.113.9 331
198.51.100.23 330
203.0.113.118 330
endpoints returning 5xx
/api/search 47
/checkout 7
slowest endpoints (min. 20 requests)
endpoint reqs avg s p95 s max s
/api/search 735 1.261 5.291 5.976
/checkout 250 0.688 1.144 1.197
/api/orders/:id 399 0.214 0.338 0.348
/login 162 0.127 0.191 0.200
/products/:id 1321 0.103 0.172 0.180
/products 846 0.096 0.145 0.150
/api/cart 622 0.066 0.106 0.110
/ 716 0.035 0.057 0.060
/static/app.js 852 0.003 0.004 0.004
/.env 66 0.002 0.003 0.003
requests per hour
00 143 ##############
01 124 ############
02 140 ##############
03 132 #############
04 162 ################
05 136 #############
06 149 ##############
07 271 ###########################
08 248 ########################
09 422 ##########################################
10 400 ########################################
11 410 #########################################
12 280 ############################
13 267 ##########################
14 277 ###########################
15 258 #########################
16 282 ############################
17 253 #########################
18 286 ############################
19 408 ########################################
20 415 #########################################
21 287 ############################
22 265 ##########################
23 135 #############Reading it the way you would on a Monday morning: the error rate is under 1%, but 47 of
the 54 server errors come from /api/search, which is also the slowest endpoint, with a
95th percentile of 5.3 s. That looks like an upstream timeout of about 5 seconds
(the 502/504 responses all took 5 to 6 s), so the search backend is the thing to
investigate. The average alone (1.26 s) would have hidden how bad the tail is. And
/.env with 66 hits is the scanner; fail2ban or a firewall rule is the fix for that,
not the report.
The previous day, from a rotated gzipped log, piped into head, then a day with no
traffic, then output to a file:
gzip logs/access.log && ls logs
./log-report.sh -d 2026-11-07 -t 3 logs/access.log.gz | head -5; echo "exit: ${PIPESTATUS[0]}"
./log-report.sh -d 2026-11-06 logs/access.log.gz; echo "exit: $?"
./log-report.sh -d 2026-11-08 -o reports/2026-11-08.txt logs/access.log.gz; echo "exit: $?"
ls -l reportsaccess.log.gz
Traffic report for 2026-11-07
requests: 250 unique IPs: 15 sent: 5.8 MiB malformed: 0
status classes
2xx 225 90.0%
exit: 0
2026-11-10T01:30:40-0500 log-report.sh[9982] ERROR: no requests for 2026-11-06 in logs/access.log.gz
exit: 1
2026-11-10T01:30:40-0500 log-report.sh[9995] INFO: report written to reports/2026-11-08.txt
exit: 0
total 4
-rw-r--r-- 1 sid sid 2243 Nov 10 01:30 2026-11-08.txtThe empty day is an error on purpose: a report that says "0 requests" usually means the log path or rotation changed, and someone should find out.
The whole report took 0.07 s for 6,400 lines when I timed it. gawk handles a few million lines a minute on ordinary hardware; beyond that, a log pipeline (Loki, Elasticsearch, ClickHouse) is the better tool.
Scheduling it
nginx rotates access.log to access.log.1 around midnight (via logrotate), so at 06:00
yesterday's traffic is in access.log.1 (plus a few minutes in access.log):
0 6 * * * /opt/ops/log-report.sh -o /srv/reports/access-$(date +\%F -d yesterday).txt /var/log/nginx/access.log.1 /var/log/nginx/access.log 2>> /var/log/ops/log-report.logNote the \% from the cron article. Mail
the file, post it to chat, or publish it on an internal page.
Tool 3: disk cleanup with safety rails
Requirements
- Delete regular files older than N days from scratch directories (upload temp dirs, render caches, export spools), then remove directories left empty.
- Default to a dry run. Deleting requires
-y. For a tool whose only job isrm, the safe mode must be the one you get by accident. - Refuse dangerous targets:
/, system directories, the home directory, symlinks, and anything that resolves to them (/etc/../usr,~/project/..). - Never follow symlinks or cross into other filesystems.
- Stop if far more files match than expected: a wrong
-dvalue should not be able to wipe a directory. - Handle any file name, including spaces and newlines; report what was freed.
The code
#!/usr/bin/env bash
#
# cleanup.sh - delete old files from scratch directories, carefully
#
# Usage: cleanup.sh [-y] [-f] [-v] [-d DAYS] [-x PATTERN]... [-m MAX] [-l LOG] DIR...
#
# Dry run is the DEFAULT: nothing is deleted unless -y is given.
# Exit codes: 0 ok, 1 failure or refused, 2 usage error, 4 missing dependency,
# 5 already running
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
DAYS=14
MAX_FILES=500
FORCE=0
DRY_RUN=1 # the safe default for a deleting tool
EXCLUDES=()
# Never clean these, or anything that resolves to them
readonly PROTECTED=(/ /bin /boot /dev /etc /home /lib /lib64 /opt /proc /root
/run /sbin /srv /sys /usr /var /var/lib /var/log "$HOME")
usage() {
cat <<EOF
Usage: $SCRIPT_NAME [options] DIR...
Delete regular files older than DAYS days under each DIR, then remove
directories left empty. Without -y it only reports what it would delete.
Options:
-d DAYS minimum age in days (default: $DAYS, at least 1)
-x PATTERN never delete files whose name matches PATTERN (repeatable)
-m MAX refuse to delete more than MAX files in one run (default: $MAX_FILES)
-f allow more than MAX files
-y really delete (default is a dry run)
-l FILE also append log lines to FILE
-v verbose
-h show this help
EOF
}
parse_args() {
local opt
while getopts ':d:x:m:l:fyvh' opt; do
case $opt in
d) DAYS=$OPTARG ;;
x) EXCLUDES+=(! -name "$OPTARG") ;;
m) MAX_FILES=$OPTARG ;;
f) FORCE=1 ;;
y) DRY_RUN=0 ;;
l) LOG_FILE=$OPTARG ;;
v) VERBOSE=1 ;;
h) usage; exit 0 ;;
:) usage_error "option -$OPTARG requires an argument" ;;
\?) usage_error "unknown option: -$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
DIRS=("$@")
(( ${#DIRS[@]} > 0 )) || usage_error "no directories given"
[[ $DAYS =~ ^[1-9][0-9]*$ ]] || usage_error "-d needs a whole number of days >= 1, got '$DAYS'"
[[ $MAX_FILES =~ ^[1-9][0-9]*$ ]] || usage_error "-m needs a positive number, got '$MAX_FILES'"
}
# check_dir DIR: set REAL_DIR to the resolved path if DIR is safe to clean, else die
# (it must not run inside $(...): die would only exit the subshell)
check_dir() {
local dir=$1 real p
[[ -d $dir ]] || die 1 "not a directory: $dir"
[[ ! -L $dir ]] || die 1 "refusing to clean a symlink: $dir"
real=$(realpath -e -- "$dir")
for p in "${PROTECTED[@]}"; do
[[ $real != "$p" ]] || die 1 "refusing to clean protected directory: $dir ($real)"
done
[[ $real == /*/* ]] || die 1 "refusing to clean a top-level directory: $real"
REAL_DIR=$real
}
# list_old DIR: NUL-separated "SIZE PATH" records for files that would be deleted
list_old() {
find -P "$1" -xdev -type f -mtime +"$((DAYS - 1))" "${EXCLUDES[@]}" -printf '%s %p\0'
}
main() {
parse_args "$@"
require_commands find realpath numfmt flock
acquire_lock "${TMPDIR:-/tmp}/cleanup.lock"
local dir real count bytes size path total_count=0 total_bytes=0
for dir in "${DIRS[@]}"; do
check_dir "$dir"
real=$REAL_DIR
count=$(list_old "$real" | tr -cd '\0' | wc -c)
if (( count > MAX_FILES && ! FORCE )); then
die 1 "$real: $count files match, more than the limit of $MAX_FILES; check -d, or use -f"
fi
bytes=0
while IFS=' ' read -r -d '' size path; do
debug "old: $path"
run rm -f -- "$path"
bytes=$((bytes + size))
done < <(list_old "$real")
if (( ! DRY_RUN )); then
find -P "$real" -xdev -mindepth 1 -type d -empty -delete
fi
info "$real: $count files, $(numfmt --to=iec "$bytes")B older than $DAYS days"
total_count=$((total_count + count))
total_bytes=$((total_bytes + bytes))
done
if (( DRY_RUN )); then
info "dry run: would delete $total_count files ($(numfmt --to=iec "$total_bytes")B); rerun with -y to delete"
else
info "deleted $total_count files, freed $(numfmt --to=iec "$total_bytes")B"
fi
}
main "$@"The safety rails, one by one:
realpath -ebefore any check. Comparing the string the user typed is useless:/etc/../usris not equal to/usr, but it is/usr. Every check runs on the resolved path.- A deny-list plus a depth rule. System directories and
$HOMEare refused by name, and/*/*rejects anything at the top level (/tmp,/data), so the cleanup always targets a specific subdirectory. find -P … -xdev -type f: never follow symlinks (-Pis the default, written out so nobody "fixes" it to-L), never descend into another mounted filesystem, and only ever delete regular files.- The count guard. The number of matches is computed first; if it exceeds
-m, the run stops before deleting anything. -mtime +$((DAYS - 1)).find -mtime +14means "more than 14 full days", i.e. 15 or more, because the age is rounded down to whole days first. Subtracting one gives "at leastDAYSdays old", which is what people expect from-d 14.- NUL-separated records (
-printf '%s %p\0',read -d '') survive any file name;IFS=' ' read -r size pathsplits only at the first space, so a path containing spaces stays intact.
Test run
A scratch area with old exports, a render cache whose name contains a space, old and new
thumbnails, a .keep marker file, and a symlink pointing at a directory of precious
config that is also old:
find scratch precious -printf '%TY-%Tm-%Td %7s %p\n' | sort -k32026-11-10 4096 precious
2026-08-12 19 precious/app.env
2026-11-10 4096 scratch
2026-11-10 11 scratch/config-link
2026-11-10 4096 scratch/exports
2026-10-01 0 scratch/exports/.keep
2026-10-01 11 scratch/exports/orders-2026-10-01.csv
2026-10-01 11 scratch/exports/orders-2026-10-02.csv
2026-10-01 11 scratch/exports/orders-2026-10-03.csv
2026-11-08 11 scratch/exports/orders-2026-11-09.csv
2026-11-10 4096 scratch/render cache
2026-11-10 4096 scratch/render cache/2026-10
2026-10-20 122880 scratch/render cache/2026-10/page 1.html
2026-10-20 122880 scratch/render cache/2026-10/page 2.html
2026-10-20 122880 scratch/render cache/2026-10/page 3.html
2026-10-20 122880 scratch/render cache/2026-10/page 4.html
2026-11-10 4096 scratch/thumbs
2026-11-10 4096 scratch/thumbs/new
2026-11-08 30720 scratch/thumbs/new/p1.jpg
2026-11-08 30720 scratch/thumbs/new/p2.jpg
2026-11-10 4096 scratch/thumbs/old
2026-10-11 30720 scratch/thumbs/old/p1.jpg
2026-10-11 30720 scratch/thumbs/old/p2.jpg
2026-10-11 30720 scratch/thumbs/old/p3.jpg
2026-10-11 30720 scratch/thumbs/old/p4.jpg
2026-10-11 30720 scratch/thumbs/old/p5.jpg
2026-10-11 30720 scratch/thumbs/old/p6.jpgRunning without -y only reports (output trimmed to a few of the 13 lines):
./cleanup.sh -x '.keep' scratch; echo "exit: $?"2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: dry-run: rm -f -- /home/sid/ops/scratch/exports/orders-2026-10-01.csv
[…]
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: dry-run: rm -f -- /home/sid/ops/scratch/thumbs/old/p6.jpg
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: dry-run: rm -f -- /home/sid/ops/scratch/render\ cache/2026-10/page\ 3.html
[…]
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: /home/sid/ops/scratch: 13 files, 661KB older than 14 days
2026-11-10T01:30:40-0500 cleanup.sh[10046] INFO: dry run: would delete 13 files (661KB); rerun with -y to delete
exit: 0With a deliberately low limit, the guard stops the run:
./cleanup.sh -m 5 -x '.keep' scratch; echo "exit: $?"2026-11-10T01:30:40-0500 cleanup.sh[10106] ERROR: /home/sid/ops/scratch: 13 files match, more than the limit of 5; check -d, or use -f
exit: 1For real, then what is left, then a second run:
./cleanup.sh -y -x '.keep' -l cleanup.log scratch; echo "exit: $?"
find scratch precious | sort
./cleanup.sh -y -x '.keep' scratch; echo "exit: $?"2026-11-10T01:30:40-0500 cleanup.sh[10119] INFO: /home/sid/ops/scratch: 13 files, 661KB older than 14 days
2026-11-10T01:30:40-0500 cleanup.sh[10119] INFO: deleted 13 files, freed 661KB
exit: 0
precious
precious/app.env
scratch
scratch/config-link
scratch/exports
scratch/exports/.keep
scratch/exports/orders-2026-11-09.csv
scratch/thumbs
scratch/thumbs/new
scratch/thumbs/new/p1.jpg
scratch/thumbs/new/p2.jpg
2026-11-10T01:30:40-0500 cleanup.sh[10169] INFO: /home/sid/ops/scratch: 0 files, 0B older than 14 days
2026-11-10T01:30:40-0500 cleanup.sh[10169] INFO: deleted 0 files, freed 0B
exit: 0Old files are gone, the emptied render cache and thumbs/old directories are gone,
the recent files and the .keep marker stayed, and precious/app.env (90 days old,
reachable through the symlink) was never touched. The second run is a no-op. Now the
inputs that must be refused:
./cleanup.sh -d 0 scratch; echo "exit: $?"
./cleanup.sh /; echo "exit: $?"
./cleanup.sh ~/projects/..; echo "exit: $?"
./cleanup.sh /etc/../usr; echo "exit: $?"
./cleanup.sh /tmp; echo "exit: $?"
./cleanup.sh scratch/config-link; echo "exit: $?"2026-11-10T01:30:40-0500 cleanup.sh[10191] ERROR: -d needs a whole number of days >= 1, got '0'
Try 'cleanup.sh -h' for more information.
exit: 2
2026-11-10T01:30:40-0500 cleanup.sh[10196] ERROR: refusing to clean protected directory: / (/)
exit: 1
2026-11-10T01:30:40-0500 cleanup.sh[10204] ERROR: refusing to clean protected directory: /home/sid/projects/.. (/home/sid)
exit: 1
2026-11-10T01:30:40-0500 cleanup.sh[10212] ERROR: refusing to clean protected directory: /etc/../usr (/usr)
exit: 1
2026-11-10T01:30:40-0500 cleanup.sh[10220] ERROR: refusing to clean a top-level directory: /tmp
exit: 1
2026-11-10T01:30:40-0500 cleanup.sh[10228] ERROR: refusing to clean a symlink: scratch/config-link
exit: 1Scheduling it
Weekly, Sunday 04:00, after the backup has run:
0 4 * * 0 /opt/ops/cleanup.sh -y -d 14 -x '.keep' -l /var/log/ops/cleanup.log /srv/shop/tmp /var/cache/shop/thumbs 2>/dev/nullThe -y is the one place it is written out: a human running the script by hand gets the
dry run, and the scheduled job is the only thing that deletes.
Where you'll use this
These three are the core of a small ops toolkit, typically installed together in
/opt/ops/ (with lib/ next to them) and scheduled from one crontab or a set of
timers. Here is the full crontab I installed on the lab machine to check that cron
accepts every line, before removing it again:
MAILTO=""
# m h dom mon dow command
30 2 * * * /opt/ops/backup.sh -b shop -d /backups/shop -k 14 -x '*.log' -x 'shop/cache' -l /var/log/ops/backup.log /srv/shop /etc/nginx 2>/dev/null
0 6 * * * /opt/ops/log-report.sh -o /srv/reports/access-$(date +\%F -d yesterday).txt /var/log/nginx/access.log.1 /var/log/nginx/access.log 2>> /var/log/ops/log-report.log
0 4 * * 0 /opt/ops/cleanup.sh -y -d 14 -x '.keep' -l /var/log/ops/cleanup.log /srv/shop/tmp /var/cache/shop/thumbs 2>/dev/nullcrontab ops.crontab && crontab -l | grep -c ops && crontab -r
systemd-analyze calendar '*-*-* 02:30:00' '*-*-* 06:00:00' 'Sun *-*-* 04:00:00' | grep -E 'Normalized|Next'3
Normalized form: *-*-* 02:30:00
Next elapse: Tue 2026-11-10 02:30:00 EST
Normalized form: *-*-* 06:00:00
Next elapse: Tue 2026-11-10 06:00:00 EST
Normalized form: Sun *-*-* 04:00:00
Next elapse: Sun 2026-11-15 04:00:00 ESTThe second command shows the matching OnCalendar= values if you prefer timers. Before
trusting any of them in cron, run each once the way cron will, with
env -i PATH=/usr/bin:/bin HOME=$HOME ./backup.sh …; I did that for the backup script
and it behaved identically. Beyond this shop, the same patterns cover database dumps
(pg_dump | gzip instead of tar, same verify/rotate logic), reports from any
line-oriented log (application logs, mail logs, firewall logs), and cleanup of CI build
directories, Docker build caches or upload spools.
Common mistakes
exit inside $(...) only exits the subshell
My first version of cleanup.sh returned the resolved path from check_dir by printing
it, and called real=$(check_dir "$dir"). Refusing / then produced this:
2026-11-10T01:28:17-0500 cleanup.sh[3598] ERROR: refusing to clean protected directory: / (/)
2026-11-10T01:28:17-0500 cleanup.sh[3598] ERROR: line 92: 'real=$(check_dir "$dir")' exited with status 1
exit: 1Command substitution runs in a subshell, so die exited only the subshell. The parent
script survived, saw a failed assignment, and its ERR trap logged a second, confusing
error. It still stopped (thanks to set -e), but without set -e it would have carried
on with an empty $real. Functions that may call exit should return results through a
global variable (REAL_DIR here) or a printf -v name, not through $(...).
Exit status 141 when you pipe into head
With set -o pipefail, ./log-report.sh … | head -5 at first produced a screenful of
errors (trimmed here):
203.0.113.42 38
2026-11-10T01:27:00-0500 log-report.sh[2710] ERROR: line 60: 'gawk -v day="$1" -v label="$2" -v top="$3" '
# nginx writes [08/Nov/2026:13:55:36 -0500]; keep only the wanted day
index($4, "[" day ":") != 1 { next }
[…]
}'' exited with status 141
2026-11-10T01:27:00-0500 log-report.sh[2710] ERROR: line 125: 'report "$logday" "$DAY" "$TOP"' exited with status 141head exits after five lines, the next write by gawk gets SIGPIPE, and 141 is 128 + 13
(SIGPIPE). pipefail reports that as a failure. For a command whose output a reader may
legitimately cut short, treat 141 as success, as log-report.sh now does. The same
output is also why the library's ERR handler prints only the first line of
$BASH_COMMAND.
Absolute paths in tar
tar -czf /tmp/abs.tar.gz /home/sid/ops/etc-shop; tar -tzf /tmp/abs.tar.gz | head -2tar: Removing leading `/' from member names
home/sid/ops/etc-shop/
home/sid/ops/etc-shop/nginx-shop.confGNU tar strips the leading / so that extracting cannot overwrite system paths, warns on
stderr every time (which pollutes cron logs), and stores the whole directory chain.
Restoring then means digging through home/sid/…. Using -C parent name for each
source, as backup.sh does, gives short, relocatable member names and no warning.
Summary
- Shared helpers belong in one sourced library once you have more than one tool; locate
it with
$(dirname "$(realpath "$0")")and tell shellcheck with# shellcheck source=. - Backups: archive to a
.partialname, verify (gzip -t,tar -t), rename, write a checksum, and only then rotate. Usetar -C dir name. Test a restore withdiff -r. - Log reports: normalize URLs (
:id, no query strings), use percentiles not just averages, pin the locale for month names, read rotated logs withzcat -f, and treat an empty day as an error. - Cleanup: dry run by default, resolve paths with
realpath -ebefore checking them, refuse protected and top-level directories and symlinks,find -P -xdev -type f, a match-count guard, and NUL-separated file lists. exitinside$(...)exits only the subshell; return values through variables.- Under
pipefail, a pipe intoheadcan yield 141 (SIGPIPE); decide explicitly whether that counts as failure. - Make the ERR trap exit with your own documented failure code, not the failed command's status.
Exercises
Write verify-backups.sh DEST that checks every *.tar.gz in DEST against its
.sha256 file and with gzip -t, prints OK or FAILED per archive, and exits 1 if
any archive fails (or none were found). Test it by truncating a copy of a backup.
Show Solution
#!/usr/bin/env bash
# verify-backups.sh DEST - check every backup in DEST against its checksum
set -uo pipefail
dest=${1:?usage: verify-backups.sh DEST}
failed=0 checked=0
shopt -s nullglob
for sum in "$dest"/*.tar.gz.sha256; do
archive=${sum%.sha256}
checked=$((checked + 1))
if (cd "$dest" && sha256sum --quiet -c "${sum##*/}") && gzip -t "$archive"; then
echo "OK ${archive##*/}"
else
echo "FAILED ${archive##*/}"
failed=$((failed + 1))
fi
done
echo "$checked checked, $failed failed"
(( checked > 0 && failed == 0 ))./verify-backups.sh backups; echo "exit: $?"
truncate -s -100 backups/shop-20261110-013037.tar.gz
./verify-backups.sh backups; echo "exit: $?"OK shop-20261110-013034.tar.gz
OK shop-20261110-013036.tar.gz
OK shop-20261110-013037.tar.gz
3 checked, 0 failed
exit: 0
OK shop-20261110-013034.tar.gz
OK shop-20261110-013036.tar.gz
shop-20261110-013037.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
FAILED shop-20261110-013037.tar.gz
3 checked, 1 failed
exit: 1Bit rot, a disk problem or an interrupted copy to remote storage all show up here. Schedule it daily on the machine that holds the copies, and alert on exit status 1.
Traffic spikes hide inside hourly totals. Add a "busiest minutes" section to
log-report.sh that lists the five minutes of the day with the most requests.
Hint
The timestamp field $4 holds [08/Nov/2026:13:55:36; characters 14 to 18 are HH:MM.
The script already has a topn function.
Show Solution
Two lines: count in the main rule, print in END.
hour[substr($4, 14, 2)]++
+ minute[substr($4, 14, 5)]++
[…]
if (err5) topn(errors, "endpoints returning 5xx", top)
+ topn(minute, "busiest minutes", 5)./ex2-log-report.sh -d 2026-11-08 logs/access.log.gz | sed -n '/busiest minutes/,/^$/p'busiest minutes
11:26 17
10:15 14
19:16 14
19:51 14
09:24 13
topn sorts numerically on the count and breaks ties alphabetically, which is why
10:15 comes before 19:16. On real traffic, compare the busiest minute with the hourly
average: 17 requests in a minute against roughly 7 per minute in that hour is a burst
worth knowing about when you size the server.
As the process management article showed,
deleting a file that a process still has open frees no space and can break the process.
Make cleanup.sh skip (with a warning) any old file that some process has open, and
make the summary count only files actually deleted. Test it with a background process
holding one of three old files open.
Show Solution
The loop gets a check before run rm, and lsof joins require_commands:
while IFS=' ' read -r -d '' size path; do
# deleting a file a process still has open frees no space (see lsof +L1)
if lsof -t -- "$path" > /dev/null 2>&1; then
warn "skipping open file: $path"
count=$((count - 1))
continue
fi
debug "old: $path"
run rm -f -- "$path"
bytes=$((bytes + size))
done < <(list_old "$real")The test: three 20-day-old temp files, one held open by a background subshell with
exec 3< scratch2/worker-tmp/job-2.tmp; sleep 5.
./ex3-cleanup.sh -y scratch2; echo "exit: $?"
ls scratch2/worker-tmp2026-11-10T01:34:34-0500 ex3-cleanup.sh[15879] WARN: skipping open file: /home/sid/ops/scratch2/worker-tmp/job-2.tmp
2026-11-10T01:34:34-0500 ex3-cleanup.sh[15879] INFO: /home/sid/ops/scratch2: 2 files, 12B older than 14 days
2026-11-10T01:34:34-0500 ex3-cleanup.sh[15879] INFO: deleted 2 files, freed 12B
exit: 0job-2.tmplsof -t FILE exits 0 when at least one process has the file open. Two caveats: without
root, lsof only sees your own processes, so run the cleanup as the user that owns the
files (or as root); and calling lsof once per file is slow for tens of thousands of
files. For large runs, collect the open paths once with lsof -Fn +D DIR and look them
up in an associative array.
Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), GNU tar 1.35, GNU Awk 5.2.1,
GNU coreutils 9.4, ShellCheck 0.9.0, November 2026. All three scripts and the exercises
were run as shown against generated test data (zero-filled stand-in files and a
deterministic synthetic access log). The crontab was installed only to confirm cron
accepts it and removed straight away; nothing ran against real /srv, /backups or
/var/log paths. Testing produced three of the fixes described above: the ERR handler's
first-line trimming and exit code, the SIGPIPE handling in the report, and check_dir
no longer running inside $(...). Paths from the test machine's scratch directories are shown shortened to /home/sid/….
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.