Capstone: A Deployment and Health-Check Toolkit
Build and run a complete Bash deployment toolkit: timestamped release directories, an atomic symlink switch, curl health checks with retries, and automatic rollback.
By The Dark DebuggerUpdated 35 min read
This is the capstone of the Bash track. We build a small but complete deployment toolkit: each release goes into its own timestamped directory, a symlink switch makes it live atomically, a health check with retries and timeouts decides whether it worked, and a wrapper rolls back automatically when it did not. Everything is run end to end, including a deliberately broken release. It uses the shared library and conventions from real-world scripts, the process control from processes, signals, and job control, and the template from writing production-ready scripts.
Why release directories
The simplest deployment is copying new files over old ones: rsync -a build/ /srv/shop/,
then restart. It works until it doesn't:
- For the seconds (or minutes, on a slow link) that the copy takes, the server runs a mix of old and new files. A request that loads a new template with an old library fails.
- If the copy dies halfway, the application is in a state that never existed anywhere, not in git and not in any build.
- Rolling back means finding and re-copying the previous build, under pressure, while the site is broken.
The release-directory layout, used by Capistrano, Deployer and many in-house tools, fixes
all three. Each deploy unpacks into a new directory, current is a symlink to the live
one, and going live (or going back) is a single atomic rename of that symlink. The
previous release is still on disk, so rollback takes a second.
The layout
After two deploys, the application root looks like this:
srv/shop
├── .deploy.lock
├── .previous
├── .release.lock
├── current -> releases/20261110T152904Z
├── deploy-history.log
├── releases
│ ├── 20261110T152902Z
│ │ ├── VERSION
│ │ ├── config -> ../../shared/config
│ │ ├── logs -> ../../shared/logs
│ │ └── public
│ │ ├── healthz
│ │ └── index.html
│ └── 20261110T152904Z
│ ├── VERSION
│ ├── config -> ../../shared/config
│ ├── logs -> ../../shared/logs
│ └── public
│ ├── healthz
│ └── index.html
├── run
│ └── app.pid
└── shared
├── config
└── logs
└── app.logreleases/<UTC timestamp>/: one directory per deploy, never modified after it is created. UTC timestamps sort chronologically and do not jump at daylight-saving changes.shared/: what must survive deploys, such as configuration with secrets, logs and user uploads. Each release links to it.current: a relative symlink to the live release. The app always runs fromcurrent. Relative, so the whole tree can be moved or restored from a backup..previous: the release that was live before the last deploy, the rollback target.deploy-history.log: an append-only record of every switch.
The toolkit
Four scripts, all built on lib/common.sh from the
previous article (logging, die, locking,
temp cleanup and an ERR trap that exits with 1), plus a build helper:
| Script | Job |
|---|---|
build-artifact.sh | Package the site as shop-VERSION.tar.gz plus a .sha256 file (stands in for your CI build) |
deploy.sh | verify, deploy, rollback and list releases |
appctl.sh | Start, stop and restart the app from current |
healthcheck.sh | Poll a URL until it is healthy, with retries and timeouts |
release.sh | The wrapper: verify, deploy, restart, check, roll back on failure |
The application and its artifacts
The "application" is a static site served by python3 -m http.server. It stands in for
any long-running service: it reads its files when it starts, so it must be restarted to
serve a new release. The health endpoint is a file, public/healthz, containing the
version, so the health check can tell "the new release is up" from "the old process is
still answering".
#!/usr/bin/env bash
# build-artifact.sh VERSION [--broken] - package the shop site as artifacts/shop-VERSION.tar.gz
# --broken leaves out the health endpoint, to simulate a release that comes up broken.
set -euo pipefail
version=${1:?usage: build-artifact.sh VERSION [--broken]}
build=$(mktemp -d)
trap 'rm -rf "$build"' EXIT
cp -r site/. "$build/"
echo "$version" > "$build/VERSION"
sed -i "s/@VERSION@/$version/" "$build/public/index.html"
if [[ ${2:-} != --broken ]]; then
echo "ok version=$version" > "$build/public/healthz"
fi
mkdir -p artifacts
tar -czf "artifacts/shop-$version.tar.gz" -C "$build" .
(cd artifacts && sha256sum "shop-$version.tar.gz" > "shop-$version.tar.gz.sha256")
echo "built artifacts/shop-$version.tar.gz"./build-artifact.sh 1.0.0 && ./build-artifact.sh 1.1.0 && ./build-artifact.sh 1.2.0 --broken
tar -tzf artifacts/shop-1.1.0.tar.gz; tar -tzf artifacts/shop-1.2.0.tar.gzbuilt artifacts/shop-1.0.0.tar.gz
built artifacts/shop-1.1.0.tar.gz
built artifacts/shop-1.2.0.tar.gz
./
./public/
./public/healthz
./public/index.html
./VERSION
./
./public/
./public/index.html
./VERSIONVersion 1.2.0 is the forced failure: a build that lost its health endpoint, standing in for any release that starts but does not work (a missing environment variable, a failed migration, a typo in a config file).
deploy.sh: releases, the atomic switch, and rollback
#!/usr/bin/env bash
#
# deploy.sh - release-directory deployments with atomic switch and rollback
#
# Usage: deploy.sh [-v] [-k KEEP] [-r APP_ROOT] deploy ARTIFACT.tar.gz
# deploy.sh [-v] [-r APP_ROOT] rollback
# deploy.sh [-r APP_ROOT] list
# deploy.sh verify ARTIFACT.tar.gz
#
# Layout under APP_ROOT:
# releases/<UTC timestamp>/ one directory per deploy
# shared/ files that survive deploys (config, logs)
# current -> releases/<id> what the app runs
# Exit codes: 0 ok, 1 failure, 2 usage error, 4 missing dependency, 5 locked
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
APP_ROOT=${APP_ROOT:-/srv/shop}
KEEP=5
usage() {
cat <<EOF
Usage: $SCRIPT_NAME [options] deploy ARTIFACT.tar.gz
$SCRIPT_NAME [options] rollback
$SCRIPT_NAME [options] list
$SCRIPT_NAME verify ARTIFACT.tar.gz
Options:
-r DIR application root (default: \$APP_ROOT or /srv/shop; now $APP_ROOT)
-k N releases to keep after a deploy (default: $KEEP)
-v verbose
-h show this help
EOF
}
# current_release: print the release id that "current" points to (empty if none)
current_release() {
local target
target=$(readlink "$APP_ROOT/current" 2>/dev/null) || return 0
printf '%s\n' "${target##*/}"
}
# releases: print release ids, oldest first
releases() {
[[ -d $APP_ROOT/releases ]] || return 0
find "$APP_ROOT/releases" -mindepth 1 -maxdepth 1 -type d \
-regextype posix-extended -regex '.*/[0-9]{8}T[0-9]{6}Z' -printf '%f\n' | sort
}
# switch_to ID: point "current" at releases/ID atomically
switch_to() {
local id=$1
[[ -d $APP_ROOT/releases/$id ]] || die 1 "no such release: $id"
ln -sfn "releases/$id" "$APP_ROOT/current.new"
mv -T "$APP_ROOT/current.new" "$APP_ROOT/current" # rename(2): atomic
echo "$(date -u +%FT%TZ) $1 $(<"$APP_ROOT/releases/$id/VERSION") ${USER:-?}" \
>> "$APP_ROOT/deploy-history.log"
}
verify_artifact() {
local artifact=$1 listing
[[ -f $artifact ]] || die 1 "artifact not found: $artifact"
if [[ -f $artifact.sha256 ]]; then
(cd "$(dirname "$artifact")" && sha256sum --quiet -c "${artifact##*/}.sha256") ||
die 1 "checksum mismatch: $artifact"
fi
listing=$(tar -tzf "$artifact") || die 1 "not a readable tar.gz: $artifact"
grep -qx './VERSION' <<< "$listing" || die 1 "artifact has no VERSION file"
grep -q '^\./public/' <<< "$listing" || die 1 "artifact has no public/ directory"
}
# previous_release: print the release that was live before the last deploy
previous_release() {
if [[ -f $APP_ROOT/.previous ]]; then cat "$APP_ROOT/.previous"; fi
}
prune() {
local current previous all=() id i
current=$(current_release)
previous=$(previous_release)
mapfile -t all < <(releases)
# newest KEEP stay; never delete the current release or the rollback target
for ((i = 0; i < ${#all[@]} - KEEP; i++)); do
id=${all[i]}
[[ $id != "$current" && $id != "$previous" ]] || continue
info "pruning old release $id"
rm -rf -- "${APP_ROOT:?}/releases/$id"
done
}
cmd_deploy() {
local artifact=${1:?usage: deploy ARTIFACT.tar.gz} id staging version
verify_artifact "$artifact"
mkdir -p "$APP_ROOT/releases" "$APP_ROOT/shared/logs" "$APP_ROOT/shared/config"
id=$(date -u +%Y%m%dT%H%M%SZ)
[[ ! -e $APP_ROOT/releases/$id ]] || die 1 "release $id already exists; deploys are once per second at most"
staging="$APP_ROOT/releases/.staging-$id"
register_tmp "$staging"
mkdir -p "$staging"
tar -xzf "$artifact" -C "$staging"
ln -s ../../shared/logs "$staging/logs"
ln -s ../../shared/config "$staging/config"
mv -T "$staging" "$APP_ROOT/releases/$id"
version=$(<"$APP_ROOT/releases/$id/VERSION")
local previous
previous=$(current_release)
if [[ -n $previous ]]; then
printf '%s\n' "$previous" > "$APP_ROOT/.previous" # the rollback target
fi
switch_to "$id"
info "deployed $version as release $id (previous: ${previous:-none})"
prune
}
cmd_rollback() {
local current previous id
current=$(current_release)
[[ -n $current ]] || die 1 "nothing is deployed"
# prefer the release that was live before the last deploy; fall back to
# the newest release older than current (e.g. for a second rollback)
previous=$(previous_release)
if [[ -z $previous || $previous == "$current" || ! -d $APP_ROOT/releases/$previous ]]; then
previous=""
while read -r id; do
if [[ $id < "$current" ]]; then previous=$id; fi
done < <(releases)
fi
[[ -n $previous ]] || die 1 "no release older than $current to roll back to"
switch_to "$previous"
rm -f "$APP_ROOT/.previous"
info "rolled back from $current ($(<"$APP_ROOT/releases/$current/VERSION")) to $previous ($(<"$APP_ROOT/releases/$previous/VERSION"))"
# a rolled-back release must never become a rollback target later
rm -rf -- "${APP_ROOT:?}/releases/$current"
}
cmd_list() {
local current id mark
current=$(current_release)
while read -r id; do
mark=" "; [[ $id == "$current" ]] && mark="*"
printf '%s %s %s\n' "$mark" "$id" "$(<"$APP_ROOT/releases/$id/VERSION")"
done < <(releases)
}
main() {
local opt
while getopts ':r:k:vh' opt; do
case $opt in
r) APP_ROOT=$OPTARG ;;
k) KEEP=$OPTARG ;;
v) VERBOSE=1 ;;
h) usage; exit 0 ;;
:) usage_error "option -$OPTARG requires an argument" ;;
\?) usage_error "unknown option: -$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
[[ $KEEP =~ ^[2-9]$|^[1-9][0-9]+$ ]] || usage_error "-k must be at least 2 (current + one to roll back to)"
require_commands tar sha256sum flock find readlink
APP_ROOT=$(realpath -m -- "$APP_ROOT")
case ${1:-} in
deploy) mkdir -p "$APP_ROOT"; acquire_lock "$APP_ROOT/.deploy.lock"; cmd_deploy "${2:-}" ;;
rollback) [[ -d $APP_ROOT/releases ]] || die 1 "nothing is deployed in $APP_ROOT"
acquire_lock "$APP_ROOT/.deploy.lock"; cmd_rollback ;;
list) cmd_list ;;
verify) verify_artifact "${2:?usage: verify ARTIFACT.tar.gz}"; info "${2##*/} looks good" ;;
*) usage_error "expected a command: deploy, rollback, list or verify" ;;
esac
}
main "$@"The parts that make it safe:
- Unpack into
.staging-ID, then rename intoreleases/ID. A deploy that dies mid-extraction leaves a staging directory (removed by the EXIT trap), never a half-populated release thatlistorrollbackcould pick. - Shared directories are linked in before the switch, so the new release is complete the moment it goes live.
- The switch is a rename.
switch_tocreates the new link ascurrent.newand thenmv -T current.new current, which is therename(2)system call. The kernel guarantees that a rename over an existing name is atomic: every process openingcurrent/…sees either the old target or the new one, never "no such file".-Tstopsmvfrom treatingcurrent(a link to a directory) as a directory to move into. - Rollback goes to the recorded previous release, not just "the next older directory". Why that matters is in common mistakes.
- Pruning never deletes
currentor the rollback target, and-kmust be at least 2, so there is always somewhere to roll back to. - The history log records every switch with time, release, version and user, which is the first thing you want during an incident ("what changed at 14:02?").
A note on ln -sfn: on the lab's GNU coreutils 9.4, ln -sfn itself already creates the
link under a temporary name and renames it into place (the ln binary imports
renameat). Older coreutils and other implementations such as BusyBox and the BSD ln
on macOS remove the old link first and then create the new one, leaving a brief moment
with no current at all. Writing the rename explicitly makes the script atomic no matter
which ln the server has.
appctl.sh: restarting the app from current
This is the appctl from process management,
adapted to the release layout and the shared library:
#!/usr/bin/env bash
#
# appctl.sh - start, stop and restart the shop app from APP_ROOT/current
#
# Usage: appctl.sh [-r APP_ROOT] [-p PORT] {start|stop|restart|status}
# The app is a static site served by python3 -m http.server, standing in for
# any long-running service that must be restarted to pick up a new release.
# Exit codes: 0 ok, 1 failure, 2 usage error, 3 not running (status)
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
APP_ROOT=${APP_ROOT:-/srv/shop}
PORT=${PORT:-18480}
pid_of_app() {
local pid
[[ -f $APP_ROOT/run/app.pid ]] || return 1
pid=$(<"$APP_ROOT/run/app.pid")
kill -0 "$pid" 2>/dev/null || return 1
tr '\0' ' ' < "/proc/$pid/cmdline" | grep -q "http.server $PORT" || return 1
printf '%s\n' "$pid"
}
start() {
local pid i
if pid=$(pid_of_app); then info "already running (PID $pid)"; return 0; fi
[[ -d $APP_ROOT/current/public ]] || die 1 "nothing deployed in $APP_ROOT/current"
mkdir -p "$APP_ROOT/run"
# cd resolves the symlink: the process keeps serving this release until restarted
(cd "$APP_ROOT/current" &&
exec nohup python3 -m http.server "$PORT" --bind 127.0.0.1 --directory public \
>> "$APP_ROOT/shared/logs/app.log" 2>&1) &
pid=$!
echo "$pid" > "$APP_ROOT/run/app.pid"
for ((i = 0; i < 50; i++)); do
kill -0 "$pid" 2>/dev/null || break
if [[ $(lsof -t -nP -iTCP:"$PORT" -sTCP:LISTEN 2>/dev/null) == "$pid" ]]; then
info "started PID $pid on port $PORT, serving $(readlink "$APP_ROOT/current")"
return 0
fi
sleep 0.1
done
rm -f "$APP_ROOT/run/app.pid"
die 1 "app failed to start; last log line: $(tail -n 1 "$APP_ROOT/shared/logs/app.log")"
}
stop() {
local pid i
if ! pid=$(pid_of_app); then info "not running"; rm -f "$APP_ROOT/run/app.pid"; return 0; fi
kill -TERM "$pid"
for ((i = 0; i < 100; i++)); do
if ! kill -0 "$pid" 2>/dev/null; then
info "stopped PID $pid"
rm -f "$APP_ROOT/run/app.pid"
return 0
fi
sleep 0.1
done
warn "PID $pid ignored SIGTERM for 10s, sending SIGKILL"
kill -KILL "$pid"
rm -f "$APP_ROOT/run/app.pid"
}
main() {
local opt pid
while getopts ':r:p:h' opt; do
case $opt in
r) APP_ROOT=$OPTARG ;;
p) PORT=$OPTARG ;;
h) echo "Usage: $SCRIPT_NAME [-r APP_ROOT] [-p PORT] {start|stop|restart|status}"; exit 0 ;;
*) usage_error "bad option" ;;
esac
done
shift $((OPTIND - 1))
require_commands python3 lsof
case ${1:-} in
start) start ;;
stop) stop ;;
restart) stop; start ;;
status)
if pid=$(pid_of_app); then
echo "running (PID $pid), serving $(readlink "$APP_ROOT/current")"
else
echo "stopped"; exit 3
fi ;;
*) usage_error "expected start, stop, restart or status" ;;
esac
}
main "$@"The line worth understanding is cd "$APP_ROOT/current". Changing into a symlinked
directory gives the process the real directory as its working directory. After the
next deploy flips current, the running process still serves its own release, which is
exactly how real applications behave: code is loaded at start-up. That is why a deploy
alone changes nothing until the restart, and why the health check must verify the
version. exec replaces the subshell with nohup, which replaces itself with python3,
so $! is the server's PID.
On a server with systemd you would replace appctl.sh with a unit
(WorkingDirectory=/srv/shop/current, ExecStart=…) and systemctl restart shop;
the rest of the toolkit stays the same.
healthcheck.sh: retries and timeouts
#!/usr/bin/env bash
#
# healthcheck.sh - poll an HTTP endpoint until it is healthy or retries run out
#
# Usage: healthcheck.sh [-v] [-e TEXT] [-r RETRIES] [-t TIMEOUT] [-w WAIT] URL
#
# Healthy means: HTTP 2xx within TIMEOUT seconds, and the body contains TEXT
# (if -e is given). Exit codes: 0 healthy, 1 unhealthy, 2 usage error,
# 4 missing dependency
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
EXPECT="" RETRIES=5 TIMEOUT=3 WAIT=2
usage() {
cat <<EOF
Usage: $SCRIPT_NAME [options] URL
Options:
-e TEXT the response body must contain TEXT
-r N attempts before giving up (default: $RETRIES)
-t SEC timeout per attempt, connect + response (default: $TIMEOUT)
-w SEC pause between attempts (default: $WAIT)
-v verbose
-h show this help
EOF
}
main() {
local opt url i out
while getopts ':e:r:t:w:vh' opt; do
case $opt in
e) EXPECT=$OPTARG ;;
r) RETRIES=$OPTARG ;;
t) TIMEOUT=$OPTARG ;;
w) WAIT=$OPTARG ;;
v) VERBOSE=1 ;;
h) usage; exit 0 ;;
:) usage_error "option -$OPTARG requires an argument" ;;
\?) usage_error "unknown option: -$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
url=${1:-}
[[ $url =~ ^https?:// ]] || usage_error "expected an http(s) URL, got '$url'"
[[ $RETRIES =~ ^[1-9][0-9]*$ ]] || usage_error "-r needs a positive number"
require_commands curl
for ((i = 1; i <= RETRIES; i++)); do
# -f: HTTP errors are failures; -sS: no progress bar, but do show errors
if out=$(curl -fsS --max-time "$TIMEOUT" --connect-timeout "$TIMEOUT" "$url" 2>&1); then
if [[ -z $EXPECT || $out == *"$EXPECT"* ]]; then
info "healthy after $i attempt(s): ${out:0:80}"
return 0
fi
warn "attempt $i/$RETRIES: body does not contain '$EXPECT': ${out:0:80}"
else
warn "attempt $i/$RETRIES: ${out:0:120}"
fi
if (( i < RETRIES )); then sleep "$WAIT"; fi
done
error "$url still unhealthy after $RETRIES attempts"
exit 1
}
main "$@"Every option matters in practice:
| curl option | Why |
|---|---|
-f (--fail) | Without it, a 404 or 500 page is a "successful" transfer and curl exits 0 |
-sS | Silent (no progress meter), but still print the error message |
--max-time | Upper bound for the whole request; a hung app must not hang the deploy |
--connect-timeout | Upper bound for connecting; without it, an address that silently drops packets waits for the kernel's TCP connect timeout, around two minutes on Linux |
The retry loop is in Bash rather than curl's own --retry, for reasons the
health check section shows with real output. The function
ends with exit 1 rather than return 1: main "$@" returning non-zero would trigger the
library's ERR trap and log a second, misleading error.
release.sh: the wrapper that rolls back
#!/usr/bin/env bash
#
# release.sh - deploy an artifact, restart the app, health-check it, and roll
# back automatically if the new release is not healthy.
#
# Usage: release.sh [-v] [-r APP_ROOT] [-p PORT] ARTIFACT.tar.gz
# Exit codes: 0 released, 1 release failed (not deployed, or rolled back),
# 2 usage error, 3 release failed AND rollback failed (page someone),
# 5 another release in progress
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
TOOLS=$(dirname "$(realpath "$0")")
# shellcheck source=lib/common.sh
source "$TOOLS/lib/common.sh"
export APP_ROOT=${APP_ROOT:-/srv/shop}
export PORT=${PORT:-18480}
main() {
local opt artifact version previous
while getopts ':r:p:vh' opt; do
case $opt in
r) APP_ROOT=$OPTARG ;;
p) PORT=$OPTARG ;;
v) VERBOSE=1 ;;
h) echo "Usage: $SCRIPT_NAME [-v] [-r APP_ROOT] [-p PORT] ARTIFACT.tar.gz"; exit 0 ;;
:) usage_error "option -$OPTARG requires an argument" ;;
\?) usage_error "unknown option: -$OPTARG" ;;
esac
done
shift $((OPTIND - 1))
artifact=${1:-}
[[ -n $artifact ]] || usage_error "no artifact given"
require_commands tar curl
mkdir -p "$APP_ROOT"
acquire_lock "$APP_ROOT/.release.lock"
"$TOOLS/deploy.sh" verify "$artifact" # fail fast, before anything changes
version=$(tar -xzOf "$artifact" ./VERSION)
previous=$("$TOOLS/deploy.sh" list | awk '$1 == "*" { print $3 }')
info "releasing $version (currently running: ${previous:-nothing})"
# {LOCK_FD}>&- : the app must not inherit our lock, or it would hold it forever
"$TOOLS/deploy.sh" deploy "$artifact"
"$TOOLS/appctl.sh" restart {LOCK_FD}>&-
if "$TOOLS/healthcheck.sh" -e "version=$version" -r 4 -w 1 -t 2 \
"http://127.0.0.1:$PORT/healthz"; then
info "release $version is live"
exit 0
fi
error "release $version failed its health check, rolling back"
if "$TOOLS/deploy.sh" rollback &&
"$TOOLS/appctl.sh" restart {LOCK_FD}>&- &&
"$TOOLS/healthcheck.sh" -e "version=$previous" -r 4 -w 1 -t 2 \
"http://127.0.0.1:$PORT/healthz"; then
error "rolled back to $previous; $version was NOT released"
exit 1
fi
error "rollback to ${previous:-nothing} failed too; the site may be down"
exit 3
}
main "$@"The flow is: verify the artifact before touching anything, deploy, restart, check that the
new version answers, and otherwise roll back, restart, and check that the old
version answers again. The exit codes distinguish "it failed but we are safely back"
(1) from "it failed and so did the rollback" (3), which is the one that should wake
someone up. The {LOCK_FD}>&- on the appctl.sh lines closes the release lock's file
descriptor for that command only; the common mistakes show what
happens without it.
All the scripts pass shellcheck (run from the toolkit directory, so the
# shellcheck source=lib/common.sh paths resolve):
(cd toolkit && shellcheck -x ./*.sh) && shellcheck build-artifact.sh && echo cleancleanRunning it end to end
Everything below ran in one session against a fresh application root, with
APP_ROOT pointing at srv/shop in the working directory and the app on port 18480.
Two good releases
toolkit/release.sh artifacts/shop-1.0.0.tar.gz; echo "exit: $?"
toolkit/release.sh artifacts/shop-1.1.0.tar.gz; echo "exit: $?"
curl -s http://127.0.0.1:18480/2026-11-10T10:29:02-0500 deploy.sh[13601] INFO: shop-1.0.0.tar.gz looks good
2026-11-10T10:29:02-0500 release.sh[13596] INFO: releasing 1.0.0 (currently running: nothing)
2026-11-10T10:29:02-0500 deploy.sh[13633] INFO: deployed 1.0.0 as release 20261110T152902Z (previous: none)
2026-11-10T10:29:02-0500 appctl.sh[13674] INFO: not running
2026-11-10T10:29:02-0500 appctl.sh[13674] INFO: started PID 13683 on port 18480, serving releases/20261110T152902Z
2026-11-10T10:29:02-0500 healthcheck.sh[13693] INFO: healthy after 1 attempt(s): ok version=1.0.0
2026-11-10T10:29:02-0500 release.sh[13596] INFO: release 1.0.0 is live
exit: 0
2026-11-10T10:29:04-0500 deploy.sh[13709] INFO: shop-1.1.0.tar.gz looks good
2026-11-10T10:29:04-0500 release.sh[13704] INFO: releasing 1.1.0 (currently running: 1.0.0)
2026-11-10T10:29:04-0500 deploy.sh[13743] INFO: deployed 1.1.0 as release 20261110T152904Z (previous: 20261110T152902Z)
2026-11-10T10:29:04-0500 appctl.sh[13785] INFO: stopped PID 13683
2026-11-10T10:29:04-0500 appctl.sh[13785] INFO: started PID 13798 on port 18480, serving releases/20261110T152904Z
2026-11-10T10:29:04-0500 healthcheck.sh[13809] INFO: healthy after 1 attempt(s): ok version=1.1.0
2026-11-10T10:29:04-0500 release.sh[13704] INFO: release 1.1.0 is live
exit: 0
<!doctype html>
<title>Northwind Outdoor Shop</title>
<h1>Northwind Outdoor Shop</h1>
<p>Release 1.1.0</p>The tree at this point is the one in the layout above, and .previous
contains 20261110T152902Z, the 1.0.0 release.
The forced failure
Now the broken 1.2.0, timed:
time toolkit/release.sh artifacts/shop-1.2.0.tar.gz; echo "exit: $?"2026-11-10T10:29:05-0500 deploy.sh[13835] INFO: shop-1.2.0.tar.gz looks good
2026-11-10T10:29:05-0500 release.sh[13827] INFO: releasing 1.2.0 (currently running: 1.1.0)
2026-11-10T10:29:05-0500 deploy.sh[13869] INFO: deployed 1.2.0 as release 20261110T152905Z (previous: 20261110T152904Z)
2026-11-10T10:29:05-0500 appctl.sh[13911] INFO: stopped PID 13798
2026-11-10T10:29:05-0500 appctl.sh[13911] INFO: started PID 13924 on port 18480, serving releases/20261110T152905Z
2026-11-10T10:29:05-0500 healthcheck.sh[13952] WARN: attempt 1/4: curl: (22) The requested URL returned error: 404
2026-11-10T10:29:06-0500 healthcheck.sh[13952] WARN: attempt 2/4: curl: (22) The requested URL returned error: 404
2026-11-10T10:29:07-0500 healthcheck.sh[13952] WARN: attempt 3/4: curl: (22) The requested URL returned error: 404
2026-11-10T10:29:08-0500 healthcheck.sh[13952] WARN: attempt 4/4: curl: (22) The requested URL returned error: 404
2026-11-10T10:29:08-0500 healthcheck.sh[13952] ERROR: http://127.0.0.1:18480/healthz still unhealthy after 4 attempts
2026-11-10T10:29:08-0500 release.sh[13827] ERROR: release 1.2.0 failed its health check, rolling back
2026-11-10T10:29:09-0500 deploy.sh[14039] INFO: rolled back from 20261110T152905Z (1.2.0) to 20261110T152904Z (1.1.0)
2026-11-10T10:29:09-0500 appctl.sh[14058] INFO: stopped PID 13924
2026-11-10T10:29:09-0500 appctl.sh[14058] INFO: started PID 14070 on port 18480, serving releases/20261110T152904Z
2026-11-10T10:29:09-0500 healthcheck.sh[14080] INFO: healthy after 1 attempt(s): ok version=1.1.0
2026-11-10T10:29:09-0500 release.sh[13827] ERROR: rolled back to 1.1.0; 1.2.0 was NOT released
real 0m3.796s
user 0m0.144s
sys 0m0.142s
exit: 1toolkit/deploy.sh list; curl -s http://127.0.0.1:18480/healthz
cat srv/shop/deploy-history.log 20261110T152902Z 1.0.0
* 20261110T152904Z 1.1.0
ok version=1.1.0
2026-11-10T15:29:02Z 20261110T152902Z 1.0.0 sid
2026-11-10T15:29:04Z 20261110T152904Z 1.1.0 sid
2026-11-10T15:29:05Z 20261110T152905Z 1.2.0 sid
2026-11-10T15:29:09Z 20261110T152904Z 1.1.0 sidFrom "deploy 1.2.0" to "1.1.0 healthy again" took under four seconds, three of which were the deliberate one-second pauses between health attempts. The broken release directory is gone, so it can never be picked as a rollback target later, but the history log still records that it was live from 15:29:05 to 15:29:09 UTC. That is the line you need when someone asks why a few requests failed at that minute. The exit status 1 is what a CI job uses to mark the pipeline as failed.
A corrupted artifact never gets deployed
An artifact truncated in transit (half the file, with the checksum of the full one):
head -c $(( $(stat -c %s artifacts/shop-1.1.0.tar.gz) / 2 )) artifacts/shop-1.1.0.tar.gz > artifacts/shop-1.1.1.tar.gz && sed 's/1.1.0/1.1.1/' artifacts/shop-1.1.0.tar.gz.sha256 > artifacts/shop-1.1.1.tar.gz.sha256
toolkit/release.sh artifacts/shop-1.1.1.tar.gz; echo "exit: $?"
toolkit/deploy.sh listshop-1.1.1.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
2026-11-10T10:29:09-0500 deploy.sh[14112] ERROR: checksum mismatch: artifacts/shop-1.1.1.tar.gz
2026-11-10T10:29:09-0500 release.sh[14107] ERROR: line 39: '"$TOOLS/deploy.sh" verify "$artifact"' exited with status 1
exit: 1
20261110T152902Z 1.0.0
* 20261110T152904Z 1.1.0Nothing was unpacked, nothing restarted. While testing I first truncated the file to a
fixed 1,000 bytes, and the "corrupted" release went live: the artifact was smaller than
1,000 bytes, so head -c had copied it whole and the checksum matched. A test that does
not fail when you expect it to is telling you something about the test.
Manual rollback and pruning
deploy.sh is also the tool for a human. Rolling back by hand (when, for example, a bug
shows up an hour after a release) is two commands:
toolkit/deploy.sh rollback && toolkit/appctl.sh restart
toolkit/deploy.sh list; curl -s http://127.0.0.1:18480/healthz
toolkit/deploy.sh rollback; echo "exit: $?"2026-11-10T10:29:09-0500 deploy.sh[14137] INFO: rolled back from 20261110T152904Z (1.1.0) to 20261110T152902Z (1.0.0)
2026-11-10T10:29:09-0500 appctl.sh[14158] INFO: stopped PID 14070
2026-11-10T10:29:09-0500 appctl.sh[14158] INFO: started PID 14170 on port 18480, serving releases/20261110T152902Z
* 20261110T152902Z 1.0.0
ok version=1.0.0
2026-11-10T10:29:09-0500 deploy.sh[14193] ERROR: no release older than 20261110T152902Z to roll back to
exit: 1Then three deploys in a row with -k 2 (keep two releases), to watch pruning:
2026-11-10T10:29:11-0500 deploy.sh[14287] INFO: deployed 1.1.0 as release 20261110T152910Z (previous: 20261110T152902Z)
2026-11-10T10:29:12-0500 deploy.sh[14330] INFO: deployed 1.1.0 as release 20261110T152912Z (previous: 20261110T152910Z)
2026-11-10T10:29:12-0500 deploy.sh[14330] INFO: pruning old release 20261110T152902Z
2026-11-10T10:29:13-0500 deploy.sh[14376] INFO: deployed 1.1.0 as release 20261110T152913Z (previous: 20261110T152912Z)
2026-11-10T10:29:13-0500 deploy.sh[14376] INFO: pruning old release 20261110T152910Ztoolkit/deploy.sh list 20261110T152912Z 1.1.0
* 20261110T152913Z 1.1.0Health checks under failure
A health check is only as good as its behavior when things go wrong. Here is
healthcheck.sh against the three failure modes you will meet, starting with the old
version still answering (the app was not restarted, or the restart failed silently):
toolkit/healthcheck.sh -e version=1.2.0 -r 2 -w 1 -t 2 http://127.0.0.1:18480/healthz; echo "exit: $?"2026-11-10T10:29:15-0500 healthcheck.sh[14687] WARN: attempt 1/2: body does not contain 'version=1.2.0': ok version=1.1.0
2026-11-10T10:29:16-0500 healthcheck.sh[14687] WARN: attempt 2/2: body does not contain 'version=1.2.0': ok version=1.1.0
2026-11-10T10:29:16-0500 healthcheck.sh[14687] ERROR: http://127.0.0.1:18480/healthz still unhealthy after 2 attempts
exit: 1A plain "is it 200?" check would have passed here and declared the release a success.
Nothing listening on the port (the app crashed on start):
toolkit/healthcheck.sh -r 3 -w 1 -t 2 http://127.0.0.1:18489/healthz; echo "exit: $?"2026-11-10T10:29:16-0500 healthcheck.sh[14703] WARN: attempt 1/3: curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
2026-11-10T10:29:17-0500 healthcheck.sh[14703] WARN: attempt 2/3: curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
2026-11-10T10:29:18-0500 healthcheck.sh[14703] WARN: attempt 3/3: curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
2026-11-10T10:29:18-0500 healthcheck.sh[14703] ERROR: http://127.0.0.1:18489/healthz still unhealthy after 3 attempts
exit: 1And the nastiest one, a process that accepts connections but never answers (deadlocked,
or stuck on a database that is down). I simulated it with a Python socket that listens
on port 18481 and never calls accept:
time toolkit/healthcheck.sh -r 2 -w 1 -t 2 http://127.0.0.1:18481/healthz; echo "exit: $?"2026-11-10T10:29:21-0500 healthcheck.sh[14841] WARN: attempt 1/2: curl: (28) Operation timed out after 2015 milliseconds with 0 bytes received
2026-11-10T10:29:24-0500 healthcheck.sh[14841] WARN: attempt 2/2: curl: (28) Operation timed out after 2008 milliseconds with 0 bytes received
2026-11-10T10:29:24-0500 healthcheck.sh[14841] ERROR: http://127.0.0.1:18481/healthz still unhealthy after 2 attempts
real 0m5.040s
user 0m0.026s
sys 0m0.035s
exit: 1Two attempts of 2 s plus a 1 s pause: 5 s, as designed. Without --max-time, curl would
have waited as long as the server held the connection, and the deploy would have hung
with it.
Why not curl --retry?
curl has built-in retries, and it is tempting to use them. Here is what they actually do against a port with nothing listening and against a 404:
time curl -fsS --retry 3 --retry-delay 1 http://127.0.0.1:18489/healthz; echo "exit: $?"
time curl -fsS --retry 3 --retry-delay 1 --retry-connrefused http://127.0.0.1:18489/healthz; echo "exit: $?"curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
real 0m0.014s
user 0m0.009s
sys 0m0.005s
exit: 7
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
real 0m3.018s
user 0m0.009s
sys 0m0.006s
exit: 7time curl -fsS --retry 3 --retry-delay 1 http://127.0.0.1:18480/missing; echo "exit: $?"
time curl -fsS --retry 3 --retry-delay 1 --retry-all-errors http://127.0.0.1:18480/missing; echo "exit: $?"curl: (22) The requested URL returned error: 404
real 0m0.015s
user 0m0.007s
sys 0m0.007s
exit: 22
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
real 0m3.025s
user 0m0.010s
sys 0m0.005s
exit: 22Plain --retry gave up immediately in both cases. It only retries errors curl considers
transient (timeouts and HTTP 408, 429, 500, 502, 503 and 504). "Connection refused", which
is exactly what you get while an app is still starting, needs --retry-connrefused, and
anything else needs --retry-all-errors (curl 7.71 or later). Even then curl cannot
check the response body for the right version. A Bash loop around a single-attempt curl
is easier to reason about, logs every attempt, and can check whatever "healthy" means for
your app. (The 3-second runs are the three one-second retry delays.)
Where you'll use this
The toolkit maps directly onto how small teams deploy without a platform:
- From CI. The pipeline builds and tests the artifact, copies it and its
.sha256to the server, and runsrelease.shover SSH. The exit code decides the pipeline status: 0 green, 1 red ("rolled back, nothing to do but fix the build"), 3 red and page the on-call engineer. The Jenkins capstone builds exactly this kind of pipeline. - Several servers. Run
release.shon each server in turn behind a load balancer, and stop at the first failure. Because each server rolls itself back, a bad release never gets past the first one. - A real app. Replace
appctl.shwithsystemctl restart shop(the unit usesWorkingDirectory=/srv/shop/current), put secrets inshared/config, and point the health check at an endpoint that checks the database and other dependencies, not just "the process is up". The deploy, switch and rollback logic does not change. - Rollback during an incident.
deploy.sh rollback && appctl.sh restarttakes seconds and needs no build system, no network access to a registry, and no memory of what the previous version was. The history log answers "what changed and when".
Common mistakes
The app inherits your lock
To show why release.sh has {LOCK_FD}>&- on the lines that start the app, I ran a copy
of it with those two redirections removed. The first release worked. The second one, with
nothing else running, failed:
toolkit/release-nofdclose.sh artifacts/shop-1.1.0.tar.gz > /dev/null 2>&1; echo "exit: $?"
toolkit/release-nofdclose.sh artifacts/shop-1.1.0.tar.gz; echo "exit: $?"
lsof -nP srv/shop/.release.lockexit: 0
2026-11-10T10:29:15-0500 release-nofdclose.sh[14653] ERROR: another run is in progress (lock: /home/sid/capstone/srv/shop/.release.lock)
exit: 5
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python3 14630 sid 10w REG 8,48 0 54918 srv/shop/.release.lockChild processes inherit open file descriptors. The release script held the lock on a
descriptor, started appctl.sh, which started the server, and the server kept descriptor
10 (the lock) open for its whole life. The flock lock lives as long as any process
holds that open file, so every later release was locked out until the app restarted.
lsof on the lock file names the culprit. The fix is to close the descriptor for the
commands that start long-lived processes, "$TOOLS/appctl.sh" restart {LOCK_FD}>&-, and
after that the lock is free as soon as release.sh exits:
toolkit/appctl.sh restart && flock -n srv/shop/.release.lock true && echo 'lock is free again'2026-11-10T10:29:15-0500 appctl.sh[14663] INFO: stopped PID 14630
2026-11-10T10:29:15-0500 appctl.sh[14663] INFO: started PID 14675 on port 18480, serving releases/20261110T152914Z
lock is free againThe same applies to any script that takes a lock and then starts a daemon, and to other inherited descriptors such as sockets and log files.
Rolling back to "the next older directory"
My first cmd_rollback picked the newest release older than current, by sorting the
directory names. That is correct only if current is always the newest release. While
testing exercise 1 below (rolling back to a specific older release), current became the
oldest release, and the next failed release rolled back to the wrong one:
2026-11-10T10:27:59-0500 ex3-release.sh[12031] INFO: releasing 1.3.0 (currently running: 1.0.0)
[…]
2026-11-10T10:28:00-0500 deploy.sh[12146] INFO: rolled back from 20261110T152759Z (1.3.0) to 20261110T152756Z (1.1.0)
[…]
2026-11-10T10:28:00-0500 healthcheck.sh[12187] WARN: attempt 1/4: body does not contain 'version=1.0.0': ok version=1.1.0
[…]
2026-11-10T10:28:03-0500 ex3-release.sh[12031] ERROR: rollback to 1.0.0 failed too; the site may be down
exit: 31.0.0 was live, the deploy of 1.3.0 failed, and the "rollback" put 1.1.0 live, a version
that had been rolled back on purpose. The health check caught it only because it checks
the version. The fix is in the final deploy.sh: record the release that was live before
each deploy in .previous, roll back to that, and fall back to sort order only for a
second rollback in a row. The general lesson: "previous" is a fact about history, so
record it; don't reconstruct it from names.
Checking only that the port answers
A health check that accepts any HTTP 200 passes when the old process is still running, when a load balancer serves a cached page, or when the app answers with an error page that happens to return 200. Check something that only the new, working release can say: its version, and ideally the result of a real dependency check.
Summary
- Deploy each release into its own
releases/<timestamp>directory, unpacked under a staging name and renamed into place; keep config, logs and uploads inshared/and link them in before going live. - Go live by renaming a new symlink over
current(ln -sfn … current.new && mv -T current.new current):rename(2)is atomic, whicheverlnthe server has. - A running process keeps its old release (its working directory is the resolved path), so a deploy needs a restart, and the health check must verify the version.
- Health checks:
curl -fsS --max-time --connect-timeoutin a Bash retry loop that checks the body. curl's own--retryskips "connection refused" and 404 unless told otherwise. - Verify artifacts (checksum, structure) before changing anything; record the previous release for rollback; never prune the current release or the rollback target; log every switch.
- Close lock descriptors (
{LOCK_FD}>&-) for commands that start long-lived processes. - Distinct exit codes turn the wrapper into a CI step: 0 released, 1 failed and rolled back, 3 failed and could not recover.
Exercises
Extend deploy.sh rollback to accept an optional release id: deploy.sh rollback 20261110T152941Z. It must refuse an id that does not exist and the id that is already
current, and behave as before when no id is given.
Show Solution
The changes, as a diff:
- local current previous id
+ local target=${1:-} current previous id
current=$(current_release)
[[ -n $current ]] || die 1 "nothing is deployed"
- # prefer the release that was live before the last deploy; fall back to
- # the newest release older than current (e.g. for a second rollback)
- previous=$(previous_release)
+ if [[ -n $target ]]; then
+ [[ -d $APP_ROOT/releases/$target ]] || die 1 "no such release: $target"
+ [[ $target != "$current" ]] || die 1 "$target is already the current release"
+ fi
+ # an explicit target wins; then the release that was live before the last
+ # deploy; then the newest release older than current
+ previous=${target:-$(previous_release)}
[…]
- acquire_lock "$APP_ROOT/.deploy.lock"; cmd_rollback ;;
+ acquire_lock "$APP_ROOT/.deploy.lock"; cmd_rollback "${2:-}" ;;With three releases deployed:
toolkit/ex1-deploy.sh list
toolkit/ex1-deploy.sh rollback 20991231T000000Z; echo "exit: $?"
toolkit/ex1-deploy.sh rollback 20261110T152944Z; echo "exit: $?"
toolkit/ex1-deploy.sh rollback 20261110T152941Z; echo "exit: $?"
toolkit/ex1-deploy.sh list 20261110T152941Z 1.0.0
20261110T152942Z 1.1.0
* 20261110T152944Z 1.1.0
2026-11-10T10:29:44-0500 ex1-deploy.sh[23212] ERROR: no such release: 20991231T000000Z
exit: 1
2026-11-10T10:29:44-0500 ex1-deploy.sh[23223] ERROR: 20261110T152944Z is already the current release
exit: 1
2026-11-10T10:29:44-0500 ex1-deploy.sh[23234] INFO: rolled back from 20261110T152944Z (1.1.0) to 20261110T152941Z (1.0.0)
exit: 0
* 20261110T152941Z 1.0.0
20261110T152942Z 1.1.0Note that current is now older than another release. This is exactly the state that
broke the first version of the rollback logic (see common mistakes), and why .previous
is recorded at deploy time.
Add -m SECONDS to healthcheck.sh: an attempt that succeeds but takes longer than the
budget counts as a failure. A release that makes the health endpoint ten times slower
should not go live.
Hint
curl -w '\n%{time_total}' appends the total time on its own line after the body. Bash
cannot compare decimals, but awk can.
Show Solution
Add MAX_SECS="" to the defaults, m: to the getopts string with
m) MAX_SECS=$OPTARG ;;, and replace the success branch of the loop:
if out=$(curl -fsS --max-time "$TIMEOUT" --connect-timeout "$TIMEOUT" \
-w '\n%{time_total}' "$url" 2>&1); then
secs=${out##*$'\n'} # last line: time_total from -w
out=${out%$'\n'*} # the body without that line...
out=${out%$'\n'} # ...and without the body's own final newline
if [[ -n $MAX_SECS ]] && awk -v t="$secs" -v m="$MAX_SECS" 'BEGIN { exit !(t > m) }'; then
warn "attempt $i/$RETRIES: too slow: ${secs}s (budget ${MAX_SECS}s)"
elif [[ -n $EXPECT && $out != *"$EXPECT"* ]]; then
warn "attempt $i/$RETRIES: body does not contain '$EXPECT': ${out:0:80}"
else
info "healthy after $i attempt(s) in ${secs}s: ${out:0:80}"
return 0
fi
elsetoolkit/ex2-healthcheck.sh -m 0.5 -e version=1.1.0 -r 2 -w 1 http://127.0.0.1:18480/healthz; echo "exit: $?"
toolkit/ex2-healthcheck.sh -m 0.00001 -r 2 -w 1 http://127.0.0.1:18480/healthz; echo "exit: $?"2026-11-10T10:30:55-0500 ex2-healthcheck.sh[25074] INFO: healthy after 1 attempt(s) in 0.004652s: ok version=1.1.0
exit: 0
2026-11-10T10:30:55-0500 ex2-healthcheck.sh[25083] WARN: attempt 1/2: too slow: 0.000876s (budget 0.00001s)
2026-11-10T10:30:56-0500 ex2-healthcheck.sh[25083] WARN: attempt 2/2: too slow: 0.002302s (budget 0.00001s)
2026-11-10T10:30:56-0500 ex2-healthcheck.sh[25083] ERROR: http://127.0.0.1:18480/healthz still unhealthy after 2 attempts
exit: 1The second call uses an impossible budget to exercise the failure path. My first version
printed an empty line after the body, because the response's own trailing newline was
left in out; the second out=${out%$'\n'} removes it. In practice, set the budget from
measurements (a few times the normal response time), not a guess.
A release can have a perfect health endpoint and still be broken: the home page is
missing, the CSS returns 404. Add -s FILE to release.sh: a file of URL paths (one per
line, # comments allowed) that must all return 2xx after the health check passes.
A smoke failure must trigger the same rollback as a health failure, and the rollback must
be smoke-tested too. Test it with an artifact that has a valid healthz but no
index.html.
Show Solution
Wrap both checks in one function and use it in both places:
SMOKE=""
# healthy VERSION: the health endpoint reports VERSION and every smoke path answers 2xx
healthy() {
local path
"$TOOLS/healthcheck.sh" -e "version=$1" -r 4 -w 1 -t 2 "http://127.0.0.1:$PORT/healthz" || return 1
[[ -n $SMOKE ]] || return 0
while read -r path; do
if [[ -z $path || $path == \#* ]]; then continue; fi
if ! curl -fsS -o /dev/null --max-time 5 "http://127.0.0.1:$PORT$path"; then
error "smoke test failed: $path"
return 1
fi
info "smoke test ok: $path"
done < "$SMOKE"
}Add s: to the getopts string with s) SMOKE=$OPTARG ;;, then replace the two
healthcheck.sh calls with healthy "$version" and healthy "$previous".
cat smoke.txt
toolkit/ex3-release.sh -s smoke.txt artifacts/shop-1.3.0.tar.gz; echo "exit: $?"# paths that must answer 2xx after every release
/index.html
/healthz
2026-11-10T10:29:45-0500 deploy.sh[23323] INFO: shop-1.3.0.tar.gz looks good
2026-11-10T10:29:45-0500 ex3-release.sh[23318] INFO: releasing 1.3.0 (currently running: 1.0.0)
2026-11-10T10:29:45-0500 deploy.sh[23357] INFO: deployed 1.3.0 as release 20261110T152945Z (previous: 20261110T152941Z)
2026-11-10T10:29:45-0500 appctl.sh[23399] INFO: stopped PID 23267
2026-11-10T10:29:45-0500 appctl.sh[23399] INFO: started PID 23411 on port 18480, serving releases/20261110T152945Z
2026-11-10T10:29:46-0500 healthcheck.sh[23421] INFO: healthy after 1 attempt(s): ok version=1.3.0
curl: (22) The requested URL returned error: 404
2026-11-10T10:29:46-0500 ex3-release.sh[23318] ERROR: smoke test failed: /index.html
2026-11-10T10:29:46-0500 ex3-release.sh[23318] ERROR: release 1.3.0 failed its checks, rolling back
2026-11-10T10:29:46-0500 deploy.sh[23435] INFO: rolled back from 20261110T152945Z (1.3.0) to 20261110T152941Z (1.0.0)
2026-11-10T10:29:46-0500 appctl.sh[23454] INFO: stopped PID 23411
2026-11-10T10:29:46-0500 appctl.sh[23454] INFO: started PID 23466 on port 18480, serving releases/20261110T152941Z
2026-11-10T10:29:46-0500 healthcheck.sh[23476] INFO: healthy after 1 attempt(s): ok version=1.0.0
2026-11-10T10:29:46-0500 ex3-release.sh[23318] INFO: smoke test ok: /index.html
2026-11-10T10:29:46-0500 ex3-release.sh[23318] INFO: smoke test ok: /healthz
2026-11-10T10:29:46-0500 ex3-release.sh[23318] ERROR: rolled back to 1.0.0; 1.3.0 was NOT released
exit: 1The health endpoint said 1.3.0 was fine; the smoke test caught the missing page and the
release was rolled back to 1.0.0, the release that was actually live before (here, after
exercise 1's targeted rollback, the recorded .previous mattered). A good release passes
both:
toolkit/ex3-release.sh -s smoke.txt artifacts/shop-1.1.0.tar.gz; echo "exit: $?"2026-11-10T10:29:46-0500 deploy.sh[23499] INFO: shop-1.1.0.tar.gz looks good
2026-11-10T10:29:46-0500 ex3-release.sh[23494] INFO: releasing 1.1.0 (currently running: 1.0.0)
2026-11-10T10:29:46-0500 deploy.sh[23533] INFO: deployed 1.1.0 as release 20261110T152946Z (previous: 20261110T152941Z)
2026-11-10T10:29:46-0500 appctl.sh[23575] INFO: stopped PID 23466
2026-11-10T10:29:46-0500 appctl.sh[23575] INFO: started PID 23587 on port 18480, serving releases/20261110T152946Z
2026-11-10T10:29:46-0500 healthcheck.sh[23597] INFO: healthy after 1 attempt(s): ok version=1.1.0
2026-11-10T10:29:46-0500 ex3-release.sh[23494] INFO: smoke test ok: /index.html
2026-11-10T10:29:46-0500 ex3-release.sh[23494] INFO: smoke test ok: /healthz
2026-11-10T10:29:46-0500 ex3-release.sh[23494] INFO: release 1.1.0 is live
exit: 0Keep the smoke list short and fast (a handful of pages that exercise templates, static assets and one database read). It runs on every release, including the emergency ones.
Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), curl 8.5.0, Python 3.12
(http.server as the stand-in app), GNU coreutils 9.4, ShellCheck 0.9.0, November 2026.
Every release, rollback and failure shown was run end to end on the lab machine in one
session, against a local application root; nothing was deployed to a real server and the
CI and multi-server usage was not tested. The sort-order rollback bug described above was
found while testing and is fixed in the listings; the inherited-lock failure was
reproduced on purpose with a modified copy of release.sh. I also
tried to demonstrate the ln -sfn versus rename difference with a race test (a reader
checking current/VERSION millions of times while a loop flipped the link); on this WSL
kernel both methods produced a few failed lookups out of millions, so the test settled
nothing and is not shown. Paths from the test machine's scratch directories are shown shortened to /home/sid/….
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.