Capstone: A Deployment and Health-Check Toolkit

Build and run a complete Bash deployment toolkit: timestamped release directories, an atomic symlink switch, curl health checks with retries, and automatic rollback.

By The Dark DebuggerUpdated 35 min read

This is the capstone of the Bash track. We build a small but complete deployment toolkit: each release goes into its own timestamped directory, a symlink switch makes it live atomically, a health check with retries and timeouts decides whether it worked, and a wrapper rolls back automatically when it did not. Everything is run end to end, including a deliberately broken release. It uses the shared library and conventions from real-world scripts, the process control from processes, signals, and job control, and the template from writing production-ready scripts.

Why release directories

The simplest deployment is copying new files over old ones: rsync -a build/ /srv/shop/, then restart. It works until it doesn't:

  • For the seconds (or minutes, on a slow link) that the copy takes, the server runs a mix of old and new files. A request that loads a new template with an old library fails.
  • If the copy dies halfway, the application is in a state that never existed anywhere, not in git and not in any build.
  • Rolling back means finding and re-copying the previous build, under pressure, while the site is broken.

The release-directory layout, used by Capistrano, Deployer and many in-house tools, fixes all three. Each deploy unpacks into a new directory, current is a symlink to the live one, and going live (or going back) is a single atomic rename of that symlink. The previous release is still on disk, so rollback takes a second.

The layout

After two deploys, the application root looks like this:

srv/shop
├── .deploy.lock
├── .previous
├── .release.lock
├── current -> releases/20261110T152904Z
├── deploy-history.log
├── releases
│   ├── 20261110T152902Z
│   │   ├── VERSION
│   │   ├── config -> ../../shared/config
│   │   ├── logs -> ../../shared/logs
│   │   └── public
│   │       ├── healthz
│   │       └── index.html
│   └── 20261110T152904Z
│       ├── VERSION
│       ├── config -> ../../shared/config
│       ├── logs -> ../../shared/logs
│       └── public
│           ├── healthz
│           └── index.html
├── run
│   └── app.pid
└── shared
    ├── config
    └── logs
        └── app.log
  • releases/<UTC timestamp>/: one directory per deploy, never modified after it is created. UTC timestamps sort chronologically and do not jump at daylight-saving changes.
  • shared/: what must survive deploys, such as configuration with secrets, logs and user uploads. Each release links to it.
  • current: a relative symlink to the live release. The app always runs from current. Relative, so the whole tree can be moved or restored from a backup.
  • .previous: the release that was live before the last deploy, the rollback target.
  • deploy-history.log: an append-only record of every switch.

The toolkit

Four scripts, all built on lib/common.sh from the previous article (logging, die, locking, temp cleanup and an ERR trap that exits with 1), plus a build helper:

ScriptJob
build-artifact.shPackage the site as shop-VERSION.tar.gz plus a .sha256 file (stands in for your CI build)
deploy.shverify, deploy, rollback and list releases
appctl.shStart, stop and restart the app from current
healthcheck.shPoll a URL until it is healthy, with retries and timeouts
release.shThe wrapper: verify, deploy, restart, check, roll back on failure

The application and its artifacts

The "application" is a static site served by python3 -m http.server. It stands in for any long-running service: it reads its files when it starts, so it must be restarted to serve a new release. The health endpoint is a file, public/healthz, containing the version, so the health check can tell "the new release is up" from "the old process is still answering".

#!/usr/bin/env bash
# build-artifact.sh VERSION [--broken] - package the shop site as artifacts/shop-VERSION.tar.gz
# --broken leaves out the health endpoint, to simulate a release that comes up broken.
set -euo pipefail
version=${1:?usage: build-artifact.sh VERSION [--broken]}
build=$(mktemp -d)
trap 'rm -rf "$build"' EXIT
 
cp -r site/. "$build/"
echo "$version" > "$build/VERSION"
sed -i "s/@VERSION@/$version/" "$build/public/index.html"
if [[ ${2:-} != --broken ]]; then
  echo "ok version=$version" > "$build/public/healthz"
fi
mkdir -p artifacts
tar -czf "artifacts/shop-$version.tar.gz" -C "$build" .
(cd artifacts && sha256sum "shop-$version.tar.gz" > "shop-$version.tar.gz.sha256")
echo "built artifacts/shop-$version.tar.gz"
./build-artifact.sh 1.0.0 && ./build-artifact.sh 1.1.0 && ./build-artifact.sh 1.2.0 --broken
tar -tzf artifacts/shop-1.1.0.tar.gz; tar -tzf artifacts/shop-1.2.0.tar.gz
built artifacts/shop-1.0.0.tar.gz
built artifacts/shop-1.1.0.tar.gz
built artifacts/shop-1.2.0.tar.gz
./
./public/
./public/healthz
./public/index.html
./VERSION
./
./public/
./public/index.html
./VERSION

Version 1.2.0 is the forced failure: a build that lost its health endpoint, standing in for any release that starts but does not work (a missing environment variable, a failed migration, a typo in a config file).

deploy.sh: releases, the atomic switch, and rollback

#!/usr/bin/env bash
#
# deploy.sh - release-directory deployments with atomic switch and rollback
#
# Usage: deploy.sh [-v] [-k KEEP] [-r APP_ROOT] deploy ARTIFACT.tar.gz
#        deploy.sh [-v] [-r APP_ROOT] rollback
#        deploy.sh [-r APP_ROOT] list
#        deploy.sh verify ARTIFACT.tar.gz
#
# Layout under APP_ROOT:
#   releases/<UTC timestamp>/   one directory per deploy
#   shared/                     files that survive deploys (config, logs)
#   current -> releases/<id>    what the app runs
# Exit codes: 0 ok, 1 failure, 2 usage error, 4 missing dependency, 5 locked
 
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
 
APP_ROOT=${APP_ROOT:-/srv/shop}
KEEP=5
 
usage() {
  cat <<EOF
Usage: $SCRIPT_NAME [options] deploy ARTIFACT.tar.gz
       $SCRIPT_NAME [options] rollback
       $SCRIPT_NAME [options] list
       $SCRIPT_NAME verify ARTIFACT.tar.gz
 
Options:
  -r DIR   application root (default: \$APP_ROOT or /srv/shop; now $APP_ROOT)
  -k N     releases to keep after a deploy (default: $KEEP)
  -v       verbose
  -h       show this help
EOF
}
 
# current_release: print the release id that "current" points to (empty if none)
current_release() {
  local target
  target=$(readlink "$APP_ROOT/current" 2>/dev/null) || return 0
  printf '%s\n' "${target##*/}"
}
 
# releases: print release ids, oldest first
releases() {
  [[ -d $APP_ROOT/releases ]] || return 0
  find "$APP_ROOT/releases" -mindepth 1 -maxdepth 1 -type d \
       -regextype posix-extended -regex '.*/[0-9]{8}T[0-9]{6}Z' -printf '%f\n' | sort
}
 
# switch_to ID: point "current" at releases/ID atomically
switch_to() {
  local id=$1
  [[ -d $APP_ROOT/releases/$id ]] || die 1 "no such release: $id"
  ln -sfn "releases/$id" "$APP_ROOT/current.new"
  mv -T "$APP_ROOT/current.new" "$APP_ROOT/current"   # rename(2): atomic
  echo "$(date -u +%FT%TZ) $1 $(<"$APP_ROOT/releases/$id/VERSION") ${USER:-?}" \
    >> "$APP_ROOT/deploy-history.log"
}
 
verify_artifact() {
  local artifact=$1 listing
  [[ -f $artifact ]] || die 1 "artifact not found: $artifact"
  if [[ -f $artifact.sha256 ]]; then
    (cd "$(dirname "$artifact")" && sha256sum --quiet -c "${artifact##*/}.sha256") ||
      die 1 "checksum mismatch: $artifact"
  fi
  listing=$(tar -tzf "$artifact") || die 1 "not a readable tar.gz: $artifact"
  grep -qx './VERSION' <<< "$listing" || die 1 "artifact has no VERSION file"
  grep -q '^\./public/' <<< "$listing" || die 1 "artifact has no public/ directory"
}
 
# previous_release: print the release that was live before the last deploy
previous_release() {
  if [[ -f $APP_ROOT/.previous ]]; then cat "$APP_ROOT/.previous"; fi
}
 
prune() {
  local current previous all=() id i
  current=$(current_release)
  previous=$(previous_release)
  mapfile -t all < <(releases)
  # newest KEEP stay; never delete the current release or the rollback target
  for ((i = 0; i < ${#all[@]} - KEEP; i++)); do
    id=${all[i]}
    [[ $id != "$current" && $id != "$previous" ]] || continue
    info "pruning old release $id"
    rm -rf -- "${APP_ROOT:?}/releases/$id"
  done
}
 
cmd_deploy() {
  local artifact=${1:?usage: deploy ARTIFACT.tar.gz} id staging version
  verify_artifact "$artifact"
  mkdir -p "$APP_ROOT/releases" "$APP_ROOT/shared/logs" "$APP_ROOT/shared/config"
 
  id=$(date -u +%Y%m%dT%H%M%SZ)
  [[ ! -e $APP_ROOT/releases/$id ]] || die 1 "release $id already exists; deploys are once per second at most"
  staging="$APP_ROOT/releases/.staging-$id"
  register_tmp "$staging"
  mkdir -p "$staging"
  tar -xzf "$artifact" -C "$staging"
  ln -s ../../shared/logs "$staging/logs"
  ln -s ../../shared/config "$staging/config"
  mv -T "$staging" "$APP_ROOT/releases/$id"
  version=$(<"$APP_ROOT/releases/$id/VERSION")
 
  local previous
  previous=$(current_release)
  if [[ -n $previous ]]; then
    printf '%s\n' "$previous" > "$APP_ROOT/.previous"   # the rollback target
  fi
  switch_to "$id"
  info "deployed $version as release $id (previous: ${previous:-none})"
  prune
}
 
cmd_rollback() {
  local current previous id
  current=$(current_release)
  [[ -n $current ]] || die 1 "nothing is deployed"
  # prefer the release that was live before the last deploy; fall back to
  # the newest release older than current (e.g. for a second rollback)
  previous=$(previous_release)
  if [[ -z $previous || $previous == "$current" || ! -d $APP_ROOT/releases/$previous ]]; then
    previous=""
    while read -r id; do
      if [[ $id < "$current" ]]; then previous=$id; fi
    done < <(releases)
  fi
  [[ -n $previous ]] || die 1 "no release older than $current to roll back to"
 
  switch_to "$previous"
  rm -f "$APP_ROOT/.previous"
  info "rolled back from $current ($(<"$APP_ROOT/releases/$current/VERSION")) to $previous ($(<"$APP_ROOT/releases/$previous/VERSION"))"
  # a rolled-back release must never become a rollback target later
  rm -rf -- "${APP_ROOT:?}/releases/$current"
}
 
cmd_list() {
  local current id mark
  current=$(current_release)
  while read -r id; do
    mark=" "; [[ $id == "$current" ]] && mark="*"
    printf '%s %s  %s\n' "$mark" "$id" "$(<"$APP_ROOT/releases/$id/VERSION")"
  done < <(releases)
}
 
main() {
  local opt
  while getopts ':r:k:vh' opt; do
    case $opt in
      r) APP_ROOT=$OPTARG ;;
      k) KEEP=$OPTARG ;;
      v) VERBOSE=1 ;;
      h) usage; exit 0 ;;
      :) usage_error "option -$OPTARG requires an argument" ;;
      \?) usage_error "unknown option: -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))
  [[ $KEEP =~ ^[2-9]$|^[1-9][0-9]+$ ]] || usage_error "-k must be at least 2 (current + one to roll back to)"
  require_commands tar sha256sum flock find readlink
  APP_ROOT=$(realpath -m -- "$APP_ROOT")
 
  case ${1:-} in
    deploy)   mkdir -p "$APP_ROOT"; acquire_lock "$APP_ROOT/.deploy.lock"; cmd_deploy "${2:-}" ;;
    rollback) [[ -d $APP_ROOT/releases ]] || die 1 "nothing is deployed in $APP_ROOT"
              acquire_lock "$APP_ROOT/.deploy.lock"; cmd_rollback ;;
    list)     cmd_list ;;
    verify)   verify_artifact "${2:?usage: verify ARTIFACT.tar.gz}"; info "${2##*/} looks good" ;;
    *)        usage_error "expected a command: deploy, rollback, list or verify" ;;
  esac
}
 
main "$@"

The parts that make it safe:

  • Unpack into .staging-ID, then rename into releases/ID. A deploy that dies mid-extraction leaves a staging directory (removed by the EXIT trap), never a half-populated release that list or rollback could pick.
  • Shared directories are linked in before the switch, so the new release is complete the moment it goes live.
  • The switch is a rename. switch_to creates the new link as current.new and then mv -T current.new current, which is the rename(2) system call. The kernel guarantees that a rename over an existing name is atomic: every process opening current/… sees either the old target or the new one, never "no such file". -T stops mv from treating current (a link to a directory) as a directory to move into.
  • Rollback goes to the recorded previous release, not just "the next older directory". Why that matters is in common mistakes.
  • Pruning never deletes current or the rollback target, and -k must be at least 2, so there is always somewhere to roll back to.
  • The history log records every switch with time, release, version and user, which is the first thing you want during an incident ("what changed at 14:02?").

A note on ln -sfn: on the lab's GNU coreutils 9.4, ln -sfn itself already creates the link under a temporary name and renames it into place (the ln binary imports renameat). Older coreutils and other implementations such as BusyBox and the BSD ln on macOS remove the old link first and then create the new one, leaving a brief moment with no current at all. Writing the rename explicitly makes the script atomic no matter which ln the server has.

appctl.sh: restarting the app from current

This is the appctl from process management, adapted to the release layout and the shared library:

#!/usr/bin/env bash
#
# appctl.sh - start, stop and restart the shop app from APP_ROOT/current
#
# Usage: appctl.sh [-r APP_ROOT] [-p PORT] {start|stop|restart|status}
# The app is a static site served by python3 -m http.server, standing in for
# any long-running service that must be restarted to pick up a new release.
# Exit codes: 0 ok, 1 failure, 2 usage error, 3 not running (status)
 
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
 
APP_ROOT=${APP_ROOT:-/srv/shop}
PORT=${PORT:-18480}
 
pid_of_app() {
  local pid
  [[ -f $APP_ROOT/run/app.pid ]] || return 1
  pid=$(<"$APP_ROOT/run/app.pid")
  kill -0 "$pid" 2>/dev/null || return 1
  tr '\0' ' ' < "/proc/$pid/cmdline" | grep -q "http.server $PORT" || return 1
  printf '%s\n' "$pid"
}
 
start() {
  local pid i
  if pid=$(pid_of_app); then info "already running (PID $pid)"; return 0; fi
  [[ -d $APP_ROOT/current/public ]] || die 1 "nothing deployed in $APP_ROOT/current"
  mkdir -p "$APP_ROOT/run"
  # cd resolves the symlink: the process keeps serving this release until restarted
  (cd "$APP_ROOT/current" &&
   exec nohup python3 -m http.server "$PORT" --bind 127.0.0.1 --directory public \
     >> "$APP_ROOT/shared/logs/app.log" 2>&1) &
  pid=$!
  echo "$pid" > "$APP_ROOT/run/app.pid"
  for ((i = 0; i < 50; i++)); do
    kill -0 "$pid" 2>/dev/null || break
    if [[ $(lsof -t -nP -iTCP:"$PORT" -sTCP:LISTEN 2>/dev/null) == "$pid" ]]; then
      info "started PID $pid on port $PORT, serving $(readlink "$APP_ROOT/current")"
      return 0
    fi
    sleep 0.1
  done
  rm -f "$APP_ROOT/run/app.pid"
  die 1 "app failed to start; last log line: $(tail -n 1 "$APP_ROOT/shared/logs/app.log")"
}
 
stop() {
  local pid i
  if ! pid=$(pid_of_app); then info "not running"; rm -f "$APP_ROOT/run/app.pid"; return 0; fi
  kill -TERM "$pid"
  for ((i = 0; i < 100; i++)); do
    if ! kill -0 "$pid" 2>/dev/null; then
      info "stopped PID $pid"
      rm -f "$APP_ROOT/run/app.pid"
      return 0
    fi
    sleep 0.1
  done
  warn "PID $pid ignored SIGTERM for 10s, sending SIGKILL"
  kill -KILL "$pid"
  rm -f "$APP_ROOT/run/app.pid"
}
 
main() {
  local opt pid
  while getopts ':r:p:h' opt; do
    case $opt in
      r) APP_ROOT=$OPTARG ;;
      p) PORT=$OPTARG ;;
      h) echo "Usage: $SCRIPT_NAME [-r APP_ROOT] [-p PORT] {start|stop|restart|status}"; exit 0 ;;
      *) usage_error "bad option" ;;
    esac
  done
  shift $((OPTIND - 1))
  require_commands python3 lsof
  case ${1:-} in
    start)   start ;;
    stop)    stop ;;
    restart) stop; start ;;
    status)
      if pid=$(pid_of_app); then
        echo "running (PID $pid), serving $(readlink "$APP_ROOT/current")"
      else
        echo "stopped"; exit 3
      fi ;;
    *) usage_error "expected start, stop, restart or status" ;;
  esac
}
 
main "$@"

The line worth understanding is cd "$APP_ROOT/current". Changing into a symlinked directory gives the process the real directory as its working directory. After the next deploy flips current, the running process still serves its own release, which is exactly how real applications behave: code is loaded at start-up. That is why a deploy alone changes nothing until the restart, and why the health check must verify the version. exec replaces the subshell with nohup, which replaces itself with python3, so $! is the server's PID.

On a server with systemd you would replace appctl.sh with a unit (WorkingDirectory=/srv/shop/current, ExecStart=…) and systemctl restart shop; the rest of the toolkit stays the same.

healthcheck.sh: retries and timeouts

#!/usr/bin/env bash
#
# healthcheck.sh - poll an HTTP endpoint until it is healthy or retries run out
#
# Usage: healthcheck.sh [-v] [-e TEXT] [-r RETRIES] [-t TIMEOUT] [-w WAIT] URL
#
# Healthy means: HTTP 2xx within TIMEOUT seconds, and the body contains TEXT
# (if -e is given). Exit codes: 0 healthy, 1 unhealthy, 2 usage error,
# 4 missing dependency
 
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
# shellcheck source=lib/common.sh
source "$(dirname "$(realpath "$0")")/lib/common.sh"
 
EXPECT="" RETRIES=5 TIMEOUT=3 WAIT=2
 
usage() {
  cat <<EOF
Usage: $SCRIPT_NAME [options] URL
 
Options:
  -e TEXT   the response body must contain TEXT
  -r N      attempts before giving up (default: $RETRIES)
  -t SEC    timeout per attempt, connect + response (default: $TIMEOUT)
  -w SEC    pause between attempts (default: $WAIT)
  -v        verbose
  -h        show this help
EOF
}
 
main() {
  local opt url i out
  while getopts ':e:r:t:w:vh' opt; do
    case $opt in
      e) EXPECT=$OPTARG ;;
      r) RETRIES=$OPTARG ;;
      t) TIMEOUT=$OPTARG ;;
      w) WAIT=$OPTARG ;;
      v) VERBOSE=1 ;;
      h) usage; exit 0 ;;
      :) usage_error "option -$OPTARG requires an argument" ;;
      \?) usage_error "unknown option: -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))
  url=${1:-}
  [[ $url =~ ^https?:// ]] || usage_error "expected an http(s) URL, got '$url'"
  [[ $RETRIES =~ ^[1-9][0-9]*$ ]] || usage_error "-r needs a positive number"
  require_commands curl
 
  for ((i = 1; i <= RETRIES; i++)); do
    # -f: HTTP errors are failures; -sS: no progress bar, but do show errors
    if out=$(curl -fsS --max-time "$TIMEOUT" --connect-timeout "$TIMEOUT" "$url" 2>&1); then
      if [[ -z $EXPECT || $out == *"$EXPECT"* ]]; then
        info "healthy after $i attempt(s): ${out:0:80}"
        return 0
      fi
      warn "attempt $i/$RETRIES: body does not contain '$EXPECT': ${out:0:80}"
    else
      warn "attempt $i/$RETRIES: ${out:0:120}"
    fi
    if (( i < RETRIES )); then sleep "$WAIT"; fi
  done
  error "$url still unhealthy after $RETRIES attempts"
  exit 1
}
 
main "$@"

Every option matters in practice:

curl optionWhy
-f (--fail)Without it, a 404 or 500 page is a "successful" transfer and curl exits 0
-sSSilent (no progress meter), but still print the error message
--max-timeUpper bound for the whole request; a hung app must not hang the deploy
--connect-timeoutUpper bound for connecting; without it, an address that silently drops packets waits for the kernel's TCP connect timeout, around two minutes on Linux

The retry loop is in Bash rather than curl's own --retry, for reasons the health check section shows with real output. The function ends with exit 1 rather than return 1: main "$@" returning non-zero would trigger the library's ERR trap and log a second, misleading error.

release.sh: the wrapper that rolls back

#!/usr/bin/env bash
#
# release.sh - deploy an artifact, restart the app, health-check it, and roll
# back automatically if the new release is not healthy.
#
# Usage: release.sh [-v] [-r APP_ROOT] [-p PORT] ARTIFACT.tar.gz
# Exit codes: 0 released, 1 release failed (not deployed, or rolled back),
#             2 usage error, 3 release failed AND rollback failed (page someone),
#             5 another release in progress
 
set -Eeuo pipefail
readonly SCRIPT_NAME=${0##*/}
TOOLS=$(dirname "$(realpath "$0")")
# shellcheck source=lib/common.sh
source "$TOOLS/lib/common.sh"
 
export APP_ROOT=${APP_ROOT:-/srv/shop}
export PORT=${PORT:-18480}
 
main() {
  local opt artifact version previous
  while getopts ':r:p:vh' opt; do
    case $opt in
      r) APP_ROOT=$OPTARG ;;
      p) PORT=$OPTARG ;;
      v) VERBOSE=1 ;;
      h) echo "Usage: $SCRIPT_NAME [-v] [-r APP_ROOT] [-p PORT] ARTIFACT.tar.gz"; exit 0 ;;
      :) usage_error "option -$OPTARG requires an argument" ;;
      \?) usage_error "unknown option: -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))
  artifact=${1:-}
  [[ -n $artifact ]] || usage_error "no artifact given"
  require_commands tar curl
 
  mkdir -p "$APP_ROOT"
  acquire_lock "$APP_ROOT/.release.lock"
  "$TOOLS/deploy.sh" verify "$artifact"       # fail fast, before anything changes
  version=$(tar -xzOf "$artifact" ./VERSION)
  previous=$("$TOOLS/deploy.sh" list | awk '$1 == "*" { print $3 }')
  info "releasing $version (currently running: ${previous:-nothing})"
 
  # {LOCK_FD}>&- : the app must not inherit our lock, or it would hold it forever
  "$TOOLS/deploy.sh" deploy "$artifact"
  "$TOOLS/appctl.sh" restart {LOCK_FD}>&-
 
  if "$TOOLS/healthcheck.sh" -e "version=$version" -r 4 -w 1 -t 2 \
       "http://127.0.0.1:$PORT/healthz"; then
    info "release $version is live"
    exit 0
  fi
 
  error "release $version failed its health check, rolling back"
  if "$TOOLS/deploy.sh" rollback &&
     "$TOOLS/appctl.sh" restart {LOCK_FD}>&- &&
     "$TOOLS/healthcheck.sh" -e "version=$previous" -r 4 -w 1 -t 2 \
       "http://127.0.0.1:$PORT/healthz"; then
    error "rolled back to $previous; $version was NOT released"
    exit 1
  fi
  error "rollback to ${previous:-nothing} failed too; the site may be down"
  exit 3
}
 
main "$@"

The flow is: verify the artifact before touching anything, deploy, restart, check that the new version answers, and otherwise roll back, restart, and check that the old version answers again. The exit codes distinguish "it failed but we are safely back" (1) from "it failed and so did the rollback" (3), which is the one that should wake someone up. The {LOCK_FD}>&- on the appctl.sh lines closes the release lock's file descriptor for that command only; the common mistakes show what happens without it.

All the scripts pass shellcheck (run from the toolkit directory, so the # shellcheck source=lib/common.sh paths resolve):

(cd toolkit && shellcheck -x ./*.sh) && shellcheck build-artifact.sh && echo clean
clean

Running it end to end

Everything below ran in one session against a fresh application root, with APP_ROOT pointing at srv/shop in the working directory and the app on port 18480.

Two good releases

toolkit/release.sh artifacts/shop-1.0.0.tar.gz; echo "exit: $?"
toolkit/release.sh artifacts/shop-1.1.0.tar.gz; echo "exit: $?"
curl -s http://127.0.0.1:18480/
2026-11-10T10:29:02-0500 deploy.sh[13601] INFO: shop-1.0.0.tar.gz looks good
2026-11-10T10:29:02-0500 release.sh[13596] INFO: releasing 1.0.0 (currently running: nothing)
2026-11-10T10:29:02-0500 deploy.sh[13633] INFO: deployed 1.0.0 as release 20261110T152902Z (previous: none)
2026-11-10T10:29:02-0500 appctl.sh[13674] INFO: not running
2026-11-10T10:29:02-0500 appctl.sh[13674] INFO: started PID 13683 on port 18480, serving releases/20261110T152902Z
2026-11-10T10:29:02-0500 healthcheck.sh[13693] INFO: healthy after 1 attempt(s): ok version=1.0.0
2026-11-10T10:29:02-0500 release.sh[13596] INFO: release 1.0.0 is live
exit: 0
2026-11-10T10:29:04-0500 deploy.sh[13709] INFO: shop-1.1.0.tar.gz looks good
2026-11-10T10:29:04-0500 release.sh[13704] INFO: releasing 1.1.0 (currently running: 1.0.0)
2026-11-10T10:29:04-0500 deploy.sh[13743] INFO: deployed 1.1.0 as release 20261110T152904Z (previous: 20261110T152902Z)
2026-11-10T10:29:04-0500 appctl.sh[13785] INFO: stopped PID 13683
2026-11-10T10:29:04-0500 appctl.sh[13785] INFO: started PID 13798 on port 18480, serving releases/20261110T152904Z
2026-11-10T10:29:04-0500 healthcheck.sh[13809] INFO: healthy after 1 attempt(s): ok version=1.1.0
2026-11-10T10:29:04-0500 release.sh[13704] INFO: release 1.1.0 is live
exit: 0
<!doctype html>
<title>Northwind Outdoor Shop</title>
<h1>Northwind Outdoor Shop</h1>
<p>Release 1.1.0</p>

The tree at this point is the one in the layout above, and .previous contains 20261110T152902Z, the 1.0.0 release.

The forced failure

Now the broken 1.2.0, timed:

time toolkit/release.sh artifacts/shop-1.2.0.tar.gz; echo "exit: $?"
2026-11-10T10:29:05-0500 deploy.sh[13835] INFO: shop-1.2.0.tar.gz looks good
2026-11-10T10:29:05-0500 release.sh[13827] INFO: releasing 1.2.0 (currently running: 1.1.0)
2026-11-10T10:29:05-0500 deploy.sh[13869] INFO: deployed 1.2.0 as release 20261110T152905Z (previous: 20261110T152904Z)
2026-11-10T10:29:05-0500 appctl.sh[13911] INFO: stopped PID 13798
2026-11-10T10:29:05-0500 appctl.sh[13911] INFO: started PID 13924 on port 18480, serving releases/20261110T152905Z
2026-11-10T10:29:05-0500 healthcheck.sh[13952] WARN: attempt 1/4: curl: (22) The requested URL returned error: 404
2026-11-10T10:29:06-0500 healthcheck.sh[13952] WARN: attempt 2/4: curl: (22) The requested URL returned error: 404
2026-11-10T10:29:07-0500 healthcheck.sh[13952] WARN: attempt 3/4: curl: (22) The requested URL returned error: 404
2026-11-10T10:29:08-0500 healthcheck.sh[13952] WARN: attempt 4/4: curl: (22) The requested URL returned error: 404
2026-11-10T10:29:08-0500 healthcheck.sh[13952] ERROR: http://127.0.0.1:18480/healthz still unhealthy after 4 attempts
2026-11-10T10:29:08-0500 release.sh[13827] ERROR: release 1.2.0 failed its health check, rolling back
2026-11-10T10:29:09-0500 deploy.sh[14039] INFO: rolled back from 20261110T152905Z (1.2.0) to 20261110T152904Z (1.1.0)
2026-11-10T10:29:09-0500 appctl.sh[14058] INFO: stopped PID 13924
2026-11-10T10:29:09-0500 appctl.sh[14058] INFO: started PID 14070 on port 18480, serving releases/20261110T152904Z
2026-11-10T10:29:09-0500 healthcheck.sh[14080] INFO: healthy after 1 attempt(s): ok version=1.1.0
2026-11-10T10:29:09-0500 release.sh[13827] ERROR: rolled back to 1.1.0; 1.2.0 was NOT released
 
real	0m3.796s
user	0m0.144s
sys	0m0.142s
exit: 1
toolkit/deploy.sh list; curl -s http://127.0.0.1:18480/healthz
cat srv/shop/deploy-history.log
  20261110T152902Z  1.0.0
* 20261110T152904Z  1.1.0
ok version=1.1.0
2026-11-10T15:29:02Z 20261110T152902Z 1.0.0 sid
2026-11-10T15:29:04Z 20261110T152904Z 1.1.0 sid
2026-11-10T15:29:05Z 20261110T152905Z 1.2.0 sid
2026-11-10T15:29:09Z 20261110T152904Z 1.1.0 sid

From "deploy 1.2.0" to "1.1.0 healthy again" took under four seconds, three of which were the deliberate one-second pauses between health attempts. The broken release directory is gone, so it can never be picked as a rollback target later, but the history log still records that it was live from 15:29:05 to 15:29:09 UTC. That is the line you need when someone asks why a few requests failed at that minute. The exit status 1 is what a CI job uses to mark the pipeline as failed.

A corrupted artifact never gets deployed

An artifact truncated in transit (half the file, with the checksum of the full one):

head -c $(( $(stat -c %s artifacts/shop-1.1.0.tar.gz) / 2 )) artifacts/shop-1.1.0.tar.gz > artifacts/shop-1.1.1.tar.gz && sed 's/1.1.0/1.1.1/' artifacts/shop-1.1.0.tar.gz.sha256 > artifacts/shop-1.1.1.tar.gz.sha256
toolkit/release.sh artifacts/shop-1.1.1.tar.gz; echo "exit: $?"
toolkit/deploy.sh list
shop-1.1.1.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
2026-11-10T10:29:09-0500 deploy.sh[14112] ERROR: checksum mismatch: artifacts/shop-1.1.1.tar.gz
2026-11-10T10:29:09-0500 release.sh[14107] ERROR: line 39: '"$TOOLS/deploy.sh" verify "$artifact"' exited with status 1
exit: 1
  20261110T152902Z  1.0.0
* 20261110T152904Z  1.1.0

Nothing was unpacked, nothing restarted. While testing I first truncated the file to a fixed 1,000 bytes, and the "corrupted" release went live: the artifact was smaller than 1,000 bytes, so head -c had copied it whole and the checksum matched. A test that does not fail when you expect it to is telling you something about the test.

Manual rollback and pruning

deploy.sh is also the tool for a human. Rolling back by hand (when, for example, a bug shows up an hour after a release) is two commands:

toolkit/deploy.sh rollback && toolkit/appctl.sh restart
toolkit/deploy.sh list; curl -s http://127.0.0.1:18480/healthz
toolkit/deploy.sh rollback; echo "exit: $?"
2026-11-10T10:29:09-0500 deploy.sh[14137] INFO: rolled back from 20261110T152904Z (1.1.0) to 20261110T152902Z (1.0.0)
2026-11-10T10:29:09-0500 appctl.sh[14158] INFO: stopped PID 14070
2026-11-10T10:29:09-0500 appctl.sh[14158] INFO: started PID 14170 on port 18480, serving releases/20261110T152902Z
* 20261110T152902Z  1.0.0
ok version=1.0.0
2026-11-10T10:29:09-0500 deploy.sh[14193] ERROR: no release older than 20261110T152902Z to roll back to
exit: 1

Then three deploys in a row with -k 2 (keep two releases), to watch pruning:

2026-11-10T10:29:11-0500 deploy.sh[14287] INFO: deployed 1.1.0 as release 20261110T152910Z (previous: 20261110T152902Z)
2026-11-10T10:29:12-0500 deploy.sh[14330] INFO: deployed 1.1.0 as release 20261110T152912Z (previous: 20261110T152910Z)
2026-11-10T10:29:12-0500 deploy.sh[14330] INFO: pruning old release 20261110T152902Z
2026-11-10T10:29:13-0500 deploy.sh[14376] INFO: deployed 1.1.0 as release 20261110T152913Z (previous: 20261110T152912Z)
2026-11-10T10:29:13-0500 deploy.sh[14376] INFO: pruning old release 20261110T152910Z
toolkit/deploy.sh list
  20261110T152912Z  1.1.0
* 20261110T152913Z  1.1.0

Health checks under failure

A health check is only as good as its behavior when things go wrong. Here is healthcheck.sh against the three failure modes you will meet, starting with the old version still answering (the app was not restarted, or the restart failed silently):

toolkit/healthcheck.sh -e version=1.2.0 -r 2 -w 1 -t 2 http://127.0.0.1:18480/healthz; echo "exit: $?"
2026-11-10T10:29:15-0500 healthcheck.sh[14687] WARN: attempt 1/2: body does not contain 'version=1.2.0': ok version=1.1.0
2026-11-10T10:29:16-0500 healthcheck.sh[14687] WARN: attempt 2/2: body does not contain 'version=1.2.0': ok version=1.1.0
2026-11-10T10:29:16-0500 healthcheck.sh[14687] ERROR: http://127.0.0.1:18480/healthz still unhealthy after 2 attempts
exit: 1

A plain "is it 200?" check would have passed here and declared the release a success.

Nothing listening on the port (the app crashed on start):

toolkit/healthcheck.sh -r 3 -w 1 -t 2 http://127.0.0.1:18489/healthz; echo "exit: $?"
2026-11-10T10:29:16-0500 healthcheck.sh[14703] WARN: attempt 1/3: curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
2026-11-10T10:29:17-0500 healthcheck.sh[14703] WARN: attempt 2/3: curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
2026-11-10T10:29:18-0500 healthcheck.sh[14703] WARN: attempt 3/3: curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
2026-11-10T10:29:18-0500 healthcheck.sh[14703] ERROR: http://127.0.0.1:18489/healthz still unhealthy after 3 attempts
exit: 1

And the nastiest one, a process that accepts connections but never answers (deadlocked, or stuck on a database that is down). I simulated it with a Python socket that listens on port 18481 and never calls accept:

time toolkit/healthcheck.sh -r 2 -w 1 -t 2 http://127.0.0.1:18481/healthz; echo "exit: $?"
2026-11-10T10:29:21-0500 healthcheck.sh[14841] WARN: attempt 1/2: curl: (28) Operation timed out after 2015 milliseconds with 0 bytes received
2026-11-10T10:29:24-0500 healthcheck.sh[14841] WARN: attempt 2/2: curl: (28) Operation timed out after 2008 milliseconds with 0 bytes received
2026-11-10T10:29:24-0500 healthcheck.sh[14841] ERROR: http://127.0.0.1:18481/healthz still unhealthy after 2 attempts
 
real	0m5.040s
user	0m0.026s
sys	0m0.035s
exit: 1

Two attempts of 2 s plus a 1 s pause: 5 s, as designed. Without --max-time, curl would have waited as long as the server held the connection, and the deploy would have hung with it.

Why not curl --retry?

curl has built-in retries, and it is tempting to use them. Here is what they actually do against a port with nothing listening and against a 404:

time curl -fsS --retry 3 --retry-delay 1 http://127.0.0.1:18489/healthz; echo "exit: $?"
time curl -fsS --retry 3 --retry-delay 1 --retry-connrefused http://127.0.0.1:18489/healthz; echo "exit: $?"
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
 
real	0m0.014s
user	0m0.009s
sys	0m0.005s
exit: 7
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18489 after 0 ms: Couldn't connect to server
 
real	0m3.018s
user	0m0.009s
sys	0m0.006s
exit: 7
time curl -fsS --retry 3 --retry-delay 1 http://127.0.0.1:18480/missing; echo "exit: $?"
time curl -fsS --retry 3 --retry-delay 1 --retry-all-errors http://127.0.0.1:18480/missing; echo "exit: $?"
curl: (22) The requested URL returned error: 404
 
real	0m0.015s
user	0m0.007s
sys	0m0.007s
exit: 22
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
 
real	0m3.025s
user	0m0.010s
sys	0m0.005s
exit: 22

Plain --retry gave up immediately in both cases. It only retries errors curl considers transient (timeouts and HTTP 408, 429, 500, 502, 503 and 504). "Connection refused", which is exactly what you get while an app is still starting, needs --retry-connrefused, and anything else needs --retry-all-errors (curl 7.71 or later). Even then curl cannot check the response body for the right version. A Bash loop around a single-attempt curl is easier to reason about, logs every attempt, and can check whatever "healthy" means for your app. (The 3-second runs are the three one-second retry delays.)

Where you'll use this

The toolkit maps directly onto how small teams deploy without a platform:

  • From CI. The pipeline builds and tests the artifact, copies it and its .sha256 to the server, and runs release.sh over SSH. The exit code decides the pipeline status: 0 green, 1 red ("rolled back, nothing to do but fix the build"), 3 red and page the on-call engineer. The Jenkins capstone builds exactly this kind of pipeline.
  • Several servers. Run release.sh on each server in turn behind a load balancer, and stop at the first failure. Because each server rolls itself back, a bad release never gets past the first one.
  • A real app. Replace appctl.sh with systemctl restart shop (the unit uses WorkingDirectory=/srv/shop/current), put secrets in shared/config, and point the health check at an endpoint that checks the database and other dependencies, not just "the process is up". The deploy, switch and rollback logic does not change.
  • Rollback during an incident. deploy.sh rollback && appctl.sh restart takes seconds and needs no build system, no network access to a registry, and no memory of what the previous version was. The history log answers "what changed and when".

Common mistakes

The app inherits your lock

To show why release.sh has {LOCK_FD}>&- on the lines that start the app, I ran a copy of it with those two redirections removed. The first release worked. The second one, with nothing else running, failed:

toolkit/release-nofdclose.sh artifacts/shop-1.1.0.tar.gz > /dev/null 2>&1; echo "exit: $?"
toolkit/release-nofdclose.sh artifacts/shop-1.1.0.tar.gz; echo "exit: $?"
lsof -nP srv/shop/.release.lock
exit: 0
2026-11-10T10:29:15-0500 release-nofdclose.sh[14653] ERROR: another run is in progress (lock: /home/sid/capstone/srv/shop/.release.lock)
exit: 5
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
python3 14630  sid   10w   REG   8,48        0 54918 srv/shop/.release.lock

Child processes inherit open file descriptors. The release script held the lock on a descriptor, started appctl.sh, which started the server, and the server kept descriptor 10 (the lock) open for its whole life. The flock lock lives as long as any process holds that open file, so every later release was locked out until the app restarted. lsof on the lock file names the culprit. The fix is to close the descriptor for the commands that start long-lived processes, "$TOOLS/appctl.sh" restart {LOCK_FD}>&-, and after that the lock is free as soon as release.sh exits:

toolkit/appctl.sh restart && flock -n srv/shop/.release.lock true && echo 'lock is free again'
2026-11-10T10:29:15-0500 appctl.sh[14663] INFO: stopped PID 14630
2026-11-10T10:29:15-0500 appctl.sh[14663] INFO: started PID 14675 on port 18480, serving releases/20261110T152914Z
lock is free again

The same applies to any script that takes a lock and then starts a daemon, and to other inherited descriptors such as sockets and log files.

Rolling back to "the next older directory"

My first cmd_rollback picked the newest release older than current, by sorting the directory names. That is correct only if current is always the newest release. While testing exercise 1 below (rolling back to a specific older release), current became the oldest release, and the next failed release rolled back to the wrong one:

2026-11-10T10:27:59-0500 ex3-release.sh[12031] INFO: releasing 1.3.0 (currently running: 1.0.0)
[…]
2026-11-10T10:28:00-0500 deploy.sh[12146] INFO: rolled back from 20261110T152759Z (1.3.0) to 20261110T152756Z (1.1.0)
[…]
2026-11-10T10:28:00-0500 healthcheck.sh[12187] WARN: attempt 1/4: body does not contain 'version=1.0.0': ok version=1.1.0
[…]
2026-11-10T10:28:03-0500 ex3-release.sh[12031] ERROR: rollback to 1.0.0 failed too; the site may be down
exit: 3

1.0.0 was live, the deploy of 1.3.0 failed, and the "rollback" put 1.1.0 live, a version that had been rolled back on purpose. The health check caught it only because it checks the version. The fix is in the final deploy.sh: record the release that was live before each deploy in .previous, roll back to that, and fall back to sort order only for a second rollback in a row. The general lesson: "previous" is a fact about history, so record it; don't reconstruct it from names.

Checking only that the port answers

A health check that accepts any HTTP 200 passes when the old process is still running, when a load balancer serves a cached page, or when the app answers with an error page that happens to return 200. Check something that only the new, working release can say: its version, and ideally the result of a real dependency check.

Summary

  • Deploy each release into its own releases/<timestamp> directory, unpacked under a staging name and renamed into place; keep config, logs and uploads in shared/ and link them in before going live.
  • Go live by renaming a new symlink over current (ln -sfn … current.new && mv -T current.new current): rename(2) is atomic, whichever ln the server has.
  • A running process keeps its old release (its working directory is the resolved path), so a deploy needs a restart, and the health check must verify the version.
  • Health checks: curl -fsS --max-time --connect-timeout in a Bash retry loop that checks the body. curl's own --retry skips "connection refused" and 404 unless told otherwise.
  • Verify artifacts (checksum, structure) before changing anything; record the previous release for rollback; never prune the current release or the rollback target; log every switch.
  • Close lock descriptors ({LOCK_FD}>&-) for commands that start long-lived processes.
  • Distinct exit codes turn the wrapper into a CI step: 0 released, 1 failed and rolled back, 3 failed and could not recover.

Exercises

Exercise 1: Roll back to a specific releasemedium

Extend deploy.sh rollback to accept an optional release id: deploy.sh rollback 20261110T152941Z. It must refuse an id that does not exist and the id that is already current, and behave as before when no id is given.

Show Solution

The changes, as a diff:

-  local current previous id
+  local target=${1:-} current previous id
   current=$(current_release)
   [[ -n $current ]] || die 1 "nothing is deployed"
-  # prefer the release that was live before the last deploy; fall back to
-  # the newest release older than current (e.g. for a second rollback)
-  previous=$(previous_release)
+  if [[ -n $target ]]; then
+    [[ -d $APP_ROOT/releases/$target ]] || die 1 "no such release: $target"
+    [[ $target != "$current" ]] || die 1 "$target is already the current release"
+  fi
+  # an explicit target wins; then the release that was live before the last
+  # deploy; then the newest release older than current
+  previous=${target:-$(previous_release)}
[…]
-              acquire_lock "$APP_ROOT/.deploy.lock"; cmd_rollback ;;
+              acquire_lock "$APP_ROOT/.deploy.lock"; cmd_rollback "${2:-}" ;;

With three releases deployed:

toolkit/ex1-deploy.sh list
toolkit/ex1-deploy.sh rollback 20991231T000000Z; echo "exit: $?"
toolkit/ex1-deploy.sh rollback 20261110T152944Z; echo "exit: $?"
toolkit/ex1-deploy.sh rollback 20261110T152941Z; echo "exit: $?"
toolkit/ex1-deploy.sh list
  20261110T152941Z  1.0.0
  20261110T152942Z  1.1.0
* 20261110T152944Z  1.1.0
2026-11-10T10:29:44-0500 ex1-deploy.sh[23212] ERROR: no such release: 20991231T000000Z
exit: 1
2026-11-10T10:29:44-0500 ex1-deploy.sh[23223] ERROR: 20261110T152944Z is already the current release
exit: 1
2026-11-10T10:29:44-0500 ex1-deploy.sh[23234] INFO: rolled back from 20261110T152944Z (1.1.0) to 20261110T152941Z (1.0.0)
exit: 0
* 20261110T152941Z  1.0.0
  20261110T152942Z  1.1.0

Note that current is now older than another release. This is exactly the state that broke the first version of the rollback logic (see common mistakes), and why .previous is recorded at deploy time.

Exercise 2: A response-time budgetmedium

Add -m SECONDS to healthcheck.sh: an attempt that succeeds but takes longer than the budget counts as a failure. A release that makes the health endpoint ten times slower should not go live.

Hint

curl -w '\n%{time_total}' appends the total time on its own line after the body. Bash cannot compare decimals, but awk can.

Show Solution

Add MAX_SECS="" to the defaults, m: to the getopts string with m) MAX_SECS=$OPTARG ;;, and replace the success branch of the loop:

    if out=$(curl -fsS --max-time "$TIMEOUT" --connect-timeout "$TIMEOUT" \
               -w '\n%{time_total}' "$url" 2>&1); then
      secs=${out##*$'\n'}       # last line: time_total from -w
      out=${out%$'\n'*}         # the body without that line...
      out=${out%$'\n'}          # ...and without the body's own final newline
      if [[ -n $MAX_SECS ]] && awk -v t="$secs" -v m="$MAX_SECS" 'BEGIN { exit !(t > m) }'; then
        warn "attempt $i/$RETRIES: too slow: ${secs}s (budget ${MAX_SECS}s)"
      elif [[ -n $EXPECT && $out != *"$EXPECT"* ]]; then
        warn "attempt $i/$RETRIES: body does not contain '$EXPECT': ${out:0:80}"
      else
        info "healthy after $i attempt(s) in ${secs}s: ${out:0:80}"
        return 0
      fi
    else
toolkit/ex2-healthcheck.sh -m 0.5 -e version=1.1.0 -r 2 -w 1 http://127.0.0.1:18480/healthz; echo "exit: $?"
toolkit/ex2-healthcheck.sh -m 0.00001 -r 2 -w 1 http://127.0.0.1:18480/healthz; echo "exit: $?"
2026-11-10T10:30:55-0500 ex2-healthcheck.sh[25074] INFO: healthy after 1 attempt(s) in 0.004652s: ok version=1.1.0
exit: 0
2026-11-10T10:30:55-0500 ex2-healthcheck.sh[25083] WARN: attempt 1/2: too slow: 0.000876s (budget 0.00001s)
2026-11-10T10:30:56-0500 ex2-healthcheck.sh[25083] WARN: attempt 2/2: too slow: 0.002302s (budget 0.00001s)
2026-11-10T10:30:56-0500 ex2-healthcheck.sh[25083] ERROR: http://127.0.0.1:18480/healthz still unhealthy after 2 attempts
exit: 1

The second call uses an impossible budget to exercise the failure path. My first version printed an empty line after the body, because the response's own trailing newline was left in out; the second out=${out%$'\n'} removes it. In practice, set the budget from measurements (a few times the normal response time), not a guess.

Exercise 3: Smoke tests as part of the gatehard

A release can have a perfect health endpoint and still be broken: the home page is missing, the CSS returns 404. Add -s FILE to release.sh: a file of URL paths (one per line, # comments allowed) that must all return 2xx after the health check passes. A smoke failure must trigger the same rollback as a health failure, and the rollback must be smoke-tested too. Test it with an artifact that has a valid healthz but no index.html.

Show Solution

Wrap both checks in one function and use it in both places:

SMOKE=""
 
# healthy VERSION: the health endpoint reports VERSION and every smoke path answers 2xx
healthy() {
  local path
  "$TOOLS/healthcheck.sh" -e "version=$1" -r 4 -w 1 -t 2 "http://127.0.0.1:$PORT/healthz" || return 1
  [[ -n $SMOKE ]] || return 0
  while read -r path; do
    if [[ -z $path || $path == \#* ]]; then continue; fi
    if ! curl -fsS -o /dev/null --max-time 5 "http://127.0.0.1:$PORT$path"; then
      error "smoke test failed: $path"
      return 1
    fi
    info "smoke test ok: $path"
  done < "$SMOKE"
}

Add s: to the getopts string with s) SMOKE=$OPTARG ;;, then replace the two healthcheck.sh calls with healthy "$version" and healthy "$previous".

cat smoke.txt
toolkit/ex3-release.sh -s smoke.txt artifacts/shop-1.3.0.tar.gz; echo "exit: $?"
# paths that must answer 2xx after every release
/index.html
/healthz
2026-11-10T10:29:45-0500 deploy.sh[23323] INFO: shop-1.3.0.tar.gz looks good
2026-11-10T10:29:45-0500 ex3-release.sh[23318] INFO: releasing 1.3.0 (currently running: 1.0.0)
2026-11-10T10:29:45-0500 deploy.sh[23357] INFO: deployed 1.3.0 as release 20261110T152945Z (previous: 20261110T152941Z)
2026-11-10T10:29:45-0500 appctl.sh[23399] INFO: stopped PID 23267
2026-11-10T10:29:45-0500 appctl.sh[23399] INFO: started PID 23411 on port 18480, serving releases/20261110T152945Z
2026-11-10T10:29:46-0500 healthcheck.sh[23421] INFO: healthy after 1 attempt(s): ok version=1.3.0
curl: (22) The requested URL returned error: 404
2026-11-10T10:29:46-0500 ex3-release.sh[23318] ERROR: smoke test failed: /index.html
2026-11-10T10:29:46-0500 ex3-release.sh[23318] ERROR: release 1.3.0 failed its checks, rolling back
2026-11-10T10:29:46-0500 deploy.sh[23435] INFO: rolled back from 20261110T152945Z (1.3.0) to 20261110T152941Z (1.0.0)
2026-11-10T10:29:46-0500 appctl.sh[23454] INFO: stopped PID 23411
2026-11-10T10:29:46-0500 appctl.sh[23454] INFO: started PID 23466 on port 18480, serving releases/20261110T152941Z
2026-11-10T10:29:46-0500 healthcheck.sh[23476] INFO: healthy after 1 attempt(s): ok version=1.0.0
2026-11-10T10:29:46-0500 ex3-release.sh[23318] INFO: smoke test ok: /index.html
2026-11-10T10:29:46-0500 ex3-release.sh[23318] INFO: smoke test ok: /healthz
2026-11-10T10:29:46-0500 ex3-release.sh[23318] ERROR: rolled back to 1.0.0; 1.3.0 was NOT released
exit: 1

The health endpoint said 1.3.0 was fine; the smoke test caught the missing page and the release was rolled back to 1.0.0, the release that was actually live before (here, after exercise 1's targeted rollback, the recorded .previous mattered). A good release passes both:

toolkit/ex3-release.sh -s smoke.txt artifacts/shop-1.1.0.tar.gz; echo "exit: $?"
2026-11-10T10:29:46-0500 deploy.sh[23499] INFO: shop-1.1.0.tar.gz looks good
2026-11-10T10:29:46-0500 ex3-release.sh[23494] INFO: releasing 1.1.0 (currently running: 1.0.0)
2026-11-10T10:29:46-0500 deploy.sh[23533] INFO: deployed 1.1.0 as release 20261110T152946Z (previous: 20261110T152941Z)
2026-11-10T10:29:46-0500 appctl.sh[23575] INFO: stopped PID 23466
2026-11-10T10:29:46-0500 appctl.sh[23575] INFO: started PID 23587 on port 18480, serving releases/20261110T152946Z
2026-11-10T10:29:46-0500 healthcheck.sh[23597] INFO: healthy after 1 attempt(s): ok version=1.1.0
2026-11-10T10:29:46-0500 ex3-release.sh[23494] INFO: smoke test ok: /index.html
2026-11-10T10:29:46-0500 ex3-release.sh[23494] INFO: smoke test ok: /healthz
2026-11-10T10:29:46-0500 ex3-release.sh[23494] INFO: release 1.1.0 is live
exit: 0

Keep the smoke list short and fast (a handful of pages that exercise templates, static assets and one database read). It runs on every release, including the emergency ones.


Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), curl 8.5.0, Python 3.12 (http.server as the stand-in app), GNU coreutils 9.4, ShellCheck 0.9.0, November 2026. Every release, rollback and failure shown was run end to end on the lab machine in one session, against a local application root; nothing was deployed to a real server and the CI and multi-server usage was not tested. The sort-order rollback bug described above was found while testing and is fixed in the listings; the inherited-lock failure was reproduced on purpose with a modified copy of release.sh. I also tried to demonstrate the ln -sfn versus rename difference with a race test (a reader checking current/VERSION millions of times while a loop flipped the link); on this WSL kernel both methods produced a few failed lookups out of millions, so the test settled nothing and is not shown. Paths from the test machine's scratch directories are shown shortened to /home/sid/….

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.