Capstone: A Build, Test, and Deploy Pipeline

A complete Jenkins delivery pipeline run end to end: tests with JUnit reports, a versioned artifact, staging, approval, production, automatic rollback, locks and notifications.

By The Dark DebuggerUpdated 31 min read

This capstone puts the whole Jenkins track together in one pipeline that builds, tests, packages and deploys a small web service through staging and production. It runs for real, including a release that breaks staging and rolls itself back. It uses multibranch and webhooks from Multibranch Pipelines and Webhooks, when, input, post and stash from Declarative Pipeline in Depth, and a folder-scoped secret as covered in Credentials and Secrets. Read those first if any of the pieces below look unfamiliar.

What we're building

Here is the brief for this pipeline, and the shape of it is common in real teams. A small team ships inventory-service, an HTTP API that reports stock levels. They want:

  • every commit on every branch built and unit-tested, with test results Jenkins can chart;
  • one versioned, fingerprinted artifact per build, the exact file that later gets deployed;
  • main deployed to staging automatically, with a smoke test against the running service;
  • a person to approve production, with a deadline, so builds don't wait forever;
  • a failed smoke test to put the previous release back on its own;
  • never two deploys to the same environment at once, and never an older build overtaking a newer one;
  • a chat message for every outcome.
StageRuns onWhat it does
Buildevery branchByte-compiles the app and stamps VERSION.BUILD_NUMBER into it
Unit testsevery branchpytest with a JUnit XML report, recorded by the junit step
Packageevery branchBuilds dist/inventory-service-X.Y.Z.tar.gz, archives and fingerprints it
Deploy to stagingmainUnder a lock: install, restart, smoke test, roll back on failure
Approve productionmaininput with a 30-minute timeout, no executor held
Deploy to productionmainUnder a lock: same deploy and smoke test, rollback on failure
PostalwaysChat notification for success, failure, abort and superseded builds

The application and the repository

The service is plain Python with no dependencies. server.py answers /health and /items/<sku>, and stock.py holds the logic. The repository looks like this:

.
├── .gitignore
├── Jenkinsfile
├── VERSION
├── app
│   ├── server.py
│   └── stock.py
├── ci
│   ├── build.sh
│   ├── deploy.sh
│   ├── environments.sh
│   ├── package.sh
│   ├── rollback.sh
│   ├── service.sh
│   └── smoke-test.sh
├── pytest.ini
└── tests
    └── test_stock.py

VERSION holds the major and minor version (1.4), and the build number becomes the patch, so build 1 is 1.4.1. pytest.ini points the tests at the app directory:

[pytest]
pythonpath = app
testpaths = tests

/health returns the running version. That one field is what lets a smoke test prove that the new release is the one answering, and not the old process that never stopped:

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            self.reply(200, {"status": "ok", "version": VERSION, "skus": len(stock.STOCK)})

Agent prerequisites. The agent needs python3, curl, jq, tar and pytest. On a normal Ubuntu agent that's apt install python3-pytest jq curl. My lab agent has no root access, so pytest 9.2.0 lives in a virtualenv, and the Jenkinsfile points at it through the PYTEST variable.

The "hosts". Staging and production are two directories on the lab machine, each running the service on its own port (18081 and 18082). In real life they'd be two servers reached over SSH. The scripts are written so that's a small change, as Where you'll use this shows.

The deployment scripts

My rule for pipelines: the Jenkinsfile decides what happens and when, and scripts in the repository do the work. You can run scripts on your laptop, shellcheck them, and reuse them from a manual rollback job. Shell code buried inside Groovy strings gets none of that.

ci/environments.sh is sourced by every other script, so environment details live in one place:

# environments.sh ENV - where each environment lives (sourced by the other scripts)
case "$1" in
  staging)    HOST_DIR="$DEPLOY_ROOT/staging";    PORT=18081 ;;
  production) HOST_DIR="$DEPLOY_ROOT/production"; PORT=18082 ;;
  *) echo "unknown environment: $1" >&2; exit 2 ;;
esac

ci/build.sh and ci/package.sh stamp the version and build the artifact. The package script lists what went into the tarball, and that turns out to matter:

#!/usr/bin/env bash
# build.sh VERSION - byte-compile the app and stamp the version into it
set -euo pipefail
version="$1"
python3 -m py_compile app/*.py
echo "$version" > app/VERSION
echo "built inventory-service $version"
#!/usr/bin/env bash
# package.sh VERSION - build the release tarball and its checksum in dist/
set -euo pipefail
version="$1"
tarball="dist/inventory-service-$version.tar.gz"
mkdir -p dist
tar -czf "$tarball" app/*.py app/VERSION
sha256sum "$tarball" > "$tarball.sha256"
echo "packaged $tarball:"
tar -tzf "$tarball" | sed 's/^/  /'

ci/deploy.sh keeps every release in its own directory and switches a current symlink. Before switching, it records the old target in previous, so a rollback is one ln:

#!/usr/bin/env bash
# deploy.sh ENV VERSION - install a release on the environment's host and restart the service
# Layout: $DEPLOY_ROOT/ENV/releases/VERSION, current -> releases/VERSION, previous = last current
set -euo pipefail
env="$1"; version="$2"
source "$(dirname "$0")/environments.sh" "$env"
mkdir -p "$HOST_DIR/releases/$version"
tar -xzf "dist/inventory-service-$version.tar.gz" -C "$HOST_DIR/releases/$version"
if [ -L "$HOST_DIR/current" ]; then
  readlink "$HOST_DIR/current" > "$HOST_DIR/previous"
fi
ln -sfn "releases/$version" "$HOST_DIR/current"
bash "$(dirname "$0")/service.sh" restart "$env"
echo "deployed $version to $env (previous: $(cat "$HOST_DIR/previous" 2>/dev/null || echo none))"

ci/service.sh stands in for systemctl restart. Note the JENKINS_NODE_COOKIE line. The Common mistakes section shows what happens without it:

#!/usr/bin/env bash
# service.sh start|stop|restart ENV - run the environment's current release in the background
set -euo pipefail
action="$1"
source "$(dirname "$0")/environments.sh" "$2"
pidfile="$HOST_DIR/service.pid"
 
stop() {
  if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then
    kill "$(cat "$pidfile")"
    for _ in 1 2 3 4 5; do kill -0 "$(cat "$pidfile")" 2>/dev/null || break; sleep 1; done
  fi
  rm -f "$pidfile"
}
 
start() {
  # JENKINS_NODE_COOKIE=dontKillMe: without it Jenkins kills the service when the build ends
  JENKINS_NODE_COOKIE=dontKillMe nohup python3 "$HOST_DIR/current/app/server.py" "$PORT" \
    >> "$HOST_DIR/service.log" 2>&1 < /dev/null &
  echo $! > "$pidfile"
}
 
case "$action" in
  start) start ;;
  stop) stop ;;
  restart) stop; start ;;
  *) echo "usage: service.sh start|stop|restart ENV" >&2; exit 2 ;;
esac

ci/smoke-test.sh retries while the service starts, then checks both status and version. If the service never answers, it prints the end of the service log. When the smoke test fails, that log is the first thing anyone asks for:

#!/usr/bin/env bash
# smoke-test.sh ENV VERSION - the running service must report status ok and the expected version
set -euo pipefail
env="$1"; expected="$2"
source "$(dirname "$0")/environments.sh" "$env"
url="http://127.0.0.1:$PORT/health"
if ! body=$(curl -fsS --retry 5 --retry-connrefused --retry-delay 1 --max-time 5 "$url"); then
  echo "smoke test FAILED: $url did not answer; last lines of service.log:"
  tail -n 4 "$HOST_DIR/service.log"
  exit 1
fi
echo "$url -> $body"
if ! jq -e --arg v "$expected" '.status == "ok" and .version == $v' <<<"$body" > /dev/null; then
  echo "smoke test FAILED: expected version $expected"
  exit 1
fi
echo "smoke test passed: $env is running $expected"

ci/rollback.sh points current back at previous, restarts the service, and runs the smoke test against the old version. A rollback you don't verify is just a hope:

#!/usr/bin/env bash
# rollback.sh ENV - point "current" back at the previous release, restart, and verify it
set -euo pipefail
env="$1"
source "$(dirname "$0")/environments.sh" "$env"
if [ ! -s "$HOST_DIR/previous" ]; then
  echo "rollback: no previous release recorded for $env" >&2
  exit 1
fi
failed=$(readlink "$HOST_DIR/current")
previous=$(cat "$HOST_DIR/previous")
ln -sfn "$previous" "$HOST_DIR/current"
bash "$(dirname "$0")/service.sh" restart "$env"
echo "rolled back $env: $failed -> $previous"
bash "$(dirname "$0")/smoke-test.sh" "$env" "${previous#releases/}"

The Jenkinsfile

This is the complete Jenkinsfile, exactly as it ran in every build in this article. jcli declarative-linter < Jenkinsfile printed Jenkinsfile successfully validated. before the first push.

pipeline {
    agent none
    options {
        buildDiscarder(logRotator(numToKeepStr: '30', artifactNumToKeepStr: '10'))
    }
    environment {
        DEPLOY_ROOT = '/home/sid/hosts'           // where "staging" and "production" live
        PYTEST      = '/home/sid/tools/py/bin/pytest' // pytest provisioned on the agent
    }
    stages {
        stage('Build and test') {
            agent any
            stages {
                stage('Build') {
                    steps {
                        script {
                            env.VERSION = "${readFile('VERSION').trim()}.${env.BUILD_NUMBER}"
                        }
                        sh 'bash ci/build.sh "$VERSION"'
                    }
                }
                stage('Unit tests') {
                    steps {
                        sh '"$PYTEST" --junitxml=reports/junit.xml'
                    }
                    post {
                        always {
                            junit 'reports/junit.xml'
                        }
                    }
                }
                stage('Package') {
                    steps {
                        sh 'bash ci/package.sh "$VERSION"'
                        archiveArtifacts artifacts: 'dist/*', fingerprint: true
                        stash name: 'release', includes: 'dist/*'
                    }
                }
                stage('Deploy to staging') {
                    when { branch 'main' }
                    options { lock('inventory-staging') }
                    steps {
                        milestone(1)
                        sh 'bash ci/deploy.sh staging "$VERSION"'
                        sh 'bash ci/smoke-test.sh staging "$VERSION"'
                    }
                    post {
                        failure {
                            sh 'bash ci/rollback.sh staging'
                        }
                    }
                }
            }
            post {
                cleanup {
                    cleanWs()
                }
            }
        }
        stage('Approve production') {
            when {
                branch 'main'
                beforeInput true
            }
            options { timeout(time: 30, unit: 'MINUTES') }
            input {
                id 'Promote'
                message 'Deploy this build to production?'
                ok 'Deploy'
                submitter 'admin'
                submitterParameter 'APPROVER'
            }
            steps {
                milestone(2)
                echo "Production deploy of ${env.VERSION} approved by ${APPROVER}"
            }
        }
        stage('Deploy to production') {
            when { branch 'main' }
            agent any
            options { lock('inventory-production') }
            steps {
                unstash 'release'
                sh 'bash ci/deploy.sh production "$VERSION"'
                sh 'bash ci/smoke-test.sh production "$VERSION"'
            }
            post {
                failure {
                    sh 'bash ci/rollback.sh production'
                }
                cleanup {
                    cleanWs()
                }
            }
        }
    }
    post {
        success {
            script { notifyChat("${env.JOB_NAME} ${env.VERSION}: deployed (${env.BUILD_URL})") }
        }
        failure {
            script { notifyChat("${env.JOB_NAME} ${env.VERSION}: FAILED, see ${env.BUILD_URL}console") }
        }
        aborted {
            script { notifyChat("${env.JOB_NAME} ${env.VERSION}: aborted (rejected or approval timed out)") }
        }
        notBuilt {
            script { notifyChat("${env.JOB_NAME} ${env.VERSION}: superseded by a newer build, not deployed") }
        }
    }
}
 
// Post a Slack-style {"text": "..."} message to the team chat webhook.
// A failed notification is reported in the log but never fails the build.
void notifyChat(String text) {
    node {
        withCredentials([string(credentialsId: 'chat-webhook', variable: 'CHAT_WEBHOOK_URL')]) {
            withEnv(["CHAT_TEXT=${text}"]) {
                int rc = sh(returnStatus: true, script: '''
                    jq -cn --arg text "$CHAT_TEXT" '{text: $text}' |
                      curl -fsS --max-time 10 -H 'Content-Type: application/json' -d @- "$CHAT_WEBHOOK_URL"
                ''')
                if (rc != 0) {
                    echo "chat notification failed (exit ${rc})"
                }
            }
        }
    }
}

The decisions that matter, from top to bottom:

  • agent none at the top, agents per stage. The approval stage has no agent. A build waiting for a human holds no executor, while a top-level agent any would tie one up for the whole 30 minutes. The four CI stages share one agent through a parent stage, so they share one workspace.
  • The version is computed once, in Build, and stored in env.VERSION. Every later stage, the artifact name and the smoke test use the same value. The smoke test can therefore prove which build is running.
  • junit sits in post { always } so failed tests are still recorded. That's the build where you most need the report.
  • archiveArtifacts ... fingerprint: true keeps the tarball and its checksum with the build, and records an MD5 fingerprint. Jenkins can then tell you which builds produced or used that exact file. stash carries the same file to the production stage, which may run in a different workspace or on a different agent. Production deploys the artifact that was tested. It never rebuilds.
  • Two locks, one per environment, so a staging deploy never waits for a production deploy. The lock step creates the resource on first use, so nothing has to be configured beforehand.
  • milestone(1) and milestone(2) stop an older build from overtaking a newer one. The concurrency section below shows them working.
  • post { failure { rollback } } on each deploy stage. If the deploy or the smoke test fails, the rollback runs in the same workspace, under the same lock.
  • when { beforeInput true } evaluates the branch condition before showing the approval prompt. Without it, a feature-branch build would ask someone to approve a production deploy, and only then skip it.
  • notifyChat never fails the build. It uses returnStatus: true. A chat outage shouldn't turn a successful production deploy red.

Setting it up

Everything lives in one folder, so the chat webhook can be a folder-scoped credential that other teams' jobs can't read. I created the folder and the secret with the CLI. In the UI the secret goes in the folder's Credentials page. jcli is the CLI wrapper from the multibranch article:

<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>chat-webhook</id>
  <description>Team chat incoming webhook URL</description>
  <secret>http://127.0.0.1:18089/hooks/T0INVENTORY/B0DEPLOYS/lab-secret-path</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>

The folder itself needs nothing more than a description:

<?xml version='1.1' encoding='UTF-8'?>
<com.cloudbees.hudson.plugins.folder.Folder>
  <description>Delivery pipelines for the inventory service</description>
</com.cloudbees.hudson.plugins.folder.Folder>
jcli create-job delivery < folder.xml
jcli create-credentials-by-xml folder::item::/delivery _ < cred.xml
jcli list-credentials folder::item::/delivery
================================================
Domain            (global)
Description       
# of Credentials  1
================================================
Id                Name
================= ==============================
chat-webhook Team chat incoming webhook URL
================================================

The webhook URL points at a 19-line Python receiver on the lab machine. It writes each POSTed body to a file and stands in for a chat service's incoming webhook, which takes the same {"text": "..."} JSON. Then I created the multibranch project delivery/inventory-service with the same config.xml and post-receive hook as in the multibranch article (with this repository's URL), so every push starts a scan.

Run 1: a clean release

I committed version 1.4 and pushed it. The hook triggered a scan, and the scan built main #1. Here are the CI stages:

[Pipeline] { (Build)
[Pipeline] script
[Pipeline] {
[Pipeline] readFile
[Pipeline] }
[Pipeline] // script
[Pipeline] sh
+ bash ci/build.sh 1.4.1
built inventory-service 1.4.1
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Unit tests)
[Pipeline] sh
+ /home/sid/tools/py/bin/pytest --junitxml=reports/junit.xml
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.2.0, pluggy-1.7.0
rootdir: /home/sid/.jenkins/workspace/_delivery_inventory-service_main
configfile: pytest.ini
testpaths: tests
collected 3 items
 
tests/test_stock.py ...                                                  [100%]
 
- generated xml file: /home/sid/.jenkins/workspace/_delivery_inventory-service_main/reports/junit.xml -
============================== 3 passed in 0.02s ===============================
Post stage
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[…]
[Pipeline] { (Package)
[Pipeline] sh
+ bash ci/package.sh 1.4.1
packaged dist/inventory-service-1.4.1.tar.gz:
  app/server.py
  app/stock.py
  app/VERSION
[Pipeline] archiveArtifacts
Archiving artifacts
Recording fingerprints
[Pipeline] stash
Stashed 2 file(s)
[…]
[Pipeline] { (Deploy to staging)
[Pipeline] lock
Trying to acquire lock on [Resource: inventory-staging]
Resource [inventory-staging] did not exist. Created.
Lock acquired on [Resource: inventory-staging], resources: [inventory-staging]
[Pipeline] {
[Pipeline] milestone
Trying to pass milestone 1
[Pipeline] sh
+ bash ci/deploy.sh staging 1.4.1
deployed 1.4.1 to staging (previous: none)
[Pipeline] sh
+ bash ci/smoke-test.sh staging 1.4.1
http://127.0.0.1:18081/health -> {"status": "ok", "version": "1.4.1", "skus": 3}
smoke test passed: staging is running 1.4.1
[Pipeline] }
Lock released on resource [Resource: inventory-staging], resources: [inventory-staging]
[…]
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done

"[Checks API] No suitable checks publisher found." is harmless. It means no GitHub-style checks integration is configured. The REST API confirms what Jenkins recorded for this build:

curl -s -u "admin:$JENKINS_PASSWORD" \
  "http://localhost:8090/job/delivery/job/inventory-service/job/main/1/testReport/api/json?tree=passCount,failCount,skipCount"
curl -s -u "admin:$JENKINS_PASSWORD" \
  "http://localhost:8090/job/delivery/job/inventory-service/job/main/1/api/json?tree=artifacts%5BrelativePath%5D" | jq -c .
{"_class":"hudson.tasks.junit.TestResult","failCount":0,"passCount":3,"skipCount":0}
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","artifacts":[{"relativePath":"dist/inventory-service-1.4.1.tar.gz"},{"relativePath":"dist/inventory-service-1.4.1.tar.gz.sha256"}]}

Then the build stopped at the approval stage. Normally someone clicks Deploy on the build page or in the stage view. From a script, you can list pending input with the wfapi endpoint that the stage view uses:

curl -s -u "admin:$JENKINS_PASSWORD" \
  http://localhost:8090/job/delivery/job/inventory-service/job/main/1/wfapi/pendingInputActions | jq .
[
  {
    "id": "Promote",
    "proceedText": "Deploy",
    "message": "Deploy this build to production?",
    "inputs": [],
    "proceedUrl": "/job/delivery/job/inventory-service/job/main/1/wfapi/inputSubmit?inputId=Promote",
    "abortUrl": "/job/delivery/job/inventory-service/job/main/1/input/Promote/abort",
    "redirectApprovalUrl": "/job/delivery/job/inventory-service/job/main/1/input/"
  }
]

I approved it with a POST to the input's proceedEmpty URL. That needs a CSRF crumb when you log in with a password, as described in the multibranch article:

CRUMB=$(curl -s -c cookies.txt -u "admin:$JENKINS_PASSWORD" \
  http://localhost:8090/crumbIssuer/api/json | jq -r '.crumbRequestField + ":" + .crumb')
curl -s -b cookies.txt -u "admin:$JENKINS_PASSWORD" -H "$CRUMB" -X POST -o /dev/null -w "approve HTTP %{http_code}\n" \
  http://localhost:8090/job/delivery/job/inventory-service/job/main/1/input/Promote/proceedEmpty
approve HTTP 200

The rest of build #1:

[Pipeline] { (Approve production)
[Pipeline] timeout
Timeout set to expire in 30 min
[Pipeline] {
[Pipeline] input
Deploy this build to production?
Deploy or Abort
Approved by admin
[Pipeline] withEnv
[Pipeline] {
[Pipeline] milestone
Trying to pass milestone 2
[Pipeline] echo
Production deploy of 1.4.1 approved by admin
[…]
[Pipeline] { (Deploy to production)
[Pipeline] lock
Trying to acquire lock on [Resource: inventory-production]
Resource [inventory-production] did not exist. Created.
Lock acquired on [Resource: inventory-production], resources: [inventory-production]
[Pipeline] {
[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/_delivery_inventory-service_main
[…]
[Pipeline] unstash
[Pipeline] sh
+ bash ci/deploy.sh production 1.4.1
deployed 1.4.1 to production (previous: none)
[Pipeline] sh
+ bash ci/smoke-test.sh production 1.4.1
http://127.0.0.1:18082/health -> {"status": "ok", "version": "1.4.1", "skus": 3}
smoke test passed: production is running 1.4.1
[…]
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/_delivery_inventory-service_main
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $CHAT_WEBHOOK_URL
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ jq -cn --arg text delivery/inventory-service/main 1.4.1: deployed (http://localhost:8090/job/delivery/job/inventory-service/job/main/1/) {text: $text}
+ curl -fsS --max-time 10 -H Content-Type: application/json -d @- ****
ok
[…]
Finished: SUCCESS

Notice the order in the production stage: the lock was acquired before the node. A stage-level lock option wraps the agent allocation, so a build waiting for the production lock doesn't sit on an executor. Also notice **** where the webhook URL would be. withCredentials masks the secret in the log, even inside set -x output.

Run 2: a broken release rolls itself back

Version 1.5 moves the stock data out of the code into app/catalog.json:

"""Stock levels for the warehouse, loaded from catalog.json next to this file."""
import json
import os
 
with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "catalog.json")) as f:
    STOCK = json.load(f)
git add -A
git commit -q -m "Load stock levels from catalog.json; version 1.5"
git show --stat --oneline HEAD
git push origin main
4165a94 Load stock levels from catalog.json; version 1.5
 VERSION          | 2 +-
 app/catalog.json | 7 +++++++
 app/stock.py     | 7 +++++--
 3 files changed, 13 insertions(+), 3 deletions(-)

The unit tests run in the repository, where catalog.json sits next to stock.py, so they passed (3 passed in 0.01s). But look at what got packaged, and at what happened in staging:

+ bash ci/package.sh 1.5.2
packaged dist/inventory-service-1.5.2.tar.gz:
  app/server.py
  app/stock.py
  app/VERSION
[…]
[Pipeline] { (Deploy to staging)
[Pipeline] lock
Trying to acquire lock on [Resource: inventory-staging]
Resource [inventory-staging] did not exist. Created.
Lock acquired on [Resource: inventory-staging], resources: [inventory-staging]
[Pipeline] {
[Pipeline] milestone
Trying to pass milestone 1
[Pipeline] sh
+ bash ci/deploy.sh staging 1.5.2
deployed 1.5.2 to staging (previous: releases/1.4.1)
[Pipeline] sh
+ bash ci/smoke-test.sh staging 1.5.2
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
smoke test FAILED: http://127.0.0.1:18081/health did not answer; last lines of service.log:
  File "/home/sid/hosts/staging/releases/1.5.2/app/stock.py", line 5, in <module>
    with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "catalog.json")) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/sid/hosts/staging/releases/1.5.2/app/catalog.json'
Post stage
[Pipeline] sh
+ bash ci/rollback.sh staging
rolled back staging: releases/1.5.2 -> releases/1.4.1
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
http://127.0.0.1:18081/health -> {"status": "ok", "version": "1.4.1", "skus": 3}
smoke test passed: staging is running 1.4.1
[Pipeline] }
Lock released on resource [Resource: inventory-staging], resources: [inventory-staging]
[…]
[Pipeline] { (Approve production)
Stage "Approve production" skipped due to earlier failure(s)
[…]
[Pipeline] { (Deploy to production)
Stage "Deploy to production" skipped due to earlier failure(s)
[…]
+ jq -cn --arg text delivery/inventory-service/main 1.5.2: FAILED, see http://localhost:8090/job/delivery/job/inventory-service/job/main/2/console {text: $text}
+ curl -fsS --max-time 10 -H Content-Type: application/json -d @- ****
ok
[…]
ERROR: script returned exit code 1
Finished: FAILURE

Everything here happened without anyone touching it:

  1. The tarball was missing catalog.json. package.sh packed app/*.py, which was fine until the app had a file that wasn't Python. The file list in the log shows it.
  2. The new process crashed on import. The smoke test's six connection attempts failed, and the script printed the traceback from service.log into the build log.
  3. post { failure } on the staging stage ran rollback.sh. That switched current back to 1.4.1, restarted the service, and verified 1.4.1 with the same smoke test. (The one refused connection is the restarted process still starting up. --retry covered it.)
  4. Approval and production were skipped, and production never saw 1.5.2.
  5. The team got {"text":"delivery/inventory-service/main 1.5.2: FAILED, see http://localhost:8090/job/delivery/job/inventory-service/job/main/2/console"}.

The state of both environments right after the build:

staging current: releases/1.4.1
production current: releases/1.4.1
{"status": "ok", "version": "1.4.1", "skus": 3}
{"status": "ok", "version": "1.4.1", "skus": 3}

The fix is one line in package.sh: ship the whole app directory, minus caches:

-tar -czf "$tarball" app/*.py app/VERSION
+tar -czf "$tarball" --exclude=__pycache__ app

Build #3 packaged the data file, passed staging, waited for approval, and went to production:

+ bash ci/package.sh 1.5.3
packaged dist/inventory-service-1.5.3.tar.gz:
  app/
  app/server.py
  app/catalog.json
  app/stock.py
  app/VERSION
[…]
deployed 1.5.3 to staging (previous: releases/1.4.1)
[Pipeline] sh
+ bash ci/smoke-test.sh staging 1.5.3
http://127.0.0.1:18081/health -> {"status": "ok", "version": "1.5.3", "skus": 5}
smoke test passed: staging is running 1.5.3
[…]
Approved by admin
[…]
Production deploy of 1.5.3 approved by admin
curl -s http://127.0.0.1:18082/health
curl -s http://127.0.0.1:18082/items/SKU-1005
{"status": "ok", "version": "1.5.3", "skus": 5}
{"sku": "SKU-1005", "qty": 8}

Unit tests can't catch "the artifact is missing a file", because they don't run the artifact. That's the smoke test's job, and it's why the smoke test runs against the deployed package, not the workspace.

Preventing concurrent deploys: locks and milestones

Two things can go wrong when builds overlap. Two deploys to the same environment can interleave, so one restarts the service halfway through the other's smoke test. Or an older build can be approved after a newer one and put old code into production. The lock and milestone steps handle one each.

To see the lock in action, I gave staging a second user: a data-refresh job that holds the same lock for 40 seconds, as a nightly job might:

// Nightly data refresh: holds the staging environment while it runs
lock('inventory-staging') {
    echo 'Refreshing staging data from the production snapshot...'
    sleep time: 40, unit: 'SECONDS'
    echo 'Refresh finished'
}

I started it, then started two builds of main (#4 and #5) a couple of seconds apart. Both waited for the refresh instead of deploying into the middle of it. This is #4:

[Pipeline] lock
Trying to acquire lock on [Resource: inventory-staging]
The resource [inventory-staging] is locked by build delivery » staging-data-refresh #1 #1 since Nov 10, 2026, 11:02 AM.
[Resource: inventory-staging] is not free, waiting for execution ...
Lock acquired on [Resource: inventory-staging], resources: [inventory-staging]
[Pipeline] milestone
Trying to pass milestone 1
deployed 1.5.4 to staging (previous: releases/1.5.3)
smoke test passed: staging is running 1.5.4
Lock released on resource [Resource: inventory-staging], resources: [inventory-staging]

#5 queued behind it and deployed 1.5.5 to staging once #4 let go. Both then waited at the approval prompt. I approved only the newer one, #5. When it passed milestone(2), Jenkins ended #4, which was still waiting at the prompt:

[Pipeline] { (Approve production)
[Pipeline] timeout
Timeout set to expire in 30 min
[Pipeline] input
Deploy this build to production?
Deploy or Abort
Superseded by delivery/inventory-service/main#5
[…]
[Pipeline] { (Deploy to production)
Stage "Deploy to production" skipped due to earlier failure(s)
[…]
+ jq -cn --arg text delivery/inventory-service/main 1.5.4: superseded by a newer build, not deployed {text: $text}
[…]
Superseded by delivery/inventory-service/main#5
Already interrupted
Finished: NOT_BUILT

Without the milestone, someone could approve #4 later and roll production back from 1.5.5 to 1.5.4 without realising. The rule a milestone enforces: once a build passes milestone N, any older build that hasn't passed it yet is cancelled.

Note the result: NOT_BUILT, not ABORTED. My first version of the Jenkinsfile only had an aborted post condition, and the superseded build sent no notification at all. That's why the final version has a separate notBuilt block.

Notifications

The chat stand-in received one message per outcome over the runs above:

{"text":"delivery/inventory-service/main 1.4.1: deployed (http://localhost:8090/job/delivery/job/inventory-service/job/main/1/)"}
{"text":"delivery/inventory-service/main 1.5.2: FAILED, see http://localhost:8090/job/delivery/job/inventory-service/job/main/2/console"}
{"text":"delivery/inventory-service/main 1.5.3: deployed (http://localhost:8090/job/delivery/job/inventory-service/job/main/3/)"}
{"text":"delivery/inventory-service/main 1.5.4: superseded by a newer build, not deployed"}
{"text":"delivery/inventory-service/main 1.5.5: deployed (http://localhost:8090/job/delivery/job/inventory-service/job/main/5/)"}

A plain HTTPS POST of {"text": ...} is what Slack and Mattermost incoming webhooks accept, so notifyChat works unchanged with a real webhook URL in the credential. Other chat tools want slightly different JSON, and only the jq line needs to change. If you'd rather use plugins: the Slack Notification plugin provides slackSend channel: '#deploys', message: '...', and Pipeline's mail step sends mail to: 'team@example.com', subject: '...', body: '...'. I didn't run either. Slack's plugin isn't installed in the lab, and mail needs an SMTP server configured under Manage Jenkins > System. Wherever the message goes, keep it short and include the version and a link to the build. That's all anyone needs at 2 a.m.

Where you'll use this

This pipeline is a scaled-down copy of how many teams deploy services to virtual machines. To move it from the lab to real servers, change the scripts, not the Jenkinsfile:

  • Hosts become SSH targets. environments.sh maps staging to deploy@staging-01.internal instead of a directory. deploy.sh copies the tarball with scp and runs the unpack and symlink switch with ssh. The Jenkinsfile wraps those steps in sshagent(['deploy-key']) from the SSH Agent plugin (not installed in my lab), and the key is a credential, as in the credentials article.
  • service.sh becomes systemctl restart inventory-service. systemd keeps the process alive, so the JENKINS_NODE_COOKIE trick is no longer needed.
  • Several servers per environment means a loop in deploy.sh. For a rolling deploy, take one server out of the load balancer, deploy, smoke test, put it back, then do the next. The lock still covers the whole environment.
  • Deploy agents. Production credentials should only exist on a dedicated agent: agent { label 'deploy-prod' } on the production stage. The agents article covers labels.
  • Containers and Kubernetes. The package stage builds and pushes an image tagged with VERSION instead of a tarball. deploy.sh runs kubectl set image and kubectl rollout status, and rollback.sh runs kubectl rollout undo. The pipeline shape (test, versioned artifact, staging with smoke test, approval, production, rollback, locks, milestones) doesn't change.
  • Many services. Once three services have this Jenkinsfile, move it into a shared library template. Each repository then keeps its ci/ scripts and a three-line Jenkinsfile.

Common mistakes

The service dies when the build finishes. Jenkins kills every process a build started when the build ends. It finds them by an environment variable it sets, JENKINS_NODE_COOKIE in Pipeline jobs. I ran this job:

node {
    sh 'nohup python3 -m http.server 18085 --bind 127.0.0.1 > http.log 2>&1 &'
    sh 'sleep 1; curl -s -o /dev/null -w "inside the build: HTTP %{http_code}\\n" http://127.0.0.1:18085/'
}
inside the build: HTTP 200

Two seconds after the build finished, from a normal shell:

curl -s -o /dev/null -w "after the build: HTTP %{http_code}\n" http://127.0.0.1:18085/
after the build: HTTP 000

nohup doesn't help: the process was killed, not hung up. With JENKINS_NODE_COOKIE=dontKillMe in front of the same command, the second check printed after the build: HTTP 200. Use that for lab setups like this one. For real services, let a service manager (systemd, a container runtime) own the process, so the build only asks for a restart.

Approving or scanning over REST gets a 403. With a password login, every POST needs a CSRF crumb:

curl -s -i -u "admin:$JENKINS_PASSWORD" -X POST \
  http://localhost:8090/job/delivery/job/inventory-service/job/main/6/input/Promote/proceedEmpty \
  | grep -E '^HTTP|<title>'
HTTP/1.1 403 Forbidden
<title>Error 403 No valid crumb was included in the request</title>

Fetch a crumb from /crumbIssuer/api/json with a cookie jar, as shown in Run 1, or log in with an API token instead, which doesn't need one.

Rebuilding in the production stage. It's tempting to run package.sh again in the production stage, "it's the same commit". But that produces a different file from the one staging tested, and the fingerprint no longer links the two. stash/unstash (or copyArtifacts from the build) deploys the tested bytes.

Only handling aborted. A build cancelled by a milestone ends as NOT_BUILT, and a pipeline that only has aborted and failure notifications says nothing about it. I hit exactly this on my first attempt. If a build can end in a way you didn't report, someone will eventually wonder where their release went.

Trusting the unit tests to cover deployment. Run 2 passed every unit test and still couldn't start. A deploy without a smoke test only proves that files were copied.

Summary

  • Keep orchestration in the Jenkinsfile and the work in versioned scripts: build, package, deploy, smoke test, rollback. Scripts can be run, linted and reused outside Jenkins.
  • Compute the version once and carry it everywhere: artifact name, deploy directory, and the /health value the smoke test checks.
  • Record tests with junit in post { always }. Archive and fingerprint the artifact, and deploy that exact file to every environment with stash/unstash.
  • Use agent none at the top so the input approval holds no executor. Give it a timeout, a submitter, and when { beforeInput true }.
  • Put a post { failure } rollback on every deploy stage, and verify the rollback with the same smoke test.
  • Use one lock per environment to serialise deploys (and maintenance jobs). Use milestone so an older build can never be approved after a newer one. Superseded builds end as NOT_BUILT.
  • Notify on every outcome (success, failure, aborted, notBuilt) without letting a notification failure fail the build. Keep webhook URLs in credentials.

Exercises

Exercise 1: What happens when nobody approves?easy

Without waiting 30 minutes, find out what result a build gets when the approval times out, and whether its aborted post block runs. Use a short pipeline with the same approval stage and a 20-second timeout.

Show Solution
pipeline {
    agent none
    stages {
        stage('Approve production') {
            options { timeout(time: 20, unit: 'SECONDS') }
            input {
                id 'Promote'
                message 'Deploy this build to production?'
                ok 'Deploy'
            }
            steps {
                echo 'approved'
            }
        }
    }
    post {
        aborted {
            echo 'Nobody approved in time; production was not touched.'
        }
    }
}
[Pipeline] { (Approve production)
[Pipeline] timeout
Timeout set to expire in 20 sec
[Pipeline] {
[Pipeline] input
Deploy this build to production?
Deploy or Abort
Cancelling nested steps due to timeout
[…]
[Pipeline] { (Declarative: Post Actions)
[Pipeline] echo
Nobody approved in time; production was not touched.
[Pipeline] End of Pipeline
Timeout has been exceeded
[…]
Finished: ABORTED

The stage-level timeout option wraps the input directive, so an unanswered approval ends the build as ABORTED, and the aborted post block runs. In the capstone pipeline, that sends the "aborted (rejected or approval timed out)" message.

Exercise 2: Reject a releaseeasy

Start a build of main, wait for the approval prompt, and reject it through the REST API instead of approving. What is the build result, what does production run afterwards, and what did the chat receive?

Show Solution

The prompt's abortUrl from pendingInputActions is .../input/Promote/abort. I POSTed to it with a crumb, the same way as the approval in Run 1:

curl -s -b cookies.txt -u "admin:$JENKINS_PASSWORD" -H "$CRUMB" -X POST -o /dev/null -w "abort HTTP %{http_code}\n" \
  http://localhost:8090/job/delivery/job/inventory-service/job/main/6/input/Promote/abort
abort HTTP 200

The end of build #6:

[Pipeline] input
Deploy this build to production?
Deploy or Abort
[…]
Stage "Deploy to production" skipped due to earlier failure(s)
[…]
Rejected by admin
[…]
Finished: ABORTED

Production was untouched, and the chat got the abort message:

production current: releases/1.5.5
{"status": "ok", "version": "1.5.5", "skus": 5}
{"text":"delivery/inventory-service/main 1.5.6: aborted (rejected or approval timed out)"}

Staging, however, now runs 1.5.6. Rejecting a production deploy doesn't undo staging. Staging is meant to hold the newest candidate.

Exercise 3: A deeper smoke testmedium

/health passing doesn't prove the data loaded correctly. Extend smoke-test.sh so it also looks up SKU-1001 and fails unless the reply contains that SKU and a numeric quantity. Test your version against the running production service.

Show Solution

Insert this before the final echo "smoke test passed..." line:

item=$(curl -fsS --max-time 5 "http://127.0.0.1:$PORT/items/SKU-1001")
echo "/items/SKU-1001 -> $item"
if ! jq -e '.sku == "SKU-1001" and (.qty | type) == "number"' <<<"$item" > /dev/null; then
  echo "smoke test FAILED: stock lookup is broken"
  exit 1
fi

Run against production (1.5.5 at that point):

bash ci/smoke-test.sh production 1.5.5; echo "exit=$?"
http://127.0.0.1:18082/health -> {"status": "ok", "version": "1.5.5", "skus": 5}
/items/SKU-1001 -> {"sku": "SKU-1001", "qty": 42}
smoke test passed: production is running 1.5.5
exit=0

I checked the type rather than the value 42. A smoke test should check that the system works, not pin down production data that changes every day.

Exercise 4: A manual rollback jobhard

Production sometimes needs to go back to a release that isn't the previous one. Write a parameterized Pipeline job, rollback-production, that takes a RELEASE parameter, refuses unknown releases (listing the ones available), switches production to the chosen one under the production lock, and verifies it with the smoke test. Run it once with no parameter and once with RELEASE=1.5.3.

Hint

Reuse ci/service.sh and ci/smoke-test.sh by checking out the repository. And watch what happens on the very first build of a job whose parameters are defined in its Jenkinsfile.

Show Solution
pipeline {
    agent any
    parameters {
        string(name: 'RELEASE', defaultValue: '', description: 'Release to put back into production, e.g. 1.4.1')
    }
    options { lock('inventory-production') }
    environment {
        DEPLOY_ROOT = '/home/sid/hosts'
    }
    stages {
        stage('Roll back production') {
            steps {
                git url: 'file:///home/sid/git/inventory-service.git', branch: 'main'
                sh '''
                    releases="$DEPLOY_ROOT/production/releases"
                    if [ -z "$RELEASE" ] || [ ! -d "$releases/$RELEASE" ]; then
                      echo "unknown release '$RELEASE'; available: $(ls "$releases" | tr '\\n' ' ')"
                      exit 1
                    fi
                    readlink "$DEPLOY_ROOT/production/current" > "$DEPLOY_ROOT/production/previous"
                    ln -sfn "releases/$RELEASE" "$DEPLOY_ROOT/production/current"
                    bash ci/service.sh restart production
                    bash ci/smoke-test.sh production "$RELEASE"
                '''
            }
        }
    }
}

The first build ran before Jenkins knew about the parameter, which is defined by running the Jenkinsfile once. So RELEASE was empty, and the guard did its job:

+ releases=/home/sid/hosts/production/releases
+ [ -z  ]
+ ls /home/sid/hosts/production/releases
+ tr 
  
+ echo unknown release ''; available: 1.4.1 1.5.3 1.5.5 
unknown release ''; available: 1.4.1 1.5.3 1.5.5 
+ exit 1
Lock released on resource [Resource: inventory-production], resources: [inventory-production]
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

The second build, with jcli build delivery/rollback-production -p RELEASE=1.5.3 -s -v:

+ releases=/home/sid/hosts/production/releases
+ [ -z 1.5.3 ]
+ [ ! -d /home/sid/hosts/production/releases/1.5.3 ]
+ readlink /home/sid/hosts/production/current
+ ln -sfn releases/1.5.3 /home/sid/hosts/production/current
+ bash ci/service.sh restart production
+ bash ci/smoke-test.sh production 1.5.3
curl: (7) Failed to connect to 127.0.0.1 port 18082 after 0 ms: Couldn't connect to server
http://127.0.0.1:18082/health -> {"status": "ok", "version": "1.5.3", "skus": 5}
smoke test passed: production is running 1.5.3
Lock released on resource [Resource: inventory-production], resources: [inventory-production]
[Pipeline] End of Pipeline
Finished: SUCCESS

Three details make this safe. The job takes the same inventory-production lock as the pipeline, so a rollback can't collide with a deploy. It writes previous before switching, so the rollback can itself be rolled back with rollback.sh. And it ends with the same smoke test as every deploy.


Tested with: Jenkins 2.580.2 (OpenJDK 21) on Ubuntu 24.04 (WSL2), Pipeline: Declarative 2.2293.v6e7193cec599, Pipeline: Input Step 568.v0c9db_2652741, Pipeline: Milestone Step 152.v6e22b_8cfc66c, Lockable Resources 1588.v7b_b_614f4202e, JUnit 1458.v38661c0638e2, Workspace Cleanup 0.49, Python 3.12.3, pytest 9.2.0, November 2026. All six main builds, the data-refresh lock holder, the approval-timeout job, the rollback job and the process-killing demo ran on the lab controller, and every log above is from those runs, trimmed where marked […]. Approvals and the rejection were sent through the REST API rather than by clicking Deploy and Abort. "Staging" and "production" were directories and ports on the lab machine rather than separate servers, and the chat service was a local stand-in for an incoming webhook. The slackSend and mail alternatives were not run. Paths from the test machine's scratch directories are shown shortened to /home/sid/…. Job, node and credential names are shown without the test run's prefixes.

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Jenkins tutorial here is tested end to end before publishing.