Capstone: A Build, Test, and Deploy Pipeline
A complete Jenkins delivery pipeline run end to end: tests with JUnit reports, a versioned artifact, staging, approval, production, automatic rollback, locks and notifications.
By The Dark DebuggerUpdated 31 min read
This capstone puts the whole Jenkins track together in one pipeline that builds, tests,
packages and deploys a small web service through staging and production. It runs for real,
including a release that breaks staging and rolls itself back. It uses multibranch and
webhooks from Multibranch Pipelines and Webhooks,
when, input, post and stash from
Declarative Pipeline in Depth, and a
folder-scoped secret as covered in Credentials and Secrets.
Read those first if any of the pieces below look unfamiliar.
What we're building
Here is the brief for this pipeline, and the shape of it is common in real teams. A small
team ships inventory-service, an HTTP API that reports stock levels. They want:
- every commit on every branch built and unit-tested, with test results Jenkins can chart;
- one versioned, fingerprinted artifact per build, the exact file that later gets deployed;
maindeployed to staging automatically, with a smoke test against the running service;- a person to approve production, with a deadline, so builds don't wait forever;
- a failed smoke test to put the previous release back on its own;
- never two deploys to the same environment at once, and never an older build overtaking a newer one;
- a chat message for every outcome.
| Stage | Runs on | What it does |
|---|---|---|
| Build | every branch | Byte-compiles the app and stamps VERSION.BUILD_NUMBER into it |
| Unit tests | every branch | pytest with a JUnit XML report, recorded by the junit step |
| Package | every branch | Builds dist/inventory-service-X.Y.Z.tar.gz, archives and fingerprints it |
| Deploy to staging | main | Under a lock: install, restart, smoke test, roll back on failure |
| Approve production | main | input with a 30-minute timeout, no executor held |
| Deploy to production | main | Under a lock: same deploy and smoke test, rollback on failure |
| Post | always | Chat notification for success, failure, abort and superseded builds |
The application and the repository
The service is plain Python with no dependencies. server.py answers /health and
/items/<sku>, and stock.py holds the logic. The repository looks like this:
.
├── .gitignore
├── Jenkinsfile
├── VERSION
├── app
│ ├── server.py
│ └── stock.py
├── ci
│ ├── build.sh
│ ├── deploy.sh
│ ├── environments.sh
│ ├── package.sh
│ ├── rollback.sh
│ ├── service.sh
│ └── smoke-test.sh
├── pytest.ini
└── tests
└── test_stock.pyVERSION holds the major and minor version (1.4), and the build number becomes the
patch, so build 1 is 1.4.1. pytest.ini points the tests at the app directory:
[pytest]
pythonpath = app
testpaths = tests/health returns the running version. That one field is what lets a smoke test prove that
the new release is the one answering, and not the old process that never stopped:
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/health":
self.reply(200, {"status": "ok", "version": VERSION, "skus": len(stock.STOCK)})Agent prerequisites. The agent needs python3, curl, jq, tar and pytest. On a
normal Ubuntu agent that's apt install python3-pytest jq curl. My lab agent has no root
access, so pytest 9.2.0 lives in a virtualenv, and the Jenkinsfile points at it through
the PYTEST variable.
The "hosts". Staging and production are two directories on the lab machine, each running the service on its own port (18081 and 18082). In real life they'd be two servers reached over SSH. The scripts are written so that's a small change, as Where you'll use this shows.
The deployment scripts
My rule for pipelines: the Jenkinsfile decides what happens and when, and scripts in the
repository do the work. You can run scripts on your laptop, shellcheck them, and reuse
them from a manual rollback job. Shell code buried inside Groovy strings gets none of that.
ci/environments.sh is sourced by every other script, so environment details live in one
place:
# environments.sh ENV - where each environment lives (sourced by the other scripts)
case "$1" in
staging) HOST_DIR="$DEPLOY_ROOT/staging"; PORT=18081 ;;
production) HOST_DIR="$DEPLOY_ROOT/production"; PORT=18082 ;;
*) echo "unknown environment: $1" >&2; exit 2 ;;
esacci/build.sh and ci/package.sh stamp the version and build the artifact. The package
script lists what went into the tarball, and that turns out to matter:
#!/usr/bin/env bash
# build.sh VERSION - byte-compile the app and stamp the version into it
set -euo pipefail
version="$1"
python3 -m py_compile app/*.py
echo "$version" > app/VERSION
echo "built inventory-service $version"#!/usr/bin/env bash
# package.sh VERSION - build the release tarball and its checksum in dist/
set -euo pipefail
version="$1"
tarball="dist/inventory-service-$version.tar.gz"
mkdir -p dist
tar -czf "$tarball" app/*.py app/VERSION
sha256sum "$tarball" > "$tarball.sha256"
echo "packaged $tarball:"
tar -tzf "$tarball" | sed 's/^/ /'ci/deploy.sh keeps every release in its own directory and switches a current symlink.
Before switching, it records the old target in previous, so a rollback is one ln:
#!/usr/bin/env bash
# deploy.sh ENV VERSION - install a release on the environment's host and restart the service
# Layout: $DEPLOY_ROOT/ENV/releases/VERSION, current -> releases/VERSION, previous = last current
set -euo pipefail
env="$1"; version="$2"
source "$(dirname "$0")/environments.sh" "$env"
mkdir -p "$HOST_DIR/releases/$version"
tar -xzf "dist/inventory-service-$version.tar.gz" -C "$HOST_DIR/releases/$version"
if [ -L "$HOST_DIR/current" ]; then
readlink "$HOST_DIR/current" > "$HOST_DIR/previous"
fi
ln -sfn "releases/$version" "$HOST_DIR/current"
bash "$(dirname "$0")/service.sh" restart "$env"
echo "deployed $version to $env (previous: $(cat "$HOST_DIR/previous" 2>/dev/null || echo none))"ci/service.sh stands in for systemctl restart. Note the JENKINS_NODE_COOKIE line. The
Common mistakes section shows what happens without it:
#!/usr/bin/env bash
# service.sh start|stop|restart ENV - run the environment's current release in the background
set -euo pipefail
action="$1"
source "$(dirname "$0")/environments.sh" "$2"
pidfile="$HOST_DIR/service.pid"
stop() {
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then
kill "$(cat "$pidfile")"
for _ in 1 2 3 4 5; do kill -0 "$(cat "$pidfile")" 2>/dev/null || break; sleep 1; done
fi
rm -f "$pidfile"
}
start() {
# JENKINS_NODE_COOKIE=dontKillMe: without it Jenkins kills the service when the build ends
JENKINS_NODE_COOKIE=dontKillMe nohup python3 "$HOST_DIR/current/app/server.py" "$PORT" \
>> "$HOST_DIR/service.log" 2>&1 < /dev/null &
echo $! > "$pidfile"
}
case "$action" in
start) start ;;
stop) stop ;;
restart) stop; start ;;
*) echo "usage: service.sh start|stop|restart ENV" >&2; exit 2 ;;
esacci/smoke-test.sh retries while the service starts, then checks both status and
version. If the service never answers, it prints the end of the service log. When the
smoke test fails, that log is the first thing anyone asks for:
#!/usr/bin/env bash
# smoke-test.sh ENV VERSION - the running service must report status ok and the expected version
set -euo pipefail
env="$1"; expected="$2"
source "$(dirname "$0")/environments.sh" "$env"
url="http://127.0.0.1:$PORT/health"
if ! body=$(curl -fsS --retry 5 --retry-connrefused --retry-delay 1 --max-time 5 "$url"); then
echo "smoke test FAILED: $url did not answer; last lines of service.log:"
tail -n 4 "$HOST_DIR/service.log"
exit 1
fi
echo "$url -> $body"
if ! jq -e --arg v "$expected" '.status == "ok" and .version == $v' <<<"$body" > /dev/null; then
echo "smoke test FAILED: expected version $expected"
exit 1
fi
echo "smoke test passed: $env is running $expected"ci/rollback.sh points current back at previous, restarts the service, and runs the
smoke test against the old version. A rollback you don't verify is just a hope:
#!/usr/bin/env bash
# rollback.sh ENV - point "current" back at the previous release, restart, and verify it
set -euo pipefail
env="$1"
source "$(dirname "$0")/environments.sh" "$env"
if [ ! -s "$HOST_DIR/previous" ]; then
echo "rollback: no previous release recorded for $env" >&2
exit 1
fi
failed=$(readlink "$HOST_DIR/current")
previous=$(cat "$HOST_DIR/previous")
ln -sfn "$previous" "$HOST_DIR/current"
bash "$(dirname "$0")/service.sh" restart "$env"
echo "rolled back $env: $failed -> $previous"
bash "$(dirname "$0")/smoke-test.sh" "$env" "${previous#releases/}"The Jenkinsfile
This is the complete Jenkinsfile, exactly as it ran in every build in this article.
jcli declarative-linter < Jenkinsfile printed Jenkinsfile successfully validated.
before the first push.
pipeline {
agent none
options {
buildDiscarder(logRotator(numToKeepStr: '30', artifactNumToKeepStr: '10'))
}
environment {
DEPLOY_ROOT = '/home/sid/hosts' // where "staging" and "production" live
PYTEST = '/home/sid/tools/py/bin/pytest' // pytest provisioned on the agent
}
stages {
stage('Build and test') {
agent any
stages {
stage('Build') {
steps {
script {
env.VERSION = "${readFile('VERSION').trim()}.${env.BUILD_NUMBER}"
}
sh 'bash ci/build.sh "$VERSION"'
}
}
stage('Unit tests') {
steps {
sh '"$PYTEST" --junitxml=reports/junit.xml'
}
post {
always {
junit 'reports/junit.xml'
}
}
}
stage('Package') {
steps {
sh 'bash ci/package.sh "$VERSION"'
archiveArtifacts artifacts: 'dist/*', fingerprint: true
stash name: 'release', includes: 'dist/*'
}
}
stage('Deploy to staging') {
when { branch 'main' }
options { lock('inventory-staging') }
steps {
milestone(1)
sh 'bash ci/deploy.sh staging "$VERSION"'
sh 'bash ci/smoke-test.sh staging "$VERSION"'
}
post {
failure {
sh 'bash ci/rollback.sh staging'
}
}
}
}
post {
cleanup {
cleanWs()
}
}
}
stage('Approve production') {
when {
branch 'main'
beforeInput true
}
options { timeout(time: 30, unit: 'MINUTES') }
input {
id 'Promote'
message 'Deploy this build to production?'
ok 'Deploy'
submitter 'admin'
submitterParameter 'APPROVER'
}
steps {
milestone(2)
echo "Production deploy of ${env.VERSION} approved by ${APPROVER}"
}
}
stage('Deploy to production') {
when { branch 'main' }
agent any
options { lock('inventory-production') }
steps {
unstash 'release'
sh 'bash ci/deploy.sh production "$VERSION"'
sh 'bash ci/smoke-test.sh production "$VERSION"'
}
post {
failure {
sh 'bash ci/rollback.sh production'
}
cleanup {
cleanWs()
}
}
}
}
post {
success {
script { notifyChat("${env.JOB_NAME} ${env.VERSION}: deployed (${env.BUILD_URL})") }
}
failure {
script { notifyChat("${env.JOB_NAME} ${env.VERSION}: FAILED, see ${env.BUILD_URL}console") }
}
aborted {
script { notifyChat("${env.JOB_NAME} ${env.VERSION}: aborted (rejected or approval timed out)") }
}
notBuilt {
script { notifyChat("${env.JOB_NAME} ${env.VERSION}: superseded by a newer build, not deployed") }
}
}
}
// Post a Slack-style {"text": "..."} message to the team chat webhook.
// A failed notification is reported in the log but never fails the build.
void notifyChat(String text) {
node {
withCredentials([string(credentialsId: 'chat-webhook', variable: 'CHAT_WEBHOOK_URL')]) {
withEnv(["CHAT_TEXT=${text}"]) {
int rc = sh(returnStatus: true, script: '''
jq -cn --arg text "$CHAT_TEXT" '{text: $text}' |
curl -fsS --max-time 10 -H 'Content-Type: application/json' -d @- "$CHAT_WEBHOOK_URL"
''')
if (rc != 0) {
echo "chat notification failed (exit ${rc})"
}
}
}
}
}The decisions that matter, from top to bottom:
agent noneat the top, agents per stage. The approval stage has no agent. A build waiting for a human holds no executor, while a top-levelagent anywould tie one up for the whole 30 minutes. The four CI stages share one agent through a parent stage, so they share one workspace.- The version is computed once, in
Build, and stored inenv.VERSION. Every later stage, the artifact name and the smoke test use the same value. The smoke test can therefore prove which build is running. junitsits inpost { always }so failed tests are still recorded. That's the build where you most need the report.archiveArtifacts ... fingerprint: truekeeps the tarball and its checksum with the build, and records an MD5 fingerprint. Jenkins can then tell you which builds produced or used that exact file.stashcarries the same file to the production stage, which may run in a different workspace or on a different agent. Production deploys the artifact that was tested. It never rebuilds.- Two locks, one per environment, so a staging deploy never waits for a production
deploy. The
lockstep creates the resource on first use, so nothing has to be configured beforehand. milestone(1)andmilestone(2)stop an older build from overtaking a newer one. The concurrency section below shows them working.post { failure { rollback } }on each deploy stage. If the deploy or the smoke test fails, the rollback runs in the same workspace, under the same lock.when { beforeInput true }evaluates the branch condition before showing the approval prompt. Without it, a feature-branch build would ask someone to approve a production deploy, and only then skip it.notifyChatnever fails the build. It usesreturnStatus: true. A chat outage shouldn't turn a successful production deploy red.
Setting it up
Everything lives in one folder, so the chat webhook can be a folder-scoped credential
that other teams' jobs can't read. I created the folder and the secret with the CLI. In
the UI the secret goes in the folder's Credentials page. jcli is the CLI wrapper
from the multibranch article:
<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
<scope>GLOBAL</scope>
<id>chat-webhook</id>
<description>Team chat incoming webhook URL</description>
<secret>http://127.0.0.1:18089/hooks/T0INVENTORY/B0DEPLOYS/lab-secret-path</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>The folder itself needs nothing more than a description:
<?xml version='1.1' encoding='UTF-8'?>
<com.cloudbees.hudson.plugins.folder.Folder>
<description>Delivery pipelines for the inventory service</description>
</com.cloudbees.hudson.plugins.folder.Folder>jcli create-job delivery < folder.xml
jcli create-credentials-by-xml folder::item::/delivery _ < cred.xml
jcli list-credentials folder::item::/delivery================================================
Domain (global)
Description
# of Credentials 1
================================================
Id Name
================= ==============================
chat-webhook Team chat incoming webhook URL
================================================The webhook URL points at a 19-line Python receiver on the lab machine. It writes each
POSTed body to a file and stands in for a chat service's incoming webhook, which takes the
same {"text": "..."} JSON. Then I created the multibranch project
delivery/inventory-service with the same config.xml and post-receive hook as in
the multibranch article (with this repository's URL), so every push starts a scan.
Run 1: a clean release
I committed version 1.4 and pushed it. The hook triggered a scan, and the scan built
main #1. Here are the CI stages:
[Pipeline] { (Build)
[Pipeline] script
[Pipeline] {
[Pipeline] readFile
[Pipeline] }
[Pipeline] // script
[Pipeline] sh
+ bash ci/build.sh 1.4.1
built inventory-service 1.4.1
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Unit tests)
[Pipeline] sh
+ /home/sid/tools/py/bin/pytest --junitxml=reports/junit.xml
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.2.0, pluggy-1.7.0
rootdir: /home/sid/.jenkins/workspace/_delivery_inventory-service_main
configfile: pytest.ini
testpaths: tests
collected 3 items
tests/test_stock.py ... [100%]
- generated xml file: /home/sid/.jenkins/workspace/_delivery_inventory-service_main/reports/junit.xml -
============================== 3 passed in 0.02s ===============================
Post stage
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[…]
[Pipeline] { (Package)
[Pipeline] sh
+ bash ci/package.sh 1.4.1
packaged dist/inventory-service-1.4.1.tar.gz:
app/server.py
app/stock.py
app/VERSION
[Pipeline] archiveArtifacts
Archiving artifacts
Recording fingerprints
[Pipeline] stash
Stashed 2 file(s)
[…]
[Pipeline] { (Deploy to staging)
[Pipeline] lock
Trying to acquire lock on [Resource: inventory-staging]
Resource [inventory-staging] did not exist. Created.
Lock acquired on [Resource: inventory-staging], resources: [inventory-staging]
[Pipeline] {
[Pipeline] milestone
Trying to pass milestone 1
[Pipeline] sh
+ bash ci/deploy.sh staging 1.4.1
deployed 1.4.1 to staging (previous: none)
[Pipeline] sh
+ bash ci/smoke-test.sh staging 1.4.1
http://127.0.0.1:18081/health -> {"status": "ok", "version": "1.4.1", "skus": 3}
smoke test passed: staging is running 1.4.1
[Pipeline] }
Lock released on resource [Resource: inventory-staging], resources: [inventory-staging]
[…]
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done"[Checks API] No suitable checks publisher found." is harmless. It means no GitHub-style checks integration is configured. The REST API confirms what Jenkins recorded for this build:
curl -s -u "admin:$JENKINS_PASSWORD" \
"http://localhost:8090/job/delivery/job/inventory-service/job/main/1/testReport/api/json?tree=passCount,failCount,skipCount"
curl -s -u "admin:$JENKINS_PASSWORD" \
"http://localhost:8090/job/delivery/job/inventory-service/job/main/1/api/json?tree=artifacts%5BrelativePath%5D" | jq -c .{"_class":"hudson.tasks.junit.TestResult","failCount":0,"passCount":3,"skipCount":0}
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","artifacts":[{"relativePath":"dist/inventory-service-1.4.1.tar.gz"},{"relativePath":"dist/inventory-service-1.4.1.tar.gz.sha256"}]}Then the build stopped at the approval stage. Normally someone clicks Deploy on the
build page or in the stage view. From a script, you can list pending input with the
wfapi endpoint that the stage view uses:
curl -s -u "admin:$JENKINS_PASSWORD" \
http://localhost:8090/job/delivery/job/inventory-service/job/main/1/wfapi/pendingInputActions | jq .[
{
"id": "Promote",
"proceedText": "Deploy",
"message": "Deploy this build to production?",
"inputs": [],
"proceedUrl": "/job/delivery/job/inventory-service/job/main/1/wfapi/inputSubmit?inputId=Promote",
"abortUrl": "/job/delivery/job/inventory-service/job/main/1/input/Promote/abort",
"redirectApprovalUrl": "/job/delivery/job/inventory-service/job/main/1/input/"
}
]I approved it with a POST to the input's proceedEmpty URL. That needs a CSRF crumb when
you log in with a password, as described in the multibranch article:
CRUMB=$(curl -s -c cookies.txt -u "admin:$JENKINS_PASSWORD" \
http://localhost:8090/crumbIssuer/api/json | jq -r '.crumbRequestField + ":" + .crumb')
curl -s -b cookies.txt -u "admin:$JENKINS_PASSWORD" -H "$CRUMB" -X POST -o /dev/null -w "approve HTTP %{http_code}\n" \
http://localhost:8090/job/delivery/job/inventory-service/job/main/1/input/Promote/proceedEmptyapprove HTTP 200The rest of build #1:
[Pipeline] { (Approve production)
[Pipeline] timeout
Timeout set to expire in 30 min
[Pipeline] {
[Pipeline] input
Deploy this build to production?
Deploy or Abort
Approved by admin
[Pipeline] withEnv
[Pipeline] {
[Pipeline] milestone
Trying to pass milestone 2
[Pipeline] echo
Production deploy of 1.4.1 approved by admin
[…]
[Pipeline] { (Deploy to production)
[Pipeline] lock
Trying to acquire lock on [Resource: inventory-production]
Resource [inventory-production] did not exist. Created.
Lock acquired on [Resource: inventory-production], resources: [inventory-production]
[Pipeline] {
[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/_delivery_inventory-service_main
[…]
[Pipeline] unstash
[Pipeline] sh
+ bash ci/deploy.sh production 1.4.1
deployed 1.4.1 to production (previous: none)
[Pipeline] sh
+ bash ci/smoke-test.sh production 1.4.1
http://127.0.0.1:18082/health -> {"status": "ok", "version": "1.4.1", "skus": 3}
smoke test passed: production is running 1.4.1
[…]
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/_delivery_inventory-service_main
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $CHAT_WEBHOOK_URL
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ jq -cn --arg text delivery/inventory-service/main 1.4.1: deployed (http://localhost:8090/job/delivery/job/inventory-service/job/main/1/) {text: $text}
+ curl -fsS --max-time 10 -H Content-Type: application/json -d @- ****
ok
[…]
Finished: SUCCESSNotice the order in the production stage: the lock was acquired before the node. A
stage-level lock option wraps the agent allocation, so a build waiting for the
production lock doesn't sit on an executor. Also notice **** where the webhook URL
would be. withCredentials masks the secret in the log, even inside set -x output.
Run 2: a broken release rolls itself back
Version 1.5 moves the stock data out of the code into app/catalog.json:
"""Stock levels for the warehouse, loaded from catalog.json next to this file."""
import json
import os
with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "catalog.json")) as f:
STOCK = json.load(f)git add -A
git commit -q -m "Load stock levels from catalog.json; version 1.5"
git show --stat --oneline HEAD
git push origin main4165a94 Load stock levels from catalog.json; version 1.5
VERSION | 2 +-
app/catalog.json | 7 +++++++
app/stock.py | 7 +++++--
3 files changed, 13 insertions(+), 3 deletions(-)The unit tests run in the repository, where catalog.json sits next to stock.py, so
they passed (3 passed in 0.01s). But look at what got packaged, and at what happened in
staging:
+ bash ci/package.sh 1.5.2
packaged dist/inventory-service-1.5.2.tar.gz:
app/server.py
app/stock.py
app/VERSION
[…]
[Pipeline] { (Deploy to staging)
[Pipeline] lock
Trying to acquire lock on [Resource: inventory-staging]
Resource [inventory-staging] did not exist. Created.
Lock acquired on [Resource: inventory-staging], resources: [inventory-staging]
[Pipeline] {
[Pipeline] milestone
Trying to pass milestone 1
[Pipeline] sh
+ bash ci/deploy.sh staging 1.5.2
deployed 1.5.2 to staging (previous: releases/1.4.1)
[Pipeline] sh
+ bash ci/smoke-test.sh staging 1.5.2
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
smoke test FAILED: http://127.0.0.1:18081/health did not answer; last lines of service.log:
File "/home/sid/hosts/staging/releases/1.5.2/app/stock.py", line 5, in <module>
with open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "catalog.json")) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/sid/hosts/staging/releases/1.5.2/app/catalog.json'
Post stage
[Pipeline] sh
+ bash ci/rollback.sh staging
rolled back staging: releases/1.5.2 -> releases/1.4.1
curl: (7) Failed to connect to 127.0.0.1 port 18081 after 0 ms: Couldn't connect to server
http://127.0.0.1:18081/health -> {"status": "ok", "version": "1.4.1", "skus": 3}
smoke test passed: staging is running 1.4.1
[Pipeline] }
Lock released on resource [Resource: inventory-staging], resources: [inventory-staging]
[…]
[Pipeline] { (Approve production)
Stage "Approve production" skipped due to earlier failure(s)
[…]
[Pipeline] { (Deploy to production)
Stage "Deploy to production" skipped due to earlier failure(s)
[…]
+ jq -cn --arg text delivery/inventory-service/main 1.5.2: FAILED, see http://localhost:8090/job/delivery/job/inventory-service/job/main/2/console {text: $text}
+ curl -fsS --max-time 10 -H Content-Type: application/json -d @- ****
ok
[…]
ERROR: script returned exit code 1
Finished: FAILUREEverything here happened without anyone touching it:
- The tarball was missing
catalog.json.package.shpackedapp/*.py, which was fine until the app had a file that wasn't Python. The file list in the log shows it. - The new process crashed on import. The smoke test's six connection attempts failed, and
the script printed the traceback from
service.loginto the build log. post { failure }on the staging stage ranrollback.sh. That switchedcurrentback to 1.4.1, restarted the service, and verified 1.4.1 with the same smoke test. (The one refused connection is the restarted process still starting up.--retrycovered it.)- Approval and production were skipped, and production never saw 1.5.2.
- The team got
{"text":"delivery/inventory-service/main 1.5.2: FAILED, see http://localhost:8090/job/delivery/job/inventory-service/job/main/2/console"}.
The state of both environments right after the build:
staging current: releases/1.4.1
production current: releases/1.4.1
{"status": "ok", "version": "1.4.1", "skus": 3}
{"status": "ok", "version": "1.4.1", "skus": 3}The fix is one line in package.sh: ship the whole app directory, minus caches:
-tar -czf "$tarball" app/*.py app/VERSION
+tar -czf "$tarball" --exclude=__pycache__ appBuild #3 packaged the data file, passed staging, waited for approval, and went to production:
+ bash ci/package.sh 1.5.3
packaged dist/inventory-service-1.5.3.tar.gz:
app/
app/server.py
app/catalog.json
app/stock.py
app/VERSION
[…]
deployed 1.5.3 to staging (previous: releases/1.4.1)
[Pipeline] sh
+ bash ci/smoke-test.sh staging 1.5.3
http://127.0.0.1:18081/health -> {"status": "ok", "version": "1.5.3", "skus": 5}
smoke test passed: staging is running 1.5.3
[…]
Approved by admin
[…]
Production deploy of 1.5.3 approved by admincurl -s http://127.0.0.1:18082/health
curl -s http://127.0.0.1:18082/items/SKU-1005{"status": "ok", "version": "1.5.3", "skus": 5}
{"sku": "SKU-1005", "qty": 8}Unit tests can't catch "the artifact is missing a file", because they don't run the artifact. That's the smoke test's job, and it's why the smoke test runs against the deployed package, not the workspace.
Preventing concurrent deploys: locks and milestones
Two things can go wrong when builds overlap. Two deploys to the same environment can
interleave, so one restarts the service halfway through the other's smoke test. Or an
older build can be approved after a newer one and put old code into production. The
lock and milestone steps handle one each.
To see the lock in action, I gave staging a second user: a data-refresh job that holds the same lock for 40 seconds, as a nightly job might:
// Nightly data refresh: holds the staging environment while it runs
lock('inventory-staging') {
echo 'Refreshing staging data from the production snapshot...'
sleep time: 40, unit: 'SECONDS'
echo 'Refresh finished'
}I started it, then started two builds of main (#4 and #5) a couple of seconds apart.
Both waited for the refresh instead of deploying into the middle of it. This is #4:
[Pipeline] lock
Trying to acquire lock on [Resource: inventory-staging]
The resource [inventory-staging] is locked by build delivery » staging-data-refresh #1 #1 since Nov 10, 2026, 11:02 AM.
[Resource: inventory-staging] is not free, waiting for execution ...
Lock acquired on [Resource: inventory-staging], resources: [inventory-staging]
[Pipeline] milestone
Trying to pass milestone 1
deployed 1.5.4 to staging (previous: releases/1.5.3)
smoke test passed: staging is running 1.5.4
Lock released on resource [Resource: inventory-staging], resources: [inventory-staging]#5 queued behind it and deployed 1.5.5 to staging once #4 let go. Both then waited at the
approval prompt. I approved only the newer one, #5. When it passed milestone(2), Jenkins
ended #4, which was still waiting at the prompt:
[Pipeline] { (Approve production)
[Pipeline] timeout
Timeout set to expire in 30 min
[Pipeline] input
Deploy this build to production?
Deploy or Abort
Superseded by delivery/inventory-service/main#5
[…]
[Pipeline] { (Deploy to production)
Stage "Deploy to production" skipped due to earlier failure(s)
[…]
+ jq -cn --arg text delivery/inventory-service/main 1.5.4: superseded by a newer build, not deployed {text: $text}
[…]
Superseded by delivery/inventory-service/main#5
Already interrupted
Finished: NOT_BUILTWithout the milestone, someone could approve #4 later and roll production back from 1.5.5 to 1.5.4 without realising. The rule a milestone enforces: once a build passes milestone N, any older build that hasn't passed it yet is cancelled.
Note the result: NOT_BUILT, not ABORTED. My first version of the Jenkinsfile only had
an aborted post condition, and the superseded build sent no notification at all. That's
why the final version has a separate notBuilt block.
Notifications
The chat stand-in received one message per outcome over the runs above:
{"text":"delivery/inventory-service/main 1.4.1: deployed (http://localhost:8090/job/delivery/job/inventory-service/job/main/1/)"}
{"text":"delivery/inventory-service/main 1.5.2: FAILED, see http://localhost:8090/job/delivery/job/inventory-service/job/main/2/console"}
{"text":"delivery/inventory-service/main 1.5.3: deployed (http://localhost:8090/job/delivery/job/inventory-service/job/main/3/)"}
{"text":"delivery/inventory-service/main 1.5.4: superseded by a newer build, not deployed"}
{"text":"delivery/inventory-service/main 1.5.5: deployed (http://localhost:8090/job/delivery/job/inventory-service/job/main/5/)"}A plain HTTPS POST of {"text": ...} is what Slack and Mattermost incoming webhooks
accept, so notifyChat works unchanged with a real webhook URL in the credential. Other
chat tools want slightly different JSON, and only the jq line needs to change. If you'd
rather use plugins: the Slack Notification plugin provides
slackSend channel: '#deploys', message: '...', and Pipeline's mail step sends
mail to: 'team@example.com', subject: '...', body: '...'. I didn't run either. Slack's
plugin isn't installed in the lab, and mail needs an SMTP server configured under
Manage Jenkins > System. Wherever the message goes, keep it short and include the version
and a link to the build. That's all anyone needs at 2 a.m.
Where you'll use this
This pipeline is a scaled-down copy of how many teams deploy services to virtual machines. To move it from the lab to real servers, change the scripts, not the Jenkinsfile:
- Hosts become SSH targets.
environments.shmapsstagingtodeploy@staging-01.internalinstead of a directory.deploy.shcopies the tarball withscpand runs the unpack and symlink switch withssh. The Jenkinsfile wraps those steps insshagent(['deploy-key'])from the SSH Agent plugin (not installed in my lab), and the key is a credential, as in the credentials article. service.shbecomessystemctl restart inventory-service. systemd keeps the process alive, so theJENKINS_NODE_COOKIEtrick is no longer needed.- Several servers per environment means a loop in
deploy.sh. For a rolling deploy, take one server out of the load balancer, deploy, smoke test, put it back, then do the next. Thelockstill covers the whole environment. - Deploy agents. Production credentials should only exist on a dedicated agent:
agent { label 'deploy-prod' }on the production stage. The agents article covers labels. - Containers and Kubernetes. The package stage builds and pushes an image tagged with
VERSIONinstead of a tarball.deploy.shrunskubectl set imageandkubectl rollout status, androllback.shrunskubectl rollout undo. The pipeline shape (test, versioned artifact, staging with smoke test, approval, production, rollback, locks, milestones) doesn't change. - Many services. Once three services have this Jenkinsfile, move it into a
shared library template. Each repository then
keeps its
ci/scripts and a three-line Jenkinsfile.
Common mistakes
The service dies when the build finishes. Jenkins kills every process a build started
when the build ends. It finds them by an environment variable it sets, JENKINS_NODE_COOKIE
in Pipeline jobs. I ran this job:
node {
sh 'nohup python3 -m http.server 18085 --bind 127.0.0.1 > http.log 2>&1 &'
sh 'sleep 1; curl -s -o /dev/null -w "inside the build: HTTP %{http_code}\\n" http://127.0.0.1:18085/'
}inside the build: HTTP 200Two seconds after the build finished, from a normal shell:
curl -s -o /dev/null -w "after the build: HTTP %{http_code}\n" http://127.0.0.1:18085/after the build: HTTP 000nohup doesn't help: the process was killed, not hung up. With
JENKINS_NODE_COOKIE=dontKillMe in front of the same command, the second check printed
after the build: HTTP 200. Use that for lab setups like this one. For real services, let
a service manager (systemd, a container runtime) own the process, so the build only asks
for a restart.
Approving or scanning over REST gets a 403. With a password login, every POST needs a CSRF crumb:
curl -s -i -u "admin:$JENKINS_PASSWORD" -X POST \
http://localhost:8090/job/delivery/job/inventory-service/job/main/6/input/Promote/proceedEmpty \
| grep -E '^HTTP|<title>'HTTP/1.1 403 Forbidden
<title>Error 403 No valid crumb was included in the request</title>Fetch a crumb from /crumbIssuer/api/json with a cookie jar, as shown in Run 1, or log in
with an API token instead, which doesn't need one.
Rebuilding in the production stage. It's tempting to run package.sh again in the
production stage, "it's the same commit". But that produces a different file from the one
staging tested, and the fingerprint no longer links the two. stash/unstash (or
copyArtifacts from the build) deploys the tested bytes.
Only handling aborted. A build cancelled by a milestone ends as NOT_BUILT, and a
pipeline that only has aborted and failure notifications says nothing about it. I hit
exactly this on my first attempt. If a build can end in a way you didn't report, someone
will eventually wonder where their release went.
Trusting the unit tests to cover deployment. Run 2 passed every unit test and still couldn't start. A deploy without a smoke test only proves that files were copied.
Summary
- Keep orchestration in the Jenkinsfile and the work in versioned scripts: build, package, deploy, smoke test, rollback. Scripts can be run, linted and reused outside Jenkins.
- Compute the version once and carry it everywhere: artifact name, deploy directory, and
the
/healthvalue the smoke test checks. - Record tests with
junitinpost { always }. Archive and fingerprint the artifact, and deploy that exact file to every environment withstash/unstash. - Use
agent noneat the top so theinputapproval holds no executor. Give it atimeout, asubmitter, andwhen { beforeInput true }. - Put a
post { failure }rollback on every deploy stage, and verify the rollback with the same smoke test. - Use one
lockper environment to serialise deploys (and maintenance jobs). Usemilestoneso an older build can never be approved after a newer one. Superseded builds end asNOT_BUILT. - Notify on every outcome (
success,failure,aborted,notBuilt) without letting a notification failure fail the build. Keep webhook URLs in credentials.
Exercises
Without waiting 30 minutes, find out what result a build gets when the approval times
out, and whether its aborted post block runs. Use a short pipeline with the same
approval stage and a 20-second timeout.
Show Solution
pipeline {
agent none
stages {
stage('Approve production') {
options { timeout(time: 20, unit: 'SECONDS') }
input {
id 'Promote'
message 'Deploy this build to production?'
ok 'Deploy'
}
steps {
echo 'approved'
}
}
}
post {
aborted {
echo 'Nobody approved in time; production was not touched.'
}
}
}[Pipeline] { (Approve production)
[Pipeline] timeout
Timeout set to expire in 20 sec
[Pipeline] {
[Pipeline] input
Deploy this build to production?
Deploy or Abort
Cancelling nested steps due to timeout
[…]
[Pipeline] { (Declarative: Post Actions)
[Pipeline] echo
Nobody approved in time; production was not touched.
[Pipeline] End of Pipeline
Timeout has been exceeded
[…]
Finished: ABORTEDThe stage-level timeout option wraps the input directive, so an unanswered approval
ends the build as ABORTED, and the aborted post block runs. In the capstone pipeline,
that sends the "aborted (rejected or approval timed out)" message.
Start a build of main, wait for the approval prompt, and reject it through the REST API
instead of approving. What is the build result, what does production run afterwards, and
what did the chat receive?
Show Solution
The prompt's abortUrl from pendingInputActions is .../input/Promote/abort. I POSTed
to it with a crumb, the same way as the approval in Run 1:
curl -s -b cookies.txt -u "admin:$JENKINS_PASSWORD" -H "$CRUMB" -X POST -o /dev/null -w "abort HTTP %{http_code}\n" \
http://localhost:8090/job/delivery/job/inventory-service/job/main/6/input/Promote/abortabort HTTP 200The end of build #6:
[Pipeline] input
Deploy this build to production?
Deploy or Abort
[…]
Stage "Deploy to production" skipped due to earlier failure(s)
[…]
Rejected by admin
[…]
Finished: ABORTEDProduction was untouched, and the chat got the abort message:
production current: releases/1.5.5
{"status": "ok", "version": "1.5.5", "skus": 5}
{"text":"delivery/inventory-service/main 1.5.6: aborted (rejected or approval timed out)"}Staging, however, now runs 1.5.6. Rejecting a production deploy doesn't undo staging. Staging is meant to hold the newest candidate.
/health passing doesn't prove the data loaded correctly. Extend smoke-test.sh so it
also looks up SKU-1001 and fails unless the reply contains that SKU and a numeric
quantity. Test your version against the running production service.
Show Solution
Insert this before the final echo "smoke test passed..." line:
item=$(curl -fsS --max-time 5 "http://127.0.0.1:$PORT/items/SKU-1001")
echo "/items/SKU-1001 -> $item"
if ! jq -e '.sku == "SKU-1001" and (.qty | type) == "number"' <<<"$item" > /dev/null; then
echo "smoke test FAILED: stock lookup is broken"
exit 1
fiRun against production (1.5.5 at that point):
bash ci/smoke-test.sh production 1.5.5; echo "exit=$?"http://127.0.0.1:18082/health -> {"status": "ok", "version": "1.5.5", "skus": 5}
/items/SKU-1001 -> {"sku": "SKU-1001", "qty": 42}
smoke test passed: production is running 1.5.5
exit=0I checked the type rather than the value 42. A smoke test should check that the system works, not pin down production data that changes every day.
Production sometimes needs to go back to a release that isn't the previous one. Write
a parameterized Pipeline job, rollback-production, that takes a RELEASE parameter,
refuses unknown releases (listing the ones available), switches production to the chosen
one under the production lock, and verifies it with the smoke test. Run it once with no
parameter and once with RELEASE=1.5.3.
Hint
Reuse ci/service.sh and ci/smoke-test.sh by checking out the repository. And watch
what happens on the very first build of a job whose parameters are defined in its
Jenkinsfile.
Show Solution
pipeline {
agent any
parameters {
string(name: 'RELEASE', defaultValue: '', description: 'Release to put back into production, e.g. 1.4.1')
}
options { lock('inventory-production') }
environment {
DEPLOY_ROOT = '/home/sid/hosts'
}
stages {
stage('Roll back production') {
steps {
git url: 'file:///home/sid/git/inventory-service.git', branch: 'main'
sh '''
releases="$DEPLOY_ROOT/production/releases"
if [ -z "$RELEASE" ] || [ ! -d "$releases/$RELEASE" ]; then
echo "unknown release '$RELEASE'; available: $(ls "$releases" | tr '\\n' ' ')"
exit 1
fi
readlink "$DEPLOY_ROOT/production/current" > "$DEPLOY_ROOT/production/previous"
ln -sfn "releases/$RELEASE" "$DEPLOY_ROOT/production/current"
bash ci/service.sh restart production
bash ci/smoke-test.sh production "$RELEASE"
'''
}
}
}
}The first build ran before Jenkins knew about the parameter, which is defined by running
the Jenkinsfile once. So RELEASE was empty, and the guard did its job:
+ releases=/home/sid/hosts/production/releases
+ [ -z ]
+ ls /home/sid/hosts/production/releases
+ tr
+ echo unknown release ''; available: 1.4.1 1.5.3 1.5.5
unknown release ''; available: 1.4.1 1.5.3 1.5.5
+ exit 1
Lock released on resource [Resource: inventory-production], resources: [inventory-production]
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILUREThe second build, with jcli build delivery/rollback-production -p RELEASE=1.5.3 -s -v:
+ releases=/home/sid/hosts/production/releases
+ [ -z 1.5.3 ]
+ [ ! -d /home/sid/hosts/production/releases/1.5.3 ]
+ readlink /home/sid/hosts/production/current
+ ln -sfn releases/1.5.3 /home/sid/hosts/production/current
+ bash ci/service.sh restart production
+ bash ci/smoke-test.sh production 1.5.3
curl: (7) Failed to connect to 127.0.0.1 port 18082 after 0 ms: Couldn't connect to server
http://127.0.0.1:18082/health -> {"status": "ok", "version": "1.5.3", "skus": 5}
smoke test passed: production is running 1.5.3
Lock released on resource [Resource: inventory-production], resources: [inventory-production]
[Pipeline] End of Pipeline
Finished: SUCCESSThree details make this safe. The job takes the same inventory-production lock as
the pipeline, so a rollback can't collide with a deploy. It writes previous before
switching, so the rollback can itself be rolled back with rollback.sh. And it ends with
the same smoke test as every deploy.
Tested with: Jenkins 2.580.2 (OpenJDK 21) on Ubuntu 24.04 (WSL2), Pipeline: Declarative 2.2293.v6e7193cec599, Pipeline: Input Step 568.v0c9db_2652741, Pipeline: Milestone Step 152.v6e22b_8cfc66c, Lockable Resources 1588.v7b_b_614f4202e, JUnit 1458.v38661c0638e2, Workspace Cleanup 0.49, Python 3.12.3, pytest 9.2.0, November 2026.
All six main builds, the data-refresh lock holder, the approval-timeout job, the rollback job and the process-killing demo ran on the lab controller, and every log above is from those runs, trimmed where marked […]. Approvals and the rejection were sent through the REST API rather than by clicking Deploy and Abort. "Staging" and "production" were directories and ports on the lab machine rather than separate servers, and the chat service was a local stand-in for an incoming webhook. The slackSend and mail alternatives were not run. Paths from the test machine's scratch directories are shown shortened to /home/sid/…. Job, node and credential names are shown without the test run's prefixes.
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Jenkins tutorial here is tested end to end before publishing.