Your First Pipeline: Jenkinsfile Basics

Write a first Jenkinsfile for a real Python project: stages, sh steps, test reports and artifacts, Pipeline from SCM, Replay, the Snippet Generator and the declarative linter.

By The Dark DebuggerUpdated 23 min read

This article rebuilds the logstats project from Freestyle Jobs, Parameters, and Triggers as a Pipeline: first as a script stored in the job, then as a Jenkinsfile committed next to the code. Along the way you will record test results and artifacts, read the console and stage view, fix a failing build, try changes with Replay, and catch mistakes with the declarative linter before they reach Jenkins. You need a controller and the jcli function from Installing Jenkins. The freestyle article explains the project itself.

Why a Jenkinsfile

At the end of the freestyle article, the logstats release process was spread over two job configurations, a shell script in a text box, three post-build actions and a trigger, none of it in git. A Jenkinsfile puts the whole process in one file in the repository:

  • It is reviewed in the same pull request as the code it builds. A change to the build process is a diff, not a mystery.
  • Each branch carries its own Jenkinsfile, so a branch that needs a new build step brings it along.
  • The build is split into named stages, so a failure says "Test failed", not "something in a 40-line shell step failed".
  • If the controller is lost, the build definition isn't.

Creating a Pipeline job

In the UI: New Item, a name, Pipeline, OK. Most of the configuration page is the same as a freestyle job (General, Triggers), but instead of build steps there is a Pipeline section with a Definition drop-down:

  • Pipeline script: the script is typed into a text box and stored in the job's config.xml. Good for experiments.
  • Pipeline script from SCM: Jenkins fetches the script from a repository, by default the file Jenkinsfile at the top level. This is what you want for real projects.

We'll start with the first, because it is the shortest path to a running pipeline.

A first pipeline

Here is the script I pasted into a new Pipeline job called logstats-inline:

pipeline {
    agent any
 
    stages {
        stage('Checkout') {
            steps {
                git url: 'file:///home/sid/logstats', branch: 'main'
            }
        }
        stage('Test') {
            steps {
                sh 'python3 tests/junit_runner.py reports/junit.xml'
            }
        }
        stage('Report') {
            steps {
                sh 'python3 logstats.py sample/access.log'
            }
        }
    }
}

Every declarative pipeline has the same skeleton:

ElementMeaning
pipeline { }Marks this as declarative Pipeline. Everything goes inside it.
agent anyRun on any available agent. Jenkins allocates an executor and a workspace for the whole pipeline.
stages { }The ordered list of stages.
stage('Test') { }One named phase. Names show up in the log, the stage view and the API.
steps { }The actual work in a stage, one step per line.
git url: ..., branch: ...Clone a repository into the workspace.
sh '...'Run a shell command (bat and powershell are the Windows equivalents). A non-zero exit fails the stage.
echo '...'Print a message to the log. No shell involved.

Build it:

jcli build logstats-inline -s -v
Started logstats-inline #1
Started from command line by admin
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/logstats-inline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] git
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning repository file:///home/sid/logstats
[…]
Checking out Revision fcc25fd24122d756fbddfdaaef4cd887768bcf9d (refs/remotes/origin/main)
[…]
Commit message: "Add a 503 to the sample log"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] sh
+ python3 tests/junit_runner.py reports/junit.xml
test_parses_a_combined_log_line (test_logstats.ParseTests.test_parses_a_combined_log_line) ... ok
test_returns_none_for_garbage (test_logstats.ParseTests.test_returns_none_for_garbage) ... ok
test_counts_status_classes (test_logstats.SummariseTests.test_counts_status_classes) ... ok
test_empty_input (test_logstats.SummariseTests.test_empty_input) ... ok
test_error_rate_is_a_percentage (test_logstats.SummariseTests.test_error_rate_is_a_percentage) ... ok
 
----------------------------------------------------------------------
Ran 5 tests in 0.000s
 
OK
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Report)
[Pipeline] sh
+ python3 logstats.py sample/access.log
requests: 11  unparsed: 1
by class: 2xx=6  3xx=1  4xx=1  5xx=3
5xx rate: 27.3%
     4  /api/orders
     3  /api/checkout
     1  /api/orders/1042
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Completed logstats-inline #1 : SUCCESS

Reading a pipeline log takes a little practice. [Pipeline] { (Test) opens a stage and [Pipeline] // stage closes it; [Pipeline] sh announces a step and its output follows; [Pipeline] node is where agent any grabbed an executor ("Jenkins" is how pipeline logs name the built-in node). Everything else is your commands and their output, exactly as in a terminal.

One habit to form right away: use single quotes around sh commands. In Groovy, '...' is a literal string, so $VARIABLE reaches the shell untouched and the shell expands it. In "...", Groovy tries to expand $VARIABLE itself, before the shell ever sees it. Common mistakes shows what that does.

Recording tests and keeping artifacts

The inline pipeline ran the tests but threw the results away. Two steps fix that:

  • junit 'reports/*.xml' parses JUnit XML and attaches the results to the build: pass/fail counts, a trend chart, per-test history. Failing tests mark the build UNSTABLE.
  • archiveArtifacts artifacts: 'dist/*.tar.gz, report.txt', fingerprint: true copies files out of the workspace into the build record, and fingerprints them so they can be traced later.

Both belong in a post section, which runs after the stages:

pipeline {
    agent any
 
    stages {
        stage('Build') {
            steps {
                sh 'python3 -m py_compile logstats.py'
                sh 'mkdir -p dist && tar czf dist/logstats-${BUILD_NUMBER}.tar.gz logstats.py'
            }
        }
        stage('Test') {
            steps {
                sh 'python3 tests/junit_runner.py reports/junit.xml'
            }
        }
        stage('Report') {
            steps {
                sh 'python3 logstats.py sample/access.log | tee report.txt'
            }
        }
    }
 
    post {
        always {
            junit 'reports/*.xml'
        }
        success {
            archiveArtifacts artifacts: 'dist/*.tar.gz, report.txt', fingerprint: true
        }
    }
}

post { always { } } runs even when a stage failed, which is exactly when you most want the test report. post { success { } } archives the package only from green builds, so a broken build can never produce something that looks deployable. Declarative Pipeline in Depth covers all the post conditions.

Notice ${BUILD_NUMBER} inside single quotes: Jenkins exports every build variable (the same ones freestyle jobs get) into the environment of sh steps, so the shell expands it.

Pipeline script from SCM

That script is the Jenkinsfile. It goes in the repository root, committed like any other file:

cd ~/logstats
git add Jenkinsfile
git commit -qm "Add Jenkinsfile"
git log --oneline
6574450 Add Jenkinsfile
ce23228 logstats: nginx access log summary with tests

The job logstats uses Pipeline script from SCM: SCM Git, the same repository URL and */main branch as before, Script Path Jenkinsfile, and Lightweight checkout ticked. This is the definition part of the XML I created the job from:

  <definition class="org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition" plugin="workflow-cps">
    <scm class="hudson.plugins.git.GitSCM" plugin="git">
      <configVersion>2</configVersion>
      <userRemoteConfigs>
        <hudson.plugins.git.UserRemoteConfig>
          <url>file:///home/sid/logstats</url>
        </hudson.plugins.git.UserRemoteConfig>
      </userRemoteConfigs>
      <branches>
        <hudson.plugins.git.BranchSpec>
          <name>*/main</name>
        </hudson.plugins.git.BranchSpec>
      </branches>
      <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
      <submoduleCfg class="empty-list"/>
      <extensions/>
    </scm>
    <scriptPath>Jenkinsfile</scriptPath>
    <lightweight>true</lightweight>
  </definition>

Lightweight checkout lets Jenkins read just the Jenkinsfile without cloning the whole repository on the controller. Here is the first build, trimmed:

jcli build logstats -s -v
Started logstats #1
Started from command line by admin
Obtained Jenkinsfile from git file:///home/sid/logstats
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/logstats
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
[…]
Checking out Revision fcc25fd24122d756fbddfdaaef4cd887768bcf9d (refs/remotes/origin/main)
[…]
[Pipeline] { (Build)
[Pipeline] sh
+ python3 -m py_compile logstats.py
[Pipeline] sh
+ mkdir -p dist
+ tar czf dist/logstats-1.tar.gz logstats.py
[…]
[Pipeline] { (Test)
[Pipeline] sh
+ python3 tests/junit_runner.py reports/junit.xml
[…]
OK
[…]
[Pipeline] { (Report)
[Pipeline] sh
+ python3 logstats.py sample/access.log
+ tee report.txt
requests: 11  unparsed: 1
[…]
[Pipeline] { (Declarative: Post Actions)
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[Pipeline] archiveArtifacts
Archiving artifacts
Recording fingerprints
[…]
Finished: SUCCESS
Completed logstats #1 : SUCCESS

Two new things appear. Obtained Jenkinsfile from git ... is Jenkins reading the script from the repository. Declarative: Checkout SCM is an automatic stage: when the pipeline came from SCM, declarative checks out the same repository and commit into the workspace before your first stage, so the Jenkinsfile no longer needs its own git step. (The Selected Git installation does not exist. Using Default line is noise from the git plugin's tool lookup and is harmless.)

Reading the results: console, stage view and tests

The job page's Stage View, and the newer Stages / Pipeline Overview pages, draw the stages as a grid or graph with each stage's duration and status. The same data comes from the wfapi endpoint, which is handy for dashboards:

curl -sS -u "admin:$JENKINS_API_TOKEN" http://localhost:8090/job/logstats/1/wfapi/describe \
  | jq -r '.stages[] | "\(.name)\t\(.status)\t\(.durationMillis) ms"'
Declarative: Checkout SCM	SUCCESS	507 ms
Build	SUCCESS	973 ms
Test	SUCCESS	877 ms
Report	SUCCESS	570 ms
Declarative: Post Actions	SUCCESS	557 ms

The junit step's results and the archived files are part of the build too:

curl -g -sS -u "admin:$JENKINS_API_TOKEN" "http://localhost:8090/job/logstats/1/testReport/api/json?tree=passCount,failCount,skipCount,duration" | jq -c .
curl -g -sS -u "admin:$JENKINS_API_TOKEN" "http://localhost:8090/job/logstats/1/api/json?tree=artifacts[relativePath]" | jq -c .
{"_class":"hudson.tasks.junit.TestResult","duration":0.0,"failCount":0,"passCount":5,"skipCount":0}
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","artifacts":[{"relativePath":"dist/logstats-1.tar.gz"},{"relativePath":"report.txt"}]}

When a test fails

A pipeline earns its keep on the bad days, so let's make one. This change counts client errors (4xx) where it should count server errors:

cd ~/logstats
sed -i 's/classes\["5xx"\] \/ total/classes["4xx"] \/ total/' logstats.py
git diff
git commit -qam "Count client errors in the error rate"
diff --git a/logstats.py b/logstats.py
index 76e4942..96af57a 100755
--- a/logstats.py
+++ b/logstats.py
@@ -31,7 +31,7 @@ def summarise(lines):
         classes[f"{rec['status'] // 100}xx"] += 1
         paths[rec["path"]] += 1
     total = sum(classes.values())
-    rate = round(100 * classes["5xx"] / total, 1) if total else 0.0
+    rate = round(100 * classes["4xx"] / total, 1) if total else 0.0
     return {"total": total, "classes": dict(sorted(classes.items())),
             "top_paths": paths.most_common(3), "error_rate": rate,
             "unparsed": unparsed}

The next build, from the Test stage on:

[Pipeline] { (Test)
[Pipeline] sh
+ python3 tests/junit_runner.py reports/junit.xml
[…]
test_error_rate_is_a_percentage (test_logstats.SummariseTests.test_error_rate_is_a_percentage) ... FAIL
 
======================================================================
FAIL: test_error_rate_is_a_percentage (test_logstats.SummariseTests.test_error_rate_is_a_percentage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/sid/.jenkins/workspace/logstats/tests/test_logstats.py", line 26, in test_error_rate_is_a_percentage
    self.assertEqual(summarise([OK, ERR])["error_rate"], 50.0)
AssertionError: 0.0 != 50.0
 
----------------------------------------------------------------------
Ran 5 tests in 0.003s
 
FAILED (failures=1)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Report)
Stage "Report" skipped due to earlier failure(s)
[Pipeline] getContext
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[…]
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE
Completed logstats #2 : FAILURE

Follow the chain. The test runner exited 1, so the sh step failed, so the Test stage failed. Declarative skipped every later stage (Stage "Report" skipped due to earlier failure(s)) but still ran post { always }, so the test report was recorded. post { success } did not run, so nothing was archived:

curl -g -sS -u "admin:$JENKINS_API_TOKEN" "http://localhost:8090/job/logstats/2/testReport/api/json?tree=passCount,failCount,suites[cases[className,name,status,errorDetails]]" \
  | jq -c '{passCount, failCount, failed: [.suites[].cases[] | select(.status != "PASSED") | {name, status, errorDetails}]}'
curl -g -sS -u "admin:$JENKINS_API_TOKEN" "http://localhost:8090/job/logstats/2/api/json?tree=result,artifacts[relativePath]" | jq -c .
{"passCount":4,"failCount":1,"failed":[{"name":"test_error_rate_is_a_percentage","status":"FAILED","errorDetails":"AssertionError: 0.0 != 50.0"}]}
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","artifacts":[],"result":"FAILURE"}

That is the behaviour you want: a red build, a precise failure, and no artifact that someone might deploy by mistake. Reverting the commit (git revert --no-edit HEAD) and building again gave a green build #3.

The result was FAILURE rather than UNSTABLE because the shell step failed before junit ever saw the report. Some teams prefer "tests failed" to show as UNSTABLE (yellow) so it can be told apart from "the build itself broke"; Exercise 4 does that.

Replay: try a change without committing

Every Pipeline build has a Replay link. It opens the script that build ran in an editor; you change it and run it again as a new build, without touching the repository. It is the fastest way to experiment with a Jenkinsfile. The CLI has the same feature, which also makes it scriptable. I saved a copy of the Jenkinsfile as Jenkinsfile.experiment with one extra line in the Report stage, counting 5xx responses with awk:

        stage('Report') {
            steps {
                sh 'python3 logstats.py sample/access.log | tee report.txt'
                sh 'awk \'$9 ~ /^5/\' sample/access.log | wc -l'
            }
        }
jcli replay-pipeline logstats -n 4 < Jenkinsfile.experiment

The command prints nothing; the new build appears in the job. Its console:

Started by user admin
Rebuilt #4
[Pipeline] Start of Pipeline
[…]
+ awk $9 ~ /^5/ sample/access.log
+ wc -l
3
[…]
Finished: SUCCESS

Two details. There is no Obtained Jenkinsfile from git line, because the script came from the replay, not the repository. And on this version the build's cause reads Rebuilt #4. Replay is for trying things: once the change works, commit it, or the next normal build silently goes back to the old Jenkinsfile.

(My first replay used grep -c " 50[0-9] " and printed 4 instead of 3, because one line's byte count was 501. That is exactly the kind of mistake Replay lets you make cheaply.)

The Snippet Generator and Directive Generator

Nobody remembers the exact syntax of every step. Each Pipeline job has a Pipeline Syntax link that opens three helper pages, all present on this version:

for p in pipeline-syntax/ directive-generator/ pipeline-syntax/globals; do
  printf '%s ' "$p"
  curl -sS -u "admin:$JENKINS_API_TOKEN" -o page.html -w '%{http_code} ' "http://localhost:8090/job/logstats/$p"
  grep -o '<title>[^<]*</title>' page.html
done
pipeline-syntax/ 200 <title>Pipeline Syntax: Snippet Generator - Jenkins</title>
directive-generator/ 200 <title>Pipeline Syntax: Directive Generator - Jenkins</title>
pipeline-syntax/globals 200 <title>Pipeline Syntax: Global Variable Reference - Jenkins</title>
  • Snippet Generator: choose a step from Sample Step (say archiveArtifacts or junit), fill in the form the same way you would in a freestyle job, click Generate Pipeline Script, and paste the one-liner into your Jenkinsfile. The list contains exactly the steps your installed plugins provide, which makes it the most reliable reference for your controller.
  • Directive Generator: the same idea for declarative sections such as options, triggers, parameters and when.
  • Global Variable Reference: documents env, params, currentBuild and scm, including every property of currentBuild.

Check a Jenkinsfile before you commit it

Pushing a commit just to find out you mistyped a section name is a slow feedback loop. The declarative linter checks a Jenkinsfile against your controller's installed plugins without running it:

jcli declarative-linter < Jenkinsfile
Jenkinsfile successfully validated.

Here is a broken one, with three mistakes beginners really make: a typo in agent, a step outside steps, and therefore no agent at all:

pipeline {
    agnet any
    stages {
        stage('Test') {
            sh 'python3 tests/junit_runner.py reports/junit.xml'
        }
    }
}
jcli declarative-linter < broken.Jenkinsfile
Errors encountered validating Jenkinsfile:
WorkflowScript: 2: Undefined section "agnet" @ line 2, column 5.
       agnet any
       ^
 
WorkflowScript: 4: Unknown stage section "sh". Starting with version 0.5, steps in a stage must be in a ‘steps’ block. @ line 4, column 9.
           stage('Test') {
           ^
 
WorkflowScript: 1: Missing required section "agent" @ line 1, column 1.
   pipeline {
   ^
 
WorkflowScript: 4: Expected one of "steps", "stages", or "parallel" for stage "Test" @ line 4, column 9.
           stage('Test') {
           ^
 

The exit code is 1 on errors and 0 on success, so the linter fits in a git pre-commit hook or a CI check. Without the CLI, the same validator answers over HTTP:

curl -sS -X POST -u "admin:$JENKINS_API_TOKEN" -F "jenkinsfile=<Jenkinsfile" http://localhost:8090/pipeline-model-converter/validate
Jenkinsfile successfully validated.

Know its limit: the linter checks structure, not step names inside steps. A Jenkinsfile that calls ech 'running tests' instead of echo passed with Jenkinsfile successfully validated. and then failed at run time with No such DSL method 'ech', exactly as shown in the introduction.

Committing the Jenkinsfile with the code

Once the Jenkinsfile lives in the repository, treat it like code:

  • Change it in the same commit as the code that needs the change. A commit that adds a dependency should also change the step that installs it, so every commit in history builds with the Jenkinsfile it was written for.
  • Review it. A Jenkinsfile runs with the agent's access to your network and with whatever credentials it asks for. A one-line change can ship your secrets somewhere; that deserves a reviewer.
  • Keep logic out of it. Twenty lines of shell inside sh '''...''' are better as ci/build.sh in the repo, called with one sh step and runnable on a laptop.
  • Lint before you push, as above.

Where you'll use this

The logstats Jenkinsfile, in the final state of this article, is a complete small CI pipeline that you can adapt to almost any project with a test runner: compile, test, report, record the tests, keep the package from green builds, and leave a clean workspace. This is the version committed to the repository:

pipeline {
    agent any
 
    stages {
        stage('Build') {
            steps {
                sh 'python3 -m py_compile logstats.py'
                sh 'mkdir -p dist && tar czf dist/logstats-${BUILD_NUMBER}.tar.gz logstats.py'
            }
        }
        stage('Test') {
            steps {
                sh 'python3 tests/junit_runner.py reports/junit.xml'
            }
        }
        stage('Report') {
            steps {
                sh 'python3 logstats.py sample/access.log | tee report.txt'
            }
        }
    }
 
    post {
        always {
            junit 'reports/*.xml'
        }
        success {
            archiveArtifacts artifacts: 'dist/*.tar.gz, report.txt', fingerprint: true
        }
        cleanup {
            cleanWs()
        }
    }
}

For a Java project the sh lines become ./mvnw -B verify and the junit path becomes target/surefire-reports/*.xml; for Node it might be npm ci and npm test with a JUnit reporter. The shape doesn't change. The declarative article grows this into a release pipeline with parallel checks and an approval before production, and the capstone takes it to a full deployment.

Common mistakes

Double quotes around shell variables. Here Groovy, not the shell, tries to expand $f:

sh "for f in dist/*.tar.gz; do echo found $f; done"
[Pipeline] { (List packages)
[Pipeline] sh
+ mkdir -p dist
+ touch dist/logstats-1.tar.gz
[Pipeline] }
[…]
groovy.lang.MissingPropertyException: No such property: f for class: groovy.lang.Binding
Finished: FAILURE

There is no Groovy variable f, so the build dies before the loop runs. (The first sh in that stage used "...${BUILD_NUMBER}..." and worked, because Groovy can resolve build variables, which makes this mistake look intermittent.) Use single quotes and let the shell do shell things. When you really need a Groovy value inside a shell command, use double quotes and escape the shell's own $ as \$.

Bash syntax in sh. The sh step runs /bin/sh, which is dash on Ubuntu. I replayed build #5 with the tarball named after a short commit, using Bash substring syntax:

sh 'mkdir -p dist && tar czf dist/logstats-${GIT_COMMIT:0:7}.tar.gz logstats.py'
[Pipeline] { (Build)
[Pipeline] sh
+ python3 -m py_compile logstats.py
[Pipeline] sh
+ mkdir -p dist
/home/sid/.jenkins/workspace/logstats@tmp/durable-bc0d59de/script.sh.copy: 1: Bad substitution
[…]
ERROR: script returned exit code 2
Finished: FAILURE

Use POSIX-only syntax, start the script with a #!/bin/bash line, or let a tool do the work; Exercise 3 has a working version.

A wrong path in junit. One missing letter (report/ instead of reports/) fails the whole build, even though every test passed:

[Pipeline] { (Declarative: Post Actions)
[Pipeline] junit
Recording test results
No test report files were found. Configuration error?
Error when executing always post condition:
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 9ab5a369-fbb2-4c1c-a937-626d9b82cd87
hudson.AbortException: No test report files were found. Configuration error?
[…]
ERROR: No test report files were found. Configuration error?
Finished: FAILURE

Check the path against the workspace (the job's Workspaces link shows it). Add allowEmptyResults: true only if "no tests ran" is genuinely acceptable.

Old files in a reused workspace. Pipeline jobs reuse their workspace just as freestyle jobs do, and the declarative checkout doesn't delete untracked files. After a few builds and replays, build #7 of logstats archived six tarballs, five of them from earlier builds:

curl -g -sS -u "admin:$JENKINS_API_TOKEN" "http://localhost:8090/job/logstats/7/api/json?tree=number,artifacts[relativePath]" | jq -c .
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","artifacts":[{"relativePath":"dist/logstats-1.tar.gz"},{"relativePath":"dist/logstats-2.tar.gz"},{"relativePath":"dist/logstats-3.tar.gz"},{"relativePath":"dist/logstats-4.tar.gz"},{"relativePath":"dist/logstats-5.tar.gz"},{"relativePath":"dist/logstats-715ee59.tar.gz"},{"relativePath":"report.txt"}],"number":7}

The fix is one post condition, cleanup, which runs last regardless of the result, and the cleanWs() step from the Workspace Cleanup plugin:

git diff
jcli declarative-linter < Jenkinsfile
git commit -qam "Clean the workspace after every build"
diff --git a/Jenkinsfile b/Jenkinsfile
index da1d79a..363788e 100755
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -27,5 +27,8 @@ pipeline {
         success {
             archiveArtifacts artifacts: 'dist/*.tar.gz, report.txt', fingerprint: true
         }
+        cleanup {
+            cleanWs()
+        }
     }
 }
Jenkinsfile successfully validated.

The first build after the change still archived the old files (the cleanup runs at the end). From the second one on, each build archived only its own package:

[Pipeline] { (Declarative: Post Actions)
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[Pipeline] archiveArtifacts
Archiving artifacts
Recording fingerprints
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
[WS-CLEANUP] done
[…]
Finished: SUCCESS
Completed logstats #9 : SUCCESS
curl -g -sS -u "admin:$JENKINS_API_TOKEN" "http://localhost:8090/job/logstats/9/api/json?tree=number,result,artifacts[relativePath]" | jq -c .
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","artifacts":[{"relativePath":"dist/logstats-9.tar.gz"},{"relativePath":"report.txt"}],"number":9,"result":"SUCCESS"}

Summary

  • A declarative Jenkinsfile is pipeline → agent → stages → stage('Name') → steps, with a post section for what must happen afterwards.
  • Pipeline script stores the script in the job; Pipeline script from SCM reads Jenkinsfile from the repository and adds an automatic Declarative: Checkout SCM stage. Use SCM for anything real.
  • Put junit in post { always } so failures still produce a report, and archiveArtifacts in post { success } so broken builds produce nothing deployable. Add cleanup { cleanWs() } so old files can't leak into new artifacts.
  • A failing sh step fails its stage, later stages are skipped, and post still runs.
  • Single-quote sh commands. sh runs /bin/sh, which is not Bash on Ubuntu.
  • Replay runs a modified script once without committing; the Snippet Generator writes step syntax for you; jcli declarative-linter checks structure, but not step names, before you push.

Exercises

Exercise 1: Add a lint stageeasy

Add a Lint stage between Build and Test that byte-compiles both the program and the tests (python3 -m py_compile logstats.py tests/*.py), and check your Jenkinsfile with the linter before committing.

Show Solution
        stage('Lint') {
            steps {
                sh 'python3 -m py_compile logstats.py tests/*.py'
            }
        }

placed after the closing brace of stage('Build'). Then:

jcli declarative-linter < Jenkinsfile
Jenkinsfile successfully validated.

The tests/*.py glob is expanded by the shell, which is one more reason the command is in single quotes.

Exercise 2: Let the linter fix it for youmedium

Run the broken Jenkinsfile from Check a Jenkinsfile before you commit it through the linter and fix every error it reports, until it validates.

Hint

There are four messages but only two real mistakes. Fix the first one and two of the messages disappear.

Show Solution
pipeline {
    agent any
    stages {
        stage('Test') {
            steps {
                sh 'python3 tests/junit_runner.py reports/junit.xml'
            }
        }
    }
}
jcli declarative-linter < fixed.Jenkinsfile
Jenkinsfile successfully validated.

agnet caused both "Undefined section" and "Missing required section agent". The bare sh caused both "Unknown stage section" and "Expected one of steps, stages, or parallel". Linter output often reports one mistake several ways; fix from the top and re-run.

Exercise 3: Name the package after the commitmedium

Change the Build stage so the tarball is called logstats-<short commit>.tar.gz (7 characters) instead of using the build number. Try it with Replay before committing.

Hint

${GIT_COMMIT:0:7} fails under /bin/sh (see Common mistakes). Git can shorten a commit itself.

Show Solution
sh 'mkdir -p dist && tar czf "dist/logstats-$(git rev-parse --short=7 HEAD).tar.gz" logstats.py'
jcli declarative-linter < Jenkinsfile.experiment
jcli replay-pipeline logstats -n 5 < Jenkinsfile.experiment
Jenkinsfile successfully validated.

The replayed build's Build stage:

[Pipeline] { (Build)
[Pipeline] sh
+ python3 -m py_compile logstats.py
[Pipeline] sh
+ mkdir -p dist
+ git rev-parse --short=7 HEAD
+ tar czf dist/logstats-715ee59.tar.gz logstats.py
[Pipeline] }

$(...) is POSIX command substitution, so it works in dash. GIT_COMMIT is also available in the environment of a Pipeline-from-SCM build, and echo "$GIT_COMMIT" | cut -c1-7 would work just as well.

Exercise 4: Make failing tests yellow, not redhard

On the branch bug/error-rate (which contains the broken commit from this article), make the pipeline finish UNSTABLE instead of FAILURE when tests fail, still record the test report, and still run the Report stage.

Hint

The build goes red because the sh step exits non-zero. The junit step marks a build unstable on its own when the report contains failures.

Show Solution
pipeline {
    agent any
    stages {
        stage('Checkout') {
            steps {
                git url: 'file:///home/sid/logstats', branch: 'bug/error-rate'
            }
        }
        stage('Test') {
            steps {
                sh 'python3 tests/junit_runner.py reports/junit.xml || true'
            }
        }
        stage('Report') {
            steps {
                sh 'python3 logstats.py sample/access.log'
            }
        }
    }
    post {
        always {
            junit 'reports/*.xml'
        }
    }
}
[Pipeline] { (Test)
[Pipeline] sh
+ python3 tests/junit_runner.py reports/junit.xml
test_error_rate_is_a_percentage (test_logstats.SummariseTests.test_error_rate_is_a_percentage) ... FAIL
[…]
FAILED (failures=1)
+ true
[…]
[Pipeline] { (Report)
[Pipeline] sh
+ python3 logstats.py sample/access.log
[…]
5xx rate: 9.1%
[…]
[Pipeline] junit
Recording test results
[…]
Finished: UNSTABLE
Completed ex-unstable #1 : UNSTABLE

|| true makes the step succeed whatever the tests did, the Report stage runs, and junit sees one failure and sets UNSTABLE. Notice the report itself: 9.1% is the buggy figure (1 client error out of 11 requests), which is why an UNSTABLE build must never be treated as releasable. The downside of || true is that a crashed test runner that writes no report is no longer caught by the shell step; the junit step then fails with "No test report files were found", which at least keeps that case red.


Tested with: Jenkins 2.580.2 (LTS) with Pipeline 608, Pipeline: Declarative 2.2293, Git 5.11.0, JUnit 1458 and Workspace Cleanup 0.49, on OpenJDK 21.0.13 (Temurin) and Python 3.12.3, Ubuntu 24.04 (WSL2), November 2026. Every pipeline, replay, linter run and API call above was executed against that controller, with jobs created from XML through the CLI; logs are trimmed where marked […]. The Snippet Generator and Directive Generator pages were confirmed over HTTP but not clicked through, so the description of their forms comes from the pages themselves rather than a recorded session. What surprised me: the linter happily validates a misspelled step name, and a replayed build's cause reads Rebuilt #4 rather than mentioning Replay. Paths from the test machine's scratch directories are shown shortened to /home/sid/…. Job, node and credential names are shown without the test run's prefixes.

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Jenkins tutorial here is tested end to end before publishing.