Agents and Distributed Builds
Move builds off the Jenkins controller: connect inbound and SSH agents, route work with labels, manage workspaces across machines, and plan agent capacity.
By The Dark DebuggerUpdated 29 min read
A single Jenkins machine running every build works until the first build fills its disk
or reads its secrets. This article moves builds onto agents: why the controller should not
build anything, how to connect inbound (WebSocket) and SSH agents, how labels route work,
how agent, node and ws decide where a stage runs, what happens to workspaces when
work spreads over several machines, and how Docker and Kubernetes agents fit in. It
assumes you know the controller/agent/executor vocabulary from
What Jenkins Is and How It Works and the agent
directive from Declarative Pipeline in Depth.
SSH agents use the SSH key credentials from Credentials and Secrets.
Why builds should not run on the controller
A fresh Jenkins install has two executors on the built-in node, the controller
itself. Every agent any pipeline can land there. Here is what such a build can see on my
test controller:
pipeline {
agent { label 'built-in' }
stages {
stage('What a build on the controller can see') {
steps {
sh '''
id -un
echo "JENKINS_HOME=$JENKINS_HOME"
ls "$JENKINS_HOME/secrets" | head -4
wc -c "$JENKINS_HOME/secrets/master.key" "$JENKINS_HOME/credentials.xml"
'''
}
}
}
}[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/agent-builtin
[…]
+ id -un
sid
+ echo JENKINS_HOME=/home/sid/.jenkins
JENKINS_HOME=/home/sid/.jenkins
+ ls /home/sid/.jenkins/secrets
+ head -4
com.cloudbees.plugins.credentials.SecretBytes.KEY
hudson.console.AnnotatedLargeText.consoleAnnotator
hudson.console.ConsoleNote.MAC
hudson.model.Job.serverCookie
+ wc -c /home/sid/.jenkins/secrets/master.key /home/sid/.jenkins/credentials.xml
256 /home/sid/.jenkins/secrets/master.key
10125 /home/sid/.jenkins/credentials.xml
10381 totalThe build runs as the same operating-system user as Jenkins, so it can read the master
key and every encrypted credential, overwrite job configurations, or fill the disk that
JENKINS_HOME lives on. Anyone who can edit a Jenkinsfile gets all of that. There are
practical reasons too: a make -j16 on the controller slows down the UI and the
scheduler for everyone, and controller upgrades become riskier when build tools are
installed on the same machine.
The rule for production: set the built-in node's executors to 0 and run every build
on an agent. In the UI that is Manage Jenkins → Nodes → Built-In Node → Configure →
Number of executors. In Configuration as Code it is one line, numExecutors: 0 under
jenkins: (see Jenkins Configuration as Code).
With no executors and no online agent, builds simply wait. This is a pipeline on my
throwaway controller, configured with numExecutors: 0 and one agent that was not
connected:
[2026-11-10T06:24:09.062Z] Started by user admin
[2026-11-10T06:24:09.132Z] [Pipeline] Start of Pipeline
[2026-11-10T06:24:09.426Z] [Pipeline] node
[2026-11-10T06:24:24.443Z] Still waiting to schedule task
[2026-11-10T06:24:24.444Z] Waiting for next available executorThe Groovy part of a pipeline (the "flyweight" task that walks through your Jenkinsfile)
still runs on the controller even with zero executors; only node blocks and their steps
need an executor. That is one reason to keep heavy logic out of Jenkinsfile Groovy and in
shell scripts that run on agents.
How agents connect
An agent is a Java process running agent.jar (the "remoting" library) on another
machine. The controller sends it work over a single connection; files in the workspace
live on the agent's disk, not on the controller. There are two ways to establish that
connection:
| Inbound agent | SSH agent | |
|---|---|---|
| Who connects | agent → controller | controller → agent |
| Transport | WebSocket over the Jenkins HTTP(S) port, or a separate TCP port | SSH |
| Needs on the agent | Java, outbound access to the Jenkins URL | Java, sshd, an account the controller can log in to |
| Launch method in the UI | "Launch agent by connecting it to the controller" | "Launch agents via SSH" |
| Good for | agents behind NAT/firewalls, Windows, containers, cloud agents | Linux servers on a network the controller can reach |
For inbound agents I recommend WebSocket: it uses the same port and TLS certificate as the web UI, so there is no extra firewall rule. The alternative is the TCP port under Manage Jenkins → Security → Agents → TCP port for inbound agents, which offers Fixed, Random and Disable.
A permanent inbound agent
Defining the node
In the UI: Manage Jenkins → Nodes → New Node, enter a Node name, choose Permanent Agent, and click Create. The configuration form then asks for:
- Number of executors: how many builds this agent runs at once.
- Remote root directory: where the agent keeps workspaces and its own files.
- Labels: space-separated words that pipelines use to pick this agent.
- Usage: "Use this node as much as possible" or "Only build jobs with label
expressions matching this node". Use the second for specialised machines so
agent anyjobs do not land on them. - Launch method: "Launch agent by connecting it to the controller", with Use WebSocket ticked.
- Availability: "Keep this agent online as much as possible".
The same node as XML, created with the CLI (set up as in Installing Jenkins):
<slave>
<name>linux-1</name>
<description>Ubuntu 24.04 build agent, connects inbound over WebSocket</description>
<remoteFS>/home/sid/agent-1</remoteFS>
<numExecutors>2</numExecutors>
<mode>EXCLUSIVE</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.slaves.JNLPLauncher">
<workDirSettings>
<disabled>false</disabled>
<internalDir>remoting</internalDir>
<failIfWorkDirIsMissing>false</failIfWorkDirIsMissing>
</workDirSettings>
<webSocket>true</webSocket>
</launcher>
<label>ubuntu-24 jdk21 maven</label>
<nodeProperties/>
</slave>java -jar jenkins-cli.jar create-node linux-1 < node.xmlEXCLUSIVE is the XML name for "Only build jobs with label expressions matching this
node", and JNLPLauncher is the historical class name of the inbound launcher. I created
a second node, linux-2, the same way, with the labels ubuntu-24 jdk21 docs.
Starting the agent
The node's page in the UI shows the exact command to run on the agent machine. On my
controller it read (I replaced the 64-character secret with <SECRET>):
Run from agent command line: (Unix)
curl -sO http://localhost:8090/jnlpJars/agent.jar
java -jar agent.jar -url http://localhost:8090/ -secret <SECRET> -name "linux-1" -webSocket -workDir "/home/sid/agent-1"The secret is what proves to the controller that this process may act as
linux-1. Passing it on the command line puts it in the process list, so the page
also offers a variant that reads it from a file with -secret @secret-file. This script
does that. It fetches the secret from the node's jenkins-agent.jnlp endpoint with an
admin API token, stores it with mode 600, and starts the agent in the background:
#!/usr/bin/env bash
# start-agent.sh NODE WORKDIR: fetch the node's secret and start an inbound agent
set -euo pipefail
NODE=$1
WORKDIR=$2
mkdir -p "$WORKDIR"
umask 077
curl -sf -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" "${JENKINS_URL}computer/${NODE}/jenkins-agent.jnlp" |
sed -n 's|^<jnlp><application-desc><argument>\([0-9a-f]*\)</argument>.*|\1|p' > "$WORKDIR/secret-file"
setsid nohup java -Xmx256m -jar agent.jar -url "$JENKINS_URL" -name "$NODE" \
-secret @"$WORKDIR/secret-file" -webSocket -workDir "$WORKDIR" \
> "$WORKDIR/agent.log" 2>&1 < /dev/null &
echo "started $NODE (pid $!)"curl -sO "${JENKINS_URL}jnlpJars/agent.jar"
./start-agent.sh linux-1 ~/agent-1
java -jar jenkins-cli.jar wait-node-online linux-1
cat ~/agent-1/agent.logstarted linux-1 (pid 35641)
Nov 09, 2026 9:52:00 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/sid/agent-1/remoting as a remoting work directory
Nov 09, 2026 9:52:00 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/sid/agent-1/remoting
Nov 09, 2026 9:52:00 PM hudson.remoting.Launcher createEngine
INFO: Setting up agent: linux-1
Nov 09, 2026 9:52:00 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3386.v353e57a_1b_ea_0
Nov 09, 2026 9:52:00 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/sid/agent-1/remoting as a remoting work directory
Nov 09, 2026 9:52:00 PM hudson.remoting.Launcher$CuiListener status
INFO: WebSocket connection open
Nov 09, 2026 9:52:01 PM hudson.remoting.Launcher$CuiListener status
INFO: Connected"Connected" is the line you are waiting for. wait-node-online returns as soon as the
controller sees the agent, which makes it handy in provisioning scripts.
On a real machine, run the agent under a dedicated jenkins user and a service manager
(a systemd unit on Linux, a Windows service on Windows) so it comes back after a reboot.
In my lab I started it by hand because the agent and the controller shared one machine.
Removing an agent
Stop the process, then delete the node. Be careful with Disconnect: for an inbound
agent it only drops the current connection, and the agent reconnects by itself. When I
ran disconnect-node, the agent log showed it reconnecting within a second:
java -jar jenkins-cli.jar disconnect-node linux-1 linux-2 -m "Decommissioning"
tail -2 ~/agent-1/agent.logNov 09, 2026 10:01:42 PM hudson.remoting.Launcher$CuiListener status
INFO: ConnectedTo take an agent out of service, either mark it offline (shown in the exercises) or stop the process for good:
pkill -f -- '-name linux-[12]'
java -jar jenkins-cli.jar delete-node linux-1 linux-2
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" "${JENKINS_URL}computer/api/json?tree=computer%5BdisplayName%5D" |
jq -r '.computer[].displayName'Built-In NodeSSH agents
With an SSH agent, the controller does the connecting: it logs in with an
"SSH Username with private key" credential, copies agent.jar over, and starts it with
the Java you point it at. You need the SSH Build Agents plugin, an sshd on the
agent, Java on the agent, and the agent's public key in the controller's trust store.
<slave>
<name>ssh-1</name>
<description>Build agent started over SSH by the controller</description>
<remoteFS>/home/deploy/jenkins-agent</remoteFS>
<numExecutors>2</numExecutors>
<mode>EXCLUSIVE</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves">
<host>127.0.0.1</host>
<port>22</port>
<credentialsId>deploy-key</credentialsId>
<javaPath>/usr/bin/java</javaPath>
<launchTimeoutSeconds>60</launchTimeoutSeconds>
<maxNumRetries>0</maxNumRetries>
<retryWaitTime>15</retryWaitTime>
<sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"/>
<tcpNoDelay>true</tcpNoDelay>
</launcher>
<label>ubuntu-24 jdk21 ssh</label>
<nodeProperties/>
</slave>The fields worth thinking about:
- credentialsId: an SSH key credential. Scope it SYSTEM; only the controller needs it, never a pipeline.
- sshHostKeyVerificationStrategy:
KnownHostsFileKeyVerificationStrategychecks the agent's host key against~/.ssh/known_hostsof the user Jenkins runs as. Never use the non-verifying strategy outside a lab; it lets anything that answers on that address receive your build and its credentials. - maxNumRetries and retryWaitTime: how hard the controller keeps trying after a failed launch.
My lab machine has no SSH server, so this agent could not actually come online. Creating it and asking the controller to connect shows what a failed launch looks like in the node's log (Nodes → ssh-1 → Log, or the URL below):
java -jar jenkins-cli.jar create-node ssh-1 < ssh-node.xml
java -jar jenkins-cli.jar connect-node ssh-1
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" "${JENKINS_URL}computer/ssh-1/logText/progressiveText?start=0"SSHLauncher{host='127.0.0.1', port=22, credentialsId='deploy-key', jvmOptions='', javaPath='/usr/bin/java', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=60, maxNumRetries=0, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[11/09/26 21:58:23] [SSH] Opening SSH connection to 127.0.0.1:22.
Connection refused (connect failed)
ERROR: Connection is not established!
java.lang.IllegalStateException: Connection is not established!
[…]
[11/09/26 21:58:38] Launch failed - cleaning up connection
[11/09/26 21:58:38] [SSH] Connection closed.The node log is the first place to look whenever an agent will not come online: it shows
the connection attempt step by step. I deleted the node afterwards with
delete-node ssh-1.
Labels and label expressions
A label is a word attached to a node. Pipelines ask for a label expression, and
Jenkins picks any online node that matches. Every node's name also works as a label.
Labels should describe capabilities, not machines: jdk21, docker, ubuntu-24,
windows, gpu.
| Operator | Meaning | Example |
|---|---|---|
a && b | both | ubuntu-24 && docs |
a || b | either | maven || docs |
!a | not | jdk21 && !docs |
( ) | grouping | (maven || docs) && !windows |
Jenkins answers "which nodes match this expression?" at /label/EXPRESSION/api/json.
This small script URL-encodes an expression and asks:
#!/usr/bin/env bash
# label-check.sh 'EXPRESSION': which nodes does a label expression match?
expr=$1
enc=$(python3 -c 'import sys, urllib.parse; print(urllib.parse.quote(sys.argv[1], safe=""))' "$expr")
printf '%-28s -> ' "$expr"
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" "${JENKINS_URL}label/${enc}/api/json?tree=nodes%5BnodeName%5D" |
jq -r '[.nodes[].nodeName | if . == "" then "(built-in)" else . end]
| if length == 0 then "(no nodes)" else join(", ") end'With linux-1 (ubuntu-24 jdk21 maven) and linux-2 (ubuntu-24 jdk21 docs)
defined (label membership does not depend on whether an agent is connected):
for e in 'maven' 'ubuntu-24' 'ubuntu-24 && docs' 'jdk21 && !docs' 'maven || docs' \
'linux-2' 'gpu' '(maven || docs) && !windows' '!docs'; do
./label-check.sh "$e"
donemaven -> linux-1
ubuntu-24 -> linux-1, linux-2
ubuntu-24 && docs -> linux-2
jdk21 && !docs -> linux-1
maven || docs -> linux-1, linux-2
linux-2 -> linux-2
gpu -> (no nodes)
(maven || docs) && !windows -> linux-1, linux-2
!docs -> linux-1, (built-in)Look at the last line. A purely negative expression matches the built-in node too,
because the controller does not have the docs label either. If the built-in node still
has executors, agent { label '!docs' } can put a build on the controller. Always anchor a
negative with a positive label: jdk21 && !docs.
Choosing where a pipeline runs
One agent for the whole pipeline
pipeline {
agent { label 'maven' }
stages {
stage('Where am I?') {
steps {
sh '''
echo "node: $NODE_NAME"
echo "labels: $NODE_LABELS"
echo "executor: $EXECUTOR_NUMBER"
echo "workspace: $WORKSPACE"
java -version 2>&1 | head -1
'''
}
}
}
}[Pipeline] node
Running on linux-1 in /home/sid/agent-1/workspace/agent-where
[…]
+ echo node: linux-1
node: linux-1
+ echo labels: jdk21 maven ubuntu-24 linux-1
labels: jdk21 maven ubuntu-24 linux-1
+ echo executor: 1
executor: 1
+ echo workspace: /home/sid/agent-1/workspace/agent-where
workspace: /home/sid/agent-1/workspace/agent-where
+ java -version
+ head -1
openjdk version "21.0.13" 2026-10-20 LTSNODE_NAME, NODE_LABELS, EXECUTOR_NUMBER and WORKSPACE are set for every step
inside a node; print them whenever you are unsure where something ran.
agent none and an agent per stage
With agent none at the top, each stage asks for its own agent. That lets one pipeline
build on a machine with the build tools and test somewhere else. Files do not follow you
from one agent to the next, so pass them explicitly with stash and unstash:
pipeline {
agent none
stages {
stage('Build') {
agent { label 'maven' }
steps {
sh '''
mkdir -p src target/classes
cat > src/Version.java <<'EOF'
public class Version {
public static void main(String[] args) {
System.out.println("payments-api 1.4.2");
}
}
EOF
"$JAVA_HOME/bin/javac" -d target/classes src/Version.java
"$JAVA_HOME/bin/jar" --create --file target/payments-api.jar --main-class Version -C target/classes .
echo "built on $NODE_NAME"
'''
stash name: 'app-jar', includes: 'target/payments-api.jar'
}
}
stage('Smoke test') {
agent { label 'docs' }
steps {
sh 'ls target || echo "no target/ here: this is a different machine"'
unstash 'app-jar'
sh 'java -jar target/payments-api.jar && echo "tested on $NODE_NAME"'
}
}
}
}[Pipeline] { (Build)
[Pipeline] node
Running on linux-1 in /home/sid/agent-1/workspace/agent-split
[…]
+ echo built on linux-1
built on linux-1
[Pipeline] stash
Stashed 1 file(s)
[…]
[Pipeline] { (Smoke test)
[Pipeline] node
Running on linux-2 in /home/sid/agent-2/workspace/agent-split
[Pipeline] {
[Pipeline] sh
+ ls target
ls: cannot access 'target': No such file or directory
+ echo no target/ here: this is a different machine
no target/ here: this is a different machine
[Pipeline] unstash
[Pipeline] sh
+ java -jar target/payments-api.jar
payments-api 1.4.2
+ echo tested on linux-2
tested on linux-2Stashes travel through the controller and are stored there for the life of the build, so
keep them small (a JAR, a test report). Big artifacts belong in an artifact repository.
With agent none, a stage that has no agent of its own cannot run sh at all (the
error is in Common mistakes). That is the right default: it forces you to say where work
happens.
A custom workspace
agent { node { … } } is the long form of label, and it accepts customWorkspace:
pipeline {
agent {
node {
label 'maven'
customWorkspace 'payments-release'
}
}
stages {
stage('Where') {
steps {
sh 'pwd'
}
}
}
}[Pipeline] node
Running on linux-1 in /home/sid/agent-1/workspace/agent-custom-ws
[Pipeline] {
[Pipeline] ws
Running in /home/sid/agent-1/payments-release
[…]
+ pwd
/home/sid/agent-1/payments-releaseA relative path is resolved against the agent's remote root directory, not against
workspace/. Use this sparingly: several jobs sharing one directory will trip over each
other.
Scripted pipeline: node, ws and dir
In scripted pipeline the same ideas are steps:
node('maven') {
echo "default workspace: ${env.WORKSPACE}"
ws('shared/payments-release') {
echo "inside ws(): ${pwd()}"
}
dir('reports') {
echo "inside dir(): ${pwd()}"
}
}[Pipeline] node
Running on linux-1 in /home/sid/agent-1/workspace/agent-scripted
[Pipeline] {
[Pipeline] echo
default workspace: /home/sid/agent-1/workspace/agent-scripted
[Pipeline] ws
Running in /home/sid/agent-1/shared/payments-release
[…]
inside ws(): /home/sid/agent-1/shared/payments-release
[…]
[Pipeline] dir
Running in /home/sid/agent-1/workspace/agent-scripted/reports
[…]
inside dir(): /home/sid/agent-1/workspace/agent-scripted/reportsnode allocates an executor and a workspace. ws allocates a different workspace
(locked, so two builds cannot use it at once). dir only changes the current directory
inside the workspace you already have.
Workspaces on agents
Each agent keeps workspaces under <remote root>/workspace/<job name>, and they persist
between builds. That makes incremental builds fast and stale files a classic source of
"works on one agent, fails on another". Three details to know:
-
Concurrent builds of the same job on the same agent get
@2,@3, … I started two builds of a job with a ten-second sleep on an agent with two executors:Running on linux-1 in /home/sid/agent-1/workspace/agent-concurrent + echo executor 1 using /home/sid/agent-1/workspace/agent-concurrent executor 1 using /home/sid/agent-1/workspace/agent-concurrentRunning on linux-1 in /home/sid/agent-1/workspace/agent-concurrent@2 + echo executor 0 using /home/sid/agent-1/workspace/agent-concurrent@2 executor 0 using /home/sid/agent-1/workspace/agent-concurrent@2Scripts that hard-code the workspace path break on the second concurrent build. Use
$WORKSPACE. -
Every workspace has a sibling
@tmpdirectory (WORKSPACE_TMP) where Jenkins keeps step scripts and credential files. -
Clean up deliberately.
deleteDir()or the Workspace Cleanup plugin'scleanWs()in apostblock gives you a fresh checkout every time, at the cost of a slower build.
Docker and Kubernetes agents
Permanent agents need someone to install and patch the build tools on them. Container agents move the toolchain into an image, so each build gets the exact JDK, Maven or Node version it asks for.
Docker
With the Docker Pipeline plugin and Docker installed on an agent, a stage can run inside a container:
pipeline {
agent {
docker {
image 'maven:3.9.9-eclipse-temurin-21'
label 'maven'
args '-v $HOME/.m2:/root/.m2'
}
}
stages {
stage('Build') {
steps {
sh 'mvn -B -ntp verify'
}
}
}
}label picks an agent that has Docker; Jenkins pulls the image there, starts a container
with the workspace mounted, and runs every sh step inside it. args adds docker run
options, here a cache for Maven downloads. agent { dockerfile true } builds the image
from a Dockerfile in the repository instead.
The declarative linter accepts this file:
java -jar jenkins-cli.jar declarative-linter < Jenkinsfile.dockerJenkinsfile successfully validated.My lab agent had no Docker, so running it fails at the first Docker command. This is
also exactly what you see when a Docker pipeline lands on an agent without Docker, which
is why the label line matters:
[Pipeline] node
Running on linux-1 in /home/sid/agent-1/workspace/agent-docker
[Pipeline] {
[Pipeline] isUnix
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
+ docker inspect -f . maven:3.9.9-eclipse-temurin-21
The command 'docker' could not be found in this WSL 2 distro.
[…]
[Pipeline] sh
+ docker pull maven:3.9.9-eclipse-temurin-21
The command 'docker' could not be found in this WSL 2 distro.
[…]
ERROR: script returned exit code 1
Finished: FAILUREThe "WSL 2 distro" wording comes from my lab machine; a plain Linux agent prints the shell's usual "docker: not found" instead.
Kubernetes
The Kubernetes plugin goes one step further: it creates a pod for each build and deletes it afterwards, so agents only exist while there is work. The cluster connection is configured under Manage Jenkins → Clouds, and a pipeline describes its pod:
pipeline {
agent {
kubernetes {
yaml '''
apiVersion: v1
kind: Pod
spec:
containers:
- name: maven
image: maven:3.9.9-eclipse-temurin-21
command: ["sleep"]
args: ["infinity"]
resources:
requests: {cpu: "1", memory: "2Gi"}
limits: {memory: "2Gi"}
'''
defaultContainer 'maven'
}
}
stages {
stage('Build') {
steps {
sh 'mvn -B -ntp verify'
}
}
}
}The plugin adds its own jnlp container running the inbound agent; defaultContainer
makes sh steps run in the maven container. The requests are what the Kubernetes
scheduler uses to place the pod, so they are your capacity planning.
I had no Kubernetes cluster and the plugin was not installed, which the linter makes
obvious. The kubernetes agent type only exists once the plugin provides it:
java -jar jenkins-cli.jar declarative-linter < Jenkinsfile.k8sErrors encountered validating Jenkinsfile:
WorkflowScript: 3: Invalid agent type "kubernetes" specified. Must be one of [any, docker, dockerfile, label, none] @ line 3, column 9.
kubernetes {
^Capacity planning
How many agents and executors do you need? Measure, then adjust. Three numbers tell you most of what you need:
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
"${JENKINS_URL}computer/api/json?tree=busyExecutors,totalExecutors,computer%5BdisplayName,numExecutors,offline,idle%5D" | jq .{
"_class": "hudson.model.ComputerSet",
"busyExecutors": 0,
"computer": [
{
"_class": "hudson.model.Hudson$MasterComputer",
"displayName": "Built-In Node",
"idle": true,
"numExecutors": 2,
"offline": false
},
{
"_class": "hudson.slaves.SlaveComputer",
"displayName": "linux-1",
"idle": true,
"numExecutors": 2,
"offline": false
},
{
"_class": "hudson.slaves.SlaveComputer",
"displayName": "linux-2",
"idle": true,
"numExecutors": 2,
"offline": false
}
],
"totalExecutors": 6
}- Busy versus total executors over a working day. If busy sits at total for long stretches, builds are queueing.
- The queue and why items wait:
/queue/api/json?tree=items[why]. "Waiting for next available executor" means you need capacity; "There are no nodes with the label …" means a typo or a missing pool. - Node health: Jenkins monitors every agent and takes it offline when it runs low on disk. The defaults, visible in the agent's API, are 1 GiB free to stay online and 2 GiB for a warning:
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" "${JENKINS_URL}computer/linux-1/api/json?tree=monitorData%5B*%5D" | jq .{
"_class": "hudson.slaves.SlaveComputer",
"monitorData": {
"hudson.node_monitors.SwapSpaceMonitor": {
"_class": "hudson.node_monitors.SwapSpaceMonitor$MemoryUsage2",
"availablePhysicalMemory": 1575145472,
"availableSwapSpace": 2147483648,
"totalPhysicalMemory": 8230989824,
"totalSwapSpace": 2147483648
},
[…]
"hudson.node_monitors.DiskSpaceMonitor": {
"_class": "hudson.node_monitors.DiskSpaceMonitorDescriptor$DiskSpace",
"timestamp": 1794279392226,
"path": "/home/sid/agent-1",
"size": 1021268914176,
"threshold": 1073741824,
"totalSize": 1081101176832,
"warningThreshold": 2147483648
},
"hudson.node_monitors.ArchitectureMonitor": "Linux (amd64)",
"hudson.node_monitors.ResponseTimeMonitor": {
"_class": "hudson.node_monitors.ResponseTimeMonitor$Data",
"timestamp": 1794279392238,
"average": 362
},
[…]
}
}Rules of thumb that hold up well:
- Executors per agent: about one per CPU core for compile-heavy work, fewer if builds are memory-hungry (two Gradle daemons at 2 GB each on an 8 GB agent is already tight). One executor per agent is the simplest setup when builds are not isolated from each other.
- Pools by capability: a general Linux pool, plus small pools for what is scarce or special (Windows, macOS, GPU, hardware test rigs). Label each pool by what it offers.
- Elastic for bursts, permanent for the special cases: Kubernetes or cloud agents absorb the 9 a.m. spike; permanent agents host the machines you cannot containerise.
- Keep the controller light: it runs the Groovy of every pipeline, serves the UI and
stores logs. Its CPU goes up with the number of concurrent pipelines even when all
shsteps run elsewhere. Controller sizing is covered in Installing Jenkins.
Where you'll use this
Most teams end up with a layout like this:
| Pool | Labels | Typical work |
|---|---|---|
| General Linux (Kubernetes pods or VMs) | linux, jdk21, docker | builds, unit tests, Docker images |
| Windows VMs | windows, msbuild | .NET builds, installer packaging |
| Deploy hosts inside the production network | deploy-prod | the only machines with routes and credentials for production |
| Special hardware | gpu, android-device | model training, device tests |
A common pattern is to fan independent checks out to different pools so they run at the same time. This is the CI stage of a payments service, with unit tests on the build pool and documentation lint on the docs pool:
pipeline {
agent none
stages {
stage('Checks') {
parallel {
stage('Unit tests') {
agent { label 'maven' }
steps {
sh 'echo "unit tests on $NODE_NAME"; sleep 10'
}
}
stage('Docs lint') {
agent { label 'docs' }
steps {
sh 'echo "docs lint on $NODE_NAME"; sleep 10'
}
}
}
}
}
}[Pipeline] { (Checks)
[Pipeline] parallel
[Pipeline] { (Branch: Unit tests)
[Pipeline] { (Branch: Docs lint)
[Pipeline] stage
[Pipeline] { (Unit tests)
[Pipeline] stage
[Pipeline] { (Docs lint)
[Pipeline] node
Running on linux-1 in /home/sid/agent-1/workspace/agent-parallel
[Pipeline] node
Running on linux-2 in /home/sid/agent-2/workspace/agent-parallel
[Pipeline] {
[Pipeline] {
[Pipeline] sh
[Pipeline] sh
+ echo unit tests on linux-1
unit tests on linux-1
+ sleep 10
+ echo docs lint on linux-2
docs lint on linux-2
+ sleep 10
[…]
Finished: SUCCESSBoth branches got a node at the same moment, on different machines. The production deploy
stage of the same pipeline would use agent { label 'deploy-prod' }, and the credentials
it needs would live in a folder only that job can see, so a compromised build agent in
the general pool never holds production secrets.
Common mistakes
A label nobody has
A typo in a label, or a pool that is all offline, does not fail the build. It waits:
pipeline {
agent { label 'gpu' }
stages {
stage('Train model') {
steps {
sh 'nvidia-smi'
}
}
}
}Started from command line by admin
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
There are no nodes with the label ‘gpu’The queue says the same thing:
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" "${JENKINS_URL}queue/api/json?tree=items%5Bid,task%5Bname%5D,why%5D" | jq .{
"_class": "hudson.model.Queue",
"items": [
{
"_class": "hudson.model.Queue$BuildableItem",
"id": 93,
"task": {
"_class": "org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask",
"name": "part of agent-gpu #2"
},
"why": "There are no nodes with the label ‘gpu’"
}
]
}Check the expression with label-check.sh, fix it, and abort the waiting build
(java -jar jenkins-cli.jar stop-builds JOB).
The only matching agent is offline
When I stopped the linux-2 process and started a job that needs docs, the build
waited with a different message:
Started from command line by admin
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘linux-2’ is offlineAs soon as the agent was started again, the same build continued on its own:
Running on linux-2 in /home/sid/agent-2/workspace/agent-docs
[…]
+ echo building docs on linux-2
building docs on linux-2
[…]
Finished: SUCCESSA pool of one is a single point of failure. Give every label at least two agents.
Wrong secret or wrong node name
An agent started with a secret that does not belong to the node name it claims keeps retrying, with a message that does not mention the secret at all. For this test I wrote 64 zeros into the secret file:
java -jar agent.jar -url "$JENKINS_URL" -name linux-1 -secret @/tmp/jkb-bad/secret-file -webSocket -workDir /tmp/jkb-bad[…]
INFO: Using Remoting version: 3386.v353e57a_1b_ea_0
[…]
Nov 09, 2026 9:57:25 PM hudson.remoting.Engine$1HeaderHandler afterResponse
WARNING: Did not receive X-Remoting-Capability header
Nov 09, 2026 9:57:25 PM hudson.remoting.Launcher$CuiListener status
INFO: Failed to connect: Handshake error.
Nov 09, 2026 9:57:25 PM hudson.remoting.Launcher$CuiListener status
INFO: Waiting 1 second before retry
[…]
INFO: Waiting 7 seconds before retryA valid secret with a misspelled -name (linux-9) produced exactly the same
"Handshake error". Each node has its own secret, so copy both the name and the secret from
the node's own page, or fetch them together as start-agent.sh does.
A stage without an agent under agent none
After switching a pipeline to agent none, it is easy to miss one stage:
pipeline {
agent none
stages {
stage('Report') {
steps {
sh 'echo "where am I?"'
}
}
}
}[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (Report)
[Pipeline] sh
[Pipeline] error
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: Attempted to execute a step that requires a node context while ‘agent none’ was specified. Be sure to specify your own ‘node { ... }’ blocks when using ‘agent none’.
Finished: FAILUREGive the stage its own agent { label '…' }. Steps such as echo, input and
milestone run without a node; anything that touches files or runs a process needs one.
Expecting files from an earlier stage
The ls target failure in the agent none example is the most common surprise in
multi-agent pipelines: the second stage ran on another machine with an empty workspace.
Use stash/unstash for small files and an artifact repository for large ones, or keep
the stages that share files on one agent.
Summary
- Builds on the built-in node run as the Jenkins user and can read
JENKINS_HOME, includingsecrets/andcredentials.xml. Set the built-in node's executors to 0 and build on agents. - Inbound agents connect to the controller (WebSocket over the normal HTTP port is the
easiest); SSH agents are started by the controller over SSH and need
sshd, Java and a verified host key on the agent. agent.jar -secret @file -name NODE -webSocket -workDir DIRstarts an inbound agent; the secret is per node. "Disconnect" does not stop an inbound agent; mark it offline or stop the process.- Label expressions use
&&,||,!and parentheses. Node names are labels too. Purely negative expressions also match the built-in node. agent noneplus per-stageagentputs stages on different machines; workspaces do not follow, so usestash/unstash. RelativecustomWorkspaceandwspaths are resolved against the agent's remote root.- Concurrent builds of one job on one agent get
@2,@3workspaces; scripts must use$WORKSPACE, never a hard-coded path. - Docker agents need Docker on the chosen agent; Kubernetes agents need the Kubernetes
plugin, which adds the
kubernetesagent type and a cloud under Manage Jenkins → Clouds. - Plan capacity from busy versus total executors and the queue's "why" messages; give every label at least two agents.
Exercises
Given linux-1 with labels ubuntu-24 jdk21 maven, linux-2 with
ubuntu-24 jdk21 docs, and a built-in node with only the built-in label, predict which
nodes match each expression, then check with label-check.sh:
jdk21 && !mavendocs || gpuubuntu-24 && jdk21 && maven!docs
Show Solution
for e in 'jdk21 && !maven' 'docs || gpu' 'ubuntu-24 && jdk21 && maven' '!docs'; do
./label-check.sh "$e"
donejdk21 && !maven -> linux-2
docs || gpu -> linux-2
ubuntu-24 && jdk21 && maven -> linux-1
!docs -> linux-1, (built-in)The fourth one is the trap: the built-in node lacks docs, so it matches !docs.
Write a declarative pipeline with agent none and one stage, Checks, containing two
parallel stages: Unit tests on any agent labelled maven and Docs lint on any agent
labelled docs. Each should print the node it ran on. Confirm from the log that they ran
on different nodes at the same time.
Show Solution
The pipeline is the one in "Where you'll use this". The two Running on lines appear
back to back, one for each node, before either sh step finishes:
[Pipeline] node
Running on linux-1 in /home/sid/agent-1/workspace/agent-parallel
[Pipeline] node
Running on linux-2 in /home/sid/agent-2/workspace/agent-parallelEach parallel branch has its own agent, so each gets its own executor and workspace.
If both labels matched the same agent with two executors, they would still run at the
same time, in …/agent-parallel and …/agent-parallel@2.
You need to upgrade the JDK on linux-2. Mark it temporarily offline with a reason,
start a job that needs the docs label and confirm it waits, check the reason through the
REST API, then bring the agent back and confirm the waiting build completes.
Hint
The CLI commands are offline-node (with -m) and online-node.
Show Solution
java -jar jenkins-cli.jar offline-node linux-2 -m "Upgrading the JDK on this host"
java -jar jenkins-cli.jar build agent-docs
sleep 20
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" "${JENKINS_URL}queue/api/json?tree=items%5Bwhy%5D" | jq -r '.items[].why'
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
"${JENKINS_URL}computer/linux-2/api/json?tree=offline,temporarilyOffline,offlineCauseReason" | jq .‘linux-2’ is offline
{
"_class": "hudson.slaves.SlaveComputer",
"offline": true,
"offlineCauseReason": "Upgrading the JDK on this host",
"temporarilyOffline": true
}java -jar jenkins-cli.jar online-node linux-2
sleep 10
java -jar jenkins-cli.jar console agent-docs 2Started from command line by admin
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘linux-2’ is offline
Running on linux-2 in /home/sid/agent-2/workspace/agent-docs
[…]
building docs on linux-2
[…]
Finished: SUCCESSUnlike disconnecting, marking a node temporarily offline keeps the agent process connected (so you can still see its logs and monitors) but stops new builds from being scheduled on it. The reason shows on the node's page for everyone else.
Create a new permanent inbound node linux-3 with one executor, the labels
ubuntu-24 docs, "Only build jobs with label expressions matching this node", and
WebSocket enabled. Start it with agent.jar, run a pipeline pinned to it by name, then
remove it completely: stop the process, wait until the controller sees it offline, delete
the node and its directory. Show that the docs label no longer includes it.
Show Solution
<slave>
<name>linux-3</name>
<description>Temporary docs agent</description>
<remoteFS>/home/sid/agent-3</remoteFS>
<numExecutors>1</numExecutors>
<mode>EXCLUSIVE</mode>
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
<launcher class="hudson.slaves.JNLPLauncher">
<webSocket>true</webSocket>
</launcher>
<label>ubuntu-24 docs</label>
<nodeProperties/>
</slave>pipeline {
agent { label 'linux-3' }
stages {
stage('Pinned') {
steps {
sh 'echo "running on $NODE_NAME with labels: $NODE_LABELS"'
}
}
}
}With the pipeline saved as a job named agent-pinned:
java -jar jenkins-cli.jar create-node linux-3 < node3.xml && echo created
./start-agent.sh linux-3 ~/agent-3
java -jar jenkins-cli.jar wait-node-online linux-3 && echo online
./label-check.sh 'docs'
java -jar jenkins-cli.jar build agent-pinned -s -v | grep -E 'running on|Finished'
pkill -f -- '-name linux-3'
java -jar jenkins-cli.jar wait-node-offline linux-3 && echo offline
java -jar jenkins-cli.jar delete-node linux-3 && echo deleted
rm -rf ~/agent-3
./label-check.sh 'docs'created
started linux-3 (pid 65698)
online
docs -> linux-2, linux-3
+ echo running on linux-3 with labels: docs ubuntu-24 linux-3
running on linux-3 with labels: docs ubuntu-24 linux-3
Finished: SUCCESS
offline
deleted
docs -> linux-2The node XML needs only the launcher's webSocket flag; the work-directory settings shown
earlier are defaults. Pinning by node name (label 'linux-3') is handy for a test
like this but a bad habit in real pipelines: when that one machine is gone, the job waits
forever.
Tested with: Jenkins 2.580.2 on OpenJDK 21 (Ubuntu 24.04 under WSL2), Remoting
3386.v353e57a_1b_ea_0, SSH Build Agents 3.1097.v868116049892, Docker Pipeline
653.v2f2c08eff0ec, November 2026. The inbound agents ran as real agent.jar processes on
the same machine as the controller, connected over WebSocket, and were deleted afterwards;
every pipeline and REST call shown was run against that controller. The zero-executor
output came from a separate throwaway controller configured with JCasC. Three things could
not run in this lab and are shown as configuration only: the SSH agent (no sshd, so it
stops at "Connection refused"), the Docker agent (no Docker; the real failure is shown),
and the Kubernetes agent (no cluster and no Kubernetes plugin; the linter output is
shown). UI field names were checked against the configuration pages served by 2.580.2,
fetched over HTTP, not clicked through. Paths from the test machine's scratch directories are shown shortened to /home/sid/…. Job, node and credential names are shown without the test run's prefixes.
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Jenkins tutorial here is tested end to end before publishing.