Credentials and Secrets
How Jenkins stores, scopes and injects secrets: the five credential types, credentials() and withCredentials, log masking, and the ways secrets still leak.
By The Dark DebuggerUpdated 27 min read
Every useful pipeline eventually needs a secret: a registry password, an SSH key for the deploy user, a kubeconfig, an API token. This article covers how Jenkins stores those secrets, the five credential types, how scopes and folders decide who can use them, how to inject them into a pipeline, and the specific ways they still end up in logs and artifacts. It assumes you can write and run a declarative pipeline, as covered in Your First Pipeline and Declarative Pipeline in Depth.
Why secrets need their own system
Picture a payments team whose deploy job needs four things: a password to push images to
the registry, an SSH key to restart the service on the app servers, a kubeconfig for the
staging cluster, and a token for the code-quality server. The quickest way to get the job
green is to paste those values into the Jenkinsfile or into job parameters. Now everyone
who can read the repository, or the job's config.xml, or the build log, has production
access.
Jenkins credentials solve three separate problems:
- Storage. Secrets live in the controller, encrypted at rest, not in Git and not in job configuration.
- Injection. A pipeline refers to a secret by ID. The value is handed to the build only for the duration of a block, as an environment variable or a temporary file.
- Masking and scope. The value is replaced by
****in the console log, and the credential is only visible to the jobs that should use it.
None of these is perfect, and the last two sections of this article show exactly where they stop protecting you.
How Jenkins stores a credential
Credentials in the global store are saved in $JENKINS_HOME/credentials.xml. Secret
fields are encrypted with keys that live in $JENKINS_HOME/secrets/. On my test
controller, the password of a username/password credential looks like this on disk:
grep -n -A4 '<id>registry</id>' "$JENKINS_HOME/credentials.xml"17: <id>registry</id>
18- <description>Container registry push account</description>
19- <username>ci-pusher</username>
20- <password>{AQAAABAAAAAgvclqzJqHpr7i6PZd+NgoSLVrLcXvF7woHdVuskTx8JTdrLxDKYtGeiOoj/olWnHU}</password>
21- </com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>The value in braces is ciphertext. The key that decrypts it is
secrets/hudson.util.Secret, itself protected by secrets/master.key. Two consequences
follow, and both matter more than the encryption itself:
- A backup of
JENKINS_HOMEthat includessecrets/is a backup of every password. Treat it like one. A backup withoutsecrets/restores credentials you can no longer decrypt. - Anyone with Overall/Administer can read every secret. The script console runs arbitrary Groovy inside the controller. This is what an administrator can do in two lines:
def xml = new File(System.getenv('JENKINS_HOME'), 'credentials.xml').text
def encrypted = (xml =~ /<id>registry<\/id>[\s\S]*?<password>(\{[^}]+\})<\/password>/)[0][1]
println "stored: ${encrypted}"
println "decrypted: ${hudson.util.Secret.decrypt(encrypted).plainText}"java -jar jenkins-cli.jar groovy = < decrypt.groovystored: {AQAAABAAAAAgvclqzJqHpr7i6PZd+NgoSLVrLcXvF7woHdVuskTx8JTdrLxDKYtGeiOoj/olWnHU}
decrypted: Wq8-tVz3-Kp2r-Ld9xEncryption at rest protects against someone who finds a copy of credentials.xml alone.
It does not protect against administrators, and it is not meant to. Keep the admin group
small.
The five credential types
These are the types provided by the Credentials, Plain Credentials and SSH Credentials plugins, which a controller set up with the suggested plugins already has:
| Type in the UI | Holds | Binding in withCredentials | Typical use |
|---|---|---|---|
| Secret text | one string | string | API tokens, webhook URLs, Sonar tokens |
| Username with password | username + password | usernamePassword | registries, databases, HTTP basic auth |
| SSH Username with private key | username + key (+ passphrase) | sshUserPrivateKey | Git over SSH, deploying to servers |
| Secret file | a file's bytes + file name | file | kubeconfig, service-account JSON, .npmrc |
| Certificate | PKCS#12 keystore + password | certificate | mTLS client certificates, code signing |
Other plugins add more types. On my controller, the add dialog also listed "GitHub App" (from the GitHub Branch Source plugin) and "X.509 Client Certificate" (from Docker Commons).
Creating credentials
In the UI
In Jenkins 2.580: Manage Jenkins → Credentials, click the System store, then the Global domain ("Credentials that should be available everywhere"), then Add Credentials. A dialog titled "Select a type of credential" lists the types from the table above; pick one, click Next, and fill in the form. For a username/password credential the form has Scope, Username, a Treat username as secret checkbox, Password, ID and Description.
Always fill in the ID yourself. If you leave it blank Jenkins generates a UUID, and
your Jenkinsfiles end up full of credentials('3f6c1e0a-…') that nobody can read.
From the command line
Clicking through forms does not scale past a handful of secrets, and it leaves no record. The Jenkins CLI can create credentials from XML. Download the CLI from your controller and point it at your server with environment variables, as set up in Installing Jenkins (use an API token from your user page, not your password):
export JENKINS_URL=http://localhost:8090/
export JENKINS_USER_ID=admin
export JENKINS_API_TOKEN=... # your API token
curl -sO "${JENKINS_URL}jnlpJars/jenkins-cli.jar"
java -jar jenkins-cli.jar who-am-iAuthenticated as: admin
Authorities:
authenticated
FACTOR_PASSWORDThis script creates one credential of each type in the global store. It reads the key, the kubeconfig and the PKCS#12 file from the current directory, so no secret material is typed into the script except the two demo passwords:
#!/usr/bin/env bash
# Creates the five demo credentials in the global store (Manage Jenkins > Credentials > System > Global credentials).
set -euo pipefail
STORE=system::system::jenkins # provider::resolver::context
DOMAIN=_ # "_" is the global domain
jcli() { java -jar jenkins-cli.jar "$@"; }
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
<scope>GLOBAL</scope>
<id>sonar-token</id>
<description>SonarQube analysis token</description>
<secret>squ_7d1e4f8a3c6b0d2e5f9c2e7a1b8d3f60e5a2c9b7</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
EOF
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>registry</id>
<description>Container registry push account</description>
<username>ci-pusher</username>
<password>Wq8-tVz3-Kp2r-Ld9x</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
<scope>GLOBAL</scope>
<id>deploy-key</id>
<description>SSH key for the deploy user on the app servers</description>
<username>deploy</username>
<privateKeySource class="com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey\$DirectEntryPrivateKeySource">
<privateKey>$(cat deploy_key)</privateKey>
</privateKeySource>
</com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
EOF
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<org.jenkinsci.plugins.plaincredentials.impl.FileCredentialsImpl>
<scope>GLOBAL</scope>
<id>kubeconfig</id>
<description>kubeconfig for the staging cluster</description>
<fileName>kubeconfig</fileName>
<secretBytes>$(base64 -w0 kubeconfig)</secretBytes>
</org.jenkinsci.plugins.plaincredentials.impl.FileCredentialsImpl>
EOF
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>
<scope>GLOBAL</scope>
<id>client-cert</id>
<description>mTLS client certificate for the artifact API</description>
<keyStoreSource class="com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl\$UploadedKeyStoreSource">
<uploadedKeystoreBytes>$(base64 -w0 client.p12)</uploadedKeystoreBytes>
</keyStoreSource>
<password>changeit</password>
</com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>
EOFThe SSH key came from ssh-keygen -t ed25519 -N '' -C 'deploy@jenkins' -f deploy_key, and
client.p12 from openssl pkcs12 -export with the password changeit. Note the escaped
\$ in the two class= attributes: inside an unquoted heredoc, an unescaped
$DirectEntryPrivateKeySource would be expanded by Bash to an empty string.
./create-credentials.sh
java -jar jenkins-cli.jar list-credentials system::system::jenkins=================================================================================================
Domain (global)
Description
# of Credentials 5
=================================================================================================
Id Name
================ ================================================================================
sonar-token SonarQube analysis token
registry Container registry push account
deploy-key SSH key for the deploy user on the app servers
kubeconfig kubeconfig (kubeconfig for the staging cluster)
client-cert O=Example Payments, CN=jenkins-ci (mTLS client certificate for the artifact API)
=================================================================================================Reading a credential back never returns the secret:
java -jar jenkins-cli.jar get-credentials-as-xml system::system::jenkins _ registry<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl plugin="credentials@1514.vb_01eb_014e7d4">
<scope>GLOBAL</scope>
<id>registry</id>
<description>Container registry push account</description>
<username>ci-pusher</username>
<password>
<secret-redacted/>
</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>A script like this is fine for a one-off migration. For a controller you rebuild regularly, define credentials in Configuration as Code instead, with the values coming from the environment; that is covered in Jenkins Configuration as Code.
Scopes: Global, System, and folders
Every credential has a scope:
- GLOBAL: usable by Jenkins itself, by agents, and by jobs and pipelines.
- SYSTEM: usable only by Jenkins itself, for example the bind account of an LDAP security realm or the key the controller uses to launch SSH agents. Pipelines cannot see it.
Separately, every credential lives in a store. The global store belongs to the controller. Every folder (from the Folders plugin) has its own store too, and a credential in a folder's store is only visible to jobs inside that folder. That is the main tool for least privilege: give each team a folder, put the team's secrets in it.
Let's prove both rules. This script creates a payments folder, a database
credential in that folder's store, and a SYSTEM-scoped LDAP credential in the global
store:
#!/usr/bin/env bash
# Folder-scoped and SYSTEM-scoped credentials for the scope demo.
set -euo pipefail
jcli() { java -jar jenkins-cli.jar "$@"; }
# A folder for the payments team
jcli create-job payments <<'EOF'
<com.cloudbees.hudson.plugins.folder.Folder plugin="cloudbees-folder">
<description>Payments team jobs</description>
</com.cloudbees.hudson.plugins.folder.Folder>
EOF
# A credential that lives in that folder's own store
jcli create-credentials-by-xml folder::item::payments _ <<'EOF'
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>payments-db</id>
<description>payments database, app user</description>
<username>payments_app</username>
<password>N7#vq/Lx+29=Tz</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
# A SYSTEM-scoped credential: for Jenkins itself, not for jobs
jcli create-credentials-by-xml system::system::jenkins _ <<'EOF'
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>SYSTEM</scope>
<id>ldap-bind</id>
<description>LDAP bind account used by the security realm</description>
<username>cn=jenkins,ou=services,dc=example,dc=com</username>
<password>Lb7-q2Rz-p9Xw</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
jcli list-credentials folder::item::payments============================================
Domain (global)
Description
# of Credentials 1
============================================
Id Name
================ ===========================
payments-db payments database, app user
============================================The same pipeline goes into two jobs: payments/db-check inside the folder, and
db-check-outside at the top level.
pipeline {
agent any
stages {
stage('Check DB credential') {
steps {
withCredentials([usernamePassword(credentialsId: 'payments-db',
usernameVariable: 'DB_USER',
passwordVariable: 'DB_PASS')]) {
sh 'echo "DB user is $DB_USER, password has ${#DB_PASS} characters"'
}
}
}
}
}Inside the folder:
[Pipeline] withCredentials
Masking supported pattern matches of $DB_USER or $DB_PASS
[Pipeline] {
[Pipeline] sh
+ echo DB user is ****, password has 14 characters
DB user is ****, password has 14 characters
[…]
Finished: SUCCESSOutside the folder, with an identical Jenkinsfile:
[Pipeline] withCredentials
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Could not find credentials entry with ID 'payments-db'
Finished: FAILUREAnd a job that tries to use the SYSTEM-scoped LDAP account fails the same way, even though the credential is in the global store:
ERROR: Could not find credentials entry with ID 'ldap-bind'
Finished: FAILURENotice that the error is identical for "does not exist", "wrong folder" and "wrong scope". When you see it, check all three before you assume a typo.
There is also a third kind of store: each user has a personal one (your user name → Credentials). Those are meant for things a user triggers with their own identity and are rarely what you want for shared pipelines.
Using credentials in the environment directive
The shortest way to use a credential in declarative pipeline is the credentials()
helper inside environment. What it sets depends on the type:
pipeline {
agent any
environment {
SONAR_TOKEN = credentials('sonar-token') // secret text
REGISTRY = credentials('registry') // username with password
KUBECONFIG = credentials('kubeconfig') // secret file
DEPLOY_KEY = credentials('deploy-key') // SSH username with private key
}
stages {
stage('What the build sees') {
steps {
sh '''
echo "SONAR_TOKEN = $SONAR_TOKEN"
echo "REGISTRY = $REGISTRY"
echo "REGISTRY_USR = $REGISTRY_USR"
echo "REGISTRY_PSW = $REGISTRY_PSW"
echo "KUBECONFIG = $KUBECONFIG"
echo "DEPLOY_KEY = $DEPLOY_KEY"
echo "DEPLOY_KEY_USR = $DEPLOY_KEY_USR"
'''
}
}
}
}[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG or $SONAR_TOKEN or $DEPLOY_KEY or $DEPLOY_KEY_PSW or $DEPLOY_KEY_USR or $REGISTRY or $REGISTRY_USR or $REGISTRY_PSW
[Pipeline] {
[Pipeline] stage
[Pipeline] { (What the build sees)
[Pipeline] sh
+ echo SONAR_TOKEN = ****
SONAR_TOKEN = ****
+ echo REGISTRY = ****
REGISTRY = ****
+ echo REGISTRY_USR = ****
REGISTRY_USR = ****
+ echo REGISTRY_PSW = ****
REGISTRY_PSW = ****
+ echo KUBECONFIG = ****
KUBECONFIG = ****
+ echo DEPLOY_KEY = ****
DEPLOY_KEY = ****
+ echo DEPLOY_KEY_USR = ****
DEPLOY_KEY_USR = ****The first line tells you which variables are being watched. The rules:
| Credential type | X = credentials('id') sets |
|---|---|
| Secret text | X = the secret |
| Username with password | X = user:password, X_USR = user, X_PSW = password |
| Secret file | X = path to a temporary copy of the file |
| SSH Username with private key | X = path to a temporary key file, X_USR = user, X_PSW = passphrase |
| Certificate | not supported (see Common mistakes) |
The _USR/_PSW split is the part people forget: $REGISTRY alone is the colon-joined
pair, which is exactly what curl -u wants and exactly what docker login does not.
Put environment at the stage level instead of the pipeline level when only one stage
needs the secret. The narrower the block, the fewer steps can see it.
withCredentials for every type
withCredentials does the same job with explicit variable names, and it covers the
certificate type too. One pipeline, one stage per type:
pipeline {
agent any
stages {
stage('Secret text') {
steps {
withCredentials([string(credentialsId: 'sonar-token', variable: 'SONAR_TOKEN')]) {
sh 'echo "token has ${#SONAR_TOKEN} characters"'
}
}
}
stage('Username and password') {
steps {
withCredentials([usernamePassword(credentialsId: 'registry',
usernameVariable: 'REG_USER',
passwordVariable: 'REG_PASS')]) {
sh 'echo "logging in to the registry as $REG_USER"'
}
}
}
stage('SSH private key') {
steps {
withCredentials([sshUserPrivateKey(credentialsId: 'deploy-key',
keyFileVariable: 'SSH_KEY',
usernameVariable: 'SSH_USER')]) {
sh '''
stat -c '%A %n' "$SSH_KEY"
ssh-keygen -l -f "$SSH_KEY"
'''
}
}
}
stage('Secret file') {
steps {
withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
sh 'grep -E "server:|namespace" "$KUBECONFIG"'
}
}
}
stage('Certificate') {
steps {
withCredentials([certificate(credentialsId: 'client-cert',
keystoreVariable: 'KEYSTORE',
passwordVariable: 'KEYSTORE_PASS',
aliasVariable: 'KEY_ALIAS')]) {
sh '''
echo "alias: $KEY_ALIAS"
openssl pkcs12 -in "$KEYSTORE" -passin env:KEYSTORE_PASS -nokeys -clcerts |
openssl x509 -noout -subject -enddate
'''
}
}
}
}
}The interesting parts of the log:
[Pipeline] { (Secret text)
[Pipeline] withCredentials
Masking supported pattern matches of $SONAR_TOKEN
[Pipeline] {
[Pipeline] sh
+ echo token has 44 characters
token has 44 characters
[…]
[Pipeline] { (SSH private key)
[Pipeline] withCredentials
Masking supported pattern matches of $SSH_KEY or $SSH_USER
[Pipeline] {
[Pipeline] sh
+ stat -c %A %n ****
-r-------- ****
+ ssh-keygen -l -f ****
256 SHA256:OUN1TRjYGX9U35bhWYbxAS3cD54b0X8In2M5wUmwqe0 ****@jenkins (ED25519)
[…]
[Pipeline] { (Secret file)
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ grep -E server:|namespace ****
server: https://k8s.staging.example.internal:6443
context: {cluster: staging, user: jenkins-deployer, namespace: payments}
[…]
[Pipeline] { (Certificate)
[Pipeline] withCredentials
Masking supported pattern matches of $KEY_ALIAS or $KEYSTORE_PASS or $KEYSTORE
[Pipeline] {
[Pipeline] sh
+ echo alias: ****
alias: ****
+ openssl pkcs12 -in **** -passin env:KEYSTORE_PASS -nokeys -clcerts
+ openssl x509 -noout -subject -enddate
subject=CN = jenkins-ci, O = Example Payments
notAfter=Nov 10 02:26:35 2027 GMTThings worth noticing:
- Key and secret files are written read-only for the owner (
-r--------) in a temporary directory next to the workspace, and the file path itself is masked. Jenkins removes them when the block ends. - Pass secrets to tools through the environment, not the command line.
openssl -passin env:KEYSTORE_PASSreads the password from the variable, so the password never appears in the process list. The same idea is behinddocker login --password-stdinandcurl --netrc-file. - Masking is a text substitution, and it is blunt. The SSH username is
deploy, so every occurrence ofdeployin the log becomes****, including the key commentdeploy@jenkins. A credential whose username isadminorjenkinswill scramble half your log. For usernames that are not secret, untick Treat username as secret on the credential (the XML field isusernameSecret). I did that for the payments database credential later in this article, and its username then printed in clear.
What masking catches
Masking works on the console output stream: whenever the exact bytes of a watched secret appear, they are replaced. On the versions I tested, it also caught the Base64 encoding of a secret. A step that built an HTTP basic-auth header printed this under the shell trace:
+ printf %s:%s **** ****
+ base64
+ AUTH=****Ol****That Ol in the middle is the Base64 of the colon between username and password; both
halves were recognised and masked. Do not rely on this. Masking does not know about any
other encoding, and it does nothing for anything that is not the console log.
How secrets still leak
This pipeline lives in the payments folder and uses the database credential, whose
password N7#vq/Lx+29=Tz contains characters that must be URL-encoded. It contains three
real mistakes:
pipeline {
agent any
stages {
stage('Migrate') {
steps {
withCredentials([usernamePassword(credentialsId: 'payments-db',
usernameVariable: 'DB_USER',
passwordVariable: 'DB_PASS')]) {
// Leak 1: Groovy string interpolation (double quotes)
sh "echo connecting as ${DB_USER} with ${DB_PASS}"
// Leak 2: a derived value, printed by the shell trace
sh '''
ENC=$(python3 -c 'import os, urllib.parse; print(urllib.parse.quote(os.environ["DB_PASS"], safe=""))')
export DATABASE_URL="postgres://$DB_USER:$ENC@db.payments.internal:5432/payments"
'''
// Leak 3: a file that outlives the build
sh 'echo "DB_PASSWORD=$DB_PASS" > migrate.env'
archiveArtifacts artifacts: 'migrate.env'
}
}
}
}
}[Pipeline] withCredentials
Masking supported pattern matches of $DB_PASS
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [DB_PASS]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ echo connecting as payments_app with ****
connecting as payments_app with ****
[Pipeline] sh
+ python3 -c import os, urllib.parse; print(urllib.parse.quote(os.environ["DB_PASS"], safe=""))
+ ENC=N7%23vq%2FLx%2B29%3DTz
+ export DATABASE_URL=postgres://payments_app:N7%23vq%2FLx%2B29%3DTz@db.payments.internal:5432/payments
[Pipeline] sh
+ echo DB_PASSWORD=****
[Pipeline] archiveArtifacts
Archiving artifacts
[…]
Finished: SUCCESSThe build is green and the log looks masked at a glance. Here is what went wrong.
Leak 1: Groovy string interpolation. In a double-quoted Groovy string, ${DB_PASS}
is replaced by Groovy before the sh step ever runs. The shell receives the password as
literal text inside its script, so it is visible in the agent's process list while the
step runs, and Jenkins has to guess that the string is sensitive. The warning is Jenkins
noticing. The fix is single quotes, so the shell expands $DB_PASS from its own
environment: sh 'echo "connecting as $DB_USER"'.
Leak 2: derived values under set -x. Jenkins runs sh steps with -xe, so every
command is echoed after expansion. The URL-encoded password N7%23vq%2FLx%2B29%3DTz is a
different string from the secret, so masking does not recognise it, and anyone with Read
access to the job can decode it. Any transformation does this: URL encoding, JSON
escaping, hex, a substring. The fix is set +x before the lines that compute derived
values.
Leak 3: files. The echo line itself was masked, but the file was not. Archived
artifacts are downloadable by anyone who can read the job:
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
"${JENKINS_URL}job/payments/job/db-migrate-leaky/lastBuild/artifact/migrate.env"DB_PASSWORD=N7#vq/Lx+29=TzThe same applies to stashes, test reports that echo configuration, and anything left in the workspace, which users with Job/Workspace permission can browse.
The corrected stage:
withCredentials([usernamePassword(credentialsId: 'payments-db',
usernameVariable: 'DB_USER',
passwordVariable: 'DB_PASS')]) {
sh 'echo "connecting as $DB_USER"'
sh '''
set +x
ENC=$(python3 -c 'import os, urllib.parse; print(urllib.parse.quote(os.environ["DB_PASS"], safe=""))')
export DATABASE_URL="postgres://$DB_USER:$ENC@db.payments.internal:5432/payments"
echo "DATABASE_URL is set (${#DATABASE_URL} characters)"
'''
}[Pipeline] withCredentials
Masking supported pattern matches of $DB_PASS
[Pipeline] {
[Pipeline] sh
+ echo connecting as payments_app
connecting as payments_app
[Pipeline] sh
+ set +x
DATABASE_URL is set (81 characters)
[Pipeline] }
[Pipeline] // withCredentialsThe username prints in clear here because this credential has Treat username as secret turned off; the password is still watched.
Least privilege in practice
Masking is a seatbelt. The real control is who can use a credential at all, and the uncomfortable rule is this:
anyone who can change what a job runs can read every credential that job can see. If
a developer can push a branch that a multibranch job builds, they can add
sh 'echo "$DB_PASS" | rev' and read the password backwards from the log. So:
- Scope credentials to folders. Production deploy credentials belong in the folder of the job that deploys to production, not in the global store.
- Separate credentials per environment.
payments-db-stagingandpayments-db-prod, never one shared account. Then a leak in a staging job costs you a staging password. - Use SYSTEM scope for anything only Jenkins needs (LDAP bind accounts, the key the controller uses to launch SSH agents).
- Give pipelines narrow, dedicated accounts. A registry account that can push one repository; a deploy user that can restart one service.
- Keep the administrator group small. As shown above, administrators can decrypt everything through the script console.
- Rotate. If a credential has ever been printed, archived or committed, change it. Updating the credential in Jenkins (same ID) updates every job that uses it.
- Protect production jobs from untrusted changes: build pull requests from forks in a folder that has no production credentials at all.
External secret stores
Jenkins' own store is fine for a few dozen secrets. Larger organisations usually keep secrets in a dedicated system and let Jenkins fetch them. There are plugins for HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager and others. They come in two styles:
- Credentials providers: secrets in the external store appear as ordinary Jenkins
credentials, so
credentials('id')andwithCredentialskeep working unchanged. This is the style I recommend, because pipelines stay portable. - Pipeline steps that fetch a secret by path during the build, for example the Vault
plugin's
withVaultstep. These tie each Jenkinsfile to one secret store.
The benefits are central rotation and audit logs, plus short-lived credentials (Vault's dynamic database users, cloud federated identities), which make a leaked value expire on its own. The costs are another system to run and a new failure mode when the secret store is unreachable. None of these plugins were installed on my test controller, so I have not tested them here; check each plugin's page on plugins.jenkins.io for its current configuration.
Where you'll use this
Credentials show up in almost every real pipeline:
- Publishing artifacts: username/password for Nexus, Artifactory or a container
registry, fed to
docker login --password-stdinor a.netrcfile. - Deploying: an SSH key for
rsync/sshto app servers, or a kubeconfig secret file forkubectland Helm. - Calling services: secret text for SonarQube, Slack or Jira tokens.
- Signing: a certificate credential for code signing or mTLS.
- Database migrations: a folder-scoped username/password per environment, as in the examples above.
Here is a complete signing pipeline of the kind you'd use to publish a release manifest that downstream systems verify. It extracts the private key from the certificate credential into the workspace's temporary directory, signs, verifies, archives the signature and the public key, and deletes the key on exit:
pipeline {
agent any
stages {
stage('Build') {
steps {
sh '''
mkdir -p dist
printf 'payments-api 1.4.2\ncommit 7188978\n' > dist/release-manifest.txt
sha256sum dist/release-manifest.txt
'''
}
}
stage('Sign') {
steps {
withCredentials([certificate(credentialsId: 'client-cert',
keystoreVariable: 'KEYSTORE',
passwordVariable: 'KEYSTORE_PASS')]) {
sh '''
set +x
umask 077
KEY="$WORKSPACE_TMP/signing.key"
trap 'rm -f "$KEY"' EXIT
openssl pkcs12 -in "$KEYSTORE" -passin env:KEYSTORE_PASS -nocerts -nodes -out "$KEY"
openssl pkcs12 -in "$KEYSTORE" -passin env:KEYSTORE_PASS -clcerts -nokeys |
openssl x509 -pubkey -noout > dist/signing-pubkey.pem
openssl dgst -sha256 -sign "$KEY" -out dist/release-manifest.sig dist/release-manifest.txt
openssl dgst -sha256 -verify dist/signing-pubkey.pem \
-signature dist/release-manifest.sig dist/release-manifest.txt
'''
}
archiveArtifacts artifacts: 'dist/*'
}
}
}
}[Pipeline] { (Sign)
[Pipeline] withCredentials
Masking supported pattern matches of $KEYSTORE_PASS or $KEYSTORE
[Pipeline] {
[Pipeline] sh
+ set +x
Verified OK
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] archiveArtifacts
Archiving artifacts
[…]
Finished: SUCCESSThe archived artifacts, listed through the REST API, contain the signature and the public key but not the private key:
curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
"${JENKINS_URL}job/release-sign/lastBuild/api/json?tree=artifacts%5BrelativePath%5D" |
jq -r '.artifacts[].relativePath'dist/release-manifest.sig
dist/release-manifest.txt
dist/signing-pubkey.pemThree habits from this example carry over to any pipeline: WORKSPACE_TMP (a directory
next to the workspace, not inside it) for anything secret you must write to disk,
umask 077 so it is created private, and trap … EXIT so it is removed even when a
command fails.
Common mistakes
"ERROR: sonar-tokn" and nothing else
A typo in the ID inside environment { X = credentials('…') } produces the least helpful
error message in Jenkins:
environment {
SONAR_TOKEN = credentials('sonar-tokn')
}Started from command line by admin
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/env-typo
[Pipeline] {
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: sonar-tokn
Finished: FAILUREThe message is just the ID it could not find. The same ID in withCredentials gives the
clearer "Could not find credentials entry with ID …". Either way, check the spelling, the
folder, and the scope.
Using a certificate with credentials()
The environment helper only knows four types:
environment {
CLIENT_CERT = credentials('client-cert')
}ERROR: No suitable binding handler could be found for type com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl. Supported types are StandardUsernamePasswordCredentials,FileCredentials,DockerServerCredentials,StringCredentials,SSHUserPrivateKey.
Finished: FAILUREUse withCredentials([certificate(...)]) for certificates, as in the signing example.
Double quotes around sh commands
Covered above, but it is the most common credential mistake in real Jenkinsfiles, so here it is in the form you'll meet it, in a registry check:
sh "curl -s -o /dev/null -w '%{http_code}' -u ${REGISTRY_USR}:${REGISTRY_PSW} ${env.JENKINS_URL}whoAmI/api/json"Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
Affected argument(s) used the following variable(s): [REGISTRY_USR, REGISTRY_PSW, REGISTRY]
See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ curl -s -o /dev/null -w %{http_code} -u **** http://localhost:8090/whoAmI/api/json
401The build still passes, which is why this survives code review. Treat the warning as an error. Exercise 2 asks you to fix it.
Summary
- Credentials are stored encrypted in
JENKINS_HOME, with keys insecrets/. Backups that includesecrets/contain every password, and administrators can decrypt everything through the script console. - The five built-in types are secret text, username with password, SSH private key, secret file and certificate. Always set a readable ID yourself.
- GLOBAL scope is for jobs, SYSTEM scope is for Jenkins itself. Folder stores limit a credential to the jobs in that folder. A wrong ID, scope or folder all give "Could not find credentials entry".
X = credentials('id')inenvironmentsetsX, plusX_USRandX_PSWfor username/password and SSH types; files become paths. Certificates needwithCredentials([certificate(...)]).- Masking replaces the exact secret (and, in current versions, its Base64 form) in the console log only. Derived values, files, artifacts and process arguments are not protected.
- Use single-quoted
shstrings,set +xaround derived secrets,WORKSPACE_TMPwithumask 077for files, and never archive anything containing a secret. - Anyone who can change a job's script can read the credentials it uses. Scope by folder and by environment, and keep production credentials away from jobs that build untrusted code.
Exercises
Create a GLOBAL secret text credential with the ID npm-token (or with your own prefix)
using create-credentials-by-xml, then write a pipeline that binds it with the
environment directive and prints its length and its value. Confirm that the value is
masked.
Show Solution
java -jar jenkins-cli.jar create-credentials-by-xml system::system::jenkins _ <<'EOF'
<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
<scope>GLOBAL</scope>
<id>npm-token</id>
<description>npm publish token</description>
<secret>npm_4kQ9zR2mX7vL1pT8wY3nB6cJ0hF5dG</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
EOFpipeline {
agent any
environment {
NPM_TOKEN = credentials('npm-token')
}
stages {
stage('Publish (dry run)') {
steps {
sh 'echo "token is ${#NPM_TOKEN} characters: $NPM_TOKEN"'
}
}
}
}[Pipeline] withCredentials
Masking supported pattern matches of $NPM_TOKEN
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Publish (dry run))
[Pipeline] sh
+ echo token is 34 characters: ****
token is 34 characters: ****Printing the length is a safe way to check that a secret arrived without revealing it.
Rewrite this step so that Jenkins no longer prints the Groovy string interpolation warning, while still calling the same URL with the same credentials:
sh "curl -s -o /dev/null -w '%{http_code}' -u ${REGISTRY_USR}:${REGISTRY_PSW} ${env.JENKINS_URL}whoAmI/api/json"Hint
Who should expand the variables: Groovy, or the shell? JENKINS_URL is also available as
an environment variable inside sh.
Show Solution
sh 'curl -s -o /dev/null -w "%{http_code}" -u "$REGISTRY_USR:$REGISTRY_PSW" "${JENKINS_URL}whoAmI/api/json"'[Pipeline] withCredentials
Masking supported pattern matches of $REGISTRY or $REGISTRY_USR or $REGISTRY_PSW
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Check registry login)
[Pipeline] sh
+ curl -s -o /dev/null -w %{http_code} -u **** http://localhost:8090/whoAmI/api/json
401With single quotes, Groovy passes the text through unchanged and the shell expands the
variables from its environment. The 401 is expected: ci-pusher is not a Jenkins user.
The point is that the credential reached curl without passing through Groovy.
Create a username/password credential with the ID deployer in the global store
(username global-deployer) and another with the same ID in the payments folder
store (username payments-deployer). Set usernameSecret to false on both so the
username prints. Run the same pipeline inside the folder and at the top level. Which
credential does each job get?
Show Solution
for pair in "system::system::jenkins global-deployer" "folder::item::payments payments-deployer"; do
set -- $pair
java -jar jenkins-cli.jar create-credentials-by-xml "$1" _ <<EOF
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
<scope>GLOBAL</scope>
<id>deployer</id>
<usernameSecret>false</usernameSecret>
<username>$2</username>
<password>not-the-point-here</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
donepipeline {
agent any
stages {
stage('Which credential?') {
steps {
withCredentials([usernamePassword(credentialsId: 'deployer',
usernameVariable: 'DEPLOY_USER',
passwordVariable: 'DEPLOY_PASS')]) {
sh 'echo "resolved deployer to user $DEPLOY_USER"'
}
}
}
}
}In payments/which-deployer:
+ echo resolved deployer to user payments-deployer
resolved deployer to user payments-deployerIn the top-level which-deployer:
+ echo resolved deployer to user global-deployer
resolved deployer to user global-deployerThe store closest to the job wins. That makes it possible to override a global default per team, and also makes it easy to confuse yourself, so reuse IDs across stores only on purpose.
Some tools only read credentials from a file. Write a pipeline stage that writes a
.netrc-format file from the registry credential, uses it with
curl --netrc-file to call ${JENKINS_URL}whoAmI/api/json, and guarantees the file is
gone when the build finishes, even if curl fails. Nothing secret may appear in the log,
and the file must be created with mode 600.
Hint
Use WORKSPACE_TMP, umask 077, set +x, and a post { always { … } } block.
Show Solution
pipeline {
agent any
stages {
stage('Call API with a netrc file') {
steps {
withCredentials([usernamePassword(credentialsId: 'registry',
usernameVariable: 'REG_USER',
passwordVariable: 'REG_PASS')]) {
sh '''
set +x
umask 077
NETRC="$WORKSPACE_TMP/netrc"
printf 'machine localhost login %s password %s\n' "$REG_USER" "$REG_PASS" > "$NETRC"
ls -l "$NETRC" | cut -d' ' -f1
curl -s -o /dev/null -w "HTTP %{http_code}\n" --netrc-file "$NETRC" "${JENKINS_URL}whoAmI/api/json"
'''
}
}
}
}
post {
always {
sh 'rm -f "$WORKSPACE_TMP/netrc"; ls -A "$WORKSPACE_TMP"'
}
}
}[Pipeline] sh
+ set +x
-rw-------
HTTP 401
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] sh
+ rm -f /home/sid/.jenkins/workspace/ex4@tmp/netrc
+ ls -A /home/sid/.jenkins/workspace/ex4@tmp
durable-fa56c283The file was private (-rw-------), set +x kept the printf line out of the log, and
after the post block only Jenkins' own durable-… control directory remains in
@tmp. Writing to WORKSPACE_TMP instead of the workspace also means a later
archiveArtifacts '**' cannot pick the file up by accident.
Tested with: Jenkins 2.580.2 on OpenJDK 21 (Ubuntu 24.04 under WSL2), Credentials
plugin 1514.vb_01eb_014e7d4, Credentials Binding 728.v902a_273b_8947, November 2026. Every
credential was created with the CLI and every pipeline was run on that controller; outputs
are pasted from the console logs, trimmed where marked. UI navigation (menu names, the
"Select a type of credential" dialog, "Treat username as secret") was checked against the
pages the same version serves, fetched over HTTP, not clicked through. Two things surprised
me: masking also caught Base64-encoded secrets, and the deploy username masked the word
deploy everywhere in the log. The external secret-store plugins were not installed and
were not tested. Paths from the test machine's scratch directories are shown shortened to /home/sid/…. Job, node and credential names are shown without the test run's prefixes.
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Jenkins tutorial here is tested end to end before publishing.