Credentials and Secrets

How Jenkins stores, scopes and injects secrets: the five credential types, credentials() and withCredentials, log masking, and the ways secrets still leak.

By The Dark DebuggerUpdated 27 min read

Every useful pipeline eventually needs a secret: a registry password, an SSH key for the deploy user, a kubeconfig, an API token. This article covers how Jenkins stores those secrets, the five credential types, how scopes and folders decide who can use them, how to inject them into a pipeline, and the specific ways they still end up in logs and artifacts. It assumes you can write and run a declarative pipeline, as covered in Your First Pipeline and Declarative Pipeline in Depth.

Why secrets need their own system

Picture a payments team whose deploy job needs four things: a password to push images to the registry, an SSH key to restart the service on the app servers, a kubeconfig for the staging cluster, and a token for the code-quality server. The quickest way to get the job green is to paste those values into the Jenkinsfile or into job parameters. Now everyone who can read the repository, or the job's config.xml, or the build log, has production access.

Jenkins credentials solve three separate problems:

  1. Storage. Secrets live in the controller, encrypted at rest, not in Git and not in job configuration.
  2. Injection. A pipeline refers to a secret by ID. The value is handed to the build only for the duration of a block, as an environment variable or a temporary file.
  3. Masking and scope. The value is replaced by **** in the console log, and the credential is only visible to the jobs that should use it.

None of these is perfect, and the last two sections of this article show exactly where they stop protecting you.

How Jenkins stores a credential

Credentials in the global store are saved in $JENKINS_HOME/credentials.xml. Secret fields are encrypted with keys that live in $JENKINS_HOME/secrets/. On my test controller, the password of a username/password credential looks like this on disk:

grep -n -A4 '<id>registry</id>' "$JENKINS_HOME/credentials.xml"
17:          <id>registry</id>
18-          <description>Container registry push account</description>
19-          <username>ci-pusher</username>
20-          <password>{AQAAABAAAAAgvclqzJqHpr7i6PZd+NgoSLVrLcXvF7woHdVuskTx8JTdrLxDKYtGeiOoj/olWnHU}</password>
21-        </com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>

The value in braces is ciphertext. The key that decrypts it is secrets/hudson.util.Secret, itself protected by secrets/master.key. Two consequences follow, and both matter more than the encryption itself:

  • A backup of JENKINS_HOME that includes secrets/ is a backup of every password. Treat it like one. A backup without secrets/ restores credentials you can no longer decrypt.
  • Anyone with Overall/Administer can read every secret. The script console runs arbitrary Groovy inside the controller. This is what an administrator can do in two lines:
def xml = new File(System.getenv('JENKINS_HOME'), 'credentials.xml').text
def encrypted = (xml =~ /<id>registry<\/id>[\s\S]*?<password>(\{[^}]+\})<\/password>/)[0][1]
println "stored:    ${encrypted}"
println "decrypted: ${hudson.util.Secret.decrypt(encrypted).plainText}"
java -jar jenkins-cli.jar groovy = < decrypt.groovy
stored:    {AQAAABAAAAAgvclqzJqHpr7i6PZd+NgoSLVrLcXvF7woHdVuskTx8JTdrLxDKYtGeiOoj/olWnHU}
decrypted: Wq8-tVz3-Kp2r-Ld9x

Encryption at rest protects against someone who finds a copy of credentials.xml alone. It does not protect against administrators, and it is not meant to. Keep the admin group small.

The five credential types

These are the types provided by the Credentials, Plain Credentials and SSH Credentials plugins, which a controller set up with the suggested plugins already has:

Type in the UIHoldsBinding in withCredentialsTypical use
Secret textone stringstringAPI tokens, webhook URLs, Sonar tokens
Username with passwordusername + passwordusernamePasswordregistries, databases, HTTP basic auth
SSH Username with private keyusername + key (+ passphrase)sshUserPrivateKeyGit over SSH, deploying to servers
Secret filea file's bytes + file namefilekubeconfig, service-account JSON, .npmrc
CertificatePKCS#12 keystore + passwordcertificatemTLS client certificates, code signing

Other plugins add more types. On my controller, the add dialog also listed "GitHub App" (from the GitHub Branch Source plugin) and "X.509 Client Certificate" (from Docker Commons).

Creating credentials

In the UI

In Jenkins 2.580: Manage Jenkins → Credentials, click the System store, then the Global domain ("Credentials that should be available everywhere"), then Add Credentials. A dialog titled "Select a type of credential" lists the types from the table above; pick one, click Next, and fill in the form. For a username/password credential the form has Scope, Username, a Treat username as secret checkbox, Password, ID and Description.

Always fill in the ID yourself. If you leave it blank Jenkins generates a UUID, and your Jenkinsfiles end up full of credentials('3f6c1e0a-…') that nobody can read.

From the command line

Clicking through forms does not scale past a handful of secrets, and it leaves no record. The Jenkins CLI can create credentials from XML. Download the CLI from your controller and point it at your server with environment variables, as set up in Installing Jenkins (use an API token from your user page, not your password):

export JENKINS_URL=http://localhost:8090/
export JENKINS_USER_ID=admin
export JENKINS_API_TOKEN=...   # your API token
curl -sO "${JENKINS_URL}jnlpJars/jenkins-cli.jar"
java -jar jenkins-cli.jar who-am-i
Authenticated as: admin
Authorities:
  authenticated
  FACTOR_PASSWORD

This script creates one credential of each type in the global store. It reads the key, the kubeconfig and the PKCS#12 file from the current directory, so no secret material is typed into the script except the two demo passwords:

#!/usr/bin/env bash
# Creates the five demo credentials in the global store (Manage Jenkins > Credentials > System > Global credentials).
set -euo pipefail
STORE=system::system::jenkins   # provider::resolver::context
DOMAIN=_                        # "_" is the global domain
 
jcli() { java -jar jenkins-cli.jar "$@"; }
 
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>sonar-token</id>
  <description>SonarQube analysis token</description>
  <secret>squ_7d1e4f8a3c6b0d2e5f9c2e7a1b8d3f60e5a2c9b7</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
EOF
 
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>registry</id>
  <description>Container registry push account</description>
  <username>ci-pusher</username>
  <password>Wq8-tVz3-Kp2r-Ld9x</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
 
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
  <scope>GLOBAL</scope>
  <id>deploy-key</id>
  <description>SSH key for the deploy user on the app servers</description>
  <username>deploy</username>
  <privateKeySource class="com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey\$DirectEntryPrivateKeySource">
    <privateKey>$(cat deploy_key)</privateKey>
  </privateKeySource>
</com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>
EOF
 
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<org.jenkinsci.plugins.plaincredentials.impl.FileCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>kubeconfig</id>
  <description>kubeconfig for the staging cluster</description>
  <fileName>kubeconfig</fileName>
  <secretBytes>$(base64 -w0 kubeconfig)</secretBytes>
</org.jenkinsci.plugins.plaincredentials.impl.FileCredentialsImpl>
EOF
 
jcli create-credentials-by-xml "$STORE" "$DOMAIN" <<EOF
<com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>client-cert</id>
  <description>mTLS client certificate for the artifact API</description>
  <keyStoreSource class="com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl\$UploadedKeyStoreSource">
    <uploadedKeystoreBytes>$(base64 -w0 client.p12)</uploadedKeystoreBytes>
  </keyStoreSource>
  <password>changeit</password>
</com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>
EOF

The SSH key came from ssh-keygen -t ed25519 -N '' -C 'deploy@jenkins' -f deploy_key, and client.p12 from openssl pkcs12 -export with the password changeit. Note the escaped \$ in the two class= attributes: inside an unquoted heredoc, an unescaped $DirectEntryPrivateKeySource would be expanded by Bash to an empty string.

./create-credentials.sh
java -jar jenkins-cli.jar list-credentials system::system::jenkins
=================================================================================================
Domain           (global)
Description      
# of Credentials 5
=================================================================================================
Id               Name
================ ================================================================================
sonar-token SonarQube analysis token
registry    Container registry push account
deploy-key  SSH key for the deploy user on the app servers
kubeconfig  kubeconfig (kubeconfig for the staging cluster)
client-cert O=Example Payments, CN=jenkins-ci (mTLS client certificate for the artifact API)
=================================================================================================

Reading a credential back never returns the secret:

java -jar jenkins-cli.jar get-credentials-as-xml system::system::jenkins _ registry
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl plugin="credentials@1514.vb_01eb_014e7d4">
  <scope>GLOBAL</scope>
  <id>registry</id>
  <description>Container registry push account</description>
  <username>ci-pusher</username>
  <password>
    <secret-redacted/>
  </password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>

A script like this is fine for a one-off migration. For a controller you rebuild regularly, define credentials in Configuration as Code instead, with the values coming from the environment; that is covered in Jenkins Configuration as Code.

Scopes: Global, System, and folders

Every credential has a scope:

  • GLOBAL: usable by Jenkins itself, by agents, and by jobs and pipelines.
  • SYSTEM: usable only by Jenkins itself, for example the bind account of an LDAP security realm or the key the controller uses to launch SSH agents. Pipelines cannot see it.

Separately, every credential lives in a store. The global store belongs to the controller. Every folder (from the Folders plugin) has its own store too, and a credential in a folder's store is only visible to jobs inside that folder. That is the main tool for least privilege: give each team a folder, put the team's secrets in it.

Let's prove both rules. This script creates a payments folder, a database credential in that folder's store, and a SYSTEM-scoped LDAP credential in the global store:

#!/usr/bin/env bash
# Folder-scoped and SYSTEM-scoped credentials for the scope demo.
set -euo pipefail
jcli() { java -jar jenkins-cli.jar "$@"; }
 
# A folder for the payments team
jcli create-job payments <<'EOF'
<com.cloudbees.hudson.plugins.folder.Folder plugin="cloudbees-folder">
  <description>Payments team jobs</description>
</com.cloudbees.hudson.plugins.folder.Folder>
EOF
 
# A credential that lives in that folder's own store
jcli create-credentials-by-xml folder::item::payments _ <<'EOF'
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>payments-db</id>
  <description>payments database, app user</description>
  <username>payments_app</username>
  <password>N7#vq/Lx+29=Tz</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
 
# A SYSTEM-scoped credential: for Jenkins itself, not for jobs
jcli create-credentials-by-xml system::system::jenkins _ <<'EOF'
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
  <scope>SYSTEM</scope>
  <id>ldap-bind</id>
  <description>LDAP bind account used by the security realm</description>
  <username>cn=jenkins,ou=services,dc=example,dc=com</username>
  <password>Lb7-q2Rz-p9Xw</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
 
jcli list-credentials folder::item::payments
============================================
Domain           (global)
Description      
# of Credentials 1
============================================
Id               Name
================ ===========================
payments-db payments database, app user
============================================

The same pipeline goes into two jobs: payments/db-check inside the folder, and db-check-outside at the top level.

pipeline {
    agent any
    stages {
        stage('Check DB credential') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'payments-db',
                                                  usernameVariable: 'DB_USER',
                                                  passwordVariable: 'DB_PASS')]) {
                    sh 'echo "DB user is $DB_USER, password has ${#DB_PASS} characters"'
                }
            }
        }
    }
}

Inside the folder:

[Pipeline] withCredentials
Masking supported pattern matches of $DB_USER or $DB_PASS
[Pipeline] {
[Pipeline] sh
+ echo DB user is ****, password has 14 characters
DB user is ****, password has 14 characters
[…]
Finished: SUCCESS

Outside the folder, with an identical Jenkinsfile:

[Pipeline] withCredentials
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Could not find credentials entry with ID 'payments-db'
Finished: FAILURE

And a job that tries to use the SYSTEM-scoped LDAP account fails the same way, even though the credential is in the global store:

ERROR: Could not find credentials entry with ID 'ldap-bind'
Finished: FAILURE

Notice that the error is identical for "does not exist", "wrong folder" and "wrong scope". When you see it, check all three before you assume a typo.

There is also a third kind of store: each user has a personal one (your user name → Credentials). Those are meant for things a user triggers with their own identity and are rarely what you want for shared pipelines.

Using credentials in the environment directive

The shortest way to use a credential in declarative pipeline is the credentials() helper inside environment. What it sets depends on the type:

pipeline {
    agent any
    environment {
        SONAR_TOKEN = credentials('sonar-token')   // secret text
        REGISTRY    = credentials('registry')      // username with password
        KUBECONFIG  = credentials('kubeconfig')    // secret file
        DEPLOY_KEY  = credentials('deploy-key')    // SSH username with private key
    }
    stages {
        stage('What the build sees') {
            steps {
                sh '''
                    echo "SONAR_TOKEN  = $SONAR_TOKEN"
                    echo "REGISTRY     = $REGISTRY"
                    echo "REGISTRY_USR = $REGISTRY_USR"
                    echo "REGISTRY_PSW = $REGISTRY_PSW"
                    echo "KUBECONFIG   = $KUBECONFIG"
                    echo "DEPLOY_KEY   = $DEPLOY_KEY"
                    echo "DEPLOY_KEY_USR = $DEPLOY_KEY_USR"
                '''
            }
        }
    }
}
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG or $SONAR_TOKEN or $DEPLOY_KEY or $DEPLOY_KEY_PSW or $DEPLOY_KEY_USR or $REGISTRY or $REGISTRY_USR or $REGISTRY_PSW
[Pipeline] {
[Pipeline] stage
[Pipeline] { (What the build sees)
[Pipeline] sh
+ echo SONAR_TOKEN  = ****
SONAR_TOKEN  = ****
+ echo REGISTRY     = ****
REGISTRY     = ****
+ echo REGISTRY_USR = ****
REGISTRY_USR = ****
+ echo REGISTRY_PSW = ****
REGISTRY_PSW = ****
+ echo KUBECONFIG   = ****
KUBECONFIG   = ****
+ echo DEPLOY_KEY   = ****
DEPLOY_KEY   = ****
+ echo DEPLOY_KEY_USR = ****
DEPLOY_KEY_USR = ****

The first line tells you which variables are being watched. The rules:

Credential typeX = credentials('id') sets
Secret textX = the secret
Username with passwordX = user:password, X_USR = user, X_PSW = password
Secret fileX = path to a temporary copy of the file
SSH Username with private keyX = path to a temporary key file, X_USR = user, X_PSW = passphrase
Certificatenot supported (see Common mistakes)

The _USR/_PSW split is the part people forget: $REGISTRY alone is the colon-joined pair, which is exactly what curl -u wants and exactly what docker login does not.

Put environment at the stage level instead of the pipeline level when only one stage needs the secret. The narrower the block, the fewer steps can see it.

withCredentials for every type

withCredentials does the same job with explicit variable names, and it covers the certificate type too. One pipeline, one stage per type:

pipeline {
    agent any
    stages {
        stage('Secret text') {
            steps {
                withCredentials([string(credentialsId: 'sonar-token', variable: 'SONAR_TOKEN')]) {
                    sh 'echo "token has ${#SONAR_TOKEN} characters"'
                }
            }
        }
        stage('Username and password') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'registry',
                                                  usernameVariable: 'REG_USER',
                                                  passwordVariable: 'REG_PASS')]) {
                    sh 'echo "logging in to the registry as $REG_USER"'
                }
            }
        }
        stage('SSH private key') {
            steps {
                withCredentials([sshUserPrivateKey(credentialsId: 'deploy-key',
                                                   keyFileVariable: 'SSH_KEY',
                                                   usernameVariable: 'SSH_USER')]) {
                    sh '''
                        stat -c '%A %n' "$SSH_KEY"
                        ssh-keygen -l -f "$SSH_KEY"
                    '''
                }
            }
        }
        stage('Secret file') {
            steps {
                withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
                    sh 'grep -E "server:|namespace" "$KUBECONFIG"'
                }
            }
        }
        stage('Certificate') {
            steps {
                withCredentials([certificate(credentialsId: 'client-cert',
                                             keystoreVariable: 'KEYSTORE',
                                             passwordVariable: 'KEYSTORE_PASS',
                                             aliasVariable: 'KEY_ALIAS')]) {
                    sh '''
                        echo "alias: $KEY_ALIAS"
                        openssl pkcs12 -in "$KEYSTORE" -passin env:KEYSTORE_PASS -nokeys -clcerts |
                            openssl x509 -noout -subject -enddate
                    '''
                }
            }
        }
    }
}

The interesting parts of the log:

[Pipeline] { (Secret text)
[Pipeline] withCredentials
Masking supported pattern matches of $SONAR_TOKEN
[Pipeline] {
[Pipeline] sh
+ echo token has 44 characters
token has 44 characters
[…]
[Pipeline] { (SSH private key)
[Pipeline] withCredentials
Masking supported pattern matches of $SSH_KEY or $SSH_USER
[Pipeline] {
[Pipeline] sh
+ stat -c %A %n ****
-r-------- ****
+ ssh-keygen -l -f ****
256 SHA256:OUN1TRjYGX9U35bhWYbxAS3cD54b0X8In2M5wUmwqe0 ****@jenkins (ED25519)
[…]
[Pipeline] { (Secret file)
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ grep -E server:|namespace ****
    server: https://k8s.staging.example.internal:6443
  context: {cluster: staging, user: jenkins-deployer, namespace: payments}
[…]
[Pipeline] { (Certificate)
[Pipeline] withCredentials
Masking supported pattern matches of $KEY_ALIAS or $KEYSTORE_PASS or $KEYSTORE
[Pipeline] {
[Pipeline] sh
+ echo alias: ****
alias: ****
+ openssl pkcs12 -in **** -passin env:KEYSTORE_PASS -nokeys -clcerts
+ openssl x509 -noout -subject -enddate
subject=CN = jenkins-ci, O = Example Payments
notAfter=Nov 10 02:26:35 2027 GMT

Things worth noticing:

  • Key and secret files are written read-only for the owner (-r--------) in a temporary directory next to the workspace, and the file path itself is masked. Jenkins removes them when the block ends.
  • Pass secrets to tools through the environment, not the command line. openssl -passin env:KEYSTORE_PASS reads the password from the variable, so the password never appears in the process list. The same idea is behind docker login --password-stdin and curl --netrc-file.
  • Masking is a text substitution, and it is blunt. The SSH username is deploy, so every occurrence of deploy in the log becomes ****, including the key comment deploy@jenkins. A credential whose username is admin or jenkins will scramble half your log. For usernames that are not secret, untick Treat username as secret on the credential (the XML field is usernameSecret). I did that for the payments database credential later in this article, and its username then printed in clear.

What masking catches

Masking works on the console output stream: whenever the exact bytes of a watched secret appear, they are replaced. On the versions I tested, it also caught the Base64 encoding of a secret. A step that built an HTTP basic-auth header printed this under the shell trace:

+ printf %s:%s **** ****
+ base64
+ AUTH=****Ol****

That Ol in the middle is the Base64 of the colon between username and password; both halves were recognised and masked. Do not rely on this. Masking does not know about any other encoding, and it does nothing for anything that is not the console log.

How secrets still leak

This pipeline lives in the payments folder and uses the database credential, whose password N7#vq/Lx+29=Tz contains characters that must be URL-encoded. It contains three real mistakes:

pipeline {
    agent any
    stages {
        stage('Migrate') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'payments-db',
                                                  usernameVariable: 'DB_USER',
                                                  passwordVariable: 'DB_PASS')]) {
                    // Leak 1: Groovy string interpolation (double quotes)
                    sh "echo connecting as ${DB_USER} with ${DB_PASS}"
 
                    // Leak 2: a derived value, printed by the shell trace
                    sh '''
                        ENC=$(python3 -c 'import os, urllib.parse; print(urllib.parse.quote(os.environ["DB_PASS"], safe=""))')
                        export DATABASE_URL="postgres://$DB_USER:$ENC@db.payments.internal:5432/payments"
                    '''
 
                    // Leak 3: a file that outlives the build
                    sh 'echo "DB_PASSWORD=$DB_PASS" > migrate.env'
                    archiveArtifacts artifacts: 'migrate.env'
                }
            }
        }
    }
}
[Pipeline] withCredentials
Masking supported pattern matches of $DB_PASS
[Pipeline] {
[Pipeline] sh
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
		 Affected argument(s) used the following variable(s): [DB_PASS]
		 See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ echo connecting as payments_app with ****
connecting as payments_app with ****
[Pipeline] sh
+ python3 -c import os, urllib.parse; print(urllib.parse.quote(os.environ["DB_PASS"], safe=""))
+ ENC=N7%23vq%2FLx%2B29%3DTz
+ export DATABASE_URL=postgres://payments_app:N7%23vq%2FLx%2B29%3DTz@db.payments.internal:5432/payments
[Pipeline] sh
+ echo DB_PASSWORD=****
[Pipeline] archiveArtifacts
Archiving artifacts
[…]
Finished: SUCCESS

The build is green and the log looks masked at a glance. Here is what went wrong.

Leak 1: Groovy string interpolation. In a double-quoted Groovy string, ${DB_PASS} is replaced by Groovy before the sh step ever runs. The shell receives the password as literal text inside its script, so it is visible in the agent's process list while the step runs, and Jenkins has to guess that the string is sensitive. The warning is Jenkins noticing. The fix is single quotes, so the shell expands $DB_PASS from its own environment: sh 'echo "connecting as $DB_USER"'.

Leak 2: derived values under set -x. Jenkins runs sh steps with -xe, so every command is echoed after expansion. The URL-encoded password N7%23vq%2FLx%2B29%3DTz is a different string from the secret, so masking does not recognise it, and anyone with Read access to the job can decode it. Any transformation does this: URL encoding, JSON escaping, hex, a substring. The fix is set +x before the lines that compute derived values.

Leak 3: files. The echo line itself was masked, but the file was not. Archived artifacts are downloadable by anyone who can read the job:

curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
    "${JENKINS_URL}job/payments/job/db-migrate-leaky/lastBuild/artifact/migrate.env"
DB_PASSWORD=N7#vq/Lx+29=Tz

The same applies to stashes, test reports that echo configuration, and anything left in the workspace, which users with Job/Workspace permission can browse.

The corrected stage:

withCredentials([usernamePassword(credentialsId: 'payments-db',
                                  usernameVariable: 'DB_USER',
                                  passwordVariable: 'DB_PASS')]) {
    sh 'echo "connecting as $DB_USER"'
    sh '''
        set +x
        ENC=$(python3 -c 'import os, urllib.parse; print(urllib.parse.quote(os.environ["DB_PASS"], safe=""))')
        export DATABASE_URL="postgres://$DB_USER:$ENC@db.payments.internal:5432/payments"
        echo "DATABASE_URL is set (${#DATABASE_URL} characters)"
    '''
}
[Pipeline] withCredentials
Masking supported pattern matches of $DB_PASS
[Pipeline] {
[Pipeline] sh
+ echo connecting as payments_app
connecting as payments_app
[Pipeline] sh
+ set +x
DATABASE_URL is set (81 characters)
[Pipeline] }
[Pipeline] // withCredentials

The username prints in clear here because this credential has Treat username as secret turned off; the password is still watched.

Least privilege in practice

Masking is a seatbelt. The real control is who can use a credential at all, and the uncomfortable rule is this: anyone who can change what a job runs can read every credential that job can see. If a developer can push a branch that a multibranch job builds, they can add sh 'echo "$DB_PASS" | rev' and read the password backwards from the log. So:

  • Scope credentials to folders. Production deploy credentials belong in the folder of the job that deploys to production, not in the global store.
  • Separate credentials per environment. payments-db-staging and payments-db-prod, never one shared account. Then a leak in a staging job costs you a staging password.
  • Use SYSTEM scope for anything only Jenkins needs (LDAP bind accounts, the key the controller uses to launch SSH agents).
  • Give pipelines narrow, dedicated accounts. A registry account that can push one repository; a deploy user that can restart one service.
  • Keep the administrator group small. As shown above, administrators can decrypt everything through the script console.
  • Rotate. If a credential has ever been printed, archived or committed, change it. Updating the credential in Jenkins (same ID) updates every job that uses it.
  • Protect production jobs from untrusted changes: build pull requests from forks in a folder that has no production credentials at all.

External secret stores

Jenkins' own store is fine for a few dozen secrets. Larger organisations usually keep secrets in a dedicated system and let Jenkins fetch them. There are plugins for HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager and others. They come in two styles:

  • Credentials providers: secrets in the external store appear as ordinary Jenkins credentials, so credentials('id') and withCredentials keep working unchanged. This is the style I recommend, because pipelines stay portable.
  • Pipeline steps that fetch a secret by path during the build, for example the Vault plugin's withVault step. These tie each Jenkinsfile to one secret store.

The benefits are central rotation and audit logs, plus short-lived credentials (Vault's dynamic database users, cloud federated identities), which make a leaked value expire on its own. The costs are another system to run and a new failure mode when the secret store is unreachable. None of these plugins were installed on my test controller, so I have not tested them here; check each plugin's page on plugins.jenkins.io for its current configuration.

Where you'll use this

Credentials show up in almost every real pipeline:

  • Publishing artifacts: username/password for Nexus, Artifactory or a container registry, fed to docker login --password-stdin or a .netrc file.
  • Deploying: an SSH key for rsync/ssh to app servers, or a kubeconfig secret file for kubectl and Helm.
  • Calling services: secret text for SonarQube, Slack or Jira tokens.
  • Signing: a certificate credential for code signing or mTLS.
  • Database migrations: a folder-scoped username/password per environment, as in the examples above.

Here is a complete signing pipeline of the kind you'd use to publish a release manifest that downstream systems verify. It extracts the private key from the certificate credential into the workspace's temporary directory, signs, verifies, archives the signature and the public key, and deletes the key on exit:

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh '''
                    mkdir -p dist
                    printf 'payments-api 1.4.2\ncommit 7188978\n' > dist/release-manifest.txt
                    sha256sum dist/release-manifest.txt
                '''
            }
        }
        stage('Sign') {
            steps {
                withCredentials([certificate(credentialsId: 'client-cert',
                                             keystoreVariable: 'KEYSTORE',
                                             passwordVariable: 'KEYSTORE_PASS')]) {
                    sh '''
                        set +x
                        umask 077
                        KEY="$WORKSPACE_TMP/signing.key"
                        trap 'rm -f "$KEY"' EXIT
                        openssl pkcs12 -in "$KEYSTORE" -passin env:KEYSTORE_PASS -nocerts -nodes -out "$KEY"
                        openssl pkcs12 -in "$KEYSTORE" -passin env:KEYSTORE_PASS -clcerts -nokeys |
                            openssl x509 -pubkey -noout > dist/signing-pubkey.pem
                        openssl dgst -sha256 -sign "$KEY" -out dist/release-manifest.sig dist/release-manifest.txt
                        openssl dgst -sha256 -verify dist/signing-pubkey.pem \
                            -signature dist/release-manifest.sig dist/release-manifest.txt
                    '''
                }
                archiveArtifacts artifacts: 'dist/*'
            }
        }
    }
}
[Pipeline] { (Sign)
[Pipeline] withCredentials
Masking supported pattern matches of $KEYSTORE_PASS or $KEYSTORE
[Pipeline] {
[Pipeline] sh
+ set +x
Verified OK
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] archiveArtifacts
Archiving artifacts
[…]
Finished: SUCCESS

The archived artifacts, listed through the REST API, contain the signature and the public key but not the private key:

curl -s -u "$JENKINS_USER_ID:$JENKINS_API_TOKEN" \
    "${JENKINS_URL}job/release-sign/lastBuild/api/json?tree=artifacts%5BrelativePath%5D" |
    jq -r '.artifacts[].relativePath'
dist/release-manifest.sig
dist/release-manifest.txt
dist/signing-pubkey.pem

Three habits from this example carry over to any pipeline: WORKSPACE_TMP (a directory next to the workspace, not inside it) for anything secret you must write to disk, umask 077 so it is created private, and trap … EXIT so it is removed even when a command fails.

Common mistakes

"ERROR: sonar-tokn" and nothing else

A typo in the ID inside environment { X = credentials('…') } produces the least helpful error message in Jenkins:

environment {
    SONAR_TOKEN = credentials('sonar-tokn')
}
Started from command line by admin
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /home/sid/.jenkins/workspace/env-typo
[Pipeline] {
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: sonar-tokn
Finished: FAILURE

The message is just the ID it could not find. The same ID in withCredentials gives the clearer "Could not find credentials entry with ID …". Either way, check the spelling, the folder, and the scope.

Using a certificate with credentials()

The environment helper only knows four types:

environment {
    CLIENT_CERT = credentials('client-cert')
}
ERROR: No suitable binding handler could be found for type com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl. Supported types are StandardUsernamePasswordCredentials,FileCredentials,DockerServerCredentials,StringCredentials,SSHUserPrivateKey.
Finished: FAILURE

Use withCredentials([certificate(...)]) for certificates, as in the signing example.

Double quotes around sh commands

Covered above, but it is the most common credential mistake in real Jenkinsfiles, so here it is in the form you'll meet it, in a registry check:

sh "curl -s -o /dev/null -w '%{http_code}' -u ${REGISTRY_USR}:${REGISTRY_PSW} ${env.JENKINS_URL}whoAmI/api/json"
Warning: A secret was passed to "sh" using Groovy String interpolation, which is insecure.
		 Affected argument(s) used the following variable(s): [REGISTRY_USR, REGISTRY_PSW, REGISTRY]
		 See https://jenkins.io/redirect/groovy-string-interpolation for details.
+ curl -s -o /dev/null -w %{http_code} -u **** http://localhost:8090/whoAmI/api/json
401

The build still passes, which is why this survives code review. Treat the warning as an error. Exercise 2 asks you to fix it.

Summary

  • Credentials are stored encrypted in JENKINS_HOME, with keys in secrets/. Backups that include secrets/ contain every password, and administrators can decrypt everything through the script console.
  • The five built-in types are secret text, username with password, SSH private key, secret file and certificate. Always set a readable ID yourself.
  • GLOBAL scope is for jobs, SYSTEM scope is for Jenkins itself. Folder stores limit a credential to the jobs in that folder. A wrong ID, scope or folder all give "Could not find credentials entry".
  • X = credentials('id') in environment sets X, plus X_USR and X_PSW for username/password and SSH types; files become paths. Certificates need withCredentials([certificate(...)]).
  • Masking replaces the exact secret (and, in current versions, its Base64 form) in the console log only. Derived values, files, artifacts and process arguments are not protected.
  • Use single-quoted sh strings, set +x around derived secrets, WORKSPACE_TMP with umask 077 for files, and never archive anything containing a secret.
  • Anyone who can change a job's script can read the credentials it uses. Scope by folder and by environment, and keep production credentials away from jobs that build untrusted code.

Exercises

Exercise 1: Create and use a secret text credentialeasy

Create a GLOBAL secret text credential with the ID npm-token (or with your own prefix) using create-credentials-by-xml, then write a pipeline that binds it with the environment directive and prints its length and its value. Confirm that the value is masked.

Show Solution
java -jar jenkins-cli.jar create-credentials-by-xml system::system::jenkins _ <<'EOF'
<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>npm-token</id>
  <description>npm publish token</description>
  <secret>npm_4kQ9zR2mX7vL1pT8wY3nB6cJ0hF5dG</secret>
</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>
EOF
pipeline {
    agent any
    environment {
        NPM_TOKEN = credentials('npm-token')
    }
    stages {
        stage('Publish (dry run)') {
            steps {
                sh 'echo "token is ${#NPM_TOKEN} characters: $NPM_TOKEN"'
            }
        }
    }
}
[Pipeline] withCredentials
Masking supported pattern matches of $NPM_TOKEN
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Publish (dry run))
[Pipeline] sh
+ echo token is 34 characters: ****
token is 34 characters: ****

Printing the length is a safe way to check that a secret arrived without revealing it.

Exercise 2: Remove the interpolation warningmedium

Rewrite this step so that Jenkins no longer prints the Groovy string interpolation warning, while still calling the same URL with the same credentials:

sh "curl -s -o /dev/null -w '%{http_code}' -u ${REGISTRY_USR}:${REGISTRY_PSW} ${env.JENKINS_URL}whoAmI/api/json"
Hint

Who should expand the variables: Groovy, or the shell? JENKINS_URL is also available as an environment variable inside sh.

Show Solution
sh 'curl -s -o /dev/null -w "%{http_code}" -u "$REGISTRY_USR:$REGISTRY_PSW" "${JENKINS_URL}whoAmI/api/json"'
[Pipeline] withCredentials
Masking supported pattern matches of $REGISTRY or $REGISTRY_USR or $REGISTRY_PSW
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Check registry login)
[Pipeline] sh
+ curl -s -o /dev/null -w %{http_code} -u **** http://localhost:8090/whoAmI/api/json
401

With single quotes, Groovy passes the text through unchanged and the shell expands the variables from its environment. The 401 is expected: ci-pusher is not a Jenkins user. The point is that the credential reached curl without passing through Groovy.

Exercise 3: Same ID, two storesmedium

Create a username/password credential with the ID deployer in the global store (username global-deployer) and another with the same ID in the payments folder store (username payments-deployer). Set usernameSecret to false on both so the username prints. Run the same pipeline inside the folder and at the top level. Which credential does each job get?

Show Solution
for pair in "system::system::jenkins global-deployer" "folder::item::payments payments-deployer"; do
  set -- $pair
  java -jar jenkins-cli.jar create-credentials-by-xml "$1" _ <<EOF
<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
  <scope>GLOBAL</scope>
  <id>deployer</id>
  <usernameSecret>false</usernameSecret>
  <username>$2</username>
  <password>not-the-point-here</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
EOF
done
pipeline {
    agent any
    stages {
        stage('Which credential?') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'deployer',
                                                  usernameVariable: 'DEPLOY_USER',
                                                  passwordVariable: 'DEPLOY_PASS')]) {
                    sh 'echo "resolved deployer to user $DEPLOY_USER"'
                }
            }
        }
    }
}

In payments/which-deployer:

+ echo resolved deployer to user payments-deployer
resolved deployer to user payments-deployer

In the top-level which-deployer:

+ echo resolved deployer to user global-deployer
resolved deployer to user global-deployer

The store closest to the job wins. That makes it possible to override a global default per team, and also makes it easy to confuse yourself, so reuse IDs across stores only on purpose.

Exercise 4: A netrc file that cleans up after itselfhard

Some tools only read credentials from a file. Write a pipeline stage that writes a .netrc-format file from the registry credential, uses it with curl --netrc-file to call ${JENKINS_URL}whoAmI/api/json, and guarantees the file is gone when the build finishes, even if curl fails. Nothing secret may appear in the log, and the file must be created with mode 600.

Hint

Use WORKSPACE_TMP, umask 077, set +x, and a post { always { … } } block.

Show Solution
pipeline {
    agent any
    stages {
        stage('Call API with a netrc file') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'registry',
                                                  usernameVariable: 'REG_USER',
                                                  passwordVariable: 'REG_PASS')]) {
                    sh '''
                        set +x
                        umask 077
                        NETRC="$WORKSPACE_TMP/netrc"
                        printf 'machine localhost login %s password %s\n' "$REG_USER" "$REG_PASS" > "$NETRC"
                        ls -l "$NETRC" | cut -d' ' -f1
                        curl -s -o /dev/null -w "HTTP %{http_code}\n" --netrc-file "$NETRC" "${JENKINS_URL}whoAmI/api/json"
                    '''
                }
            }
        }
    }
    post {
        always {
            sh 'rm -f "$WORKSPACE_TMP/netrc"; ls -A "$WORKSPACE_TMP"'
        }
    }
}
[Pipeline] sh
+ set +x
-rw-------
HTTP 401
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] sh
+ rm -f /home/sid/.jenkins/workspace/ex4@tmp/netrc
+ ls -A /home/sid/.jenkins/workspace/ex4@tmp
durable-fa56c283

The file was private (-rw-------), set +x kept the printf line out of the log, and after the post block only Jenkins' own durable-… control directory remains in @tmp. Writing to WORKSPACE_TMP instead of the workspace also means a later archiveArtifacts '**' cannot pick the file up by accident.


Tested with: Jenkins 2.580.2 on OpenJDK 21 (Ubuntu 24.04 under WSL2), Credentials plugin 1514.vb_01eb_014e7d4, Credentials Binding 728.v902a_273b_8947, November 2026. Every credential was created with the CLI and every pipeline was run on that controller; outputs are pasted from the console logs, trimmed where marked. UI navigation (menu names, the "Select a type of credential" dialog, "Treat username as secret") was checked against the pages the same version serves, fetched over HTTP, not clicked through. Two things surprised me: masking also caught Base64-encoded secrets, and the deploy username masked the word deploy everywhere in the log. The external secret-store plugins were not installed and were not tested. Paths from the test machine's scratch directories are shown shortened to /home/sid/…. Job, node and credential names are shown without the test run's prefixes.

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Jenkins tutorial here is tested end to end before publishing.