Loops, Functions, and Reading Input
Repeat work with for, while and until, read files and CSVs line by line without losing data, prompt users safely, and organise scripts into functions with local variables and main().
By The Dark DebuggerUpdated 33 min read
This article adds repetition and structure to the scripts from Your First Scripts: Arguments, Exit Codes, and Conditionals. We'll cover every loop Bash has, the correct way to read a file line by line (and the two traps that silently lose data), interactive input with read and select, and functions. It assumes you're comfortable with if, case, [[ ]] and exit codes from that article. Arrays and arithmetic get their own article next, Arrays, Associative Arrays, and Arithmetic; here we only use the simple $(( )) form to add numbers.
Why loops and functions
Most real scripting jobs are "do this for every X": every photo in a folder, every user in a CSV export, every host in an inventory, every attempt until a service answers. Without loops you copy and paste. Without functions, a 200-line script becomes one long block where a variable set on line 20 is silently overwritten on line 150.
By the end of this article you'll have three tools that show up in real work: a batch photo renamer with a dry-run mode, a user-provisioning script driven by a CSV, and a retry function with exponential backoff.
for loops
Looping over a list
The basic form runs the body once per word:
for host in web01 web02 db01; do echo "checking $host"; donechecking web01
checking web02
checking db01Written over several lines in a script, the same loop looks like this:
for host in web01 web02 db01; do
echo "checking $host"
doneLeaving out the in … part loops over the script's arguments, exactly like for arg in "$@":
set -- "access log.txt" error.log
for arg; do echo "arg: $arg"; donearg: access log.txt
arg: error.logLooping over files with a glob
A glob in the list expands to the matching filenames, each one a separate word even when it contains spaces. This is the correct way to process files:
ls logs
for f in logs/*.log; do echo "$f: $(wc -l < "$f") lines"; done api.log 'batch run.log' scheduler.log worker.log
logs/api.log: 3 lines
logs/batch run.log: 1 lines
logs/scheduler.log: 0 lines
logs/worker.log: 3 linesNotice "$f" is quoted inside the loop. The glob produced logs/batch run.log as one word; an unquoted $f would split it again.
When a glob matches nothing, Bash leaves it as the literal pattern, so the loop runs once with a filename that doesn't exist:
for f in empty/*.csv; do echo "processing $f"; done
shopt -s nullglob
for f in empty/*.csv; do echo "processing $f"; done
echo "loop done"processing empty/*.csv
loop doneWith shopt -s nullglob, a pattern with no matches expands to nothing and the loop body never runs. I put shopt -s nullglob near the top of any script that loops over globs. The alternative is a guard inside the loop, [[ -e $f ]] || continue, which we'll use in the recursive example later.
Brace ranges and C-style loops
Brace expansion generates sequences before the loop even starts (see Wildcards, Globbing, and Brace Expansion):
echo {1..5}
echo {01..12}
echo {0..100..25}
echo {z..a}1 2 3 4 5
01 02 03 04 05 06 07 08 09 10 11 12
0 25 50 75 100
z y x w v u t s r q p o n m l k j i h g f e d c b a{01..12} keeps the zero padding, which is handy for month directories. The catch: brace expansion happens before variables are expanded, so a variable can't be an endpoint:
n=5
echo {1..$n}{1..5}When the bounds are variables, use the C-style loop, or seq:
for ((i = 1; i <= n; i++)); do printf '%s ' "$i"; done; echo
seq -s ' ' 1 "$n"1 2 3 4 5
1 2 3 4 5Inside (( )) variables don't need a $, and i++ adds one. The arithmetic rules are covered properly in the next article.
Don't loop over $(cat file) or $(ls)
A for loop splits its list on whitespace, so feeding it a file's contents gives you words, not lines:
cat servers.txt
for line in $(cat servers.txt); do echo "line: $line"; doneweb01 eu-west-1
web02 eu-west-1
db01 us-east-1
line: web01
line: eu-west-1
line: web02
line: eu-west-1
line: db01
line: us-east-1$(ls) has the same problem with filenames:
for f in $(ls logs); do echo "file: $f"; donefile: api.log
file: batch
file: run.log
file: scheduler.log
file: worker.logFor files, use a glob (for f in logs/*). For lines, use while read, which is the next big topic.
while and until
while repeats as long as its condition command succeeds; until repeats as long as it fails. The condition is any command, just as with if.
attempt=1
while [[ $attempt -le 3 ]]; do echo "attempt $attempt"; attempt=$((attempt + 1)); doneattempt 1
attempt 2
attempt 3until reads naturally for "wait for something to become true". This script waits for a deploy marker file that a background job creates after three seconds (the ( … ) & runs that job in the background; Processes, Signals, and Job Control explains & in detail):
#!/usr/bin/env bash
# wait-ready.sh - poll until a deploy marker file appears
rm -f ready.flag
(sleep 3; touch ready.flag) & # stand-in for "another process finishes deploying"
until [[ -f ready.flag ]]; do
echo "$(date +%T) waiting for ready.flag"
sleep 1
done
echo "$(date +%T) ready"./wait-ready.sh21:41:22 waiting for ready.flag
21:41:23 waiting for ready.flag
21:41:24 waiting for ready.flag
21:41:25 readyA polling loop without a time limit can hang a script forever if the thing never happens. Exercise 3 adds a timeout, and the retry function later in this article caps the number of attempts.
while true; do …; done is an intentional infinite loop. It's how long-running workers are written, and it always needs a break or exit somewhere inside.
break and continue
continue skips to the next iteration; break leaves the loop. This script reports on each log until it finds the first one containing errors, skipping empty files:
#!/usr/bin/env bash
for f in logs/*.log; do
if [[ ! -s $f ]]; then
echo "skipping empty $f"
continue
fi
if grep -q ERROR "$f"; then
echo "first log with errors: $f"
break
fi
echo "no errors in $f"
doneno errors in logs/api.log
no errors in logs/batch run.log
skipping empty logs/scheduler.log
first log with errors: logs/worker.logIn nested loops, break only leaves the innermost one. break 2 leaves two levels:
#!/usr/bin/env bash
for host in web01 web02 web03; do
for port in 80 443 8080; do
if [[ $host == web02 && $port == 443 ]]; then
echo "found $host:$port, stopping both loops"
break 2
fi
echo "checked $host:$port"
done
done
echo "after the loops"checked web01:80
checked web01:443
checked web01:8080
checked web02:80
found web02:443, stopping both loops
after the loopsReading a file line by line
The correct idiom is worth memorising exactly as written:
while IFS= read -r line; do
…
done < file.txtEach part is there to fix a specific problem.
Why IFS= and -r
read normally strips leading and trailing whitespace (IFS= turns that off) and treats backslashes as escape characters (-r turns that off). Here is a file with indentation and a Windows path, and what each version of the loop does with it:
cat -A paths.txt
while read line; do echo "[$line]"; done < paths.txt
while IFS= read -r line; do echo "[$line]"; done < paths.txt indented four spaces$
^Iindented with a tab$
C:\Users\deploy\new_release$
[indented four spaces]
[indented with a tab]
[C:Usersdeploynew_release]
[ indented four spaces]
[ indented with a tab]
[C:\Users\deploy\new_release]The plain read lost the indentation (which matters in YAML or Python) and ate every backslash. Always write IFS= read -r.
The missing last line
Files produced by some editors and tools don't end with a newline. read returns failure when it hits end-of-file, even if it read some text first, so the loop exits before processing the last line:
while IFS= read -r host; do echo "host: $host"; done < hosts-no-newline.txt
while IFS= read -r host || [[ -n $host ]]; do echo "host: $host"; done < hosts-no-newline.txthost: web01
host: web02
host: web01
host: web02
host: web03web03 disappeared in the first loop. || [[ -n $host ]] keeps going one more time when the final read returned text without a newline.
Splitting fields: CSV and other delimited files
Give read several variable names and it splits the line on IFS, putting any leftover text in the last variable. Setting IFS=, just for the read command splits on commas:
while IFS=, read -r username email team active; do echo "$username ($team) active=$active"; done < users.csvusername (team) active=active
asha (platform) active=yes
ben (payments) active=no
chen (platform) active=yes
dmitri (data) active=yes
farah (payments) active=yesThe header came through as data. Two clean ways to skip it: read one line before the loop, inside a { } group that shares the same input, or feed the loop from tail -n +2 with process substitution (< <(command)), which we'll explain in a moment. _ is the conventional name for a field you don't care about:
{ read -r header; while IFS=, read -r username email team active; do echo "$username ($team) active=$active"; done; } < users.csv
while IFS=, read -r username _ team _; do echo "$username is in $team"; done < <(tail -n +2 users.csv)asha (platform) active=yes
ben (payments) active=no
chen (platform) active=yes
dmitri (data) active=yes
farah (payments) active=yes
asha is in platform
ben is in payments
chen is in platform
dmitri is in data
farah is in paymentsThis simple splitting is fine for CSVs you control. A CSV with quoted fields containing commas ("Smith, Jane") needs a real parser; awk with FPAT or a few lines of Python will save you a lot of pain there.
The pipe trap: variables that vanish
This is the most common loop bug in Bash. Count the active users by piping grep into the loop:
#!/usr/bin/env bash
active=0
grep ',yes$' users.csv | while IFS=, read -r username _; do
active=$((active + 1))
echo "inside the loop: active=$active ($username)"
done
echo "after the loop: active=$active"./count-active.shinside the loop: active=1 (asha)
inside the loop: active=2 (chen)
inside the loop: active=3 (dmitri)
inside the loop: active=4 (farah)
after the loop: active=0Each part of a pipeline runs in its own subshell, a copy of the script's shell. The loop counted to 4 in the copy, and the copy disappeared when the pipeline ended. There are two fixes.
Process substitution keeps the loop in the main shell and puts the command on the other side. <(grep …) behaves like a file containing the command's output, and the first < redirects it into the loop:
#!/usr/bin/env bash
active=0
while IFS=, read -r username _; do
active=$((active + 1))
done < <(grep ',yes$' users.csv)
echo "after the loop: active=$active"after the loop: active=4shopt -s lastpipe tells Bash to run the last part of a pipeline in the main shell. It only works when job control is off, which is the case in scripts but not at an interactive prompt:
#!/usr/bin/env bash
shopt -s lastpipe
active=0
grep ',yes$' users.csv | while IFS=, read -r username _; do
active=$((active + 1))
done
echo "after the loop: active=$active"after the loop: active=4I use process substitution: it works everywhere, including when you paste the loop into a terminal to test it.
When something inside the loop reads stdin
The loop's input is redirected from the file, and every command inside the loop inherits that input. Anything that reads stdin will eat lines meant for read. A confirmation prompt is the classic case:
#!/usr/bin/env bash
while IFS= read -r host; do
read -rp "Restart $host? [y/N] " answer
echo "host=$host answer=$answer"
done < hosts.txt./restart-hosts.shhost=web01 answer=web02
host=web03 answer=The inner read took web02 from the file as its "answer", and it never prompted at all, because read -p only shows the prompt when input comes from a terminal. ssh, ffmpeg and anything else that reads stdin cause the same skipped-lines bug in loops (for ssh, the -n option stops it reading stdin).
The fix is to give the file its own file descriptor. 3< hosts.txt opens the file as descriptor 3, and read -r host <&3 reads from it, leaving stdin connected to the keyboard:
#!/usr/bin/env bash
while IFS= read -r host <&3; do
read -rp "Restart $host? [y/N] " answer
echo "host=$host answer=$answer"
done 3< hosts.txtRestart web01? [y/N] y
host=web01 answer=y
Restart web02? [y/N] n
host=web02 answer=n
Restart web03? [y/N] y
host=web03 answer=yread: prompts, passwords, and timeouts
read has a handful of options worth knowing:
| Option | Effect |
|---|---|
-r | don't treat backslashes specially (always use it) |
-p "text" | show a prompt (only when reading from a terminal) |
-s | silent: don't echo what is typed, for passwords |
-t N | give up after N seconds; the exit status is then greater than 128 |
-n N | return after N characters, without waiting for Enter |
-a name | split the input into an array |
-d c | stop at character c instead of newline (-d '' means a NUL byte) |
This script asks for a host with a default, a password, and a confirmation that times out:
#!/usr/bin/env bash
read -rp "Database host [localhost]: " db_host
db_host=${db_host:-localhost}
read -rsp "Password for $db_host: " db_pass
echo # -s also swallows the Enter key, so print the newline ourselves
echo "Connecting to $db_host (password is ${#db_pass} characters long)"
read -rt 5 -p "Run migrations now? [y/N] " answer
status=$?
if [[ $status -gt 128 ]]; then
echo
echo "no answer after 5 seconds (read returned $status), assuming no"
answer=n
fi
echo "migrations: $answer"Two runs. In the first I pressed Enter at the host prompt, typed a password, and then waited; in the second I answered everything:
Database host [localhost]:
Password for localhost:
Connecting to localhost (password is 11 characters long)
Run migrations now? [y/N]
no answer after 5 seconds (read returned 142), assuming no
migrations: nDatabase host [localhost]: db.internal
Password for db.internal:
Connecting to db.internal (password is 8 characters long)
Run migrations now? [y/N] y
migrations: yThe password never appeared on screen. The timeout status was 142, which is 128 + 14 (SIGALRM). Timeouts matter for scripts that might run unattended: a prompt with no -t waits forever when nobody is there.
-n 1 gives you a single-keypress prompt, handy for yes/no questions:
#!/usr/bin/env bash
read -rn 1 -p "Delete 3 old snapshots? [y/N] " key
echo
case $key in
y|Y) echo "deleting" ;;
*) echo "keeping them" ;;
esacDelete 3 old snapshots? [y/N] y
deleting-a splits a line into an array, which is a quick way to break up a delimited string:
IFS=, read -ra fields <<< "web01,10.0.1.11,eu-west-1"
echo "${#fields[@]} fields; the IP is ${fields[1]}"3 fields; the IP is 10.0.1.11(<<< is a here-string: it feeds the string to the command's stdin.)
select menus
select prints a numbered menu, reads a choice using the PS3 prompt, and sets the loop variable to the chosen item (or to empty for an invalid choice, with the raw input in $REPLY). It loops until you break:
#!/usr/bin/env bash
PS3="Deploy to which environment? "
select env in staging production quit; do
case $env in
staging|production)
echo "Deploying to $env"
break
;;
quit)
echo "Cancelled"
break
;;
*)
echo "Invalid choice '$REPLY', pick a number from the list"
;;
esac
doneI typed 5, then 1:
1) staging
2) production
3) quit
Deploy to which environment? 5
Invalid choice '5', pick a number from the list
Deploy to which environment? 1
Deploying to stagingselect is fine for small internal tools. Anything that might run from cron or CI should take its choices as arguments instead, because there is nobody there to type a number.
Functions
Defining and calling a function
A function is a named block of commands. Inside it, $1, $2, $# and "$@" refer to the function's arguments; $0 is still the script name, and FUNCNAME holds the function's own name:
#!/usr/bin/env bash
greet() {
echo "$FUNCNAME got $# argument(s): $*"
echo "first: $1, script: $0"
}
greet "Ada Lovelace" admin
greetgreet got 2 argument(s): Ada Lovelace admin
first: Ada Lovelace, script: ./greet-fn.sh
greet got 0 argument(s):
first: , script: ./greet-fn.shYou'll also see function greet { … }. It's a Bash/Ksh-only spelling of the same thing; I use greet() { … } because it also works in sh. Either way, the function must be defined before the line that calls it runs (see Common mistakes).
local: keep variables inside the function
Variables in Bash are global by default, including ones assigned inside a function. This is how a helper function silently clobbers its caller's data:
#!/usr/bin/env bash
count_errors() {
file=$1
n=$(grep -c ERROR "$file")
echo "$file: $n error(s)"
}
file="report.txt"
count_errors logs/worker.log
echo "after the call: file=$file n=$n"logs/worker.log: 1 error(s)
after the call: file=logs/worker.log n=1The caller's file was overwritten, and n leaked out. Declaring them local fixes both:
count_errors() {
local file=$1 n
n=$(grep -c ERROR "$file")
echo "$file: $n error(s)"
}logs/worker.log: 1 error(s)
after the call: file=report.txt n=Make every variable in a function local unless you deliberately want to set a global. One subtlety that catches experienced people: local is itself a command with its own exit status, so combining it with a command substitution hides failures:
#!/usr/bin/env bash
load_config() {
local config=$(cat /etc/myapp/config.yml)
echo "load_config: status after cat is $?"
}
load_config_fixed() {
local config
config=$(cat /etc/myapp/config.yml)
echo "load_config_fixed: status after cat is $?"
}
load_config
load_config_fixedcat: /etc/myapp/config.yml: No such file or directory
load_config: status after cat is 0
cat: /etc/myapp/config.yml: No such file or directory
load_config_fixed: status after cat is 1In the first version, $? is the status of local, which succeeded. Declare first, assign on a separate line, whenever you need the status. Error Handling, Traps, and Debugging shows how this also defeats set -e.
Returning results: exit status vs output
Functions have two output channels, and they're for different things:
return Nsets the exit status (0–255). Use it for success or failure, so the function works withif,&&and||.- stdout carries data. The caller captures it with
$(function_name).
A function whose last command is a test returns that test's status, which makes predicate functions tidy:
#!/usr/bin/env bash
too_big() { return 300; }
not_number() { return abc; }
too_big; echo "too_big returned $?"
not_number; echo "not_number returned $?"
is_valid_email() {
[[ $1 =~ ^[^@[:space:]]+@[^@[:space:]]+\.[a-z]{2,}$ ]]
}
if is_valid_email "asha.k@example.com"; then
echo "asha.k@example.com looks valid"
fi
is_valid_email "chen.wei@example" || echo "chen.wei@example is not valid"too_big returned 44
./returns.sh: line 3: return: abc: numeric argument required
not_number returned 2
asha.k@example.com looks valid
chen.wei@example is not validreturn 300 wrapped to 44 (300 − 256), and return abc is an error. return is never a way to hand back a number like a count or a percentage; print it instead.
Printing data has its own trap. When a function's output is captured, everything it prints to stdout is captured, including log messages:
#!/usr/bin/env bash
log() {
echo "[$(date +%T)] $*"
}
disk_usage() {
log "checking $1"
df --output=pcent "$1" | tail -n 1 | tr -dc '0-9'
}
usage=$(disk_usage /)
echo "usage is '$usage'"usage is '[21:41:35] checking /
1'The log line ended up inside usage, and any [ "$usage" -ge 90 ] check would now fail with "integer expression expected". Diagnostics belong on stderr, which $( ) doesn't capture. Changing the log body to echo "[$(date +%T)] $*" >&2 gives:
[21:41:35] checking /
usage is '1'The log line still reaches the terminal, and the value is clean. Make this a rule for every script: data to stdout, messages to stderr.
Recursion
A function can call itself. Because each call has its own local variables, recursion works naturally for tree-shaped data, such as directories:
#!/usr/bin/env bash
# walk.sh DIR - print a directory tree using a recursive function
walk() {
local dir=$1 indent=$2 entry
for entry in "$dir"/*; do
[[ -e $entry ]] || continue # empty directory: the glob stayed literal
echo "${indent}${entry##*/}"
if [[ -d $entry ]]; then
walk "$entry" "$indent "
fi
done
}
walk "${1:-.}" ""./walk.sh projectREADME.md
src
lib
log.sh
main.sh
tests
test_main.sh${entry##*/} strips everything up to the last /, leaving the name. Without local, the inner call would overwrite entry and dir, and the outer loop would go wrong after the first subdirectory. In practice find or tree does this job faster; recursion in Bash is for the occasional tree-shaped problem, not heavy lifting.
Organising a script around main()
Once a script has a few functions, give it a consistent shape:
- Shebang and a comment saying what the script does and how to call it.
- Settings (constants,
shoptoptions). - Functions, small and single-purpose.
- A
mainfunction holding the top-level logic. - One last line:
main "$@".
Every function is defined before anything runs, the top-level flow reads like a table of contents, and main's variables can be local too. There's also a practical benefit: Bash reads scripts as it executes them, so editing a long-running script mid-run can make it execute garbage. When the file ends with main "$@", Bash has already parsed the whole of main before running it. The provisioning script in the next section follows this shape.
Where you'll use this
Batch-renaming files
Camera and phone exports produce names like IMG_0042.JPG that say nothing about when the photo was taken. This script renames them to 2026-10-31_0042.jpg using each file's modification date, prints what it would do by default, and only renames with --apply. It also refuses to overwrite a file that already exists:
#!/usr/bin/env bash
# rename-photos.sh [--apply] DIR
# Rename camera files like IMG_0042.JPG to 2026-10-31_0042.jpg using each
# file's modification date. Without --apply it only prints what it would do.
shopt -s nullglob nocaseglob
apply=no
if [[ $1 == --apply ]]; then
apply=yes
shift
fi
if [[ $# -ne 1 || ! -d $1 ]]; then
echo "Usage: rename-photos.sh [--apply] DIR" >&2
exit 2
fi
dir=$1
count=0
for photo in "$dir"/IMG_*.jpg; do
name=${photo##*/} # IMG_0042.JPG
number=${name#IMG_} # 0042.JPG
number=${number%.*} # 0042
new_name="$(date -r "$photo" +%F)_$number.jpg"
if [[ -e $dir/$new_name ]]; then
echo "skip $name: $new_name already exists" >&2
continue
fi
echo "$name -> $new_name"
if [[ $apply == yes ]]; then
mv -n -- "$photo" "$dir/$new_name"
fi
count=$((count + 1))
done
if [[ $apply == yes ]]; then
echo "$count file(s) renamed"
else
echo "$count file(s) would be renamed; run again with --apply"
fils -l --time-style=long-iso photos
./rename-photos.sh photos
./rename-photos.sh --apply photos
ls photostotal 0
-rw-r--r-- 1 sid sid 0 2026-11-02 09:30 2026-11-02_0101.jpg
-rw-r--r-- 1 sid sid 0 2026-10-31 14:02 IMG_0042.JPG
-rw-r--r-- 1 sid sid 0 2026-10-31 14:05 IMG_0043.JPG
-rw-r--r-- 1 sid sid 0 2026-11-02 09:30 IMG_0101.jpg
-rw-r--r-- 1 sid sid 0 2026-11-02 10:00 notes.txt
IMG_0042.JPG -> 2026-10-31_0042.jpg
IMG_0043.JPG -> 2026-10-31_0043.jpg
skip IMG_0101.jpg: 2026-11-02_0101.jpg already exists
2 file(s) would be renamed; run again with --apply
IMG_0042.JPG -> 2026-10-31_0042.jpg
IMG_0043.JPG -> 2026-10-31_0043.jpg
skip IMG_0101.jpg: 2026-11-02_0101.jpg already exists
2 file(s) renamed
2026-10-31_0042.jpg 2026-10-31_0043.jpg 2026-11-02_0101.jpg IMG_0101.jpg notes.txtnocaseglob lets IMG_*.jpg match both .JPG and .jpg. The ${name#IMG_} and ${number%.*} expansions strip a prefix and a suffix without calling any external program. mv -n (no-clobber) is a second safety net behind the explicit -e check, and -- stops a filename beginning with - from being read as an option. The same skeleton (glob, compute a new name, check, dry-run by default) works for rotating logs, normalising upload names, or moving reports into dated folders.
Processing a CSV export
HR or an identity system hands you a CSV of users; you need to create accounts for the active ones and reject bad rows. This script uses everything above: a while read loop fed by process substitution, continue to skip rows, small functions, and main. It prints the useradd commands instead of running them, so you can review them first (and because creating users needs root, which my test lab doesn't have):
#!/usr/bin/env bash
# provision-users.sh CSV - print useradd commands for the active users in CSV
# Expected header: username,email,team,active
is_valid_email() {
[[ $1 =~ ^[^@[:space:]]+@[^@[:space:]]+\.[a-z]{2,}$ ]]
}
provision() {
local username=$1 email=$2 team=$3
echo "useradd -m -c '$email' -G '$team' '$username'"
}
main() {
local csv=$1 username email team active
local created=0 skipped=0
if [[ ! -r $csv ]]; then
echo "provision-users.sh: cannot read '$csv'" >&2
return 1
fi
while IFS=, read -r username email team active; do
if [[ $active != yes ]]; then
echo "skip $username: inactive" >&2
skipped=$((skipped + 1))
continue
fi
if ! is_valid_email "$email"; then
echo "skip $username: invalid email '$email'" >&2
skipped=$((skipped + 1))
continue
fi
provision "$username" "$email" "$team"
created=$((created + 1))
done < <(tail -n +2 "$csv")
echo "$created to create, $skipped skipped" >&2
}
main "$@"./provision-users.sh users.csvuseradd -m -c 'asha.k@example.com' -G 'platform' 'asha'
skip ben: inactive
skip chen: invalid email 'chen.wei@example'
useradd -m -c 'dmitri.p@example.com' -G 'data' 'dmitri'
useradd -m -c 'farah.n@example.com' -G 'payments' 'farah'
3 to create, 2 skippedBecause the commands go to stdout and the commentary goes to stderr, you can capture a clean, reviewable file of commands:
./provision-users.sh users.csv 2>/dev/null > create-users.sh
cat create-users.shuseradd -m -c 'asha.k@example.com' -G 'platform' 'asha'
useradd -m -c 'dmitri.p@example.com' -G 'data' 'dmitri'
useradd -m -c 'farah.n@example.com' -G 'payments' 'farah'The counters work because the loop reads from < <(tail …) rather than a pipe. Written as tail -n +2 "$csv" | while …, the summary line would always say 0 to create, 0 skipped.
Retrying a flaky command with backoff
Deploy scripts constantly wait for something: a service to come up after a restart, a load balancer health check, a package mirror that drops connections. A retry function wraps any command and runs it until it succeeds, doubling the wait each time:
#!/usr/bin/env bash
# retry.sh - wait for local services, retrying with exponential backoff
# retry MAX_ATTEMPTS COMMAND [ARGS...]
# Runs COMMAND until it succeeds, waiting 1s, 2s, 4s, ... between attempts.
retry() {
local max=$1 attempt=1 delay=1
shift
until "$@"; do
if [[ $attempt -ge $max ]]; then
echo "retry: giving up after $attempt attempts: $*" >&2
return 1
fi
echo "retry: attempt $attempt failed, retrying in ${delay}s" >&2
sleep "$delay"
attempt=$((attempt + 1))
delay=$((delay * 2))
done
}
main() {
local start=$SECONDS
retry 5 curl -fsS -o /dev/null http://127.0.0.1:8765/ || return 1
echo "service on 8765 healthy after $((SECONDS - start))s"
retry 3 curl -fsS -o /dev/null http://127.0.0.1:8799/ || echo "service on 8799 never came up"
}
main "$@"To test it for real, I started a small web server on port 8765 that only comes up after 2.5 seconds, and left port 8799 with nothing listening:
(sleep 2.5; exec python3 -m http.server 8765 --bind 127.0.0.1 >/dev/null 2>&1) & sleep 0.2
./retry.sh[1] 9416
curl: (7) Failed to connect to 127.0.0.1 port 8765 after 0 ms: Couldn't connect to server
retry: attempt 1 failed, retrying in 1s
curl: (7) Failed to connect to 127.0.0.1 port 8765 after 0 ms: Couldn't connect to server
retry: attempt 2 failed, retrying in 2s
service on 8765 healthy after 4s
curl: (7) Failed to connect to 127.0.0.1 port 8799 after 0 ms: Couldn't connect to server
retry: attempt 1 failed, retrying in 1s
curl: (7) Failed to connect to 127.0.0.1 port 8799 after 0 ms: Couldn't connect to server
retry: attempt 2 failed, retrying in 2s
curl: (7) Failed to connect to 127.0.0.1 port 8799 after 0 ms: Couldn't connect to server
retry: giving up after 3 attempts: curl -fsS -o /dev/null http://127.0.0.1:8799/
service on 8799 never came upThe key line is until "$@": the function runs whatever command and arguments it was given, intact, because of the quotes. That makes retry reusable for curl, ssh, pg_isready, aws s3 cp or anything else that signals failure through its exit status. SECONDS is a built-in variable counting seconds since the script started. The [1] 9416 line is the interactive shell reporting the background job.
Common mistakes
Calling a function before it's defined
#!/usr/bin/env bash
check_disk /
check_disk() {
df -h "$1" | tail -n 1
}./early.sh: line 2: check_disk: command not foundBash executes the file top to bottom, and at line 2 check_disk doesn't exist yet. Define functions first, or use the main "$@" layout so that every call happens after every definition.
Using local outside a function
local x=1bash: local: can only be used in a functionThis usually happens when code is moved out of a function into the top level of a script. At the top level, just assign the variable.
A CSV from Excel where nothing matches
users-excel.csv has the same rows as users.csv, but was saved with Windows line endings. Every user is skipped:
./provision-users.sh users-excel.csvskip asha: inactive
skip ben: inactive
skip chen: inactive
skip dmitri: inactive
skip farah: inactive
0 to create, 5 skippedThe last field of every line is yes\r, not yes, so [[ $active != yes ]] is always true. The carriage return is invisible in the output. Strip it as you read (active=${active%$'\r'} inside the loop) or clean the file first with sed -i 's/\r$//' users-excel.csv.
Looping over $(ls) or $(cat)
Covered above, but it's common enough to repeat with the error you'll actually see:
for f in $(ls logs); do wc -l "logs/$f"; done3 logs/api.log
wc: logs/batch: No such file or directory
wc: logs/run.log: No such file or directory
0 logs/scheduler.log
3 logs/worker.logThe glob version handles the space correctly:
for f in logs/*; do wc -l "$f"; done3 logs/api.log
1 logs/batch run.log
0 logs/scheduler.log
3 logs/worker.logSummary
for x in listloops over words;for f in dir/*is the safe way to loop over files. Turn onshopt -s nullglobso a glob that matches nothing produces no iterations instead of a literal*.- Brace ranges like
{01..12}can't use variables; usefor ((i = 1; i <= n; i++))orseqwhen the bounds are variables. Never loop over$(cat file)or$(ls). whileruns while its condition command succeeds,untilwhile it fails.continueskips an iteration,breakleaves the loop,break 2leaves two nested loops. Every polling loop needs a limit.- Read lines with
while IFS= read -r line; do … done < file. Add|| [[ -n $line ]]for files without a final newline, andIFS=,with several variable names to split fields. - A pipe into
whileruns the loop in a subshell, so variables set inside vanish. Feed the loop withdone < <(command)instead. Commands inside the loop that read stdin steal its lines; read the file on descriptor 3 (read -r line <&3…done 3< file). read -pprompts,-shides input,-ttimes out with a status above 128,-n 1takes a single key.selectbuilds numbered menus for interactive tools.- In functions, make every variable
local, and declare it on a separate line when you need the exit status of a command substitution. Usereturnfor success or failure, stdout for data, and stderr for log messages. - Structure scripts as settings, then functions, then
main, ending withmain "$@".
Exercises
Write log-summary.sh [DIR] that prints, for each .log file in DIR (default: the current directory), its line count and the number of lines containing ERROR. Empty files should be reported as empty without running wc or grep. If there are no .log files at all, print no .log files in DIR.
Hint
printf '%-22s %4d lines\n' … lines up columns. With nullglob on, you can set a flag inside the loop to know whether it ran at all.
Show Solution
#!/usr/bin/env bash
# log-summary.sh [DIR] - lines and ERROR count for every .log file in DIR
shopt -s nullglob
dir=${1:-.}
found=0
for f in "$dir"/*.log; do
found=1
if [[ ! -s $f ]]; then
echo "$f: empty"
continue
fi
printf '%-22s %4d lines %3d errors\n' "$f" "$(wc -l < "$f")" "$(grep -c ERROR "$f")"
done
if [[ $found -eq 0 ]]; then
echo "no .log files in $dir"
fi./log-summary.sh logs
./log-summary.sh emptylogs/api.log 3 lines 0 errors
logs/batch run.log 1 lines 0 errors
logs/scheduler.log: empty
logs/worker.log 3 lines 1 errors
no .log files in emptywc -l < "$f" (input redirection) prints only the number; wc -l "$f" would also print the name. The file with a space in its name works because "$f" is quoted everywhere.
Using users.csv from this article, write team-emails.sh TEAM [CSV] that prints the email address of every active user in TEAM, one per line, then prints N active member(s) in TEAM to stderr. The count must be correct, so think about where the loop runs. Skip the header. With no arguments, print a usage message and exit 2.
Show Solution
#!/usr/bin/env bash
# team-emails.sh TEAM [CSV] - print the emails of active members of TEAM
if [[ $# -lt 1 ]]; then
echo "Usage: team-emails.sh TEAM [CSV]" >&2
exit 2
fi
team=$1
csv=${2:-users.csv}
count=0
{
read -r _header
while IFS=, read -r username email user_team active; do
if [[ $user_team == "$team" && $active == yes ]]; then
echo "$email"
count=$((count + 1))
fi
done
} < "$csv"
echo "$count active member(s) in $team" >&2./team-emails.sh platform
./team-emails.sh payments
./team-emails.sh; echo "exit $?"asha.k@example.com
chen.wei@example
2 active member(s) in platform
farah.n@example.com
1 active member(s) in payments
Usage: team-emails.sh TEAM [CSV]
exit 2The { … } < "$csv" group runs in the current shell, so count survives the loop, and the first read consumes the header from the same input. "$team" is quoted on the right of == so a team name containing * would be compared literally rather than as a pattern.
Write a function wait_for_file PATH TIMEOUT that checks once a second for PATH to exist. It prints PATH appeared after Ns and returns 0 when it does, or prints an error to stderr and returns 1 after TIMEOUT seconds. Test it with a file that a background job creates after 2 seconds, and with one that never appears.
Show Solution
#!/usr/bin/env bash
# wait_for_file PATH TIMEOUT_SECONDS - return 0 when PATH exists, 1 on timeout
wait_for_file() {
local path=$1 timeout=$2 waited=0
while [[ ! -e $path ]]; do
if [[ $waited -ge $timeout ]]; then
echo "wait_for_file: $path did not appear within ${timeout}s" >&2
return 1
fi
sleep 1
waited=$((waited + 1))
done
echo "$path appeared after ${waited}s"
}
rm -f backup.done
(sleep 2; touch backup.done) &
wait_for_file backup.done 10
wait_for_file restore.done 3 || echo "giving up on restore"./wait-for-file.shbackup.done appeared after 2s
wait_for_file: restore.done did not appear within 3s
giving up on restoreBecause it returns a proper status, the caller decides what a timeout means: || echo here, || exit 1 in a deploy script. All three variables are local, so calling it twice in a row doesn't leak waited between calls.
Write a function rotate FILE KEEP that rotates a log the way logrotate does: delete FILE.KEEP, rename FILE.(KEEP-1) to FILE.KEEP, and so on down to FILE.1 → FILE.2, then rename FILE to FILE.1 and create a new empty FILE. Return 1 with a message if FILE doesn't exist. Rotate four times with KEEP=3 and confirm that only the three newest copies remain, in the right order.
Hint
The renames must go from the highest number down, or each mv overwrites the copy you're about to move. A C-style loop can count down: for ((i = keep - 1; i >= 1; i--)).
Show Solution
#!/usr/bin/env bash
# rotate FILE KEEP - shift FILE.1 -> FILE.2 ... and FILE -> FILE.1, keeping KEEP copies
rotate() {
local file=$1 keep=$2 i
[[ -f $file ]] || { echo "rotate: no such file: $file" >&2; return 1; }
rm -f -- "$file.$keep" # the oldest copy falls off the end
for ((i = keep - 1; i >= 1; i--)); do
if [[ -e $file.$i ]]; then
mv -- "$file.$i" "$file.$((i + 1))"
fi
done
mv -- "$file" "$file.1"
: > "$file" # start a fresh, empty log
}
rotate "$@"printf 'first\n' > app.log
./rotate.sh app.log 3; ls app.log*
echo "second" > app.log; ./rotate.sh app.log 3
echo "third" > app.log; ./rotate.sh app.log 3
echo "fourth" > app.log; ./rotate.sh app.log 3
ls app.log*
head app.log.*
./rotate.sh missing.log 3; echo "exit $?"app.log app.log.1
app.log app.log.1 app.log.2 app.log.3
==> app.log.1 <==
fourth
==> app.log.2 <==
third
==> app.log.3 <==
second
rotate: no such file: missing.log
exit 1first was rotated out on the fourth run, exactly as intended. : > "$file" truncates or creates a file using the no-op command :. A real logging application keeps its old file open after the rename, which is why logrotate either signals the application to reopen its log or uses copytruncate; this exercise covers the file shuffling, not that coordination.
Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), November 2026. Every loop, script and exercise above was run as shown; the interactive examples (read, select, the restart prompt) were typed into a real terminal session, and the retry test ran against a real python3 -m http.server listener. The useradd commands are printed rather than executed, because creating accounts needs root, which the test machine didn't have.
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.