Loops, Functions, and Reading Input

Repeat work with for, while and until, read files and CSVs line by line without losing data, prompt users safely, and organise scripts into functions with local variables and main().

By The Dark DebuggerUpdated 33 min read

This article adds repetition and structure to the scripts from Your First Scripts: Arguments, Exit Codes, and Conditionals. We'll cover every loop Bash has, the correct way to read a file line by line (and the two traps that silently lose data), interactive input with read and select, and functions. It assumes you're comfortable with if, case, [[ ]] and exit codes from that article. Arrays and arithmetic get their own article next, Arrays, Associative Arrays, and Arithmetic; here we only use the simple $(( )) form to add numbers.

Why loops and functions

Most real scripting jobs are "do this for every X": every photo in a folder, every user in a CSV export, every host in an inventory, every attempt until a service answers. Without loops you copy and paste. Without functions, a 200-line script becomes one long block where a variable set on line 20 is silently overwritten on line 150.

By the end of this article you'll have three tools that show up in real work: a batch photo renamer with a dry-run mode, a user-provisioning script driven by a CSV, and a retry function with exponential backoff.

for loops

Looping over a list

The basic form runs the body once per word:

for host in web01 web02 db01; do echo "checking $host"; done
checking web01
checking web02
checking db01

Written over several lines in a script, the same loop looks like this:

for host in web01 web02 db01; do
  echo "checking $host"
done

Leaving out the in … part loops over the script's arguments, exactly like for arg in "$@":

set -- "access log.txt" error.log
for arg; do echo "arg: $arg"; done
arg: access log.txt
arg: error.log

Looping over files with a glob

A glob in the list expands to the matching filenames, each one a separate word even when it contains spaces. This is the correct way to process files:

ls logs
for f in logs/*.log; do echo "$f: $(wc -l < "$f") lines"; done
 api.log  'batch run.log'   scheduler.log   worker.log
logs/api.log: 3 lines
logs/batch run.log: 1 lines
logs/scheduler.log: 0 lines
logs/worker.log: 3 lines

Notice "$f" is quoted inside the loop. The glob produced logs/batch run.log as one word; an unquoted $f would split it again.

When a glob matches nothing, Bash leaves it as the literal pattern, so the loop runs once with a filename that doesn't exist:

for f in empty/*.csv; do echo "processing $f"; done
shopt -s nullglob
for f in empty/*.csv; do echo "processing $f"; done
echo "loop done"
processing empty/*.csv
loop done

With shopt -s nullglob, a pattern with no matches expands to nothing and the loop body never runs. I put shopt -s nullglob near the top of any script that loops over globs. The alternative is a guard inside the loop, [[ -e $f ]] || continue, which we'll use in the recursive example later.

Brace ranges and C-style loops

Brace expansion generates sequences before the loop even starts (see Wildcards, Globbing, and Brace Expansion):

echo {1..5}
echo {01..12}
echo {0..100..25}
echo {z..a}
1 2 3 4 5
01 02 03 04 05 06 07 08 09 10 11 12
0 25 50 75 100
z y x w v u t s r q p o n m l k j i h g f e d c b a

{01..12} keeps the zero padding, which is handy for month directories. The catch: brace expansion happens before variables are expanded, so a variable can't be an endpoint:

n=5
echo {1..$n}
{1..5}

When the bounds are variables, use the C-style loop, or seq:

for ((i = 1; i <= n; i++)); do printf '%s ' "$i"; done; echo
seq -s ' ' 1 "$n"
1 2 3 4 5 
1 2 3 4 5

Inside (( )) variables don't need a $, and i++ adds one. The arithmetic rules are covered properly in the next article.

Don't loop over $(cat file) or $(ls)

A for loop splits its list on whitespace, so feeding it a file's contents gives you words, not lines:

cat servers.txt
for line in $(cat servers.txt); do echo "line: $line"; done
web01 eu-west-1
web02 eu-west-1
db01 us-east-1
line: web01
line: eu-west-1
line: web02
line: eu-west-1
line: db01
line: us-east-1

$(ls) has the same problem with filenames:

for f in $(ls logs); do echo "file: $f"; done
file: api.log
file: batch
file: run.log
file: scheduler.log
file: worker.log

For files, use a glob (for f in logs/*). For lines, use while read, which is the next big topic.

while and until

while repeats as long as its condition command succeeds; until repeats as long as it fails. The condition is any command, just as with if.

attempt=1
while [[ $attempt -le 3 ]]; do echo "attempt $attempt"; attempt=$((attempt + 1)); done
attempt 1
attempt 2
attempt 3

until reads naturally for "wait for something to become true". This script waits for a deploy marker file that a background job creates after three seconds (the ( … ) & runs that job in the background; Processes, Signals, and Job Control explains & in detail):

#!/usr/bin/env bash
# wait-ready.sh - poll until a deploy marker file appears
rm -f ready.flag
(sleep 3; touch ready.flag) &    # stand-in for "another process finishes deploying"
 
until [[ -f ready.flag ]]; do
  echo "$(date +%T) waiting for ready.flag"
  sleep 1
done
echo "$(date +%T) ready"
./wait-ready.sh
21:41:22 waiting for ready.flag
21:41:23 waiting for ready.flag
21:41:24 waiting for ready.flag
21:41:25 ready

A polling loop without a time limit can hang a script forever if the thing never happens. Exercise 3 adds a timeout, and the retry function later in this article caps the number of attempts.

while true; do …; done is an intentional infinite loop. It's how long-running workers are written, and it always needs a break or exit somewhere inside.

break and continue

continue skips to the next iteration; break leaves the loop. This script reports on each log until it finds the first one containing errors, skipping empty files:

#!/usr/bin/env bash
for f in logs/*.log; do
  if [[ ! -s $f ]]; then
    echo "skipping empty $f"
    continue
  fi
  if grep -q ERROR "$f"; then
    echo "first log with errors: $f"
    break
  fi
  echo "no errors in $f"
done
no errors in logs/api.log
no errors in logs/batch run.log
skipping empty logs/scheduler.log
first log with errors: logs/worker.log

In nested loops, break only leaves the innermost one. break 2 leaves two levels:

#!/usr/bin/env bash
for host in web01 web02 web03; do
  for port in 80 443 8080; do
    if [[ $host == web02 && $port == 443 ]]; then
      echo "found $host:$port, stopping both loops"
      break 2
    fi
    echo "checked $host:$port"
  done
done
echo "after the loops"
checked web01:80
checked web01:443
checked web01:8080
checked web02:80
found web02:443, stopping both loops
after the loops

Reading a file line by line

The correct idiom is worth memorising exactly as written:

while IFS= read -r line; do
  …
done < file.txt

Each part is there to fix a specific problem.

Why IFS= and -r

read normally strips leading and trailing whitespace (IFS= turns that off) and treats backslashes as escape characters (-r turns that off). Here is a file with indentation and a Windows path, and what each version of the loop does with it:

cat -A paths.txt
while read line; do echo "[$line]"; done < paths.txt
while IFS= read -r line; do echo "[$line]"; done < paths.txt
    indented four spaces$
^Iindented with a tab$
C:\Users\deploy\new_release$
[indented four spaces]
[indented with a tab]
[C:Usersdeploynew_release]
[    indented four spaces]
[	indented with a tab]
[C:\Users\deploy\new_release]

The plain read lost the indentation (which matters in YAML or Python) and ate every backslash. Always write IFS= read -r.

The missing last line

Files produced by some editors and tools don't end with a newline. read returns failure when it hits end-of-file, even if it read some text first, so the loop exits before processing the last line:

while IFS= read -r host; do echo "host: $host"; done < hosts-no-newline.txt
while IFS= read -r host || [[ -n $host ]]; do echo "host: $host"; done < hosts-no-newline.txt
host: web01
host: web02
host: web01
host: web02
host: web03

web03 disappeared in the first loop. || [[ -n $host ]] keeps going one more time when the final read returned text without a newline.

Splitting fields: CSV and other delimited files

Give read several variable names and it splits the line on IFS, putting any leftover text in the last variable. Setting IFS=, just for the read command splits on commas:

while IFS=, read -r username email team active; do echo "$username ($team) active=$active"; done < users.csv
username (team) active=active
asha (platform) active=yes
ben (payments) active=no
chen (platform) active=yes
dmitri (data) active=yes
farah (payments) active=yes

The header came through as data. Two clean ways to skip it: read one line before the loop, inside a { } group that shares the same input, or feed the loop from tail -n +2 with process substitution (< <(command)), which we'll explain in a moment. _ is the conventional name for a field you don't care about:

{ read -r header; while IFS=, read -r username email team active; do echo "$username ($team) active=$active"; done; } < users.csv
while IFS=, read -r username _ team _; do echo "$username is in $team"; done < <(tail -n +2 users.csv)
asha (platform) active=yes
ben (payments) active=no
chen (platform) active=yes
dmitri (data) active=yes
farah (payments) active=yes
asha is in platform
ben is in payments
chen is in platform
dmitri is in data
farah is in payments

This simple splitting is fine for CSVs you control. A CSV with quoted fields containing commas ("Smith, Jane") needs a real parser; awk with FPAT or a few lines of Python will save you a lot of pain there.

The pipe trap: variables that vanish

This is the most common loop bug in Bash. Count the active users by piping grep into the loop:

#!/usr/bin/env bash
active=0
grep ',yes$' users.csv | while IFS=, read -r username _; do
  active=$((active + 1))
  echo "inside the loop: active=$active ($username)"
done
echo "after the loop: active=$active"
./count-active.sh
inside the loop: active=1 (asha)
inside the loop: active=2 (chen)
inside the loop: active=3 (dmitri)
inside the loop: active=4 (farah)
after the loop: active=0

Each part of a pipeline runs in its own subshell, a copy of the script's shell. The loop counted to 4 in the copy, and the copy disappeared when the pipeline ended. There are two fixes.

Process substitution keeps the loop in the main shell and puts the command on the other side. <(grep …) behaves like a file containing the command's output, and the first < redirects it into the loop:

#!/usr/bin/env bash
active=0
while IFS=, read -r username _; do
  active=$((active + 1))
done < <(grep ',yes$' users.csv)
echo "after the loop: active=$active"
after the loop: active=4

shopt -s lastpipe tells Bash to run the last part of a pipeline in the main shell. It only works when job control is off, which is the case in scripts but not at an interactive prompt:

#!/usr/bin/env bash
shopt -s lastpipe
active=0
grep ',yes$' users.csv | while IFS=, read -r username _; do
  active=$((active + 1))
done
echo "after the loop: active=$active"
after the loop: active=4

I use process substitution: it works everywhere, including when you paste the loop into a terminal to test it.

When something inside the loop reads stdin

The loop's input is redirected from the file, and every command inside the loop inherits that input. Anything that reads stdin will eat lines meant for read. A confirmation prompt is the classic case:

#!/usr/bin/env bash
while IFS= read -r host; do
  read -rp "Restart $host? [y/N] " answer
  echo "host=$host answer=$answer"
done < hosts.txt
./restart-hosts.sh
host=web01 answer=web02
host=web03 answer=

The inner read took web02 from the file as its "answer", and it never prompted at all, because read -p only shows the prompt when input comes from a terminal. ssh, ffmpeg and anything else that reads stdin cause the same skipped-lines bug in loops (for ssh, the -n option stops it reading stdin).

The fix is to give the file its own file descriptor. 3< hosts.txt opens the file as descriptor 3, and read -r host <&3 reads from it, leaving stdin connected to the keyboard:

#!/usr/bin/env bash
while IFS= read -r host <&3; do
  read -rp "Restart $host? [y/N] " answer
  echo "host=$host answer=$answer"
done 3< hosts.txt
Restart web01? [y/N] y
host=web01 answer=y
Restart web02? [y/N] n
host=web02 answer=n
Restart web03? [y/N] y
host=web03 answer=y

read: prompts, passwords, and timeouts

read has a handful of options worth knowing:

OptionEffect
-rdon't treat backslashes specially (always use it)
-p "text"show a prompt (only when reading from a terminal)
-ssilent: don't echo what is typed, for passwords
-t Ngive up after N seconds; the exit status is then greater than 128
-n Nreturn after N characters, without waiting for Enter
-a namesplit the input into an array
-d cstop at character c instead of newline (-d '' means a NUL byte)

This script asks for a host with a default, a password, and a confirmation that times out:

#!/usr/bin/env bash
read -rp "Database host [localhost]: " db_host
db_host=${db_host:-localhost}
 
read -rsp "Password for $db_host: " db_pass
echo    # -s also swallows the Enter key, so print the newline ourselves
 
echo "Connecting to $db_host (password is ${#db_pass} characters long)"
 
read -rt 5 -p "Run migrations now? [y/N] " answer
status=$?
if [[ $status -gt 128 ]]; then
  echo
  echo "no answer after 5 seconds (read returned $status), assuming no"
  answer=n
fi
echo "migrations: $answer"

Two runs. In the first I pressed Enter at the host prompt, typed a password, and then waited; in the second I answered everything:

Database host [localhost]: 
Password for localhost: 
Connecting to localhost (password is 11 characters long)
Run migrations now? [y/N] 
no answer after 5 seconds (read returned 142), assuming no
migrations: n
Database host [localhost]: db.internal
Password for db.internal: 
Connecting to db.internal (password is 8 characters long)
Run migrations now? [y/N] y
migrations: y

The password never appeared on screen. The timeout status was 142, which is 128 + 14 (SIGALRM). Timeouts matter for scripts that might run unattended: a prompt with no -t waits forever when nobody is there.

-n 1 gives you a single-keypress prompt, handy for yes/no questions:

#!/usr/bin/env bash
read -rn 1 -p "Delete 3 old snapshots? [y/N] " key
echo
case $key in
  y|Y) echo "deleting" ;;
  *)   echo "keeping them" ;;
esac
Delete 3 old snapshots? [y/N] y
deleting

-a splits a line into an array, which is a quick way to break up a delimited string:

IFS=, read -ra fields <<< "web01,10.0.1.11,eu-west-1"
echo "${#fields[@]} fields; the IP is ${fields[1]}"
3 fields; the IP is 10.0.1.11

(<<< is a here-string: it feeds the string to the command's stdin.)

select menus

select prints a numbered menu, reads a choice using the PS3 prompt, and sets the loop variable to the chosen item (or to empty for an invalid choice, with the raw input in $REPLY). It loops until you break:

#!/usr/bin/env bash
PS3="Deploy to which environment? "
select env in staging production quit; do
  case $env in
    staging|production)
      echo "Deploying to $env"
      break
      ;;
    quit)
      echo "Cancelled"
      break
      ;;
    *)
      echo "Invalid choice '$REPLY', pick a number from the list"
      ;;
  esac
done

I typed 5, then 1:

1) staging
2) production
3) quit
Deploy to which environment? 5
Invalid choice '5', pick a number from the list
Deploy to which environment? 1
Deploying to staging

select is fine for small internal tools. Anything that might run from cron or CI should take its choices as arguments instead, because there is nobody there to type a number.

Functions

Defining and calling a function

A function is a named block of commands. Inside it, $1, $2, $# and "$@" refer to the function's arguments; $0 is still the script name, and FUNCNAME holds the function's own name:

#!/usr/bin/env bash
greet() {
  echo "$FUNCNAME got $# argument(s): $*"
  echo "first: $1, script: $0"
}
 
greet "Ada Lovelace" admin
greet
greet got 2 argument(s): Ada Lovelace admin
first: Ada Lovelace, script: ./greet-fn.sh
greet got 0 argument(s): 
first: , script: ./greet-fn.sh

You'll also see function greet { … }. It's a Bash/Ksh-only spelling of the same thing; I use greet() { … } because it also works in sh. Either way, the function must be defined before the line that calls it runs (see Common mistakes).

local: keep variables inside the function

Variables in Bash are global by default, including ones assigned inside a function. This is how a helper function silently clobbers its caller's data:

#!/usr/bin/env bash
count_errors() {
  file=$1
  n=$(grep -c ERROR "$file")
  echo "$file: $n error(s)"
}
 
file="report.txt"
count_errors logs/worker.log
echo "after the call: file=$file n=$n"
logs/worker.log: 1 error(s)
after the call: file=logs/worker.log n=1

The caller's file was overwritten, and n leaked out. Declaring them local fixes both:

count_errors() {
  local file=$1 n
  n=$(grep -c ERROR "$file")
  echo "$file: $n error(s)"
}
logs/worker.log: 1 error(s)
after the call: file=report.txt n=

Make every variable in a function local unless you deliberately want to set a global. One subtlety that catches experienced people: local is itself a command with its own exit status, so combining it with a command substitution hides failures:

#!/usr/bin/env bash
load_config() {
  local config=$(cat /etc/myapp/config.yml)
  echo "load_config: status after cat is $?"
}
 
load_config_fixed() {
  local config
  config=$(cat /etc/myapp/config.yml)
  echo "load_config_fixed: status after cat is $?"
}
 
load_config
load_config_fixed
cat: /etc/myapp/config.yml: No such file or directory
load_config: status after cat is 0
cat: /etc/myapp/config.yml: No such file or directory
load_config_fixed: status after cat is 1

In the first version, $? is the status of local, which succeeded. Declare first, assign on a separate line, whenever you need the status. Error Handling, Traps, and Debugging shows how this also defeats set -e.

Returning results: exit status vs output

Functions have two output channels, and they're for different things:

  • return N sets the exit status (0–255). Use it for success or failure, so the function works with if, && and ||.
  • stdout carries data. The caller captures it with $(function_name).

A function whose last command is a test returns that test's status, which makes predicate functions tidy:

#!/usr/bin/env bash
too_big()    { return 300; }
not_number() { return abc; }
 
too_big;    echo "too_big returned $?"
not_number; echo "not_number returned $?"
 
is_valid_email() {
  [[ $1 =~ ^[^@[:space:]]+@[^@[:space:]]+\.[a-z]{2,}$ ]]
}
 
if is_valid_email "asha.k@example.com"; then
  echo "asha.k@example.com looks valid"
fi
is_valid_email "chen.wei@example" || echo "chen.wei@example is not valid"
too_big returned 44
./returns.sh: line 3: return: abc: numeric argument required
not_number returned 2
asha.k@example.com looks valid
chen.wei@example is not valid

return 300 wrapped to 44 (300 − 256), and return abc is an error. return is never a way to hand back a number like a count or a percentage; print it instead.

Printing data has its own trap. When a function's output is captured, everything it prints to stdout is captured, including log messages:

#!/usr/bin/env bash
log() {
  echo "[$(date +%T)] $*"
}
 
disk_usage() {
  log "checking $1"
  df --output=pcent "$1" | tail -n 1 | tr -dc '0-9'
}
 
usage=$(disk_usage /)
echo "usage is '$usage'"
usage is '[21:41:35] checking /
1'

The log line ended up inside usage, and any [ "$usage" -ge 90 ] check would now fail with "integer expression expected". Diagnostics belong on stderr, which $( ) doesn't capture. Changing the log body to echo "[$(date +%T)] $*" >&2 gives:

[21:41:35] checking /
usage is '1'

The log line still reaches the terminal, and the value is clean. Make this a rule for every script: data to stdout, messages to stderr.

Recursion

A function can call itself. Because each call has its own local variables, recursion works naturally for tree-shaped data, such as directories:

#!/usr/bin/env bash
# walk.sh DIR - print a directory tree using a recursive function
walk() {
  local dir=$1 indent=$2 entry
  for entry in "$dir"/*; do
    [[ -e $entry ]] || continue      # empty directory: the glob stayed literal
    echo "${indent}${entry##*/}"
    if [[ -d $entry ]]; then
      walk "$entry" "$indent    "
    fi
  done
}
 
walk "${1:-.}" ""
./walk.sh project
README.md
src
    lib
        log.sh
    main.sh
tests
    test_main.sh

${entry##*/} strips everything up to the last /, leaving the name. Without local, the inner call would overwrite entry and dir, and the outer loop would go wrong after the first subdirectory. In practice find or tree does this job faster; recursion in Bash is for the occasional tree-shaped problem, not heavy lifting.

Organising a script around main()

Once a script has a few functions, give it a consistent shape:

  1. Shebang and a comment saying what the script does and how to call it.
  2. Settings (constants, shopt options).
  3. Functions, small and single-purpose.
  4. A main function holding the top-level logic.
  5. One last line: main "$@".

Every function is defined before anything runs, the top-level flow reads like a table of contents, and main's variables can be local too. There's also a practical benefit: Bash reads scripts as it executes them, so editing a long-running script mid-run can make it execute garbage. When the file ends with main "$@", Bash has already parsed the whole of main before running it. The provisioning script in the next section follows this shape.

Where you'll use this

Batch-renaming files

Camera and phone exports produce names like IMG_0042.JPG that say nothing about when the photo was taken. This script renames them to 2026-10-31_0042.jpg using each file's modification date, prints what it would do by default, and only renames with --apply. It also refuses to overwrite a file that already exists:

#!/usr/bin/env bash
# rename-photos.sh [--apply] DIR
# Rename camera files like IMG_0042.JPG to 2026-10-31_0042.jpg using each
# file's modification date. Without --apply it only prints what it would do.
shopt -s nullglob nocaseglob
 
apply=no
if [[ $1 == --apply ]]; then
  apply=yes
  shift
fi
 
if [[ $# -ne 1 || ! -d $1 ]]; then
  echo "Usage: rename-photos.sh [--apply] DIR" >&2
  exit 2
fi
 
dir=$1
count=0
 
for photo in "$dir"/IMG_*.jpg; do
  name=${photo##*/}            # IMG_0042.JPG
  number=${name#IMG_}          # 0042.JPG
  number=${number%.*}          # 0042
  new_name="$(date -r "$photo" +%F)_$number.jpg"
 
  if [[ -e $dir/$new_name ]]; then
    echo "skip $name: $new_name already exists" >&2
    continue
  fi
 
  echo "$name -> $new_name"
  if [[ $apply == yes ]]; then
    mv -n -- "$photo" "$dir/$new_name"
  fi
  count=$((count + 1))
done
 
if [[ $apply == yes ]]; then
  echo "$count file(s) renamed"
else
  echo "$count file(s) would be renamed; run again with --apply"
fi
ls -l --time-style=long-iso photos
./rename-photos.sh photos
./rename-photos.sh --apply photos
ls photos
total 0
-rw-r--r-- 1 sid sid 0 2026-11-02 09:30 2026-11-02_0101.jpg
-rw-r--r-- 1 sid sid 0 2026-10-31 14:02 IMG_0042.JPG
-rw-r--r-- 1 sid sid 0 2026-10-31 14:05 IMG_0043.JPG
-rw-r--r-- 1 sid sid 0 2026-11-02 09:30 IMG_0101.jpg
-rw-r--r-- 1 sid sid 0 2026-11-02 10:00 notes.txt
IMG_0042.JPG -> 2026-10-31_0042.jpg
IMG_0043.JPG -> 2026-10-31_0043.jpg
skip IMG_0101.jpg: 2026-11-02_0101.jpg already exists
2 file(s) would be renamed; run again with --apply
IMG_0042.JPG -> 2026-10-31_0042.jpg
IMG_0043.JPG -> 2026-10-31_0043.jpg
skip IMG_0101.jpg: 2026-11-02_0101.jpg already exists
2 file(s) renamed
2026-10-31_0042.jpg  2026-10-31_0043.jpg  2026-11-02_0101.jpg  IMG_0101.jpg  notes.txt

nocaseglob lets IMG_*.jpg match both .JPG and .jpg. The ${name#IMG_} and ${number%.*} expansions strip a prefix and a suffix without calling any external program. mv -n (no-clobber) is a second safety net behind the explicit -e check, and -- stops a filename beginning with - from being read as an option. The same skeleton (glob, compute a new name, check, dry-run by default) works for rotating logs, normalising upload names, or moving reports into dated folders.

Processing a CSV export

HR or an identity system hands you a CSV of users; you need to create accounts for the active ones and reject bad rows. This script uses everything above: a while read loop fed by process substitution, continue to skip rows, small functions, and main. It prints the useradd commands instead of running them, so you can review them first (and because creating users needs root, which my test lab doesn't have):

#!/usr/bin/env bash
# provision-users.sh CSV - print useradd commands for the active users in CSV
# Expected header: username,email,team,active
 
is_valid_email() {
  [[ $1 =~ ^[^@[:space:]]+@[^@[:space:]]+\.[a-z]{2,}$ ]]
}
 
provision() {
  local username=$1 email=$2 team=$3
  echo "useradd -m -c '$email' -G '$team' '$username'"
}
 
main() {
  local csv=$1 username email team active
  local created=0 skipped=0
 
  if [[ ! -r $csv ]]; then
    echo "provision-users.sh: cannot read '$csv'" >&2
    return 1
  fi
 
  while IFS=, read -r username email team active; do
    if [[ $active != yes ]]; then
      echo "skip $username: inactive" >&2
      skipped=$((skipped + 1))
      continue
    fi
    if ! is_valid_email "$email"; then
      echo "skip $username: invalid email '$email'" >&2
      skipped=$((skipped + 1))
      continue
    fi
    provision "$username" "$email" "$team"
    created=$((created + 1))
  done < <(tail -n +2 "$csv")
 
  echo "$created to create, $skipped skipped" >&2
}
 
main "$@"
./provision-users.sh users.csv
useradd -m -c 'asha.k@example.com' -G 'platform' 'asha'
skip ben: inactive
skip chen: invalid email 'chen.wei@example'
useradd -m -c 'dmitri.p@example.com' -G 'data' 'dmitri'
useradd -m -c 'farah.n@example.com' -G 'payments' 'farah'
3 to create, 2 skipped

Because the commands go to stdout and the commentary goes to stderr, you can capture a clean, reviewable file of commands:

./provision-users.sh users.csv 2>/dev/null > create-users.sh
cat create-users.sh
useradd -m -c 'asha.k@example.com' -G 'platform' 'asha'
useradd -m -c 'dmitri.p@example.com' -G 'data' 'dmitri'
useradd -m -c 'farah.n@example.com' -G 'payments' 'farah'

The counters work because the loop reads from < <(tail …) rather than a pipe. Written as tail -n +2 "$csv" | while …, the summary line would always say 0 to create, 0 skipped.

Retrying a flaky command with backoff

Deploy scripts constantly wait for something: a service to come up after a restart, a load balancer health check, a package mirror that drops connections. A retry function wraps any command and runs it until it succeeds, doubling the wait each time:

#!/usr/bin/env bash
# retry.sh - wait for local services, retrying with exponential backoff
 
# retry MAX_ATTEMPTS COMMAND [ARGS...]
# Runs COMMAND until it succeeds, waiting 1s, 2s, 4s, ... between attempts.
retry() {
  local max=$1 attempt=1 delay=1
  shift
 
  until "$@"; do
    if [[ $attempt -ge $max ]]; then
      echo "retry: giving up after $attempt attempts: $*" >&2
      return 1
    fi
    echo "retry: attempt $attempt failed, retrying in ${delay}s" >&2
    sleep "$delay"
    attempt=$((attempt + 1))
    delay=$((delay * 2))
  done
}
 
main() {
  local start=$SECONDS
  retry 5 curl -fsS -o /dev/null http://127.0.0.1:8765/ || return 1
  echo "service on 8765 healthy after $((SECONDS - start))s"
 
  retry 3 curl -fsS -o /dev/null http://127.0.0.1:8799/ || echo "service on 8799 never came up"
}
 
main "$@"

To test it for real, I started a small web server on port 8765 that only comes up after 2.5 seconds, and left port 8799 with nothing listening:

(sleep 2.5; exec python3 -m http.server 8765 --bind 127.0.0.1 >/dev/null 2>&1) & sleep 0.2
./retry.sh
[1] 9416
curl: (7) Failed to connect to 127.0.0.1 port 8765 after 0 ms: Couldn't connect to server
retry: attempt 1 failed, retrying in 1s
curl: (7) Failed to connect to 127.0.0.1 port 8765 after 0 ms: Couldn't connect to server
retry: attempt 2 failed, retrying in 2s
service on 8765 healthy after 4s
curl: (7) Failed to connect to 127.0.0.1 port 8799 after 0 ms: Couldn't connect to server
retry: attempt 1 failed, retrying in 1s
curl: (7) Failed to connect to 127.0.0.1 port 8799 after 0 ms: Couldn't connect to server
retry: attempt 2 failed, retrying in 2s
curl: (7) Failed to connect to 127.0.0.1 port 8799 after 0 ms: Couldn't connect to server
retry: giving up after 3 attempts: curl -fsS -o /dev/null http://127.0.0.1:8799/
service on 8799 never came up

The key line is until "$@": the function runs whatever command and arguments it was given, intact, because of the quotes. That makes retry reusable for curl, ssh, pg_isready, aws s3 cp or anything else that signals failure through its exit status. SECONDS is a built-in variable counting seconds since the script started. The [1] 9416 line is the interactive shell reporting the background job.

Common mistakes

Calling a function before it's defined

#!/usr/bin/env bash
check_disk /
 
check_disk() {
  df -h "$1" | tail -n 1
}
./early.sh: line 2: check_disk: command not found

Bash executes the file top to bottom, and at line 2 check_disk doesn't exist yet. Define functions first, or use the main "$@" layout so that every call happens after every definition.

Using local outside a function

local x=1
bash: local: can only be used in a function

This usually happens when code is moved out of a function into the top level of a script. At the top level, just assign the variable.

A CSV from Excel where nothing matches

users-excel.csv has the same rows as users.csv, but was saved with Windows line endings. Every user is skipped:

./provision-users.sh users-excel.csv
skip asha: inactive
skip ben: inactive
skip chen: inactive
skip dmitri: inactive
skip farah: inactive
0 to create, 5 skipped

The last field of every line is yes\r, not yes, so [[ $active != yes ]] is always true. The carriage return is invisible in the output. Strip it as you read (active=${active%$'\r'} inside the loop) or clean the file first with sed -i 's/\r$//' users-excel.csv.

Looping over $(ls) or $(cat)

Covered above, but it's common enough to repeat with the error you'll actually see:

for f in $(ls logs); do wc -l "logs/$f"; done
3 logs/api.log
wc: logs/batch: No such file or directory
wc: logs/run.log: No such file or directory
0 logs/scheduler.log
3 logs/worker.log

The glob version handles the space correctly:

for f in logs/*; do wc -l "$f"; done
3 logs/api.log
1 logs/batch run.log
0 logs/scheduler.log
3 logs/worker.log

Summary

  • for x in list loops over words; for f in dir/* is the safe way to loop over files. Turn on shopt -s nullglob so a glob that matches nothing produces no iterations instead of a literal *.
  • Brace ranges like {01..12} can't use variables; use for ((i = 1; i <= n; i++)) or seq when the bounds are variables. Never loop over $(cat file) or $(ls).
  • while runs while its condition command succeeds, until while it fails. continue skips an iteration, break leaves the loop, break 2 leaves two nested loops. Every polling loop needs a limit.
  • Read lines with while IFS= read -r line; do … done < file. Add || [[ -n $line ]] for files without a final newline, and IFS=, with several variable names to split fields.
  • A pipe into while runs the loop in a subshell, so variables set inside vanish. Feed the loop with done < <(command) instead. Commands inside the loop that read stdin steal its lines; read the file on descriptor 3 (read -r line <&3 … done 3< file).
  • read -p prompts, -s hides input, -t times out with a status above 128, -n 1 takes a single key. select builds numbered menus for interactive tools.
  • In functions, make every variable local, and declare it on a separate line when you need the exit status of a command substitution. Use return for success or failure, stdout for data, and stderr for log messages.
  • Structure scripts as settings, then functions, then main, ending with main "$@".

Exercises

Exercise 1: Summarise every log fileeasy

Write log-summary.sh [DIR] that prints, for each .log file in DIR (default: the current directory), its line count and the number of lines containing ERROR. Empty files should be reported as empty without running wc or grep. If there are no .log files at all, print no .log files in DIR.

Hint

printf '%-22s %4d lines\n' … lines up columns. With nullglob on, you can set a flag inside the loop to know whether it ran at all.

Show Solution
#!/usr/bin/env bash
# log-summary.sh [DIR] - lines and ERROR count for every .log file in DIR
shopt -s nullglob
dir=${1:-.}
found=0
 
for f in "$dir"/*.log; do
  found=1
  if [[ ! -s $f ]]; then
    echo "$f: empty"
    continue
  fi
  printf '%-22s %4d lines %3d errors\n' "$f" "$(wc -l < "$f")" "$(grep -c ERROR "$f")"
done
 
if [[ $found -eq 0 ]]; then
  echo "no .log files in $dir"
fi
./log-summary.sh logs
./log-summary.sh empty
logs/api.log              3 lines   0 errors
logs/batch run.log        1 lines   0 errors
logs/scheduler.log: empty
logs/worker.log           3 lines   1 errors
no .log files in empty

wc -l < "$f" (input redirection) prints only the number; wc -l "$f" would also print the name. The file with a space in its name works because "$f" is quoted everywhere.

Exercise 2: Active members of a teammedium

Using users.csv from this article, write team-emails.sh TEAM [CSV] that prints the email address of every active user in TEAM, one per line, then prints N active member(s) in TEAM to stderr. The count must be correct, so think about where the loop runs. Skip the header. With no arguments, print a usage message and exit 2.

Show Solution
#!/usr/bin/env bash
# team-emails.sh TEAM [CSV] - print the emails of active members of TEAM
if [[ $# -lt 1 ]]; then
  echo "Usage: team-emails.sh TEAM [CSV]" >&2
  exit 2
fi
team=$1
csv=${2:-users.csv}
count=0
 
{
  read -r _header
  while IFS=, read -r username email user_team active; do
    if [[ $user_team == "$team" && $active == yes ]]; then
      echo "$email"
      count=$((count + 1))
    fi
  done
} < "$csv"
 
echo "$count active member(s) in $team" >&2
./team-emails.sh platform
./team-emails.sh payments
./team-emails.sh; echo "exit $?"
asha.k@example.com
chen.wei@example
2 active member(s) in platform
farah.n@example.com
1 active member(s) in payments
Usage: team-emails.sh TEAM [CSV]
exit 2

The { … } < "$csv" group runs in the current shell, so count survives the loop, and the first read consumes the header from the same input. "$team" is quoted on the right of == so a team name containing * would be compared literally rather than as a pattern.

Exercise 3: Wait for a file, with a timeoutmedium

Write a function wait_for_file PATH TIMEOUT that checks once a second for PATH to exist. It prints PATH appeared after Ns and returns 0 when it does, or prints an error to stderr and returns 1 after TIMEOUT seconds. Test it with a file that a background job creates after 2 seconds, and with one that never appears.

Show Solution
#!/usr/bin/env bash
# wait_for_file PATH TIMEOUT_SECONDS - return 0 when PATH exists, 1 on timeout
wait_for_file() {
  local path=$1 timeout=$2 waited=0
  while [[ ! -e $path ]]; do
    if [[ $waited -ge $timeout ]]; then
      echo "wait_for_file: $path did not appear within ${timeout}s" >&2
      return 1
    fi
    sleep 1
    waited=$((waited + 1))
  done
  echo "$path appeared after ${waited}s"
}
 
rm -f backup.done
(sleep 2; touch backup.done) &
wait_for_file backup.done 10
wait_for_file restore.done 3 || echo "giving up on restore"
./wait-for-file.sh
backup.done appeared after 2s
wait_for_file: restore.done did not appear within 3s
giving up on restore

Because it returns a proper status, the caller decides what a timeout means: || echo here, || exit 1 in a deploy script. All three variables are local, so calling it twice in a row doesn't leak waited between calls.

Exercise 4: Rotate a log filehard

Write a function rotate FILE KEEP that rotates a log the way logrotate does: delete FILE.KEEP, rename FILE.(KEEP-1) to FILE.KEEP, and so on down to FILE.1 → FILE.2, then rename FILE to FILE.1 and create a new empty FILE. Return 1 with a message if FILE doesn't exist. Rotate four times with KEEP=3 and confirm that only the three newest copies remain, in the right order.

Hint

The renames must go from the highest number down, or each mv overwrites the copy you're about to move. A C-style loop can count down: for ((i = keep - 1; i >= 1; i--)).

Show Solution
#!/usr/bin/env bash
# rotate FILE KEEP - shift FILE.1 -> FILE.2 ... and FILE -> FILE.1, keeping KEEP copies
rotate() {
  local file=$1 keep=$2 i
  [[ -f $file ]] || { echo "rotate: no such file: $file" >&2; return 1; }
 
  rm -f -- "$file.$keep"                    # the oldest copy falls off the end
  for ((i = keep - 1; i >= 1; i--)); do
    if [[ -e $file.$i ]]; then
      mv -- "$file.$i" "$file.$((i + 1))"
    fi
  done
  mv -- "$file" "$file.1"
  : > "$file"                               # start a fresh, empty log
}
 
rotate "$@"
printf 'first\n' > app.log
./rotate.sh app.log 3; ls app.log*
echo "second" > app.log; ./rotate.sh app.log 3
echo "third" > app.log; ./rotate.sh app.log 3
echo "fourth" > app.log; ./rotate.sh app.log 3
ls app.log*
head app.log.*
./rotate.sh missing.log 3; echo "exit $?"
app.log  app.log.1
app.log  app.log.1  app.log.2  app.log.3
==> app.log.1 <==
fourth
 
==> app.log.2 <==
third
 
==> app.log.3 <==
second
rotate: no such file: missing.log
exit 1

first was rotated out on the fourth run, exactly as intended. : > "$file" truncates or creates a file using the no-op command :. A real logging application keeps its old file open after the rename, which is why logrotate either signals the application to reopen its log or uses copytruncate; this exercise covers the file shuffling, not that coordination.


Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), November 2026. Every loop, script and exercise above was run as shown; the interactive examples (read, select, the restart prompt) were typed into a real terminal session, and the retry test ran against a real python3 -m http.server listener. The useradd commands are printed rather than executed, because creating accounts needs root, which the test machine didn't have.

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.