Redirection and Pipes
How Bash wires stdin, stdout and stderr: send output to files, merge or discard errors, feed input with here-documents, and build pipelines that fail loudly.
By The Dark DebuggerUpdated 31 min read
Every command you run in Bash reads from one place and writes to two others, and
redirection is how you decide what those places are. This article covers the three
standard streams, every redirection operator you will meet in real scripts,
here-documents and here-strings, pipes, tee, pipeline exit status and process
substitution. It assumes you can move around the filesystem and are comfortable with
variables and quoting; the next article,
command substitution, chaining, and exit codes,
builds directly on it.
Why redirection matters
Picture a nightly backup job written as one line:
tar -cf - ./app-data | gzip > backup.tar.gzOne night the app-data directory is missing because a deploy moved it. Here is what
happens when I run that line with the directory gone:
tar: ./app-data: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errorsecho "exit status: $?"
ls -l backup.tar.gzexit status: 0
-rw-r--r-- 1 sid sid 45 Nov 9 21:26 backup.tar.gzExit status 0 means "success", so the cron job reports nothing. The backup file exists, so a check like "does the file exist?" passes too. It is a 45-byte gzip of nothing. You find out the day you need to restore.
Three things went wrong, and each one is a redirection topic: the error messages went to
a terminal nobody was watching (stderr), the exit status of the pipeline only reflected
the last command (gzip), and > happily created an empty archive. By the end of this
article you will know how to fix all three.
The three standard streams
When a program starts, it inherits three open file descriptors (small integers the kernel uses to refer to open files):
| FD | Name | Default in a terminal | Used for |
|---|---|---|---|
| 0 | stdin | keyboard | input the program reads |
| 1 | stdout | terminal | normal output (the data) |
| 2 | stderr | terminal | errors, warnings, progress messages |
You can see them. On Linux, /proc/self/fd lists the file descriptors of the process
reading it. Run in a terminal, all three point at the terminal device:
ls -l /proc/self/fdtotal 0
lrwx------ 1 sid sid 64 Nov 9 21:26 0 -> /dev/pts/4
lrwx------ 1 sid sid 64 Nov 9 21:26 1 -> /dev/pts/4
lrwx------ 1 sid sid 64 Nov 9 21:26 2 -> /dev/pts/4
lr-x------ 1 sid sid 64 Nov 9 21:26 3 -> /proc/8591/fd(FD 3 is the directory ls opened to read the listing.) Now the same command with its
streams rearranged:
ls -l /proc/self/fd < /dev/null 2> /dev/null | cattotal 0
lr-x------ 1 sid sid 64 Nov 9 21:26 0 -> /dev/null
l-wx------ 1 sid sid 64 Nov 9 21:26 1 -> pipe:[40830]
l-wx------ 1 sid sid 64 Nov 9 21:26 2 -> /dev/null
lr-x------ 1 sid sid 64 Nov 9 21:26 3 -> /proc/8597/fdThat is the whole idea of redirection in one picture. ls did nothing differently; the
shell pointed FD 0 at /dev/null, FD 1 at a pipe, and FD 2 at /dev/null before
starting it. Programs just write to "FD 1" and "FD 2" and never know where the bytes
end up.
The split between stdout and stderr is what lets you capture data without capturing
noise. Watch what a pipe carries when ls gets one good path and one bad one:
ls -l /etc/hostname /etc/nonexistent | wc -lls: cannot access '/etc/nonexistent': No such file or directory
1The error went straight to the terminal; only the one line of real output went
through the pipe to wc -l. That is by design, and it is the behaviour you want most
of the time.
Writing output to files: > and >>
> sends stdout to a file. It creates the file if needed and truncates it to zero
length first if it exists:
echo "deploy started" > deploy.log
echo "migrations applied" > deploy.log
cat deploy.logmigrations appliedThe first line is gone. (Typing > where you meant >> is an easy way to lose a log
file.) >> appends instead:
echo "deploy started" > deploy.log
echo "migrations applied" >> deploy.log
echo "deploy finished" >> deploy.log
cat deploy.logdeploy started
migrations applied
deploy finishedA redirection with no command just opens the file, which is the shortest way to create an empty file or empty an existing log without deleting it (useful when a running program still has the file open):
> empty.txt
ls -l empty.txt-rw-r--r-- 1 sid sid 0 Nov 9 21:26 empty.txtThe truncation happens when the shell sets up the redirection, before the command runs. Keep that sentence in mind; it explains the most destructive mistake in this article (see Common mistakes).
Redirecting errors: 2> and 2>>
Put a file descriptor number in front of > to redirect that descriptor. > on its
own is shorthand for 1>. So 2> redirects stderr:
ls /etc/hostname /etc/nonexistent 2> errors.txt
cat errors.txt/etc/hostname
ls: cannot access '/etc/nonexistent': No such file or directoryThe first line is stdout (still on the terminal); the second came from cat errors.txt.
You can redirect both streams to different files in one command:
ls /etc/hostname /etc/nonexistent > found.txt 2> errors.txt
cat found.txt errors.txt/etc/hostname
ls: cannot access '/etc/nonexistent': No such file or directoryThe everyday use is getting clean results out of a command that complains a lot.
find over /etc as a normal user hits directories it may not read:
find /etc -name '*.conf' | wc -lfind: ‘/etc/credstore’: Permission denied
find: ‘/etc/ssl/private’: Permission denied
find: ‘/etc/polkit-1/rules.d’: Permission denied
find: ‘/etc/credstore.encrypted’: Permission denied
133Notice the count was right both times: the errors never entered the pipe. They only
clutter the screen. To hide them, send FD 2 to /dev/null:
find /etc -name '*.conf' 2> /dev/null | wc -l133Be deliberate about 2> /dev/null. It hides every error, including the one that
tells you why the result is wrong. In scripts I prefer 2>> errors.log so the messages
are still somewhere when you need them. (find itself is covered in
finding files.)
In your own scripts, error messages belong on stderr too. >&2 is shorthand for
1>&2: "send this command's stdout wherever stderr goes":
echo "error: config file not found" >&2error: config file not foundIt looks identical on screen, but anyone running yourscript > out.txt will still see
the error instead of finding it buried in the output file.
Sending both streams to one place
2>&1 means "make FD 2 a copy of whatever FD 1 is right now". Combined with >, it
puts everything in one file, which is what you want for a job log:
ls /etc/hostname /etc/nonexistent > all.txt 2>&1
cat all.txtls: cannot access '/etc/nonexistent': No such file or directory
/etc/hostnameWhy the order matters
Redirections are processed left to right, and 2>&1 copies FD 1's destination at
that moment. Swap the order and you get a different result:
ls /etc/hostname /etc/nonexistent 2>&1 > all.txt
cat all.txtls: cannot access '/etc/nonexistent': No such file or directory
/etc/hostnameThe error printed on the terminal and only /etc/hostname landed in the file. Step by
step:
2>&1: FD 1 still points at the terminal, so FD 2 now points at the terminal too (it already did; nothing visible changes).> all.txt: FD 1 now points at the file. FD 2 is not affected; it was a copy, not a link.
Read 2>&1 as "FD 2 goes where FD 1 goes now", not "FD 2 follows FD 1 forever", and
the order rule stops being something to memorise.
This same left-to-right rule gives you a useful trick: pipe only the errors and throw the normal output away. Point FD 2 at the pipe first, then point FD 1 elsewhere:
ls /etc/hostname /etc/nonexistent 2>&1 > /dev/null | wc -l1Shorthands: &>, &>> and |&
Bash has shorter spellings for the common cases:
ls /etc/hostname /etc/nonexistent &> all.txt
ls /etc/hostname /etc/nonexistent |& wc -l2&> file is the same as > file 2>&1, &>> file appends both streams, and
cmd1 |& cmd2 is the same as cmd1 2>&1 | cmd2. They are Bash-specific; if a script has to run
under /bin/sh (dash on Debian and Ubuntu), use the long forms.
Discarding output with /dev/null
/dev/null is a device file that accepts any amount of data and throws it away.
Reading from it returns end-of-file immediately:
wc -c < /dev/null0The classic use is asking a question where only the exit status matters. command -v
(covered in the next article) prints the path of a
command if it exists; we don't want the path, just the yes/no:
command -v tar > /dev/null && echo "tar is installed"
command -v kubectl > /dev/null || echo "kubectl is missing"tar is installed
kubectl is missing< /dev/null is useful too: it guarantees a command can't sit waiting for keyboard
input, which matters in cron jobs and CI where nobody is there to type.
Protecting files with noclobber
If > truncating files makes you nervous, turn on noclobber. With it set, > refuses
to overwrite an existing file:
set -o noclobber
echo "v2" > deploy.logbash: deploy.log: cannot overwrite existing fileWhen you really do mean to overwrite, >| forces it. Appending with >> is still
allowed:
echo "v2" >| deploy.log
echo "more" >> deploy.log
cat deploy.logv2
moreset +o noclobber turns it off again. Some people put set -o noclobber in their
~/.bashrc; I think it is a reasonable safety net for interactive shells, but don't rely
on it in scripts, because a script run by someone else won't have your settings.
Reading input from a file: <
< connects a file to a command's stdin. Many commands accept a filename argument
anyway, so the difference is subtle but real. Compare:
wc -l /etc/passwd
wc -l < /etc/passwd28 /etc/passwd
28With <, wc never sees a filename; it just reads bytes from FD 0. That is handy when
you want only the number, for example in count=$(wc -l < file).
Some commands only read stdin. tr, which translates characters, is the one
everyone trips over:
tr '[:lower:]' '[:upper:]' deploy.logtr: extra operand ‘deploy.log’
Try 'tr --help' for more information.tr '[:lower:]' '[:upper:]' < deploy.logV2
MOREThe other place you will use < constantly is feeding a file to a while read loop,
one line at a time:
printf '%s\n' web01 web02 db01 > hosts.txt
while read -r host; do echo "checking $host"; done < hosts.txtchecking web01
checking web02
checking db01Loops get their own article, loops, functions, and reading input.
Here-documents: multi-line input inline
A here-document feeds several lines of text to a command's stdin, written right in
the script. Everything between <<WORD and a line containing only WORD becomes the
input:
cat <<EOF
Report for user: $USER
Home directory: $HOME
Generated on: $(date +%F)
EOFReport for user: sid
Home directory: /home/sid
Generated on: 2026-11-09EOF is only a convention; any word works. With an unquoted delimiter, the body is
treated like a double-quoted string: variables, $(...) command substitution and
backslash escapes are expanded.
Quoted delimiter: no expansion
Quote the delimiter (any part of it, 'EOF' is the usual style) and the body is passed
through literally:
cat <<'EOF'
Report for user: $USER
Home directory: $HOME
Generated on: $(date +%F)
EOFReport for user: $USER
Home directory: $HOME
Generated on: $(date +%F)My rule: quote the delimiter by default, and only drop the quotes when you actually
want expansion. Use the quoted form whenever the body is itself code (a script, SQL,
an awk program) that contains $.
Mixing expanded and literal $
Config files often need both: your shell variables filled in, and the target program's
own $variables left alone. Use an unquoted delimiter and escape the dollars that must
survive:
domain=shop.example.com
cat > site.conf <<EOF
server {
listen 80;
server_name $domain;
location / {
try_files \$uri \$uri/ =404;
}
}
EOF
cat site.confserver {
listen 80;
server_name shop.example.com;
location / {
try_files $uri $uri/ =404;
}
}cat > file <<EOF is the standard way to write a whole file from a script. Notice the
redirections combine: > sends stdout to site.conf, <<EOF feeds stdin.
Indented here-documents: <<-
Inside a function or if block you want the body indented with the rest of the code.
<<- strips leading tab characters (not spaces) from each body line and from the
closing delimiter. Here is a script where the indentation is real tabs; cat -A shows
them as ^I:
cat -A motd.sh#!/usr/bin/env bash$
print_banner() {$
^Icat <<-EOF$
^IMaintenance window: Saturday 02:00-04:00 UTC$
^IContact: ops@example.com$
^IEOF$
}$
print_banner$bash motd.shMaintenance window: Saturday 02:00-04:00 UTC
Contact: ops@example.comIf your editor converts tabs to spaces, <<- silently stops working; the result is in
Common mistakes. Copying the example from a web page usually turns
tabs into spaces too, so type the tabs yourself.
Feeding other programs
Anything that reads stdin can take a here-document: psql, mysql, ssh host bash,
or an interpreter. The - tells Python to read the program from stdin:
python3 - <<'EOF'
import platform
print("python", platform.python_version())
EOFpython 3.12.3Here-strings: one line of input
A here-string, <<< word, feeds a single string (plus a trailing newline) to stdin.
It is the tidy way to hand a variable to a command that only reads stdin:
read -r method path proto <<< "POST /api/orders HTTP/1.1"
echo "$path"/api/ordersIFS=, read -r id date region <<< "1001,2026-11-01,north"
echo "order $id shipped to $region"order 1001 shipped to northNote the trailing newline Bash adds. It is usually what you want, but it does count:
wc -c <<< "abc"4Always quote the word (<<< "$var") for the same reasons you quote any expansion; see
variables and quoting.
Pipes: connecting commands
A pipe, cmd1 | cmd2, connects stdout of cmd1 to stdin of cmd2. This is the
mechanism behind the Unix philosophy: small programs that each do one job, reading
text and writing text, combined on the command line into something bigger.
Here is a pipeline built one stage at a time: which login shells are configured on
this machine, and how many accounts use each? Field 7 of /etc/passwd is the shell:
cut -d: -f7 /etc/passwd | head -n 4/bin/bash
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologinGroup identical lines and count them (uniq needs sorted input):
cut -d: -f7 /etc/passwd | sort | uniq -c 2 /bin/bash
1 /bin/false
1 /bin/sync
24 /usr/sbin/nologinSort by the count, largest first:
cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn 24 /usr/sbin/nologin
2 /bin/bash
1 /bin/sync
1 /bin/falseBuild pipelines exactly like this: run the first stage, look at the output, add one stage, look again. Every tool used here gets a full treatment in searching and filtering text.
All stages run at the same time
A pipeline is not "run A, save the output, then run B". The shell starts every stage at
once and the kernel moves data between them through a small buffer. Two sleep 2
commands in a pipeline take two seconds, not four:
time (sleep 2 | sleep 2)
real 0m2.005s
user 0m0.002s
sys 0m0.001sThis is also why yes | head -n 3 finishes even though yes prints forever: when
head has its three lines it exits, and the next write by yes kills it with
SIGPIPE.
yes "y" | head -n 3y
y
yStreaming means pipelines work on files far larger than memory, and you see the first results before the last line has been read.
Each stage runs in a subshell
Each part of a pipeline runs in a child process (a subshell). Variables set inside a
pipeline stage vanish when the stage ends. This catches everyone who pipes into a
while read loop; it is in Common mistakes, and
the next article explains subshells properly.
tee: save a copy and keep going
tee reads stdin and writes it both to stdout and to one or more files, like a T-shaped
pipe fitting. Use it to keep a copy of what flows through a pipeline:
ls -l /etc/*.conf | tee conf-list.txt | wc -l
head -n 3 conf-list.txt26
-rw-r--r-- 1 root root 3444 Jul 5 2023 /etc/adduser.conf
-rw-r--r-- 1 root root 6862 Jun 17 08:34 /etc/ca-certificates.conf
-rw-r--r-- 1 root root 2967 Apr 12 2024 /etc/debconf.confDropping a tee into the middle of a pipeline is also the best way to debug one: you
see the final result and get a snapshot of an intermediate stage:
cut -d: -f7 /etc/passwd | sort | tee shells-sorted.txt | uniq -c | sort -rn | head -n 2
wc -l shells-sorted.txt 24 /usr/sbin/nologin
2 /bin/bash
28 shells-sorted.txttee -a appends instead of truncating, so some-command | tee -a job.log shows output
live and logs it. Add > /dev/null after tee when you want only the file.
Writing to root-owned files: the sudo tee pattern
This is the redirection gotcha that confuses people for years. Say you want to add a
line to /etc/hosts, which only root can write:
echo "192.0.2.10 db.internal" >> /etc/hostsbash: /etc/hosts: Permission deniedReasonable next try:
sudo echo "192.0.2.10 db.internal" >> /etc/hostsbash: /etc/hosts: Permission deniedSame error, and notice it says bash:, not sudo:. The redirection belongs to your
shell, which opens /etc/hosts as you, fails, and never even runs sudo. Only echo
would have been elevated. The fix is to make the process that opens the file the
privileged one, and tee is a program that opens files:
echo "192.0.2.10 db.internal" | sudo tee -a /etc/hosts > /dev/nullsudo tee -a opens the file as root and appends; > /dev/null stops tee echoing the
line back at you. (I could not run this last command in my test environment, which has
no sudo password; the two failures above are real output.) See
file permissions and ownership for why root owns these
files in the first place.
Pipeline exit status: PIPESTATUS and pipefail
Every command returns an exit status: 0 for success, anything else for failure (the next article covers this in depth). A pipeline's status is, by default, the status of its last command only:
false | true
echo $?
true | false
echo $?0
1That default is how the backup at the top of this article "succeeded". It also bites in reporting pipelines. Counting errors in a log that doesn't exist:
grep ERROR app.log | wc -l
echo "exit status: $?"grep: app.log: No such file or directory
0
exit status: 0A monitoring script would read that as "zero errors, all good".
PIPESTATUS: every stage's status
Bash records the status of each command of the most recent pipeline in the array
PIPESTATUS. Read it immediately, because the next command overwrites it:
grep ERROR app.log | sort | wc -l
echo "${PIPESTATUS[@]}"grep: app.log: No such file or directory
0
2 0 0grep returned 2 (error), sort and wc returned 0.
set -o pipefail: make the pipeline fail
With pipefail on, a pipeline's status is the status of the last command that
failed (or 0 if all succeeded):
set -o pipefail
grep ERROR app.log | sort | wc -l
echo "exit status: $?"grep: app.log: No such file or directory
0
exit status: 2And the backup from the start of the article now reports what happened:
set -o pipefail
tar -cf - ./app-data | gzip > backup.tar.gz
echo "exit status: $?"tar: ./app-data: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
exit status: 2Put set -o pipefail at the top of every script you write. It is part of the
set -euo pipefail header covered in
error handling, traps, and debugging.
The pipefail surprise: exit status 141
pipefail has one sharp edge. When a downstream command stops reading early (head,
grep -q, grep -m 1), the upstream command is killed by SIGPIPE, and that counts as
a failure:
set -o pipefail
seq 1 1000000 | head -n 1
echo "exit status: $? PIPESTATUS: ${PIPESTATUS[*]}"1
exit status: 141 PIPESTATUS: 141 0141 is 128 + 13, and 13 is the signal number of SIGPIPE. The output was correct; the
status says otherwise. With pipefail off, the same pipeline returns 0. It only happens
when the upstream command still has output to write after the reader exits, so it
depends on data size, which makes it a nasty intermittent failure in scripts. When you
see 141 in a pipefail script, look for a head or grep -q downstream; the usual
fixes are to let the reader consume everything (grep -c instead of grep -q, awk 'NR==1' instead of head -n 1) or to check PIPESTATUS yourself for that pipeline.
Process substitution: <( ) and >( )
Some commands insist on filenames, not stdin. diff compares two files; you can't
pipe two things into one stdin. Process substitution runs a command and hands its output
to another command as if it were a file:
echo <(true)/dev/fd/63That path is what the receiving command sees: a file descriptor the shell connected to the inner command's output. Here it compares two unsorted server lists without creating temporary sorted copies:
printf '%s\n' web01 web02 db01 db02 cache01 > inventory.txt
printf '%s\n' db01 web02 web01 lb01 > monitored.txt
diff <(sort inventory.txt) <(sort monitored.txt)1d0
< cache01
3c2
< db02
---
> lb01cache01 and db02 are in the inventory but not monitored; lb01 is monitored but not
in the inventory.
>( ) works the other way: it gives a command a "file" to write to, and whatever is
written there becomes stdin of the inner command. Combined with tee, it lets one pass
over the data feed several consumers. This compresses a log and counts its 404s in one
read:
tee >(gzip > access.log.gz) < access.log | grep -c ' 404 '9One caveat: the command inside >( ) runs in the background, and Bash does not wait
for it. If the next line of your script uses access.log.gz, it may not be completely
written yet. For anything a later step depends on, use a plain pipeline or a temporary
file.
Process substitution is a Bash feature (not POSIX sh) and needs /dev/fd, which every
Linux and macOS system has.
Opening your own file descriptors with exec
exec with only redirections (no command) changes the current shell's own file
descriptors from that point on. Two patterns appear in real scripts.
Send everything the rest of the script prints to a log:
exec >> job.log 2>&1And open an extra descriptor (3 to 9 are free for you) for a file you write to many
times, then close it with 3>&-:
exec 3>> audit.log
echo "user sid rotated logs" >&3
exec 3>&-
echo "user sid restarted nginx" >&3bash: 3: Bad file descriptorThe last line fails because FD 3 is closed, which proves the close worked. Opening the
file once is cheaper than >> audit.log on every line and, more usefully, keeps the
log separate from stdout and stderr.
Quick reference
| Syntax | Meaning |
|---|---|
cmd > file | stdout to file (truncate first) |
cmd >> file | stdout appended to file |
cmd 2> file | stderr to file |
cmd > file 2>&1 | stdout and stderr to file (order matters) |
cmd &> file / cmd &>> file | same, Bash shorthand (truncate / append) |
cmd >&2 | stdout to wherever stderr goes |
cmd < file | stdin from file |
cmd <<EOF / <<'EOF' | here-document (expanded / literal) |
cmd <<-EOF | here-document with leading tabs stripped |
cmd <<< "$var" | here-string |
cmd1 | cmd2 | stdout of cmd1 to stdin of cmd2 |
cmd1 |& cmd2 | stdout and stderr of cmd1 to cmd2 |
cmd >| file | overwrite even with noclobber set |
<(cmd) / >(cmd) | process substitution (read from / write to) |
exec 3> file, exec 3>&- | open / close a file descriptor in the current shell |
Where you'll use this
Redirection is in almost every script you will write. The common jobs:
- Cron jobs have no terminal. Anything not redirected is mailed to the user or lost,
so the standard crontab line ends with
>> /path/to/job.log 2>&1. Scheduling is covered in system monitoring and scheduling. - Generating config files in provisioning scripts and Dockerfiles with
cat > file <<'EOF'. - Reports and audits built from pipelines, saved with
teeso a human sees them and a file keeps them. - CI pipelines, where
set -o pipefailis the difference between a red build and a green build that shipped nothing.
Here is a complete script that uses most of this article: it archives a log directory,
logs everything it does, detects failure anywhere in the tar | gzip pipeline, and
writes a summary with a here-document.
#!/usr/bin/env bash
# backup-logs.sh - archive a log directory; built to run unattended from cron.
set -o pipefail
src=${1:-./logs}
dest=./backups
archive="$dest/logs-$(date +%Y%m%d-%H%M%S).tar.gz"
partial="$archive.partial"
joblog=./backup-job.log
mkdir -p "$dest"
# From here on, stdout and stderr of every command go to the job log.
exec >> "$joblog" 2>&1
echo "[$(date '+%F %T')] backing up $src"
# Write to a temporary name; only a complete archive gets the real name.
if tar -cf - "$src" | gzip > "$partial"; then
mv "$partial" "$archive"
echo "[$(date '+%F %T')] OK: $archive ($(du -h "$archive" | cut -f1))"
else
echo "[$(date '+%F %T')] FAILED: tar or gzip failed, removing $partial"
rm -f "$partial"
exit 1
fi
cat > "$dest/LATEST.txt" <<EOF
archive: $archive
source: $src
entries: $(tar -tzf "$archive" | wc -l)
EOFRun it once normally and once against a directory that doesn't exist:
./backup-logs.sh
echo "exit: $?"
./backup-logs.sh ./no-such-dir
echo "exit: $?"exit: 0
exit: 1Nothing appeared on the terminal except our own echo lines, because the script sent
all of its output to the job log:
cat backup-job.log[2026-11-09 21:28:00] backing up ./logs
[2026-11-09 21:28:00] OK: ./backups/logs-20261109-212800.tar.gz (4.0K)
[2026-11-09 21:28:00] backing up ./no-such-dir
tar: ./no-such-dir: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
[2026-11-09 21:28:00] FAILED: tar or gzip failed, removing ./backups/logs-20261109-212800.tar.gz.partialls backups
cat backups/LATEST.txtLATEST.txt
logs-20261109-212800.tar.gz
archive: ./backups/logs-20261109-212800.tar.gz
source: ./logs
entries: 3The .partial step is there because of something that happened while I tested this.
My first version wrote straight to $archive and deleted it on failure. Both test runs
happened in the same second, so they computed the same archive name: the failing run's
gzip > "$archive" truncated the good archive from the first run, and its cleanup then
deleted it. Writing to a temporary name and renaming only on success means a failed run
can never damage a good file. mv within one filesystem is a rename, so the real name
only ever points at a complete archive.
In a crontab the script would be scheduled like this (the cron article walks through installing it):
30 2 * * * /home/sid/bin/backup-logs.sh /var/log/shopBecause the script redirects its own output with exec, the crontab line doesn't need
>> log 2>&1, though adding it costs nothing and catches errors that happen before the
exec line.
Common mistakes
Sorting a file into itself
printf '%s\n' web02 db01 web01 > servers.txt
sort servers.txt > servers.txt
wc -c servers.txt0 servers.txtThe file is now empty. The shell processed > servers.txt first, truncating the file,
and only then started sort, which read an empty file. The same happens with grep,
sed, cut or anything else when the input and output are the same file. Write to a
new file and rename it, or use a tool's own output option:
printf '%s\n' web02 db01 web01 > servers.txt
sort -o servers.txt servers.txt
cat servers.txtdb01
web01
web02sort -o reads all its input before opening the output file, so it is safe. For other
commands, cmd file > file.tmp && mv file.tmp file is the portable pattern.
A here-document that never ends
If the closing delimiter has leading spaces (with <<), trailing spaces, or the body is
indented with spaces under <<-, Bash doesn't recognise the end marker and keeps
reading to the end of the file. This script is indented with four spaces:
cat banner-spaces.sh#!/usr/bin/env bash
print_banner() {
cat <<-EOF
Maintenance window: Saturday 02:00-04:00 UTC
EOF
}
print_banner
echo "script finished"bash banner-spaces.sh; echo "exit: $?"banner-spaces.sh: line 8: warning: here-document at line 3 delimited by end-of-file (wanted `EOF')
banner-spaces.sh: line 9: syntax error: unexpected end of file
exit: 2The here-document swallowed the closing } and everything after it, so the function
was never closed. Fix it by indenting with tabs (<<- only strips tabs) or by putting
the closing EOF at column 0 with nothing after it. cat -A shows tabs as ^I and
line ends as $, which makes stray whitespace visible.
Piping into while read and losing the variable
last=none
printf '%s\n' web01 web02 db01 | while read -r host; do last=$host; done
echo "last=$last"last=noneThe loop ran in a subshell (it is a pipeline stage), set its own copy of last, and
exited. Feed the loop with process substitution instead, so the loop runs in the
current shell:
while read -r host; do last=$host; done < <(printf '%s\n' web01 web02 db01)
echo "last=$last"last=db01The space in < <( matters: it is a < redirection whose source is a <( ) process
substitution.
Summary
- Every process has stdin (0), stdout (1) and stderr (2). Redirection changes where those descriptors point before the command starts; the command never knows.
>truncates first,>>appends,2>redirects errors,<supplies input. Never redirect output to a file that the same command is reading.- Redirections are processed left to right:
> file 2>&1captures both streams,2>&1 > filedoes not.&>and|&are Bash shorthands. - Here-documents feed multi-line input; quote the delimiter (
<<'EOF') to stop expansion, use<<-only with tab indentation. Here-strings (<<<) feed one string. - A pipeline's stages run concurrently in subshells; only stdout flows through
|. - By default a pipeline's exit status is its last command's. Use
set -o pipefailin scripts, readPIPESTATUSright after the pipeline, and expect 141 whenheadorgrep -qcloses a pipe early. sudo cmd > filefails because your shell opens the file; use| sudo tee file.<(cmd)turns a command's output into a filename for tools likediffandcomm.
Exercises
Run ls on /etc/hostname /etc/passwd /etc/nope /var/nothing. Send the paths that
exist to found.txt and the error messages to missing.txt, then count the lines in
each. Finally, run the same ls again, appending both streams to a single all.log.
Show Solution
ls /etc/hostname /etc/passwd /etc/nope /var/nothing > found.txt 2> missing.txt
wc -l found.txt missing.txt 2 found.txt
2 missing.txt
4 totalls /etc/hostname /etc/passwd /etc/nope /var/nothing >> all.log 2>&1
cat all.logls: cannot access '/etc/nope': No such file or directory
ls: cannot access '/var/nothing': No such file or directory
/etc/hostname
/etc/passwd>> all.log 2>&1 appends stdout, then points stderr at the same place. &>> all.log
does the same in Bash.
Using shell variables app=shop and retention=14, write a here-document that creates
shop.logrotate with this content. The app name and retention must come from the
variables, but $(cat /run/nginx.pid) must appear literally in the file:
/var/log/shop/*.log {
daily
rotate 14
compress
missingok
postrotate
kill -USR1 $(cat /run/nginx.pid)
endscript
}Hint
An unquoted delimiter expands everything; a quoted one expands nothing. You need the unquoted form plus one backslash.
Show Solution
app=shop
retention=14
cat > "$app.logrotate" <<EOF
/var/log/$app/*.log {
daily
rotate $retention
compress
missingok
postrotate
kill -USR1 \$(cat /run/nginx.pid)
endscript
}
EOF
cat shop.logrotate/var/log/shop/*.log {
daily
rotate 14
compress
missingok
postrotate
kill -USR1 $(cat /run/nginx.pid)
endscript
}Escaping the $ as \$ stops the command substitution from running now; logrotate's
own shell runs it later.
The pipeline grep ' 500 ' logs/missing.log | sort | head -n 1 reads a file that does
not exist. Print the exit status of every stage, then make the pipeline as a whole
report the failure.
Show Solution
grep ' 500 ' logs/missing.log | sort | head -n 1
echo "${PIPESTATUS[@]}"grep: logs/missing.log: No such file or directory
2 0 0set -o pipefail
grep ' 500 ' logs/missing.log | sort | head -n 1
echo "pipeline exit: $?"grep: logs/missing.log: No such file or directory
pipeline exit: 2PIPESTATUS must be read on the very next command. Without pipefail, $? would be
0, the status of head.
inventory.txt lists the servers you own and monitored.txt lists the servers your
monitoring system watches. Neither file is sorted. Using comm (which needs sorted
input and prints lines unique to file 1, unique to file 2, and common lines in three
columns) and process substitution, print (a) servers that are not monitored, and (b)
monitored servers that are not in the inventory.
printf '%s\n' web01 web02 db01 db02 cache01 > inventory.txt
printf '%s\n' db01 web02 web01 lb01 > monitored.txtHint
comm -23 a b hides columns 2 and 3, leaving lines only in a.
Show Solution
comm -23 <(sort inventory.txt) <(sort monitored.txt)cache01
db02comm -13 <(sort inventory.txt) <(sort monitored.txt)lb01Each <(sort ...) becomes a /dev/fd/N path that comm opens like a file, so the
sorted copies never touch the disk.
Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), November 2026. Every command
was run as shown and the output pasted unchanged; bash's own error messages were
captured from an interactive shell so they read as they do at a prompt. The
sudo tee command was not run because the test account has no sudo password (the
Permission denied failures before it are real), and the crontab line was not
installed. The exit status 141 example depends on the pipe buffer filling, which
seq 1 1000000 reliably does.
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.