Redirection and Pipes

How Bash wires stdin, stdout and stderr: send output to files, merge or discard errors, feed input with here-documents, and build pipelines that fail loudly.

By The Dark DebuggerUpdated 31 min read

Every command you run in Bash reads from one place and writes to two others, and redirection is how you decide what those places are. This article covers the three standard streams, every redirection operator you will meet in real scripts, here-documents and here-strings, pipes, tee, pipeline exit status and process substitution. It assumes you can move around the filesystem and are comfortable with variables and quoting; the next article, command substitution, chaining, and exit codes, builds directly on it.

Why redirection matters

Picture a nightly backup job written as one line:

tar -cf - ./app-data | gzip > backup.tar.gz

One night the app-data directory is missing because a deploy moved it. Here is what happens when I run that line with the directory gone:

tar: ./app-data: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
echo "exit status: $?"
ls -l backup.tar.gz
exit status: 0
-rw-r--r-- 1 sid sid 45 Nov  9 21:26 backup.tar.gz

Exit status 0 means "success", so the cron job reports nothing. The backup file exists, so a check like "does the file exist?" passes too. It is a 45-byte gzip of nothing. You find out the day you need to restore.

Three things went wrong, and each one is a redirection topic: the error messages went to a terminal nobody was watching (stderr), the exit status of the pipeline only reflected the last command (gzip), and > happily created an empty archive. By the end of this article you will know how to fix all three.

The three standard streams

When a program starts, it inherits three open file descriptors (small integers the kernel uses to refer to open files):

FDNameDefault in a terminalUsed for
0stdinkeyboardinput the program reads
1stdoutterminalnormal output (the data)
2stderrterminalerrors, warnings, progress messages

You can see them. On Linux, /proc/self/fd lists the file descriptors of the process reading it. Run in a terminal, all three point at the terminal device:

ls -l /proc/self/fd
total 0
lrwx------ 1 sid sid 64 Nov  9 21:26 0 -> /dev/pts/4
lrwx------ 1 sid sid 64 Nov  9 21:26 1 -> /dev/pts/4
lrwx------ 1 sid sid 64 Nov  9 21:26 2 -> /dev/pts/4
lr-x------ 1 sid sid 64 Nov  9 21:26 3 -> /proc/8591/fd

(FD 3 is the directory ls opened to read the listing.) Now the same command with its streams rearranged:

ls -l /proc/self/fd < /dev/null 2> /dev/null | cat
total 0
lr-x------ 1 sid sid 64 Nov  9 21:26 0 -> /dev/null
l-wx------ 1 sid sid 64 Nov  9 21:26 1 -> pipe:[40830]
l-wx------ 1 sid sid 64 Nov  9 21:26 2 -> /dev/null
lr-x------ 1 sid sid 64 Nov  9 21:26 3 -> /proc/8597/fd

That is the whole idea of redirection in one picture. ls did nothing differently; the shell pointed FD 0 at /dev/null, FD 1 at a pipe, and FD 2 at /dev/null before starting it. Programs just write to "FD 1" and "FD 2" and never know where the bytes end up.

The split between stdout and stderr is what lets you capture data without capturing noise. Watch what a pipe carries when ls gets one good path and one bad one:

ls -l /etc/hostname /etc/nonexistent | wc -l
ls: cannot access '/etc/nonexistent': No such file or directory
1

The error went straight to the terminal; only the one line of real output went through the pipe to wc -l. That is by design, and it is the behaviour you want most of the time.

Writing output to files: > and >>

> sends stdout to a file. It creates the file if needed and truncates it to zero length first if it exists:

echo "deploy started" > deploy.log
echo "migrations applied" > deploy.log
cat deploy.log
migrations applied

The first line is gone. (Typing > where you meant >> is an easy way to lose a log file.) >> appends instead:

echo "deploy started" > deploy.log
echo "migrations applied" >> deploy.log
echo "deploy finished" >> deploy.log
cat deploy.log
deploy started
migrations applied
deploy finished

A redirection with no command just opens the file, which is the shortest way to create an empty file or empty an existing log without deleting it (useful when a running program still has the file open):

> empty.txt
ls -l empty.txt
-rw-r--r-- 1 sid sid 0 Nov  9 21:26 empty.txt

The truncation happens when the shell sets up the redirection, before the command runs. Keep that sentence in mind; it explains the most destructive mistake in this article (see Common mistakes).

Redirecting errors: 2> and 2>>

Put a file descriptor number in front of > to redirect that descriptor. > on its own is shorthand for 1>. So 2> redirects stderr:

ls /etc/hostname /etc/nonexistent 2> errors.txt
cat errors.txt
/etc/hostname
ls: cannot access '/etc/nonexistent': No such file or directory

The first line is stdout (still on the terminal); the second came from cat errors.txt. You can redirect both streams to different files in one command:

ls /etc/hostname /etc/nonexistent > found.txt 2> errors.txt
cat found.txt errors.txt
/etc/hostname
ls: cannot access '/etc/nonexistent': No such file or directory

The everyday use is getting clean results out of a command that complains a lot. find over /etc as a normal user hits directories it may not read:

find /etc -name '*.conf' | wc -l
find: ‘/etc/credstore’: Permission denied
find: ‘/etc/ssl/private’: Permission denied
find: ‘/etc/polkit-1/rules.d’: Permission denied
find: ‘/etc/credstore.encrypted’: Permission denied
133

Notice the count was right both times: the errors never entered the pipe. They only clutter the screen. To hide them, send FD 2 to /dev/null:

find /etc -name '*.conf' 2> /dev/null | wc -l
133

Be deliberate about 2> /dev/null. It hides every error, including the one that tells you why the result is wrong. In scripts I prefer 2>> errors.log so the messages are still somewhere when you need them. (find itself is covered in finding files.)

In your own scripts, error messages belong on stderr too. >&2 is shorthand for 1>&2: "send this command's stdout wherever stderr goes":

echo "error: config file not found" >&2
error: config file not found

It looks identical on screen, but anyone running yourscript > out.txt will still see the error instead of finding it buried in the output file.

Sending both streams to one place

2>&1 means "make FD 2 a copy of whatever FD 1 is right now". Combined with >, it puts everything in one file, which is what you want for a job log:

ls /etc/hostname /etc/nonexistent > all.txt 2>&1
cat all.txt
ls: cannot access '/etc/nonexistent': No such file or directory
/etc/hostname

Why the order matters

Redirections are processed left to right, and 2>&1 copies FD 1's destination at that moment. Swap the order and you get a different result:

ls /etc/hostname /etc/nonexistent 2>&1 > all.txt
cat all.txt
ls: cannot access '/etc/nonexistent': No such file or directory
/etc/hostname

The error printed on the terminal and only /etc/hostname landed in the file. Step by step:

  1. 2>&1: FD 1 still points at the terminal, so FD 2 now points at the terminal too (it already did; nothing visible changes).
  2. > all.txt: FD 1 now points at the file. FD 2 is not affected; it was a copy, not a link.

Read 2>&1 as "FD 2 goes where FD 1 goes now", not "FD 2 follows FD 1 forever", and the order rule stops being something to memorise.

This same left-to-right rule gives you a useful trick: pipe only the errors and throw the normal output away. Point FD 2 at the pipe first, then point FD 1 elsewhere:

ls /etc/hostname /etc/nonexistent 2>&1 > /dev/null | wc -l
1

Shorthands: &>, &>> and |&

Bash has shorter spellings for the common cases:

ls /etc/hostname /etc/nonexistent &> all.txt
ls /etc/hostname /etc/nonexistent |& wc -l
2

&> file is the same as > file 2>&1, &>> file appends both streams, and cmd1 |& cmd2 is the same as cmd1 2>&1 | cmd2. They are Bash-specific; if a script has to run under /bin/sh (dash on Debian and Ubuntu), use the long forms.

Discarding output with /dev/null

/dev/null is a device file that accepts any amount of data and throws it away. Reading from it returns end-of-file immediately:

wc -c < /dev/null
0

The classic use is asking a question where only the exit status matters. command -v (covered in the next article) prints the path of a command if it exists; we don't want the path, just the yes/no:

command -v tar > /dev/null && echo "tar is installed"
command -v kubectl > /dev/null || echo "kubectl is missing"
tar is installed
kubectl is missing

< /dev/null is useful too: it guarantees a command can't sit waiting for keyboard input, which matters in cron jobs and CI where nobody is there to type.

Protecting files with noclobber

If > truncating files makes you nervous, turn on noclobber. With it set, > refuses to overwrite an existing file:

set -o noclobber
echo "v2" > deploy.log
bash: deploy.log: cannot overwrite existing file

When you really do mean to overwrite, >| forces it. Appending with >> is still allowed:

echo "v2" >| deploy.log
echo "more" >> deploy.log
cat deploy.log
v2
more

set +o noclobber turns it off again. Some people put set -o noclobber in their ~/.bashrc; I think it is a reasonable safety net for interactive shells, but don't rely on it in scripts, because a script run by someone else won't have your settings.

Reading input from a file: <

< connects a file to a command's stdin. Many commands accept a filename argument anyway, so the difference is subtle but real. Compare:

wc -l /etc/passwd
wc -l < /etc/passwd
28 /etc/passwd
28

With <, wc never sees a filename; it just reads bytes from FD 0. That is handy when you want only the number, for example in count=$(wc -l < file).

Some commands only read stdin. tr, which translates characters, is the one everyone trips over:

tr '[:lower:]' '[:upper:]' deploy.log
tr: extra operand ‘deploy.log’
Try 'tr --help' for more information.
tr '[:lower:]' '[:upper:]' < deploy.log
V2
MORE

The other place you will use < constantly is feeding a file to a while read loop, one line at a time:

printf '%s\n' web01 web02 db01 > hosts.txt
while read -r host; do echo "checking $host"; done < hosts.txt
checking web01
checking web02
checking db01

Loops get their own article, loops, functions, and reading input.

Here-documents: multi-line input inline

A here-document feeds several lines of text to a command's stdin, written right in the script. Everything between <<WORD and a line containing only WORD becomes the input:

cat <<EOF
Report for user: $USER
Home directory:  $HOME
Generated on:    $(date +%F)
EOF
Report for user: sid
Home directory:  /home/sid
Generated on:    2026-11-09

EOF is only a convention; any word works. With an unquoted delimiter, the body is treated like a double-quoted string: variables, $(...) command substitution and backslash escapes are expanded.

Quoted delimiter: no expansion

Quote the delimiter (any part of it, 'EOF' is the usual style) and the body is passed through literally:

cat <<'EOF'
Report for user: $USER
Home directory:  $HOME
Generated on:    $(date +%F)
EOF
Report for user: $USER
Home directory:  $HOME
Generated on:    $(date +%F)

My rule: quote the delimiter by default, and only drop the quotes when you actually want expansion. Use the quoted form whenever the body is itself code (a script, SQL, an awk program) that contains $.

Mixing expanded and literal $

Config files often need both: your shell variables filled in, and the target program's own $variables left alone. Use an unquoted delimiter and escape the dollars that must survive:

domain=shop.example.com
cat > site.conf <<EOF
server {
    listen 80;
    server_name $domain;
    location / {
        try_files \$uri \$uri/ =404;
    }
}
EOF
cat site.conf
server {
    listen 80;
    server_name shop.example.com;
    location / {
        try_files $uri $uri/ =404;
    }
}

cat > file <<EOF is the standard way to write a whole file from a script. Notice the redirections combine: > sends stdout to site.conf, <<EOF feeds stdin.

Indented here-documents: <<-

Inside a function or if block you want the body indented with the rest of the code. <<- strips leading tab characters (not spaces) from each body line and from the closing delimiter. Here is a script where the indentation is real tabs; cat -A shows them as ^I:

cat -A motd.sh
#!/usr/bin/env bash$
print_banner() {$
^Icat <<-EOF$
^IMaintenance window: Saturday 02:00-04:00 UTC$
^IContact: ops@example.com$
^IEOF$
}$
print_banner$
bash motd.sh
Maintenance window: Saturday 02:00-04:00 UTC
Contact: ops@example.com

If your editor converts tabs to spaces, <<- silently stops working; the result is in Common mistakes. Copying the example from a web page usually turns tabs into spaces too, so type the tabs yourself.

Feeding other programs

Anything that reads stdin can take a here-document: psql, mysql, ssh host bash, or an interpreter. The - tells Python to read the program from stdin:

python3 - <<'EOF'
import platform
print("python", platform.python_version())
EOF
python 3.12.3

Here-strings: one line of input

A here-string, <<< word, feeds a single string (plus a trailing newline) to stdin. It is the tidy way to hand a variable to a command that only reads stdin:

read -r method path proto <<< "POST /api/orders HTTP/1.1"
echo "$path"
/api/orders
IFS=, read -r id date region <<< "1001,2026-11-01,north"
echo "order $id shipped to $region"
order 1001 shipped to north

Note the trailing newline Bash adds. It is usually what you want, but it does count:

wc -c <<< "abc"
4

Always quote the word (<<< "$var") for the same reasons you quote any expansion; see variables and quoting.

Pipes: connecting commands

A pipe, cmd1 | cmd2, connects stdout of cmd1 to stdin of cmd2. This is the mechanism behind the Unix philosophy: small programs that each do one job, reading text and writing text, combined on the command line into something bigger.

Here is a pipeline built one stage at a time: which login shells are configured on this machine, and how many accounts use each? Field 7 of /etc/passwd is the shell:

cut -d: -f7 /etc/passwd | head -n 4
/bin/bash
/usr/sbin/nologin
/usr/sbin/nologin
/usr/sbin/nologin

Group identical lines and count them (uniq needs sorted input):

cut -d: -f7 /etc/passwd | sort | uniq -c
      2 /bin/bash
      1 /bin/false
      1 /bin/sync
     24 /usr/sbin/nologin

Sort by the count, largest first:

cut -d: -f7 /etc/passwd | sort | uniq -c | sort -rn
     24 /usr/sbin/nologin
      2 /bin/bash
      1 /bin/sync
      1 /bin/false

Build pipelines exactly like this: run the first stage, look at the output, add one stage, look again. Every tool used here gets a full treatment in searching and filtering text.

All stages run at the same time

A pipeline is not "run A, save the output, then run B". The shell starts every stage at once and the kernel moves data between them through a small buffer. Two sleep 2 commands in a pipeline take two seconds, not four:

time (sleep 2 | sleep 2)
 
real	0m2.005s
user	0m0.002s
sys	0m0.001s

This is also why yes | head -n 3 finishes even though yes prints forever: when head has its three lines it exits, and the next write by yes kills it with SIGPIPE.

yes "y" | head -n 3
y
y
y

Streaming means pipelines work on files far larger than memory, and you see the first results before the last line has been read.

Each stage runs in a subshell

Each part of a pipeline runs in a child process (a subshell). Variables set inside a pipeline stage vanish when the stage ends. This catches everyone who pipes into a while read loop; it is in Common mistakes, and the next article explains subshells properly.

tee: save a copy and keep going

tee reads stdin and writes it both to stdout and to one or more files, like a T-shaped pipe fitting. Use it to keep a copy of what flows through a pipeline:

ls -l /etc/*.conf | tee conf-list.txt | wc -l
head -n 3 conf-list.txt
26
-rw-r--r-- 1 root root  3444 Jul  5  2023 /etc/adduser.conf
-rw-r--r-- 1 root root  6862 Jun 17 08:34 /etc/ca-certificates.conf
-rw-r--r-- 1 root root  2967 Apr 12  2024 /etc/debconf.conf

Dropping a tee into the middle of a pipeline is also the best way to debug one: you see the final result and get a snapshot of an intermediate stage:

cut -d: -f7 /etc/passwd | sort | tee shells-sorted.txt | uniq -c | sort -rn | head -n 2
wc -l shells-sorted.txt
     24 /usr/sbin/nologin
      2 /bin/bash
28 shells-sorted.txt

tee -a appends instead of truncating, so some-command | tee -a job.log shows output live and logs it. Add > /dev/null after tee when you want only the file.

Writing to root-owned files: the sudo tee pattern

This is the redirection gotcha that confuses people for years. Say you want to add a line to /etc/hosts, which only root can write:

echo "192.0.2.10 db.internal" >> /etc/hosts
bash: /etc/hosts: Permission denied

Reasonable next try:

sudo echo "192.0.2.10 db.internal" >> /etc/hosts
bash: /etc/hosts: Permission denied

Same error, and notice it says bash:, not sudo:. The redirection belongs to your shell, which opens /etc/hosts as you, fails, and never even runs sudo. Only echo would have been elevated. The fix is to make the process that opens the file the privileged one, and tee is a program that opens files:

echo "192.0.2.10 db.internal" | sudo tee -a /etc/hosts > /dev/null

sudo tee -a opens the file as root and appends; > /dev/null stops tee echoing the line back at you. (I could not run this last command in my test environment, which has no sudo password; the two failures above are real output.) See file permissions and ownership for why root owns these files in the first place.

Pipeline exit status: PIPESTATUS and pipefail

Every command returns an exit status: 0 for success, anything else for failure (the next article covers this in depth). A pipeline's status is, by default, the status of its last command only:

false | true
echo $?
true | false
echo $?
0
1

That default is how the backup at the top of this article "succeeded". It also bites in reporting pipelines. Counting errors in a log that doesn't exist:

grep ERROR app.log | wc -l
echo "exit status: $?"
grep: app.log: No such file or directory
0
exit status: 0

A monitoring script would read that as "zero errors, all good".

PIPESTATUS: every stage's status

Bash records the status of each command of the most recent pipeline in the array PIPESTATUS. Read it immediately, because the next command overwrites it:

grep ERROR app.log | sort | wc -l
echo "${PIPESTATUS[@]}"
grep: app.log: No such file or directory
0
2 0 0

grep returned 2 (error), sort and wc returned 0.

set -o pipefail: make the pipeline fail

With pipefail on, a pipeline's status is the status of the last command that failed (or 0 if all succeeded):

set -o pipefail
grep ERROR app.log | sort | wc -l
echo "exit status: $?"
grep: app.log: No such file or directory
0
exit status: 2

And the backup from the start of the article now reports what happened:

set -o pipefail
tar -cf - ./app-data | gzip > backup.tar.gz
echo "exit status: $?"
tar: ./app-data: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
exit status: 2

Put set -o pipefail at the top of every script you write. It is part of the set -euo pipefail header covered in error handling, traps, and debugging.

The pipefail surprise: exit status 141

pipefail has one sharp edge. When a downstream command stops reading early (head, grep -q, grep -m 1), the upstream command is killed by SIGPIPE, and that counts as a failure:

set -o pipefail
seq 1 1000000 | head -n 1
echo "exit status: $?  PIPESTATUS: ${PIPESTATUS[*]}"
1
exit status: 141  PIPESTATUS: 141 0

141 is 128 + 13, and 13 is the signal number of SIGPIPE. The output was correct; the status says otherwise. With pipefail off, the same pipeline returns 0. It only happens when the upstream command still has output to write after the reader exits, so it depends on data size, which makes it a nasty intermittent failure in scripts. When you see 141 in a pipefail script, look for a head or grep -q downstream; the usual fixes are to let the reader consume everything (grep -c instead of grep -q, awk 'NR==1' instead of head -n 1) or to check PIPESTATUS yourself for that pipeline.

Process substitution: <( ) and >( )

Some commands insist on filenames, not stdin. diff compares two files; you can't pipe two things into one stdin. Process substitution runs a command and hands its output to another command as if it were a file:

echo <(true)
/dev/fd/63

That path is what the receiving command sees: a file descriptor the shell connected to the inner command's output. Here it compares two unsorted server lists without creating temporary sorted copies:

printf '%s\n' web01 web02 db01 db02 cache01 > inventory.txt
printf '%s\n' db01 web02 web01 lb01 > monitored.txt
diff <(sort inventory.txt) <(sort monitored.txt)
1d0
< cache01
3c2
< db02
---
> lb01

cache01 and db02 are in the inventory but not monitored; lb01 is monitored but not in the inventory.

>( ) works the other way: it gives a command a "file" to write to, and whatever is written there becomes stdin of the inner command. Combined with tee, it lets one pass over the data feed several consumers. This compresses a log and counts its 404s in one read:

tee >(gzip > access.log.gz) < access.log | grep -c ' 404 '
9

One caveat: the command inside >( ) runs in the background, and Bash does not wait for it. If the next line of your script uses access.log.gz, it may not be completely written yet. For anything a later step depends on, use a plain pipeline or a temporary file.

Process substitution is a Bash feature (not POSIX sh) and needs /dev/fd, which every Linux and macOS system has.

Opening your own file descriptors with exec

exec with only redirections (no command) changes the current shell's own file descriptors from that point on. Two patterns appear in real scripts.

Send everything the rest of the script prints to a log:

exec >> job.log 2>&1

And open an extra descriptor (3 to 9 are free for you) for a file you write to many times, then close it with 3>&-:

exec 3>> audit.log
echo "user sid rotated logs" >&3
exec 3>&-
echo "user sid restarted nginx" >&3
bash: 3: Bad file descriptor

The last line fails because FD 3 is closed, which proves the close worked. Opening the file once is cheaper than >> audit.log on every line and, more usefully, keeps the log separate from stdout and stderr.

Quick reference

SyntaxMeaning
cmd > filestdout to file (truncate first)
cmd >> filestdout appended to file
cmd 2> filestderr to file
cmd > file 2>&1stdout and stderr to file (order matters)
cmd &> file / cmd &>> filesame, Bash shorthand (truncate / append)
cmd >&2stdout to wherever stderr goes
cmd < filestdin from file
cmd <<EOF / <<'EOF'here-document (expanded / literal)
cmd <<-EOFhere-document with leading tabs stripped
cmd <<< "$var"here-string
cmd1 | cmd2stdout of cmd1 to stdin of cmd2
cmd1 |& cmd2stdout and stderr of cmd1 to cmd2
cmd >| fileoverwrite even with noclobber set
<(cmd) / >(cmd)process substitution (read from / write to)
exec 3> file, exec 3>&-open / close a file descriptor in the current shell

Where you'll use this

Redirection is in almost every script you will write. The common jobs:

  • Cron jobs have no terminal. Anything not redirected is mailed to the user or lost, so the standard crontab line ends with >> /path/to/job.log 2>&1. Scheduling is covered in system monitoring and scheduling.
  • Generating config files in provisioning scripts and Dockerfiles with cat > file <<'EOF'.
  • Reports and audits built from pipelines, saved with tee so a human sees them and a file keeps them.
  • CI pipelines, where set -o pipefail is the difference between a red build and a green build that shipped nothing.

Here is a complete script that uses most of this article: it archives a log directory, logs everything it does, detects failure anywhere in the tar | gzip pipeline, and writes a summary with a here-document.

#!/usr/bin/env bash
# backup-logs.sh - archive a log directory; built to run unattended from cron.
set -o pipefail
 
src=${1:-./logs}
dest=./backups
archive="$dest/logs-$(date +%Y%m%d-%H%M%S).tar.gz"
partial="$archive.partial"
joblog=./backup-job.log
 
mkdir -p "$dest"
 
# From here on, stdout and stderr of every command go to the job log.
exec >> "$joblog" 2>&1
 
echo "[$(date '+%F %T')] backing up $src"
 
# Write to a temporary name; only a complete archive gets the real name.
if tar -cf - "$src" | gzip > "$partial"; then
    mv "$partial" "$archive"
    echo "[$(date '+%F %T')] OK: $archive ($(du -h "$archive" | cut -f1))"
else
    echo "[$(date '+%F %T')] FAILED: tar or gzip failed, removing $partial"
    rm -f "$partial"
    exit 1
fi
 
cat > "$dest/LATEST.txt" <<EOF
archive: $archive
source:  $src
entries: $(tar -tzf "$archive" | wc -l)
EOF

Run it once normally and once against a directory that doesn't exist:

./backup-logs.sh
echo "exit: $?"
./backup-logs.sh ./no-such-dir
echo "exit: $?"
exit: 0
exit: 1

Nothing appeared on the terminal except our own echo lines, because the script sent all of its output to the job log:

cat backup-job.log
[2026-11-09 21:28:00] backing up ./logs
[2026-11-09 21:28:00] OK: ./backups/logs-20261109-212800.tar.gz (4.0K)
[2026-11-09 21:28:00] backing up ./no-such-dir
tar: ./no-such-dir: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
[2026-11-09 21:28:00] FAILED: tar or gzip failed, removing ./backups/logs-20261109-212800.tar.gz.partial
ls backups
cat backups/LATEST.txt
LATEST.txt
logs-20261109-212800.tar.gz
archive: ./backups/logs-20261109-212800.tar.gz
source:  ./logs
entries: 3

The .partial step is there because of something that happened while I tested this. My first version wrote straight to $archive and deleted it on failure. Both test runs happened in the same second, so they computed the same archive name: the failing run's gzip > "$archive" truncated the good archive from the first run, and its cleanup then deleted it. Writing to a temporary name and renaming only on success means a failed run can never damage a good file. mv within one filesystem is a rename, so the real name only ever points at a complete archive.

In a crontab the script would be scheduled like this (the cron article walks through installing it):

30 2 * * * /home/sid/bin/backup-logs.sh /var/log/shop

Because the script redirects its own output with exec, the crontab line doesn't need >> log 2>&1, though adding it costs nothing and catches errors that happen before the exec line.

Common mistakes

Sorting a file into itself

printf '%s\n' web02 db01 web01 > servers.txt
sort servers.txt > servers.txt
wc -c servers.txt
0 servers.txt

The file is now empty. The shell processed > servers.txt first, truncating the file, and only then started sort, which read an empty file. The same happens with grep, sed, cut or anything else when the input and output are the same file. Write to a new file and rename it, or use a tool's own output option:

printf '%s\n' web02 db01 web01 > servers.txt
sort -o servers.txt servers.txt
cat servers.txt
db01
web01
web02

sort -o reads all its input before opening the output file, so it is safe. For other commands, cmd file > file.tmp && mv file.tmp file is the portable pattern.

A here-document that never ends

If the closing delimiter has leading spaces (with <<), trailing spaces, or the body is indented with spaces under <<-, Bash doesn't recognise the end marker and keeps reading to the end of the file. This script is indented with four spaces:

cat banner-spaces.sh
#!/usr/bin/env bash
print_banner() {
    cat <<-EOF
    Maintenance window: Saturday 02:00-04:00 UTC
    EOF
}
print_banner
echo "script finished"
bash banner-spaces.sh; echo "exit: $?"
banner-spaces.sh: line 8: warning: here-document at line 3 delimited by end-of-file (wanted `EOF')
banner-spaces.sh: line 9: syntax error: unexpected end of file
exit: 2

The here-document swallowed the closing } and everything after it, so the function was never closed. Fix it by indenting with tabs (<<- only strips tabs) or by putting the closing EOF at column 0 with nothing after it. cat -A shows tabs as ^I and line ends as $, which makes stray whitespace visible.

Piping into while read and losing the variable

last=none
printf '%s\n' web01 web02 db01 | while read -r host; do last=$host; done
echo "last=$last"
last=none

The loop ran in a subshell (it is a pipeline stage), set its own copy of last, and exited. Feed the loop with process substitution instead, so the loop runs in the current shell:

while read -r host; do last=$host; done < <(printf '%s\n' web01 web02 db01)
echo "last=$last"
last=db01

The space in < <( matters: it is a < redirection whose source is a <( ) process substitution.

Summary

  • Every process has stdin (0), stdout (1) and stderr (2). Redirection changes where those descriptors point before the command starts; the command never knows.
  • > truncates first, >> appends, 2> redirects errors, < supplies input. Never redirect output to a file that the same command is reading.
  • Redirections are processed left to right: > file 2>&1 captures both streams, 2>&1 > file does not. &> and |& are Bash shorthands.
  • Here-documents feed multi-line input; quote the delimiter (<<'EOF') to stop expansion, use <<- only with tab indentation. Here-strings (<<<) feed one string.
  • A pipeline's stages run concurrently in subshells; only stdout flows through |.
  • By default a pipeline's exit status is its last command's. Use set -o pipefail in scripts, read PIPESTATUS right after the pipeline, and expect 141 when head or grep -q closes a pipe early.
  • sudo cmd > file fails because your shell opens the file; use | sudo tee file.
  • <(cmd) turns a command's output into a filename for tools like diff and comm.

Exercises

Exercise 1: Separate results from errorseasy

Run ls on /etc/hostname /etc/passwd /etc/nope /var/nothing. Send the paths that exist to found.txt and the error messages to missing.txt, then count the lines in each. Finally, run the same ls again, appending both streams to a single all.log.

Show Solution
ls /etc/hostname /etc/passwd /etc/nope /var/nothing > found.txt 2> missing.txt
wc -l found.txt missing.txt
  2 found.txt
  2 missing.txt
  4 total
ls /etc/hostname /etc/passwd /etc/nope /var/nothing >> all.log 2>&1
cat all.log
ls: cannot access '/etc/nope': No such file or directory
ls: cannot access '/var/nothing': No such file or directory
/etc/hostname
/etc/passwd

>> all.log 2>&1 appends stdout, then points stderr at the same place. &>> all.log does the same in Bash.

Exercise 2: Generate a logrotate configmedium

Using shell variables app=shop and retention=14, write a here-document that creates shop.logrotate with this content. The app name and retention must come from the variables, but $(cat /run/nginx.pid) must appear literally in the file:

/var/log/shop/*.log {
    daily
    rotate 14
    compress
    missingok
    postrotate
        kill -USR1 $(cat /run/nginx.pid)
    endscript
}
Hint

An unquoted delimiter expands everything; a quoted one expands nothing. You need the unquoted form plus one backslash.

Show Solution
app=shop
retention=14
cat > "$app.logrotate" <<EOF
/var/log/$app/*.log {
    daily
    rotate $retention
    compress
    missingok
    postrotate
        kill -USR1 \$(cat /run/nginx.pid)
    endscript
}
EOF
cat shop.logrotate
/var/log/shop/*.log {
    daily
    rotate 14
    compress
    missingok
    postrotate
        kill -USR1 $(cat /run/nginx.pid)
    endscript
}

Escaping the $ as \$ stops the command substitution from running now; logrotate's own shell runs it later.

Exercise 3: Find the failing stagemedium

The pipeline grep ' 500 ' logs/missing.log | sort | head -n 1 reads a file that does not exist. Print the exit status of every stage, then make the pipeline as a whole report the failure.

Show Solution
grep ' 500 ' logs/missing.log | sort | head -n 1
echo "${PIPESTATUS[@]}"
grep: logs/missing.log: No such file or directory
2 0 0
set -o pipefail
grep ' 500 ' logs/missing.log | sort | head -n 1
echo "pipeline exit: $?"
grep: logs/missing.log: No such file or directory
pipeline exit: 2

PIPESTATUS must be read on the very next command. Without pipefail, $? would be 0, the status of head.

Exercise 4: Audit two server lists without temp fileshard

inventory.txt lists the servers you own and monitored.txt lists the servers your monitoring system watches. Neither file is sorted. Using comm (which needs sorted input and prints lines unique to file 1, unique to file 2, and common lines in three columns) and process substitution, print (a) servers that are not monitored, and (b) monitored servers that are not in the inventory.

printf '%s\n' web01 web02 db01 db02 cache01 > inventory.txt
printf '%s\n' db01 web02 web01 lb01 > monitored.txt
Hint

comm -23 a b hides columns 2 and 3, leaving lines only in a.

Show Solution
comm -23 <(sort inventory.txt) <(sort monitored.txt)
cache01
db02
comm -13 <(sort inventory.txt) <(sort monitored.txt)
lb01

Each <(sort ...) becomes a /dev/fd/N path that comm opens like a file, so the sorted copies never touch the disk.


Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), November 2026. Every command was run as shown and the output pasted unchanged; bash's own error messages were captured from an interactive shell so they read as they do at a prompt. The sudo tee command was not run because the test account has no sudo password (the Permission denied failures before it are real), and the crontab line was not installed. The exit status 141 example depends on the pipe buffer filling, which seq 1 1000000 reliably does.

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.