Searching and Filtering Text: grep, cut, sort, uniq
Answer real questions from logs and CSV files with grep, cut, sort, uniq, wc, tr, head, tail, paste and column, building up to a complete access-log report.
By The Dark DebuggerUpdated 31 min read
This is the first article in the text-processing part of the Bash path. It covers the
small filters you will combine every day: grep in depth (including regular
expressions), cut, sort, uniq, wc, tr, head, tail, paste and column.
You should be comfortable with redirection and pipes;
everything here is used inside pipelines. The next two articles cover the heavier
tools, sed and
awk.
Why these tools still matter
Picture a shop's checkout failing intermittently. Someone asks: is it one client, is it a bot, when did it start, which endpoint? You have SSH access to the web server and its access log, and no dashboard. Every one of those questions is a one-line pipeline of the tools in this article, and they run in seconds on logs with millions of lines, because each tool streams its input instead of loading it.
They are also everywhere: in Docker images, CI runners, recovery shells and every Linux server you will ever log into. Learning them once pays off for decades.
Sample data
Every example uses three files. Create them in an empty directory so you can follow along. First, an nginx access log in the standard "combined" format:
cat > access.log <<'EOF'
203.0.113.42 - - [09/Nov/2026:08:01:12 +0000] "GET / HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
203.0.113.42 - - [09/Nov/2026:08:01:13 +0000] "GET /css/main.css HTTP/1.1" 200 1834 "https://shop.example.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
10.0.0.5 - - [09/Nov/2026:08:02:00 +0000] "GET /healthz HTTP/1.1" 200 2 "-" "kube-probe/1.30"
198.51.100.7 - - [09/Nov/2026:08:02:40 +0000] "GET /products/42 HTTP/1.1" 200 7342 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0.0.0 Safari/537.36"
198.51.100.7 - - [09/Nov/2026:08:02:41 +0000] "GET /img/products/42.jpg HTTP/1.1" 304 0 "https://shop.example.com/products/42" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0.0.0 Safari/537.36"
192.0.2.15 - - [09/Nov/2026:08:05:02 +0000] "GET /wp-login.php HTTP/1.1" 404 162 "-" "python-requests/2.31.0"
192.0.2.15 - - [09/Nov/2026:08:05:02 +0000] "GET /.env HTTP/1.1" 404 162 "-" "python-requests/2.31.0"
192.0.2.15 - - [09/Nov/2026:08:05:03 +0000] "GET /.git/config HTTP/1.1" 404 162 "-" "python-requests/2.31.0"
192.0.2.15 - - [09/Nov/2026:08:05:03 +0000] "GET /admin/config.php HTTP/1.1" 404 162 "-" "python-requests/2.31.0"
192.0.2.15 - - [09/Nov/2026:08:05:04 +0000] "GET /phpmyadmin/ HTTP/1.1" 404 162 "-" "python-requests/2.31.0"
192.0.2.15 - - [09/Nov/2026:08:05:04 +0000] "GET /xmlrpc.php HTTP/1.1" 404 162 "-" "python-requests/2.31.0"
192.0.2.15 - - [09/Nov/2026:08:05:05 +0000] "GET /wp-admin/ HTTP/1.1" 404 162 "-" "python-requests/2.31.0"
192.0.2.15 - - [09/Nov/2026:08:05:05 +0000] "GET /wp-login.php HTTP/1.1" 404 162 "-" "python-requests/2.31.0"
203.0.113.42 - - [09/Nov/2026:08:07:55 +0000] "GET /products?page=2 HTTP/1.1" 200 15003 "https://shop.example.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
10.0.0.5 - - [09/Nov/2026:08:12:00 +0000] "GET /healthz HTTP/1.1" 200 2 "-" "kube-probe/1.30"
198.51.100.23 - - [09/Nov/2026:08:15:21 +0000] "POST /api/orders HTTP/1.1" 201 412 "-" "curl/8.5.0"
198.51.100.23 - - [09/Nov/2026:08:15:58 +0000] "POST /api/orders HTTP/1.1" 500 89 "-" "curl/8.5.0"
203.0.113.9 - - [09/Nov/2026:08:20:13 +0000] "GET /robots.txt HTTP/1.1" 200 67 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
203.0.113.9 - - [09/Nov/2026:08:20:14 +0000] "GET /products/42 HTTP/1.1" 200 7342 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
10.0.0.5 - - [09/Nov/2026:08:22:00 +0000] "GET /healthz HTTP/1.1" 200 2 "-" "kube-probe/1.30"
203.0.113.42 - - [09/Nov/2026:09:03:37 +0000] "GET /cart HTTP/1.1" 200 3921 "https://shop.example.com/products?page=2" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
203.0.113.42 - - [09/Nov/2026:09:04:10 +0000] "POST /api/orders HTTP/1.1" 201 398 "https://shop.example.com/cart" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
198.51.100.23 - - [09/Nov/2026:09:10:44 +0000] "POST /api/orders HTTP/1.1" 502 157 "-" "curl/8.5.0"
198.51.100.23 - - [09/Nov/2026:09:11:02 +0000] "POST /api/orders HTTP/1.1" 201 405 "-" "curl/8.5.0"
198.51.100.7 - - [09/Nov/2026:09:12:09 +0000] "GET /account HTTP/1.1" 301 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0.0.0 Safari/537.36"
198.51.100.7 - - [09/Nov/2026:09:12:10 +0000] "GET /account/ HTTP/1.1" 200 4410 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0.0.0 Safari/537.36"
10.0.0.5 - - [09/Nov/2026:09:32:00 +0000] "GET /healthz HTTP/1.1" 200 2 "-" "kube-probe/1.30"
192.0.2.77 - - [09/Nov/2026:09:40:31 +0000] "GET /favicon.ico HTTP/1.1" 404 162 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) Mobile/15E148 Safari/604.1"
198.51.100.7 - - [09/Nov/2026:10:01:15 +0000] "GET /img/logo.png HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0.0.0 Safari/537.36"
198.51.100.7 - - [09/Nov/2026:10:01:16 +0000] "GET /products/42 HTTP/1.1" 200 7342 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0.0.0 Safari/537.36"
203.0.113.42 - - [09/Nov/2026:10:05:49 +0000] "GET /orders/1042 HTTP/1.1" 200 2876 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
198.51.100.23 - - [09/Nov/2026:10:06:30 +0000] "GET /api/orders/1042 HTTP/1.1" 200 1500 "-" "curl/8.5.0"
203.0.113.42 - - [09/Nov/2026:10:09:02 +0000] "GET /logout HTTP/1.1" 302 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
EOFSplit on spaces, each line has these fields. The numbers matter, because cut and
awk address fields by position:
| Field | Example | Meaning |
|---|---|---|
| 1 | 203.0.113.42 | client IP |
| 4 | [09/Nov/2026:08:01:12 | timestamp (with the bracket) |
| 6 | "GET | method (with the quote) |
| 7 | /products/42 | path |
| 9 | 200 | HTTP status |
| 10 | 5120 | response size in bytes |
Second, an application log with two Java stack traces:
cat > app.log <<'EOF'
2026-11-09 08:15:57 INFO [http-8080-3] POST /api/orders received (order 1043)
2026-11-09 08:15:57 INFO [http-8080-3] reserving stock for order 1043
2026-11-09 08:15:58 WARN [http-8080-3] payment gateway slow: 2950 ms
2026-11-09 08:15:58 ERROR [http-8080-3] order 1043 failed
java.net.SocketTimeoutException: Read timed out
at com.example.orders.PaymentClient.charge(PaymentClient.java:88)
at com.example.orders.OrderService.place(OrderService.java:142)
2026-11-09 08:15:58 INFO [http-8080-3] stock released for order 1043
2026-11-09 09:04:10 INFO [http-8080-1] POST /api/orders received (order 1044)
2026-11-09 09:04:10 INFO [http-8080-1] order 1044 placed
2026-11-09 09:10:44 ERROR [http-8080-2] order 1045 failed
java.sql.SQLTransientConnectionException: orders-pool - Connection is not available
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
2026-11-09 09:11:02 INFO [http-8080-2] order 1046 placed
EOFThird, a CSV export of orders:
cat > orders.csv <<'EOF'
order_id,date,region,product,qty,unit_price
1001,2026-11-01,north,keyboard,2,49.99
1002,2026-11-01,south,monitor,1,189.00
1003,2026-11-02,west,mouse,5,19.50
1004,2026-11-02,north,monitor,2,189.00
1005,2026-11-03,east,headset,1,79.00
1006,2026-11-03,south,keyboard,3,49.99
1007,2026-11-04,west,monitor,1,189.00
1008,2026-11-04,north,mouse,10,19.50
1009,2026-11-05,east,keyboard,1,49.99
1010,2026-11-05,south,headset,2,79.00
1011,2026-11-06,north,headset,1,79.00
1012,2026-11-06,west,keyboard,4,49.99
EOFgrep: find the lines you care about
grep PATTERN FILE... prints every line that contains a match for the pattern:
grep 'POST' access.log198.51.100.23 - - [09/Nov/2026:08:15:21 +0000] "POST /api/orders HTTP/1.1" 201 412 "-" "curl/8.5.0"
198.51.100.23 - - [09/Nov/2026:08:15:58 +0000] "POST /api/orders HTTP/1.1" 500 89 "-" "curl/8.5.0"
203.0.113.42 - - [09/Nov/2026:09:04:10 +0000] "POST /api/orders HTTP/1.1" 201 398 "https://shop.example.com/cart" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"
198.51.100.23 - - [09/Nov/2026:09:10:44 +0000] "POST /api/orders HTTP/1.1" 502 157 "-" "curl/8.5.0"
198.51.100.23 - - [09/Nov/2026:09:11:02 +0000] "POST /api/orders HTTP/1.1" 201 405 "-" "curl/8.5.0"Always put the pattern in single quotes. Patterns are full of characters the shell
would otherwise interpret (*, $, [, |, spaces), and single quotes pass them to
grep untouched.
The options you'll use every day
Count matching lines with -c, show line numbers with -n, and invert the
match (lines that don't match) with -v:
grep -c 'POST' access.log
grep -n 'ERROR' app.log5
4:2026-11-09 08:15:58 ERROR [http-8080-3] order 1043 failed
11:2026-11-09 09:10:44 ERROR [http-8080-2] order 1045 failedgrep -v 'INFO' app.log2026-11-09 08:15:58 WARN [http-8080-3] payment gateway slow: 2950 ms
2026-11-09 08:15:58 ERROR [http-8080-3] order 1043 failed
java.net.SocketTimeoutException: Read timed out
at com.example.orders.PaymentClient.charge(PaymentClient.java:88)
at com.example.orders.OrderService.place(OrderService.java:142)
2026-11-09 09:10:44 ERROR [http-8080-2] order 1045 failed
java.sql.SQLTransientConnectionException: orders-pool - Connection is not available
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)-i ignores case. Here the user agent says Googlebot but we typed it in lowercase
(the cut at the end just keeps the IP and path; it is explained below):
grep -i 'googlebot' access.log | cut -d' ' -f1,7203.0.113.9 /robots.txt
203.0.113.9 /products/42Whole words: -w
How many requests failed with HTTP 500? The obvious command gives the wrong answer:
grep -c 500 access.log
grep 500 access.log | cut -d' ' -f9,103
200 15003
500 89
200 1500grep matches text, not fields: 15003 and 1500 (response sizes) both contain
500. -w only matches where the pattern is a whole word, with no letter, digit or
underscore on either side:
grep -cw 500 access.log1-w fixed this one, but a size of exactly 500 bytes would still match. The robust
approach is to anchor the pattern to its position in the line; in this log format the
status always follows the closing quote of the request, so '" 500 ' is precise. We'll
use that below.
Only the matching part: -o
-o prints just the text that matched, one match per line, instead of the whole line.
Combined with a counting pipeline it extracts values:
grep -o 'python-requests/[0-9.]*' access.log | sort | uniq -c 8 python-requests/2.31.0Context: -A, -B, -C
An error line is rarely enough; you want the stack trace after it or the warning
before it. -A N prints N lines after each match, -B N before, and -C N both:
grep -A 3 'ERROR' app.log2026-11-09 08:15:58 ERROR [http-8080-3] order 1043 failed
java.net.SocketTimeoutException: Read timed out
at com.example.orders.PaymentClient.charge(PaymentClient.java:88)
at com.example.orders.OrderService.place(OrderService.java:142)
--
2026-11-09 09:10:44 ERROR [http-8080-2] order 1045 failed
java.sql.SQLTransientConnectionException: orders-pool - Connection is not available
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
2026-11-09 09:11:02 INFO [http-8080-2] order 1046 placedgrep -B 1 'ERROR' app.log2026-11-09 08:15:58 WARN [http-8080-3] payment gateway slow: 2950 ms
2026-11-09 08:15:58 ERROR [http-8080-3] order 1043 failed
--
2026-11-09 09:04:10 INFO [http-8080-1] order 1044 placed
2026-11-09 09:10:44 ERROR [http-8080-2] order 1045 failedThe -- lines separate non-adjacent groups. -B 1 shows the slow-payment warning right
before the first failure, which is exactly the clue you would want at 2am.
-m N stops after N matches, which is useful on huge files when you only need to know
the first occurrence:
grep -m 1 'ERROR' app.log2026-11-09 08:15:58 ERROR [http-8080-3] order 1043 failedSearching many files: -r, -l, -L, -h
Say the database host db01.internal is being retired and you need every config that
still mentions it. A conf.d directory holds orders.conf, cache.conf,
reports.conf and a stray backup, env.sh.bak. -r searches recursively, and with
-n you get file, line number and line:
grep -rn 'db01.internal' conf.d/conf.d/orders.conf:2:host = db01.internal
conf.d/reports.conf:2:dsn = postgres://report@db01.internal:5432/orders
conf.d/env.sh.bak:1:DB_HOST=db01.internal-l lists only the names of files that match (ideal for feeding to another command),
--include restricts which files are searched, and -L lists the files that don't
match:
grep -rl 'db01.internal' conf.d/
grep -rl --include='*.conf' 'db01.internal' conf.d/
grep -L 'db01.internal' conf.d/*conf.d/orders.conf
conf.d/reports.conf
conf.d/env.sh.bak
conf.d/orders.conf
conf.d/reports.conf
conf.d/cache.confWhen you give grep more than one file, it prefixes each line with the file name;
-h turns that off and -H forces it on:
grep 'order 1043' app.log access.logapp.log:2026-11-09 08:15:57 INFO [http-8080-3] POST /api/orders received (order 1043)
app.log:2026-11-09 08:15:57 INFO [http-8080-3] reserving stock for order 1043
app.log:2026-11-09 08:15:58 ERROR [http-8080-3] order 1043 failed
app.log:2026-11-09 08:15:58 INFO [http-8080-3] stock released for order 1043Finally, -q prints nothing and only sets the exit status (0 for a match, 1 for none),
which is what you use in if statements; see
exit codes.
Regular expressions for grep
The pattern is a regular expression: a small language for describing text. You only need a handful of pieces to cover most real searches.
| Pattern | Matches |
|---|---|
. | any single character |
* | zero or more of the previous item |
^ / $ | start / end of the line |
[abc], [0-9] | one character from the set or range |
[^0-9] | one character not in the set |
[[:digit:]], [[:alpha:]], [[:space:]], [[:upper:]] | POSIX character classes |
\b, \<, \> | word boundary, start of word, end of word (GNU) |
+, ?, {n,m}, |, ( ) | one or more, optional, repetition, alternation, grouping (ERE) |
Basic vs extended: grep vs grep -E
grep uses basic regular expressions (BRE) by default, where + ? { } | ( ) are
ordinary characters unless you put a backslash in front of them. grep -E uses
extended regular expressions (ERE), where they are special without backslashes.
Server errors (status 5xx), three ways:
grep -E '" (500|502|503) ' access.log | cut -d' ' -f1,7,9198.51.100.23 /api/orders 500
198.51.100.23 /api/orders 502grep '" 5[0-9]\{2\} ' access.log | cut -d' ' -f1,7,9
grep '" 5[0-9]{2} ' access.log | cut -d' ' -f1,7,9
grep -E '" 5[0-9]{2} ' access.log | cut -d' ' -f1,7,9198.51.100.23 /api/orders 500
198.51.100.23 /api/orders 502
198.51.100.23 /api/orders 500
198.51.100.23 /api/orders 502The first (BRE with \{2\}) and third (ERE with {2}) found both lines. The second
printed nothing: in BRE, {2} means the literal characters {2}. No error, just
no output. My advice is to use grep -E whenever the pattern has any of
+ ? { } | ( ); it saves you a whole class of silent mistakes.
An IP address extractor shows several pieces together. ^ anchors to the start of the
line, [0-9]{1,3} is one to three digits, and (\.[0-9]{1,3}){3} is "a dot and one to
three digits", three times:
grep -oE '^[0-9]{1,3}(\.[0-9]{1,3}){3}' access.log | sort -u10.0.0.5
192.0.2.15
192.0.2.77
198.51.100.23
198.51.100.7
203.0.113.42
203.0.113.9Anchors and escaping dots
. matches any character, and an unanchored pattern matches anywhere in the line. Both
bite when you search for an IP address. Looking for client 192.0.2.1:
grep -c '192.0.2.1' access.log
grep -c '^192\.0\.2\.1 ' access.log8
0The first pattern matched the eight requests from 192.0.2.15, because nothing said
the address had to end after the 1. The second escapes the dots (\. is a literal
dot), anchors to the start of the line, and requires the space that follows the IP.
Client 192.0.2.1 made no requests at all.
Anchors and classes also find structure. Every normal log line starts with a date, so lines that don't start with one are stack-trace continuation lines:
grep -vE '^[0-9]{4}-[0-9]{2}-[0-9]{2} ' app.logjava.net.SocketTimeoutException: Read timed out
at com.example.orders.PaymentClient.charge(PaymentClient.java:88)
at com.example.orders.OrderService.place(OrderService.java:142)
java.sql.SQLTransientConnectionException: orders-pool - Connection is not available
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)And alternation with anchors pulls the section headers and port settings out of a
config file (we'll use this app.conf again in the
sed article):
grep -E '^\[|^port' app.conf[server]
port = 8080
[database]
port = 5432
[cache]
port = 6379
[metrics]
port = 9100Fixed strings: -F
When the thing you are searching for is literal text full of regex characters, such as
a thread name in brackets, a version number or an IP, use -F and nothing is special:
grep -c '[http-8080-3]' app.loggrep: Invalid range endgrep -cF '[http-8080-3]' app.log5Brackets make a character set, and inside it p-8 is a range from p to 8, which
is backwards. That one failed loudly. The quiet version is worse:
grep -c '[ERROR]' app.log
grep -c 'ERROR' app.log13
2[ERROR] means "any one of the letters E, R or O", which nearly every line contains.
GNU grep also has -P for Perl-compatible regular expressions, which adds shortcuts like
\d (digit) and lookarounds:
grep -P '\d{4} ms' app.log2026-11-09 08:15:58 WARN [http-8080-3] payment gateway slow: 2950 ms-P is not available in BSD/macOS grep, so keep it out of scripts that must be
portable.
cut: pick out columns
cut extracts fields separated by a single delimiter character (-d), or character
positions (-c):
cut -d' ' -f1 access.log | head -n 3
cut -d' ' -f1,9 access.log | head -n 3203.0.113.42
203.0.113.42
10.0.0.5
203.0.113.42 200
203.0.113.42 200
10.0.0.5 200Field lists accept ranges: -f2- is "field 2 to the end", -f1-3 the first three:
head -n 4 orders.csv | cut -d, -f3,5
head -n 3 orders.csv | cut -d, -f2-region,qty
north,2
south,1
west,5
date,region,product,qty,unit_price
2026-11-01,north,keyboard,2,49.99
2026-11-01,south,monitor,1,189.00GNU cut adds --complement (everything except the listed fields) and
--output-delimiter:
head -n 3 orders.csv | cut -d, --complement -f2,6
head -n 3 orders.csv | cut -d, -f1,4 --output-delimiter=' -> 'order_id,region,product,qty
1001,north,keyboard,2
1002,south,monitor,1
order_id -> product
1001 -> keyboard
1002 -> monitor-c cuts by character position, handy for fixed-width data like the timestamp at the
start of each app.log line:
cut -c1-19 app.log | head -n 22026-11-09 08:15:57
2026-11-09 08:15:57Where cut stops being the right tool
cut has three limits worth knowing before they surprise you:
-
Output order is file order.
-f9,1prints field 1 then field 9; you can't reorder columns withcut.head -n 2 access.log | cut -d' ' -f9,1203.0.113.42 200 203.0.113.42 200 -
Every delimiter counts. Command output aligned with runs of spaces, like
df, produces empty fields:df -P / df -P / | cut -d' ' -f5Filesystem 1024-blocks Used Available Capacity Mounted on /dev/sdd 1055762868 4529796 997529600 1% /Two empty lines: field 5 is an empty string between two of the padding spaces. Squeeze the spaces first with
tr -s ' '(below), or useawk, which splits on runs of whitespace by default. -
It knows nothing about quoting. A CSV field like
"Smith, Jane"contains the delimiter; see Common mistakes.
sort: put lines in order
sort sorts lines as text, character by character. That is not what you want for
numbers:
printf '%s\n' 10 9 100 25 | sort
printf '%s\n' 10 9 100 25 | sort -n10
100
25
9
9
10
25
100-n compares numerically, -r reverses. For human-readable sizes like 27M and
580K, use -h. Here are the four largest directories under /usr/share, first
correctly and then with -n, which ignores the units:
du -sh /usr/share/* 2> /dev/null | sort -h | tail -n 427M /usr/share/locale
31M /usr/share/python-babel-localedata
42M /usr/share/vim
46M /usr/share/iconsdu -sh /usr/share/* 2> /dev/null | sort -n | tail -n 4580K /usr/share/libthai
612K /usr/share/lintian
708K /usr/share/perl5
1008K /usr/share/dbus-1For version numbers, -V understands that 1.10 comes after 1.9:
printf '%s\n' v1.10.0 v1.9.2 v1.2.0 | sort
printf '%s\n' v1.10.0 v1.9.2 v1.2.0 | sort -Vv1.10.0
v1.2.0
v1.9.2
v1.2.0
v1.9.2
v1.10.0Sorting by a column: -t and -k
-t sets the field separator and -k chooses the key. Largest quantity first (skipping
the header with tail -n +2, covered below):
tail -n +2 orders.csv | sort -t, -k5,5nr | head -n 31008,2026-11-04,north,mouse,10,19.50
1003,2026-11-02,west,mouse,5,19.50
1012,2026-11-06,west,keyboard,4,49.99-k5,5nr reads as "the key starts at field 5 and ends at field 5; compare it
numerically, reversed". The ,5 matters. -k5 alone means "from field 5 to the end
of the line". That doesn't change a single-key sort much, but it breaks multi-key
sorts. Sort by region, then by quantity, largest first:
tail -n +2 orders.csv | sort -t, -k3 -k5nr1005,2026-11-03,east,headset,1,79.00
1009,2026-11-05,east,keyboard,1,49.99
1011,2026-11-06,north,headset,1,79.00
1001,2026-11-01,north,keyboard,2,49.99
1004,2026-11-02,north,monitor,2,189.00
1008,2026-11-04,north,mouse,10,19.50
1010,2026-11-05,south,headset,2,79.00
1006,2026-11-03,south,keyboard,3,49.99
1002,2026-11-01,south,monitor,1,189.00
1012,2026-11-06,west,keyboard,4,49.99
1007,2026-11-04,west,monitor,1,189.00
1003,2026-11-02,west,mouse,5,19.50Wrong: within north, the order with quantity 10 is last. -k3 compared
north,headset,... against north,mouse,... all the way to the end of the line, so
those keys were never equal and the second key was never used. With both keys bounded:
tail -n +2 orders.csv | sort -t, -k3,3 -k5,5nr1005,2026-11-03,east,headset,1,79.00
1009,2026-11-05,east,keyboard,1,49.99
1008,2026-11-04,north,mouse,10,19.50
1001,2026-11-01,north,keyboard,2,49.99
1004,2026-11-02,north,monitor,2,189.00
1011,2026-11-06,north,headset,1,79.00
1006,2026-11-03,south,keyboard,3,49.99
1010,2026-11-05,south,headset,2,79.00
1002,2026-11-01,south,monitor,1,189.00
1003,2026-11-02,west,mouse,5,19.50
1012,2026-11-06,west,keyboard,4,49.99
1007,2026-11-04,west,monitor,1,189.00Always write keys as -kN,N.
Stable sorts: -s
When two lines have equal keys, sort breaks the tie by comparing the whole line, and
-r reverses that last-resort comparison too. -s (stable) turns the tiebreak off, so
equal lines keep their input order. Compare the two mouse orders here:
tail -n +2 orders.csv | sort -t, -r -k4,4 | head -n 2
tail -n +2 orders.csv | sort -t, -r -s -k4,4 | head -n 21008,2026-11-04,north,mouse,10,19.50
1003,2026-11-02,west,mouse,5,19.50
1003,2026-11-02,west,mouse,5,19.50
1008,2026-11-04,north,mouse,10,19.50Without -s, the tie was broken by the whole line in reverse, so 1008 came first. With
-s, they stayed in file (date) order. Use -s when the input order means something,
such as time order in a log.
Other sort options worth knowing
-uoutputs only the first of each run of equal lines: a sorted list of distinct values.-ffolds case. In theCandC.UTF-8locales, sorting compares byte values, so every uppercase letter comes before every lowercase one (Apple Cherry apple banana, below). Locales such asen_US.UTF-8use dictionary-style collation instead, so the same input sorts differently on a machine with another locale. If a script's output order must be identical everywhere, runsortwithLC_ALL=C.-o filewrites the result to a file, and is safe even when that file is the input.-cchecks whether input is already sorted, and reports the first line out of order.
tail -n +2 orders.csv | cut -d, -f3 | sort -u
printf '%s\n' banana Apple apple Cherry | sorteast
north
south
west
Apple
Cherry
apple
bananauniq: collapse and count duplicates
uniq removes adjacent duplicate lines, and uniq -c prefixes each line with how
many times it repeated. The word "adjacent" is the entire lesson of this section. Here
is uniq -c on the client IPs, straight from the log:
cut -d' ' -f1 access.log | uniq -c 2 203.0.113.42
1 10.0.0.5
2 198.51.100.7
8 192.0.2.15
1 203.0.113.42
1 10.0.0.5
2 198.51.100.23
2 203.0.113.9
1 10.0.0.5
2 203.0.113.42
2 198.51.100.23
2 198.51.100.7
1 10.0.0.5
1 192.0.2.77
2 198.51.100.7
1 203.0.113.42
1 198.51.100.23
1 203.0.113.42Useless: every IP appears several times because uniq only merges lines that are next
to each other. Sort first, so identical lines become adjacent:
cut -d' ' -f1 access.log | sort | uniq -c 4 10.0.0.5
8 192.0.2.15
1 192.0.2.77
5 198.51.100.23
6 198.51.100.7
7 203.0.113.42
2 203.0.113.9Then sort the counts, numerically and largest first. This four-stage pattern,
sort | uniq -c | sort -rn, is the single most useful pipeline in this article. It
answers "what are the most common X?" for any X you can cut out of a line:
cut -d' ' -f1 access.log | sort | uniq -c | sort -rn 8 192.0.2.15
7 203.0.113.42
6 198.51.100.7
5 198.51.100.23
4 10.0.0.5
2 203.0.113.9
1 192.0.2.77uniq -d prints only lines that occur more than once (paths requested repeatedly), and
uniq -u only lines that occur exactly once. uniq -i ignores case.
cut -d' ' -f7 access.log | sort | uniq -d/api/orders
/healthz
/products/42
/wp-login.phpwc: count lines, words and bytes
wc access.log
wc -l access.log app.log
wc -l < access.log 33 497 4472 access.log
33 access.log
14 app.log
47 total
33Plain wc prints lines, words and bytes. -l, -w and -c select one. Reading from
stdin (< file) leaves the file name off, which is what you want inside $( ).
-c counts bytes and -m counts characters, which differ for anything outside
ASCII:
printf 'café\n' | wc -c
printf 'café\n' | wc -m6
5é is two bytes in UTF-8.
tr: translate, squeeze and delete characters
tr works on single characters and only reads stdin. It maps characters in the first
set to the corresponding ones in the second:
tr 'a-z' 'A-Z' <<< "post /api/orders"
head -n 3 orders.csv | tr ',' '\t'POST /API/ORDERS
order_id date region product qty unit_price
1001 2026-11-01 north keyboard 2 49.99
1002 2026-11-01 south monitor 1 189.00-s squeezes runs of a repeated character into one, which is the fix for cut on
space-aligned output:
echo "web01 up 12d" | tr -s ' '
df -P / | tr -s ' ' | cut -d' ' -f5web01 up 12d
Capacity
1%-d deletes characters. Its most common job is stripping the carriage returns from
files created on Windows (see Common mistakes): tr -d '\r'.
Add -c to complement the set, so tr -cd '[:print:]\n' deletes everything that isn't
printable.
head and tail: the start and end of a file
head -n N prints the first N lines and tail -n N the last N:
head -n 3 orders.csv
tail -n 2 app.logorder_id,date,region,product,qty,unit_price
1001,2026-11-01,north,keyboard,2,49.99
1002,2026-11-01,south,monitor,1,189.00
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
2026-11-09 09:11:02 INFO [http-8080-2] order 1046 placedtail -n +N means "starting at line N", so tail -n +2 skips a header line. You have
seen it throughout this article:
tail -n +2 orders.csv | head -n 21001,2026-11-01,north,keyboard,2,49.99
1002,2026-11-01,south,monitor,1,189.00head -c N takes bytes instead of lines.
Following a growing file: tail -f
tail -f prints the end of a file and then keeps waiting, printing new lines as they
are appended. It is how you watch a log while reproducing a bug. Here a background job
appends a line one second after tail starts, and timeout 3 stops tail after three
seconds (normally you press Ctrl-C):
cp app.log live.log
(sleep 1; echo "2026-11-09 10:20:00 ERROR [http-8080-1] order 1047 failed" >> live.log) &
timeout 3 tail -n 2 -f live.log at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
2026-11-09 09:11:02 INFO [http-8080-2] order 1046 placed
2026-11-09 10:20:00 ERROR [http-8080-1] order 1047 failedUse tail -F (capital F) for logs that get rotated: it notices when the file is renamed
and replaced, and reopens the new one. tail -f log | grep --line-buffered ERROR
filters live output; --line-buffered makes grep pass each line on immediately
instead of in blocks.
paste: join lines side by side
paste merges files line by line, separated by tabs:
printf '%s\n' web01 web02 db01 > hosts.txt
printf '%s\n' 10.0.1.11 10.0.1.12 10.0.2.21 > ips.txt
paste hosts.txt ips.txt
paste -d' ' ips.txt hosts.txtweb01 10.0.1.11
web02 10.0.1.12
db01 10.0.2.21
10.0.1.11 web01
10.0.1.12 web02
10.0.2.21 db01The second form is a ready-made /etc/hosts fragment. With -s (serial), paste
joins all lines of one input into a single line, which turns a column into a
comma-separated list or an arithmetic expression:
paste -sd, hosts.txt
tail -n +2 orders.csv | cut -d, -f5 | paste -sd+
tail -n +2 orders.csv | cut -d, -f5 | paste -sd+ | bcweb01,web02,db01
2+1+5+2+1+3+1+10+1+2+1+4
33bc is a calculator that reads an expression from stdin; 33 units were ordered in
total. (In the awk article the same sum is one
short command.) Each - argument reads one line from stdin, so paste - - - -
reassembles every four lines into one row:
printf '%s\n' host web01 status up host db01 status down | paste - - - -host web01 status up
host db01 status downcolumn: align output into a table
column -t lines up whitespace-separated columns; -s sets the input separator:
head -n 5 orders.csv | column -t -s,order_id date region product qty unit_price
1001 2026-11-01 north keyboard 2 49.99
1002 2026-11-01 south monitor 1 189.00
1003 2026-11-02 west mouse 5 19.50
1004 2026-11-02 north monitor 2 189.00It is the last stage of many report pipelines. Requests per client and status code:
cut -d' ' -f1,9 access.log | sort | uniq -c | sort -rn | head -n 6 | column -t8 192.0.2.15 404
5 203.0.113.42 200
4 10.0.0.5 200
3 198.51.100.7 200
2 203.0.113.9 200
2 198.51.100.7 304Where you'll use this
These filters do most of the "quick answer" work in operations and data handling:
- Incident response: who is hitting the server, which endpoint is failing, when it
started. The scanner at
192.0.2.15(eight 404s in three seconds probing for/wp-login.phpand/.env) is exactly what these pipelines expose. - Migrations and audits:
grep -rnacross a config tree before you rename a host, a database or an environment variable, so nothing is missed. - Data checks: distinct values in a CSV column, duplicate IDs (
sort | uniq -d), row counts before and after an import. - CI and scripts:
grep -qas a test,wc -las a threshold,sort -uto normalise lists before comparing them.
Here is a complete report script built from the pipelines above. Every section is one pipeline; the script adds headers and argument checking:
#!/usr/bin/env bash
# access-report.sh - quick traffic summary from an nginx/Apache combined-format log.
set -o pipefail
log=${1:?usage: access-report.sh ACCESS_LOG}
[[ -r $log ]] || { echo "cannot read $log" >&2; exit 1; }
total=$(wc -l < "$log")
clients=$(cut -d' ' -f1 "$log" | sort -u | wc -l)
echo "== $log: $total requests from $clients clients"
echo; echo "== Top 5 clients"
cut -d' ' -f1 "$log" | sort | uniq -c | sort -rn | head -n 5
echo; echo "== Status codes"
cut -d' ' -f9 "$log" | sort | uniq -c | sort -rn
echo; echo "== Requests per hour"
cut -d: -f2 "$log" | sort | uniq -c
echo; echo "== Top 404 paths"
grep '" 404 ' "$log" | cut -d' ' -f7 | sort | uniq -c | sort -rn | head -n 5
echo; echo "== Server errors (5xx)"
grep -E '" 5[0-9]{2} ' "$log" | cut -d' ' -f1,4,7,9 | tr -d '[' | column -tThe "requests per hour" line uses a trick worth stealing: with : as the delimiter,
field 2 of ...[09/Nov/2026:08:01:12 +0000]... is the hour.
./access-report.sh access.log== access.log: 33 requests from 7 clients
== Top 5 clients
8 192.0.2.15
7 203.0.113.42
6 198.51.100.7
5 198.51.100.23
4 10.0.0.5
== Status codes
15 200
9 404
3 201
2 304
1 502
1 500
1 302
1 301
== Requests per hour
20 08
8 09
5 10
== Top 404 paths
2 /wp-login.php
1 /xmlrpc.php
1 /wp-admin/
1 /phpmyadmin/
1 /favicon.ico
== Server errors (5xx)
198.51.100.23 09/Nov/2026:08:15:58 /api/orders 500
198.51.100.23 09/Nov/2026:09:10:44 /api/orders 502Reading it: a quarter of all traffic is a scanner, and both server errors are POST /api/orders from the same API client, 55 minutes apart. That is a lead for the payment
gateway timeout we saw in app.log. The ${1:?...} expansion makes the script refuse
to run without an argument:
./access-report.sh
./access-report.sh nope.log./access-report.sh: line 4: 1: usage: access-report.sh ACCESS_LOG
cannot read nope.logWhen a report like this grows conditions ("only count 5xx between 08:00 and 09:00", "sum bytes per client"), switch to awk; the awk article rewrites this report in one program. Scheduling it daily is covered in real-world scripts.
Common mistakes
Windows line endings break $ anchors
A CSV exported from a Windows tool ends every line with \r\n. The \r is invisible,
but it sits between your text and the end of the line:
printf 'order_id,region\r\n1001,north\r\n1002,south\r\n' > export.csv
grep -c 'north$' export.csv
cat -A export.csv0
order_id,region^M$
1001,north^M$
1002,south^M$cat -A shows the carriage return as ^M. Strip it and the anchor works:
tr -d '\r' < export.csv > export-unix.csv
grep -c 'north$' export-unix.csv1The same \r makes sort -u treat north and north\r as different values and makes
numeric comparisons fail. When results look impossible, cat -A is the first thing to
run.
cut on CSV with quoted fields
cat > customers.csv <<'EOF'
id,name,city
17,"Smith, Jane",Leeds
18,Raj Patel,Bristol
EOF
cut -d, -f3 customers.csvcity
Jane"
Bristolcut doesn't understand quotes, so the comma inside "Smith, Jane" shifted every
field on that line. If a CSV can contain quoted fields, use a real CSV parser. Python's
standard library has one and is installed almost everywhere:
python3 -c 'import csv, sys; [print(row[2]) for row in csv.reader(sys.stdin)]' < customers.csvcity
Leeds
BristolGNU awk can also handle simple quoted CSV (see awk).
Counting without sorting first
uniq -c without sort gave us eighteen lines of partial counts for seven IPs earlier.
The output looks plausible, which is what makes it dangerous. If the input is not
already grouped, the pattern is always sort | uniq -c.
Sorting numbers as text, and unbounded keys
sort without -n puts 100 before 25, and -k3 without ,3 silently ignores
every later key. Both were shown above with real output; the habit that prevents both is
writing every key as -kN,N plus a type letter: -k5,5n.
Matching inside other fields
grep 500 counted response sizes 15003 and 1500 as server errors. Anchor patterns
to the structure of the line ('" 500 '), use -w, or extract the field first with
cut and match on that.
Summary
grepoptions to know cold:-i -v -n -c -l -L -r -w -o -m -q, context with-A/-B/-C,-Efor extended regex,-Ffor literal text.- Basic regex treats
+ ? { } | ( )literally unless escaped;grep -Emakes them special. Escape dots (\.) and anchor (^,$, trailing space) when matching IPs and codes. cut -d X -fNextracts fields split on a single character; it can't reorder fields, handle runs of spaces (tr -s ' 'first) or quoted CSV.sort -nfor numbers,-hfor sizes,-Vfor versions,-t, -kN,Nfor columns (always bound the key),-sto keep input order for ties,-ufor distinct values.uniqonly merges adjacent lines:sort | uniq -c | sort -rnis the frequency-count pipeline.wc -l < filegives a bare count;trtranslates, squeezes (-s) and deletes (-d '\r');tail -n +2skips a header;tail -Ffollows rotated logs.paste -sd,joins lines,paste a bmerges files side by side,column -taligns the final output.
Exercises
How many requests came from 192.0.2.15, and which paths did it request, most frequent
first? Make sure your pattern can't match another address that merely starts with the
same digits.
Show Solution
grep -c '^192\.0\.2\.15 ' access.log
grep '^192\.0\.2\.15 ' access.log | cut -d' ' -f7 | sort | uniq -c | sort -rn8
2 /wp-login.php
1 /xmlrpc.php
1 /wp-admin/
1 /phpmyadmin/
1 /favicon.ico
1 /admin/config.php
1 /.git/config
1 /.envThe ^ and the trailing space pin the match to the whole first field; \. makes the
dots literal. Paths like /.env and /.git/config are classic probes for leaked
secrets.
The user agent is the last quoted string on each line and contains spaces, so splitting
on spaces won't extract it. Count requests per user agent, most frequent first, using
only cut, sort and uniq.
Hint
Which character surrounds the user agent? Count how many of them come before it.
Show Solution
cut -d'"' -f6 access.log | sort | uniq -c | sort -rn 8 python-requests/2.31.0
7 Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0
6 Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0.0.0 Safari/537.36
5 curl/8.5.0
4 kube-probe/1.30
2 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
1 Mozilla/5.0 (iPhone; CPU iPhone OS 18_1 like Mac OS X) Mobile/15E148 Safari/604.1With " as the delimiter, field 2 is the request, field 4 the referrer and field 6 the
user agent. Choosing the delimiter cleverly often saves a regex, the same trick the
report script uses with : to get the hour.
From orders.csv, list how many orders each product appeared in, highest count first,
with ties in alphabetical order. The header line must not be counted.
Show Solution
tail -n +2 orders.csv | cut -d, -f4 | sort | uniq -c | sort -k1,1nr -k2,2 4 keyboard
3 headset
3 monitor
2 mouseThe final sort has two keys: field 1 (the count) numerically reversed, then field 2 (the
name) as text. uniq -c output is separated by spaces, which is sort's default
separator, so no -t is needed.
Produce an aligned table of how many 4xx and 5xx responses each client received, broken down by status code, largest count first.
Show Solution
grep -E '" [45][0-9]{2} ' access.log | cut -d' ' -f1,9 | sort | uniq -c | sort -rn | column -t8 192.0.2.15 404
1 198.51.100.23 502
1 198.51.100.23 500
1 192.0.2.77 404'" [45][0-9]{2} ' matches a status from 400 to 599 in the right position, cut keeps
IP and status, sort | uniq -c counts each pair, sort -rn orders by count, and
column -t aligns the result.
Tested with: GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), November 2026, with GNU grep
3.11, GNU coreutils 9.4 (cut, sort, uniq, wc, tr, head, tail, paste) and
column from util-linux. Every command was run against the sample files above and the
output pasted unchanged; the shell's locale was C.UTF-8, which is why uppercase sorts
before lowercase. The /usr/share sizes come from that machine and will differ on
yours.
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.