awk: Processing Columns and Records
Use awk to filter, sum, count and reformat columns in logs, CSV files and command output, from one-line field extraction to complete single-pass reports.
By The Dark DebuggerUpdated 29 min read
awk is a small programming language built for one job: reading text as records
(lines) made of fields (columns), and doing something with the fields. This article
covers how awk splits input, patterns and actions, BEGIN and END, formatted output,
variables, associative arrays for counting and grouping, string functions, rebuilding
records, and joining two files. It assumes you know the filters in
searching and filtering text and the regular
expressions used by sed. After this, the path moves
on to writing scripts.
Why awk
In the previous article a sort | uniq -c pipeline
told us which clients made the most requests. Now the question changes: which clients
downloaded the most bytes? That needs a sum per client, and none of cut, sort or
uniq can add numbers. You could write a Bash loop, but it would start several
processes per line and crawl on a real log.
In awk it's one line, and it runs in a single pass over the file:
awk '{ sent[$1] += $10 } END { for (ip in sent) printf "%8d %s\n", sent[ip], ip }' access.log | sort -rn 29152 203.0.113.42
19094 198.51.100.7
7409 203.0.113.9
2563 198.51.100.23
1296 192.0.2.15
162 192.0.2.77
8 10.0.0.5By the end of this article you'll be able to read every part of that line, and write the longer versions you need for real reports.
The examples use the access.log, app.log and orders.csv sample files created in
searching and filtering text, and the
app.conf file from the sed article. The system
awk on Ubuntu here is GNU awk (gawk); where something is gawk-only, I say so.
Records and fields
awk reads its input one record at a time (by default, one line) and splits each record into fields (by default, separated by runs of spaces and tabs). Inside the program:
| Name | Meaning |
|---|---|
$0 | the whole current record |
$1, $2, ... | field 1, field 2, ... |
NF | number of fields in the current record |
$NF | the last field; $(NF-1) is the one before it |
NR | number of records read so far, across all files |
FNR | record number within the current file |
FILENAME | name of the current input file |
An awk program is a list of pattern { action } pairs. The simplest program is an
action with no pattern, which runs for every line:
awk '{ print $1 }' access.log | head -n 3
awk '{ print $1, $9, $10 }' access.log | head -n 3203.0.113.42
203.0.113.42
10.0.0.5
203.0.113.42 200 5120
203.0.113.42 200 1834
10.0.0.5 200 2The comma in print $1, $9 matters: it inserts the output field separator (a space
by default). Without it, awk concatenates the values:
awk '{ print $1 $9 }' access.log | head -n 2203.0.113.42200
203.0.113.42200NF varies from line to line. In this log the user agent has a different number of
words per browser:
awk '{ print NF }' access.log | sort -n | uniq -c 17 12
2 15
7 18
6 19
1 23That's why fields at the end of a log line are addressed from the end, with $NF:
awk '{ print $(NF-1), $NF }' access.log | head -n 1Gecko/20100101 Firefox/131.0"Default splitting beats cut on aligned output
With the default separator, awk treats any run of spaces and tabs as one separator and
ignores leading and trailing whitespace. That is exactly what command output aligned
into columns needs, and exactly what cut gets wrong:
echo " web01 up 12d " | awk '{ print $2 }'
echo " web01 up 12d " | cut -d' ' -f2up
cut returned the empty string between the two leading spaces. This is why awk is the
usual tool for parsing ps, df, ls -l and docker ps output.
Choosing the field separator: -F
-F sets the input field separator. A single character is used literally:
awk -F, '{ print $3 }' orders.csv | head -n 3
awk -F: '{ print $1, $7 }' /etc/passwd | head -n 3region
north
south
root /bin/bash
daemon /usr/sbin/nologin
bin /usr/sbin/nologinSplitting on the double quote is a neat trick for this log format: field 2 is the request, 4 the referrer, 6 the user agent.
awk -F'"' '{ print $2 }' access.log | head -n 2GET / HTTP/1.1
GET /css/main.css HTTP/1.1A longer separator is a regular expression. [][] is a bracket expression
containing ] and [, so this splits on either square bracket and pulls out the
timestamp:
awk -F'[][]' '{ print $2 }' access.log | head -n 209/Nov/2026:08:01:12 +0000
09/Nov/2026:08:01:13 +0000The output separator is the variable OFS. Set it with -v:
awk -F, -v OFS='\t' '{ print $1, $3, $4 }' orders.csv | head -n 3order_id region product
1001 north keyboard
1002 south monitorPatterns: choosing which lines to act on
A pattern in front of an action limits it to matching lines. A pattern with no action
prints the matching lines, like grep. Patterns can be:
- a comparison on fields:
$9 >= 500,$3 == "north" - a regex against the whole line:
/POST/ - a regex against one field, with
~(matches) or!~(doesn't match):$7 ~ /^\/api\// - a combination with
&&,||and! - a range,
pattern1, pattern2, like sed's ranges
awk '$9 >= 500 { print $1, $7, $9 }' access.log198.51.100.23 /api/orders 500
198.51.100.23 /api/orders 502This is the precise version of what took grep -E '" 5[0-9]{2} ' in the previous
article: a numeric comparison on exactly the status field, with no risk of matching a
response size.
awk '$7 ~ /^\/api\// { print $1, $6, $7, $9 }' access.log198.51.100.23 "POST /api/orders 201
198.51.100.23 "POST /api/orders 500
203.0.113.42 "POST /api/orders 201
198.51.100.23 "POST /api/orders 502
198.51.100.23 "POST /api/orders 201
198.51.100.23 "GET /api/orders/1042 200awk -F, '$3 == "north" && $5 >= 2' orders.csv1001,2026-11-01,north,keyboard,2,49.99
1004,2026-11-02,north,monitor,2,189.00
1008,2026-11-04,north,mouse,10,19.50Ranges work with any patterns, including line numbers:
awk 'NR == 2, NR == 4' orders.csv
awk '/^\[database\]/,/^$/' app.conf1001,2026-11-01,north,keyboard,2,49.99
1002,2026-11-01,south,monitor,1,189.00
1003,2026-11-02,west,mouse,5,19.50
[database]
host = db01.internal
port = 5432
name = orders
user = orders_app
max_connections = 50
Note that strings in awk use double quotes, so "north" inside a single-quoted
program is fine.
BEGIN and END
BEGIN { ... } runs before the first line is read, and END { ... } after the last.
BEGIN is for headers and setup; END is where totals get printed:
awk 'BEGIN { print "ip,status" } { print $1 "," $9 }' access.log | head -n 3
awk 'END { print NR }' access.logip,status
203.0.113.42,200
203.0.113.42,200
33Variables need no declaration, and an unset variable is 0 in a numeric context and
"" in a string context. So summing a column is just +=:
awk '{ bytes += $10 } END { print bytes, "bytes" }' access.log59684 bytesprintf: formatted output
print is quick; printf gives you control. It works like C's printf: a format string,
then the values. It does not add a newline, so end the format with \n:
awk '{ bytes += $10 } END { printf "%.1f KiB in %d requests\n", bytes / 1024, NR }' access.log58.3 KiB in 33 requests| Format | Prints |
|---|---|
%s | a string |
%d | an integer (decimals are truncated) |
%.2f | a number with 2 decimal places |
%8d, %8.2f | right-aligned in 8 characters |
%-10s | left-aligned in 10 characters |
%% | a literal % |
Widths turn output into aligned columns:
awk -F, 'NR > 1 { printf "%-6s %-9s %3d x %7.2f = %8.2f\n", $1, $4, $5, $6, $5 * $6 }' orders.csv | head -n 41001 keyboard 2 x 49.99 = 99.98
1002 monitor 1 x 189.00 = 189.00
1003 mouse 5 x 19.50 = 97.50
1004 monitor 2 x 189.00 = 378.00sprintf works the same way but returns the string instead of printing it, which you
need when building a value.
Passing values in from the shell
Never paste a shell variable into the program text. Pass it with -v, which defines an
awk variable before the program starts:
min=5000
awk -v limit="$min" '$10 > limit { print $10, $7 }' access.log5120 /
7342 /products/42
15003 /products?page=2
7342 /products/42
7342 /products/42Inside awk, limit is just a variable; the program stays in single quotes and nothing
is re-interpreted by the shell. Environment variables are also readable through the
ENVIRON array:
STATUS=502 awk '$9 == ENVIRON["STATUS"] { print $1, $4 }' access.log198.51.100.23 [09/Nov/2026:09:10:44What goes wrong when you don't do this is in Common mistakes.
Arrays: counting, summing, grouping
awk arrays are associative: the index can be any string, such as an IP address, a
status code or a region. You don't declare them or size them; referencing
count["x"] creates the element with value 0. That makes three idioms trivial:
- count:
count[key]++ - sum:
total[key] += value - group:
list[key] = list[key] " " value
for (key in array) loops over the keys. Requests per client:
awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' access.log8 192.0.2.15
6 198.51.100.7
4 10.0.0.5
1 192.0.2.77
7 203.0.113.42
2 203.0.113.9
5 198.51.100.23That is the same result as sort | uniq -c, in no particular order. The order of
for (key in array) is unspecified: not insertion order, not sorted. Pipe the output
through sort:
awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' access.log | sort -rn8 192.0.2.15
7 203.0.113.42
6 198.51.100.7
5 198.51.100.23
4 10.0.0.5
2 203.0.113.9
1 192.0.2.77gawk can sort the loop itself through the special PROCINFO["sorted_in"] setting; that
is gawk-only, so I use sort in anything that might run on another awk:
awk '{ count[$1]++ } END { PROCINFO["sorted_in"] = "@val_num_desc"; for (ip in count) print count[ip], ip }' access.log | head -n 38 192.0.2.15
7 203.0.113.42
6 198.51.100.7Summing works the same way. Revenue per region is quantity times unit price, summed per
region, skipping the header with NR > 1:
awk -F, 'NR > 1 { revenue[$3] += $5 * $6 } END { for (r in revenue) printf "%-6s %8.2f\n", r, revenue[r] }' orders.csv | sort -k2,2nrnorth 751.98
south 496.97
west 486.46
east 128.99Keys can combine several fields. Concatenate them into one string:
awk '{ pair[$1 " " $9]++ } END { for (k in pair) print pair[k], k }' access.log | sort -rn | head -n 48 192.0.2.15 404
5 203.0.113.42 200
4 10.0.0.5 200
3 198.51.100.7 200The !seen[$0]++ idiom
This one-liner removes duplicate lines while keeping the original order, which
sort -u can't do:
awk '!seen[$1]++ { print $1 }' access.log203.0.113.42
10.0.0.5
198.51.100.7
192.0.2.15
198.51.100.23
203.0.113.9
192.0.2.77Read it slowly: seen[$1]++ returns the old value (0 the first time an IP appears,
then 1, 2, ...) and increments it. !0 is true, so the pattern matches only the first
time each key is seen. Written as awk '!seen[$0]++' file, it de-duplicates whole lines.
To test whether a key exists without creating it, use (key in array); that matters in
the two-file join below.
Conditions and control flow
Actions can use if/else, while, for and the ternary operator, with C syntax:
awk '{ if ($9 >= 500) class = "server error"; else if ($9 >= 400) class = "client error"; else class = "ok"; print class }' access.log | sort | uniq -c 9 client error
22 ok
2 server errorawk '{ print $1, ($9 >= 400 ? "FAIL" : "ok") }' access.log | head -n 7 | tail -n 3198.51.100.7 ok
192.0.2.15 FAIL
192.0.2.15 FAILTwo statements control the flow through the input:
nextstops processing the current line and moves to the next one.NR == 1 { next }skips a header.exitstops reading input entirely (theENDblock still runs). Use it to stop at the first match in a huge file.
awk -F, 'NR == 1 { next } { n++ } END { print n, "orders" }' orders.csv
awk '$9 >= 500 { print "first server error on line " NR ": " $7; exit }' access.log12 orders
first server error on line 17: /api/ordersA variable used as a flag tracks state across lines. This prints the [database]
section of app.conf as environment variables. -F' = ' splits on the literal
= string; every header line resets the flag, and only lines in the right section
with exactly two fields are printed:
awk -F' = ' '/^\[/ { in_db = ($0 == "[database]"); next } in_db && NF == 2 { print "DB_" toupper($1) "=" $2 }' app.confDB_HOST=db01.internal
DB_PORT=5432
DB_NAME=orders
DB_USER=orders_app
DB_MAX_CONNECTIONS=50Compare that with the nested sed braces needed for the same section in the
sed article. Once logic depends on state, awk reads
better.
String functions
| Function | Returns |
|---|---|
length(s) | length of s (length alone means length($0)) |
substr(s, start, len) | substring; positions start at 1 |
index(s, t) | position of t in s, or 0 |
split(s, arr, sep) | splits s into arr[1], arr[2]...; returns the count |
sub(re, rep, target) | replaces the first match in target (default $0); returns 1 or 0 |
gsub(re, rep, target) | replaces every match |
tolower(s), toupper(s) | case conversion |
match(s, re) | position of the match, and sets RSTART and RLENGTH |
sprintf(fmt, ...) | formatted string |
The hour is characters 14 and 15 of field 4 ([09/Nov/2026:08:01:12), or the second
piece when field 4 is split on colons. Both give requests per hour:
awk '{ print substr($4, 14, 2) }' access.log | sort | uniq -c
awk '{ split($4, t, ":"); print t[2] }' access.log | sort | uniq -c 20 08
8 09
5 10
20 08
8 09
5 10split is the more robust of the two: it doesn't break if a field's width changes.
Its return value is the number of pieces, here the depth of each path plus one:
awk '{ n = split($7, parts, "/"); print n, $7 }' access.log | head -n 42 /
3 /css/main.css
2 /healthz
3 /products/42sub and gsub edit a field (or $0) in place. Strip the bracket from the timestamp,
and normalise the method's case after removing its quote:
awk '{ sub(/^\[/, "", $4); print $4 }' access.log | head -n 2
awk '{ gsub(/"/, "", $6); print tolower($6) }' access.log | sort | uniq -c09/Nov/2026:08:01:12
09/Nov/2026:08:01:13
28 get
5 postmatch finds a regex anywhere in a string and sets RSTART (where it starts) and
RLENGTH (how long it is), which substr then uses. Every order number mentioned in
app.log:
awk 'match($0, /order [0-9]+/) { print substr($0, RSTART + 6, RLENGTH - 6) }' app.log | sort -u1043
1044
1045
1046The + 6 and - 6 skip the six characters of order . gawk has a cleaner form: a third
argument to match receives the capture groups. It's convenient, but it's a gawk
extension. Here are gawk and then mawk (the default awk on some Debian-based images)
running the same program:
awk 'match($0, /order ([0-9]+) failed/, m) { print m[1] }' app.log
mawk 'match($0, /order ([0-9]+) failed/, m) { print m[1] }' app.log1043
1045
mawk: line 1: syntax error at or near ,If a script might run in a minimal container, check which awk it has (awk --version
for gawk, awk -W version for mawk) or stick to POSIX functions.
Changing fields: OFS and rebuilding $0
You can assign to fields. When you do, awk rebuilds $0 by joining all fields with
OFS. Until you change a field, $0 is the original line untouched, which surprises
everyone the first time:
head -n 3 orders.csv | awk -F, -v OFS='\t' '{ print }'
head -n 3 orders.csv | awk -F, -v OFS='\t' '{ $1 = $1; print }'order_id,date,region,product,qty,unit_price
1001,2026-11-01,north,keyboard,2,49.99
1002,2026-11-01,south,monitor,1,189.00
order_id date region product qty unit_price
1001 2026-11-01 north keyboard 2 49.99
1002 2026-11-01 south monitor 1 189.00$1 = $1 changes nothing visible but forces the rebuild, so the commas become tabs.
That is the standard way to convert separators.
Adding a computed column is print $0, value. Handle the header separately, and format
numbers yourself, because awk prints non-integers with up to six significant digits and
drops trailing zeros:
head -n 4 orders.csv | awk -F, -v OFS=, 'NR == 1 { print $0, "total"; next } { print $0, $5 * $6 }'
head -n 4 orders.csv | awk -F, -v OFS=, 'NR == 1 { print $0, "total"; next } { print $0, sprintf("%.2f", $5 * $6) }'order_id,date,region,product,qty,unit_price,total
1001,2026-11-01,north,keyboard,2,49.99,99.98
1002,2026-11-01,south,monitor,1,189.00,189
1003,2026-11-02,west,mouse,5,19.50,97.5
order_id,date,region,product,qty,unit_price,total
1001,2026-11-01,north,keyboard,2,49.99,99.98
1002,2026-11-01,south,monitor,1,189.00,189.00
1003,2026-11-02,west,mouse,5,19.50,97.50Forget the header guard and awk does arithmetic on it. A 10% price rise:
head -n 3 orders.csv | awk -F, -v OFS=, '{ $6 = $6 * 1.10; print }'order_id,date,region,product,qty,0
1001,2026-11-01,north,keyboard,2,54.989
1002,2026-11-01,south,monitor,1,207.9The header's unit_price became 0 (a non-numeric string is 0 in arithmetic), and the
prices need sprintf("%.2f", ...). Assigning to NF truncates (or extends) the
record:
head -n 3 orders.csv | awk -F, -v OFS=, '{ NF = 3; print }'order_id,date,region
1001,2026-11-01,north
1002,2026-11-01,southProcessing two files: NR == FNR
awk reads every file on its command line in turn. NR keeps counting across files, but
FNR restarts at 1 for each file:
awk '{ print NR, FNR, FILENAME }' orders.csv customers.csv | sed -n '12,15p'12 12 orders.csv
13 13 orders.csv
14 1 customers.csv
15 2 customers.csvSo NR == FNR is true only while reading the first file. That gives the classic
join idiom: load the first file into an array, next, then use the array while
processing the second. Here is a cost list per product:
cat > products.csv <<'EOF'
product,category,unit_cost
keyboard,peripherals,31.00
monitor,displays,142.00
mouse,peripherals,8.75
headset,audio,51.50
EOFJoin it to the orders to compute the margin on each order:
awk -F, 'NR == FNR { cost[$1] = $3; next } FNR > 1 { printf "%s %-9s margin %7.2f\n", $1, $4, ($6 - cost[$4]) * $5 }' products.csv orders.csv | head -n 41001 keyboard margin 37.98
1002 monitor margin 47.00
1003 mouse margin 53.75
1004 monitor margin 94.00The first block runs for products.csv only and builds cost["keyboard"] = 31.00 and
so on (the header line also loads a harmless cost["product"]). next stops those
lines reaching the second block. FNR > 1 skips the orders header.
Real data has gaps. Add an order for a product with no cost entry and check for it with
in, which tests for a key without creating it:
cp orders.csv orders-new.csv && echo "1013,2026-11-07,east,webcam,2,64.00" >> orders-new.csv
awk -F, 'NR == FNR { cost[$1] = $3; next } FNR > 1 && !($4 in cost) { print "no cost for product \"" $4 "\" in order " $1 }' products.csv orders-new.csvno cost for product "webcam" in order 1013Without that check, cost["webcam"] would silently be 0 and the webcam order would
show a margin equal to its full price.
Quoted CSV with gawk's FPAT
-F, breaks on CSV fields that contain commas inside quotes, just like cut. gawk can
describe fields by what they contain instead of what separates them, with FPAT: a
field is either a run of non-commas or a quoted string:
awk -F, '{ print $3 }' customers.csv
awk -v FPAT='([^,]*)|("[^"]*")' '{ print $3 }' customers.csv
awk -v FPAT='([^,]*)|("[^"]*")' '{ print $2 }' customers.csvcity
Jane"
Bristol
city
Leeds
Bristol
name
"Smith, Jane"
Raj PatelThe quotes stay part of the field, and this pattern doesn't handle escaped quotes ("")
or newlines inside fields. It's fine for well-behaved exports; for anything else, use a
real CSV parser such as Python's csv module. (gawk 5.3 added a --csv option; the
5.2 release tested here doesn't have it.)
awk programs in files
Once a program spans more than a line or two, put it in a file and run it with
awk -f. A #!/usr/bin/awk -f first line makes it directly executable. The
Where you'll use this section has a complete example.
When to use awk, and when not
| Task | Best tool |
|---|---|
| Find lines containing text | grep |
| Extract a column split on one character | cut (or awk) |
| Columns separated by runs of spaces | awk |
| Substitute text, edit files in place | sed |
| Count occurrences of a value | sort | uniq -c, or awk |
| Sum, average, min or max per group | awk |
| Conditions across several columns | awk |
| Join two files on a key | awk (or join if both are sorted) |
| Nested data (JSON), real CSV, dates, HTTP calls | a real language: Python, or jq for JSON |
| More than ~30 lines of awk | probably a real language |
The deciding question is usually: am I doing arithmetic or keeping state across lines? If yes, awk. If the program needs data structures beyond one level of arrays, error handling, or tests, it has outgrown awk.
Where you'll use this
- Log reports: per-client bytes, error rates per hour, slow-request lists, all in one pass over files too big for a spreadsheet.
- CSV and TSV reshaping: aggregating exports, reordering or adding columns, converting separators.
- Parsing command output in monitoring scripts:
df,ps,free,ss,docker stats. - Quick joins between an inventory list and a status list, or orders and prices.
A one-pass access log report
The previous article built a report from five
pipelines, each reading the whole log. This awk program reads it once and adds things
pipelines can't do easily: bytes sent and an error percentage. Save it as
access-report.awk:
#!/usr/bin/awk -f
# access-report.awk - one-pass summary of a combined-format access log.
# usage: awk -f access-report.awk access.log
{
requests++
bytes += $10
hits[$1]++
sent[$1] += $10
status_class[substr($9, 1, 1) "xx"]++
if ($9 >= 500) server_errors++
hour[substr($4, 14, 2)]++
}
END {
if (requests == 0) { print "no requests"; exit 1 }
printf "%d requests from %d clients, %.1f KiB sent, %.1f%% server errors\n",
requests, length(hits), bytes / 1024, 100 * server_errors / requests
print "\nby status class:"
for (c in status_class) printf " %s %4d\n", c, status_class[c] | "sort"
close("sort")
print "\nby hour:"
for (h in hour) printf " %s:00 %4d\n", h, hour[h] | "sort"
close("sort")
print "\ntop clients by requests:"
cmd = "sort -k2,2nr | head -n 5"
for (ip in hits) printf " %-15s %3d req %7d bytes\n", ip, hits[ip], sent[ip] | cmd
close(cmd)
}awk -f access-report.awk access.log33 requests from 7 clients, 58.3 KiB sent, 6.1% server errors
by status class:
2xx 18
3xx 4
4xx 9
5xx 2
by hour:
08:00 20
09:00 8
10:00 5
top clients by requests:
192.0.2.15 8 req 1296 bytes
203.0.113.42 7 req 29152 bytes
198.51.100.7 6 req 19094 bytes
198.51.100.23 5 req 2563 bytes
10.0.0.5 4 req 8 bytesThree techniques here are worth stealing:
printf ... | "sort"sends output to a command, and awk keeps that pipe open for later prints using the same command string.close()flushes it, so the sorted section appears before the next heading.length(hits)is the number of keys in an array. It's supported by gawk and mawk (when I ran this file withmawk -f, the first line came out identical), though older awks lack it.- The
requests == 0guard: on an empty log the percentage would divide by zero.
chmod +x access-report.awk
./access-report.awk access.log | head -n 1
awk -f access-report.awk /dev/null
echo "exit: $?"33 requests from 7 clients, 58.3 KiB sent, 6.1% server errors
no requests
exit: 1CSV aggregation
Orders, revenue, average and largest order per region, with a header row:
awk -F, 'NR > 1 { n[$3]++; total[$3] += $5 * $6; if ($5 * $6 > max[$3]) max[$3] = $5 * $6 } END { printf "%-6s %6s %9s %9s %9s\n", "region", "orders", "revenue", "average", "largest"; for (r in n) printf "%-6s %6d %9.2f %9.2f %9.2f\n", r, n[r], total[r], total[r] / n[r], max[r] }' orders.csvregion orders revenue average largest
east 2 128.99 64.50 79.00
west 3 486.46 162.15 199.96
south 3 496.97 165.66 189.00
north 4 751.98 188.00 378.00Parsing df for a disk-space alert
A cron job that warns when any filesystem crosses a threshold is a classic awk task.
Here is real df output from my test machine, limited to ext4 and tmpfs filesystems:
df -P -t ext4 -t tmpfsFilesystem 1024-blocks Used Available Capacity Mounted on
/dev/sdb 98720 98720 0 100% /usr/lib/linux-tools/6.18.54.1-microsoft-standard-WSL2
none 4019036 4 4019032 1% /mnt/wsl
/dev/sdd 1055762868 4741336 997318060 1% /
none 4019036 32 4019004 1% /mnt/wslg
none 4019036 520 4018516 1% /run
none 4019036 0 4019036 0% /run/lock
none 4019036 67184 3951852 2% /run/shm
tmpfs 803804 24 803780 1% /run/user/1000One filesystem is genuinely 100% full (a small read-only image the system mounts). The
alert, with the threshold passed in by -v:
df -P -t ext4 -t tmpfs | awk -v limit=90 'NR > 1 && $5 + 0 >= limit { printf "%s is %s full (%d MiB free)\n", $6, $5, $4 / 1024 }'/usr/lib/linux-tools/6.18.54.1-microsoft-standard-WSL2 is 100% full (0 MiB free)-P asks df for the POSIX output format, which keeps each filesystem on one line so
the field numbers are stable. The + 0 is not decoration; the next section shows what
happens without it. Scheduling a check like this is covered in
system monitoring and cron.
Common mistakes
Comparing 85% with a number
The obvious alert filter on the same df output:
df -P -t ext4 -t tmpfs | awk '$5 > 80 { print $6, $5 }'Mounted CapacityIt printed the header and missed the full filesystem. 100% doesn't look like a
number, so awk compared it to 80 as a string: "1" sorts before "8", so
"100%" > "80" is false, while "Capacity" > "80" is true because letters sort after
digits. For the same reason a filesystem at 9% would trigger the alert:
echo "9%" | awk '$1 > 80 { print $1, "looks bigger than 80" }'9% looks bigger than 80$5 + 0 forces a numeric conversion (awk reads the leading 100 and ignores the %),
and NR > 1 skips the header. When a comparison gives impossible results, check whether
the field is really a number.
Double quotes around the program
awk "{ print $1 }" access.log | head -n 1203.0.113.42 - - [09/Nov/2026:08:01:12 +0000] "GET / HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"Inside double quotes, the shell expanded $1 (its own first argument, empty at the
prompt) before awk started, so awk ran { print } and printed whole lines. Always put
awk programs in single quotes and bring shell values in with -v.
Counting the header
Average quantity per order, first by dividing by NR:
awk -F, '{ sum += $5 } END { print sum / NR }' orders.csv
awk -F, 'NR > 1 { sum += $5; n++ } END { print sum / n }' orders.csv2.53846
2.75The header contributes 0 to the sum but 1 to NR, so the first average is wrong, and
nothing about the number says so. Skip the header and count the rows you actually
used.
Dividing by zero when nothing matched
awk -F, '$3 == "central" { sum += $5; n++ } END { print sum / n }' orders.csvawk: cmd. line:1: (FILENAME=orders.csv FNR=13) fatal: division by zero attemptedNo region is called central, so n was never set. In a monitoring script this error
replaces the output you were waiting for. Guard the division:
awk -F, '$3 == "central" { sum += $5; n++ } END { if (n) print sum / n; else print "no orders for central" }' orders.csvno orders for centralExpecting for (k in a) to be sorted
The unsorted loop output earlier came out in neither insertion nor sorted order. Treat
the order as random and pipe through sort (or use gawk's PROCINFO["sorted_in"] when
you know the script runs on gawk).
Summary
- awk splits each line into fields:
$1...$NF, with$0the whole line,NFthe field count,NRthe line number (andFNRper file). Default splitting handles runs of whitespace;-Fsets a character or a regex. - A program is
pattern { action }pairs; a pattern alone prints matches, an action alone runs on every line.BEGINruns before input,ENDafter. print a, busesOFS;print a bconcatenates.printfgives widths and decimals and needs an explicit\n.- Associative arrays do the heavy lifting:
count[k]++,sum[k] += v,!seen[k]++,(k in arr). Loop order is unspecified; sort the output. - Pass shell values with
-v name="$value"; keep the program in single quotes. - Assigning to a field rebuilds
$0withOFS;$1 = $1converts separators. Format numbers withsprintf("%.2f")and guard headers withNR > 1. NR == FNR { ...; next }loads the first file to join against the second.- Force numbers with
+ 0when fields carry units like%, and guard divisions.
Exercises
Using /etc/passwd (fields separated by :; field 1 is the user, 6 the home directory,
7 the shell), print the user name and home directory of every account whose shell is
exactly /bin/bash.
Show Solution
awk -F: '$7 == "/bin/bash" { print $1, $6 }' /etc/passwdroot /root
sid /home/sid== compares the whole field, so /bin/bash doesn't match /usr/bin/bash or
/bin/bash5. Your output will list the accounts on your own machine.
Total the response bytes (field 10) for each HTTP status code in access.log, largest
total first.
Show Solution
awk '{ sent[$9] += $10 } END { for (s in sent) print sent[s], s }' access.log | sort -rn56765 200
1458 404
1215 201
157 502
89 500
0 304
0 302
0 301Redirects (301, 302) and "not modified" responses (304) send no body, so they total 0.
For each region in orders.csv, print the average order value (quantity times unit
price), with two decimal places, sorted by region name.
Show Solution
awk -F, 'NR > 1 { n[$3]++; total[$3] += $5 * $6 } END { for (r in n) printf "%-6s %7.2f\n", r, total[r] / n[r] }' orders.csv | sorteast 64.50
north 188.00
south 165.66
west 162.15Two arrays keyed by region: one counts orders, one sums values. Dividing in END
gives the average. n[r] can't be zero here, because a key only exists if an order
created it.
For each hour in access.log, print the number of requests, the number of error
responses (status 400 or above) and the error percentage to one decimal place, in hour
order.
Hint
Use one array for totals and one for errors, both keyed by the hour (substr($4, 14, 2)).
An hour with no errors has no entry in the errors array; what does awk return for it?
Show Solution
awk '{ h = substr($4, 14, 2); total[h]++; if ($9 >= 400) errors[h]++ } END { for (h in total) printf "%s:00 %3d requests %3d errors %5.1f%%\n", h, total[h], errors[h], 100 * errors[h] / total[h] }' access.log | sort08:00 20 requests 9 errors 45.0%
09:00 8 requests 2 errors 25.0%
10:00 5 requests 0 errors 0.0%Looping over total (not errors) guarantees every hour appears; errors["10"] was
never set, so it evaluates to 0, which %3d prints as 0. The 08:00 spike is the
scanner's 404 burst.
Tested with: GNU Awk 5.2.1 and GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), November
2026, plus mawk for the portability checks. Every command was run as shown against the
sample files and the output pasted unchanged. The df output is from the WSL2 test
machine itself, which is why the mount points look unusual; the 100% filesystem is
WSL's read-only tools image, which made a convenient real example of a full disk.
Written by The Dark Debugger
Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.