awk: Processing Columns and Records

Use awk to filter, sum, count and reformat columns in logs, CSV files and command output, from one-line field extraction to complete single-pass reports.

By The Dark DebuggerUpdated 29 min read

awk is a small programming language built for one job: reading text as records (lines) made of fields (columns), and doing something with the fields. This article covers how awk splits input, patterns and actions, BEGIN and END, formatted output, variables, associative arrays for counting and grouping, string functions, rebuilding records, and joining two files. It assumes you know the filters in searching and filtering text and the regular expressions used by sed. After this, the path moves on to writing scripts.

Why awk

In the previous article a sort | uniq -c pipeline told us which clients made the most requests. Now the question changes: which clients downloaded the most bytes? That needs a sum per client, and none of cut, sort or uniq can add numbers. You could write a Bash loop, but it would start several processes per line and crawl on a real log.

In awk it's one line, and it runs in a single pass over the file:

awk '{ sent[$1] += $10 } END { for (ip in sent) printf "%8d %s\n", sent[ip], ip }' access.log | sort -rn
   29152 203.0.113.42
   19094 198.51.100.7
    7409 203.0.113.9
    2563 198.51.100.23
    1296 192.0.2.15
     162 192.0.2.77
       8 10.0.0.5

By the end of this article you'll be able to read every part of that line, and write the longer versions you need for real reports.

The examples use the access.log, app.log and orders.csv sample files created in searching and filtering text, and the app.conf file from the sed article. The system awk on Ubuntu here is GNU awk (gawk); where something is gawk-only, I say so.

Records and fields

awk reads its input one record at a time (by default, one line) and splits each record into fields (by default, separated by runs of spaces and tabs). Inside the program:

NameMeaning
$0the whole current record
$1, $2, ...field 1, field 2, ...
NFnumber of fields in the current record
$NFthe last field; $(NF-1) is the one before it
NRnumber of records read so far, across all files
FNRrecord number within the current file
FILENAMEname of the current input file

An awk program is a list of pattern { action } pairs. The simplest program is an action with no pattern, which runs for every line:

awk '{ print $1 }' access.log | head -n 3
awk '{ print $1, $9, $10 }' access.log | head -n 3
203.0.113.42
203.0.113.42
10.0.0.5
203.0.113.42 200 5120
203.0.113.42 200 1834
10.0.0.5 200 2

The comma in print $1, $9 matters: it inserts the output field separator (a space by default). Without it, awk concatenates the values:

awk '{ print $1 $9 }' access.log | head -n 2
203.0.113.42200
203.0.113.42200

NF varies from line to line. In this log the user agent has a different number of words per browser:

awk '{ print NF }' access.log | sort -n | uniq -c
     17 12
      2 15
      7 18
      6 19
      1 23

That's why fields at the end of a log line are addressed from the end, with $NF:

awk '{ print $(NF-1), $NF }' access.log | head -n 1
Gecko/20100101 Firefox/131.0"

Default splitting beats cut on aligned output

With the default separator, awk treats any run of spaces and tabs as one separator and ignores leading and trailing whitespace. That is exactly what command output aligned into columns needs, and exactly what cut gets wrong:

echo "  web01    up     12d  " | awk '{ print $2 }'
echo "  web01    up     12d  " | cut -d' ' -f2
up
 

cut returned the empty string between the two leading spaces. This is why awk is the usual tool for parsing ps, df, ls -l and docker ps output.

Choosing the field separator: -F

-F sets the input field separator. A single character is used literally:

awk -F, '{ print $3 }' orders.csv | head -n 3
awk -F: '{ print $1, $7 }' /etc/passwd | head -n 3
region
north
south
root /bin/bash
daemon /usr/sbin/nologin
bin /usr/sbin/nologin

Splitting on the double quote is a neat trick for this log format: field 2 is the request, 4 the referrer, 6 the user agent.

awk -F'"' '{ print $2 }' access.log | head -n 2
GET / HTTP/1.1
GET /css/main.css HTTP/1.1

A longer separator is a regular expression. [][] is a bracket expression containing ] and [, so this splits on either square bracket and pulls out the timestamp:

awk -F'[][]' '{ print $2 }' access.log | head -n 2
09/Nov/2026:08:01:12 +0000
09/Nov/2026:08:01:13 +0000

The output separator is the variable OFS. Set it with -v:

awk -F, -v OFS='\t' '{ print $1, $3, $4 }' orders.csv | head -n 3
order_id	region	product
1001	north	keyboard
1002	south	monitor

Patterns: choosing which lines to act on

A pattern in front of an action limits it to matching lines. A pattern with no action prints the matching lines, like grep. Patterns can be:

  • a comparison on fields: $9 >= 500, $3 == "north"
  • a regex against the whole line: /POST/
  • a regex against one field, with ~ (matches) or !~ (doesn't match): $7 ~ /^\/api\//
  • a combination with &&, || and !
  • a range, pattern1, pattern2, like sed's ranges
awk '$9 >= 500 { print $1, $7, $9 }' access.log
198.51.100.23 /api/orders 500
198.51.100.23 /api/orders 502

This is the precise version of what took grep -E '" 5[0-9]{2} ' in the previous article: a numeric comparison on exactly the status field, with no risk of matching a response size.

awk '$7 ~ /^\/api\// { print $1, $6, $7, $9 }' access.log
198.51.100.23 "POST /api/orders 201
198.51.100.23 "POST /api/orders 500
203.0.113.42 "POST /api/orders 201
198.51.100.23 "POST /api/orders 502
198.51.100.23 "POST /api/orders 201
198.51.100.23 "GET /api/orders/1042 200
awk -F, '$3 == "north" && $5 >= 2' orders.csv
1001,2026-11-01,north,keyboard,2,49.99
1004,2026-11-02,north,monitor,2,189.00
1008,2026-11-04,north,mouse,10,19.50

Ranges work with any patterns, including line numbers:

awk 'NR == 2, NR == 4' orders.csv
awk '/^\[database\]/,/^$/' app.conf
1001,2026-11-01,north,keyboard,2,49.99
1002,2026-11-01,south,monitor,1,189.00
1003,2026-11-02,west,mouse,5,19.50
[database]
host = db01.internal
port = 5432
name = orders
user = orders_app
max_connections = 50
 

Note that strings in awk use double quotes, so "north" inside a single-quoted program is fine.

BEGIN and END

BEGIN { ... } runs before the first line is read, and END { ... } after the last. BEGIN is for headers and setup; END is where totals get printed:

awk 'BEGIN { print "ip,status" } { print $1 "," $9 }' access.log | head -n 3
awk 'END { print NR }' access.log
ip,status
203.0.113.42,200
203.0.113.42,200
33

Variables need no declaration, and an unset variable is 0 in a numeric context and "" in a string context. So summing a column is just +=:

awk '{ bytes += $10 } END { print bytes, "bytes" }' access.log
59684 bytes

printf: formatted output

print is quick; printf gives you control. It works like C's printf: a format string, then the values. It does not add a newline, so end the format with \n:

awk '{ bytes += $10 } END { printf "%.1f KiB in %d requests\n", bytes / 1024, NR }' access.log
58.3 KiB in 33 requests
FormatPrints
%sa string
%dan integer (decimals are truncated)
%.2fa number with 2 decimal places
%8d, %8.2fright-aligned in 8 characters
%-10sleft-aligned in 10 characters
%%a literal %

Widths turn output into aligned columns:

awk -F, 'NR > 1 { printf "%-6s %-9s %3d x %7.2f = %8.2f\n", $1, $4, $5, $6, $5 * $6 }' orders.csv | head -n 4
1001   keyboard    2 x   49.99 =    99.98
1002   monitor     1 x  189.00 =   189.00
1003   mouse       5 x   19.50 =    97.50
1004   monitor     2 x  189.00 =   378.00

sprintf works the same way but returns the string instead of printing it, which you need when building a value.

Passing values in from the shell

Never paste a shell variable into the program text. Pass it with -v, which defines an awk variable before the program starts:

min=5000
awk -v limit="$min" '$10 > limit { print $10, $7 }' access.log
5120 /
7342 /products/42
15003 /products?page=2
7342 /products/42
7342 /products/42

Inside awk, limit is just a variable; the program stays in single quotes and nothing is re-interpreted by the shell. Environment variables are also readable through the ENVIRON array:

STATUS=502 awk '$9 == ENVIRON["STATUS"] { print $1, $4 }' access.log
198.51.100.23 [09/Nov/2026:09:10:44

What goes wrong when you don't do this is in Common mistakes.

Arrays: counting, summing, grouping

awk arrays are associative: the index can be any string, such as an IP address, a status code or a region. You don't declare them or size them; referencing count["x"] creates the element with value 0. That makes three idioms trivial:

  • count: count[key]++
  • sum: total[key] += value
  • group: list[key] = list[key] " " value

for (key in array) loops over the keys. Requests per client:

awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' access.log
8 192.0.2.15
6 198.51.100.7
4 10.0.0.5
1 192.0.2.77
7 203.0.113.42
2 203.0.113.9
5 198.51.100.23

That is the same result as sort | uniq -c, in no particular order. The order of for (key in array) is unspecified: not insertion order, not sorted. Pipe the output through sort:

awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' access.log | sort -rn
8 192.0.2.15
7 203.0.113.42
6 198.51.100.7
5 198.51.100.23
4 10.0.0.5
2 203.0.113.9
1 192.0.2.77

gawk can sort the loop itself through the special PROCINFO["sorted_in"] setting; that is gawk-only, so I use sort in anything that might run on another awk:

awk '{ count[$1]++ } END { PROCINFO["sorted_in"] = "@val_num_desc"; for (ip in count) print count[ip], ip }' access.log | head -n 3
8 192.0.2.15
7 203.0.113.42
6 198.51.100.7

Summing works the same way. Revenue per region is quantity times unit price, summed per region, skipping the header with NR > 1:

awk -F, 'NR > 1 { revenue[$3] += $5 * $6 } END { for (r in revenue) printf "%-6s %8.2f\n", r, revenue[r] }' orders.csv | sort -k2,2nr
north    751.98
south    496.97
west     486.46
east     128.99

Keys can combine several fields. Concatenate them into one string:

awk '{ pair[$1 " " $9]++ } END { for (k in pair) print pair[k], k }' access.log | sort -rn | head -n 4
8 192.0.2.15 404
5 203.0.113.42 200
4 10.0.0.5 200
3 198.51.100.7 200

The !seen[$0]++ idiom

This one-liner removes duplicate lines while keeping the original order, which sort -u can't do:

awk '!seen[$1]++ { print $1 }' access.log
203.0.113.42
10.0.0.5
198.51.100.7
192.0.2.15
198.51.100.23
203.0.113.9
192.0.2.77

Read it slowly: seen[$1]++ returns the old value (0 the first time an IP appears, then 1, 2, ...) and increments it. !0 is true, so the pattern matches only the first time each key is seen. Written as awk '!seen[$0]++' file, it de-duplicates whole lines.

To test whether a key exists without creating it, use (key in array); that matters in the two-file join below.

Conditions and control flow

Actions can use if/else, while, for and the ternary operator, with C syntax:

awk '{ if ($9 >= 500) class = "server error"; else if ($9 >= 400) class = "client error"; else class = "ok"; print class }' access.log | sort | uniq -c
      9 client error
     22 ok
      2 server error
awk '{ print $1, ($9 >= 400 ? "FAIL" : "ok") }' access.log | head -n 7 | tail -n 3
198.51.100.7 ok
192.0.2.15 FAIL
192.0.2.15 FAIL

Two statements control the flow through the input:

  • next stops processing the current line and moves to the next one. NR == 1 { next } skips a header.
  • exit stops reading input entirely (the END block still runs). Use it to stop at the first match in a huge file.
awk -F, 'NR == 1 { next } { n++ } END { print n, "orders" }' orders.csv
awk '$9 >= 500 { print "first server error on line " NR ": " $7; exit }' access.log
12 orders
first server error on line 17: /api/orders

A variable used as a flag tracks state across lines. This prints the [database] section of app.conf as environment variables. -F' = ' splits on the literal = string; every header line resets the flag, and only lines in the right section with exactly two fields are printed:

awk -F' = ' '/^\[/ { in_db = ($0 == "[database]"); next } in_db && NF == 2 { print "DB_" toupper($1) "=" $2 }' app.conf
DB_HOST=db01.internal
DB_PORT=5432
DB_NAME=orders
DB_USER=orders_app
DB_MAX_CONNECTIONS=50

Compare that with the nested sed braces needed for the same section in the sed article. Once logic depends on state, awk reads better.

String functions

FunctionReturns
length(s)length of s (length alone means length($0))
substr(s, start, len)substring; positions start at 1
index(s, t)position of t in s, or 0
split(s, arr, sep)splits s into arr[1], arr[2]...; returns the count
sub(re, rep, target)replaces the first match in target (default $0); returns 1 or 0
gsub(re, rep, target)replaces every match
tolower(s), toupper(s)case conversion
match(s, re)position of the match, and sets RSTART and RLENGTH
sprintf(fmt, ...)formatted string

The hour is characters 14 and 15 of field 4 ([09/Nov/2026:08:01:12), or the second piece when field 4 is split on colons. Both give requests per hour:

awk '{ print substr($4, 14, 2) }' access.log | sort | uniq -c
awk '{ split($4, t, ":"); print t[2] }' access.log | sort | uniq -c
     20 08
      8 09
      5 10
     20 08
      8 09
      5 10

split is the more robust of the two: it doesn't break if a field's width changes. Its return value is the number of pieces, here the depth of each path plus one:

awk '{ n = split($7, parts, "/"); print n, $7 }' access.log | head -n 4
2 /
3 /css/main.css
2 /healthz
3 /products/42

sub and gsub edit a field (or $0) in place. Strip the bracket from the timestamp, and normalise the method's case after removing its quote:

awk '{ sub(/^\[/, "", $4); print $4 }' access.log | head -n 2
awk '{ gsub(/"/, "", $6); print tolower($6) }' access.log | sort | uniq -c
09/Nov/2026:08:01:12
09/Nov/2026:08:01:13
     28 get
      5 post

match finds a regex anywhere in a string and sets RSTART (where it starts) and RLENGTH (how long it is), which substr then uses. Every order number mentioned in app.log:

awk 'match($0, /order [0-9]+/) { print substr($0, RSTART + 6, RLENGTH - 6) }' app.log | sort -u
1043
1044
1045
1046

The + 6 and - 6 skip the six characters of order . gawk has a cleaner form: a third argument to match receives the capture groups. It's convenient, but it's a gawk extension. Here are gawk and then mawk (the default awk on some Debian-based images) running the same program:

awk 'match($0, /order ([0-9]+) failed/, m) { print m[1] }' app.log
mawk 'match($0, /order ([0-9]+) failed/, m) { print m[1] }' app.log
1043
1045
mawk: line 1: syntax error at or near ,

If a script might run in a minimal container, check which awk it has (awk --version for gawk, awk -W version for mawk) or stick to POSIX functions.

Changing fields: OFS and rebuilding $0

You can assign to fields. When you do, awk rebuilds $0 by joining all fields with OFS. Until you change a field, $0 is the original line untouched, which surprises everyone the first time:

head -n 3 orders.csv | awk -F, -v OFS='\t' '{ print }'
head -n 3 orders.csv | awk -F, -v OFS='\t' '{ $1 = $1; print }'
order_id,date,region,product,qty,unit_price
1001,2026-11-01,north,keyboard,2,49.99
1002,2026-11-01,south,monitor,1,189.00
order_id	date	region	product	qty	unit_price
1001	2026-11-01	north	keyboard	2	49.99
1002	2026-11-01	south	monitor	1	189.00

$1 = $1 changes nothing visible but forces the rebuild, so the commas become tabs. That is the standard way to convert separators.

Adding a computed column is print $0, value. Handle the header separately, and format numbers yourself, because awk prints non-integers with up to six significant digits and drops trailing zeros:

head -n 4 orders.csv | awk -F, -v OFS=, 'NR == 1 { print $0, "total"; next } { print $0, $5 * $6 }'
head -n 4 orders.csv | awk -F, -v OFS=, 'NR == 1 { print $0, "total"; next } { print $0, sprintf("%.2f", $5 * $6) }'
order_id,date,region,product,qty,unit_price,total
1001,2026-11-01,north,keyboard,2,49.99,99.98
1002,2026-11-01,south,monitor,1,189.00,189
1003,2026-11-02,west,mouse,5,19.50,97.5
order_id,date,region,product,qty,unit_price,total
1001,2026-11-01,north,keyboard,2,49.99,99.98
1002,2026-11-01,south,monitor,1,189.00,189.00
1003,2026-11-02,west,mouse,5,19.50,97.50

Forget the header guard and awk does arithmetic on it. A 10% price rise:

head -n 3 orders.csv | awk -F, -v OFS=, '{ $6 = $6 * 1.10; print }'
order_id,date,region,product,qty,0
1001,2026-11-01,north,keyboard,2,54.989
1002,2026-11-01,south,monitor,1,207.9

The header's unit_price became 0 (a non-numeric string is 0 in arithmetic), and the prices need sprintf("%.2f", ...). Assigning to NF truncates (or extends) the record:

head -n 3 orders.csv | awk -F, -v OFS=, '{ NF = 3; print }'
order_id,date,region
1001,2026-11-01,north
1002,2026-11-01,south

Processing two files: NR == FNR

awk reads every file on its command line in turn. NR keeps counting across files, but FNR restarts at 1 for each file:

awk '{ print NR, FNR, FILENAME }' orders.csv customers.csv | sed -n '12,15p'
12 12 orders.csv
13 13 orders.csv
14 1 customers.csv
15 2 customers.csv

So NR == FNR is true only while reading the first file. That gives the classic join idiom: load the first file into an array, next, then use the array while processing the second. Here is a cost list per product:

cat > products.csv <<'EOF'
product,category,unit_cost
keyboard,peripherals,31.00
monitor,displays,142.00
mouse,peripherals,8.75
headset,audio,51.50
EOF

Join it to the orders to compute the margin on each order:

awk -F, 'NR == FNR { cost[$1] = $3; next } FNR > 1 { printf "%s %-9s margin %7.2f\n", $1, $4, ($6 - cost[$4]) * $5 }' products.csv orders.csv | head -n 4
1001 keyboard  margin   37.98
1002 monitor   margin   47.00
1003 mouse     margin   53.75
1004 monitor   margin   94.00

The first block runs for products.csv only and builds cost["keyboard"] = 31.00 and so on (the header line also loads a harmless cost["product"]). next stops those lines reaching the second block. FNR > 1 skips the orders header.

Real data has gaps. Add an order for a product with no cost entry and check for it with in, which tests for a key without creating it:

cp orders.csv orders-new.csv && echo "1013,2026-11-07,east,webcam,2,64.00" >> orders-new.csv
awk -F, 'NR == FNR { cost[$1] = $3; next } FNR > 1 && !($4 in cost) { print "no cost for product \"" $4 "\" in order " $1 }' products.csv orders-new.csv
no cost for product "webcam" in order 1013

Without that check, cost["webcam"] would silently be 0 and the webcam order would show a margin equal to its full price.

Quoted CSV with gawk's FPAT

-F, breaks on CSV fields that contain commas inside quotes, just like cut. gawk can describe fields by what they contain instead of what separates them, with FPAT: a field is either a run of non-commas or a quoted string:

awk -F, '{ print $3 }' customers.csv
awk -v FPAT='([^,]*)|("[^"]*")' '{ print $3 }' customers.csv
awk -v FPAT='([^,]*)|("[^"]*")' '{ print $2 }' customers.csv
city
 Jane"
Bristol
city
Leeds
Bristol
name
"Smith, Jane"
Raj Patel

The quotes stay part of the field, and this pattern doesn't handle escaped quotes ("") or newlines inside fields. It's fine for well-behaved exports; for anything else, use a real CSV parser such as Python's csv module. (gawk 5.3 added a --csv option; the 5.2 release tested here doesn't have it.)

awk programs in files

Once a program spans more than a line or two, put it in a file and run it with awk -f. A #!/usr/bin/awk -f first line makes it directly executable. The Where you'll use this section has a complete example.

When to use awk, and when not

TaskBest tool
Find lines containing textgrep
Extract a column split on one charactercut (or awk)
Columns separated by runs of spacesawk
Substitute text, edit files in placesed
Count occurrences of a valuesort | uniq -c, or awk
Sum, average, min or max per groupawk
Conditions across several columnsawk
Join two files on a keyawk (or join if both are sorted)
Nested data (JSON), real CSV, dates, HTTP callsa real language: Python, or jq for JSON
More than ~30 lines of awkprobably a real language

The deciding question is usually: am I doing arithmetic or keeping state across lines? If yes, awk. If the program needs data structures beyond one level of arrays, error handling, or tests, it has outgrown awk.

Where you'll use this

  • Log reports: per-client bytes, error rates per hour, slow-request lists, all in one pass over files too big for a spreadsheet.
  • CSV and TSV reshaping: aggregating exports, reordering or adding columns, converting separators.
  • Parsing command output in monitoring scripts: df, ps, free, ss, docker stats.
  • Quick joins between an inventory list and a status list, or orders and prices.

A one-pass access log report

The previous article built a report from five pipelines, each reading the whole log. This awk program reads it once and adds things pipelines can't do easily: bytes sent and an error percentage. Save it as access-report.awk:

#!/usr/bin/awk -f
# access-report.awk - one-pass summary of a combined-format access log.
# usage: awk -f access-report.awk access.log
{
    requests++
    bytes += $10
    hits[$1]++
    sent[$1] += $10
    status_class[substr($9, 1, 1) "xx"]++
    if ($9 >= 500) server_errors++
    hour[substr($4, 14, 2)]++
}
END {
    if (requests == 0) { print "no requests"; exit 1 }
    printf "%d requests from %d clients, %.1f KiB sent, %.1f%% server errors\n",
        requests, length(hits), bytes / 1024, 100 * server_errors / requests
 
    print "\nby status class:"
    for (c in status_class) printf "  %s %4d\n", c, status_class[c] | "sort"
    close("sort")
 
    print "\nby hour:"
    for (h in hour) printf "  %s:00 %4d\n", h, hour[h] | "sort"
    close("sort")
 
    print "\ntop clients by requests:"
    cmd = "sort -k2,2nr | head -n 5"
    for (ip in hits) printf "  %-15s %3d req %7d bytes\n", ip, hits[ip], sent[ip] | cmd
    close(cmd)
}
awk -f access-report.awk access.log
33 requests from 7 clients, 58.3 KiB sent, 6.1% server errors
 
by status class:
  2xx   18
  3xx    4
  4xx    9
  5xx    2
 
by hour:
  08:00   20
  09:00    8
  10:00    5
 
top clients by requests:
  192.0.2.15        8 req    1296 bytes
  203.0.113.42      7 req   29152 bytes
  198.51.100.7      6 req   19094 bytes
  198.51.100.23     5 req    2563 bytes
  10.0.0.5          4 req       8 bytes

Three techniques here are worth stealing:

  • printf ... | "sort" sends output to a command, and awk keeps that pipe open for later prints using the same command string. close() flushes it, so the sorted section appears before the next heading.
  • length(hits) is the number of keys in an array. It's supported by gawk and mawk (when I ran this file with mawk -f, the first line came out identical), though older awks lack it.
  • The requests == 0 guard: on an empty log the percentage would divide by zero.
chmod +x access-report.awk
./access-report.awk access.log | head -n 1
awk -f access-report.awk /dev/null
echo "exit: $?"
33 requests from 7 clients, 58.3 KiB sent, 6.1% server errors
no requests
exit: 1

CSV aggregation

Orders, revenue, average and largest order per region, with a header row:

awk -F, 'NR > 1 { n[$3]++; total[$3] += $5 * $6; if ($5 * $6 > max[$3]) max[$3] = $5 * $6 } END { printf "%-6s %6s %9s %9s %9s\n", "region", "orders", "revenue", "average", "largest"; for (r in n) printf "%-6s %6d %9.2f %9.2f %9.2f\n", r, n[r], total[r], total[r] / n[r], max[r] }' orders.csv
region orders   revenue   average   largest
east        2    128.99     64.50     79.00
west        3    486.46    162.15    199.96
south       3    496.97    165.66    189.00
north       4    751.98    188.00    378.00

Parsing df for a disk-space alert

A cron job that warns when any filesystem crosses a threshold is a classic awk task. Here is real df output from my test machine, limited to ext4 and tmpfs filesystems:

df -P -t ext4 -t tmpfs
Filesystem     1024-blocks    Used Available Capacity Mounted on
/dev/sdb             98720   98720         0     100% /usr/lib/linux-tools/6.18.54.1-microsoft-standard-WSL2
none               4019036       4   4019032       1% /mnt/wsl
/dev/sdd        1055762868 4741336 997318060       1% /
none               4019036      32   4019004       1% /mnt/wslg
none               4019036     520   4018516       1% /run
none               4019036       0   4019036       0% /run/lock
none               4019036   67184   3951852       2% /run/shm
tmpfs               803804      24    803780       1% /run/user/1000

One filesystem is genuinely 100% full (a small read-only image the system mounts). The alert, with the threshold passed in by -v:

df -P -t ext4 -t tmpfs | awk -v limit=90 'NR > 1 && $5 + 0 >= limit { printf "%s is %s full (%d MiB free)\n", $6, $5, $4 / 1024 }'
/usr/lib/linux-tools/6.18.54.1-microsoft-standard-WSL2 is 100% full (0 MiB free)

-P asks df for the POSIX output format, which keeps each filesystem on one line so the field numbers are stable. The + 0 is not decoration; the next section shows what happens without it. Scheduling a check like this is covered in system monitoring and cron.

Common mistakes

Comparing 85% with a number

The obvious alert filter on the same df output:

df -P -t ext4 -t tmpfs | awk '$5 > 80 { print $6, $5 }'
Mounted Capacity

It printed the header and missed the full filesystem. 100% doesn't look like a number, so awk compared it to 80 as a string: "1" sorts before "8", so "100%" > "80" is false, while "Capacity" > "80" is true because letters sort after digits. For the same reason a filesystem at 9% would trigger the alert:

echo "9%" | awk '$1 > 80 { print $1, "looks bigger than 80" }'
9% looks bigger than 80

$5 + 0 forces a numeric conversion (awk reads the leading 100 and ignores the %), and NR > 1 skips the header. When a comparison gives impossible results, check whether the field is really a number.

Double quotes around the program

awk "{ print $1 }" access.log | head -n 1
203.0.113.42 - - [09/Nov/2026:08:01:12 +0000] "GET / HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0"

Inside double quotes, the shell expanded $1 (its own first argument, empty at the prompt) before awk started, so awk ran { print } and printed whole lines. Always put awk programs in single quotes and bring shell values in with -v.

Counting the header

Average quantity per order, first by dividing by NR:

awk -F, '{ sum += $5 } END { print sum / NR }' orders.csv
awk -F, 'NR > 1 { sum += $5; n++ } END { print sum / n }' orders.csv
2.53846
2.75

The header contributes 0 to the sum but 1 to NR, so the first average is wrong, and nothing about the number says so. Skip the header and count the rows you actually used.

Dividing by zero when nothing matched

awk -F, '$3 == "central" { sum += $5; n++ } END { print sum / n }' orders.csv
awk: cmd. line:1: (FILENAME=orders.csv FNR=13) fatal: division by zero attempted

No region is called central, so n was never set. In a monitoring script this error replaces the output you were waiting for. Guard the division:

awk -F, '$3 == "central" { sum += $5; n++ } END { if (n) print sum / n; else print "no orders for central" }' orders.csv
no orders for central

Expecting for (k in a) to be sorted

The unsorted loop output earlier came out in neither insertion nor sorted order. Treat the order as random and pipe through sort (or use gawk's PROCINFO["sorted_in"] when you know the script runs on gawk).

Summary

  • awk splits each line into fields: $1...$NF, with $0 the whole line, NF the field count, NR the line number (and FNR per file). Default splitting handles runs of whitespace; -F sets a character or a regex.
  • A program is pattern { action } pairs; a pattern alone prints matches, an action alone runs on every line. BEGIN runs before input, END after.
  • print a, b uses OFS; print a b concatenates. printf gives widths and decimals and needs an explicit \n.
  • Associative arrays do the heavy lifting: count[k]++, sum[k] += v, !seen[k]++, (k in arr). Loop order is unspecified; sort the output.
  • Pass shell values with -v name="$value"; keep the program in single quotes.
  • Assigning to a field rebuilds $0 with OFS; $1 = $1 converts separators. Format numbers with sprintf("%.2f") and guard headers with NR > 1.
  • NR == FNR { ...; next } loads the first file to join against the second.
  • Force numbers with + 0 when fields carry units like %, and guard divisions.

Exercises

Exercise 1: Who logs in with bash?easy

Using /etc/passwd (fields separated by :; field 1 is the user, 6 the home directory, 7 the shell), print the user name and home directory of every account whose shell is exactly /bin/bash.

Show Solution
awk -F: '$7 == "/bin/bash" { print $1, $6 }' /etc/passwd
root /root
sid /home/sid

== compares the whole field, so /bin/bash doesn't match /usr/bin/bash or /bin/bash5. Your output will list the accounts on your own machine.

Exercise 2: Bytes by status codemedium

Total the response bytes (field 10) for each HTTP status code in access.log, largest total first.

Show Solution
awk '{ sent[$9] += $10 } END { for (s in sent) print sent[s], s }' access.log | sort -rn
56765 200
1458 404
1215 201
157 502
89 500
0 304
0 302
0 301

Redirects (301, 302) and "not modified" responses (304) send no body, so they total 0.

Exercise 3: Average order value per regionmedium

For each region in orders.csv, print the average order value (quantity times unit price), with two decimal places, sorted by region name.

Show Solution
awk -F, 'NR > 1 { n[$3]++; total[$3] += $5 * $6 } END { for (r in n) printf "%-6s %7.2f\n", r, total[r] / n[r] }' orders.csv | sort
east     64.50
north   188.00
south   165.66
west    162.15

Two arrays keyed by region: one counts orders, one sums values. Dividing in END gives the average. n[r] can't be zero here, because a key only exists if an order created it.

Exercise 4: Error rate per hourhard

For each hour in access.log, print the number of requests, the number of error responses (status 400 or above) and the error percentage to one decimal place, in hour order.

Hint

Use one array for totals and one for errors, both keyed by the hour (substr($4, 14, 2)). An hour with no errors has no entry in the errors array; what does awk return for it?

Show Solution
awk '{ h = substr($4, 14, 2); total[h]++; if ($9 >= 400) errors[h]++ } END { for (h in total) printf "%s:00 %3d requests %3d errors %5.1f%%\n", h, total[h], errors[h], 100 * errors[h] / total[h] }' access.log | sort
08:00  20 requests   9 errors  45.0%
09:00   8 requests   2 errors  25.0%
10:00   5 requests   0 errors   0.0%

Looping over total (not errors) guarantees every hour appears; errors["10"] was never set, so it evaluates to 0, which %3d prints as 0. The 08:00 spike is the scanner's 404 burst.


Tested with: GNU Awk 5.2.1 and GNU Bash 5.2.21 on Ubuntu 24.04 (WSL2), November 2026, plus mawk for the portability checks. Every command was run as shown against the sample files and the output pasted unchanged. The df output is from the WSL2 test machine itself, which is why the mount points look unusual; the 100% filesystem is WSL's read-only tools image, which made a convenient real example of a full disk.

Was this page helpful?
The Dark Debugger

Written by The Dark Debugger

Engineer documenting the things that took me too long to learn, so they take you less time. Every Bash tutorial here is tested end to end before publishing.