awk & sed

Both read text a line at a time and are built for use in a pipeline, but they solve different shapes of problem.

• sed (stream editor) thinks in terms of lines: find lines matching something, then substitute, delete, insert, or print them. Reach for it when the unit of work is "this line".
• awk thinks in terms of fields: it automatically splits every line into columns and gives you variables, arithmetic, and associative arrays to work with them. Reach for it when the unit of work is "this column" — sums, counts, reformatting CSV-ish data, log analysis.

They're frequently chained together: grep to find the lines, awk to pull out and compute over a column, sed to reshape the text of what's left.

An awk program is a series of pattern { action } pairs. For every input line, awk checks each pattern in order and runs the matching action; a bare pattern with no action defaults to { print }, and a bare action with no pattern runs on every line.

echo "alice 30 engineer" | awk '{ print $1, $3 }'
# alice engineer

Before running any action, awk splits the current line ($0) into fields on whitespace by default: $1 is the first field, $2 the second, and so on up to $NF, the last one.

awk '{ sum += $2 } END { print sum }' sales.txt
awk '{ if ($2 > max) max = $2 } END { print max }' sales.txt
awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' log.txt

Variables in awk don't need to be declared; an unseen variable starts as an empty string/zero. This makes running totals, counts, and maximums trivial one-liners — the pattern above (accumulate in the main block, report in END) covers most of what people reach for awk to do.

awk's arrays are always associative (string-keyed), even when you use them like a numeric-indexed array — the keys are just stringified numbers under the hood.

function average(arr,    sum, n, k) {   # extra params after the real one are local vars, by convention
  for (k in arr) { sum += arr[k]; n++ }
  return n ? sum / n : 0
}

BEGIN {
  scores["alice"] = 90
  scores["bob"] = 75
  print average(scores)
}

sed reads input one line at a time into its pattern space, runs your commands against it, then (unless told otherwise) prints the result and moves to the next line. The single most common command is substitution.

echo "hello world" | sed 's/world/bash/'
# hello bash

s/pattern/replacement/ replaces only the first match on each line by default.

Any sed command can be limited to specific lines by prefixing it with an address.

sed -E 's/(foo|bar)baz/\1/' file -E (or the older -r) enables extended regular expressions: unescaped ( ), +, ?, and | for alternation, instead of needing backslash-escaped \( \) \+ \? in sed's default basic regex mode

Beyond the pattern space (the current line), sed keeps a second, separate buffer called the hold space that persists across lines. Five commands move data between them: h copies pattern space into hold space, H appends it instead of overwriting, g copies hold space into pattern space, G appends it, and x swaps the two outright. This is what makes sed capable of operating across lines, not just within one.

sed -n '1!G;h;$p' file

This is the classic sed one-liner for reversing a file's line order (equivalent to tac): on every line except the first, it prepends whatever has accumulated in the hold space so far (1!G), saves the growing, reversed result back to the hold space (h), and only prints once it reaches the last line ($p), by which point the whole file has been built up in reverse.