Text Processing with awk and sed

awk and sed are two powerful text processing tools that complement grep.

1. awk: Field Extraction and Calculations

awk processes text line by line and splits each line into fields using whitespace (or a custom delimiter) as the separator. Fields are referenced as $1, $2, $3, etc.

# Print only the 1st (IP) and 7th (URL path) columns from Nginx access log
awk '{print $1, $7}' /var/log/nginx/access.log

# Sum the total bytes served (column 10 in a standard Nginx log format)
awk '{total += $10} END {print "Total bytes:", total}' /var/log/nginx/access.log

# Use a comma as field separator for CSV files
awk -F',' '{print $2, $3}' data.csv

2. sed: Stream Editing and In-Place Substitution

sed applies text transformations on a stream line by line. The most common operation is the s/pattern/replacement/ substitute command:

# Replace "localhost" with "api.example.com" and print to stdout
sed 's/localhost/api.example.com/' config.txt

# Replace all occurrences on each line (global flag g)
sed 's/foo/bar/g' file.txt

# Edit the file in-place (modifies the actual file)
sed -i 's/DEBUG/INFO/g' /etc/app/config.conf

# Delete lines containing a specific pattern
sed '/^#/d' config.conf  # Remove all comment lines starting with #

3. Combining awk and sed in Pipelines

# Extract error lines, then clean up the timestamp prefix
grep "ERROR" app.log | awk '{print $3, $4, $5}' | sed 's/\[//g'