Module 1.8 — Pipes and text filters

One small command rarely answers a real question. A chain of small commands often does. The pipe symbol | lets the output of one command become the input of the next, and a handful of tiny filters (grep, wc, sort, uniq, cut) turn a long log into a short, readable answer.

Sam's tidewatch logs have grown past what anyone wants to read line by line. This module shows how to ask them questions.

By the end you should be able to:

• explain what | connects, and read a pipeline from left to right;
• filter lines with grep, including -i, -n, and -v;
• count with wc -l, order with sort, collapse with uniq, and pick columns with cut -d -f;
• say clearly what a pipe does not carry (stderr, unless you redirect it).

Scroll Core ideas → worked examples → Quiz. Plan on roughly 60 minutes, with a terminal open in ~/cli-practice/tidewatch/.

Core ideas — small filters, joined by pipes

In the last module you learned that every command has a stream of output and that you can send that stream to a file. This module takes the next natural step: sending that stream not to a file but straight into another command. The tool that does this is the pipe, written |.

The idea behind it is old and very practical. Instead of building one huge program that can do everything, Unix gives you many small programs that each do one thing well, and lets you snap them together like sections of garden hose. One command finds the lines you care about, the next counts them, the next sorts them. Nobody had to write a "find and count and sort" command; you assemble it from parts you already own.

---

What the pipe actually does

command_one | command_two

The | takes the stdout of command_one and delivers it as the stdin of command_two. That is the entire definition. Everything else in this module is an application of that one sentence.

A concrete picture helps. Suppose command_one is cat logs/events.log. It would normally pour the file onto your screen. With a pipe after it, the text does not reach your screen at all; it flows into the next command instead, and only the last command in the chain prints to the screen (unless you redirect it).

Three small facts are worth keeping in mind:

1. No temporary files. The data flows between commands directly. You do not have to save an intermediate result and clean it up.
2. Commands run together. The second command can start working on the first lines before the first command has finished producing the last ones.
3. Each command is blind to the others. grep does not know it is being fed by cat; it just reads stdin and writes stdout. This is exactly what makes the pieces swappable.

---

Reading a pipeline: left to right

When you meet a long line full of |, read it like a sentence from left to right, one stage at a time. Ask of each stage: "what comes in, what goes out?"

grep WARN logs/events.log | wc -l

Stage one, grep WARN logs/events.log, reads the log and keeps only lines containing WARN. Stage two, wc -l, receives those kept lines and counts them. The answer is a single number: how many warnings the log holds.

A good way to build your own pipelines is one stage at a time. Run the first command alone and look at its output. Add | second and look again. Add the third and look again. If something goes wrong, you know exactly which stage introduced the problem, because it was the last thing you added.

---

Finding lines — grep

grep reads text and prints only the lines that contain a pattern. In its simplest form the pattern is just a word.

grep WARN logs/events.log

Every line with the exact letters WARN appears, and every other line is dropped. grep can read a named file (as above) or read stdin when it sits after a pipe:

cat logs/events.log | grep WARN

Both give the same result. When you already know the file name, passing it directly is shorter and is the more common style.

Four options to know first:

| Option | Meaning | Example | |--------|---------|---------| | (none) | case-sensitive match | grep WARN finds WARN, not warn | | -i | ignore case | grep -i info finds INFO and info | | -n | show line numbers | grep -n WARN prefixes each hit with its line number and a colon | | -v | invert: show lines that do not match | grep -v INFO hides the routine lines |

Case sensitivity is a frequent surprise. If you search for info and the log says INFO, you will get nothing, and you might wrongly conclude there is no such line. When a search returns nothing and you expected something, try -i before doubting your data.

-n is a gift when you want to jump to a spot in a long file; the number tells you which line to open. -v is the "everything except" switch. It shines when you want to remove noise: grep -v INFO strips the ordinary entries so only the interesting ones remain.

Also worth knowing: grep -c pattern file prints just the count of matching lines. It is a shortcut for grep pattern file | wc -l.

One more honest note: if grep finds nothing, it prints nothing, and it quietly reports failure to the shell through its exit status. No output does not mean the command broke; it often just means "no matches."

---

Counting — wc -l

wc stands for "word count", but the option you will use most is -l, which counts lines.

wc -l logs/events.log

When it reads a named file, wc -l prints the count and the file name. When it reads from a pipe, it prints only the count, since there is no file name to show:

grep ERROR logs/events.log | wc -l

Counting lines is a good partner for grep because grep produces one output line per match. So "lines out of grep" equals "number of matches." Keep that link in mind: it is how you turn a filter into a question like "how many?"

---

Ordering — sort

sort reads lines and prints them back in order, by default alphabetically from the first character of each line.

sort data/raw/readings.csv

A few useful switches:

• -r reverses the order.
• -n compares by numeric value instead of by letters. This matters: alphabetically 9 comes after 10 (because 1 sorts before 9), but numerically 9 comes before 10.
• -u removes duplicates as it sorts (we will meet a more flexible tool for that next).

sort does not change the original file. It prints a sorted copy to stdout. If you want a sorted file, redirect with > to a new name, as you learned in the last module.

---

Collapsing neighbours — uniq

uniq removes repeated lines, but with an important catch: it only notices duplicates that sit right next to each other. This is why uniq is almost always used after sort, which brings identical lines together.

sort names.txt | uniq

The combination sort | uniq gives you each distinct line once. And one option makes it a counting tool:

sort names.txt | uniq -c

With -c, each distinct line is shown with a count of how many times it occurred. This sort | uniq -c pair is one of the most useful two-step recipes in daily work: "what values appear in this column, and how often?"

If you forget the sort step and the repeats are scattered, uniq will leave them all in place, because they were never adjacent. That is not a bug in uniq; it is simply how it was designed.

---

Picking columns — a light touch of cut

Many files have fields separated by a character, such as a comma in a CSV file. cut extracts chosen fields from each line.

cut -d, -f1 data/raw/readings.csv

Read the options as: -d, means "the delimiter is a comma", and -f1 means "give me field number 1." Fields are counted from 1, not 0. You can pick more than one field with -f1,3, which keeps fields 1 and 3.

For a space-separated file, you tell cut the delimiter is a space, usually quoted so the shell does not get confused: cut -d' ' -f2.

cut is deliberately simple. It splits on one exact character, so it is a poor fit for files where the spacing is uneven or where a comma can appear inside a quoted value. For tidy files like the ones in this course, it is perfect.

---

Putting it together

Here is a typical pipeline: "which gauges appear in the readings file, and how many times each?"

cut -d, -f1 data/raw/readings.csv | grep -v gauge | sort | uniq -c

Reading left to right:

1. cut -d, -f1 keeps only the first column (the gauge letter).
2. grep -v gauge drops the header line, which contains the word gauge.
3. sort groups equal letters together.
4. uniq -c collapses each group and prints its count.

Four tiny commands, one useful answer, and no file written along the way.

---

Do not overclaim: what pipes carry

A pipe connects stdout to stdin, and nothing more. It does not carry stderr. If the first command prints an error message, that message goes to your screen as usual, not into the next command.

ls missing_folder | wc -l

Here ls complains on stderr, so the complaint appears on your screen, while wc -l receives nothing on stdin and prints 0. This is a correct behaviour, but it can mislead you if you read only the 0 and ignore the message above it.

If you truly want errors to flow through the pipe as well, you redirect stderr into stdout first, using what you learned last module:

ls missing_folder 2>&1 | wc -l

Now the error message becomes ordinary text on the stream, and wc -l counts it (1 line). Do this only when you mean it. Most of the time, you want errors kept out of the data flow.

Two further cautions. A pipeline's visible success is judged by its last command by default, so a failure early on may be invisible in the final status. And pipes pass text; the filters here do not understand what the text means, so sort does not know that A,06:00,112 is a reading, only that it is a line.

---

Common pitfalls

• Forgetting sort before uniq. Repeats that are not neighbours survive.
• Sorting numbers without -n. You will get 10 before 9.
• Case-sensitive searches that find nothing. Try -i.
• Assuming errors travel through |. They do not unless you write 2>&1.
• Treating cut as a CSV parser. It splits on a single character and nothing smarter.
• Reading the pipeline all at once. Go stage by stage, or build it that way.

---

Self-study checklist

• Say in one sentence what | connects.
• Read grep WARN logs/events.log | wc -l aloud, one stage at a time.
• Explain the difference between grep info, grep -i info, and grep -v info.
• Use grep -n and explain what the number before the colon means.
• Say why uniq is usually preceded by sort, and what -c adds.
• Extract the second column of a comma-separated file with cut.
• Explain why ls missing_folder | wc -l shows an error and a 0.

Worked examples — asking the logs questions

A week of gauge activity has piled up in tidewatch/, and Sam wants answers without reading every line. Each scene gives a situation, the commands, an interpretation, and a takeaway. Read each pipeline left to right.

First, a small setup so you can follow along:

cd ~/cli-practice
mkdir -p tidewatch/logs tidewatch/data/raw
printf '06:00 INFO gauge A reading 112\n06:05 WARN gauge C silent\n07:00 INFO gauge B reading 140\n07:05 ERROR cable fault at gauge B\n08:00 info gauge A reading 131\n08:10 WARN gauge C silent\n09:00 INFO gauge A reading 149\n09:30 WARN gauge C silent\n' > tidewatch/logs/events.log
printf 'gauge,time,level_cm\nA,06:00,112\nB,06:00,140\nA,07:00,118\nB,07:00,143\nA,08:00,131\nC,08:00,90\nA,09:00,149\nB,09:00,151\n' > tidewatch/data/raw/readings.csv
cd tidewatch

---

Scene 1 — How many warnings, and where?

Scene. Sam wants to know whether gauge C is a repeat offender.

grep WARN logs/events.log
grep -n WARN logs/events.log
grep WARN logs/events.log | wc -l

06:05 WARN gauge C silent
08:10 WARN gauge C silent
09:30 WARN gauge C silent
2:06:05 WARN gauge C silent
6:08:10 WARN gauge C silent
8:09:30 WARN gauge C silent
       3

Interpretation. The first command keeps three lines. The second adds the line number and a colon in front of each (the first number is the position in the file; the clock time that follows is part of the original line). The third pipes the same three lines into wc -l, which prints their count. Gauge C has complained three times in under four hours.

Takeaway. grep turns a long file into the lines that matter, and | wc -l turns those lines into a number.

---

Scene 2 — A search that finds too little

Scene. Sam counts the routine entries, but the number looks too small.

grep INFO logs/events.log | wc -l
grep -i info logs/events.log | wc -l
grep -i info logs/events.log

       3
       4
06:00 INFO gauge A reading 112
07:00 INFO gauge B reading 140
08:00 info gauge A reading 131
09:00 INFO gauge A reading 149

Interpretation. One entry was written in lowercase (info), so the case-sensitive search skipped it. With -i, the fourth line appears. This is a real hazard in real logs: capitalisation drifts, and a missing line is easy to blame on the data instead of the search.

Takeaway. When a count seems low, try grep -i before you doubt the file.

---

Scene 3 — Removing the noise

Scene. Sam wants to see only the entries that are not routine readings.

grep -v -i info logs/events.log

06:05 WARN gauge C silent
07:05 ERROR cable fault at gauge B
08:10 WARN gauge C silent
09:30 WARN gauge C silent

Interpretation. -v inverts the match, and -i ignores case, so every line mentioning info in any capitalisation is dropped. What remains is the list of problems: three warnings and one error. Combining options is normal; here, they simply stack.

Sam then asks a follow-up: which kinds of entry exist, and how common is each?

cut -d' ' -f2 logs/events.log | sort | uniq -c

   1 ERROR
   3 INFO
   3 WARN
   1 info

Interpretation. cut -d' ' -f2 takes the second space-separated word of each line, which is the level. sort brings equal levels together, and uniq -c counts each group. Notice INFO and info appear as two separate groups, the same case issue as before, now visible in a summary.

Takeaway. grep -v removes what you do not want. The sort | uniq -c pair summarises what is left.

---

Scene 4 — Columns and counts from a CSV file

Scene. Sam wants to know how many readings each gauge has contributed, most active first.

cut -d, -f1 data/raw/readings.csv | grep -v gauge | sort | uniq -c | sort -rn

   4 A
   3 B
   1 C

Interpretation. Read the stages one by one. cut extracts the gauge column, including the header word gauge. grep -v gauge removes the header. sort groups the letters, and uniq -c counts each group. The final sort -rn orders the counts numerically and in reverse, so the largest comes first. Gauge A has been the most reliable, and C the least.

If Sam had left out the first sort, the repeated letters would be scattered (A, B, A, B...) and uniq -c would have reported one-line groups, a classic mistake.

Takeaway. Build a pipeline one stage at a time, and always sort before uniq.

---

Scene 5 — Errors do not travel through the pipe

Scene. Sam mistypes a folder name while counting files.

ls tidewatch_typo | wc -l
ls tidewatch_typo 2>&1 | wc -l
ls tidewatch_typo 2> /dev/null | wc -l

ls: tidewatch_typo: No such file or directory
       0
       1
       0

Interpretation. In the first command, the complaint went straight to the screen (stderr is not piped), and wc -l received nothing, printing 0. The second merged stderr into stdout, so the complaint flowed through the pipe and was counted as one line. The third discarded the complaint, leaving a quiet 0.

The lesson is about reading honestly. A bare 0 could mean "the folder is empty," "the folder does not exist," or "the error was hidden." Only the message on screen told Sam which. The pipe did exactly what it promises and no more: it connected stdout to stdin.

Takeaway. A pipe carries stdout only. If a count looks wrong, look for an error on the screen before trusting the number.