Filters

The underlying principles of Linux are: every item should do one thing only, and you can easily join these items together. Think of it as a set of building blocks you can put together however you like to build anything you want. This page illustrates how filters are used to implement these.

Within the context of the Linux command line, a filter is a program that accepts textual data and transforms it in a particular way. Filters are a way to take raw data, either produced by another program or stored in a file, and manipulate it to be displayed in a way more suited to what we are after.

These filters often have various command-line options modifying their behavior, so it is always good to check out the Manual pages for a filter to see what is available.

In the examples below, input to the programs is provided by a file. In the Piping and redirection section, you will see that you can provide input via other means that add a lot more power.

Let's dive in and introduce you to some filters. Remember, the examples here only give you a taste of what is possible with these commands. Make sure you explore and use your creativity to see what else you can do with them.

For each of the demonstrations below, the following file is going to be used as an example. It contains a list of content purely to make the examples easier to understand but realize that they will work the same with absolutely any other textual data. Also, remember that the file is actually specified as a path. So, you can use absolute and relative paths and wildcards.

user@bash: cat mysampledata.txt
Fred apples 20
Susy oranges 5
Mark watermellons 12
Robert pears 4
Terry oranges 9
Lisa peaches 7
Susy oranges 12
Mark grapes 39
Anne mangoes 7
Greg pineapples 3
Oliver rockmellons 2
Betty limes 14
user@bash:

head

The head program prints the first so many lines of its input. By default, those are the first ten lines, but you can modify the behavior with a command-line argument.

head [-number of lines to print] [path]

Example 1:

The example below illustrates the head's default behavior.

user@bash: head mysampledata.txt
Fred apples 20
Susy oranges 5
Mark watermellons 12
Robert pears 4
Terry oranges 9
Lisa peaches 7
Susy oranges 12
Mark grapes 39
Anne mangoes 7
Greg pineapples 3
user@bash:

Example 2:

The example below specifies a preset number of lines.

user@bash: head -4 mysampledata.txt
Fred apples 20
Susy oranges 5
Mark watermellons 12
Robert pears 4
user@bash:

tail

The tail program is the opposite of the head one. It prints the last so many lines of its input. By default, those are the last ten lines, but you can modify the behavior with a command-line argument.

tail [-number of lines to print] [path]

Example 1

The example below illustrates the default tailbehavior.

user@bash: tail mysampledata.txt
Mark watermellons 12
Robert pears 4
Terry oranges 9
Lisa peaches 7
Susy oranges 12
Mark grapes 39
Anne mangoes 7
Greg pineapples 3
Oliver rockmellons 2
Betty limes 14
user@bash:

Example 2

The example below specifies a preset number of lines.

user@bash: tail -3 mysampledata.txt
Greg pineapples 3
Oliver rockmellons 2
Betty limes 14
user@bash:

sort

The sort program sorts the input. By default, it is done alphabetically, but there are many options available to modify the sorting mechanism. Be sure to check out its manual page to see everything it can do.

sort [-options] [path]

Example

user@bash: sort mysampledata.txt
Anne mangoes 7
Betty limes 14
Fred apples 20
Greg pineapples 3
Lisa peaches 7
Mark grapes 39
Mark watermellons 12
Oliver rockmellons 2
Robert pears 4
Susy oranges 12
Susy oranges 5
Terry oranges 9
user@bash:

nl

nl stands for number lines, and the program does just that.

nl [-options] [path]

Example 1

Below is an example of basic formatting.

user@bash: nl mysampledata.txt
1 Fred apples 20
2 Susy oranges 5
3 Mark watermellons 12
4 Robert pears 4
5 Terry oranges 9
6 Lisa peaches 7
7 Susy oranges 12
8 Mark grapes 39
9 Anne mangoes 7
10 Greg pineapples 3
11 Oliver rockmellons 2
12 Betty limes 14
user@bash:

Example 2

The basic formatting is ok, but sometimes you are after something a little different. With a few command-line options, nl is happy to oblige.

user@bash: nl -s '. ' -w 10 mysampledata.txt
         1. Fred apples 20
         2. Susy oranges 5
         3. Mark watermellons 12
         4. Robert pears 4
         5. Terry oranges 9
         6. Lisa peaches 7
         7. Susy oranges 12
         8. Mark grapes 39
         9. Anne mangoes 7
        10. Greg pineapples 3
        11. Oliver rockmellons 2
        12. Betty limes 14
user@bash:

The above example comprises two command-line options. The first one -s specifies what is to be printed after the number. The second one -w specifies how much padding to put before the numbers.

For the first one, you need to include a space as part of what is printed. Because spaces are normally used as a separator of characters on the command line, you need a way of specifying that the space is a part of your argument and not just in-between arguments. To do that, include the argument surrounded by quotes.

wc

wc stands for word count, and it does just that plus counting characters and lines. By default, it gives you a count of all three items. However, using command-line options, you can limit it to just what you are after.

wc [-options] [path]

Example 1

The example below illustrates the default behavior.

user@bash: wc mysampledata.txt
12 36 195 mysampledata.txt
user@bash:

Example 2

Sometimes you just want one of these values. -l gives you lines only, whereas -w—words, and -m—characters. The example below outputs just a line count.

user@bash: wc -l mysampledata.txt
12 mysampledata.txt
user@bash:

Example 3

You can combine command-line arguments too. The example below gives you both lines and words.

user@bash: wc -lw mysampledata.txt
12 36 mysampledata.txt
user@bash:

cut

Use cut if your content is separated into fields (columns), and you only want certain fields.

cut [-options] [path]

In the sample file, you have data in three columns—a name, fruit, and amount. Let's say you only want the first column.

user@bash: cut -f 1 -d ' ' mysampledata.txt
Fred
Susy
Mark
Robert
Terry
Lisa
Susy
Mark
Anne
Greg
Oliver
Betty
user@bash:

cut defaults to using the TAB character as a separator to identify fields. In this file, a single space is used instead, so you need to tell cut to use that instead.

The separator character can be anything you like. For instance, in a CSV file, the separator is typically a comma (,). This is what the -doption does: you include the space within single quotes, so it knows this is part of the argument.

The -f option allows you to specify which field or fields you would like. If you want two or more fields, separate them with a comma as below.

user@bash: cut -f 1,2 -d ' ' mysampledata.txt
Fred apples
Susy oranges
Mark watermellons
Robert pears
Terry oranges
Lisa peaches
Susy oranges
Mark grapes
Anne mangoes
Greg pineapples
Oliver rockmellons
Betty limes
user@bash:

sed

sed stands for Stream Editor, and it effectively allows us to do a search and replace on data.

sed <expression> [path]

s/search/replace/g

The initial s stands for substitute and specifies the action to perform. Between the first and second slashes (/), place what you are searching for, and between the second and third ones—what you wish to replace it with.

The g at the end stands for global and is optional. If you omit it, it will only replace the first instance of search on each line. With the g option, you replace every instance of search on each line.

Example

Say, you ran out of oranges and wanted to give those people bananas instead.

user@bash: sed 's/oranges/bananas/g' mysampledata.txt
Fred apples 20
Susy bananas 5
Mark watermellons 12
Robert pears 4
Terry bananas 9
Lisa peaches 7
Susy bananas 12
Mark grapes 39
Anne mangoes 7
Greg pineapples 3
Oliver rockmellons 2
Betty limes 14
user@bash:

It's important to note that sed does not identify words but strings of characters. Try running the example above yourself but replace oranges with es, and you'll see what is meant.

The search term is also actually something called a regular expression—a means to define a pattern similar to wildcards.

Also, note that the expression is within single quotes. It was done so that any characters included in it with a special meaning on the command line don't get interpreted and acted upon by the command line but instead get passed through to sed.

A common mistake is to forget the single quotes, in which case you can get strange behavior from the command line. If this happens, you may need to press CTRL+c to cancel the program and get back to the prompt.

uniq

uniq stands for unique, and its job is to remove duplicate lines from data. However, one limitation is that those lines must be adjacent—that means one after the other.

uniq [options] [path]

Example

Let's say that the sample file was actually generated from another sales program, but after a software update, it had buggy output.

user@bash: cat mysampledata.txt
Fred apples 20
Susy oranges 5
Susy oranges 5
Susy oranges 5
Mark watermellons 12
Robert pears 4
Terry oranges 9
Lisa peaches 7
Susy oranges 12
Mark grapes 39
Mark grapes 39
Anne mangoes 7
Greg pineapples 3
Oliver rockmellons 2
Betty limes 14
user@bash:

You can easily fix that using uniq.

user@bash: uniq mysampledata.txt
Fred apples 20
Susy oranges 5
Mark watermellons 12
Robert pears 4
Terry oranges 9
Lisa peaches 7
Susy oranges 12
Mark grapes 39
Anne mangoes 7
Greg pineapples 3
Oliver rockmellons 2
Betty limes 14
user@bash:

tac

The tac program is actually cat in reverse. It was named this as it does the opposite of cat. Given data, it prints the last line first, through to the first one.

tac [path]

Example

Let's assume the sample file is generated by writing each new order to the end of the file. As a result, the most recent orders are at the end of the file. You would like it the other way so that the most recent orders are always at the top.

user@bash: tac mysampledata.txt
Betty limes 14
Oliver rockmellons 2
Greg pineapples 3
Anne mangoes 7
Mark grapes 39
Susy oranges 12
Lisa peaches 7
Terry oranges 9
Robert pears 4
Mark watermellons 12
Susy oranges 5
Fred apples 20
user@bash:

Others

Here are two other programs that are worth investigating:

• awk
• diff

They are quite powerful but also more complex than the programs listed above.

Activities for practicing

Let's mangle some data:

• First off, you may want to make a file with data similar to the sample file.
• Now play with each of the programs you looked at above. Make sure you use both relative and absolute paths.
• Have a look at the Manual page for each of the programs and try at least two of the command-line options for them.