Piping and redirection

This article highlights how you can join together filters from the previous articles to enable more powerful data manipulation.

Every program you run on the command line has three data streams connected to it automatically:

• STDIN (0): standard input, which is data fed into the program.
• STDOUT (1): standard output, which is data printed by the program, defaults to the terminal.
• STDERR (2): standard error for error messages, also defaults to the terminal.

Piping and redirection are the means by which you can connect these streams between programs and files to direct data in interesting and useful ways.

These mechanisms are demonstrated below with several examples. However, they work with every program on the command line, not just the ones used in the examples.

Redirecting to file

Normally, you get output on the screen, which is convenient most of the time. Sometimes though, you may wish to save it to a file to keep as a record, feed into another system, or send to someone else.

The greater than (>) operator indicates to the command line that you wish the program output (or whatever it sends to STDOUT) to be saved to a file instead of printed to the screen.

Example

user@bash: ls
barry.txt bob example.png firstfile foo1 video.mpeg
user@bash: ls > myoutput
user@bash: ls
barry.txt bob example.png firstfile foo1 myoutput video.mpeg
user@bash: cat myoutput
barry.txt
bob
example.png
firstfile
foo1
myoutput
video.mpeg

Let's break down the example above:

• With line 1, you start off by seeing what's in your current directory.

• At line 3, you run the same command, but this time you use > to tell the terminal to save the output to the myoutput file.

  Note that you don't need to create the file before saving to it. The terminal creates it automatically if it does not exist.

• At line 4, as you can see, the new file is created.

• At line 6, you can see what was saved in there.

Some observations

In the above example, the output saved in the file was one file per line instead of all across one line when printed to the screen.

The reason for this is that the screen is a known width, and the program can format its output to suit that. When you are redirecting, it may be to a file, or it could be somewhere else, so the safest option is to format it as one entry per line. This also allows you to manipulate that data easier later on as you see further down the page.

When piping and redirecting, the actual data is always the same, but the formatting of that data can be slightly different from what is normally printed to the screen.

Note that the file you created for saving the data is also in the listing. The way the mechanism works, the file is created first (if it does not exist already), and then the program is run, and output is saved to the file.

Saving to existing file

If you redirect to a file that does not exist, the file is created automatically for you. If you save to a file that already exists, its content is cleared first, and then the new output is saved to it.

user@bash: cat myoutput
barry.txt
bob
example.png
firstfile
foo1
myoutput
video.mpeg
user@bash: wc -l barry.txt > myoutput
user@bash: cat myoutput
7 barry.txt

You can instead get the new data to be appended to the file by using the double greater than (>>) operator.

user@bash: cat myoutput
7 barry.txt
user@bash: ls >> myoutput
user@bash: cat myoutput
7 barry.txt
barry.txt
bob
example.png
firstfile
foo1
myoutput
video.mpeg

Redirecting from file

If you use the less than operator (<), you can send data the other way. You read data from the file and feed it into the program via its STDIN stream.

user@bash: wc -l myoutput
8 myoutput
user@bash: wc -l < myoutput
8

As you have seen in previous sections, many programs allow you to supply a file as a command-line argument, and it will read and process the contents of that file. Given this, you may be asking why you would need to use this operator. The above example illustrates a subtle but useful difference.

When you run wc, supplying the file to process as a command-line argument, the output from the program includes the name of the processed file. When you run it, redirecting the contents of the file into wc, the file name is not printed. This is because whenever you use redirection or piping, the data is sent anonymously. So in the above example, wc received some content to process, but it has no knowledge of where it came from, so it may not print this information. As a result, this mechanism is often used to get ancillary data (which may not be required) not to be printed.

You can easily combine the two forms of redirection you have seen so far into a single command, as in the example below.

user@bash: wc -l < barry.txt > myoutput
user@bash: cat myoutput
7

Redirecting STDERR

Now let's look at the third stream, which is Standard Error or STDERR. The three streams actually have numbers associated with them, as specified in the brackets in the list at the top of the page. You can use these numbers to identify the streams.

STDERR is stream number 2. If you place a number before the > operator, it redirects the stream. If you don't use a number (as in the previous examples above), it defaults to stream 1.

user@bash: ls -l video.mpg blah.foo
ls: cannot access blah.foo: No such file or directory
-rwxr--r-- 1 ryan users 6 May 16 09:14 video.mpg
user@bash: ls -l video.mpg blah.foo 2> errors.txt
-rwxr--r-- 1 ryan users 6 May 16 09:14 video.mpg
user@bash: cat errors.txt
ls: cannot access blah.foo: No such file or directory

To save both normal output and error messages to a single file, redirect the STDERR stream to the STDOUT one and redirect STDOUT to a file.

Redirect to a file first, then redirect the error stream. To identify the redirection to a stream, add & in front of the stream number. Otherwise, it would redirect to a file called 1.

user@bash: ls -l video.mpg blah.foo > myoutput 2>&1
user@bash: cat myoutput
ls: cannot access blah.foo: No such file or directory
-rwxr--r-- 1 ryan users 6 May 16 09:14 video.mpg

Piping

So far, you've dealt with sending data to and from files. Now let's take a look at the mechanism for sending data from one program to another. It's called piping, and the operator you use is (|) (found above the backslash (\) key on most keyboards). The operator feeds the output from the program on the left as input to the program on the right. In the example below, you list only the first three files in the directory.

user@bash: ls
barry.txt bob example.png firstfile foo1 myoutput video.mpeg
user@bash: ls | head -3
barry.txt
bob
example.png

You can pipe as many programs together as you like. In the below example, you have the output piped to tail to get only the third file.

user@bash: ls | head -3 | tail -1
example.png

Any command-line arguments you supply for a program must be next to that program.

tip

For better results, build pipes up incrementally. Run the first program and make sure it provides the output you were expecting. Then, add the second program and check again before adding the third one, and so on.

You can combine pipes and redirection, too.

user@bash: ls | head -3 | tail -1 > myoutput
user@bash: cat myoutput
example.png

More examples

Below are some more examples to give an idea of the things you can do with piping. All programs used in the examples are programs you have seen before. However, the examples contain some command-line arguments that were not covered yet. Look up relevant Manual pages to find out what they do. Also, try the commands yourself, building up incrementally to see exactly what each step is doing.

Example 1

In this example, the listing of a directory is sorted so that all directories are listed first.

user@bash: ls -l /etc | tail -n +2 | sort
drwxrwxr-x 3 nagios nagcmd 4096 Mar 29 08:52 nagios
drwxr-x--- 2 news news 4096 Jan 27 02:22 news
drwxr-x--- 2 root mysql 4096 Mar 6 22:39 mysql
...

Example 2

In this example, you feed the output of a program into the less program so that you can view it easier.

user@bash: ls -l /etc | less
(Full screen of output you may scroll. Try it yourself to see.)

Example 3

Identify all files in your home directory for which the group has the write permission.

user@bash: ls -l ~ | grep '^.....w'
drwxrwxr-x 3 ryan users 4096 Jan 21 04:12 dropbox

Example 4

Create a listing of every user who owns a file in a given directory and how many files and directories they own.

user@bash: ls -l /projects/ghosttrail | tail -n +2 | sed 's/\s\s*/ /g' | cut -d ' ' -f 3 | sort | uniq -c
8 anne
34 harry
37 tina
18 ryan

Activities for practicing

Let's mangle some data:

• First off, experiment with saving the output from various commands to a file. Overwrite the file and append to it as well. Make sure you are using a both absolute and relative paths as you go.
• Now, see if you can list only the 20th last file in the /etc directory.
• Finally, see if you can get a count of how many files and directories you have the execute permission for in your home directory.