Regular expressions and grep
The section describes the usage of the grep filter in combination with a concept called regular expressions or res for short.
tip
The best approach to master res is to go over the material and experiment on the command line a little, leave it for a day or three, and then come back and have another go. You will be surprised, but it will start to make more sense the second time.
Regular expressions are similar to the wildcards. They allow us to create a pattern. However, they are a bit more powerful.
Res are typically used to identify and manipulate specific pieces of data, for instance, if you want to identify every line containing an email address or an URL in a data set.
In this section, they are used with grep, but many other programs utilize them as well, including sed and vi, and many programming languages make use of them too.
For more information, read the regular expression tutorial.
important
The characters used in regular expressions are the same as those used in wildcards. Their behavior is slightly different, however. A common mistake is to forget this and get their functions mixed up.
egrep
egrep is a program to search a given data set and print every line containing a specific pattern. It is an extension of the grep program.
Its name is odd but based upon a command that did a similar function called ed in a text editor. The program has many command-line options to modify its behavior, so it's worth checking out its Manual page. For example, the -v option tells grep to print every line that does not match a pattern.
egrep [command line options] <pattern> [path]
In the examples below, a similar sample file is used as in the Filters section. It is included below as a reference.
user@bash: cat mysampledata.txt
Fred apples 20
Susy oranges 5
Mark watermellons 12
Robert pears 4
Terry oranges 9
Lisa peaches 7
Susy oranges 12
Mark grapes 39
Anne mangoes 7
Greg pineapples 3
Oliver rockmellons 2
Betty limes 14
Example 1
Let's say you want to identify every line that contains the mellon string.
user@bash: egrep 'mellon' mysampledata.txt
Mark watermellons 12
Oliver rockmellons 2
The basic egrep behavior is that it prints the entire line for every line containing a string of characters matching the given pattern. Note that you are not searching for a word but a string of characters.
Also, note that the pattern is within quotes. This is not always required, but it is safer to get in the habit of always using them. They are required if your pattern contains characters that have a special meaning on the command line.
Example 2
Sometimes, it is essential to know not only which lines matched but their line number as well.
user@bash: egrep -n 'mellon' mysampledata.txt
3:Mark watermellons 12
11:Oliver rockmellons 2
Example 3
Or maybe you are not interested in seeing the matched lines but wish to know how many lines did match.
user@bash: egrep -c 'mellon' mysampledata.txt
2
Learning regular expressions
The best way to learn regular expressions is to give the examples a try yourself, then modify them slightly to test your understanding.
It is common to make mistakes in your patterns while you are learning. When this happens, typically, every line is matched, or no lines are matched, or you get an obscure set. Don't worry if this happens—you can easily go back and have another go. Remember you may hit the up arrow on your keyboard to get at your recent commands and also modify them, so you don't need to retype the whole command each time.
If you're not getting the output you would like, here are some basic strategies:
- First off, check for typos.
- Re-read the content here. Maybe what you thought a particular operator did was slightly different from what it actually does. In this case, while you are re-reading, you can notice a point you may have missed the first time.
- Break your pattern down into components and test each of these individually. This helps to get a feel for which parts of the pattern are right and which parts you need to adjust.
- Examine your output. Your current pattern may not have worked the way you want, but you can still learn from it. Looking at what you actually did match and using it to help understand what actually did happen helps to work out what you should try changing to get closer to what you actually want.
Debuggex is an online tool that allows you to experiment with regular expressions and visualize their behavior.
Regular expression overview
Below are the basic building blocks of res:
.(dot)—a single character?—the preceding character matches 0 or 1 times only.*—the preceding character matches 0 or more times.+—the preceding character matches 1 or more times.{n}—the preceding character matches exactlyntimes.{n,m}—the preceding character matches at leastntimes and not more thanmtimes.[agd]—the character is one of those included within the square brackets.[\^agd]—the character is not one of those included within the square brackets.[c-f]—the dash within the square brackets operates as a range. In this case, it means either the letters c, d, e, or f.()—allows you to group several characters to behave as one.|(pipe symbol)—the logical OR operation.\^—matches the beginning of the line.\$— matches the end of the line.
Examples with regular expressions
Example 1
Let's say you wish to identify any line with two or more vowels in a row. In the example below, the
multiplier {2,} applies to the preceding item, which is the range.
user@bash: egrep '[aeiou]{2,}' mysampledata.txt
Robert pears 4
Lisa peaches 7
Anne mangoes 7
Greg pineapples 3
Example 2
Identify any line with 2 on it, which is not the end of the line. In this example, the multiplier + applies to the ., which is any character.
user@bash: egrep '2.+' mysampledata.txt
Fred apples 20
Example 3
Find the lines where the number 2 is the last character on the line.
user@bash: egrep '2$' mysampledata.txt
Mark watermellons 12
Susy oranges 12
Oliver rockmellons 2
Example 4
Now, identify each line containing either or, or is, or go.
user@bash: egrep 'or|is|go' mysampledata.txt
Susy oranges 5
Terry oranges 9
Lisa peaches 7
Susy oranges 12
Anne mangoes 7
Example 5
Maybe you wish to see orders for everyone whose name begins with A to K.
user@bash: egrep '^[A-K]' mysampledata.txt
Fred apples 20
Anne mangoes 7
Greg pineapples 3
Betty limes 14
Activities for practicing
Let's identify some information:
- First off, you may want to make a file with data similar to the sample file.
- Now, play with some of the examples you looked at above.
- Have a look at the Manual page for
egrepand try at least two of the command-line options for the program.