Bash scripting

The article is a brief introduction to Bash scripting. Also, it brings together a lot of what you learnt from previous articles—you'll see them referred to often.

A Bash script in computing terms is similar to a script in theatrical terms. It is a document stating what to say and do. Here, instead of the script being read and acted upon by a person, it is read and acted upon (or executed) by the computer.

A Bash script allows you to define a series of actions that the computer will then perform without you having to enter the commands yourself. If a particular task is done often, or it is repetetive, then a script can be a useful tool. Anything you can run on the command line, you can place into a script, and they will behave the same. Vice versa, anything you can put into a script, you can run on the command line and again—it will perform the same.

The above statement is important to understand when creating scripts. When testing different parts of your script as you're building it, it is often easiest to just run your commands directly on the command line.

A script is just a plain text file, and it can have any name you like. You create them the same way you would any other text file, with just a plain old text editor, such as Vim.

Example

Below is a simple script. The recommendation is to create a similar file yourself and run it to get a feel for how scripts work.

This script prints a message to the screen using a program called echo and gives you a listing of what is in your current directory.

user@bash: cat myscript.sh
#!/bin/bash
# A simple demonstration script
# Ryan 25/4/2018

user@bash: echo Here are the files in your current directory:
ls
user@bash: 
user@bash: ls -l myscript.sh
-rwxr-xr-x 1 ryan users 2 Jun 4 2012 myscript.sh
user@bash: ./myscript.sh
Here are the files in your current directory:
barry.txt bob example.png firstfile foo1 myoutput video.mpeg
user@bash: 

Let's break down the example above:

• On line 1, you start off by having a look at our script. Linux is an extensionless system, so it is not required for scripts to have a .sh extension. However, it is common to put them on to make them easy to identify.
• Line 2 is the line that should come first in the script. This line identifies which interpreter should be used. The first two characters are referred to as a shebang. After that (important, no spaces) is the path to the interpreter.
• On lines 3 and 4, you can see comments. Anything following a # is a comment. The interpreter does not run this. It is just here for your benefit. It is good practice to include your name, the date you wrote the script, as well as a one-line description of what it does at the top of the script.
• On line 6, you introduce a program called echo. It merely prints whatever you place after it as command-line arguments to the screen. Useful for printing messages.
• On line 7, you print the contents of your current directory.
• On line 9, permissions are demonstrated. A script must have the execute permission before you can run it.
• On line 12, you finally run the script.
• On lines 13 and 14, you can find the output from running (or executing) the script.

Important points

Shebang

The very first line of a script should tell the system which interpreter to use on this file. This must be the very first line of the script. It is also important that there are no spaces.

The first two characters #! (shebang) tell the system that, directly after it, a path follows to the interpreter to be used. If you don't know where your interpreter is located, you can use the which program to find out the location.

which <program>

Example

user@bash: which bash
/bin/bash
user@bash: which ls
/usr/bin/ls

If you leave this line out, the Bash script may still work. Most shells, Bash included, assume they are the interpreter if one is not specified. However, it is good practice always to include the interpreter. Later on, you or someone else can run your script in conditions under which Bash is not the shell currently in use, and this could lead to undesirable outcomes.

Name

Linux is an extensionless system. That means you can call your script whatever you like, and it will not affect its running in any way. While it is typical to put a .sh extension on our scripts, this is purely for convenience and is not required. You could name the script above simply myscript or even myscript.jpg, and it would still run quite happily.

Comments

A comment is just a note in the script that does not get run, it is merely there for your benefit. Comments are easy to put in: place a hash (#), then anything after that is considered a comment. A comment can be a whole line or at the end of a line.

user@bash: cat myscript.sh
#!/bin/bash
# A comment which takes up a whole line
ls # A comment at the end of the line
user@bash: 

It is common practice to include a comment at the top of a script with a brief description of what the script does, who wrote it, and when. These are just basic things that people often wish to know about a script.

For the rest of the script, it is not necessary to comment on every line. For most lines, it is self-explanatory what they do. Only put comments in for important lines or to explain a particular command whose operation may not be immediately apparent.

Why the ./

Linux is set up the way it is, largely for logical reasons. This peculiarity makes the system a bit safer.

When you type a command on the command line, the system runs through a preset series of directories, looking for the program you specified. You can find out these directories by looking at the PATH variable.

user@bash: echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin

The system looks in the first directory, and if it finds the program, it runs it. If not, it checks the second directory and so on. Directories are separated by a colon (:).

The system does not look in any directories apart from these. It doesn't even look in your current directory. However, you can override this behavior by supplying a path. When you do so, the system effectively says: "Ah, you've told me where to look to find the script, so I'll ignore PATH and go straight to the location you've specified instead."

You remember from the Basic navigation article, a full stop (.) represents your current directory. So, when you say ./myscript.sh, you are telling the system to look in your current directory to find the script. You could have used an absolute path as well (/home/ryan/linuxtutorialwork/myscript.sh ), and it would have worked exactly the same, or a relative path if you are not currently in the same directory as the script (../linuxtutorialwork/myscript.sh).

If it were possible to run scripts in your current directory without this mechanism, it would be easy, for instance, for someone to create a malicious script in a particular directory and name it ls or something similar. People would inadvertently run it if they wanted to see what was in that directory.

Permissions

A script must have the execute permission before you can run it, even if you are the owner of the file. For safety reasons, you don't have the execute permission by default, so you have to add it. A good command to run to ensure your script is set up right is chmod 755 <script>.

Variables

A variable is a container for a simple piece of data. They are useful if you need to work out a particular thing and then use it later on. Variables are easy to set and refer to, but they have a specific syntax you must follow exactly for them to work:

• When you set a variable, specify its name followed directly by the equals sign (=) followed directly by the value. No spaces on either side of the = sign.
• When you refer to a variable, place a dollar sign ($) before the variable name.

Example

user@bash: cat variableexample.sh
#!/bin/bash
# A simple demonstration of variables
# Ryan 25/4/2018
name='Ryan'
user@bash: echo Hello $name
user@bash: ./variableexample.sh
Hello Ryan
user@bash: 

Command-line arguments and more

When you run a script, there are several variables that get set automatically for you. Here are some of them:

• $0: script name.
• $1 to $9: any command-line arguments given to the script. $1 is the first argument, $2 is the second, and so on.
• $#: how many command-line arguments are given to the script.
• $*: all command-line arguments.

There are other variables, but these should be enough to get you going for now. See the example below to understand their usage.

user@bash: cat morevariables.sh
#!/bin/bash
# A simple demonstration of variables
# Ryan 25/4/2018
echo My name is $0, and I have been given $# command-line arguments
echo Here they are: $*
echo And the 2nd command-line argument is $2
user@bash: 
user@bash: ./morevariables.sh bob fred sally
My name is morevariables.sh, and I have been given 3 command line arguments
Here they are: bob fred sally
And the 2nd command-line argument is fred
user@bash: 

Backticks

It is also possible to save the output of a command to a variable, and the mechanism to use for the purpose is the backtick (\). Note it is a backtick, not a single quote. Typically, you can find the backtick on the keyboard to the left of the 1 (one) key. Here is an example:

user@bash: cat backticks.sh
#!/bin/bash
# A simple demonstration of using backticks
# Ryan 25/4/2018
lines=`cat $1 | wc -l`
echo The number of lines in the file $1 is $lines
user@bash: 
user@bash: ./backticks.sh testfile.txt
The number of lines in the file testfile.txt is 12
user@bash: 

Sample backup script

Now let's put the stuff you've learnt so far into a script that does something useful.

The below script backs up projects kept in separate folders within the projects directory in the home directory. The backup copies of the projects are saved to and kept in dated folders within the projectbackups directory, also in the home directory.

#!/bin/bash
# Backs up a single project directory
# Ryan 25/4/2018
date=`date +%F`
mkdir ~/projectbackups/$1_$date
cp -R ~/projects/$1 ~/projectbackups/$1_$date
echo Backup of $1 completed
user@bash: 
user@bash: ./projectbackup.sh ocelot
Backup of ocelot completed
user@bash: 

The script above uses relative paths, which makes it more generic. In this case, if a workmate wishes to use it, you can give them a copy, and it will work just as well for them without modification. You should always think about making your scripts flexible and generic so they may easily be used by other users or adapted to similar situations. The more reusable your scripts are, the more time goes on, the less work you have to do.

If statements

So the above backup script makes your life a little easier, but what if you make a mistake? The script may fall over in a mess of error messages. In the example below, if statements are introduced.

user@bash: cat projectbackup.sh
#!/bin/bash
# Backs up a single project directory
# Ryan 25/4/2018

if [ $# != 1 ]
then
    echo Usage: A single argument which is the directory to backup
    exit
fi
if [ ! -d ~/projects/$1 ]
then
    echo 'The given directory does not seem to exist (possible typo?)'
    exit
fi
date=`date +%F`

# Do we already have a backup folder for todays date?
if [ -d ~/projectbackups/$1_$date ]
then
    echo 'This project has already been backed up today, overwrite?'
    read answer
    if [ $answer != 'y' ]
    then
        exit
    fi
else
    mkdir ~/projectbackups/$1_$date
fi
cp -R ~/projects/$1 ~/projectbackups/$1_$date
echo Backup of $1 completed
user@bash: 

Let's break down the example above:

• Line 6 is your first if statement. The formatting is important. Note where the spaces are as they are required for it to work properly. In this statement, you are asking if the number of arguments ($#) is not equal to (!=) one.
• Line 8: if not, then the script has not been properly invoked. Print a message explaining how it should be used.
• Line 9: because the script has not been invoked properly, you wish to exit the script before going any further.
• Line 10: to indicate the end of an if statement, you have a single line which has fi (if backwards) on it.
• Line 11: If statements can test a lot of different things. Here, the exclamation mark (!) means not, -d means the path exists and is a directory. So the line reads as follows: If the given directory does not exist.
• Line 22: it is possible to ask the user for input. For that, use the read command. read takes a single argument, which is the variable to store the answer in.
• Line 23: let's see how the user responded and act accordingly.

You'll notice that certain lines are indented. This is not necessary but is generally considered a good practice as it makes the code a lot easier to read.

If statements make use of the test command. To know all the different comparisons you can perform, have a look at the Manual page for test.

Loops

Bash loops are very useful. This section highlights different loop formats available to you and discusses when and why you may want to use each of them.

Loops allow you to take a series of commands and keep re-running them until a particular situation is reached. They are helpful in automating repetitive tasks.

There are three basic loop structures in Bash scripting described in detail below. There are also a few statements you can use to control the operation of the loops.

while loops

One of the easiest loops to work with is while. The loop says, while an expression is true, keep executing these lines of code. It has the following format:

while [<some test>]
do
<commands>
done

Similar to if statements, the test is placed between square brackets ([ ]).

while_loop.sh

In the example below, you print the numbers from one through to ten.

#!/bin/bash
# Basic while loop

counter=1
while [ $counter -le 10 ]
do
echo $counter
((counter++))
done
echo All done

Let's break down the example above:

• On line 4, the variable counter is initialized with its starting value.

• Line 5 says, while the test is true (the counter is less than or equal to ten), let's do the following commands.

• On line 7, place any commands you like. In the example above, echo is used as it's an easy way to illustrate what is going on.

• On line 8, using the double brackets, you can increase the counter value by one.

• Line 9 is the bottom of the loop, so the script goes back to line 5 and performs the test again. If the test is true, then it executes the commands. If the test is false, then it continues executing any commands following done.

./while_loop.sh
1
2
3
4
5
6
7
8
9
10
All done

A common mistake is what's called an off-by-one error. In the example above, you could have put -lt as opposed to -le (less than as opposed to less than or equal). Had you done this, it would have printed up until nine. The mistake is easy to make but also easy to fix once you've identified it.

until loops

The until loop is fairly similar to the while one. The difference is that it executes the commands within it until the test becomes true.

until [<some test>]
do
<commands>
done

until_loop.sh

#!/bin/bash
# Basic until loop
counter=1
until [ $counter -gt 10 ]
do
echo $counter
((counter++))
done
echo All done

As you can see in the example above, the syntax is almost exactly the same as the while loop (just replace while with until). You can also create a script that does exactly the same as the while example above by changing the test accordingly.

So you may be wondering why bother having the two different kinds of loops? In fact, you don't have to. The while loop would be able to handle every scenario. Sometimes, however, it is easier to read if we phrase it with until rather than while.

Think about the following statement: "Leave the towel on the line until it's dry." You could have said: "Leave the towel on the line while it is not dry." Or: "Leave the towel on the line while it is wet."

But they just don't seem as elegant and easy to understand. So by having both while and until, you can pick whichever one makes the most sense to you. As a result, you end up with code that is easier for you to understand when you read it.

You should always strive for clean, obvious, and elegant code when writing Bash scripts.

for loops

The for loop is a little bit different from the previous two one. It says for each of the items in a given list: "perform the given set of commands." It has the following syntax:

for var in <list>
do
<commands>
done

The for loop works as follows:

1. Takes each item in the list one after the other.
2. Assigns that item as the var variable value.
3. Executes the commands between do and done.
4. Goes back to the top, grabs the next item in the list, and repeats over.

The list is defined as a series of strings separated by spaces.

for_loop.sh

#!/bin/bash
# Basic for loop
names='Stan Kyle Cartman'
for name in $names
do
echo $name
done
echo All done

Let's break down the above example:

• On line 4, you create a simple list, which is a series of names.

• On line 6, each item in the $names list is assigned to the $name variable, and then the commands are executed that follow do.

• On line 8, echo prints the name to the screen just to show that the mechanism works. Here, you can have as many commands as you like.

• On line 11, echo prints another command to show that the Bash script continued execution as expected after all items in the list were processed.

./for_loop.sh
Stan
Kyle
Cartman
All done

Ranges

You can also process a series of numbers.

for_loop_series.sh

#!/bin/bash
# Basic range in for loop

for value in {1..5}
do
echo $value
done
echo All done

On line 4, when specifying a range like this, make sure there are no spaces between the curly brackets ({ }). If there are, it is not seen as a range but as a list of items.

./for_loop_series.sh
1
2
3
4
5
All done

When specifying a range, you can specify any number you like for the starting and ending values. The first value may also be larger than the second one, in which case it counts down.

It is also possible to specify a value to increase or decrease by each time. You do this by adding another two dots (..) and the value to step by.

for_loop_stepping.sh

#!/bin/bash
# Basic range with steps for loop

for value in {10..0..2}
do
echo $value
done
echo All done

./for_loop.sh
10
8
6
4
2
0
All done

One of the more useful applications of the for loop is processing of a file set. To do this, use wildcards. Let's say you want to convert a series of HTML files over to PHP files.

convert_html_to_php.sh

#!/bin/bash
# Make a php copy of any html files
for value in $1/*.html
do
cp $value $1/$( basename -s .html $value ).php
done

Controlling loops: break and continue

Most of the time, your loops are going through in a smooth and orderly manner. Sometimes, however, you may need to intervene and alter their running slightly. There are two statements you can issue to do this.

break

The break statement tells Bash to leave the loop straight away. It may be that there is a normal situation that should cause the loop to end, but there are also exceptional situations in which it should end. For instance, maybe you are copying files, but if the free disk space gets below a certain level, you should stop copying.

copy_files.sh

#!/bin/bash
# Make a backup set of files
for value in $1/*
do
used=$( df $1 | tail -1 | awk '{ print $5 }' | sed 's/%//' )
if [ $used -gt 90 ]
then
echo Low disk space 1>&2
break
fi
cp $value $1/backup/
done

continue

The continue statement tells Bash to stop running through this loop iteration and begin the next iteration.

Sometimes, there are circumstances that stop you from going any further. For instance, maybe you are using the loop to process a series of files, but if you stumble upon a file you don't have the read permission for, you should not try to process it.

copy_check.sh

#!/bin/bash
# Make a backup set of files
for value in $1/*
do
if [ ! -r $value ]
then
echo $value not readable 1>&2
continue
fi
cp $value $1/backup/
done

Activities for practicing

To solve these activities, bring together your skills and knowledge from this article and previous ones.

• First off, think about writing your backup script. You can make it as simple or complex as you like. Maybe start with a simple one and improve it progressively.

• Now see if you can write a script that will give you a report about a given directory. Things you could report include the following:

  • How many files are there in the directory?
  • How many folders are there in the directory?
  • What is the biggest file?
  • What is the most recently modified or created file?
  • A list of people who own files in the directory.
  • Anything else you can think of.