Over the course of the term you have seen a number of examples where we have used the shell to issue commands. There is a wide universe of powerful tools available through the shell in Unix systems, and we have only just begun to tap the power of the shell and the utility programs. Another powerful aspect of the shell is shell scripting, which allows us to write programs in a shell scripting language to automate common tasks.
In these notes I am going to walk you through the development process for a simple shell script. Along the way we are going to learn the basics of shell scripting and get an introduction to some powerful utility programs.
Suppose you are a system administrator in a Unix system. One of the applications running on your system is the folder watcher system that we have been developing. A common maintenance task that an administrator would have to perform with this system is adding or removing watched folders. Of course, an adminstrator could always do this by editing the /etc/fwd.conf file, stopping the fwd application, and then restarting it. Alternatively, a skilled system administrator would just construct a script that could automate these tasks, making it possible to do all of these things by just issuing a single command.
In its simplest form, the problem of adding a new watched folder to the fwd system involves adding a single line of text to the configuration file. We could try to do this from the shell by typing the command
"newTag /path/to/new/folder" > /etc/fwd.conf
There are two things wrong with this. The first is that when you type a line in the shell the shell will try to interpret what you have typed as a command. This will produce an error message:
bash: newTag /path/to/new/folder: No such file or directory
What we need here is a command that will generate the desired text. The > operator will then redirect the output of that command to the conf file. The echo command is what we need to do this:
echo "newTag /path/to/new/folder" > /etc/fwd.conf
This is also not quite right, since the redirection operator > ends up replacing the entire contents of the file in question with the one line we wanted to add. What we need instead here is to use an alternate operator, >>, which appends text to the file instead of wiping out the previous contents.
echo "newTag /path/to/new/folder" >> /etc/fwd.conf
This will do what we want, but may be an incomplete solution. A problem that this may not address is that the tag we are trying to use here may already be in use in the conf file. We should start by searching the existing conf file to see whether or not that is the case. The powerful grep function is our friend here. grep can search a file for a pattern that you supply and then print all of the lines in the file where that pattern appears.
grep newTag /etc/fwd.conf
This will work, but may produce more output than we wanted. This command prints all of the lines in fwd.conf where newTag appears. What we really want to see instead is all of the lines where newTag appears at the start of the line. To do this, we can run the search with a pattern that includes the special symbol ^. Patterns that start with ^ will return only lines where newTag appears at the start of the line:
grep "^newTag " /etc/fwd.conf
The search pattern here also includes a space after newTag, so that we don't accidentally catch lines that start with tags like newTag1 or newTag52.
Now that we have developed a couple of useful shell commands to help with our task, it is time to start constructing a script that will run these commands for us.
Below is the text for our first simple script, simple.sh.
#!/bin/bash exists=$(grep "^$1 " /etc/fwd.conf) if [ -z $exists ] then echo "$1 $2" >> /etc/fwd.conf kill -2 $(cat /run/fwd.pid) /opt/fwd/fwd else echo "Tag $1 already exists." fi
Here are some things to note in this code:
1. The first line in the script
#!/bin/bash
specifies what shell program we would like to run this script in. Many Unix systems have several different shell programs installed, and each of these alternative shells has its own syntax for shell scripting. To ensure that our script will run correctly, we have to specify which shell program to run the script in. The bash shell is the one of the most popular and widely used shells, and you can find a lot of documentation and information online on how to use the bash scripting language.
2. At various places in the script you will see the expressions $1 and $2. These are variable evaluations. The variables in question here are command line variables, whose values are command line parameters that the user will supply when they run the script. For example, if the user invokes this script by running the command
./first.sh newTag /path/to/new/folder
Then $1 will evaluate to newTag and $2 will evaluate to /path/to/new/folder.
3. The next line in the script sets up a variable exists. Variables in a script can contain text or integers. In most cases we set up variables to hold text. The value for this particular variable comes from what is called a command substitution. The expression $(<cmd>) runs the command <cmd> and then evaluates to the text that would be printed by that command. The command that we are running in this case is a grep search in the /etc/fwd.conf file that looks for lines that start with the tag that we are interested in. If the search finds a line, the expression will evaluate to the text of that line. If the search finds no lines, the expression will evaluate to an empty string. When the shell evaluates a command substitution expression it will first replace any variable evaluations in the command with their values and then run the command.
4. This script also contains a simple example of an if/else statement. The test in an if/else statement is supplied by a condition. The condition [ -z $exists ] that we used here checks whether the string stored in the exists variable is empty. If it is, this means that the new tag that we want to insert does not already exist and it is safe to add it to the fwd.conf file. If the exists variable is non-empty we instead just want to print an error message. You can read more about conditions in bash scripting on this page.
5. After successfully updating the conf file we will want to restart the fwd server. To do that we need to run the kill command with the pid of the fwd application. To get that pid we use a command substitution that runs the cat command to print the contents of the /run/fwd.pid file.
Before we can run the script we just created we have to do one additional step. To run a script file we have to make the file executable. We do this by running the chmod command in a terminal to change the file permissions for the file:
chmod +x simple.sh
This adds the execute permission to the file. To run our script we then would invoke the same way we would invoke any other program:
./simple.sh newTag /path/to/folder
Now that we have a script that does most of what we need done, we should go ahead and add a few additional features to make it more correct. In particular, we want to add these features:
Here is an expanded version of the script that adds these features:
#!/bin/bash
tag=$(grep -w "^$1" /etc/fwd.conf)
path=$(grep -w "$2\$" /etc/fwd.conf)
restart="true"
if [ -n "$tag" ]
then
read -p "Tag $1 already exists. Replace it?[y/n]" yn
if [ yn != "n" ]
then
rest=$(grep -v "^$1" /etc/fwd.conf)
echo $rest > /etc/fwd.conf
echo "$1 $2" >> /etc/fwd.conf
else
restart="false"
fi
else
if [ -n "$path" ]
then
read -p "Path $2 already exists. Replace it?[y/n]" yn
if [ yn != "n" ]
then
rest=$(grep -v " $2\$" /etc/fwd.conf)
echo $rest > /etc/fwd.conf
echo "$1 $2" >> /etc/fwd.conf
else
restart="false"
else
echo "$1 $2" >> /etc/fwd.conf
fi
fi
if [ $restart == "true" ]
then
kill -2 $(cat /run/fwd.pid)
/opt/fwd/fwd
fi
This version of the script starts by doing two separate searches with grep. The first search locates entries that use the same tag as $1, while the second search finds lines that end with the same path as $2. Putting a $ at the end of a grep pattern finds that pattern when it occurs at the end of a line. Since the $ character has a special meaning in shell scripts we have to escape that character as \$ to use it in the grep command.
If we find that the tag the user wants to add exists already we will have to ask the user whether or not they want to replace it. To do this we use the read command to ask the user for input and then put the user's response into the yn variable.
Should the user want to replace the existing entry that uses the same tag, we have to then get every line but that line from the file. To do this we can make use of the -v option in grep, which will return every line in the file but the lines that match the pattern.
Another common use for shell scripts is to generate reports that summarize data from data collections such as log files. In this next example we are going to construct a script that gives a summary of what files had changes recorded in the fwd.log file and how many times those files are modified. For example, if the fwd.log file contains the entries
a one 8980 b five 9002 a two 9022 a two 9025 a one 9067 a one 9077 b three 9234 b three 9344 b five 9546 b five 9607 b five 9714 a two 9898
the reporting script would generate a report that looks like this:
a one 3 a two 3 b five 4 b three 2
In the notes below I am going to be developing a script that can generate this report from the fwd.log file. I will start the process of developing this script, and then leave the rest of the work to you as a homework assignment.
An obvious ingredient in our script is a loop that can iterate through the contents of the fwd.log file. Our first simple example script demonstrates how to write a simple loop that reads lines of input.
#!/bin/bash while read line do echo "$line" >> output.txt done
This example demonstrates the use of the bash while loop. The command that controls this loop is
read line
which invokes the bash read command to read a line of input from the standard input and then store that line in a variable named line. In the body of the loop we take each line read and append it to a text file named output.txt.
When you run this script it will read lines of text that you type into the terminal. The while loop will stop when you terminate the read by pressing the control-d key combination. After running the script you can open the output.txt file to see that the text that you typed has been stored in the file.
In the next example we are going to construct a similar while loop. The difference this time is that we are going to use redirection to have the while loop read from a file instead of reading from the standard input.
#!/bin/bash while read line do echo "$line" done < fwd.log
The expression < fwd.log after the end of the while loop is a file redirection that redirects the contents of the fwd.log file into the while loop so that the read command will be reading from that file instead of from the standard input. When you run this script it will print the contents of the fwd.log file to the output.
The next example demonstrates a trick that you can do with the read command. In the last example we used the read command to read a line of text from the input file into the line variable. In this example we will instead read the three things that appear on each line of the log file into three separate variables.
#!/bin/bash while read tag file time do echo "$tag $file" done < fwd.log
When you run this script it will print just the first two columns of the log file to the console.
At this point you have a start on building the reporting script we want to construct. I am now going to leave the rest of the script to you as a homework exercise. Here are some additional things your script should do: