The name of the program awk is made from the initials of three authors of the language, namely Alfred Aho, Peter Weinberger and Brian Kernighan. It is not very clear why they selected the name awk instead of kaw or wak!

The following are different ways to use awk:

As seen previously, the awk instruction consists of patterns, actions, or a combination of both.

Actions will be enclosed in curly brackets. Actions can contain many statements separated by a semicolon or a newline.

awk commands can be on the command line or in the awk script file. The input lines could be received from keyboard, pipe, or a file.

$ cat people.txt

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977

$ awk '/Martin/' people.txt

Fred Martin  6500  22/7/1982

$ cat people.txt

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977

$ awk '{print $1}' people.txt

Bill
Fred
Julie
Marie
Tom

$ cat people.txt

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977

$ awk '/Martin/{print $1, $2}' people.txt
Fred Martin 

We can use the output of any other Linux command as an input to the awk program. We need to use the pipe to send an output of other command as the input to the awk program.

The syntax is as follows:

$ command | awk 'pattern'
$ command | awk '{action}'
$ command | awk 'patter	n {action}'

For example:

$ cat people.txt | awk '$3 > 6500'

The output is as follows:

Bill Thomas  8000  08/9/1968
Tom Walker   7000  14/1/1977

This prints all lines, in which field 3 is greater than 6500.

For example:

$ cat people.txt | awk '/1972$/{print $1, $2}'

The output is as follows:

Marie Jones

This prints fields 1 and 2 of the lines, which ends with the 1972 pattern:

$ cat people.txt | awk '$3 > 6500 {print $1, $2}'

This prints fields 1 and 2 of the line, in which the third field is greater than 6500.

Let's understand how the awk program processes every line. We will consider a simple file, sample.txt:

$ cat sample.txt
Happy Birth Day
We should live every day.

Let's consider the following awk command:

$ awk '{print $1, $3}' 
sample.txt

The following diagram shows, how the awk will process every line in memory:

The explanation about the preceding diagram is as follows:

When awk checks an action as '{print $1, $3}', it tells awk to print the first and third fields. Fields will be separated by space. The command will be as follows:

$ awk '{print $1, $3}' sample.txt

The output will be as follows:

Happy Day
We live

the explanation of the output is as follows:

We can put awk commands in a file. We will need to use the -f option before using the awk script file name to inform about using the awk script file for all processing instructions. awk will copy the first line from the data file to be processed in $0, and then, it will apply all processing instructions on that record. Then, it will discard that record and load the next line from the data file. This way, it will proceed till the last line of the data file. If the action is not specified, the pattern matching lines will be printed on screen. If the pattern is not specified, then the specified action will be performed on all lines of the data file.

For example:

$ cat people.txt
Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977
$ cat awk_script 
/Martin/{print $1, $2}

Enter the next command as follows:

$ awk -f awk_script people.txt

The output is as follows:

Fred Martin

The awk command file contains the Martin pattern and it specifies the action of printing fields 1 and 2 of the line, matching the pattern. Therefore, it has printed the first and second fields of the line, containing the Martin pattern.

Every line terminated by the newline is called record and every word separated by white space is called field. We will learn more about them in this section.

$ cat people.txt

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977
$ awk '{print $0}' people.txt

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977

$ awk 
'{print}' people.txt

$ cat people.txt

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977
$ awk '{print NR, $0}'  people.txt

1 Bill Thomas  8000  08/9/1968
2 Fred Martin  6500  22/7/1982
3 Julie Moore  4500  25/2/1978
4 Marie Jones  6000  05/8/1972
5 Tom Walker   7000  14/1/1977

$1    $2        $3       $4
Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977
$ awk '{print NR, $1, $2, $4}' people.txt

1 Bill Thomas 08/9/1968
2 Fred Martin 22/7/1982
3 Julie Moore 25/2/1978
4 Marie Jones 05/8/1972
5 Tom Walker  14/1/1977

$ cat people.txt

Bill Thomas:8000:08/9/1968
Fred Martin:6500:22/7/1982
Julie Moore:4500:25/2/1978
Marie Jones:6000:05/8/1972
Tom Walker:7000:14/1/1977
$ awk -F: '/Marie/{print $1, $2}' people.txt

Marie Jones 6000

$ awk –F'[ :\t]'  '{print $1, $2, $3}' people.txt

While executing commands using awk, we need to define patterns and actions. Let's learn more about them in this section.

$ cat people.txt

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977
$ awk '/Bill/' people.txt

Bill Thomas  8000  08/9/1968

$ awk '$3 > 5000' people.txt

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977

pattern{ action statement; action statement; .. }
     or
pattern
{    action statement
       action statement
}

$ awk '/Bill/{print $1, $2 ", Happy Birth Day !"}' people.txt

Bill Thomas, Happy Birth Day !

The regular expressions is a pattern enclosed in forward slashes. Regular expression can contain metacharacters. If the pattern matches any string in the record, then the condition is true and any associated action, if mentioned, will be executed. If no action is specified, then simply the record is printed on screen.

Metacharacters used in awk regular expressions are as follows:

In the following example, all lines containing regular expression "Moore" will be searched and matching record's field 1 and 2 will be displayed on screen:

$ awk  '/Moore/{print $1, $2}' people.txt

The output is as follows:

Julie Moore

Whenever we need to write multiple patterns and actions in a statement, then it is more convenient to write a script file. The script file will contain patterns and actions. If multiple commands are on the same line, then those should be separated by a semicolon; otherwise, we need to write them on separate lines. The comment line will start by using the pound (#) sign.

For example:

$ cat people.txt

The output is as folllows:

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977

(The awk script)

$ cat report

The output is as follows:

/Bill/{print "Birth date of " $1, $2 " is " $4}
/^Julie/{print $1, $2 " has a salary of  $" $3 "."}
/Marie/{print NR, $0}

Enter the next command as follows:

$ awk -f report people.txt

The output will be:

Birth date of Bill Thomas is 08/9/1968
Julie Moore has a salary of $4500.
4 Marie Jones  6000  05/8/1972

In this example, the awk command is followed by the -f option, which specifies the script file as record and then processes all the commands on the text file people.txt.

In this script, regular expression Bill is matched, then print text, field 1, field 2, and then print the birth date information. If the regular expression Julie is matched at the start of the line, then print her salary information. If regular expression Marie is matched, then print the record number NR and print the complete record.

We can simply declare a variable in the awk script, without even any initialization. Variables can be of type string, number, or floating type and so on. There is no type declaration required like in C programming. awk will find out the type of variable by its right-hand side data type during initialization or its usage in the script.

Uninitialized variables will have the value 0 or strings will have a value null such as "", depending on how it is used inside scripts:

name = "Ganesh"

The variable name is of the string type:

j++

The variable j is a number. Variable j is initialized to zero and it is incremented by one:

value = 50

The variable value is a number with initial value 50.

The technique to modify the string type variable to the number type is as follows:

name + 0

The technique to modify the number type variable to the string type is as follows:

value " "

User-defined variables can be made up of letters, digits, and underscores. The variable cannot start with a digit.

In awk programming, the if statement is used for decision making. The syntax is as follows:

if (conditional-expression)
  action1
else
  action2

If the condition is true, then action1 will be performed, else action2 will be performed. This is very similar to C programming if constructs.

An example of using the if statement in the awk command is as follows:

$ cat person.txt

The output is as follows:

Bill Thomas  8000  08/9/1968
Fred Martin  6500  22/7/1982
Julie Moore  4500  25/2/1978
Marie Jones  6000  05/8/1972
Tom Walker   7000  14/1/1977
$ awk '{
if ($3 > 7000) { print "person with salary more than 7000 is \n", $1, " " , $2;}
}' people.txt

The output is as follows:

person with salary more than 7000 is
Bill Thomas

In this example, field 3 is checked for greater than 7000 in every record. If field 3 is greater than 7000 for any record, then the action of printing the name of the person and value of third record will be done.

The for loop is used for doing certain actions repetitively. The syntax is as follows:

for(initialization; condition; increment/decrement)
actions

Initially, a variable is initialized. Then, the condition is checked, if it is true, then action or actions enclosed in curly brackets are performed. Then, the variable is incremented or decremented. Again, the condition is checked. If the condition is true, then actions are performed, otherwise, the loop is terminated.

An example of the awk command with the for loop is as follows:

$ awk '{ for( i = 1; i <= NF; i++) print NF, $i }' people.txt

Initially, the i variable is initialized to 1. Then, the condition is checked to see whether i is less than NF. If true, then the action of printing NF and the field is performed. Then i is incremented by one. Again, the condition is checked if it is true or false. If true, then it will perform actions again; otherwise, it will terminate the looping activity.

Similar to C programming, awk has a while loop for doing the tasks repeatedly. while will check for the condition. If the condition is true, then actions will be performed. If condition is false, then it will terminate the loop.

The syntax is as follows:

  while(condition)
    actions

An example of using the while construct in awk is as follows:

$ cat people.txt
$ awk '{ i  = 1; while ( i <= NF ) { print NF, $i ; i++ } }' people.txt

NF is the number of fields in the record. The variable i is initialized to 1. Then, while i is smaller or equal to NF, the print action will be performed. The print command will print fields from the record from the file people.txt. In the action block, i is incremented by one. The while construct will perform the action repeatedly until i is less than or equal to NF.

The do while loop is similar to while loop; but the difference is, even if the condition is true, at least once the action will be performed unlike the while loop.

The syntax is as follows:

do
action
while (condition)

After the action or actions are performed, the condition is checked again. If the condition is true, then the action will be performed again, otherwise, the loop will be terminated.

The following is an example of using the do while loop:

$ cat awk_script
BEGIN {
  do {
    ++x
    print x
  } while ( x <= 4 )
}
$ awk -f awk_script
1
2
3
4
5

In this example, x is incremented to 1 and value of x is printed. Then the condition is checked to see whether x is less than or equal to 4. If the condition is true, then the action is performed again.