Whenever you use sed commands on a text file, sed reads the first line of the file and stores it in a temporary buffer called pattern space. The sed processes this pattern space buffer as per commands given by the user. Then, it prints the output on screen. This line from the pattern space is then removed and the next line of the file is loaded in the pattern space. In this way, it processes all the lines one by one. This line-by-line processing is continued till the last line of the file. As the sed commands are processed on the temporary buffer or pattern space, the original line is not modified. Therefore, we say sed is a nondestructive buffer.

While using sed, regular expressions are enclosed in forward slashes. As grep and sed use regular expressions and metacharacters for searching patterns in the file. For example:

sed -n '/Regular_Expression/p' filename
sed -n '/Mango/p' filename

This will print lines matching the Mango pattern:

sed -n 's/RE/replacement string/' filename
sed -n 's/Mango/Apple/' filename

This will find the line containing the Mango pattern and then the Mango pattern will be replaced by the Apple text. This modified line will be shown on screen and the original file will be unchanged.

The following is a summary of various metacharacters and it's usage in sed:

We can specify which line or number of lines the pattern search and commands are to be applied on while using the sed commands. If line numbers are not specified, then the pattern search and commands will be applied on all lines of the input file.

The line numbers on which commands are to be applied are called address. Address can be a single line number or range of lines in which the starting number of the line and the ending number of the range will be separated by commas. Ranges can be composed of numbers, regular expressions or a combination of both.

The sed commands specify actions such as printing, removing, replacing, and so on.

The syntax is as follows:

sed 'command' filename(s
)

Example:

$ cat myfile |  sed '1,3d'

Otherwise it can be:

sed '1,3d' myfile

This will delete lines 1 to 3:

sed -n '/[Aa]pple/p' item.list

If the Apple or apple pattern is found in the item.list file, then those lines will be printed on screen and the original file myfile will be unchanged.

To negate the command, the exclamation character (!) can be used.

Example:

sed '/Apple/d' item.list

This tells sed to delete all the lines containing the Apple pattern.

Consider the following example:

sed '/Apple/!d' item.list

This will delete all the lines except the line containing the Apple pattern.

The sed is a nondestructive editor. This means the output of sed is displayed on screen; but the original file is unchanged. If we want to modify the file, then we can redirect the output of the sed command to the file. Deleting lines is illustrated in the following examples:

$ sed '1,3d' datafile >  tempfile
$ mv tempfile newfile

In this example, we have deleted lines 1 to 3 and stored the output in tempfile. Then, we have to rename tempfile to newfile.

By default, the action of the sed command is to print the pattern space, such as every line which is copied in buffer, and then print the result of processing on it. Therefore, the sed output will consist of all lines along with the processed line by sed. If we do not want the default pattern space line to be printed, then we need to give the –n option. Therefore, we should use the –n option and the p command together to see the result of the sed processed output.

Here is an example:

$ cat country.txt

The output is as follows:

Country  Capital     ISD Code
USA      Washington  1
China    Beijing     86
Japan    Tokyo       81
India    Delhi       91
$ sed '/USA/p'country.txt

The output is as follows:

Country  Capital     ISD Code
USA      Washington   1
USA      Washington   1
China    Beijing     86
Japan    Tokyo       81
India    Delhi       91

All the lines from the file are printed by default and the lines with the USA pattern are also printed:

$ sed –n '/USA/p' country.txt

The output is as follows:

USA  Washington  1

As we have given the –n option, sed has suppressed default printing of all lines from the country file; but has printed the line that contains the text pattern USA.

The d command is used to delete lines. After sed copies a line from a file and puts it into a pattern buffer, it processes commands on that line, and finally, displays the contents of the pattern buffer on screen. When the d command is issued, the line currently in the pattern buffer is removed, not displayed which is shown as follows:

$ cat country.txt
Country  Capital     ISD Code
USA      Washington   1
China    Beijing     86
Japan    Tokyo       81
India    Delhi       91
$ sed '3d' country.txt

The output is as follows:

Country      Capital     ISD Code
USA          Washington   1
Japan        Tokyo       81
India        Delhi       91

Here is the explanation.

The output will contain all the lines except the third line. The third line is deleted by the following command:

$ sed '3,$d' country.txt

The output is as follows:

Country       Capital     ISD Code
USA           Washington  1

This will delete third line to the last line. The dollar sign in the address indicates the last line. The comma is called ra nge operator.

$ sed '$d' country.txt

The output is as follows:

Country       Capital     ISD Code
USA           Washington   1
China         Beijing     86
Japan         Tokyo       81

Here is the explanation.

This deletes the last line. All lines except lines will be displayed.

Here is an example:
$ sed '/Japan/d' country.txt

The output is as follows:

Country       Capital     ISD Code
USA           Washington    1
China         Beijing      86
India         Delhi        91

The line containing the Japan pattern is deleted. All other lines are printed:

$ sed '/Japan/!d' country.txt

The output is as follows:

Japan       Tokyo         81

This has deleted all the lines that do not contain Japan.

Let's see a few more examples with the delete command.

This will delete line 4 and the next five lines:

$ sed '4,+5d'

This will keep lines 1 to 5 and delete all the other lines:

$ sed '1,5!d'

This will delete lines 1, 4, 7, and so on:

$ sed '1~3d'

Starting from 1, every third line step increments. The number that follows the tilde is what is called the step increment. The step increment indicates the following:

$ sed '2~2d'

This will delete every other line starting with line 2 to be deleted.

If we want to substitute the text by new text, then we can use commands. After the forward slash, the regular expression is enclosed and then the text to be substituted is placed. If the g option is used, then substitution will happen globally, meaning that it will be applied in the full document. Otherwise, only the first instance will be substituted:

$ cat shopping.txt

The output is as follows:

Product     Quantity   Unit_Price  Total_Cost
Apple        2           3          6
Orange       2           .8         1.6
Papaya       2           1.5        3
Chicken      3           5         15
Cashew       1     
     10         10
$ sed 's/Cashew/Almonds/g' shopping.txt

The output is as follows:

Product     Quantity   Unit_Price  Total_Cost 
Apple       2           3           6
Orange      2            .8         1.6
Papaya      2           1.5         3
Chicken     3           5          15
Almonds     1          10          10

The s command has replaced Cashew by Almonds. The g flag at the end indicates that the substitution is to be applied globally. Otherwise, it will be applied to the first pattern match only.

The following substitution command will replace two digit numbers at the end of the line with .5 appended to them:

$ sed 's/[0–9][0–9]$/&.5/' shopping.txt

The output is as follows:

Product     Quantity  Unit_Price  Total_Cost
Apple       2          3            6
Orange      2           .8          1.6
Papaya      2          1.5          3
Chicken     3          5           15.5
Cashew      1         10           10.5

The ampersand in the search pattern represents the exact pattern found. This will be replaced by the exact pattern with .5 appended to it.

To use the sed effectively, we should be clear about how to define range. Range is typically two addresses in a file as follows:

As mentioned, we can specify the range as numbers, pattern, or a combination of both.

For example:

$ cat country.txt
Country  Capital     ISD Code
USA      Washington    1
China    Beijing      86
Japan    Tokyo        81
India    Delhi        91
$ sed -n '/USA/,/Japan/p' country.txt

The output is as follows:

USA     Washington     1
China   Beijing       86
Japan   Tokyo         81

In this example, all the lines between addresses starting from USA and until the pattern Japan will be printed on screen.

For example:

$ sed -n '2,/India/p' country.txt

The output is as follows:

USA      Washington    1
China    Beijing      86
Japan    Tokyo        81
India    Delhi        91

In this example, line 2 to the pattern India, are printed on screen.

For example:

$ sed '/Apple/,/Papaya/s/$/**   Out of Stock   **/' shopping.txt

The output is as follows:

Product  Quantity  Unit_Price  Total_Cost
Apple    2          3             6**       Out of Stock    **
Orange   2           .8           1.6**     Out of Stock    **
Papaya   2          1.5           3**       Out of Stock    **
Chicken  3          5            15
Cashew  1          10            10

In this example. for all the lines between the Apple and Papaya patterns, the end of line will be replaced by the ** Out of Stock ** string.

If we need to perform multiple editing by the same command, then we can use the –e command. Each edit command should be separated by the –e command. The sed will apply each editing command separated by –e on the pattern space before loading the next line in the pattern space:

$ cat shopping.txt

The output is as follows:

Product     Quantity  Unit_Price  Total_Cost
Apple       2          3            6
Orange      2           .8          1.6
Papaya      2          1.5          3
Chicken     3          5           15
Cashew      1         10           10

For example:

sed -e '5d' -e 's/Cashew/Almonds/' shopping.txt

The output is as follows:

Product  Quantity  Unit_Price  Total_Cost
Apple    2          3           6
Orange   2           .8          1.6
Papaya   2          1.5          3
Almonds  1         10          10

Initially, the command for deleting the fifth line is called, then, the next substitution command to replace Cashew by Almonds is processed.

If we need to insert text from another file into the file, which is processed by sed, then we can use the r command. We can insert text from another file to the specified location:

For example:

$ cat new.txt

The output will be:

*********************************
    Apples are out of stock

$ sed '/Apple/r new.txt' shopping.txt

The output is as follows:

Product     Quantity  Unit_Price  Total_Cost
Apple       2         3           6
*********************************
    Apples are out of stock
*********************************
Orange      2          .8         1.6
Papaya      2         1.5         3
Chicken     3         5          15
Cashew      1        10          10

The explanation is that, this command has added the content of the new.txt file after the line containing the Apple pattern.

The sed command for write is w. Using this command, we can write lines from one file to another file.

For example:

$ cat new.txt 

The output is as follows:

new is a empty file

$ sed -n '/Chicken/w new.txt' shopping.txt
$ cat new.txt 
Chicken    3    5    15

After the w command, we specify the file to which we will perform the write operation. In this example, the line containing the Chicken pattern is written to the new.txt file.

The a command is used for appending. When the append command is used, it appends the text after the line in the pattern space, in which the pattern is matched. The backslash should be placed immediately after the a command. On the next line, the text to be appended is to be placed.

For example:

$ cat shopping.txt

The output is as follows:

Product     Quantity  Unit_Price  Total_Cost
Apple       2          3           6
Orange      2           .8         1.6
Papaya      2          1.5         3
Chicken     3          5          15
Cashew      1         10          10

$ sed '/Orange/a\
**** Buy one get one free offer on this item ! ****' shopping.txt

The output is as follows:

Product   Quantity  Unit_Price  Total_Cost
Apple     2          3           6
Orange    2           .8         1.6
**** Buy one get one free offer on this item ! ****
Papaya    2          1.5         3
Chicken   3          5          15
Cashew    1         10          10

The new text **** Buy one get one free offer on this item ! **** is appended after the line containing the Orange pattern.

The i command is used for inserting text above the current pattern space line. When we use the append command, new text is inserted after the current line which is in the pattern buffer. In this similar-to-append command, the backslash is inserted after the i command.

For example:

$ cat shopping.txt
Product     Quantity  Unit_Price  Total_Cost
Apple       2          3            6
Orange      2           .8          1.6
Papaya      2          1.5          3
Chicken     3          5           15
Cashew      1         10           10 
$ sed '/Apple/i\                                         
       New Prices will apply from Next month ! ' shopping.txt

The output is as follows:

Product     Quantity  Unit_Price  Total_Cost
       New Prices will apply from Next month ! 
Apple          2      3            6
Orange         2       .8          1.6
Papaya         2      1.5          3
Chicken        3      5           15
Cashew         1     10           10

In this example, the new text New Prices will be applied from next month! is inserted before the line containing the Apple pattern. Please check the i command and the backslash following it.

The c command is the change command. It allows the sed to modify or change existing text with new text. The old text is overwritten with the new:

$ cat shopping.txt

The output is as follows:

Product     Quantity  Unit_Price  Total_Cost
Apple        2         3           6
Orange       2          .8         1.6
Papaya       2         1.5         3
Chicken      3         5          15
Cashew       1        10          10

For example:

$ sed '/Papaya/c\
  Papaya is out of stock today !' shopping.txt

The output is as follows:

Product  Quantity  Unit_Price  Total_Cost
Apple     2         3           6
Orange    2         .8          1.6
  Papaya is out of stock today !
Chicken   3        5           15
Cashew    1       10           10

In this example, the line containing the expression Papaya is changed by the new line Papaya is out of stock today!.

The command transform is similar to the Linux tr command. The characters are translated as per character sequence given. For example, y/ABC/abc/ will convert lowercase abc into uppercase ABC.

For example:

$ cat shopping.txt

The output will be:

Product     Quantity  Unit_Price  Total_Cost
Apple        2         3           6
Orange       2          .8         1.6
Papaya       2         1.5         3
Chicken      3         5          15
Cashew       1        10          10
$ sed '2,4y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' shopping.txt

The output will be:

Product  Quantity  Unit_Price  Total_Cost
APPLE       2        3           6
ORANGE      2        .8          1.6
PAPAYA      2       1.5          3
Chicken     3       5           15
Cashew      1      10           10

In this example, on lines 2, 3 and 4, all the lowercase letters are converted to uppercase letters.

The q command is used for quitting the sed processing without proceeding to the next lines:

$ cat shopping.txt

The output will be:

Product     Quantity  Unit_Price  Total_Cost
Apple         2        3           6
Orange        2        .8          1.6
Papaya        2       1.5          3
Chicken       3       5           15
Cashew        1      10           10

For example:

$ sed '3q' shopping.txt

The output will be:

Product  Quantity  Unit_Price  Total_Cost
Apple     2         3          6
Orange    2         .8         1.6

In this example, after printing the first to third lines, sed quits further processing.

We have already seen that the sed has pattern buffer. The sed has one more type of buffer called holding buffer. By the h command, we can inform sed to store the pattern buffer in the holding buffer. And whenever we need the line that is stored in the pattern buffer, we can get it by the g command, that is, get the buffer.

For example:

$ sed -e '/Product/h' -e '$g' shopping.txt

The output is as follows:

Product  Quantity  Unit_Price  Total_Cost
Apple     2          3           6
Orange    2          .8          1.6
Papaya    2         1.5          3
Chicken   3         5            15
Cashew    1        10            10
Product  Quantity  Unit_Price  Total_Cost

In this example, the line containing the Product pattern is stored in the holding buffer by the h command. Then, the next editing command indicates to the sed to get the line from the holding buffer when the last line of the file is reached and appends the line from the holding buffer after the last line of the file.

The x is an exchange command. By using this command, we can exchange the holding buffer with the current line in the pattern buffer.

For example:

$ sed -e '/Apple/h'  -e '/Cashew/x' shopping.txt

The output is as follows:

Product  Quantity  Unit_Price  Total_Cost
Apple      2         3          6
Orange     2        .8          1.6
Papaya     2       1.5          3
Chicken    3       5           15
Apple      2       3            6

In this example, the line with the Apple pattern is stored in the holding buffer. When the pattern with Cashew is found, that line will be exchanged by the holding buffer.

The sed script file contains a list of sed commands in a file. To inform the sed about our script file, we should use the –f option before the script file name. If the sed commands are not separated by a new line, then every command should be separated by a colon ":". We have to take care that there should not be any training white space after every command in the sed script file; otherwise, the sed will give an error. sed takes each line in the pattern buffer and then, it will process all commands on that line. After this line is processed, the next line will be loaded in the pattern buffer. For the continuation of any sed command, which cannot be fitted in one line, we need to add one backslash at the end of the line to inform about continuation.

For example:

$ cat shopping1.txt

The output is as follows:

Product   Quantity  Unit_Price
Apple      200       3
Orange     200        .8
Papaya     100       1.5
Chicken     65       5
Cashew      50      10
April, third week 
$ cat stock 

The output is as follows:

# This is my first sed script by : 
1i\
Stock status report
/Orange/a\
Fresh Oranges are not available in this season. \
Fresh Oranges will be available from next month
/Chicken/c\
**********************************************************\
We will not be stocking this item for next few weeks.\
**********************************************************
$d

Enter the next command as follows:

$ sed -f stock shopping1.txt

The output is as follows:

Stock status report
Product   Quantity  Unit_Price
Apple      200       3
Orange     200        .8
Fresh Oranges are not available in this season. 
Fresh Oranges will be available from next month
Papaya     100       1.5
**********************************************************
We will not be stocking this item for next few weeks.
**********************************************************
Cashew      50     10

In this script, the following processing has taken place: