[Chapter 2] Understanding Basic Operations
Author: Dale Dougherty & Arnold Robbins
Date: 1998-08-03T20:49:23Z

#

| Chapter 2 |

# 2. Understanding Basic Operations

Contents:
Awk, by Sed and Grep, out of Ed
Command-Line Syntax
Using sed
Using awk
Using sed and awk Together

If you are starting out to learn sed and awk, you can benefit from looking at how much they have in common.

- They are invoked using similar syntax.

- They are both stream-oriented, reading input from text files one line at a time and directing the result to standard output.

- They use regular expressions for pattern matching.

- They allow the user to specify instructions in a script.

One reason they have so much in common is that their origins can be found in the same line editor, `ed`. In this chapter, we begin by taking a brief look at `ed` and show how sed and awk were logical steps towards the creation of a programmable editor.

Where sed and awk differ is in the kind of instructions that control the work they do. Make no mistake - this is a major difference, and it affects the kinds of tasks that can best be performed with these programs.

This chapter looks at the command-line syntax of sed and awk and the basic structure of scripts. It also offers a tutorial, using a mailing list, that will give you a taste of script writing. It is valuable to see sed and awk scripts side-by-side before you concentrate on either one of them.

## 2.1 Awk, by Sed and Grep, out of Ed

You can trace the lineage of awk to sed and `grep`, and through those two programs to `ed`, the original UNIX line editor.

Have you ever used a line editor? If so, it will be much easier for you to understand the line orientation of sed and awk. If you have used `vi`, a full-screen editor, then you are familiar with a number of commands that are derived from its underlying line editor, `ex` (which in turn is a superset of the features in `ed`).

Let's look at some basic operations using the line editor `ed`. Don't worry - this is an exercise intended to help you learn sed and awk, not an attempt to convince you of the wonders of line editors. The `ed` commands that are shown in this exercise are identical to the `sed` commands you'll learn later on. Feel free to experiment with `ed` on your own to get a sense of how it works. (If you're already familiar with `ed`, feel free to skip to the next section.)

To use a line editor, you work on one line at a time. It is important to know what line you are positioned at in the file. When you open a file using `ed`, it displays the number of characters in the file and positions you at the last line.

>     $ ed test
>     339

There is no prompt. If you enter a command that `ed` does not understand, it prints a question mark as an error message. You can enter the print command, `p`, to display the current line.

>     p
>     label on the first box.

By default, a command affects only the current line. To make an edit, you move to the line that you want to edit and then apply the command. To move to a line, you specify its address. An address might consist of a line number, a symbol indicating a specific position in the file, or a regular expression. You can go to the first line by entering the line number 1. Then you can enter the delete command to remove that line.

>     1
>     You might think of a regular expression
>     d

Entering "1" makes the first line the current line, displaying it on the screen. The delete command in `ed` is `d` and here it deletes the current line. Rather than moving to a line and then editing it, you can prefix an editing command with an address that indicates which line or range of lines is the object of the command. If you enter "1d", the first line would be deleted.

You can also specify a regular expression as an address. To delete a line containing the word "regular," you could issue this command:

>     /regular/d

where slashes delimit the regular expression and "regular" is the string you want to match. This command deletes the first line containing "regular" and makes the line following it the current line.

> NOTE: Make sure you understand that the delete command deletes the whole line. It does not just delete the word "regular" on the line.

To delete all the lines that contain the regular expression, you'd prefix the command with the letter `g` for global.

>     g/regular/d

The global command makes all lines that match the regular expression the object of the specified command.

Deleting text can take you only so far. Substituting text (replacing one bit of text with another) is much more interesting. The substitution command, `s`, in `ed` is:

>     [address]s/pattern/replacement/flag

pattern is a regular expression that matches a string in the current line to be replaced by replacement. For example, the following command replaces the first occurrence of "regular" with "complex" on the current line.

>     s/regular/complex/

No address is specified, so it affects only the first occurrence on the current line. It is an error if "regular" is not found on the current line. To look for multiple occurrences on the same line, you must specify `g` as a flag:

>     s/regular/complex/g

This command changes all occurrences on the current line. An address must be specified to direct this command to act upon more than the current line. The following substitution command specifies an address:

>     /regular/s/regular/complex/g

This command affects the first line that matches the address in the file. Remember, the first "regular" is an address and the second is a pattern to match for the substitution command. To make it apply to all lines, use the global command, putting `g` before the address.

>     g/regular/s/regular/complex/g

Now the substitution is made everywhere - all occurrences on all lines.

> NOTE: Note the different meanings of "g." The "g" at the beginning is the global command that means make the changes on all lines matched by the address. The "g" at the end is a flag that means change each occurrence on a line, not just the first.

The address and the pattern need not be the same.

>     g/regular expression/s/regular/complex/g

On any line that contains the string "regular expression," replace "regular" with "complex." If the address and the pattern are the same, you can tell `ed` by specifying two consecutive delimiters (//).

>     g/regular/s//complex/g

In this example, "regular" is specified as the address and the pattern to be matched for substitution is the same. If it seems that we've covered these commands quickly and that there is a lot to absorb, don't worry. We will be covering these commands again later on.

The familiar UNIX utility `grep` is derived from the following global command in `ed`:

>     g/re/p

which stands for "global regular expression print." Grep is a line-editing command that has been extracted from `ed` and made available as an external program. It is hard-wired to perform one editing command. It takes the regular expression as an argument on the command line and uses it as the address of lines to print. Here's an example, looking for lines matching "box":

>     $ grep 'box' test
>     You are given a series of boxes, the first one labeled "A",
>     label on the first box.

It prints all lines matching the regular expression.

One more interesting feature of `ed` is the ability to script your edits, placing them in a separate file and directing them as input to the line editor. For instance, if a series of commands were put in a file named ed-script, the following command executes the script:

>     ed test < ed-script

This feature makes `ed` a programmable editor - that is, you can script any action that you might perform manually.

Sed was created as a special-purpose editor that was meant to execute scripts exclusively; unlike `ed`, it cannot be used interactively. Sed differs from `ed` primarily in that it is stream-oriented. By default, all of the input to sed passes through and goes to standard output. The input file itself is not changed. If you actually do want to alter the input file, you typically use the shell mechanism for output redirection, and when you are satisfied with the edits you've made, replace the original file with the modified version.

`ed` is not stream-oriented and changes are made to the file itself. An `ed` script must contain commands to save the file and quit the editor. It produces no output to the screen, except what may be generated by a specific command.

The stream orientation of sed has a major impact on how addressing is applied. In `ed`, a command without an address affects only the current line. Sed goes through the file, a line at a time, such that each line becomes the current line, and the commands are applied to it. The result is that sed applies a command without an address to every line in the file.

Look at the following substitution command:

>     s/regular/complex/

If you entered this command interactively in `ed`, you'd substitute "complex" for the first occurrence of "regular" on the current line. In an `ed` script, if this was the first command in the script, it would be applied only to the last line of the file (`ed`'s default current line). However, in a sed script, the same command applies to all lines. That is, sed commands are implicitly global. In sed, the previous example has the same result as the following global command in `ed`:

>     g/regular/s//complex/

> NOTE: Understanding the difference between current-line addressing in `ed` and global-line addressing in sed is very important. In `ed` you use addressing to expand the number of lines that are the object of a command; in sed, you use addressing to restrict the number of lines affected by a command.

Sed also was designed with a number of additional commands that support script writing. We will look at many of these commands in Chapter 6, Advanced sed Commands.

Awk was developed as a programmable editor that, like sed, is stream-oriented and interprets a script of editing commands. Where awk departs from sed is in discarding the line-editor command set. It offers in its place a programming language modeled on the C language. The `print` statement replaces the `p` command, for example. The concept of addressing is carried over, such that:

>     /regular/ { print }

prints those lines matching "regular". The braces ({}) surround a series of one or more statements that are applied to the same address.

The advantage of using a programming language in scripts is that it offers many more ways to control what the programmable editor can do. Awk offers expressions, conditional statements, loops, and other programming constructs.

One of the most distinctive features of awk is that it parses, or breaks up, each input line and makes individual words available for processing with a script. (An editor such as `vi` also recognizes words, allowing you to move word by word, or make a word the object of an action, but these features can only be used interactively.)

Although awk was designed as a programmable editor, users found that awk scripts could do a wide range of other tasks as well. The authors of awk never imagined it would be used to write large programs. But, recognizing that awk was being used in this way, the authors revised the language, creating `nawk` to offer more support for writing larger programs and tackling general-purpose programming problems. This new version, with minor improvements, is now codified by the POSIX standard.

1.4 Four Hurdles to Mastering sed and awk |  | 2.2 Command-Line Syntax