# sed: modify your text without opening your files

> **Note: Specialist technique**
>
> sed is a Unix command-line tool for editing many text files in one pass. It remains practical for one-off, repository-wide changes to text sources; in a Git repository, review the result with `git diff` before committing it. See [Git: from file to content](https://docs.redaction-technique.org/en/tech-writing-process/git-from-file-to-content/).

## Objective

Unix clones such as Linux are rarely used to manage technical documentation. This is unusual when you consider the plethora of tools available on these platforms for manipulating text in various ways - tools that complement [Python's regular expression library](https://docs.redaction-technique.org/en/tutorials/python-regular-expressions/) for batch text manipulation.

The tutorial builds up the commands step by step on one sentence from Molière. To see them applied to a whole file at once, go to [Much ado...](#much-ado).

Consider the dialogue between Mr. Jourdain and his Professor of Philosophy in Molière's *The Shopkeeper Turned Gentleman*. We will work from the English rendering below, adapted from Charles Heron Wall's prose translation:

**MR. JOURDAIN:**

*[...] I should like to put into a note: "Fair Marchioness, your beautiful eyes make me die of love"; but I want it phrased in a gallant way, turned nicely.*

[...]

**PROFESSOR OF PHILOSOPHY:**

*They may be put, first of all, as you said: Fair Marchioness, your beautiful eyes make me die of love. Or: Of love die make me, fair Marchioness, your beautiful eyes. Or: Your beautiful eyes of love make me, fair Marchioness, die. Or: Die of love your beautiful eyes, fair Marchioness, make me. Or: Me make your beautiful eyes die, fair Marchioness, of love.*

## Swap words with awk

Let's start by displaying the original sentence in a terminal:

```bash
awk '1'
```

Now we need to swap the words in the sentence to create a new one. For a simple transposition, you might find it easier to use `awk`. `awk` doesn't deal with lines, but with the fields of a record (of a line), delimited by spaces by default. In other words, `awk` treats text like a database. It can easily display the whole line or just one or more fields in any desired order. Fields are indicated in the form `$n`, where `n` indicates the position of the field in the line, starting from the left. So `$1` indicates the first field, `$2` the second, and so forth, up to `$10` for the tenth word. `$0` corresponds to the whole line.

So we're going to give Mr. Jourdain's declaration of love as input to a one-line `awk` program, using the pipeline redirection symbol (`|`).

```bash
awk '{print $9" "$10" "$8" "$6" "$7" "$1" "$2" "$3" "$4" "$5}'
```

The output of the `echo` command is not displayed. What is displayed is the output of the `awk` program, of which the output of the `echo` command, Mr. Jourdain's declaration of love, was the input.

However, the final output is not what was intended. The fields do not correspond exactly to words: the period is stuck to `love`, the comma to `Marchioness`, and the capitalization is wrong. The `awk` command therefore needs to be refined.

## Reorder words with sed

It's simpler to turn to `sed`. `sed` selects sets of characters in lines, either quoted literally or via metacharacters in regular expressions. A well-known regular expression metacharacter is `*`, meaning "zero or more of the preceding item": `a*` matches zero or more `a`s, and `.*` - any character, repeated - matches an arbitrary run of characters.

> **Note: Shell glob vs. regex**
>
> Don't confuse the regex `*` with the shell glob `*`. On the command line, `ls *.rst` uses `*` on its own to mean "any sequence of characters." In a regular expression the bare `*` only repeats whatever precedes it; it's `.*` that behaves like the shell glob.

### Capture words with back references

`sed` also supports back references, which display the value corresponding to a previously found literal or regular expression at the desired location. Mr. Jourdain's declaration of love contains ten words. `sed`, however, offers only nine back references, `\1` through `\9`: it reads `\10` as `\1` followed by a literal `0`, so a tenth word cannot be addressed directly. Fortunately, the last two words - `of love` - always travel together in the professor's reorderings, so we can capture them as a single group and stay within the nine-reference limit.

```bash
sed "s/PATTERN/\9 \8 \6 \7, \1 \2, \3 \4 \5/"
```

### Match whole words

We've run into the same problem: the regular expression `.*` doesn't correspond to a word, but to a series of characters, including punctuation. We must then use the `\<.*\>` form, which corresponds to a word, like the ones Mr. Jourdain uses to make prose. We're going to use escape characters (backslash `\`) so that the `<` and `>` signs are not interpreted literally under certain consoles, but as metacharacters with a special function. The final group, `\(\<.*\> \<.*\>\)`, captures two words at once - the closing phrase `of love`:

> **Caution: GNU sed only**
>
> The word-boundary markers `\<` and `\>`, and the case operators `\u` / `\l` used further down, are GNU sed extensions. The default `sed` on BSD/macOS doesn't support them - install GNU sed (`brew install gnu-sed`) and run it as `gsed`.

```bash
export p="\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)"
sed "s/$p/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/"
```

We could also use the `[[:alpha:]]*` form, which is more readable but less concise:

> **Tip: Why the final period survives**
>
> Notice the trailing period is still there in the output, even though no replacement pattern ever mentions it. That's because sed's `s` command only rewrites the span it matches, and the capture groups stop at the word boundary (`\>`) just before the period. Anything a substitution doesn't match is passed through verbatim.

### Fix capitalization

That's better, but we've got a capitalization problem. So we're going to use the judiciously placed `\u` and `\l` operators. First, we'll export some variables to make the script more concise and readable:

```bash
sed "s/\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/"
```

We can now easily redistribute the back references to get all the professor's variations:

```bash
sed "s/\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)/\u\3 \4 \5 \9 \6 \7, \l\1 \2, \8/"
```

```bash
sed "s/\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)/\u\8 \9 \3 \4 \5, \l\1 \2, \6 \7/"
```

```bash
sed "s/\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)/\u\7 \6 \3 \4 \5 \8, \l\1 \2, \9/"
```

## Molière and GNU/Linux

Let's rewrite the dialogue between Mr. Jourdain and his Professor of Philosophy in geek style:

**MR. JOURDAIN:**

I'd like to show her on the standard output:

```console
$ Fair Marchioness, your beautiful eyes make me die of love.
```

> **Note**
>
> This is Mr. Jourdain's naive attempt - like the man who spoke prose without knowing it, he types the sentence straight at the prompt. Run as-is, the shell would try to execute `Fair` as a command and fail; the working version, with `echo` in front, comes just below.

But I wish it were put in a gallant way, that it were turned nicely.

**PROFESSOR OF PHILOSOPHY:**

They can be put first as you said:

```bash
echo "Fair Marchioness, your beautiful eyes make me die of love."
```

Or:

```bash
export declaration="Fair Marchioness, your beautiful eyes make me die of love."
echo $declaration
```

Or:

```bash
export p="\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)"
sed "s/$p/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/"
```

Or else:

```bash
export p="\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)"
sed "s/$p/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/"
```

Or else:

```bash
export p="\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)"
sed "s/$p/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/"
```

Or else:

```bash
export p="\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)"
sed "s/$p/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/"
```

Every command above ran GNU sed. Here is that very binary - compiled from its C source to WebAssembly - running real GNU sed, live in this page. Edit the sentences and rephrase them yourself:

That same word-shuffling idea doesn't have to stay in the shell. Here it is running as real Python, live in this page:

## Much ado...

Admittedly, much ado about next to nothing, you might say. But imagine a file containing 1000 sentences of the same structure.

This may look contrived, but structurally identical sentences are common in technical documentation, for the sake of stylistic consistency.

To carry out our tests on a sample, let's place the two sentences above in a file:

Let's place the various `sed` commands in a different script each. We reuse the pattern `$p`, this time ending with the generic two-word group `\(\<.*\> \<.*\>\)` so it captures `with bitterness` and `with drunkenness` just as well as `of love`:

Now let's loop through all the `sed` scripts on all the lines in the file:

```bash
for (( i=1; i<5; i++ )); do
   while read s;
    do echo "$s" |
     sed -f moliere$i.sed ;
    done < variations.txt
   done
```

And there it is. In just a few moments, without ever opening a single file, we apply a series of complex operations to an indefinite number of sentences of the same structure. This is not feasible with a word processor or any other tool with a graphical interface or with binary files - which is one reason why [text-based source formats](https://docs.redaction-technique.org/en/tech-writing-process/source-format/) are so well-suited for automated workflows.

## Related articles

- [Create different documents from the same sources using Jinja](https://docs.redaction-technique.org/en/tutorials/conditional-text-jinja/)
- [Automatically insert data into a reStructuredText file](https://docs.redaction-technique.org/en/tutorials/auto-insert-data-restructuredtext/)
- [The Raspberry Pi 3 as a documentation platform](https://docs.redaction-technique.org/en/tutorials/raspberry-pi-documentation-platform/)

---

Source: https://docs.redaction-technique.org/en/tutorials/sed-text-editing/