sed: modify your text without opening your files
Unix clones such as Linux are rarely used to manage technical documentation. This is unusual when you consider the plethora of tools available on these platforms for manipulating text in various ways - tools that complement Python’s regular expression library for batch text manipulation.
Consider the dialogue between Mr. Jourdain and his Professor of Philosophy in Molière’s The Shopkeeper Turned Gentleman. We will work from the English rendering below, adapted from Charles Heron Wall’s prose translation:
MR. JOURDAIN:
[…] I should like to put into a note: “Fair Marchioness, your beautiful eyes make me die of love”; but I want it phrased in a gallant way, turned nicely.
[…]
PROFESSOR OF PHILOSOPHY:
They may be put, first of all, as you said: Fair Marchioness, your beautiful eyes make me die of love. Or: Of love die make me, fair Marchioness, your beautiful eyes. Or: Your beautiful eyes of love make me, fair Marchioness, die. Or: Die of love your beautiful eyes, fair Marchioness, make me. Or: Me make your beautiful eyes die, fair Marchioness, of love.
Let’s start by displaying the original sentence in a terminal:
echo "Fair Marchioness, your beautiful eyes make me die of love."
Now we need to swap the words in the sentence to create a new one. For a simple transposition, you might find it easier to use awk. awk doesn’t deal with lines, but with the fields of a record (of a line), delimited by spaces by default. In other words, awk treats text like a database. It can easily display the whole line or just one or more fields in any desired order. Fields are indicated in the form $n, where n indicates the position of the field in the line, starting from the left. So $1 indicates the first field, $2 the second, and so forth, up to $10 for the tenth word. $0 corresponds to the whole line.
So we’re going to give Mr. Jourdain’s declaration of love as input to a one-line awk program, using the pipeline redirection symbol (|).
awk '{print $9" "$10" "$8" "$6" "$7" "$1" "$2" "$3" "$4" "$5}'The output of the echo command is not displayed. What is displayed is the output of the awk program, of which the output of the echo command, Mr. Jourdain’s declaration of love, was the input.
However, the final output is not what was intended. The fields do not correspond exactly to words: the period is stuck to love, the comma to Marchioness, and the capitalization is wrong. The awk command therefore needs to be refined.
It’s simpler to turn to sed. sed selects sets of characters in lines, either quoted literally or via metacharacters in regular expressions. A well-known regular expression metacharacter is *, meaning “zero or more of the preceding item”: a* matches zero or more as, and .* - any character, repeated - matches an arbitrary run of characters.
sed also supports back references, which display the value corresponding to a previously found literal or regular expression at the desired location. Mr. Jourdain’s declaration of love contains ten words. sed, however, offers only nine back references, \1 through \9: it reads \10 as \1 followed by a literal 0, so a tenth word cannot be addressed directly. Fortunately, the last two words - of love - always travel together in the professor’s reorderings, so we can capture them as a single group and stay within the nine-reference limit.
sed "s/\(.*\) \(.*\), \(.*\) \(.*\) \(.*\) \(.*\) \(.*\) \(.*\) \(of .*\)/\9 \8 \6 \7, \1 \2, \3 \4 \5/"
We’ve run into the same problem: the regular expression .* doesn’t correspond to a word, but to a series of characters, including punctuation. We must then use the \<.*\> form, which corresponds to a word, like the ones Mr. Jourdain uses to make prose. We’re going to use escape characters (backslash \) so that the < and > signs are not interpreted literally under certain consoles, but as metacharacters with a special function. The final group, \(\<.*\> \<.*\>\), captures two words at once - the closing phrase of love:
export p="\(\<.*\>\) \(\<.*\>\), \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\>\) \(\<.*\> \<.*\>\)"
sed "s/$p/\9 \8 \6 \7, \1 \2, \3 \4 \5/"
We could also use the [[:alpha:]]* form, which is more readable but less concise:
export a="[[:alpha:]]"
export n="\($a*\) \($a*\), \($a*\) \($a*\) \($a*\) \($a*\) \($a*\) \($a*\) \($a* $a*\)"
sed "s/$n/\9 \8 \6 \7, \1 \2, \3 \4 \5/"
That’s better, but we’ve got a capitalization problem. So we’re going to use the judiciously placed \u and \l operators. First, we’ll export some variables to make the script more concise and readable:
export w="\(\<.*\>\)"
export m="$w $w, $w $w $w $w $w $w"
sed "s/$m \(\<.*\> \<.*\>\)/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/"
We can now easily redistribute the back references to get all the professor’s variations:
sed "s/$m \(\<.*\> \<.*\>\)/\u\3 \4 \5 \9 \6 \7, \l\1 \2, \8/"
sed "s/$m \(\<.*\> \<.*\>\)/\u\8 \9 \3 \4 \5, \l\1 \2, \6 \7/"
sed "s/$m \(\<.*\> \<.*\>\)/\u\7 \6 \3 \4 \5 \8, \l\1 \2, \9/"
Molière and GNU/Linux
Section titled “Molière and GNU/Linux”Let’s rewrite the dialogue between Mr. Jourdain and his Professor of Philosophy in geek style:
MR. JOURDAIN:
I’d like to show her on the standard output:
Fair Marchioness, your beautiful eyes make me die of love.But I wish it were put in a gallant way, that it were turned nicely.
PROFESSOR OF PHILOSOPHY:
They can be put first as you said:
Or:
Or:
export w="\(\<.*\>\)"
export m="$w $w, $w $w $w $w $w $w"
sed "s/$m \(\<.*\> \<.*\>\)/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/"
Or else:
sed "s/$m \(\<.*\> \<.*\>\)/\u\3 \4 \5 \9 \6 \7, \l\1 \2, \8/"
Or else:
sed "s/$m \(\<.*\> \<.*\>\)/\u\8 \9 \3 \4 \5, \l\1 \2, \6 \7/"
Or else:
sed "s/$m \(\<.*\> \<.*\>\)/\u\7 \6 \3 \4 \5 \8, \l\1 \2, \9/"
Every command above ran GNU sed. Here is that very binary - compiled from its C source to WebAssembly - running real GNU sed, live in this page. Edit the sentences and rephrase them yourself:
The ~250 KB binary is fetched once and cached. Each phrasing is a separate sed substitution over your lines - the same command the tutorial builds, running for real, client-side.
That same word-shuffling idea doesn’t have to stay in the shell. Here it is running as real Python, live in this page:
The runtime is fetched on first load, so the dot stays amber for a few seconds while genuine CPython boots in your browser. Any line that isn't ten words is flagged instead of rephrased - everything runs client-side, no server involved.
Much ado…
Section titled “Much ado…”Admittedly, much ado about next to nothing, you might say. But imagine a file containing 1000 sentences of the same structure.
This may look contrived, but structurally identical sentences are common in technical documentation, for the sake of stylistic consistency.
To carry out our tests on a sample, let’s place the two sentences above in a file:
echo "Dear doctor, these great misfortunes make you weep with bitterness." > variations.txt
echo "Vast ocean, the strong swell makes you pitch with drunkenness." >> variations.txt
Let’s place the various sed commands in a different script each. We reuse the pattern $p, this time ending with the generic two-word group \(\<.*\> \<.*\>\) so it captures with bitterness and with drunkenness just as well as of love:
export p="$m \(\<.*\> \<.*\>\)"
echo "s/$p/\u\9 \8 \6 \7, \l\1 \2, \3 \4 \5/" > moliere1.sed
echo "s/$p/\u\3 \4 \5 \9 \6 \7, \l\1 \2, \8/" > moliere2.sed
echo "s/$p/\u\8 \9 \3 \4 \5, \l\1 \2, \6 \7/" > moliere3.sed
echo "s/$p/\u\7 \6 \3 \4 \5 \8, \l\1 \2, \9/" > moliere4.sed
Now let’s loop through all the sed scripts on all the lines in the file:
for (( i=1; i<5; i++ )); do
while read s;
do echo "$s" |
sed -f moliere$i.sed ;
done < variations.txt
doneAnd there it is. In just a few moments, without ever opening a single file, we apply a series of complex operations to an indefinite number of sentences of the same structure. This is not feasible with a word processor or any other tool with a graphical interface or with binary files - which is one reason why text-based source formats are so well-suited for automated workflows.