docs(parameter expansion): prefix/suffix removal uses pattern-lists

[mgkurtz]

Abstract

After updating a shell script, it ceased to work for people with ksh as system shell (pdfjam/pdfjam#99). The incompatibility was caused by ksh interpreting | as OR operator in a construct such as ${parameter%word|separated|by|bars}, while other shells interpret | literally. This behaviour persists even after set -o posix.

Example

Here is some shell session that removes the smallest match for |* from x|y resulting in x, which works for at least bash, dash and zsh:

$ a="x|y"
$ echo ${a%|*}'
x

In ksh, the |* matches the empty string and removes nothing:

$ a="x|y"
$ echo ${a%|*}'
x|y

$ set -o posix
$ a="x|y"
$ echo ${a%|*}'
x|y

Resolving as script author

One can escape the special meaning of | using a backslash, which works in all tested shells including ksh:

$ a="x|y"
$ echo ${a%\|*}'
x

The POSIX requirements

The parameter expansion section of the POSIX Shell Command Language requires

The following four varieties of parameter expansion provide for character substring processing. In each case, pattern matching notation (see 2.14 Pattern Matching Notation), rather than regular expression notation, shall be used to evaluate the patterns. […]

${parameter%[word]}
Remove Smallest Suffix Pattern. The word shall be expanded to produce a pattern. The parameter expansion shall then result in parameter, with the smallest portion of the suffix matched by the pattern deleted. […]

Wish

Either

• a) change the behaviour to what the other shells do, or
• b) keep and document the behaviour and change it to the POSIX compliant way when asking for POSIX standard mode.

[jghub]

AFAICS, POSIX does only define behaviour of ${parameter%pattern} type constructs for standard glob patterns ("words"). OR/alternation is not part of standard patterns (something like ls pattern1|pattern2 would not work that way, obviously...). so it seems every shell can interpret ${parameter%pattern1|pattern2*} as it prefers and it is at least not a bug and not a question of posix-compatibility.

ksh is treating pattern1|pattern2 as a "pattern-list" (as per its manpage) and such lists can be used in extended globs like @(pattern-list). and | in such lists is the OR operator as you observed. it seems, for convenience (depending on POV, of course), ksh interprets ${parameter%pattern1|pattern2*} just in that way (as equivalent to @(pattern-list)): a OR pattern-list:

set -v
s='ab|cd|ef'
echo "    $s"
    ab|cd|ef
echo

echo "    ${s%|*}"
    ab|cd|ef

echo "    ${s%|e*}"
    ab|cd|ef
echo

echo "    ${s%\|e*}"
    ab|cd
echo

echo "    ${s%d|e*}"
    ab|cd|
echo "    ${s%@(d|e)*}"
    ab|cd|

echo "    ${s%d\|e*}"
    ab|c
set +v

your case thus uses a pattern-list equal to @(|) (empty OR empty) and that matches the rightmost occurrence in ${s%pattern-list*} -- which is end of string. so no change.

I did not find explicit mention of this behaviour in the manpage (nor in the korn/bolsky book) but I strongly presume this is as it is intended to be in ksh (if it is, one probably should consider to augment the manpage??). the incompatibility, then, is just between bash (etc.) and ksh.

so regarding your requests my personal opinion would be:

• changing behaviour to follow the other shells: I would not think this is desirable
• keep and document current behaviour: yes (so manpage could/should explain this).
• change behaviour when set -o posix is set: since it seems to have nothing to do with POSIX: no

[mgkurtz]

Thanks @jghub for your extensive answer. I double checked in the POSIX standard, and agree that ksh’s described behaviour is compatible with POSIX.
Section 2.14 of POSIX Shell & Utilities specifies (with my highlighting):

An ordinary character is a pattern that shall match itself. In a pattern, or part of one, where a shell-quoting <backslash> can be used, an ordinary character can be any character in the supported character set except for NUL, those special shell characters in 2.2 Quoting that require quoting, and the three special pattern characters described below. […] If any character (ordinary, shell special, or pattern special) is quoted, or escaped with a <backslash>, that pattern shall match the character itself.

As we can use a backslash in the word part of ${parameter%word}, the above does only specify what the pattern \|* matches and not what |* matches. According to that, ksh indeed has the liberty, to interpret |, &, ( and ) as it likes in the pattern word.

Other shells do not use this liberty, so a=xxx; ${a%%*(x)} yields an empty string with ksh and xxx with other shells, I tested. This is fine and a nice ksh feature. So, in ${parameter%word}, word can use pattern-lists and gets interpreted as @(word). If such a sentence were part of the man page, this would have helped me a lot in debugging.