|Anonymous | Login||2021-02-28 06:53 UTC|
|Main | My View | View Issues | Change Log | Docs|
|Viewing Issue Simple Details|
|ID||Category||Severity||Type||Date Submitted||Last Update|
|0001342||[1003.1(2016/18)/Issue7+TC2] Shell and Utilities||Objection||Error||2020-05-05 08:50||2021-01-29 11:21|
|Priority||normal||Resolution||Accepted As Marked|
|Final Accepted Text||Note: 0005123|
|Summary||0001342: Aliases in command substitutions are handled differently when the command subst is quoted vs not quoted|
A 1997 interpretation resulted in the addition of the words:
not including the alias substitutions in Section 2.3.1,
to the rules for processing a command substitution embedded within double
quotes (XCU 2.2.3).
No similar change was made to section 2.3 (bullet point 5) which handles
the general rules for token recognition when a command substitution is
This means that different rules apply to recognising a command substitution
that happens in a double-quoted string than one that appears unquoted.
This is neither rational, nor what shells implement.
Apart from the unfortunate difference made to the two cases, the 1997
interpretation was almost certainly correct for its time - the relevant shells
of the mid 1990's did not expand aliases while searching for the terminating ')'
of a $() command substitution (backquoted command substitutions are much
simpler and have none of these issues). Shells of the time didn't parse
the command substitution at all while searching for the ')' - they relied
upon '(' and ')' counting.
In the intervening 20-something years that technique has been shown to be
fatally flawed, not only do (some - unlikely but possible) aliases potentially
break this, but case statements (not using the optional '(' preceding the
pattern) and here documents (which can contain almost anything - but can
only be recognised as being a here document by parsing the accompanying
command) break this simple scheme - shells must fully parse a $() command
substitution to have any hope of correctly finding the terminating ')' in
Note that whether the shell retains the results of this parse, or simply
copies the command substitution string literally, saving it to be parsed
again later, is immaterial to this issue - the correct ')' must be located
to properly terminate the command substitution, whatever is done with the
text of the command substitution, or the results of parsing it, during that
In line 74271, on page 2346 of XCU 2.2.3 (2016 edition) delete the words:
not including the alias substitutions in Section 2.3.1,
This is needed regardless of any other changes that might be made, as even
if it is (was) the correct behaviour, it was applied in the wrong place.
That is all that is needed to handle this issue.
However, as an option, in XCU 2.3, bullet point 5, on page 2348, after
the sentence (lines 74774-6):
While processing the characters, if instances of expansions or quoting are
found nested within the substitution, the shell shall recursively process
them in the manner specified for the construct that is found.
add a new sentence:
It is unspecified whether aliases [xref XCU 2.3.1] shall be expanded
while performing this process.
It needs to be unspecified, rather than "shall not process" as most shells
do process aliases at this point, as they are when the command substitution
is eventually parsed and executed, if different rules apply strange hard
to fathom errors can occur.
However, not all shells do, so making it unspecified might be the wise choice
for now - even if such shells tend to be broken in this regard. Since alias
processing when the command substitution was in a double quoted string was
(or is) prohibited by the standard, suddenly making it mandated when there are,
or might be, some shells which actually implemented this prohibition would
also seem harsh.
If this option is adopted, two further changes should be made.
In the application notes (wherever they are for this) there should be an
Applications shall not include aliases which contain unbalanced
syntax components in any $() command substitution.
What that means can be expanded if deemed appropriate. The idea is that
trivial aliases "alias l=ls" (etc) are harmless, and not a problem, it is
only ones like "alias switch=case", "alias subshell='('", or
"alias forever='while true do;'" that cause problems. The "unbalanced"
is because an alias like "alias set13='(exit 13)'" is not a problem,
despite having parentheses in its value.
And second, in the Future Directions (wherever that is for sh) it should
A future version of this standard may require alias expansion
while scanning for the terminating ')' in a $() command substitution.
Last, some kind of explanation of all that happened here should be added
in the rationale (somewhere in XRAT I assume).
An alternative option open to the group who decides these things would be
to simply delete aliases from the standard completely. Since I doubt this
one will happen, I won't bother providing the changes needed to implement it.
We also need to do something about the statement for $() command substitution that "Any valid shell script can be used for command". This is not true because a shell script which defines an alias which changes the syntax, and then uses it, might not be parsed correctly without the alias command being executed.
For example a script containing:
alias subshell='(' subshell echo foo ) subshell echo bar )works fine (note you need POSIXLY_CORRECT set for bash), but:
echo $( alias subshell='(' subshell echo foo ) subshell echo bar ) )does not.
Re Note: 0004856
I suspect that's already handled by the (new, I forget which bug it is)
wording that specifies when alias commands take effect.
This is just the same issue as prevents
if [ $SHELL = myshell ]; then
xx foo bar
from working. Or from defining, and then using, an alias in a function.
Aliases are evil - let's delete them!!!
Re Note: 0004859
Bug 0000953 does indeed change that text, but the new text still has the same problem. It now says:
It is unspecified whether command is parsed and executed as a program (as for a shell script) or is parsed as a single compound_list that is executed after the entire command has been parsed. With the $(command) form any valid program can be used for command, except a program consisting solely of redirections which produces unspecified results.
OK, we can add text that makes it clear (just in case it wasn't
already obvious) that the entire command in a command substitution is
parsed before any of it is executed. And given that, alias commands
in the command substitution cannot work - the command substitution is
a subshell environment, so the parent shell cannot be affected, and
any aliases defined there get defined too late to be used within the
Alias in a sub-shell block '(' code ')' have the same uselessness.
Aliases in general are largely useless, let's delete them!!!
The standard states that alias substitutions are applied when finding the terminating ')' of a command substitution when the command substitution is not within double quotes, but they are not applied when the command substitution is within double quotes, and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor.
Historically some shells used simple parenthesis counting to find the terminating ')' and therefore did not account for aliases. However, such shells never conformed to POSIX, which has always required recursive parsing (see XCU 2.3 item 5). It is likely that observing the behavior of these non-conforming shells was what led to the erroneous addition of "not including the alias substitutions in Section 2.3.1" to XCU 2.2.3 after PASC interpretation 1003.2 #157.
Notes to the Editor (not part of this interpretation):
On page 2346 line 74721 section 2.2.3, delete:
not including the alias substitutions in Section 2.3.1,
On page 2357 line 75182 section 2.6.3, change:
Command substitution allows the output of a command to be substituted in place of the command name itself. Command substitution shall occur when the command is enclosed as followsto:
Command substitution allows the output of one or more commands to be substituted in place of the commands themselves. Command substitution shall occur when command(s) are enclosed as follows
On page 2357 line 75190 section 2.6.3, change:
with the standard output of the commandto:
with the standard output of the command(s)
On page 2358 lines 75202-75204 section 2.6.3, after applying bug 953, change:
It is unspecified whether command is parsed and executed as a program (as for a shell script) or is parsed as a single compound_list that is executed after the entire command has been parsed. With the $(command) form any valid program can be used for command, except a program consisting solely of redirections which produces unspecified results.to:
It is unspecified whether the commands string is parsed and executed incrementally as a program (as for a shell script), or is parsed as a single compound_list that is executed after the string has been completely parsed. In addition, it is unspecified whether the terminating ')' of the $(commands) form can result from alias substitution. With the $(commands) form any syntactically correct program can be used for commands, except that:
Throughout section 2.6.3 change any remaining occurrences of "command" to "commands" or "the commands string" as appropriate.
On page 3730 line 127967 section C.2.6.3, change:
While the newer "$()" form can process any kind of valid embedded script, the backquoted form ...to:
While the newer "$()" form can process any kind of valid embedded script (with a few caveats; see below), the backquoted form ...
On page 3730 line 127985 section C.2.6.3, change:
If command is of the form ...to:
If the commands string is of the form ...
On page 3730 line 127990 section C.2.6.3, after:
... it is unspecified to have a script within "$()" that has only redirections.add the following new paragraph:
In POSIX.2-1992 the $(commands) form of command substitution only had unspecified behavior for a commands string consisting solely of redirections. However, two additional unspecified cases have since been added with relation to aliases:
|Interpretation proposed: 2 December 2020|
|Interpretation approved: 4 Jan 2021|
|2020-05-05 08:50||kre||New Issue|
|2020-05-05 08:50||kre||Name||=> Robert Elz|
|2020-05-05 08:50||kre||Section||=> XCU 2.2.3|
|2020-05-05 08:50||kre||Page Number||=> 2346|
|2020-05-05 08:50||kre||Line Number||=> 74721|
|2020-05-05 10:57||geoffclare||Note Added: 0004856|
|2020-05-05 15:36||kre||Note Added: 0004859|
|2020-05-05 15:52||geoffclare||Note Added: 0004860|
|2020-05-05 15:53||geoffclare||Relationship added||related to 0000953|
|2020-05-05 17:43||kre||Note Added: 0004863|
|2020-11-30 17:04||geoffclare||Note Added: 0005123|
|2020-11-30 17:07||geoffclare||Interp Status||=> Pending|
|2020-11-30 17:07||geoffclare||Final Accepted Text||=> Note: 0005123|
|2020-11-30 17:07||geoffclare||Status||New => Interpretation Required|
|2020-11-30 17:07||geoffclare||Resolution||Open => Accepted As Marked|
|2020-11-30 17:07||geoffclare||Tag Attached: tc3-2008|
|2020-12-02 17:04||agadmin||Interp Status||Pending => Proposed|
|2020-12-02 17:04||agadmin||Note Added: 0005137|
|2021-01-04 12:14||agadmin||Interp Status||Proposed => Approved|
|2021-01-04 12:14||agadmin||Note Added: 0005190|
|2021-01-29 11:21||geoffclare||Status||Interpretation Required => Applied|
|Mantis 1.1.6[^] Copyright © 2000 - 2008 Mantis Group|