Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001342 [1003.1(2016/18)/Issue7+TC2] Shell and Utilities Objection Error 2020-05-05 08:50 2020-05-05 17:43
Reporter kre View Status public  
Assigned To
Priority normal Resolution Open  
Status New  
Name Robert Elz
Organization
User Reference
Section XCU 2.2.3
Page Number 2346
Line Number 74721
Interp Status ---
Final Accepted Text
Summary 0001342: Aliases in command substitutions are handled differently when the command subst is quoted vs not quoted
Description A 1997 interpretation resulted in the addition of the words:

   not including the alias substitutions in Section 2.3.1,

to the rules for processing a command substitution embedded within double
quotes (XCU 2.2.3).

No similar change was made to section 2.3 (bullet point 5) which handles
the general rules for token recognition when a command substitution is
detected, unquoted.

This means that different rules apply to recognising a command substitution
that happens in a double-quoted string than one that appears unquoted.

This is neither rational, nor what shells implement.

Apart from the unfortunate difference made to the two cases, the 1997
interpretation was almost certainly correct for its time - the relevant shells
of the mid 1990's did not expand aliases while searching for the terminating ')'
of a $() command substitution (backquoted command substitutions are much
simpler and have none of these issues). Shells of the time didn't parse
the command substitution at all while searching for the ')' - they relied
upon '(' and ')' counting.

In the intervening 20-something years that technique has been shown to be
fatally flawed, not only do (some - unlikely but possible) aliases potentially
break this, but case statements (not using the optional '(' preceding the
pattern) and here documents (which can contain almost anything - but can
only be recognised as being a here document by parsing the accompanying
command) break this simple scheme - shells must fully parse a $() command
substitution to have any hope of correctly finding the terminating ')' in
all cases.

Note that whether the shell retains the results of this parse, or simply
copies the command substitution string literally, saving it to be parsed
again later, is immaterial to this issue - the correct ')' must be located
to properly terminate the command substitution, whatever is done with the
text of the command substitution, or the results of parsing it, during that
process.
Desired Action In line 74271, on page 2346 of XCU 2.2.3 (2016 edition) delete the words:

    not including the alias substitutions in Section 2.3.1,

This is needed regardless of any other changes that might be made, as even
if it is (was) the correct behaviour, it was applied in the wrong place.

That is all that is needed to handle this issue.

However, as an option, in XCU 2.3, bullet point 5, on page 2348, after
the sentence (lines 74774-6):

    While processing the characters, if instances of expansions or quoting are
    found nested within the substitution, the shell shall recursively process
    them in the manner specified for the construct that is found.

add a new sentence:

    It is unspecified whether aliases [xref XCU 2.3.1] shall be expanded
    while performing this process.

It needs to be unspecified, rather than "shall not process" as most shells
do process aliases at this point, as they are when the command substitution
is eventually parsed and executed, if different rules apply strange hard
to fathom errors can occur.

However, not all shells do, so making it unspecified might be the wise choice
for now - even if such shells tend to be broken in this regard. Since alias
processing when the command substitution was in a double quoted string was
(or is) prohibited by the standard, suddenly making it mandated when there are,
or might be, some shells which actually implemented this prohibition would
also seem harsh.

If this option is adopted, two further changes should be made.

In the application notes (wherever they are for this) there should be an
admonition

     Applications shall not include aliases which contain unbalanced
     syntax components in any $() command substitution.

What that means can be expanded if deemed appropriate. The idea is that
trivial aliases "alias l=ls" (etc) are harmless, and not a problem, it is
only ones like "alias switch=case", "alias subshell='('", or
"alias forever='while true do;'" that cause problems. The "unbalanced"
is because an alias like "alias set13='(exit 13)'" is not a problem,
despite having parentheses in its value.

And second, in the Future Directions (wherever that is for sh) it should
say

     A future version of this standard may require alias expansion
     while scanning for the terminating ')' in a $() command substitution.

Last, some kind of explanation of all that happened here should be added
in the rationale (somewhere in XRAT I assume).


An alternative option open to the group who decides these things would be
to simply delete aliases from the standard completely. Since I doubt this
one will happen, I won't bother providing the changes needed to implement it.
Tags No tags attached.
Attached Files

- Relationships
related to 0000953Appliedajosey 1003.1(2013)/Issue7+TC1 Alias expansion is under-specified 

-  Notes
(0004856)
geoffclare (manager)
2020-05-05 10:57

We also need to do something about the statement for $() command substitution that "Any valid shell script can be used for command". This is not true because a shell script which defines an alias which changes the syntax, and then uses it, might not be parsed correctly without the alias command being executed.
For example a script containing:
alias subshell='('
subshell echo foo )
subshell echo bar )
works fine (note you need POSIXLY_CORRECT set for bash), but:
echo $(
alias subshell='('
subshell echo foo )
subshell echo bar )
)
does not.
(0004859)
kre (reporter)
2020-05-05 15:36

Re Note: 0004856

I suspect that's already handled by the (new, I forget which bug it is)
wording that specifies when alias commands take effect.

This is just the same issue as prevents

if [ $SHELL = myshell ]; then
    alias xx=mycommand
    xx foo bar
fi

from working. Or from defining, and then using, an alias in a function.

Aliases are evil - let's delete them!!!
(0004860)
geoffclare (manager)
2020-05-05 15:52

Re Note: 0004859

Bug 0000953 does indeed change that text, but the new text still has the same problem. It now says:
It is unspecified whether command is parsed and executed as a program (as for a shell script) or is parsed as a single compound_list that is executed after the entire command has been parsed. With the $(command) form any valid program can be used for command, except a program consisting solely of redirections which produces unspecified results.
(0004863)
kre (reporter)
2020-05-05 17:43

OK, we can add text that makes it clear (just in case it wasn't
already obvious) that the entire command in a command substitution is
parsed before any of it is executed. And given that, alias commands
in the command substitution cannot work - the command substitution is
a subshell environment, so the parent shell cannot be affected, and
any aliases defined there get defined too late to be used within the
command substitution.

Alias in a sub-shell block '(' code ')' have the same uselessness.

Aliases in general are largely useless, let's delete them!!!

- Issue History
Date Modified Username Field Change
2020-05-05 08:50 kre New Issue
2020-05-05 08:50 kre Name => Robert Elz
2020-05-05 08:50 kre Section => XCU 2.2.3
2020-05-05 08:50 kre Page Number => 2346
2020-05-05 08:50 kre Line Number => 74721
2020-05-05 10:57 geoffclare Note Added: 0004856
2020-05-05 15:36 kre Note Added: 0004859
2020-05-05 15:52 geoffclare Note Added: 0004860
2020-05-05 15:53 geoffclare Relationship added related to 0000953
2020-05-05 17:43 kre Note Added: 0004863


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker