Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001123 [1003.1(2013)/Issue7+TC1] Shell and Utilities Objection Clarification Requested 2017-03-04 07:56 2019-10-31 11:23
Reporter kre View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Robert Elz
Organization
User Reference
Section 2.6
Page Number 2353
Line Number 75006-75007
Interp Status ---
Final Accepted Text Note: 0004082
Summary 0001123: Problematic specification of execution environment for word expansions
Description Section 2.6 says ...

    The expansions described in this section shall occur in the same shell
    environment as that in which the command is executed.

which makes no sense, first because the expansions are not always being
performed on behalf of a command about to be executed (consider the case
of expanding one of the PSn prompts) and second, because at the time the
expansions are done, we do not yet know what command is to be executed,
and hence, we do not know which execution environment is to be used.

Consider:
                case "${something}" in
                a) command=exec;;
                b) command=echo;;
                *) command=;;
                esac

                ${command} ls "${X=3}"

which environment is to be used when expanding ${command} ?

if we had something=a then because exec is a special builtin, the current
(command line processing) environment is to be used. In the other cases,
a new execution environment is created. But until we have expanded ${command}
we don't know which.

In this particular example, it probably makes no real difference, but it is
easy to come up with others (not involving exec) where it does matter.
Desired Action Right now, no idea... Completely re-do section 2.6 to make it rational
would be a good first step.
Tags tc3-2008
Attached Files

- Relationships
related to 0001193New 1003.1(2016)/Issue7+TC2 Brace expansion and {var}>file redirects in the shell 
has duplicate 0001297Closed 1003.1(2016)/Issue7+TC2 Potentially misleading wording wrt removal of empty fields 

-  Notes
(0004070)
kre (reporter)
2018-08-05 21:45
edited on: 2018-08-06 15:29

Since this report must be reaching the head of the queue to be discussed soon,
I thought I'd do a little experiment to see what the standard should actually
say for this ...

I ran the following little script (which was /tmp/T)

      V=${C:=XYZZY} $A ${B=-l} ${F:=foo.c} > ${R:=/tmp/junk} 2>/dev/null

      printf 'A<%s> B<%s> C<%s> F<%s> R<%s> V<%s>\n' \
              "$A" "${B}" "${C}" "${F}" "${R}" "${V}"

in a loop like:

      for S in 'bash -o posix' bash <....>
      do
           printf '%s: ' "${S}"
           A=: $S /tmp/T
           printf '%s: ' "${S}"
           A=ls $S /tmp/T
     done

(using every shell I have available that I remembered ...)
and obtained the following results:

bash -o posix: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
bash -o posix: A<ls> B<-l> C<XYZZY> F<foo.c> R<> V<>
bash: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
bash: A<ls> B<-l> C<XYZZY> F<foo.c> R<> V<>
bosh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
bosh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
dash: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
dash: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
fbsh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
fbsh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
ksh93: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
ksh93: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
ksh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
ksh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
mksh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
mksh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
pdksh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
pdksh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
sh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
sh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
yash: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
yash: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
zsh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<>
zsh: A<ls> B<-l> C<> F<foo.c> R<> V<>


Most of those shells are obvious to all of you I think, "sh" is the
NetBSD shell, "ksh" is /bin/ksh on NetBSD which is a very very close
relation of pdksh (and I'd be surprised if the results were different,
and they aren't...) fbsh is a slightly old now version of the FreeBSD
shell.

I included the V= output just for interest, what happens to V is
unrelated to the current issue, and is best ignored here.

What this shows is that *all* shells (or all I tested, I have no access
to either ksh88 or some old SysVR4 /bin/sh) expand the command word and
its args in the current shell environment, not that of the command to be
executed, which I think is unavoidable (as mentioned in the original
bug report). That is the values of B and F.

Further all shells with the exception of zsh (which was not in its posix
emulation mode, as I am not a zsh user, and have no idea how to make that
happen correctly, it has no "-o posix") expand the variable assignment in
the current shell execution environment, only zsh does it in the envoronment
of the command to be executed. This is the value of C.

And lastly, all shells but bash (in posix mode, or not) and zsh evaluate
expansions in redirects in the current shell environment, not in the
environment of the command to be executed (but as you will observe I did
not test a here document expansion for this, that one might be different.)
This is the value of R.

When reviewing this, in order to propose some new text to replace what is
broken, I also noticed the following lines... (lines 75008-75010)

    If the complete expansion appropriate for a word results in an empty
    field, that empty field shall be deleted from the list of fields that
    form the completely expanded command,

which are fine up to there, but continue:

    unless the original word contained single-quote or double-quote
    characters.

which I am sure is not what was intended. Consider the (whole) word:
        ${F:+"xxx"}
If F is unset or null, the whole word (field) should be deleted, regardless
of the "" characters that are present - only quote characters that are
actually processed as part of the expansion matter for this purpose.

This could be a whole new bug report, but it seems a bit over the top to
not just fix it all in one action (the following lines, 75011..75019 are
the subject of another issue, so those can be handled with that one.)

To handle all of this, the following note will propose some new text to
replace (all of) lines 75006-75010, though I am not really happy with the
wording of most of it - at least it should provide a starting point for
discussion.

(0004071)
kre (reporter)
2018-08-05 21:46

Replace lines 75006 - 75010 (section 2.6, page 2353) with the following

        When the expansions in this section are performed other
        than in the context of preparing a command for execution
        they shall be carried out in the current shell execution
        environment.

        When expanding words for command about to be executed, and
        the word will be the command name, or an argument to the command,
        the expansions shall be carried out in the current shell
        execution environment (the environment for the command to be
        executed is unknown until the command word is known).

        When processing the words in a command about to be executed,
        which are used with variable assignments or redirects for that
        command, it is unspecified whether the expansions are carried
        out in the current execution environment or in the environment
        of the command about to be executed.

        If the complete expansion appropriate for a word results in an
        empty field, that empty field shall be deleted from the list of
        fields that form the completely expanded command, unless the
        original word contained single-quote or double-quote characters
        which were processed as part of expanding the word.
(0004072)
geoffclare (manager)
2018-08-06 11:01

Re: Note: 0004070
For completeness, here are the results for ksh88 (same on Solaris 11 and HP-UX):

$ A=: sh /tmp/T
A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY>
$ A=ls sh /tmp/T
A<ls> B<-l> C<> F<foo.c> R<> V<>
(0004073)
kre (reporter)
2018-08-06 15:06

Thanks for note 4072.

Upon reflection, the proposed wording for the final paragraph is all
wrong, and more complex than it needs to be. Instead of what I suggested
for that final paragraph, instead please consider

       If the result of an expansion from steps 1, 2, or 3 above results
       in a completely empty field, that field shall be deleted. However
       if step 4 produces an empty field, that field shall be retained.

No need to mention quote characters at all, as empty fields are all deleted
before the quote characters are, a field that is "" (literally) is not
empty. Also no real need to say that filename expansion cannot make
empty fields, if it were possible, they would be deleted!

But while considering that, I also realised that line 74998

        The order of word expansion shall be as follows:

also needs work (apart from the missing plural of expansion).

So, also please consider replacing that line (74998) with:

        While not all of the following word expansions are always
        performed, those that are performed occur in the following order:

or something like it. Eg: case patterns do not do filename expansion,
or field splitting, and subject to the resolution of another issue,
probably (hopefully) (once again) will not do quote removal either.
PSn expansions don't do them all either.
(0004082)
geoffclare (manager)
2018-08-10 09:11
edited on: 2018-08-30 16:11

Revised proposal based on mailing list discussion...

On page 2353 line 74991-74992 section 2.6, change:
This section describes the various expansions that are performed on words. Not all expansions are performed on every word, as explained in the following sections.
to:
This section describes the various expansions that are performed on words. Not all expansions are performed on every word, as explained in the following sections and elsewhere in this chapter. The expansions that are performed for a given word shall be performed in the following order:

On page 2353 line 74998 section 2.6, delete:
The order of word expansion shall be as follows:

Move lines 74999-75005 to follow line 74992.

On page 2353 line 75003 section 2.6, delete:
, unless IFS is null

On page 2353 line 75006-75007 section 2.6, change:
The expansions described in this section shall occur in the same shell environment as that in which the command is executed.
to:
When the expansions in this section are performed other than in the context of preparing a command for execution, they shall be carried out in the current shell execution environment.

When expanding words for command about to be executed, and the word will be the command name or an argument to the command, the expansions shall be carried out in the current shell execution environment. (The environment for the command to be executed is unknown until the command word is known.)

When expanding the words in a command about to be executed that are used with variable assignments or redirections, it is unspecified whether the expansions are carried out in the current execution environment or in the environment of the command about to be executed.

On page 2353 line 75008-75010 section 2.6, delete:
If the complete expansion appropriate for a word results in an empty field, that empty field shall be deleted from the list of fields that form the completely expanded command, unless the original word contained single-quote or double-quote characters.

On page 2359 line 75266 section 2.6.5, append a new sentence to the 2nd paragraph:
If no fields are delimited, for example if the input is empty or consists entirely of IFS white space, the result shall be zero fields (rather than an empty field).

On page 2359 line 75273 section 2.6.5, change:
If the value of IFS is null, no field splitting shall be performed.
to:
If the value of IFS is null, field splitting shall have no effect, except that if the input is empty the result shall be zero fields.



- Issue History
Date Modified Username Field Change
2017-03-04 07:56 kre New Issue
2017-03-04 07:56 kre Name => Robert Elz
2017-03-04 07:56 kre Section => 2.6
2017-03-04 07:56 kre Page Number => 2353
2017-03-04 07:56 kre Line Number => 75006-75007
2018-08-05 21:45 kre Note Added: 0004070
2018-08-05 21:46 kre Note Added: 0004071
2018-08-05 21:48 kre Note Edited: 0004070
2018-08-05 21:49 kre Note Edited: 0004070
2018-08-05 21:52 kre Note Edited: 0004070
2018-08-06 11:01 geoffclare Note Added: 0004072
2018-08-06 15:06 kre Note Added: 0004073
2018-08-06 15:29 kre Note Edited: 0004070
2018-08-10 09:11 geoffclare Note Added: 0004082
2018-08-10 09:12 geoffclare Relationship added related to 0001193
2018-08-10 11:36 geoffclare Note Edited: 0004082
2018-08-10 11:54 geoffclare Note Edited: 0004082
2018-08-24 09:14 geoffclare Note Edited: 0004082
2018-08-30 16:09 geoffclare Note Edited: 0004082
2018-08-30 16:10 geoffclare Note Edited: 0004082
2018-08-30 16:11 geoffclare Note Edited: 0004082
2018-08-30 16:14 geoffclare Interp Status => ---
2018-08-30 16:14 geoffclare Final Accepted Text => Note: 0004082
2018-08-30 16:14 geoffclare Status New => Resolved
2018-08-30 16:14 geoffclare Resolution Open => Accepted As Marked
2018-08-30 16:14 geoffclare Tag Attached: tc3-2008
2019-10-15 10:11 geoffclare Relationship added has duplicate 0001297
2019-10-31 11:23 geoffclare Status Resolved => Applied


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker