0001123: Problematic specification of execution environment for word expansions

Notes
(0004070) kre (reporter) 2018-08-05 21:45 edited on: 2018-08-06 15:29	Since this report must be reaching the head of the queue to be discussed soon, I thought I'd do a little experiment to see what the standard should actually say for this ... I ran the following little script (which was /tmp/T) V=${C:=XYZZY} $A ${B=-l} ${F:=foo.c} > ${R:=/tmp/junk} 2>/dev/null printf 'A<%s> B<%s> C<%s> F<%s> R<%s> V<%s>\n' \ "$A" "${B}" "${C}" "${F}" "${R}" "${V}" in a loop like: for S in 'bash -o posix' bash <....> do printf '%s: ' "${S}" A=: $S /tmp/T printf '%s: ' "${S}" A=ls $S /tmp/T done (using every shell I have available that I remembered ...) and obtained the following results: bash -o posix: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> bash -o posix: A<ls> B<-l> C<XYZZY> F<foo.c> R<> V<> bash: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> bash: A<ls> B<-l> C<XYZZY> F<foo.c> R<> V<> bosh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> bosh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> dash: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> dash: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> fbsh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> fbsh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> ksh93: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> ksh93: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> ksh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> ksh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> mksh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> mksh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> pdksh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> pdksh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> sh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> sh: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> yash: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> yash: A<ls> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> zsh: A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<> zsh: A<ls> B<-l> C<> F<foo.c> R<> V<> Most of those shells are obvious to all of you I think, "sh" is the NetBSD shell, "ksh" is /bin/ksh on NetBSD which is a very very close relation of pdksh (and I'd be surprised if the results were different, and they aren't...) fbsh is a slightly old now version of the FreeBSD shell. I included the V= output just for interest, what happens to V is unrelated to the current issue, and is best ignored here. What this shows is that all shells (or all I tested, I have no access to either ksh88 or some old SysVR4 /bin/sh) expand the command word and its args in the current shell environment, not that of the command to be executed, which I think is unavoidable (as mentioned in the original bug report). That is the values of B and F. Further all shells with the exception of zsh (which was not in its posix emulation mode, as I am not a zsh user, and have no idea how to make that happen correctly, it has no "-o posix") expand the variable assignment in the current shell execution environment, only zsh does it in the envoronment of the command to be executed. This is the value of C. And lastly, all shells but bash (in posix mode, or not) and zsh evaluate expansions in redirects in the current shell environment, not in the environment of the command to be executed (but as you will observe I did not test a here document expansion for this, that one might be different.) This is the value of R. When reviewing this, in order to propose some new text to replace what is broken, I also noticed the following lines... (lines 75008-75010) If the complete expansion appropriate for a word results in an empty field, that empty field shall be deleted from the list of fields that form the completely expanded command, which are fine up to there, but continue: unless the original word contained single-quote or double-quote characters. which I am sure is not what was intended. Consider the (whole) word: ${F:+"xxx"} If F is unset or null, the whole word (field) should be deleted, regardless of the "" characters that are present - only quote characters that are actually processed as part of the expansion matter for this purpose. This could be a whole new bug report, but it seems a bit over the top to not just fix it all in one action (the following lines, 75011..75019 are the subject of another issue, so those can be handled with that one.) To handle all of this, the following note will propose some new text to replace (all of) lines 75006-75010, though I am not really happy with the wording of most of it - at least it should provide a starting point for discussion.

(0004071) kre (reporter) 2018-08-05 21:46	Replace lines 75006 - 75010 (section 2.6, page 2353) with the following When the expansions in this section are performed other than in the context of preparing a command for execution they shall be carried out in the current shell execution environment. When expanding words for command about to be executed, and the word will be the command name, or an argument to the command, the expansions shall be carried out in the current shell execution environment (the environment for the command to be executed is unknown until the command word is known). When processing the words in a command about to be executed, which are used with variable assignments or redirects for that command, it is unspecified whether the expansions are carried out in the current execution environment or in the environment of the command about to be executed. If the complete expansion appropriate for a word results in an empty field, that empty field shall be deleted from the list of fields that form the completely expanded command, unless the original word contained single-quote or double-quote characters which were processed as part of expanding the word.

(0004072) geoffclare (manager) 2018-08-06 11:01	Re: Note: 0004070 For completeness, here are the results for ksh88 (same on Solaris 11 and HP-UX): $ A=: sh /tmp/T A<:> B<-l> C<XYZZY> F<foo.c> R</tmp/junk> V<XYZZY> $ A=ls sh /tmp/T A<ls> B<-l> C<> F<foo.c> R<> V<>

(0004073) kre (reporter) 2018-08-06 15:06	Thanks for note 4072. Upon reflection, the proposed wording for the final paragraph is all wrong, and more complex than it needs to be. Instead of what I suggested for that final paragraph, instead please consider If the result of an expansion from steps 1, 2, or 3 above results in a completely empty field, that field shall be deleted. However if step 4 produces an empty field, that field shall be retained. No need to mention quote characters at all, as empty fields are all deleted before the quote characters are, a field that is "" (literally) is not empty. Also no real need to say that filename expansion cannot make empty fields, if it were possible, they would be deleted! But while considering that, I also realised that line 74998 The order of word expansion shall be as follows: also needs work (apart from the missing plural of expansion). So, also please consider replacing that line (74998) with: While not all of the following word expansions are always performed, those that are performed occur in the following order: or something like it. Eg: case patterns do not do filename expansion, or field splitting, and subject to the resolution of another issue, probably (hopefully) (once again) will not do quote removal either. PSn expansions don't do them all either.

(0004082) geoffclare (manager) 2018-08-10 09:11 edited on: 2018-08-30 16:11	Revised proposal based on mailing list discussion... On page 2353 line 74991-74992 section 2.6, change: This section describes the various expansions that are performed on words. Not all expansions are performed on every word, as explained in the following sections. to: This section describes the various expansions that are performed on words. Not all expansions are performed on every word, as explained in the following sections and elsewhere in this chapter. The expansions that are performed for a given word shall be performed in the following order: On page 2353 line 74998 section 2.6, delete: The order of word expansion shall be as follows: Move lines 74999-75005 to follow line 74992. On page 2353 line 75003 section 2.6, delete: , unless IFS is null On page 2353 line 75006-75007 section 2.6, change: The expansions described in this section shall occur in the same shell environment as that in which the command is executed. to: When the expansions in this section are performed other than in the context of preparing a command for execution, they shall be carried out in the current shell execution environment. When expanding words for command about to be executed, and the word will be the command name or an argument to the command, the expansions shall be carried out in the current shell execution environment. (The environment for the command to be executed is unknown until the command word is known.) When expanding the words in a command about to be executed that are used with variable assignments or redirections, it is unspecified whether the expansions are carried out in the current execution environment or in the environment of the command about to be executed. On page 2353 line 75008-75010 section 2.6, delete: If the complete expansion appropriate for a word results in an empty field, that empty field shall be deleted from the list of fields that form the completely expanded command, unless the original word contained single-quote or double-quote characters. On page 2359 line 75266 section 2.6.5, append a new sentence to the 2nd paragraph: If no fields are delimited, for example if the input is empty or consists entirely of IFS white space, the result shall be zero fields (rather than an empty field). On page 2359 line 75273 section 2.6.5, change: If the value of IFS is null, no field splitting shall be performed. to: If the value of IFS is null, field splitting shall have no effect, except that if the input is empty the result shall be zero fields.

Issue History
Date Modified	Username	Field	Change
2017-03-04 07:56	kre	New Issue
2017-03-04 07:56	kre	Name	=> Robert Elz
2017-03-04 07:56	kre	Section	=> 2.6
2017-03-04 07:56	kre	Page Number	=> 2353
2017-03-04 07:56	kre	Line Number	=> 75006-75007
2018-08-05 21:45	kre	Note Added: 0004070
2018-08-05 21:46	kre	Note Added: 0004071
2018-08-05 21:48	kre	Note Edited: 0004070
2018-08-05 21:49	kre	Note Edited: 0004070
2018-08-05 21:52	kre	Note Edited: 0004070
2018-08-06 11:01	geoffclare	Note Added: 0004072
2018-08-06 15:06	kre	Note Added: 0004073
2018-08-06 15:29	kre	Note Edited: 0004070
2018-08-10 09:11	geoffclare	Note Added: 0004082
2018-08-10 09:12	geoffclare	Relationship added	related to 0001193
2018-08-10 11:36	geoffclare	Note Edited: 0004082
2018-08-10 11:54	geoffclare	Note Edited: 0004082
2018-08-24 09:14	geoffclare	Note Edited: 0004082
2018-08-30 16:09	geoffclare	Note Edited: 0004082
2018-08-30 16:10	geoffclare	Note Edited: 0004082
2018-08-30 16:11	geoffclare	Note Edited: 0004082
2018-08-30 16:14	geoffclare	Interp Status	=> ---
2018-08-30 16:14	geoffclare	Final Accepted Text	=> Note: 0004082
2018-08-30 16:14	geoffclare	Status	New => Resolved
2018-08-30 16:14	geoffclare	Resolution	Open => Accepted As Marked
2018-08-30 16:14	geoffclare	Tag Attached: tc3-2008
2019-10-15 10:11	geoffclare	Relationship added	has duplicate 0001297
2019-10-31 11:23	geoffclare	Status	Resolved => Applied

Aardvark Mark IV