Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000500 [1003.1(2008)/Issue 7] Shell and Utilities Objection Error 2011-10-12 11:02 2020-05-05 14:44
Reporter geoffclare View Status public  
Assigned To ajosey
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Geoff Clare
Organization The Open Group
User Reference
Section 2.2.3 Double-Quotes
Page Number 2299
Line Number 72397
Interp Status Approved
Final Accepted Text Note: 0001000
Summary 0000500: Special meaning of <backslash> inside double quotes
Description XCU Section 2.2.3 says:

    The <backslash> shall retain its special meaning as an escape
    character (see Section 2.2.1, on page 2298) only when followed by
    one of the following characters when considered special:

    $ ` " \ <newline>

This doesn't work right for backslash itself. Consider:

    printf '%s\n' "\\."

To decide whether the first backslash in the double quotes is special
we need to know whether the character that follows (the second
backslash) is being considered special. The second backslash is not
special because it is not followed by $ ` " \ or <newline>. So, it
would appear that the first backslash should not be special (because
it is followed by a character that is not being considered special).
By this interpretation the output should contain both backslashes, but
in practice it contains one.

When this issue came up originally I thought that the phrase "when
considered special" is there because of <newline>, since it is
special when used in a line continuation but otherwise not special,
and suggested that the handling of <newline> should be described
separately. However, since the presence of the backslash in a line
continuation is what makes the <newline> special, I think that the
description works fine if "when considered special" is just removed.

(This was originally raised on the mailing list in March 2010 but
didn't ever get submitted as a bug.)
Desired Action Delete "when considered special".
Tags issue8
Attached Files

- Relationships
related to 0000221Appliedajosey poor wording about even quotes in double quoted parameter expansion 

-  Notes
(0000990)
wpollock (reporter)
2011-10-12 17:31

This is probably a Bash bug, but consider the problem this shows:

$ echo hi!
hi!

$ echo 'hi!'
hi!

$ echo "hi!"
bash: !": event not found

$ echo "hi\!"
hi\!

$ set +H # turn off history expansion

$ echo "hi!"
hi!

$ echo "hi\!"
hi\!

$ bash -version
GNU bash, version 4.2.10(1)-release (i386-redhat-linux-gnu)

The point is that history expansion is not mentioned in the standard here, so the Bash maintainer claims "!" should be a meta-character and didn't agree this was a bug. If the standard wants to allow or deny shells such as bash to define additional metacharacters, then the wording here needs to be updated to something such as:

option A:
The <backslash> shall retain its special meaning as an escape
    character (see Section 2.2.1, on page 2298) only when followed by
    one of the following characters:

option B:
The <backslash> shall retain its special meaning as an escape
    character (see Section 2.2.1, on page 2298) only when followed by
    any character considered special. The following 5 (five) characters are
    considered special:

The intent of option B is to leave the door open for other characters to be declared special. I don't like this option however, because it doesnt limit the set of special characters, so a script writer can't depend on backslash behavior, as shown in the Bash example (which shows that Bash considers "!" partially special). I believe all modern (and some historical) shells seem to have some form of history expansion, I would prefer something such as this:

option C:
The <backslash> shall retain its special meaning as an escape
    character (see Section 2.2.1, on page 2298) only when followed by
    one of the following characters considered "special":

           $ ` " \ <newline>

    Additionally, if a history expansion character is defined, it shall
    also be considered as a special character.

(Not sure if the use of "shall" is correct in that last sentence.)

In addition, maybe it is time to add an enhancement for history expansion to the standard, but that's a problem for another bug. Option C (my preferred solution) is probably not valid unless history expansion is defined. So I believe that only leaves option A, for now.
(0000991)
geoffclare (manager)
2011-10-13 09:21

(Response to Note: 0000990)

Bash configurations that have history expansion enabled are
non-conforming, but the standard allows for that through the concept
of "conforming environments". Implementations are allowed to support
non-conforming environments, and even have one as the default, as
long as they document how to obtain a conforming environment (see
the end of XBD section 2.1.1).

So there is no need to take account of history expansion in the
standard. (And even if there was, allowing ! to be special in
double quotes would not be the right approach, since ! operates
at a different level, as can be seen from the fact that it reports
the double quote as an event it did not find.)
(0000998)
geoffclare (manager)
2011-10-21 11:06

Note that 0000221 makes extensive changes to the paragraph at page
2299 line 72382, hence the changes below do not need to cover the
behaviour of backslash and double-quote within "${...}".

On page 2298 line 72363 section 2.2.1 change:

    If a <newline> follows the <backslash>

to:

    If a <newline> immediately follows the <backslash>

On page 2299 line 72396-72398 section 2.2.3 change:

    The <backslash> shall retain its special meaning as an escape
    character (see Section 2.2.1, on page 2298) only when followed by
    one of the following characters when considered special:

    $ ` " \ <newline>

to:

    Outside of "$(...)" and "${...}" the <backslash> shall retain its
    special meaning as an escape character (see Section 2.2.1, on page
    2298) only when immediately followed by one of the following
    characters:

    $ ` \ <newline>

    or by a double-quote character that would otherwise be considered
    special (see Section 2.6.4 on page NNNN and Section 2.7.4 on
    page NNNN).

On page 2299 line 72399 section 2.2.3 change:

    The application shall ensure that a double-quote is preceded by a
    <backslash> to be included within double-quotes.

to:

    The application shall ensure that a double-quote that is not
    within "$(...)" nor within "${...}" is immediately preceded by
    a <backslash> in order to be included within double-quotes.
(0001000)
msbrown (manager)
2011-11-03 15:13

Interpretation response
------------------------
The standard is unclear on this issue, and no conformance distinction can be made between alternative implementations based on this. This is being referred to the sponsor, with a suggested set of corrections in Note: 0000998 .


Notes to the Editor (not part of this interpretation):
-------------------------------------------------------
If approved, please use the corrective wording supplied in Note: 0000998
(0001297)
ajosey (manager)
2012-06-29 16:18

Interpretation proposed 29 June 2012 for final 45 day review
(0001338)
ajosey (manager)
2012-08-30 09:10

Interpretation approved 30 Aug 2012
(0004218)
kre (reporter)
2019-01-23 05:42

The approved text for this bug is incorrect too. The last paragraph...

    The application shall ensure that a double-quote that is not
    within "$(...)" nor within "${...}" is immediately preceded by
    a <backslash> in order to be included within double-quotes

is insufficient. The "not considered special" stuff relates to
expansions occurring in here documents and arithmetic contexts,
where it is stated that the word is interpreted as if it were in
double quoted, except that the double quote character is not
considered special.

A \ before a " in those contexts is simply a \ before a ", so
to include the " it should simply be included, and the application
most certainly should not ne inserting a \ before it.

Fixing this might be as simple as prefixing the paragraph with
"When double-quote is considered special" ...
(0004792)
geoffclare (manager)
2020-03-05 09:45

Re: Note: 0004218

The last paragraph does not apply to here documents because it says "in order to be included within double-quotes", and here documents are not within double-quotes.

It is also not intended to apply to $((...)) since there is no reason for an application to want to achieve the goal of including a " inside $((...)) as it would be a syntax error.

Having said that, the situation for $((...)) within double quotes is interesting. As far as I can tell the standard does not describe how the shell should behave when given something like:

echo "$((1"+"2))"

Ksh and bash seem to treat the second " as paired with the first, and the third as starting new double-quotes that end at the fourth, thus they output 3. The other shells I tried report an error.

So for ksh and bash the last paragraph is actually true: in order to include a " inside a $((...)) that is within double-quotes, and thus produce the expected syntax error, the " does need to be preceded by backslash.

In conclusion, I think that the existing resolution should be applied as-is and if anyone thinks the issue of how to parse "$((1"+"2))" is worth making explicitly unspecified, instead implicitly unspecified as it is now, then they should raise it as a separate bug.

- Issue History
Date Modified Username Field Change
2011-10-12 11:02 geoffclare New Issue
2011-10-12 11:02 geoffclare Status New => Under Review
2011-10-12 11:02 geoffclare Assigned To => ajosey
2011-10-12 11:02 geoffclare Name => Geoff Clare
2011-10-12 11:02 geoffclare Organization => The Open Group
2011-10-12 11:02 geoffclare Section => 2.2.3 Double-Quotes
2011-10-12 11:02 geoffclare Page Number => 2299
2011-10-12 11:02 geoffclare Line Number => 72397
2011-10-12 11:02 geoffclare Interp Status => ---
2011-10-12 17:31 wpollock Note Added: 0000990
2011-10-13 09:21 geoffclare Note Added: 0000991
2011-10-21 10:44 geoffclare Relationship added related to 0000221
2011-10-21 11:06 geoffclare Note Added: 0000998
2011-11-03 15:13 msbrown Interp Status --- => Proposed
2011-11-03 15:13 msbrown Note Added: 0001000
2011-11-03 15:13 msbrown Status Under Review => Interpretation Required
2011-11-03 15:13 msbrown Resolution Open => Accepted As Marked
2011-11-03 15:13 msbrown Desired Action Updated
2011-11-03 15:13 msbrown Final Accepted Text => Note: 0001000
2011-11-03 15:13 msbrown Tag Attached: issue8
2011-11-03 15:14 msbrown Interp Status Proposed => Pending
2012-06-29 16:18 ajosey Interp Status Pending => Proposed
2012-06-29 16:18 ajosey Note Added: 0001297
2012-08-30 09:10 ajosey Interp Status Proposed => Approved
2012-08-30 09:10 ajosey Note Added: 0001338
2019-01-23 05:42 kre Note Added: 0004218
2020-03-05 09:45 geoffclare Note Added: 0004792
2020-05-05 14:44 geoffclare Status Interpretation Required => Applied


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker