View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001191 | 1003.1(2016/18)/Issue7+TC2 | Shell and Utilities | public | 2018-04-13 14:33 | 2024-06-11 09:08 |
| Reporter | geoffclare | Assigned To | |||
| Priority | normal | Severity | Editorial | Type | Omission |
| Status | Closed | Resolution | Accepted As Marked | ||
| Name | Geoff Clare | ||||
| Organization | The Open Group | ||||
| User Reference | |||||
| Section | 2.2 Quoting | ||||
| Page Number | 2346 | ||||
| Line Number | 74697 | ||||
| Interp Status | --- | ||||
| Final Accepted Text | See 0001191:0004278. | ||||
| Summary | 0001191: The list of special characters in 2.2 is missing '-' and '!' | ||||
| Description | Section 2.2 lists special characters that either always or sometimes need quoting if they are to represent themselves. It is missing the characters '-' and '!' which are sometimes special (in bracket expressions). This appears to be an editorial oversight, since the purpose of this introductory text seems to be to warn shell script writers about which characters they need to think about quoting if they want them to be treated literally, rather than stating a normative requirement about which characters can be special. Luckily the lists here are not referenced by places such as 2.2.3; had that been the case, this would be a non-editorial matter. The text here also misuses the term "may", which is supposed to indicate "a feature or behavior that is optional for an implementation" (XBD 1.5). | ||||
| Desired Action | Change:and the following may need to be quoted under certain circumstances. That is, these characters may be special depending on conditions described elsewhere in this volume of POSIX.1-2008:to:* ? [ # ~ = % and the following might need to be quoted under certain circumstances. That is, these characters are sometimes special depending on conditions described elsewhere in this volume of POSIX.1-2008:* ? [ - ! # ~ = % | ||||
| Tags | tc3-2008 | ||||
| related to | 0001193 | Closed | Brace expansion and {var}>file redirects in the shell |
|
|
I suppose '^' ought to be included as well, since leaving it unquoted when it is the first character in a bracket expression produces unspecified results. Not sure whether it would be okay just to include it in the second list, or if it needs a separate statement (because of the unspecified results thing). |
|
|
As that is only listing what chars require it sometimes, not why's, to me just including '^' ok... Xrefs in XRAT for this section to the why's for each char may be desirable. |
|
|
If you include ^, !, -, then you should also include ] as I take it it's linked to the recent discussion whereby when used in wildcards in shells (and possibly also in fnmatch with backslash), quoting ^, !, -, ] removes their special meaning within bracket expressions. Also { and } should be quoted under some circumstance in many shells (bash (also in sh mode, I don't know if that applies to certified ones like on macOS), ksh93, zsh, pdksh, mksh (except with -o posix)) for brace expansion at least, it may be time to acknowledge it though it may warrant a separate ticket. I would say it's also time to *require* ^ be understood as equivalent to !. That's what most people (who haven't been exposed to ancient Bourne or ksh88 shells) expect nowadays. It's silly to deviate from regexps here when ^ is no longer a pipe operator in POSIX shells. |
|
|
Re 0001191:0003973 I agree ] should be in the list. I think { and } should be raised separately as it's questionable whether brace expansion is allowed by the standard. Requiring ^ to be equivalent to ! would also best be raised as a separate bug. |
|
|
Why would brace expansion not be allowed by the standard? `{' is a reserved word, not an operator, and does not qualify as a reserved word when used in a brace expansion context. |
|
|
Re: 0001191:0003976 Currently, AFAICT, POSIX requires echo {a,b} to output "{a,b}" and echo {fd}>&2 to output "{fd}" on stderr, making pdksh/ksh93/bash non-conformant (zsh disables those in sh mode).IIRC, there's some wording that says the "{" reserved word is only recognised as such in command position. But even then "{echo foo}" where we have a "{echo" word and not "{" is still required to run the "{echo" command with "foo}" as argument. |
|
|
(Sorry, I sent this only to Geoff first.) If it's an ordinary character, any additional interpretation falls outside the standard. A brace expansion is a WORD, and as long as the shell obeys the Posix word expansion and quoting rules, it's simply an extension. |
|
|
Re: 0001191:0003978 The shell cannot have carte blanche to make any character it likes have a special meaning as an extension. Otherwise how are shell script writers supposed to know what characters they need to quote? Are you saying they need to quote everything they don't want treated as special? (Which would be contrary to what's obviously intended by the text that is the subject of this bug.) |
|
|
Re: 3979: That is the situation we find ourselves in today. A script writer who expects portability should be, and usually is, aware of extensions in the various existing shells. Quoting is an easy remedy. Look at the other parallel discussions: all shells have extensions, and different ones to boot. Script writers are already aware of them. If you like, we can have a list of characters that script writers should be aware of, including `{', in the same spirit as the standard leaving the effects of common builtin extensions or characters following a $ that the standard doesn't cover as explicitly unspecified. |
|
|
Re: 0001191:0003980 valid extensions of shell syntax use the characters that the standard says are always special in ways that do not fit the POSIX syntax. For example the <<< redirection operator is a valid extension because < is an operator. Likewise process substitution using <(some_command) because < and ( are operators, and !(foo) pathname expansion because ( is an operator. Or they extend a construct that is specified by POSIX in a way that POSIX doesn't specify a behaviour for, such as ${var/foo/bar} which is valid because / can't be a character in a variable name. Brace expansion is the first time I have encountered an extension (of shell syntax) that does not appear to be allowed by POSIX. |
|
|
There's the long-deprecated $[expr] (explicitly unspecified in 2.6), csh-style history expansion (not a special character), and possibly some single-character parameter expansions that other shells implement (also explicitly unspecified). |
|
|
Re: 0001191:0003981 Other extensions that clash with POSIX that come to mind: - extra keywords that can't be used as function names (POSIX reserves some of ksh's) - special variables. POSIX lists a few like SECONDS, but not others like TMOUT, LD_PRELOAD and can't possibly list them all. - zsh's %?foo that is not taken as a glob (won't expand to %xfoo (to allow kill %?job unquoted) - csh-style history expansion (!, ^), still enabled by bash when running as sh, though that's only for interactive shells - echo options (-e from Unix V8 1981, -E from bash 1992, combinations like -nn, -ne) supported by many shells, a problem because there's no "--" to end the options with echo (though zsh supports "-" as a (clashing) extension). Here I'd say POSIX should be amended to allow at least echo -[eEn]* (note the * instead of +) to be unspecified, at it's clearly a case where POSIX fails to describe the current reality (or even the reality from 25 years ago). - &> redirection operator (foo &> bar per POSIX is "foo & > bar [nocommand]") (unlikely to be a problem in practice). - echo **/* (zsh) - .* expansion not including . and .. (pdksh/zsh) (though it's actually the behaviour most people would like) - commands defined as alias in ksh that can't be used as function name. |
|
|
Re: 0001191:0003982 Ah yes, history expansion. I'd forgotten that one. It's also not a valid POSIX extension, but since it only affects interactive shells and is easy to turn off, it's not nearly as much of an issue. |
|
|
Re: 0001191:0003983 the context is extensions that cause scripts to need extra quoting, so I think most of those are not relevant. |
|
|
Brace expansion is also easy to turn off, at least in bash (set +B), and it's easy to enable history expansion in non-interactive shells (set -H). It's a question of defaults. |
|
|
Re: 0001191:0003986 just to be clear, although I believe brace expansion is not currently allowed by POSIX, I am in favour of changing the standard to allow it. I am working on a new bug with the necessary changes. |
|
|
Change on Page 2346 Lines 74697-74699: and the following may need to be quoted under certain circumstances. That is, these characters may be special depending on conditions described elsewhere in this volume of POSIX.1-2008: * ? [ # ~ = % to: and the following might need to be quoted under certain circumstances. That is, these characters are sometimes special depending on conditions described elsewhere in this volume of POSIX.1-2008: * ? [ ] ^ - ! # ~ = % |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2018-04-13 14:33 | geoffclare | New Issue | |
| 2018-04-13 14:33 | geoffclare | Name | => Geoff Clare |
| 2018-04-13 14:33 | geoffclare | Organization | => The Open Group |
| 2018-04-13 14:33 | geoffclare | Section | => 2.2 Quoting |
| 2018-04-13 14:33 | geoffclare | Page Number | => 2346 |
| 2018-04-13 14:33 | geoffclare | Line Number | => 74697 |
| 2018-04-13 14:33 | geoffclare | Interp Status | => --- |
| 2018-04-13 14:44 | geoffclare | Note Added: 0003956 | |
| 2018-04-19 05:54 | shware_systems | Note Added: 0003971 | |
| 2018-04-19 05:55 | shware_systems | Note Edited: 0003971 | |
| 2018-04-19 15:32 | stephane | Note Added: 0003973 | |
| 2018-04-20 09:52 | geoffclare | Note Added: 0003975 | |
| 2018-04-20 13:30 | chet_ramey | Note Added: 0003976 | |
| 2018-04-20 13:39 | stephane | Note Added: 0003977 | |
| 2018-04-20 13:52 | chet_ramey | Note Added: 0003978 | |
| 2018-04-20 14:09 | geoffclare | Note Added: 0003979 | |
| 2018-04-20 14:35 | chet_ramey | Note Added: 0003980 | |
| 2018-04-20 14:56 | geoffclare | Note Added: 0003981 | |
| 2018-04-20 15:06 | stephane | Note Edited: 0003977 | |
| 2018-04-20 15:22 | geoffclare | Note Edited: 0003981 | |
| 2018-04-20 15:25 | chet_ramey | Note Added: 0003982 | |
| 2018-04-20 15:40 | stephane | Note Added: 0003983 | |
| 2018-04-20 15:43 | geoffclare | Note Added: 0003984 | |
| 2018-04-20 15:43 | stephane | Note Edited: 0003983 | |
| 2018-04-20 15:44 | geoffclare | Note Edited: 0003984 | |
| 2018-04-20 15:46 | geoffclare | Note Added: 0003985 | |
| 2018-04-20 15:46 | chet_ramey | Note Added: 0003986 | |
| 2018-04-20 16:05 | geoffclare | Note Added: 0003987 | |
| 2018-04-23 08:19 | geoffclare | Note Edited: 0003981 | |
| 2018-04-23 08:22 | geoffclare | Note Edited: 0003981 | |
| 2019-03-07 16:17 | geoffclare | Relationship added | related to 0001193 |
| 2019-03-07 16:26 | Don Cragun | Note Added: 0004278 | |
| 2019-03-07 16:27 | Don Cragun | Final Accepted Text | => See 0001191:0004278. |
| 2019-03-07 16:27 | Don Cragun | Status | New => Resolved |
| 2019-03-07 16:27 | Don Cragun | Resolution | Open => Accepted As Marked |
| 2019-03-07 16:28 | Don Cragun | Note Edited: 0004278 | |
| 2019-03-07 16:29 | Don Cragun | Tag Attached: tc3-2008 | |
| 2019-11-08 11:42 | geoffclare | Status | Resolved => Applied |
| 2024-06-11 09:08 | agadmin | Status | Applied => Closed |
