View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001976 | 1003.1(2024)/Issue8 | Shell and Utilities | public | 2026-03-13 21:10 | 2026-09-24 18:54 |
| Reporter | navi | Assigned To | |||
| Priority | normal | Severity | Editorial | Type | Enhancement Request |
| Status | New | Resolution | Open | ||
| Name | navi | ||||
| Organization | |||||
| User Reference | |||||
| Section | XCU: 2. Shell Command Language | ||||
| Page Number | -- | ||||
| Line Number | -- | ||||
| Interp Status | |||||
| Final Accepted Text | |||||
| Summary | 0001976: specify array variables and syntax | ||||
| Description | POSIX sh specifies one array per function, the argument array many sh implementations support having array variables past the argument array due to posix shell specifying a syntax for the argument array, sh implementations with array extensions naturally landed on similar and compatible syntax mksh, ksh93, bash, zsh, osh, yash, all support the following syntax: foo=(a b "c d" e) foo+=(f g) "${foo[@]}" # a b "c d" e f g "${foo[*]}" # "a b c d e f g" "${foo[expr]}" # indexed by expr, where expr is evaluated like $(( expr )) "${#foo[@]}" # 6, the lenght of foo additionally, mksh, ksh, and zsh, support declaring arrays as such: set -A name -- values notably ksh does not implement the name=(values) syntax, but has no other syntax that would conflict with it either ash and dash do not currently implement any sort of arrays, but also have no syntax that conflicts with the existing extensions | ||||
| Desired Action | rough first draft of changes: XBD 4.25: specify varname=(value) as assigning an array variable specify varname[expr]=value as assigning a entry in the array XCU: 2.5: specify array variables as containing an array of arbitrary byte sequences, except for null byte XCU: 2.6.2: specify ${parameter[@]} and ${parameter[*]} expansions matching 2.5.2 special parameters specify ${parameter[expr]} expansion, where expr is arithimetically expanded, then parameter is expanded matching 2.5.1 positional parameters | ||||
| Tags | No tags attached. | ||||
|
|
IMO, specifying arrays in POSIX sh is both impossible (see https://unix.stackexchange.com/questions/238080/test-for-array-support-by-shell/238129#238129 for details) and undesirable. Arrays would be yet another kludge added to a clunky shell design. Arrays/lists should have been the primary variable type in shells and several shells like rc and derivatives or fish have understood that. Unix had a chance to have a shell with clean list variable support with rc (from plan9 / Unix v10 in the late 80s; intended to be the successor of the Bourne shell then), but it's too late now. For sh, you'd want to at least disable implicit word splitting globbing upon unquoted parameter expansion (like with the set +o shwordsplit +o globsubst of the zsh implementation of sh) and certainly not go with the Korn design (as copied by bash). The fact that in all Bourne-like shell with arrays, indices are interpreted as arithmetic expressions complicates the parsing and tokenising significantly and is source of all sorts of bugs and vulnerabilities in particular if you want to allow array elements in any places that expect an lvalue. If you find yourself needing arrays in a sh script, you're probably using the wrong tool: consider switching to perl, ruby or other proper interpreted language, or a shell without so much of a baggage. |
|
|
I'm not sure I agree with that conclusion. The point of POSIX should not be to define a clean shell language. As you point out, it's too late for that and this is visible for more than just arrays. But shell scripts are not used for writing complex software. The reality of the situation is that (a) questions like the one you've linked to occur because people rely on arrays in practice, (b) support for arrays exists in numerous shells and having a common standard is more desirable than each new shell diverging in some new way, and (c) scripts currently need to bake kludges into their scripts like the ones you suggest in your answer you've linked to. I don't think "use another language" is the appropriate answer here but rather that the goal of POSIX should be to provide a uniform way to use the features people rely so that scripts can at least be written in a portable manner. |
|
|
since more words were asked for, this is my best attempt at a mostly complete wording, though i could have missed a few details i don't think i can edit the `desired action` field, so i'm adding the new wording here, and feedback appreciated. the += operator was left out, it could be added at a later time while also being specified for strings ---- XCU: ----------------------------- 2.5: Parameters and Variables ----------------------------- on page 2478, line 80337, change: A parameter is set if it has one or more assigned values -------------------------- 2.6.2: Parameter Expansion -------------------------- on page 2487, line 80692, change: ${#parameter} ${#parameter[word]} String Length. The shortest decimal representation of the length in characters of the first value of parameter, starting from zero, or the value denominated by word if word evaluates to one or more digits, shall be substituted. on page 2487, after line 80695, add: ${#parameter[@]} or ${#parameter[*]} Array Length. The decimal representation of the number of set values of parameter shall be substituted. If parameter is unset and set -u is in effect, the expansion shall fail." on page 2487, after line 80716, add: ${parameter[@]} Expands to the values of parameter, starting from zero, initially producing one field for each value that is set. When the expansion occurs in a context where field splitting will be performed, any empty fields may be discarded and each of the non-empty fields shall be further split as described in Section 2.6.5. When the expansion occurs within double-quotes, the behavior is unspecified unless one of the following is true: • Field splitting as described in Section 2.6.5 would be performed if the expansion were not within double-quotes (regardless of whether field splitting would have any effect; for example, if IFS is null). • The double-quotes are within the word of a ${parameter :−word} or a ${parameter :+word} expansion (with or without the <colon>; see Section 2.6.2) which would have been subject to field splitting if parameter had been expanded instead of word. If one of these conditions is true, the initial fields shall be retained as separate fields, except that if the parameter being expanded was embedded within a word, the first field shall be joined with the beginning part of the original word and the last field shall be joined with the end part of the original word. In all other contexts the results of the expansion are unspecified. If parameter has no values, the expansion shall generate zero fields, even when it is within double-quotes; however, if the expansion is embedded within a word which contains one or more other parts that expand to a quoted null string, these null string(s) shall still produce an empty field, except that if the other parts are all within the same double-quotes as the expression, it is unspecified whether the result is zero fields or one empty field. ${parameter[*]} Expands to the values of parameter, starting from zero, initially producing one field for each positional parameter that is set. When the expansion occurs in a context where field splitting will be performed, any empty fields may be discarded and each of the non-empty fields shall be further split as described in Section 2.6.5. When the expansion occurs in a context where field splitting will not be performed, the initial fields shall be joined to form a single field with the value of each parameter separated by the first character of the IFS variable if IFS contains at least one character, or separated by a <space> if IFS is unset, or with no separation if IFS is set to a null string ${parameter[word]} If word evaluates to one or more digits, the value of parameter denoted by word shall be substituted. Otherwise, the first value of parameter, starting from zero, shall be substituted. ----------------------------- 2.9.1.2: Variable Assignments ----------------------------- on page 2501, after line 81278, add: If value starts with an unquoted open parenthesis and no command results, all characters following the open parenthesis to the matching closing parenthesis shall be split into assignments. If any assignment has the form [word]=value, the behaviour is undefined. Each assignment shall be expanded as a word (see Section 2.6.2) and Field splitting (see Section 2.6.5) shall be performed on any fields generated by the expansion, each resulting field shall be assigned to as values to the variable, in order, starting from zero. Then the variable shall be given the array attribute, and it's export attribute shall be removed. If value starts with an unquoted open parenthesis and a command name results, the behaviour is undefined. ------------------------------------------- 2.10.2.7: Assignment preceding command name ------------------------------------------- on page 2514, line 81790, change: If all the characters in the TOKEN preceding the first such <equals-sign> form a valid name (see XBD Section 3.216, on page 63), or a valid name, followed by open bracket, word, and close bracket, the token ASSIGNMENT_WORD shall be returned. on page 2514, line 81797, change: If a returned ASSIGNMENT_WORD token begins with a valid name and is not followed by open-bracket, word, close-bracket, assignment of the value after the first <equal-sign> to the first value of name, starting from zero, shall occur as specified in Section 2.9.1. on page 2514, after line 82800, add: If a returned ASSIGNMENT_WORD token begin with a valid name followed by open-bracket, word, close-bracket, and if word evaluates to one or more digits, the assignment of value to the parameter value denoted by word shall occur as specified in Section 2.9.1, otherwise if word does not evaluate to one or more digits, the assignment of the first value of parameter, starting from zero, shall occur. Then the variable shall be set as an array, and it's export attribute shall be unset. If a returned ASSIGNMENT_WORD token begin with a valid name followed by open-bracket, word, close-bracket, and value begins with an unquoted open-parenthesis, the behaviour is undefined. -------------------------------- 2.15: Special Built-In Utilities -------------------------------- on page 2538, after line 82777, add: If the variable corresponding to a specified name has the array attribute, it shall not be given the export attribute. alternatively: If the variable corresponding to a specified name has the array attribute, the behaviour is undefined. on page 2571, after line 83910, add: If name has the form `name[word]`, if word evaluates to one or more digits, the value of `name` denominated by word shall be unset, otherwise it's first value, starting from zero, shall be unset. ----- ----- a few notes: for array length expansions: --- since zsh and pdksh derivatives counts gaps in Array Length expansions, it may be necessary to also include: "If there are unset values in parameter between set values, the behaviour is undefined." although zsh does already differ from posix in numerous points, it could also emulate this behaviour in sh mode. i am unsure if pdksh or derivatives could do so --- yash has quite a few changes that differ from all other shells, but as they do not make arrays available when ran as `sh`, it shouldn't be a problem to implement a similar syntax to the other shells when in sh mode --- zsh and ksh derivatives allow `set -A array [values...]` to set a variable as an array, in addition to `array=()` bash does not support set -A, but also has no conflicting -A flag in set, so specifying both array=(), which is the most commonly used form, and set -A, could be considered. |
|
|
This is a very poor idea - sh does not need arrays. If some kind of complex data type, more than the generic string that is all sh currently has, were to be added, the obvious one to add would be a list, not an array. No-one has (yet) seen a need to implement that, that I'm aware of anyway, so even lists can't be a particularly missing feature, though sh deals with lists of things all the time. That some shells (beginning long ago) came to the conclusion that no programming language could be complete without having arrays as one of its data types is of no relevance. Were arrays ever to be added as a standard feature, the time to do that would have been many versions ago, probably when the first SUS versions were being created, not now. Further, the first line of the Description of the issue is incorrect: POSIX sh specifies one array per function, the argument array There is no such thing in POSIX sh, the positional parameters are not an array, not now, not ever. This is obvious, given that some shells which have arrays, have invented a well known (to them) variable of an array type, to hold the positional parameters. If the POSIX specified positional parameters were an array, that would never have been needed. Reject this proposal. |
|
|
kre, what, in your view, makes a list different from an array in the proposed context? Compound assignment syntax, expansion semantics, something else? |
|
|
Re: 0001976:0007504 ... lists have no indexing, and can have lists as values of elements. A sane implementation would not use var assigns to create/manipulate lists, but new built-in commands. Expansions could be something as is done for arrays, ${var} al;ways returning the head of the list, and perhaps ${@var} ${*var} returning some forms of the entire list - I haven't seen a need to attempt any kind of implementation of this, so I haven't spent any time working out the appropriate additions - in my view, none of it is needed. |
|
|
OK, so more like classic Lisp than, say, what rc calls lists (which are more just indexed arrays). |
|
|
Yes, though as I said, I haven't encountered enough of a need for anything at all in this area yet, so the semantics etc are more just a dream than anything concrete. Potentially just simple FIFO lists might be all that was needed, along with search, cut, insert, and duplicate mechanisms (with just string values). But until a need arises, something reasonable to implement in sh, and which could really benefit from this, unless someone else does it, we'll never know. Certainly no lisp style notation though, nor any of its complexities (no code implemented as lists...) For now, in the applications I have which use lists (and I have several) just using functions, with the lists implemented as a sequence of quoted strings, suitable for manipulation with methods like inlist() { local X A="$1"; shift; eval set -- "$@" ; for X do [ "$X" = "$A" ] && return 0; done; return 1; } (which I just recreated from memory, so I might have some details/syntax wrong). That (and several more funcs for other operations) has always been enough. All my scripts like this run "fast enough", they'd be faster with more built in, but they are good enough as they are (some of them run those functions thousands of times over reasonably big lists - still entirely fast enough for me, and what slowness is usually doing external commands - copying files, running make, ...) |
|
|
And oops - that inllist() was a wacky cross-over of 2 different things, it would usually be just eval set -- \$"$2" (used as -- inlist "$value" listname ) - a version with the expanded ${listname} as args also exists, that has no need of the eval set at all. I knew I shouldn't trust memory! |
|
|
A few comments on 0001976:0007485 First, here I'll only consider these 4 array array implementations: - zsh in zsh emulation (the intended array design there) - zsh in sh emulation (array design bent to be closer to that of ksh). - ksh and clones such as pdksh and derivatives or (yes, I know there is quite a bit of variation between all of them, and between versions thereof, but they agree on the main lines). - yash (disabled in strict posix mode). > ----------------------------- > 2.5: Parameters and Variables > ----------------------------- > > on page 2478, line 80337, change: > A parameter is set if it has one or more assigned values That's at odds with the ksh design (the only one with sparse arrays and array indexes starting at 0) where a parameter that is an array is considered set (for set -o nounset or ${array+set} for instance) if and only if its element of index 0 is assigned a value (empty or not). And with the zsh and yash designs where an array is considered set if assigned any list, including the empty list. > -------------------------- > 2.6.2: Parameter Expansion > -------------------------- > > on page 2487, line 80692, change: > > ${#parameter} ${#parameter[word]} > String Length. > The shortest decimal representation of the length in characters of the first > value of parameter, starting from zero, or the value denominated by word if > word evaluates to one or more digits, shall be substituted. In yash, ${#array} (short for ${#array[@]} like $array is short for ${array[@]}) is the list of the length of elements of the array: $ yash -c 'a=("" a bb); echo "${#a}"' 0 1 2 In zsh, the number of elements in the array: $ zsh -c 'a=("" a bb); echo ${#a}' 3 In ksh, the length of the element of index 0 if set or 0 otherwise: $ ksh -c 'a[3]=aa; echo ${#a}' 0 For zsh or yash where arrays and scalars are separate types, ${scalar[n]} expands to the nth *character* or $scalar There's also variation in behaviour across shells if "word" evaluates to 0 or starts with 0 (where the number may be interpreted as octal). > on page 2487, after line 80695, add: > > ${#parameter[@]} or ${#parameter[*]} > Array Length. The decimal representation of the number of set values of > parameter shall be substituted. If parameter is unset and set -u is in effect, > the expansion shall fail." In yash, ${#array[@]} is again a list of lengths of array elements, and ${#array[*]} is the length of ${array[*]}, ie the length of the concatenation of all array elements with the first character of $I F S in between The number of elements is in ${array[#]} there. > on page 2487, after line 80716, add: > > ${parameter[@]} > Expands to the values of parameter, starting from zero, initially producing one > field for each value that is set. Only ksh array design (and zsh --emulate [k]sh for compatibility) has array indexes starting at 0 (and sparse arrays, not in zsh --emulate [k]sh). [...] > ${parameter[word]} > If word evaluates to one or more digits, the value of parameter denoted by word > shall be substituted. > Otherwise, the first value of parameter, starting from zero, shall be > substituted. In all aforementioned array designs, "word" is interpreted as an arithmetic expression. ${array[foo]} will expand to the first element if foo is an expression that evaluates to 1 (0 in ksh), such as after foo=1 or foo=4/4 > ----------------------------- > 2.9.1.2: Variable Assignments > ----------------------------- > > on page 2501, after line 81278, add: > > If value starts with an unquoted open parenthesis and no command results, all > characters following the open parenthesis to the matching closing parenthesis > shall be split into assignments. If any assignment has the form [word]=value, > the behaviour is undefined. Note that the var=(values) syntax, initially from zsh is not supported in ksh88 or pdksh derivatives other than mksh. In ksh93 var=() and other variations defines $var as a compound variable, not an array variable. You need a typeset -a var=() there. $ ksh -c 'a=(b=1) b=() c=(get() uname); typeset -p a b c' typeset -C a=(b=1) typeset -C b=() typeset -C c=Linux You don't say how the "characters [...] shall be split" nor how the parenthesis matching is meant to be done. Here it's more about tokenisation of what's inside like how the arguments in a simple command are tokenised except newlines are also allowed (and comments on each line). Tokens such as ";" may be allowed in some shells under some circumstance. > Each assignment shall be expanded as a word (see Section 2.6.2) and Field > splitting (see Section 2.6.5) shall be performed on any fields generated by the > expansion, each resulting field shall be assigned to as values to the variable, > in order, starting from zero. Then the variable shall be given the array > attribute, and it's export attribute shall be removed. Again, it's starting with 1 in most shells, just like $@ starts at $1. yash supports exporting arrays in which case the corresponding environment string is made of the concatenation of elements with colons in between. $ yash -c 'export a; a=(foo bar); printenv a' foo:bar [...] > If all the characters in the TOKEN preceding the first such <equals-sign> form > a valid name (see XBD Section 3.216, on page 63), or a valid name, followed by > open bracket, word, and close bracket, the token ASSIGNMENT_WORD shall be > returned. [...] Note ksh currently treats: a[1 + 1]=foo the same as a[1+1]=foo making it currently non-compliant, as that's currently required to run the a[1 command with + and 1]=foo as arguments. zsh complains about the unmatched [ / ] glob operators in the "a[1" and "1]=foo" words. yash doesn't support the a[word]=value notation. You need to use its array builtin for that. In zsh array[1,3]=(values) assigns array slices. In ksh93, that's either assigning to array[1] if in a locale where the decimal radix is "," or to array[3] otherwise ("," arithmetic operator). Also in zsh, for a scalar variable scalar[3]=value changes the 3rd *character* of $scalar (assuming $scalar's length is at least 3, it appends the value otherwise). [...] > If a returned ASSIGNMENT_WORD token begin with a valid name followed by open-bracket, word, close-bracket, and if word evaluates to one or more digits, the assignment of value to the parameter value denoted by word shall occur as specified in Section 2.9.1, otherwise if word does not evaluate to one or more digits, the assignment of the first value of parameter, starting from zero, shall occur. Then the variable shall be set as an array, and it's export attribute shall be unset. same as above, in a[word]=value (except in yash where that syntax is not supported), word is interpreted as an arithmetic expression. [...] > -------------------------------- > 2.15: Special Built-In Utilities > -------------------------------- > > on page 2538, after line 82777, add: > > If the variable corresponding to a specified name has the array attribute, it shall not be given the export attribute. In ksh88, arrays and scalars are not really differentiated, it's more that $var is short for ${var[0]}. It remains the case to some extent in clones and derivatives. ksh93 and mksh at least still allow you to export an array, and it's then exported with the value of the element of index 0 if set. [...] > on page 2571, after line 83910, add: > > If name has the form `name[word]`, if word evaluates to one or more digits, the value of `name` denominated by > word shall be unset, otherwise it's first value, starting from zero, shall be unset. Same again about arithmetic expressions. [...] > for array length expansions: > --- > since zsh and pdksh derivatives counts gaps in Array Length expansions, it may be necessary to also include: zsh arrays like the arrays of virtually all shells (bourne $@, csh, rc, es, fish, yash...) or languages are not sparse, they never have "gaps". In zsh a=() a[3]=foo Is equivalent to a=('' '' foo) # a=('' '' '' foo) in [k]sh emulation mksh's ${#a[@]} is now no longer 1 + the-highest-index. I don't know when that changed or if that's also the case of some other pdksh derivatives. yash won't let you set the element of index 3 is the element has fewer than 2 elements. [...] > yash has quite a few changes that differ from all other shells, but as they do not make arrays available when ran as `sh`, it shouldn't be a problem to implement a similar syntax to the other shells when in sh mode [...] I wouldn't call that "changes", it just didn't intend to copy the ksh array design which arguably is by far the worst across all shells with array support. It would be very sad to have to force the ksh array design onto other sh implementations. But again, adding arrays to sh can only make it very clunky unless like zsh you fix quite a few of the design issues especially the implicit split+glob upon expansions (which was a bit like the Bourne shell's ersatz for arrays). Now, maybe POSIX could consider specifying zsh's set +o shwordsplit +o globsubst to address that and specify a subset of the zsh array design, which would be a good start but I expect objected by maintainer of all shells that already have a different array design. [...] > zsh and ksh derivatives allow `set -A array [values...]` to set a variable as an array, in addition to `array=()` [...] Yes, in ksh, that was *only* with set -A. zsh introduced the array=(...) syntax (obviously inspired from csh's set a = (...) or perl's @a = (...) or rc's a = (...)), "copied" and extended by ksh93 (adding several other types using that syntax) and later bash. > bash does not support set -A, but also has no conflicting -A flag in set, so specifying both array=(), which is the most commonly used form, and set -A, could be considered. Missing from the specification proposal are things like: - array elements for builtins that take lvalues, like read 'a[word]', unset 'a[word]' - how to reference array elements in arithmetic expressions, like in `$(( a[b + $(cmd)] += 1 ))`. That's a mine field. Array support in bash/ksh/zsh is why their arithmetic expressions end up being ACE vulnerabilities. For example bash -c 'echo "$((expr))"' will run reboot if run after export expr='a[$(reboot)]' - read -A (pendant of set -A; bash annoyingly changed it to read -a). My position remains that it's hopeless trying to specify arrays for sh. Yes, lists should have been the primary variable type. Lists are primarily what you manipulate in a shell (list of command arguments, or files, etc). Shells like rc or fish (and even csh to some extent though csh remains unusable for plenty of other separate reasons) which came later understood that and made list variables first citizens, but sh started on the wrong foot, and all awkward array designs that have been added on top are incompatible with each other and irreconcilable IMO. I hope my comments above make it clear to everyone. You can find some complement of information in my post at https://unix.stackexchange.com/questions/238080/test-for-array-support-by-shell/238129#238129 POSIX could specify rc which has a much better design but I wouldn't expect many to rush and use it as a result. |
|
|
($I F S spelled as such because without the spaces, the comment is rejected with 403 Forbidden; was that not meant to have been fixed?) |
|
|
Re: 0001976:0007509 > Missing from the specification proposal are things like: > - array elements for builtins that take lvalues, like read 'a[word]', unset 'a[word]' > - how to reference array elements in arithmetic expressions, like in `$(( a[b + $(cmd)] += 1 ))`. That's a mine field. Array support in bash/ksh/zsh is why their arithmetic expressions end up being ACE vulnerabilities. For example bash -c 'echo "$((expr))"' will run reboot if run after export expr='a[$(reboot)]' > - read -A (pendant of set -A; bash annoyingly changed it to read -a). Also, if POSIX were to specify an array design that allows sparse arrays à la ksh (or in other words, sort of associative arrays with keys being positive integers), we'd need a way to get the list of keys of that associative array such as the ${!array[@]} of ksh93 and bash (mksh as well in recent versions), similar to the ${(k)assoc} of zsh for associative arrays. Awkwardly, bash, ksh have no operator to copy their arrays preserving the keys. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2026-03-13 21:10 | navi | New Issue | |
| 2026-04-24 07:17 | stephane | Note Added: 0007423 | |
| 2026-04-25 06:04 | Love4Boobies | Note Added: 0007424 | |
| 2026-08-12 15:36 | navi | Note Added: 0007485 | |
| 2026-09-21 16:18 | kre | Note Added: 0007503 | |
| 2026-09-21 16:20 | kre | Note Edited: 0007503 | |
| 2026-09-21 16:21 | kre | Note Edited: 0007503 | |
| 2026-09-21 16:47 | chet_ramey | Note Added: 0007504 | |
| 2026-09-21 17:53 | kre | Note Added: 0007505 | |
| 2026-09-21 19:24 | chet_ramey | Note Added: 0007506 | |
| 2026-09-21 23:12 | kre | Note Added: 0007507 | |
| 2026-09-21 23:18 | kre | Note Added: 0007508 | |
| 2026-09-21 23:19 | kre | Note Edited: 0007508 | |
| 2026-09-23 19:04 | stephane | Note Added: 0007509 | |
| 2026-09-23 19:05 | stephane | Note Added: 0007510 | |
| 2026-09-23 19:15 | stephane | Note Added: 0007511 | |
| 2026-09-24 18:54 | stephane | Note Edited: 0007509 |
