View Issue Details

IDProjectCategoryView StatusLast Update
00019131003.1(2024)/Issue8Shell and Utilitiespublic2025-09-25 11:34
Reportercalestyo Assigned To 
PrioritynormalSeverityEditorialTypeEnhancement Request
Status NewResolutionOpen 
NameChristoph Anton Mitterer
Organization
User ReferenceShell & Utilities
Section2.7.5, 2.7.6
Page Number2497
Line Number81097-81118
Interp Status
Final Accepted Text
Summary0001913: clarify/define the meaning of n<&n and m>&m redirections
DescriptionHey.

This originated from a thread that started at
https://collaboration.opengroup.org/operational/mailarch.php?soph=N&action=show&archive=austin-group-l&num=38114&limit=100&offset=100&sid=

In short:
The motivation was whether it's possible to portably differentiate whether a non-zero exit status originated from a utility or failed redirection on that.
The idea was that, with the clarifications from #1879 and assuming that there is at least on exit status which a utility is known not to use, a construct like the following:
   ( command exec <some redirections> || exit 125; utility )
where the subshell is merely to clean up any redirections, can be used to differentiate between a redirection error (which above would be indicated by 125) or a non-zero exit status from the utility.

For file descriptors less than or equal to 2, this should already be guaranteed to work portably, as POSIX demands those to be passed on to the utility.
For FDs greater than two, this is no longer guaranteed, though.

An idea was brought up by Harald van Dijk, that n<&n and m>&m could be used e.g.:
   ( command exec <some redirections> || exit 125; utility n<&n... m>&m... )
in order to "manually" pass on on any such FDs.

Questions remained:
a) Does that behaviour even follow from POSIX (in an obvious way where it's really clear that shells should behave like this, not just some wobbly way)
b) Does it even solve the original problem, or could e.g. such a n<&n respectively m>&m fail itself (not e.g. because a file doesn't exist, but because of something like resource exhaustion, etc.)

When the original thread was continued a bit later at:
https://collaboration.opengroup.org/operational/mailarch.php?soph=N&action=show&archive=austin-group-l&num=38279&limit=100&offset=0&sid=
it apparently turned out that n<&n and m>&m redirections are not defined because POSIX uses the wordings:
"The redirection operator: [n]<&word shall duplicate one input file descriptor from another"
respectively
"The redirection operator: [n]>&word shall duplicate one output file descriptor from another"
i.e. one from another, implying the two must not be the same, as Geoff Clare pointed out.
Desired Action1. It shall be clarified whether or not the n<&n and m>&m forms are covered by the current wording of the standard.
(One could perhaps also interpret the "one from another" as the "one" being the FD from the utility's PoV, and the "another" being the FD of the same number from the shell's PoV.)

2. If possible - i.e. no conflicting behaviour of shell implementations - it would be nice if the definition could be change in such a way, that it actually allows portably for the goals described above.
I have no strong opinion one how this definition should look like, one suggestion on the list was that implementations that the n<&n and m>&m forms shalll be no-ops for implementations which *do* pass on FDs > 2, and for those that don't, it shall have the meaning of explicitly passing the given FD on.

It would be even nicer, if the standard mention in a sentence that this is explicitly meant to be usable for the purpose of differentiating between utility and redirection errors.

Thanks,
Chris.
TagsNo tags attached.

Activities

larryv

2025-03-12 07:00

reporter   bugnote:0007111

Desired Action:
It would be even nicer, if the standard mention in a sentence that this is explicitly meant to be usable for the purpose of differentiating between utility and redirection errors.
The standard does not exist for your personal benefit, so I do not think it should bless your highly specific use case. However, it would make sense to briefly mention the closing-high-FDs behavior as motivation. Something along the lines of Geoff's wording in the thread would be more than sufficient:
So I would support updating the standard to require that n<&n and n>&n are always a no-op if fd n is open, except that if the shell normally closes fds > 2, that were opened with exec, when it executes a non-built-in utility, then applying n<&n or n>&n to such commands causes fd n to remain open.

calestyo

2025-03-13 02:41

reporter   bugnote:0007112

The standard does not exist for your personal benefit, so I do not think it should bless your highly specific use case.

I don't think I was asking to change it to "my personal benefit" (actually my own use case doesn't need FDs > 2, so I'm already happy with that.).



However, it would make sense to briefly mention the closing-high-FDs behavior as motivation. Something along the lines of Geoff's wording in the thread would be more than sufficient:

What I at least would want to avoid is that people might ever come across this issue or the corresponding mailing list thread and assume that this is now the way to portably differentiate between utility non-zero exit status and redirection error, when wouldn't be really the case.

If you don't think it is, fine for me,... we can still make the change here, but should also mention that this cannot be expected to portably allow the above.

If you think it is and if you think that "briefly mentioning the closing-high-FDs behavior as motivation" is enough to also make sure that this is understood by any shell implementer, then I'm all good.
My idea to mention the deeper purpose was merely for the case that would be needed.

geoffclare

2025-03-13 16:12

manager   bugnote:0007115

Last edited: 2025-03-20 14:49

In the Mar 13, 2025 teleconference the following wording was agreed, but the bug is being left open for now for feedback.

On page 2497 line 81097-81118 replace sections 2.7.5 and 2.7.6 with:
2.7.5 Duplicating a File Descriptor
The redirection operators:
[n]<&word
and:
[n]>&word
shall duplicate one input file descriptor or output file descriptor, respectively, from another, or shall close one. If word evaluates to one or more digits, the file descriptor denoted by n, or standard input or standard output, respectively, if n is not specified, shall be made to be a copy of the file descriptor denoted by word; if the digits in word do not represent an already open file descriptor, a redirection error shall result (see Section 2.8.1); if the file descriptor denoted by word represents an open file descriptor that is not open for input or open for output, respectively, a redirection error may result. If word and n evaluate to the same open file descriptor, or if n is not specified and word evaluates to 0 or 1, respectively, no duplication shall occur; if the shell would have closed the file descriptor because it was opened using exec and has a value greater than 2, when the redirection is being performed in a command that will execute a non-built-in utility, the file descriptor shall instead remain open when the utility is executed. If word evaluates to '−', then file descriptor n, or standard input or standard output, respectively, if n is not specified, shall be closed. Attempts to close a file descriptor that is not open shall not constitute an error. If word evaluates to something else, the behavior is unspecified.

(Added after the meeting) A change to merge XRAT C.2.7.5 and C.2.7.6 will also be needed, plus a change to lines 135455-135459 in C.2.9.3.

calestyo

2025-03-13 17:48

reporter   bugnote:0007117

The only two things that came to my mind:
1) Over-pedantic and probably not really needed to take care of:
This text is now for both <& and >& ... and the wording uses "or standard input or standard output" as well as "or if n is not specified and word evaluates to 0 or 1",... so what about things like <1 or >0 ?

2) As far as I understand the wording about the n = word case, wouldn't that also mean that e.g.:
   exec 8<some file; exec 8<8; utility
would now be expected to keep FD 8 open when invoking utility, and not just a construct like:
   exec 8<some file; utility 8<8
?

larryv

2025-03-13 20:20

reporter   bugnote:0007119

0001913:0007117:
so what about things like <1 or >0 ?
The former is covered by section 2.7.1 and redirects stdin from a file literally named "1"; the latter is covered by section 2.7.2 and redirects stdout to a file literally named "0". This report isn't about those forms of redirection.
As far as I understand the wording about the n = word case, wouldn't that also mean that e.g.:
   exec 8<some file; exec 8<8; utility
would now be expected to keep FD 8 open when invoking utility
I don't see how, assuming you actually mean
exec 8<somefile; exec 8<&8; utility
The proposed wording says (emphasis mine):
if the shell would have closed the file descriptor because it was opened using exec and has a value greater than 2, when the redirection is being performed in a command that will execute a non-built-in utility, the file descriptor shall instead remain open when the utility is executed
Your second exec command isn't executing a non-built-in utility.

larryv

2025-03-13 20:43

reporter   bugnote:0007120

0001913:0007112:
What I at least would want to avoid is that people might ever come across this issue or the corresponding mailing list thread and assume that this is now the way to portably differentiate between utility non-zero exit status and redirection error, when wouldn't be really the case.
Why not? Is it because the redirections can't be absolutely guaranteed to succeed, as Geoff mentioned on the mailing list?
If it is a no-op then it can't fail. So the only possible failure case would be in the "remain open" requirement. In practice this will involve calling fcntl() to clear the FD_CLOEXEC flag, which could indeed fail because of something like resource exhaustion, but I don't see that it increases the likelihood of internal shell failure significantly. Any command execution can fail within the shell because of resource exhaustion (e.g. fork() failure) before it gets as far as doing the exec.
https://www.mail-archive.com/austin-group-l@opengroup.org/msg13640.html
If you don't think it is, fine for me,... we can still make the change here, but should also mention that this cannot be expected to portably allow the above.
I don't think the standard should mention your use case at all.
If you think it is and if you think that "briefly mentioning the closing-high-FDs behavior as motivation" is enough to also make sure that this is understood by any shell implementer, then I'm all good.
Alright. The readers I had in mind were application developers who might otherwise assume that n<&n serves no purpose whatsoever.

geoffclare

2025-03-18 12:30

manager   bugnote:0007125

> This text is now for both <& and >& ... and the wording uses "or standard input or standard output" as well as "or if n is not specified and word evaluates to 0 or 1",

The proposed text is careful to use "respectively" where necessary.

> so what about things like <1 or >0 ?

I assume this was meant to be "like <&1 or >&0", in which case the answer is "a redirection error may result" (assuming that 0 is open readonly and 1 is open writeonly).

geoffclare

2025-09-25 11:34

manager   bugnote:0007277

Suggested Interpretation response
------------------------
The standard is unclear on this issue, and no conformance distinction can be made between alternative implementations based on this. This is being referred to the sponsor.

Rationale:
-------------

As the submitter points out, the wording in the standard "one input file descriptor from another" implies that they are two different file descriptors, and so the behavior of n<&n is currently unspecified (implicitly). Likewise for m>&m.

There is an example in XRAT C.2.9.3 Lists (under "Asynchronous AND-OR Lists") which shows that the intention is for <&0 (and therefore also 0<&0) to be a no-op, but since this is non-normative it does not affect the requirements of the standard.


Notes to the Editor (not part of this interpretation):
-------------------------------------------------------

On page 2497 line 81097-81118 replace sections 2.7.5 and 2.7.6 with:
2.7.5 Duplicating a File Descriptor
The redirection operators:
[n]<&word
and:
[n]>&word
shall duplicate one input file descriptor or output file descriptor, respectively, from another, or shall close one.

If word consists of one or more decimal digits which evaluate to a value not equal to n (or 0 or 1, respectively, if n is not specified), the file descriptor denoted by n (or standard input or standard output, respectively, if n is not specified) shall be made to be a copy of the file descriptor denoted by word. If the digits in word do not represent an already open file descriptor, a redirection error shall result (see [xref to 2.8.1]). If the file descriptor denoted by word represents an open file descriptor that is not open for input or open for output, respectively, a redirection error may result.

If word and n evaluate to the same open file descriptor, or if n is not specified and word evaluates to 0 or 1, respectively, no duplication shall occur. If the shell would have closed the file descriptor because it was opened using [xref to exec] and has a value greater than 2, when the redirection is being performed in a command that will execute a non-built-in utility the file descriptor shall instead remain open when the utility is executed.

If word evaluates to '-', then file descriptor n (or standard input or standard output, respectively, if n is not specified) shall be closed. Attempts to close a file descriptor that is not open shall not constitute an error.

If word evaluates to something else, the behavior is unspecified.
and renumber 2.7.7 to 2.7.6.

On page 2506 line 81501 section 2.9.3.1 Asynchronous AND-OR Lists, change:
If, and only if, job control is disabled, the standard input for the subshell in which an asynchronous AND-OR list is executed shall initially be assigned to an open file description that behaves as if /dev/null had been opened for reading only. This initial assignment shall be overridden by any explicit redirection of standard input within the AND-OR list.
to:
If, and only if, job control is disabled, the standard input for the subshell in which an asynchronous AND-OR list is executed shall be assigned to an open file description that behaves as if /dev/null had been opened for reading only, except that:

  • This assignment shall not be performed if there is any explicit redirection of standard input, other than <tt><&0</tt> (or equivalent), within the AND-OR list.

  • This assignment need not be performed if the AND-OR list is within a compound command and either there is any explicit redirection, other than <tt><&0</tt> (or equivalent), of the compound command's standard input or the compound command follows the control operator '|'.

On page 3893 line 135146 section C.2.7.5, change the section heading:
Duplicating an Input File Descriptor
to:
Duplicating a File Descriptor

After page 3893 line 135152 section C.2.7.5, add a paragraph:
If word and n evaluate to the same open file descriptor, the operation is a no-op except in shells which set the close-on-exec flag for file descriptors greater than 2 opened using [xref to exec]. In these shells, a redirection of this form can be used to clear the close-on-exec flag so that the file descriptor will remain open when executing a non-built-in utility. For example:
exec 3<infile 4>outfile
utility 3<&3 4>&4
One use for this feature, together with command and exec, is to differentiate between utility and redirection errors. For example:
( command exec 3<infile 4>outfile || exit 128; utility 3<&3 4>&4 )
case $? in
0)   # success
     ;;
128) # redirection error
     ...
     ;;
*)   # utility error
     ...
     ;;
esac
(This assumes utility is known not to use 128 as an exit status and that the shell does not detect an internal error such as resource exhaustion.)

On page 3893 line 135155-135158, delete section C.2.7.6 Duplicating an Output File Descriptor

On page 3894 line 135159, change section number C.2.7.7 to C.2.7.6

On page 3901 line 135455 section C.2.9.3 Lists, change:
Since the connection of the input to the equivalent of /dev/null is considered to occur before redirections, the following script would produce no output:
exec < /etc/passwd
cat <&0 &
wait
to:
The assignment of standard input from an open file description that behaves like /dev/null is not overridden by an explicit <tt><&0</tt> redirection because this redirection does not perform a duplication and thus has no effect on where standard input comes from. This was the original Korn Shell behavior but was not clearly required by versions of this standard earlier than Issue 8 TC1, although in all those versions there was rationale stating that the following script would produce no output:
exec < /etc/passwd
cat <&0 &
wait

Issue History

Date Modified Username Field Change
2025-03-12 03:33 calestyo New Issue
2025-03-12 07:00 larryv Note Added: 0007111
2025-03-13 02:41 calestyo Note Added: 0007112
2025-03-13 16:12 geoffclare Note Added: 0007115
2025-03-13 17:48 calestyo Note Added: 0007117
2025-03-13 20:20 larryv Note Added: 0007119
2025-03-13 20:43 larryv Note Added: 0007120
2025-03-14 09:44 geoffclare Note Edited: 0007115
2025-03-18 12:30 geoffclare Note Added: 0007125
2025-03-20 14:49 geoffclare Note Edited: 0007115
2025-09-25 11:34 geoffclare Note Added: 0007277