View Issue Details

IDProjectCategoryView StatusLast Update
00019131003.1(2024)/Issue8Shell and Utilitiespublic2025-03-20 14:49
Reportercalestyo Assigned To 
PrioritynormalSeverityEditorialTypeEnhancement Request
Status NewResolutionOpen 
NameChristoph Anton Mitterer
Organization
User ReferenceShell & Utilities
Section2.7.5, 2.7.6
Page Number2497
Line Number81097-81118
Interp Status
Final Accepted Text
Summary0001913: clarify/define the meaning of n<&n and m>&m redirections
DescriptionHey.

This originated from a thread that started at
https://collaboration.opengroup.org/operational/mailarch.php?soph=N&action=show&archive=austin-group-l&num=38114&limit=100&offset=100&sid=

In short:
The motivation was whether it's possible to portably differentiate whether a non-zero exit status originated from a utility or failed redirection on that.
The idea was that, with the clarifications from #1879 and assuming that there is at least on exit status which a utility is known not to use, a construct like the following:
   ( command exec <some redirections> || exit 125; utility )
where the subshell is merely to clean up any redirections, can be used to differentiate between a redirection error (which above would be indicated by 125) or a non-zero exit status from the utility.

For file descriptors less than or equal to 2, this should already be guaranteed to work portably, as POSIX demands those to be passed on to the utility.
For FDs greater than two, this is no longer guaranteed, though.

An idea was brought up by Harald van Dijk, that n<&n and m>&m could be used e.g.:
   ( command exec <some redirections> || exit 125; utility n<&n... m>&m... )
in order to "manually" pass on on any such FDs.

Questions remained:
a) Does that behaviour even follow from POSIX (in an obvious way where it's really clear that shells should behave like this, not just some wobbly way)
b) Does it even solve the original problem, or could e.g. such a n<&n respectively m>&m fail itself (not e.g. because a file doesn't exist, but because of something like resource exhaustion, etc.)

When the original thread was continued a bit later at:
https://collaboration.opengroup.org/operational/mailarch.php?soph=N&action=show&archive=austin-group-l&num=38279&limit=100&offset=0&sid=
it apparently turned out that n<&n and m>&m redirections are not defined because POSIX uses the wordings:
"The redirection operator: [n]<&word shall duplicate one input file descriptor from another"
respectively
"The redirection operator: [n]>&word shall duplicate one output file descriptor from another"
i.e. one from another, implying the two must not be the same, as Geoff Clare pointed out.
Desired Action1. It shall be clarified whether or not the n<&n and m>&m forms are covered by the current wording of the standard.
(One could perhaps also interpret the "one from another" as the "one" being the FD from the utility's PoV, and the "another" being the FD of the same number from the shell's PoV.)

2. If possible - i.e. no conflicting behaviour of shell implementations - it would be nice if the definition could be change in such a way, that it actually allows portably for the goals described above.
I have no strong opinion one how this definition should look like, one suggestion on the list was that implementations that the n<&n and m>&m forms shalll be no-ops for implementations which *do* pass on FDs > 2, and for those that don't, it shall have the meaning of explicitly passing the given FD on.

It would be even nicer, if the standard mention in a sentence that this is explicitly meant to be usable for the purpose of differentiating between utility and redirection errors.

Thanks,
Chris.
TagsNo tags attached.

Activities

larryv

2025-03-12 07:00

reporter   bugnote:0007111

Desired Action:
It would be even nicer, if the standard mention in a sentence that this is explicitly meant to be usable for the purpose of differentiating between utility and redirection errors.
The standard does not exist for your personal benefit, so I do not think it should bless your highly specific use case. However, it would make sense to briefly mention the closing-high-FDs behavior as motivation. Something along the lines of Geoff's wording in the thread would be more than sufficient:
So I would support updating the standard to require that n<&n and n>&n are always a no-op if fd n is open, except that if the shell normally closes fds > 2, that were opened with exec, when it executes a non-built-in utility, then applying n<&n or n>&n to such commands causes fd n to remain open.

calestyo

2025-03-13 02:41

reporter   bugnote:0007112

The standard does not exist for your personal benefit, so I do not think it should bless your highly specific use case.

I don't think I was asking to change it to "my personal benefit" (actually my own use case doesn't need FDs > 2, so I'm already happy with that.).



However, it would make sense to briefly mention the closing-high-FDs behavior as motivation. Something along the lines of Geoff's wording in the thread would be more than sufficient:

What I at least would want to avoid is that people might ever come across this issue or the corresponding mailing list thread and assume that this is now the way to portably differentiate between utility non-zero exit status and redirection error, when wouldn't be really the case.

If you don't think it is, fine for me,... we can still make the change here, but should also mention that this cannot be expected to portably allow the above.

If you think it is and if you think that "briefly mentioning the closing-high-FDs behavior as motivation" is enough to also make sure that this is understood by any shell implementer, then I'm all good.
My idea to mention the deeper purpose was merely for the case that would be needed.

geoffclare

2025-03-13 16:12

manager   bugnote:0007115

Last edited: 2025-03-20 14:49

In the Mar 13, 2025 teleconference the following wording was agreed, but the bug is being left open for now for feedback.

On page 2497 line 81097-81118 replace sections 2.7.5 and 2.7.6 with:
2.7.5 Duplicating a File Descriptor
The redirection operators:
[n]<&word
and:
[n]>&word
shall duplicate one input file descriptor or output file descriptor, respectively, from another, or shall close one. If word evaluates to one or more digits, the file descriptor denoted by n, or standard input or standard output, respectively, if n is not specified, shall be made to be a copy of the file descriptor denoted by word; if the digits in word do not represent an already open file descriptor, a redirection error shall result (see Section 2.8.1); if the file descriptor denoted by word represents an open file descriptor that is not open for input or open for output, respectively, a redirection error may result. If word and n evaluate to the same open file descriptor, or if n is not specified and word evaluates to 0 or 1, respectively, no duplication shall occur; if the shell would have closed the file descriptor because it was opened using exec and has a value greater than 2, when the redirection is being performed in a command that will execute a non-built-in utility, the file descriptor shall instead remain open when the utility is executed. If word evaluates to '−', then file descriptor n, or standard input or standard output, respectively, if n is not specified, shall be closed. Attempts to close a file descriptor that is not open shall not constitute an error. If word evaluates to something else, the behavior is unspecified.

(Added after the meeting) A change to merge XRAT C.2.7.5 and C.2.7.6 will also be needed, plus a change to lines 135455-135459 in C.2.9.3.

calestyo

2025-03-13 17:48

reporter   bugnote:0007117

The only two things that came to my mind:
1) Over-pedantic and probably not really needed to take care of:
This text is now for both <& and >& ... and the wording uses "or standard input or standard output" as well as "or if n is not specified and word evaluates to 0 or 1",... so what about things like <1 or >0 ?

2) As far as I understand the wording about the n = word case, wouldn't that also mean that e.g.:
   exec 8<some file; exec 8<8; utility
would now be expected to keep FD 8 open when invoking utility, and not just a construct like:
   exec 8<some file; utility 8<8
?

larryv

2025-03-13 20:20

reporter   bugnote:0007119

0001913:0007117:
so what about things like <1 or >0 ?
The former is covered by section 2.7.1 and redirects stdin from a file literally named "1"; the latter is covered by section 2.7.2 and redirects stdout to a file literally named "0". This report isn't about those forms of redirection.
As far as I understand the wording about the n = word case, wouldn't that also mean that e.g.:
   exec 8<some file; exec 8<8; utility
would now be expected to keep FD 8 open when invoking utility
I don't see how, assuming you actually mean
exec 8<somefile; exec 8<&8; utility
The proposed wording says (emphasis mine):
if the shell would have closed the file descriptor because it was opened using exec and has a value greater than 2, when the redirection is being performed in a command that will execute a non-built-in utility, the file descriptor shall instead remain open when the utility is executed
Your second exec command isn't executing a non-built-in utility.

larryv

2025-03-13 20:43

reporter   bugnote:0007120

0001913:0007112:
What I at least would want to avoid is that people might ever come across this issue or the corresponding mailing list thread and assume that this is now the way to portably differentiate between utility non-zero exit status and redirection error, when wouldn't be really the case.
Why not? Is it because the redirections can't be absolutely guaranteed to succeed, as Geoff mentioned on the mailing list?
If it is a no-op then it can't fail. So the only possible failure case would be in the "remain open" requirement. In practice this will involve calling fcntl() to clear the FD_CLOEXEC flag, which could indeed fail because of something like resource exhaustion, but I don't see that it increases the likelihood of internal shell failure significantly. Any command execution can fail within the shell because of resource exhaustion (e.g. fork() failure) before it gets as far as doing the exec.
https://www.mail-archive.com/austin-group-l@opengroup.org/msg13640.html
If you don't think it is, fine for me,... we can still make the change here, but should also mention that this cannot be expected to portably allow the above.
I don't think the standard should mention your use case at all.
If you think it is and if you think that "briefly mentioning the closing-high-FDs behavior as motivation" is enough to also make sure that this is understood by any shell implementer, then I'm all good.
Alright. The readers I had in mind were application developers who might otherwise assume that n<&n serves no purpose whatsoever.

geoffclare

2025-03-18 12:30

manager   bugnote:0007125

> This text is now for both <& and >& ... and the wording uses "or standard input or standard output" as well as "or if n is not specified and word evaluates to 0 or 1",

The proposed text is careful to use "respectively" where necessary.

> so what about things like <1 or >0 ?

I assume this was meant to be "like <&1 or >&0", in which case the answer is "a redirection error may result" (assuming that 0 is open readonly and 1 is open writeonly).

Issue History

Date Modified Username Field Change
2025-03-12 03:33 calestyo New Issue
2025-03-12 07:00 larryv Note Added: 0007111
2025-03-13 02:41 calestyo Note Added: 0007112
2025-03-13 16:12 geoffclare Note Added: 0007115
2025-03-13 17:48 calestyo Note Added: 0007117
2025-03-13 20:20 larryv Note Added: 0007119
2025-03-13 20:43 larryv Note Added: 0007120
2025-03-14 09:44 geoffclare Note Edited: 0007115
2025-03-18 12:30 geoffclare Note Added: 0007125
2025-03-20 14:49 geoffclare Note Edited: 0007115