Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001016 [1003.1(2013)/Issue7+TC1] Shell and Utilities Editorial Enhancement Request 2015-12-28 13:52 2020-06-11 11:16
Reporter izabera View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Isabella
Organization ---
User Reference ---
Section 2.7.2
Page Number http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_02 [^]
Line Number last paragraph
Interp Status Approved
Final Accepted Text Note: 0003493
Summary 0001016: race condition with set -C
Description "Output redirection using the '>' format shall fail if the noclobber
option is set (see the description of set -C) and the file named by the
expansion of word exists and is a regular file."

This is the current description, found here:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_02 [^]

The "and is a regular file" part means that the shell has to stat() the
file, and then try to open() it with O_EXCL.
Pro: this doesn't cause an error: echo x > /dev/null
Con: obvious race condition.



One common use of set -C is to implement a simple file locking mechanism,
but this is impossible to do safely. Suppose cmd1 and cmd2 are running
the following code:

set -C
echo "$$" > mylock || exit
...actual work here...


This is what can happen:

         cmd1         |          cmd2         |         attacker
----------------------+-----------------------+-------------------------
                      |                       | create fifo named mylock
stat mylock           |                       |
it's a fifo           |                       |
                      |                       | remove mylock
                      | stat mylock           |
                      | not found             |
                      | open w/ O_EXCL        |
                      | success, lock is mine |
open w/o O_EXCL       |                       |
success, lock is mine |                       |
summon cthulhu        | kill babies           | evil laugh



By skimming the code in various shells, it looks like this is well known
and it was just ignored. I understand that this behaviour is historic
but I don't think the standard should force shells to implement a race.
Desired Action My proposal is to change that part to something along these lines:
"If the noclobber option is set (see the description of set -C), output
redirection using the '>' format shall fail if the file named by the
expansion of word exists and is a regular file. If it's not a regular
file, the results are unspecified."

Con: this may cause an error: echo x > /dev/null
Pro: a shell that implements -C without a race can be POSIX-compliant.
Tags issue8
Attached Files

- Relationships
related to 0001364Applied Issue 8 drafts use of noclobber with files in /tmp 

-  Notes
(0002990)
shware_systems (reporter)
2015-12-31 05:58

Shell output redirection is not a context requiring the use of the O_EXCL flag. That isn't obvious, granted, but is covered by what the standard requires, afaict.

To avoid races the expected usage as I see it is using openat(hroot, file, O_RDWR | O_CREAT, r--r--r--) as first step for the shell doing cmd1 and cmd2, not stat().
If this succeeds the open reference prevents a deletion by attacker in example.
Once opened stat() can be used safely to check if it is a writable regular file and simply abort if so, after closing it, when set -C in effect. If writable and a special file, truncate() can be used on it.
A failure to write a byte when still marked rdonly indicates the file was allocated by a parallel create() so abort after close() should also occur; the other process is the lock owner.
After a successful write(), a dup(); reopen(O_APPEND | O_TRUNC, rw-rw-rw-); close(hdup); also blocks deletion attempts and creator is ready to write to it. The dup() may be superfluous, but ensures any pending parallel ops don't succeed on the implied close of reopen().

That's the gist, anyways. While far from complete, especially for additional device types, I believe it shows the creations can be serialized without races.
(0002991)
jilles (reporter)
2015-12-31 12:52

The shell can avoid the race condition in the scenario with the fifo by checking (using fstat()) whether the opened file is not a regular file, in the case that the original stat() reported that the file was not a regular file and open() was called without O_EXCL (and without O_TRUNC). If the file is a regular file, there was a race condition and the file should be closed and "file already exists" should be reported. If the original stat() reported that the file did not exist, O_EXCL should be used and no additional checking is necessary. This is implemented in FreeBSD sh.

The approach from shware_systems does not work since an open reference does not prevent removal of a file name from a directory. Also, approaches based on file permissions will not work for root since root can open files for writing regardless of permissions.

Approaches based on hard links may help for NFSv2 but will break on non-standard-compliant filesystems that do not support hard links. This is outside the scope of the standard but I think filesystems that do not support hard links are more common nowadays than filesystems that do not support O_EXCL properly.

There is a minor issue in that O_EXCL will not create a file through a symlink, which a normal O_CREAT will (i.e., after ln -s file1 link1; rm -f file1, it is possible to >link1 to create file1). A strict reading of the standard implies that O_EXCL cannot be used but shells almost universally use O_EXCL. Therefore, noclobber > need not work if the target is a symlink to a file that does not exist but could be created.
(0002992)
shware_systems (reporter)
2015-12-31 14:21

It does not prevent entirely a hard link dirent from being modified, yes, though it can be argued it should for unlinks and renames when that dirent was accessed successfully by open(), but the file description and data the dirent references by inode does stay; the same as if two hard links to that data exist. Both need to be unlinked before the data gets removed is my understanding.
That's what makes the *at() interfaces plausible, as they hold open files of type dir, as an extension of type regular from the logical perspective, so they won't be deleted.
(0002993)
nick (manager)
2015-12-31 17:05

(NB ... Description edited to add tags to improve readability)
(0003446)
geoffclare (manager)
2016-10-20 16:40

(Response to Note: 0002991)

The FreeBSD sh method described here does not behave correctly in the case where the initial stat() fails and then open() with O_EXCL fails, and the file that was created in between was not a regular file (e.g. a FIFO). It needs to open the file (without O_CREAT or O_EXCL) and use fstat() to see if it is a regular file.

You then have the problem of what to do if this open() fails.
(0003481)
geoffclare (manager)
2016-10-31 16:23
edited on: 2016-10-31 16:33

In the Oct 27 teleconference we agreed that we should change the standard
to match what modern shells are doing, and I took an action to propose
wording changes. However, while working on the changes, it occurred to
me that there is a better solution: add an O_NOCLOBBER flag to open().
Adding this to the standard would require that it is added to an
implementation first and therefore I present two options below, one which
requires O_NOCLOBBER and one which allows either current behaviour or (the
equivalent of) O_NOCLOBBER but encourages using O_NOCLOBBER.

Rather than choose between these options when resolving this bug, we
should defer the choice until the time comes to apply this bug during
work on the Issue 8 drafts. If, by then, O_NOCLOBBER has been added to
open() by at least one widely used implementation, we should require it
(by applying option 2).

The detailed changes for the two options are below, but first there
are two minor editorial changes that should be made regardless of which
option ends up being applied.

(All page and line numbers are for the 2016 edition.)

On page 2361 line 75346 section 2.7.3, change:

    ... called with the O_APPEND flag.

to:

    ... called with the O_APPEND flag set.

On page 2410 line 77125 section 2.14 set, change:

    Prevent existing files from being overwritten ...

to:

    Prevent existing regular files from being overwritten ...


Option 1: encourage O_NOCLOBBER

On page 2361 line 75337 section 2.7.2, change:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of word to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.
to:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is either a regular file or a symbolic link that resolves to a regular file; it may also fail if the file is a symbolic link that does not resolve to an existing file. Implementations need not perform the check for the existence and type of the file and the creation of the file if it does not exist as a single atomic operation; however, creation of the file shall be performed as if the open() function as defined in the System Interfaces volume of POSIX.1-2008 was called with the O_CREAT and O_EXCL flags set, except that if the file exists and is a symbolic link, the open operation need not fail unless the symbolic link resolves to an existing regular file. Performing this step atomically ensures that creation of lock files using '>' with noclobber set is reliable. Since the overall operation may involve a race condition, redirection failure caused by concurrent creation or removal of files (particularly non-regular files) may be diagnosed misleadingly by the shell when it reports a redirection failure.

In all other cases (noclobber not set, redirection using '>' does not fail for the reasons stated above, or redirection using the ">|" format), output redirection shall cause the file whose name results from the expansion of word to be opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created as an empty file; otherwise, it shall be opened as if the open() function was called with the O_TRUNC flag set.


Cross-volume change to XBD ...

On page 241 line 8132 section <fcntl.h> change FUTURE DIRECTIONS from:
None.
to:
A future version of this standard may add an O_NOCLOBBER file creation flag - see the FUTURE DIRECTIONS section for [xref to open()].


Cross-volume change to XSH ...

On page 1416 line 47098 section open() change FUTURE DIRECTIONS from:
None.
to:
A future version of this standard may add an O_NOCLOBBER flag, specified as follows, for use by shells when the noclobber option is set (see [xref to XRAT C.2.7.2]):

O_NOCLOBBER
If O_CREAT and O_NOCLOBBER are set, open() shall fail if the file exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_NOCLOBBER and O_CREAT set or with O_EXCL and O_CREAT set. If O_NOCLOBBER and O_CREAT are set, and the file exists and is either a non-regular file or a symbolic link that resolves to a non-regular file, the file shall be opened as if neither flag was set. If O_NOCLOBBER and O_CREAT are set, and path names a symbolic link that does not resolve to an existing file, an empty file shall be created such that path resolves to the newly created file. If O_NOCLOBBER is set and O_CREAT is not set, the result is undefined.


Cross-volume change to XRAT ...

On page 3736 line 128235 section C.2.7.2, change:
There is no additional rationale provided for this section.
to:
Earlier versions of this standard did not require redirection using '>' when noclobber is set to perform the file creation step as an atomic operation. Historical shells just called stat() to check if a regular file existed and then called creat(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files. Many shell implementations improved on this by using open() with the O_CREAT and O_EXCL flags set as one step in a multi-step process which still meant that an existing non-regular file (for example /dev/null, /dev/tty, or a FIFO) was opened successfully. However, the methods employed still involved a race condition and could produce misleading diagnostics if there is concurrent creation or removal of files.

An ideal solution would be an O_NOCLOBBER flag for open() which the shell could use in order to perform the entire operation atomically, and implementations are encouraged to adopt this solution, adding the flag as described in the FUTURE DIRECTIONS sections of [xref to XSH open()], and using it in the implementation's POSIX shell and in other shells. Authors of portable shells should make use of <tt>#ifdef O_NOCLOBBER</tt> so that it is used on implementations that provide it.

If O_NOCLOBBER is not used, shells can use one of the following methods:
  1. The "stat first" method.
    • a. Call stat() and if the file exists and is a regular file, the redirection fails. Otherwise:

    • b. Call open() without O_CREAT or O_TRUNC to open an existing file. If the open succeeds, use fstat() to check whether the opened file is a regular file. If it is, close it and fail the redirection. If it is a non-regular file, the redirection succeeds. Otherwise:

    • c. Call open() with O_CREAT|O_EXCL. The redirection succeeds or fails depending on whether the open succeeds or fails.

  2. The "exclusive create first" method.
    • a. Call open() with O_CREAT|O_EXCL. If the open succeeds, the redirection succeeds. If the open fails with [EMFILE] or [ENFILE], use stat() to check whether a regular file exists; if it does, fail the redirection. Otherwise:

    • b. Call open() without O_CREAT or O_TRUNC to open an existing file. If the open succeeds, use fstat() to check whether the opened file is a regular file. If it is, close it and fail the redirection. If it is a non-regular file, the redirection succeeds. If the second open fails, the redirection fails with a diagnostic based on the errno value set by the first open.

    (A minor variation of this method could also be used whereby step 2.b is only done if the open() in step 2.a fails with [EEXIST].)


Method 1 is in widespread use. Method 2 has not been observed exactly as described, although an implementation which omits the stat() in step 2.a has been observed. Without the stat(), this method has a problem in that if a regular file exists but the open() fails with [EMFILE] or [ENFILE] instead of [EEXIST] (which is to be expected if those conditions exist, because detecting [EEXIST] is more expensive), then the shell will give an incorrect diagnostic. (Reporting that no file descriptors are available implies that a non-regular file exists, because the shell tried to open the file and it is not supposed to open an existing regular file.)

A variant of method 1 which omits the initial stat() call has also been observed; this has the same problem with [EMFILE] and [ENFILE]. With the stat(), this misleading diagnostic can also happen, but only if a regular file is created in the timing window between steps 1.a and 1.b, which makes it an allowed case. (The standard allows a misleading diagnostic when there is concurrent creation or removal of files.)

Both methods have cases where a misleading diagnostic is given when a non-regular file is concurrently created or removed. With method 1 it occurs if no file exists at steps 1.a and 1.b, and a non-regular file is created before step 1.c. With method 2 it occurs if a non-regular file exists at step 2.a and is removed before step 2.b. (In both cases, the diagnostic misleadingly implies that a regular file exists).

Both methods differ from historical shell behavior in that the redirection fails if there is an existing symbolic link whose target does not exist, instead of the link's target being created as a regular file. The standard developers consider this to be of less importance than ensuring that the creation of lock files is reliable.



Option 2: require O_NOCLOBBER

On page 2361 line 75337 section 2.7.2, change:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of word to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.
to:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be performed as a single atomic operation, as if the open() function as defined in the System Interfaces volume of POSIX.1-2008 was called with the O_CREAT and O_NOCLOBBER flags set. Performing the operation atomically ensures that creation of lock files using '>' with noclobber set is reliable.

In all other cases (noclobber not set, redirection using '>' does not fail for the reasons stated above, or redirection using the ">|" format), output redirection shall cause the file whose name results from the expansion of word to be opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created as an empty file; otherwise, it shall be opened as if the open() function was called with the O_TRUNC flag set.


Cross-volume change to XBD ...

On page 239 line 8036 section <fcntl.h> add:
O_NOCLOBBER Prevent existing regular files from being overwritten.


Cross-volume changes to XSH ...

On page 1409 line 46805 section open(), change:
... naming the same filename in the same directory with O_EXCL and O_CREAT set.
to:
... naming the same filename in the same directory with O_EXCL and O_CREAT set or with O_NOCLOBBER and O_CREAT set.


On page 1409 line 46809 section open(), add:
O_NOCLOBBER
If O_CREAT and O_NOCLOBBER are set, open() shall fail if the file exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_NOCLOBBER and O_CREAT set or with O_EXCL and O_CREAT set. If O_NOCLOBBER and O_CREAT are set, and the file exists and is either a non-regular file or a symbolic link that resolves to a non-regular file, the file shall be opened as if neither flag was set. If O_NOCLOBBER and O_CREAT are set, and path names a symbolic link that does not resolve to an existing file, an empty file shall be created such that path resolves to the newly created file. If O_NOCLOBBER is set and O_CREAT is not set, the result is undefined.


Cross-volume change to XRAT ...

On page 3736 line 128235 section C.2.7.2, change:
There is no additional rationale provided for this section.
to:
Earlier versions of this standard did not require redirection using '>' when noclobber is set to open the file as if by calling open() with the O_CREAT and O_NOCLOBBER flags set. Historical shells just called stat() to check if a regular file existed and then called creat(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files. Many shell implementations improved on this by using open() with the O_CREAT and O_EXCL flags set as one step in a multi-step process which still meant that an existing non-regular file (for example /dev/null, /dev/tty, or a FIFO) was opened successfully. However, the methods employed still involved a race condition and could produce misleading diagnostics if there is concurrent creation or removal of files. They also differed from historical shell behavior in that the redirection failed if there is an existing symbolic link whose target does not exist, instead of the link's target being created as a regular file. The ideal solution is to use the O_NOCLOBBER flag for open(), which this standard now requires.


(0003485)
rhansen (manager)
2016-11-10 17:33
edited on: 2016-11-11 15:30

We discussed this during the 2016-11-03 and 2016-11-10 teleconferences and came up with the following revised version of the wording from Note: 0003481:

(All page and line numbers are for the 2016 edition.)

Common Changes

On page 2361 line 75346 section 2.7.3, change:
... called with the O_APPEND flag.
to:
... called with the O_APPEND flag set.

On page 2410 line 77125 section 2.14 set, change:
Prevent existing files from being overwritten ...
to:
Prevent existing regular files from being overwritten ...

Option 1: encourage O_NOCLOBBER

On page 2361 line 75337 section 2.7.2, change:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of word to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.
to:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is either a regular file or a symbolic link that resolves to a regular file; it may also fail if the file is a symbolic link that does not resolve to an existing file. The check for existence, file creation, and open operations shall be performed atomically as is done by the open() function as defined in the System Interfaces volume of POSIX.1-2008 when the O_CREAT and O_EXCL flags are set, except that if the file exists and is a symbolic link, the open operation need not fail with [EEXIST] unless the symbolic link resolves to an existing regular file. Performing these operations atomically ensures that the creation of lock files and unique (often temporary) files is reliable. The check for the type of the file need not be performed atomically with the check for existence, file creation, and open operations. If not, there is a potential race condition that may result in a misleading shell diagnostic message when redirection fails. See [xref to rationale] for more details.

In all other cases (noclobber not set, redirection using '>' does not fail for the reasons stated above, or redirection using the ">|" format), output redirection shall cause the file whose name results from the expansion of word to be opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created as an empty file; otherwise, it shall be opened as if the open() function was called with the O_TRUNC flag set.

Cross-volume change to XBD ...

On page 241 line 8132 section <fcntl.h> change FUTURE DIRECTIONS from:
None.
to:
A future version of this standard may add an O_NOCLOBBER file creation flag - see the FUTURE DIRECTIONS section for [xref to open()].

Cross-volume change to XSH ...

On page 1416 line 47098 section open() change FUTURE DIRECTIONS from:
None.
to:
A future version of this standard may add an O_NOCLOBBER flag, specified as follows, for use by shells when the noclobber option is set (see [xref to XRAT C.2.7.2]):

O_NOCLOBBER
If O_CREAT and O_NOCLOBBER are set, open() shall fail if the file exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_NOCLOBBER and O_CREAT set or with O_EXCL and O_CREAT set. If O_NOCLOBBER and O_CREAT are set, and the file exists and is either a non-regular file or a symbolic link that resolves to a non-regular file, the file shall be opened as if neither flag was set. If O_NOCLOBBER and O_CREAT are set, and path names a symbolic link that does not resolve to an existing file, an empty file shall be created such that path resolves to the newly created file. If O_NOCLOBBER is set and O_CREAT is not set, the result is undefined.

Cross-volume change to XRAT ...

On page 3736 line 128235 section C.2.7.2, change:
There is no additional rationale provided for this section.
to:
Earlier versions of this standard did not require redirection using '>' when noclobber is set to perform the file creation step as an atomic operation. Historical shells just called stat() to check if a regular file existed and then called creat(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files. Many shell implementations improved on this by using open() with the O_CREAT and O_EXCL flags set as one step in a multi-step process which still meant that an existing non-regular file (for example /dev/null, /dev/tty, or a FIFO) was opened successfully. However, the methods employed still involved a race condition and could produce misleading diagnostics if there is concurrent creation or removal of files.

An ideal solution would be an O_NOCLOBBER flag for open() which the shell could use in order to perform the entire operation atomically, and implementations are encouraged to adopt this solution, adding the flag as described in the FUTURE DIRECTIONS sections of [xref to XSH open()], and using it in the implementation's POSIX shell and in other shells. Authors of portable shells should make use of <tt>#ifdef O_NOCLOBBER</tt> so that it is used on implementations that provide it.

If O_NOCLOBBER is not used, shells can use one of the following methods:
  1. The "stat first" method.
    • a. Call stat() and if the file exists and is a regular file, the redirection fails. Otherwise:
    • b. Call open() without O_CREAT or O_TRUNC to open an existing file. If the open succeeds, use fstat() to check whether the opened file is a regular file. If it is, close it and fail the redirection. If it is a non-regular file, the redirection succeeds. Otherwise:
    • c. Call open() with O_CREAT|O_EXCL. The redirection succeeds or fails depending on whether the open succeeds or fails.
  2. The "exclusive create first" method.
    • a. Call open() with O_CREAT|O_EXCL. If the open succeeds, the redirection succeeds. If the open fails with [EMFILE] or [ENFILE], use stat() to check whether a regular file exists; if it does, fail the redirection. Otherwise:
    • b. Call open() without O_CREAT or O_TRUNC to open an existing file. If the open succeeds, use fstat() to check whether the opened file is a regular file. If it is, close it and fail the redirection. If it is a non-regular file, the redirection succeeds. If the second open fails, the redirection fails with a diagnostic based on the errno value set by the first open.
    (A minor variation of this method could also be used whereby step 2.b is only done if the open() in step 2.a fails with [EEXIST].)

Method 1 is in widespread use. Method 2 has not been observed exactly as described, although an implementation which omits the stat() in step 2.a has been observed. Without the stat(), this method has a problem in that if a regular file exists but the open() fails with [EMFILE] or [ENFILE] instead of [EEXIST] (which is to be expected if those conditions exist, because detecting [EEXIST] is more expensive), then the shell will give an incorrect diagnostic. (Reporting that no file descriptors are available implies that a non-regular file exists, because the shell tried to open the file and it is not supposed to open an existing regular file.)

A variant of method 1 which omits the initial stat() call has also been observed; this has the same problem with [EMFILE] and [ENFILE]. With the stat(), this misleading diagnostic can also happen, but only if a regular file is created in the timing window between steps 1.a and 1.b, which makes it an allowed case. (The standard allows a misleading diagnostic when there is concurrent creation or removal of files.)

Both methods have cases where a misleading diagnostic is given when a non-regular file is concurrently created or removed. With method 1 it occurs if no file exists at steps 1.a and 1.b, and a non-regular file is created before step 1.c. With method 2 it occurs if a non-regular file exists at step 2.a and is removed before step 2.b. (In both cases, the diagnostic misleadingly implies that a regular file exists).

Both methods differ from historical shell behavior in that the redirection fails if there is an existing symbolic link whose target does not exist, instead of the link's target being created as a regular file. The standard developers consider this to be of less importance than ensuring that the creation of lock files is reliable.

Option 2: require O_NOCLOBBER

On page 2361 line 75337 section 2.7.2, change:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is a regular file. Otherwise, redirection using the '>' or ">|" formats shall cause the file whose name results from the expansion of word to be created and opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created; otherwise, it shall be truncated to be an empty file after being opened.
to:
Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be performed as a single atomic operation, as if the open() function as defined in the System Interfaces volume of POSIX.1-2008 was called with the O_CREAT and O_NOCLOBBER flags set. Performing the operation atomically ensures that creation of lock files and unique (often temporary) files using '>' with noclobber set is reliable.

In all other cases (noclobber not set, redirection using '>' does not fail for the reasons stated above, or redirection using the ">|" format), output redirection shall cause the file whose name results from the expansion of word to be opened for output on the designated file descriptor, or standard output if none is specified. If the file does not exist, it shall be created as an empty file; otherwise, it shall be opened as if the open() function was called with the O_TRUNC flag set.

Cross-volume change to XBD ...

On page 239 line 8036 section <fcntl.h> add:
O_NOCLOBBER Prevent existing regular files from being overwritten.

Cross-volume changes to XSH ...

On page 1409 line 46805 section open(), change:
... naming the same filename in the same directory with O_EXCL and O_CREAT set.
to:
... naming the same filename in the same directory with O_EXCL and O_CREAT set or with O_NOCLOBBER and O_CREAT set.

On page 1409 line 46809 section open(), add:
O_NOCLOBBER
If O_CREAT and O_NOCLOBBER are set, open() shall fail if the file exists and is either a regular file or a symbolic link that resolves to a regular file. The check for the existence and type of the file and the creation of the file if it does not exist shall be atomic with respect to other threads executing open() naming the same filename in the same directory with O_NOCLOBBER and O_CREAT set or with O_EXCL and O_CREAT set. If O_NOCLOBBER and O_CREAT are set, and the file exists and is either a non-regular file or a symbolic link that resolves to a non-regular file, the file shall be opened as if neither flag was set. If O_NOCLOBBER and O_CREAT are set, and path names a symbolic link that does not resolve to an existing file, an empty file shall be created such that path resolves to the newly created file. If O_NOCLOBBER is set and O_CREAT is not set, the result is undefined.

Cross-volume change to XRAT ...

On page 3736 line 128235 section C.2.7.2, change:
There is no additional rationale provided for this section.
to:
Earlier versions of this standard did not require redirection using '>' when noclobber is set to open the file as if by calling open() with the O_CREAT and O_NOCLOBBER flags set. Historical shells just called stat() to check if a regular file existed and then called creat(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files. Many shell implementations improved on this by using open() with the O_CREAT and O_EXCL flags set as one step in a multi-step process which still meant that an existing non-regular file (for example /dev/null, /dev/tty, or a FIFO) was opened successfully. However, the methods employed still involved a race condition and could produce misleading diagnostics if there is concurrent creation or removal of files. They also differed from historical shell behavior in that the redirection failed if there is an existing symbolic link whose target does not exist, instead of the link's target being created as a regular file. The ideal solution is to use the O_NOCLOBBER flag for open(), which this standard now requires.


(0003486)
stephane (reporter)
2016-11-11 21:26
edited on: 2016-11-12 13:43

The part about adding O_NOCLOBBER is unreasonable in my opinion, that would be adding a feature to the kernel (and probably difficult to implement without for instance modifying network file system protocols) just to work around a short-coming in a misfeature of one application (sh).

I can't think of any other reason why one may want to use such a O_NOCLOBBER (O_EXCL only if the file is regular) flag.

"set -C" is the solution the sh developers came up with to allow users to redirect without clobbering files, and then realised it wasn't convenient when redirecting to /dev/null or named pipes. The problem here is not as much that a O_NOCLOBBER is missing to make that reliable, but that "set -C" feature in the first place which is using a global setting where users only need to affect the behaviour of *some* redirections.

Most sh features that make use of global settings (IFS, CDPATH, noglob, errexit, nounset, even LC_*) are a problem and source of headaches to write reusable code, especially considering that POSIX sh has no local (let alone private) scope for those.

Here, it could make a lot more sense to have a dedicated redirection operator that uses O_EXCL than having a O_CLOBBER flag so that only some redirections only are affected by a global set -C setting.

Something like:

    cmd (excl)> newfile 2> /dev/null

To allow more flags (O_NOFOLLOW, O_CLOEXEC and the ability to turn on/off some of the flags for <>, >>, > redirections are also badly missing).

Like above, if we wanted to be pedantic, we'd want to write:

   cmd (excl)> newfile 2(nocreate)> /dev/null

to avoid creating a /dev/null regular file if it wasn't there in the first place.

There are far more important race conditions to address IMO (even at kernel level like a rename() that doesn't clobber the destination (see renameat2 on Linux)) than this one. Like the missing mv/cp/ln -T/-n, the find -execdir, mktemp, cd with nofollow...

(0003487)
shware_systems (reporter)
2016-11-14 07:01

It isn't just one application; sh is the focus as where an example of the desired behavior has defined semantics. Collateral effects on any app making use of system() to invoke "sh -c command-pipe-with-redirects", or utilities like awk and make with shell-like behavior also potentially can have this sort of race condition. This potential makes this more a defect of open(), as where atomicity can be guaranteed, than simply the shell, where it can't, as a feature many apps might want access to, but no implementation currently provides.

While O_NOCLOBBER, as a design proposal, and being more explicit about the atomicity that can be required of noclobber, can be seen as more band-aid than solution to the general problem, it involves adaptations of known techniques that makes it viable (if barely) as a Future Direction recommendation. Should it be more general was brought up as nice in theory, and quickly rejected as being too much invention to even make an option. Something like that would involve new techniques, as you begin to touch upon, that don't have any sort of implementation yet. Just adding the hooks so new file types can advertise what degree of guarding against race conditions they need (are they more like regular files than pipes, or neither, sort of thing) I don't see as trivial.

Last, afaik for many settings there is local scope, if the application where local behavior is needed is run in a subshell. The parent environment is unaffected for exported variable changes made in that subshell, at least.
(0003488)
geoffclare (manager)
2016-11-14 09:30

> I can't think of any other reason why one may want to use such a O_NOCLOBBER (O_EXCL only if the file is regular) flag.

My feeling is quite the opposite. I suspect there are certain types of application that currently use O_EXCL to ensure an existing output file will not be overwritten, but the author would have used O_NOCLOBBER instead if it had existed.
(0003489)
stephane (reporter)
2016-11-14 10:10

Re: Note: 0003488

Geoff, would you mind giving an example.

From my experience, O_EXCL/set -C is used in contexts like:

- exclusive access to a file: special treatment of non-regular files defeats it if a non-regular file might be there.
- security: make sure we a new file in the right place in the presence of potential attackers putting symlinks in place of the target file. Again, the special treatment of non-regular files defeats it.
- creating temporary files. As seen with Martijn's attempt to come up with a POSIX mktemp sh function, that also gets in the way.

In all those cases, using a proper O_EXCL interface like with zsh, socat or GNU dd would not have those issues.
(0003490)
stephane (reporter)
2016-11-14 10:28

Re: Note: 0003487

awk, make don't have shell-like behaviour in that instance. They don't have a global noclobber setting that needs to be worked around for non-regular files. They call a shell in their system().

Note that while POSIX has no proper interface to O_EXCL in the shell and utilities, there are a few in the wild.

For instance, some equivalents of my suggested "cmd (excl)> file" could be:

cmd | GNU-dd status=none bs=64k conv=excl of=file

cmd | socat -u - open:file,creat,excl

(those not ideal because of the pipe and extra process (and subshell environment)).

zsh's (also works in sh mode):

zmodload zsh/system
(sysopen -u 1 -w -o excl file && cmd)

Not as good as a (excl)> file redirection operator as you need either to use a subshell for the local scope of the redirection like above, or do the fd saving and restoring by hand (with the current limitation that zsh has no API to dup a fd with O_CLOEXEC yet).

> Last, afaik for many settings there is local scope, if the application where local behavior is needed is run in a subshell. The parent environment is unaffected for exported variable changes made in that subshell, at least.

subshell means local context *for everything* which is not we want in many situations in my context of "reusable code". For instance, how would I do a:

excl_open() {
  command eval "exec $1(excl)> \"\$2\""
}

excl_open 3 file

with "set -C"?
(0003491)
geoffclare (manager)
2016-11-14 10:47

Re: Note: 0003489

Your list does not include what I believe is the reason set -C was invented: avoiding accidentally overwriting an existing regular file when executing a command that is intended to create a new regular file.

Setting -C for the shell provides the wanted protection when using redirection to create files, but there are applications which take an output file name on the command line instead of writing the output to stdout. In order to provide equivalent protection for these files, the application creating the file needs to mimic what the shell does for set -C.

That's the obvious case, but there may well be others. Another I had in mind but haven't fully thought through is a "save file" dialog in interactive applications.
(0003492)
shware_systems (reporter)
2016-11-14 15:55
edited on: 2016-11-14 16:17

Re: Note: 3490
"awk, make don't have shell-like behaviour in that instance. They don't have a global noclobber setting that needs to be worked around for non-regular files. They call a shell in their system()."

and those calls aren't prohibited from allowing "set -C; rest; of; script" to be passed as argument. It is the behavior of the application scripts that it is a factor for, not so much the engine executable.

"excl_open() {
  command eval "exec $1(excl)> \"\$2\""
}"
I think should be more like

excl_pipe() {
  ( set +C; command eval "exec $1(excl)>\"\$2\""; $( $1>&$1 $3) );
}
called as
excl_pipe 3 file "pipe statements";
# set -C/+C state reverts when subshell exits, exit code of pipe reflected as result of subshell and then of function.

to be consistent with what I'm thinking. The "pipe statements" can include resetting IFS or CDPATH as well, before the first pipe command, and these will be in effect just for the duration of the pipe.

(0003493)
geoffclare (manager)
2016-11-17 16:18

Interpretation response
------------------------
The standard states the requirements for redirection with '>' when set -C is in effect, and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor.

Rationale:
-------------
Historical shells just called stat() to check if a regular file existed and then called creat(). The operation thus involved a race condition which meant that it could not be used for reliable creation of lock files or unique (often temporary) files. Modern shell implementations are instead using open() with the O_CREAT and O_EXCL flags set as one step in a multi-step process, although this makes their behavior differ from the behavior required by the standard in certain respects.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------
Make the changes in Note: 0003485, choosing between option 1 and option 2 during work on the Issue 8 drafts.
(0003518)
ajosey (manager)
2016-12-15 18:09

Interpretation proposed: 15 Dec 2016
(0003554)
ajosey (manager)
2017-01-18 15:25

Interpretation Approved: 18 Jan 2017
(0003655)
McDutchie (reporter)
2017-03-29 00:58

"Output redirection using the '>' format shall fail if the noclobber option is set (see the description of set -C) and the file named by the expansion of word exists and is either a regular file or a symbolic link that resolves to a regular file." [...] "Performing these operations atomically ensures that the creation of lock files and unique (often temporary) files is reliable."

If noclobber doesn't block on writing to an existing FIFO or device file, then an attacker on another user account, if they guess the name before the file is created, can subvert the creation of a unique temporary file in a shared directory (e.g. /tmp) by putting a FIFO or device file in the way. With a FIFO they could even steal some data. So the statement about reliable unique temporary files is accurate only for private directories. You may want to add this for clarity.
(0004887)
geoffclare (manager)
2020-06-11 11:16

I have applied Option 1 but included "Notes to Reviewers" about the potential for changing to Option 2.

- Issue History
Date Modified Username Field Change
2015-12-28 13:52 izabera New Issue
2015-12-28 13:52 izabera Name => Isabella
2015-12-28 13:52 izabera Organization => ---
2015-12-28 13:52 izabera User Reference => ---
2015-12-28 13:52 izabera Section => 2.7.2
2015-12-28 13:52 izabera Page Number => http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_02 [^]
2015-12-28 13:52 izabera Line Number => last paragraph
2015-12-31 05:58 shware_systems Note Added: 0002990
2015-12-31 12:52 jilles Note Added: 0002991
2015-12-31 14:21 shware_systems Note Added: 0002992
2015-12-31 17:05 nick Interp Status => ---
2015-12-31 17:05 nick Note Added: 0002993
2015-12-31 17:05 nick Description Updated
2016-10-20 16:40 geoffclare Note Added: 0003446
2016-10-31 16:23 geoffclare Note Added: 0003481
2016-10-31 16:28 geoffclare Note Edited: 0003481
2016-10-31 16:33 geoffclare Note Edited: 0003481
2016-11-10 17:33 rhansen Note Added: 0003485
2016-11-10 17:36 rhansen Note Edited: 0003485
2016-11-10 17:37 rhansen Note Edited: 0003485
2016-11-11 15:24 rhansen Note Edited: 0003485
2016-11-11 15:25 rhansen Note Edited: 0003485
2016-11-11 15:27 rhansen Note Edited: 0003485
2016-11-11 15:29 rhansen Note Edited: 0003485
2016-11-11 15:29 rhansen Note Edited: 0003485
2016-11-11 15:30 rhansen Note Edited: 0003485
2016-11-11 21:26 stephane Note Added: 0003486
2016-11-12 13:43 stephane Note Edited: 0003486
2016-11-14 07:01 shware_systems Note Added: 0003487
2016-11-14 09:30 geoffclare Note Added: 0003488
2016-11-14 10:10 stephane Note Added: 0003489
2016-11-14 10:28 stephane Note Added: 0003490
2016-11-14 10:47 geoffclare Note Added: 0003491
2016-11-14 15:55 shware_systems Note Added: 0003492
2016-11-14 16:17 shware_systems Note Edited: 0003492
2016-11-17 16:18 geoffclare Note Added: 0003493
2016-11-17 16:20 geoffclare Interp Status --- => Pending
2016-11-17 16:20 geoffclare Final Accepted Text => Note: 0004393
2016-11-17 16:20 geoffclare Status New => Interpretation Required
2016-11-17 16:20 geoffclare Resolution Open => Accepted As Marked
2016-11-17 16:20 geoffclare Final Accepted Text Note: 0004393 => Note: 0003493
2016-11-17 16:20 geoffclare Tag Attached: issue8
2016-12-15 18:09 ajosey Interp Status Pending => Proposed
2016-12-15 18:09 ajosey Note Added: 0003518
2017-01-18 15:25 ajosey Interp Status Proposed => Approved
2017-01-18 15:25 ajosey Note Added: 0003554
2017-03-29 00:58 McDutchie Note Added: 0003655
2020-06-11 11:16 geoffclare Note Added: 0004887
2020-06-11 11:16 geoffclare Status Interpretation Required => Applied
2020-07-06 10:46 geoffclare Relationship added related to 0001364


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker