Austin Group Defect Tracker - ISSUES

0000217: The "single subshell" rule for $(( is not sufficient

In the description of command substitution the standard says:

If the command substitution consists of a single subshell, such as:

$( (command) )

a conforming application shall separate the "$(" and '(' into two
tokens (that is, separate them with white space). This is required
to avoid any ambiguities with arithmetic expansion.

In a discussion on the mailing list in December 2009, Philip Guenther
demonstrated that there are ambiguous cases other than the "single
subshell" case. He gave the example:

cat=1; EOH=3; echo $(( cat < + ( (
EOH
) && ( cat < ) ) + 1 +
EOH
))

Current shell implementations treat this as an arithmetic expansion.

Thus the requirement on applications quoted above should be extended
to cover all ambiguous cases, not just the "single subshell" case.

The same requirement should also apply to nested subshells, so that
shells which implement the "(( arithmetic expression ))" extension
can apply the same disambiguation rules consistently to $((...)) and
((...)).

0000225: ENXIO and 'may fail'

The standard is inconsistent on when ENXIO, in relation to a request outside the capabilities of a device, is required rather than optional. This text is associated with actions related to read() and write(). After auditing all uses of ENXIO associated with this phrase, the only uses that were marked as 'shall fail' are fseek/fseeko, fsetpos, and pread. The case of pread is already being dealt with in 0000218.

From an application standpoint, the application must already be prepared to deal with an optional error. From an implementation standpoint, relaxing ENXIO to optional has no impact if the situation was already being detected, and eases compliance if it was not. From a conformance test standpoint, it is not easy to write a test that reliably and intentionally requests an action beyond the capability of a device, so there is nothing lost by relaxing the requirement.

Other uses of ENXIO, such as open() failing if an underlying device is not present, should remain as 'shall fail'.

0000224: awk -F and FS

There are a couple of problems with the description of the awk -F option:

-F ERE
Define the input field separator to be the extended regular
expression ERE, before any input is read

1. By a strict reading, the ERE option-argument is used directly as
an extended regular expression, but in practice the option-argument
is handled as if it were a string literal to be assigned to the FS
variable. This makes a difference to the treatment of backslash
characters:

$ awk -F '\\\\' 'BEGIN { print FS }'
\\

Here the ERE used as input field separator is \\ not \\\\. It is
the same as for:

$ awk 'BEGIN { FS="\\\\"; print FS }'
\\

2. The input field separator (and consequently the FS variable) should
be set before execution of any BEGIN actions, not just before any
input is read, so that applications can use FS within BEGIN actions.
E.g.:

BEGIN { OFS=FS }

0000219: Missing statement about pwrite() on file incapable of seeking

The read() page has a statement at the end of the DESCRIPTION:

"An attempt to perform a pread() on a file that is incapable of
seeking shall result in an error."

There is no equivalent statement for pwrite() on the write() page.

0000223: mq_* routines use char * for data buffer

The header provides the following functions:

ssize_t mq_receive(mqd_t, char *, size_t, unsigned *);
int mq_send(mqd_t, const char *, size_t, unsigned);
int mq_setattr(mqd_t, const struct mq_attr *restrict,
struct mq_attr *restrict);
ssize_t mq_timedreceive(mqd_t, char *restrict, size_t,
unsigned *restrict, const struct timespec *restrict);
int mq_timedsend(mqd_t, const char *, size_t, unsigned,
const struct timespec *);

For some reason I've never understood why these routines use char *'s for data buffers, while functions like write() and send() use a void *. Message queues don't use strings. Message queues are binary safe.

0000222: C.2.6.2 needs a reference to C.2.2.3

Section C.2.2.3 Double-Quotes contains a lot of rationale about
parameter expansion (within double-quotes). Readers looking for
rationale about parameter expansion go to C.2.6.2, so it would
be useful if that section had a reference to C.2.2.3.

0000218: Inconsistencies in pread() and read() errors

The error conditions on the read() page associated with pipes/FIFOs
and sockets have the same inconsistencies between pread() and read(),
and between pread() and lseek(), as were reported for pwrite() in
0000215.

In addition, there are duplicate EOVERFLOW and ENXIO errors for
pread(), one in the "these functions" part and one specific to
pread(). The suggested changes remove the pread()-specific one
in each case, as the other one seems more correct.

0000221: poor wording about even quotes in double quoted parameter expansion

Section 2.2.3 Double-Quotes says:

Within the string of characters from an enclosed "${" to the
matching '}', an even number of unescaped double-quotes or
single-quotes, if any, shall occur.

This wording is poor for two reasons:

1. It is a requirement on the application, but is not worded as such.

2. It is ambiguous, in that it could be taken to mean the total
number of unescaped single-quotes and double-quotes, taken together,
must be even. The intention is for the requirement to apply
separately to single-quotes and double-quotes.

0000144: Standard lacks a (possibly XSI) interface to associate a session to a TTY; tcsetsid()

According to the General Terminal Interface chapter, it is implementation defined whether a session associates to a TTY when O_NOCTTY is not passed to open():

The controlling terminal for a session is allocated by the session leader in an implementation-defined manner. If a session leader has no controlling terminal, and opens a terminal device file that is not already associated with a session without using the O_NOCTTY option (see open()), it is implementation-defined whether the terminal becomes the controlling terminal of the session leader.

Unfortunately, the standard does not specify a way to associate them when the implementation does not do this by default (i.e. the BSDs). This means it's practically impossible to implement a network login service or terminal emulator only using the interfaces described in POSIX.

0000183: Broken link for break on HTML version of standard, for break in section C2.4

The section containing

The restriction on ending a name with a is to allow future implementations that support named labels for flow control; see the RATIONALE for the break built-in utility.

contains a broken link for the word break.

0000180: code formatting

I was looking up for loops (to see which expansions are done)
and I noticed the formatting of multi-line code snippets
was missing some newlines:

The format for the for loop is as follows:

for name [ in [word ... ]]do
compound-listdone

The same can be observed for the case, while, and other descriptions
of formats in that section.

0000175: HTML problem in XCU7 sleep example

(Originally reported by Konrad Schwarz on 12 Jan 2009)

> while true
> do
>     command    sleep 37
> done
>
> should be formatted as
>
> while true
> do
>     command
>     sleep 37
> done

0000188: thread-safe getenv()

The standard says that getenv() need not be thread-safe. Since there
is no getenv_r() function, this makes obtaining environment variable
values somewhat inconvenient in multi-threaded conforming applications.
I imagine there are a large number of applications that don't bother
to be careful with getenv() and just assume it is thread-safe.
It probably is thread-safe on most, if not all, implementations that
support POSIX threads.

The reason getenv() is not required to be thread-safe is stated in
the rationale:

"The getenv() function is inherently not thread-safe because it
returns [sic] a value pointing to static data."

(I added "[sic]" because it should say "may return" not "returns".)

I think it likely that the only POSIX implementations where getenv()
copies the value to an internal buffer are UNIX implementations where
the UNIX APIs have been added to an O/S that was not originally UNIX.
It's possible none of these implement POSIX threads, or if they do
then they may have made getenv() thread-safe by using per-thread
internal buffers instead of a single buffer.

Therefore I think the standard should require getenv() to be
thread-safe. If there are any implementations which copy to an
internal buffer, and either have POSIX threads or want to add POSIX
threads in the future, they can conform simply by using per-thread
internal buffers.

0000187: Specification of c99 -I option should introduce the concept of system directories

POSIX says: "For headers whose names are enclosed in angle brackets ("<>"), the header shall be searched for only in directories named in -I options and then in the usual places."

However some implementations such as GCC (e.g. used via c99 under Linux) do not behave in that way, and developers regard that as a defect in POSIX. See e.g.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40442 [^]

which has been closed with the comment "Not a GCC bug, the POSIX list generally agreed the effects of reordering system directories should be unspecified or undefined."

0000185: setenv failure case

There is a typo in the EINVAL error for setenv.

Meanwhile, many existing BSD implementations crash or give EFAULT, rather than failing with EINVAL, when invoking setenv(NULL,"",0). Therefore, use of a null pointer is already non-portable in practice, and removing the requirement for a null pointer check allows for a faster implementation. Omission of a requirement for a check for a null pointer could invalidate existing applications, but the likelihood of a compliant application expecting setenv(NULL,"",0) to fail with EINVAL is minimal (given that such an application could crash when ported to BSD, and that wasting the time on a call known to not update environ seems pointless). On the other hand, omitting the requirement would allow more existing implementations to be compliant, without penalizing any existing implementation that performs the null pointer check, since the standard already has an overriding statement that if not explicitly mentioned otherwise, use of a null pointer gives unspecified behavior. Therefore, this proposal includes a change to relax the requirement regarding a null pointer check.

This is orthogonal to issue 167, although it touches the same interfaces.

Meanwhile, note that neither putenv("=") nor getenv("") are required to fail with EINVAL, although in practice, behavior varies widely on whether environ can successfully be managed if it contains an entry whose first character is '='. The desired action makes no attempt to change this.

0000173: Executing programs with "bad" file descriptors 0, 1 or 2

XSH7 section 2.5 states:

"At program start-up, three streams are predefined and need not
be opened explicitly: standard input (for reading conventional
input), standard output (for writing conventional output), and
standard error (for writing diagnostic output)."

The standard also provides the means to execute programs with fd 0
not open for reading, or fd 1 or 2 not open for writing. However,
it does not fully address the issue of what happens to stdin, stdout
and stderr when a conforming application is executed in such a way.
The security aspects of the issue were addressed in SUSv3 TC1, which
resulted in implementations being allowed to automatically open these
fds when executing set-user-ID or set-group-ID programs (although it
only says this for exec, not for posix_spawn or the shell). At the
same time a note was added to exec() application usage saying:

"Applications should not depend on file descriptors 0, 1, and 2
being closed after an exec. A future version may allow these file
descriptors to be automatically opened for any process."

I think we should make this change to allow implementations to do the
automatic open for any process. However, the issue would still remain
of what happens on implementations which do not open them, and what
happens if they are open but not readable (fd 0) or not writable (fd 1
or 2). In practice, I believe what happens is that the stdin, stdout
and stderr streams are created and their underlying file descriptors
are set to 0, 1 and 2 respectively, regardless of whether they are
open or closed, and regardless of whether fd 0 is readable or fd 1 or
2 is writable.

At first sight, it would seem that a simple solution is just to
modify section 2.5 to describe this existing practice. The problem
with this solution is that the guarantee in section 2.5 about stdin,
stdout and stderr being open (and readable/writable) at program
startup comes from the C Standard. I don't believe a modification
of this nature would be a valid extension to the C Standard - it
would create a conflict.

I believe an appropriate solution would be to state that if a program
is executed with fd 0 not open for reading or with fd 1 or 2 not open
for writing then this means the program is not executed in a conforming
environment. Thus the program is not required to behave as described
in the standards (C and POSIX).

There is also a similar issue for the standard utilities, in that
they may misbehave in odd ways if executed with fd 0 not open for
reading or with fd 1 or 2 not open for writing. The solution for
applications could be used to solve this issue as well.

0000215: Inconsistencies in pwrite() and write() errors

The error conditions on the pwrite() page associated with pipes/FIFOs
and sockets are inconsistently specified between pwrite() and write(),
and between pwrite() and lseek().

* Line 71229 specifies EAGAIN when O_NONBLOCK flag is set and pwrite()
or write() would block, regardless of the file type. This conflicts
with line 71251 which allows EWOULDBLOCK as an alternative error
for write() when the file descriptor refers to a socket. There is
no equivalent "EAGAIN or EWOULDBLOCK" error specified for pwrite().

* EPIPE for a pipe/FIFO is specified for both pwrite() and write(),
but EPIPE for a socket is only specified for write().

* ECONNRESET, EACCES, ENETDOWN, and ENETUNREACH are only specified
for write().

* EINVAL for a negative offset is required for pwrite() on all file
types, whereas for lseek() it is only required for regular files,
block special files, and directories. (I.e. implementations are
allowed to support lseek() to negative offsets on character
special files.)

* ESPIPE for pwrite() is only specified for a pipe/FIFO but should
also be specified for a socket (cf. lseek()).

When deciding how to correct these problems, a precedence issue
between the ESPIPE and other pwrite() errors needs to be
considered. Normally the standard does not specify precedence
between different error conditions - if this convention is
followed, then the full set of write() errors associated with
pipes, FIFOs and sockets should also be specified for pwrite(),
to allow those conditions to be detected before ESPIPE. However,
in practice it seems unlikely that any implementation of pwrite()
would get as far as detecting whether the write operation would
block, or is to a pipe/FIFO with no readers, or is to a socket
that has been shut down for writing etc., without having detected
the ESPIPE condition. The ESPIPE can be detected using only the
file type, whereas these other conditions require further
file-type-dependent information (i.e. they can only be detected
after the file type has been determined). Therefore the suggested
changes remove the existing pipe/FIFO/socket-related errors for
pwrite(), other than ESPIPE, instead of adding the missing ones.

In addition there are a couple of editorial problems that would be
worth addressing. Line 71230 refers to "the write() operation", but
the error applies to both write() and pwrite(); and, the pwrite()
"shall fail" block should be moved to before the "may fail" errors.

0000213: Definition of pututxline() is slightly too restrictive to allow flexible implementations

We at The FreeBSD Project very recently migrated to utmpx for storing user accounting information. We entirely removed the existing utmp interface, because it would be very impractical for us to support both interfaces.

When implementing utmpx I noticed the following. The behaviour of pututxline() is specified as follows:

"If the process has appropriate privileges, the pututxline() function shall write out the structure into the user accounting database. It shall use getutxid() to search for a record that satisfies the request. If this search succeeds, then the entry shall be replaced. Otherwise, a new entry shall be made at the end of the user accounting database."

Now this definition comes with two problems:

- It basically disallows an implementation which does not use the same file descriptor for the reading and the writing interface of the database. It clearly says that getutxid() must be used to search for a record that satisfies the request, while one could easily imagine an implementation which does not do this. For example, Solaris and NetBSD may spawn a setuid application to perform logging when unprivileged. This process may use getutxid() internally, but the point is that its side-effects are not noticeable by the calling process.

- A more serious problem with this definition is that the usage of getutxid() would blow up the database. Say, a login service that uses utmpx uses random values for ut_id to reduce the likelihood of collisions. Then this would mean each login session will have a DEAD_PROCESS entry that will remain in the database indefinitely. We at FreeBSD use a slightly improved allocation algorithm. When writing a USER_PROCESS, LOGIN_PROCESS or INIT_PROCESS record, it searches the database for an entry which has the same value for ut_id. If none is found, it overwrites an arbitrary DEAD_PROCESS record. This means the database will never become bigger than the maximum amount of concurrent sessions.

0000210: fsblkcnt_t and fsfilcnt_t incorrectly marked XSI

The statvfs structure, where fsblkcnt_t and fsfilcnt_t are used, is now in base. Therefore these type definitions should be, too.

0000205: Shell pipeline connection requirement too vague

The requirement for the way commands in a pipeline are connected
is stated as:

"The standard output of all but the last command shall be
connected to the standard input of the next command."

This is too vague; it needs to say precisely how they are
connected. Otherwise implementations could, for example,
connect the commands using a pseudo-terminal which would
cause some commands to behave differently (such as ls
writing multi-column output).

0000216: Control characters must have single-byte encodings

The design of the general terminal interface is such that the
characters which perform the terminal functions ERASE, KILL,
INTR, etc. have to be single-byte characters, since they are
assigned/obtained using a single numeric value in the
termios.c_cc[] array.

Currently only those control characters that are in the portable
character set (alert, backspace, tab, carriage-return, newline,
vertical-tab, form-feed) have an explicit requirement that they
are represented in a single byte (P128 L3590). However, the
absence of any mention of a way of encoding multiple bytes into
termios.c_cc[] entries means that there is an implicit requirement
for the other control characters as well. The requirement should
be stated explicitly.

In addition, there is a minor editorial problem in the beginning
of section 6.2 where it says "The POSIX locale contains the
characters in Table 6-1 (on page 125), which have the properties
listed in Section 7.3.1 (on page 139)." Since 7.3.1 specifies
properties for both the portable character set (Table 6-1) and the
control character set (Table 6-2) in the POSIX locale, both tables
should be mentioned in section 6.2.

0000214: "dot" should mention the effect of `return'

The description of "dot" never mentions `return'.

0000212: editorial errors in

TIMER_ABSTIME is not specified in C99, and is only used by clock_nanosleep and timer_settime, both of which are marked CX.

There is a typo in the description of getdate_err.

0000211: inttypes should provide wchar_t

In the move to make all of the standard headers self-contained (as an extension to the C99 behavior of requiring inclusion of prerequisite headers), we overlooked the use of wchar_t in the declaration of wcstoimax and wcstoumax.

0000209: lockf is XSI, constants not

lockf() is still (and should be) XSI. The constants to use with lockf() are not marked as such anymore. This should be changed.