Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000854 [1003.1(2013)/Issue7+TC1] Shell and Utilities Objection Error 2014-07-02 07:29 2020-03-27 17:22
Reporter philip-guenther View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied  
Name Philip Guenther
Organization OpenBSD
User Reference
Section XCU 2.9.1.1
Page Number 2339
Line Number 74397-74398
Interp Status Approved
Final Accepted Text See Note: 0002495
Summary 0000854: requirement for additional built-in utilities to be searched for via $PATH was not and is not existing practice
Description According to XCU 2.9.1.1, built-in utilities that are neither special built-ins (exec, set, etc) nor the regular ones listed in 2.9.1.1 part 1.c (alias, bg, etc) can only be found by searching $PATH, and that if not present in $PATH then attempts to execute them must fail with status 127.

That does not match existing practice, even of certified products like Solaris 10 and OS X Version 10.9. Instead, it appears that existing implementations just treat all non-special built-ins as found in part 1.c.
Desired Action Delete the requirement for other standard utility built-ins to be searched for in $PATH. Non-special built-ins must be considered after functions. In 2.9.1.1 part 1.c, state that the listed utilities must be invoked without searching $PATH, and that any other standard utility may be invoked without searching $PATH.

Consider changing the label on table 1-5 in XCU 1.6 to something like "Required Regular Built-In Utilities", or maybe "Minimal set of regular ...".
Tags issue8
Attached Files

- Relationships
related to 0000705Closed 1003.1(2013)/Issue7+TC1 Add hash, type and ulimit to regular built-in utilities 
has duplicate 0001389Closed 1003.1(2016/18)/Issue7+TC2 Shell command search procedure is incorrect 
related to 0000901Closed 1003.1(2013)/Issue7+TC1 reserve _POSIX* shell option namespace for future use 
related to 0000654Closedajosey 1003.1(2013)/Issue7+TC1 unclear behavior of in-line variable assignments preceding functions, special built-ins 
related to 0001445Applied Issue 8 drafts Update "special built-in" to match changes from bug #854 

-  Notes
(0002453)
Don Cragun (manager)
2014-12-01 06:39
edited on: 2014-12-04 16:33

Although the bug report requested changes to the XCU volumne of the standard, the changes below are to be applied to the XBD and XRAT volumes...

Change the description of PATH in XBD section 8, P178, L5711-5729 from:
This variable shall represent the sequence of path prefixes that certain functions and utilities apply in searching for an executable file known only by a filename. The prefixes shall be separated by a <colon> (’:’). When a nonzero-length prefix is applied to this filename, a <slash> shall be inserted between the prefix and the filename if the prefix did not end in <slash>. A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent <colon> characters ("::"), as an initial <colon> preceding the rest of the list, or as a trailing <colon> following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH. The list shall be searched from beginning to end, applying the filename to each prefix, until an executable file with the specified name and appropriate execution permissions is found. If the pathname being sought contains a <slash>, the search through the path prefixes shall not be performed. If the pathname begins with a <slash>, the specified path is resolved (see Section 4.12, on page 111). If PATH is unset or is set to null, the path search is implementation-defined.

Since <colon> is a separator in this context, directory names that might be used in PATH should not include a <colon> character.

to:
This variable shall represent the sequence of path prefixes that certain functions and utilities apply in searching for built-ins and executable files known only by a filename. The prefixes shall be separated by a <colon> (’:’). When a nonzero-length prefix is applied to this filename, a <slash> shall be inserted between the prefix and the filename if the prefix did not end in <slash>. A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent <colon> characters ("::"), as an initial <colon> preceding the rest of the list, or as a trailing <colon> following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH. The list shall be searched from beginning to end, applying the filename to each prefix, until an executable file with the specified name and appropriate execution permissions is found; or, if the shell is performing the search, a built-in associated with that prefix and filename has been found. If the pathname being sought contains a <slash>, the search through the path prefixes shall not be performed. If the pathname begins with a <slash>, the specified path is resolved (see Section 4.12, on page 111). If PATH is unset or is set to null, or if a path prefix in PATH contains a <percent-sign> character ('%'), the path search is implementation-defined.

Since <colon> is a separator in this context, directory names that might be used in PATH should not include a <colon> character. Since <percent-sign> may have an implementation-defined meaning when searching for built-in utilities, directory names in PATH that will be used to search for non-built-in utilities should not contain a <percent-sign> character.


Add the following paragraphs to the end of the rationale for the PATH environment variable in XRAT subclause A.8.3 after P3499, L118222:
The standard specifies that (when no <slash> character is included in a command pathname) special built-in utilities and the standard utilities listed in the table in XREF to XCU subclause 2.9.1.1 Command Search and Execution item 1.c. are not subject to a search using PATH. All other standard utilities, even if implemented as shell built-ins, are required to be found by searching PATH. This means that if a shell includes a built-in for a standard utility that is not in the list above, a user is allowed to write a utility that will override that built-in. The standard also requires that all standard utilities can be executed by commands like:
find . -type d -exec printf 'Found directory: %s\n' '{}' +

So, other than differences caused by using different shell execution environments, a standard utility that is implemented as a built-in and the non-built-in version of that standard utility are both required to behave as the standard specifies. But, if a non-standard utility is found in PATH before the standard utility's location in PATH, the non-standard utility must be invoked rather than the built-in. For instance, if the shell includes a built-in printf utility (which most shells do), PATH is initialized using:
PATH="$HOME/bin:$(command -p getconf PATH)"

and
$HOME/bin/printf
is an executable file containing:
command -p printf 'In %s with args:\n' "${0##*/}" >&2
command -p printf '\t%s\n' "$@" >&2
command -V printf >&2
command -Vp printf >&2
command -p printf "$@"

then the command:
printf '%s %s\n' HOME "$HOME" PATH "$PATH"

should produce output similar to:
In printf with args:
    %s %s\n
    HOME
    /Users/dwc
    PATH
    /Users/dwc/bin:/usr/bin:/bin:/usr/sbin:/sbin
printf is a tracked alias for /Users/dwc/bin/printf
printf is a shell builtin
HOME /Users/dwc
PATH /Users/dwc/bin:/usr/bin:/bin:/usr/sbin:/sbin


The current version of the Korn shell installs built-ins into the shell using a builtin utility that allows the built-in to be associated with the pathname of the non-built-in version of that utility. (Unfortunately, some implementations that use ksh93 as their standard sh utility, do not make use of this feature and incorrectly install built-ins for standard utilities that are not associated with a PATH search. And, most other shells incorrectly always use a built-in utility if one is installed, even when it should be overridden by a PATH search that should find the non-standard version of a utility with the name of that built-in.) Some other shells use a <percent-sign> character in a directory pathname in PATH to indicate one or more directories that should be used when processing PATH to determine when standard utilities (other than those listed in the table in XCU subclause 2.9.1.1 item 1.c.) should be found. The POSIX.1-201x revision of the standard allows either of these methods to be used to install built-ins that meet the requirements stated in XCU subclause 2.9.1.1 by making the behavior of the built-in path search implementation-defined when a <percent-sign> character is found in PATH.


(0002454)
wpollock (reporter)
2014-12-01 07:17

Typo in 2nd para of new rational additions:

"...using a eifferent shell execution environments..."

should probably be:

"...using different shell execution environments..."
(0002455)
Don Cragun (manager)
2014-12-01 07:36
edited on: 2014-12-04 08:24

The typo noted in Note: 0002454 as been corrected in place in Note: 0002453.

Two other corrections changing:
The list shall be searched from beginning to end, applying the filename to each prefix, until an executable file with the specified name and appropriate execution permissions is found. If the shell is performing the search, a built-in associated with the filename then the shell shall select that built-in

in the description of PATH processing to:
The list shall be searched from beginning to end, applying the filename to each prefix, until an executable file with the specified name and appropriate execution permissions is found; or, if the shell is performing the search, a built-in associated with that prefix and filename has been found.

and changing:
There is a proposal being considered for the dash shell that uses a <percent-sign> character in a directory pathname in PATH to indicate one or more directories that should be used when processing PATH to determine when standard utilities (other than those listed in the table in XCU subclause 2.9.1.1 item 1.c.) should be found.

in the RATIONALE to:
Some other shells use a <percent-sign> character in a directory pathname in PATH to indicate one or more directories that should be used when processing PATH to determine when standard utilities (other than those listed in the table in XCU subclause 2.9.1.1 item 1.c.) should be found.

have also been made in place in Note: 0002453.

(0002477)
rhansen (manager)
2014-12-05 08:37
edited on: 2014-12-11 17:41

Some existing implementations always run the built-in no matter what. This isn't ideal; users should always be able to override an ordinary built-in (which I define as a regular built-in that is not in the list in XCU 2.9.1.1 bullet 1.c.). The rationale (XRAT C.2.9.1 page 3694 lines 126360-126371) makes it clear that the intention has been to require implementations to associate ordinary built-ins with a directory so that the behavior is largely indistinguishable from a non-built-in. However, there are a couple of problems:
  • The existing normative text does not match the intended behavior described in the rationale. The normative text does not make it clear that ordinary built-ins are associated with a directory, and it says that if a non-built-in is found during a PATH search then the built-in is invoked regardless of the order of directories in PATH or the built-in's associated directory (if any). This is almost equivalent to the historical behavior of "always execute the built-in if there is one", with the only difference being the behavior when a non-built-in doesn't exist (issue 7 says to return 126 in this case).
  • Many current implementations don't behave in the intended way, probably because of the wording of the current normative text. They instead skip the PATH search and execute the built-in no matter what. ksh seems to be the only exception.

The proposed wording in Note: 0002453 addresses the former problem, but not the latter. Is that what we really want?

It's unfortunate that some existing implementations always execute the built-in without a PATH search, but given its popularity I think it's better to officially acknowledge the behavior while strongly discouraging it. For example:
On page 46 lines 1440-1443 (XBD 3.83 definition of Built-In Utility), change:
A utility implemented within a shell. The utilities referred to as special built-ins have special qualities. Unless qualified, the term "built-in" includes the special built-in utilities. Regular built-ins are not required to be actually built into the shell on the implementation, but they do have special command-search qualities.

Note: Special Built-In Utilities are defined in detail in XCU Section 2.14 (on page 2356). Regular Built-In Utilities are defined in detail in XCU Section 2.9.1.1 (on page 2339).
to:
A utility implemented within a shell. There are two main types of built-in utilities: special built-ins and regular built-ins. Unless qualified, the term "built-in" includes both types. The utilities referred to as special built-ins have special qualities. Regular built-ins are not required to be actually built into the shell on the implementation, but they usually have special command-search qualities or affect the current execution environment.

Note: Special Built-In Utilities are defined in detail in XCU Section 2.14 (on page 2356). Regular Built-In Utilities are defined in detail in XCU Section 1.6 (on page 2319).
On page 65 after line 1901 insert a new section:
3.201 Intrinsic Utility

A utility that is not subject to a PATH search during command search, usually implemented as a regular built-in utility.

Note: Intrinsic Utilities are defined in detail in XCU Section 1.7 (on page 2319).
On page 84 after line 2374 insert a new section:
3.318 Regular Built-In Utility (or Regular Built-In)

See Built-In Utility in Section 3.83 (on page 46).
On page 2318 lines 73544-73553 (XCU 1.6 Built-In Utilities), change:
The utilities named in Table 1-5 are frequently provided in built-in form. All of the utilities named in the table have special properties in terms of command search order within the shell, as described in Section 2.9.1.1 (on page 2339).

Table 1-5 Regular Built-In Utilities
  • alias
  • bg
  • cd
  • command
  • false
  • fc
  • fg
  • getopts
  • jobs
  • kill
  • newgrp
  • pwd
  • read
  • true
  • umask
  • unalias
  • wait

However, all of the standard utilities, including the regular built-ins in the table, but not the special built-ins [...]
to:
The intrinsic utilities described in Section 1.7 (on page 2318) are frequently provided as regular built-ins.

However, all of the standard utilities, including the regular built-ins but not the special built-ins [...]
On page 2319 after line 73556 insert a new section:
1.7 Intrinsic Utilities

The utilities in Table 1-5 are intrinsic utilities. As described in Section 2.9.1.1 (on page 2339), these utilities are not subject to a PATH search during command search and execution.

Table 1-5 Regular Built-In Utilities
  • alias
  • bg
  • cd
  • command
  • false
  • fc
  • fg
  • getopts
  • jobs
  • kill
  • newgrp
  • pwd
  • read
  • true
  • umask
  • unalias
  • wait

Whether any additional utility is considered an intrinsic utility is implementation-defined. Because applications are unable to override an intrinsic utility with a utility from PATH, implementations are strongly encouraged to minimize the number of additional intrinsic utilities.
On page 2339 lines 74388-74389 (XCU 2.9.1.1 bullet 1.c.), change:
If the command name matches the name of a utility listed in the following table, that utility shall be invoked.
  • alias
  • bg
  • cd
  • command
  • false
  • fc
  • fg
  • getopts
  • jobs
  • kill
  • newgrp
  • pwd
  • read
  • true
  • umask
  • unalias
  • wait
to:
If the command name matches the name of an intrinsic utility (see Section 1.17 on page 2319), that utility shall be invoked.
On page 3694 lines 126360-126366 (rationale for XCU 2.9.1.1), change:
The sequence selected for the Shell and Utilities volume of POSIX.1-2008 acknowledges that special built-ins cannot be overridden, but gives the programmer full control over which versions of other utilities are executed. It provides a means of suppressing function lookup (via the command utility) for the user's own functions and ensures that any regular built-ins or functions provided by the implementation are under the control of the path search. The mechanisms for associating built-ins or functions with executable files in the path are not specified by the Shell and Utilities volume of POSIX.1-2008, [...]
to:
The sequence selected for the Shell and Utilities volume of POSIX.1-2008 acknowledges that special built-ins cannot be overridden, but gives the programmer full control over which versions of other utilities are executed (with some exceptions). It provides a means of suppressing function lookup (via the command utility) for the user's own functions and, with the exception of the intrinsic utilities (see XCU Section 1.7), ensures that any regular built-ins or functions provided by the implementation are under the control of the path search. The mechanisms for associating non-intrinsic built-ins or functions with executable files in the path are not specified by the Shell and Utilities volume of POSIX.1-2008, [...]
Also, the phrase "at this point in the path search" in XCU 2.9.1.1 bullet 1.d.i.a. is confusing. How about:
On page 2340 lines 74397-74398 (XCU 2.9.1.1 bullet 1.d.i.a.), change:
If the system has implemented the utility as a regular built-in or as a shell function, it shall be invoked at this point in the path search.
to:
If the system has implemented the utility as a built-in or as a shell function, and the built-in or function is associated with the directory that was most recently tested during the successful PATH search, that built-in or function shall be invoked.


(0002479)
chet_ramey (reporter)
2014-12-05 14:03

The normative text is nonsensical and doesn't reflect existing or
historical practice; that's probably why it's been widely ignored.

As the original description notes, a strict reading of the standard, even
including the rationale, prohibits all shell builtins other than the
special builtins and the ones listed in the table in 2.9.1.1 unless they
are overriding an existing executable file, and requires that builtins
override executables.

So, a strict reading renders ksh93's `print', bash's `enable' or `help',
dash's `local', zsh's `disable', and mksh's `realpath' all non-conforming.
That's probably why few shells implement the normative text.

The proposed changes are not compelling, since they don't provide any
portable mechanism to do much of anything. They do provide a way for
a shell to preserve historical behavior: the shell associates regular
builtins with a magical initial entry in $PATH.

I suggest that we only require that a builtin version of a standard
command (e.g., test) behave as a superset of what the standard requires --
which we do -- and codify existing and historical practice. If you want
to "strongly discourage" existing behavior, we need to provide some actual
mechanism to accomplish what people have been doing for years.

Otherwise, while env(1) has its limitations, it can be used to execute
utilities while skipping builtins.
(0002487)
Don Cragun (manager)
2014-12-06 22:51
edited on: 2014-12-08 15:03

In response to Note: 0002479:
The C standard and the system interfaces volume of POSIX.1-2008 and its predecessors provide reserved namespaces for use by application writers, by standards, and by system implementors.

Unfortunately, other than reserving some variable names starting with POSIX and _POSIX, and the shell keywords, special built-ins, and standard utilities, there are few, if any, reserved namespaces in the shell and utilities volume of POSIX.1-2008 and its predecessors. And, due to lots of diverging existing practice, and the first two Utility Syntax Guidelines:
  1. Utility names should be between two and nine characters, inclusive.

  2. Utility names should include lowercase letters (the lower character classification) and digits only from the portable character set.
using leading underscores and case differences can't be used to differentiate utility namespaces like function namespaces are reserved in the C Standard and the System Interfaces volume of POSIX.1-2008.

The standard requires that standard utilities (other than special built-ins and those listed in XCU 2.9.1.1 point 1.c) must be able to be overridden by an executable file found in a directory listed in PATH before the directories containing the standard utilities. However, the standard allows a conforming shell to add keywords, special built-ins, and regular built-ins (for utilities not defined by the standard) to the list of built-ins found before evaluating PATH and to those found while searching PATH. So the print, enable, help, local, disable, and realpath built-ins implemented by ksh93, dash, zsh, and mksh (without searching PATH) do not violate the standards. However, if standard utilities (such as echo and printf) are implemented as built-ins, they are required to be associated with a PATH search and to be able to be overridden by a user supplied executable installed in a directory in PATH before the directories provided by the output from:
getconf PATH


(0002495)
rhansen (manager)
2014-12-12 03:09
edited on: 2015-01-08 16:50

Interpretation response
------------------------
The standard states that if a PATH search for a utility is successful, the built-in version of that utility (if present) will always be executed, and if the PATH search is not successful then attempts to execute the utility must fail with exit status 127. Conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor.

Rationale:
-------------
Many existing implementations execute a regular built-in without performing a PATH search. This behavior does not match the normative text, and it does not allow script authors to override regular built-in utilities via a specially crafted PATH. In addition, the rationale explains that the intention is to allow authors to override built-ins by modifying PATH, but this is not what the normative text says.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------
On page 46 lines 1440-1443 (XBD 3.83 definition of Built-In Utility), change:
A utility implemented within a shell. The utilities referred to as special built-ins have special qualities. Unless qualified, the term "built-in" includes the special built-in utilities. Regular built-ins are not required to be actually built into the shell on the implementation, but they do have special command-search qualities.

Note: Special Built-In Utilities are defined in detail in XCU Section 2.14 (on page 2356). Regular Built-In Utilities are defined in detail in XCU Section 2.9.1.1 (on page 2339).
to:
A utility implemented within a shell. There are two main types of built-in utilities: special built-ins and regular built-ins. Unless qualified, the term "built-in" includes both types. The utilities referred to as special built-ins have special qualities. Regular built-ins are not required to be actually built into the shell on the implementation, but they usually have special command-search qualities or affect the current execution environment.

Note: Special Built-In Utilities are defined in detail in XCU Section 2.14 (on page 2356). Regular Built-In Utilities are defined in detail in XCU Section 1.6 (on page 2318).
On page 65 after line 1901 insert a new section:
3.201 Intrinsic Utility

A utility that is not subject to a PATH search during command search, usually implemented as a regular built-in utility.

Note: Intrinsic Utilities are defined in detail in XCU Section 1.7 (on page 2318).
On page 84 after line 2374 insert a new section:
3.318 Regular Built-In Utility (or Regular Built-In)

See Built-In Utility in Section 3.83 (on page 46).
Change the description of PATH in XBD section 8, P178, L5711-5729 from:
This variable shall represent the sequence of path prefixes that certain functions and utilities apply in searching for an executable file known only by a filename. The prefixes shall be separated by a <colon> (’:’). When a nonzero-length prefix is applied to this filename, a <slash> shall be inserted between the prefix and the filename if the prefix did not end in <slash>. A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent <colon> characters ("::"), as an initial <colon> preceding the rest of the list, or as a trailing <colon> following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH. The list shall be searched from beginning to end, applying the filename to each prefix, until an executable file with the specified name and appropriate execution permissions is found. If the pathname being sought contains a <slash>, the search through the path prefixes shall not be performed. If the pathname begins with a <slash>, the specified path is resolved (see Section 4.12, on page 111). If PATH is unset or is set to null, the path search is implementation-defined.

Since <colon> is a separator in this context, directory names that might be used in PATH should not include a <colon> character.
to:
This variable shall represent the sequence of path prefixes that certain functions and utilities apply in searching for built-ins and executable files known only by a filename. The prefixes shall be separated by a <colon> (’:’). When a nonzero-length prefix is applied to this filename, a <slash> shall be inserted between the prefix and the filename if the prefix did not end in <slash>. A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent <colon> characters ("::"), as an initial <colon> preceding the rest of the list, or as a trailing <colon> following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH. The list shall be searched from beginning to end, applying the filename to each prefix, until either an executable file with the specified name and appropriate execution permissions is found or, if the shell is performing the search, a built-in associated with that prefix and filename is found. If the pathname being sought contains a <slash>, the search through the path prefixes shall not be performed. If the pathname begins with a <slash>, the specified path shall be resolved (see Section 4.12, on page 111). If PATH is unset or is set to null, or if a path prefix in PATH contains a <percent-sign> character ('%'), the path search is implementation-defined.

Since <colon> is a separator in this context, directory names that might be used in PATH should not include a <colon> character. Since <percent-sign> may have an implementation-defined meaning when searching for built-in utilities, directory names in PATH that will be used to search for non-built-in utilities should not contain a <percent-sign> character.
On page 2318 lines 73544-73553 (XCU 1.6 Built-In Utilities), change:
The utilities named in Table 1-5 are frequently provided in built-in form. All of the utilities named in the table have special properties in terms of command search order within the shell, as described in Section 2.9.1.1 (on page 2339).
Table 1-5 Regular Built-In Utilities
  • alias
  • bg
  • cd
  • command
  • false
  • fc
  • fg
  • getopts
  • jobs
  • kill
  • newgrp
  • pwd
  • read
  • true
  • umask
  • unalias
  • wait
However, all of the standard utilities, including the regular built-ins in the table, but not the special built-ins [...]
to:
The intrinsic utilities described in Section 1.7 (on page 2318) are frequently provided as regular built-ins.

However, all of the standard utilities, including the regular built-ins but not the special built-ins [...]
On page 2318 after line 73556 insert a new section:
1.7 Intrinsic Utilities

As described in Section 2.9.1.1 (on page 2339), intrinsic utilities are not subject to a PATH search during command search and execution. The utilities in Table 1-5 shall be intrinsic utilities.
Table 1-5 Intrinsic Utilities
  • alias
  • bg
  • cd
  • command
  • fc
  • fg
  • getopts
  • hash
  • jobs
  • kill
  • read
  • [XSI]type[/XSI]
  • [XSI]ulimit[/XSI]
  • umask
  • unalias
  • wait
Whether any additional utility is considered an intrinsic utility is implementation-defined. Because applications are unable to override an intrinsic utility with a utility from PATH, implementations should not make any utility an intrinsic utility beyond the utilities in Table 1-5.
On page 2339 lines 74388-74389 (XCU 2.9.1.1 bullet 1.c.), change:
If the command name matches the name of a utility listed in the following table, that utility shall be invoked.
  • alias
  • bg
  • cd
  • command
  • false
  • fc
  • fg
  • getopts
  • jobs
  • kill
  • newgrp
  • pwd
  • read
  • true
  • umask
  • unalias
  • wait
to:
If the command name matches the name of an intrinsic utility (see Section 1.7 on page 2318), that utility shall be invoked.
On page 2340 lines 74397-74398 (XCU 2.9.1.1 bullet 1.d.i.a.), change:
If the system has implemented the utility as a regular built-in or as a shell function, it shall be invoked at this point in the path search.
to:
If the system has implemented the utility as a built-in or as a shell function, and the built-in or function is associated with the directory that was most recently tested during the successful PATH search, that built-in or function shall be invoked.
Add the following paragraphs to the end of the rationale for the PATH environment variable in XRAT subclause A.8.3 after P3499, L118222:
The standard specifies that (when no <slash> character is included in a command pathname) special built-in utilities and intrinsic utilities are not subject to a search using PATH. All other standard utilities, even if implemented as shell built-ins, are required to be found by searching PATH. This means that if a shell includes a built-in for a standard utility that is not intrinsic, a user can write a utility that will override that built-in. The standard also requires that all standard utilities can be executed by commands like:
find . -type d -exec printf 'Found directory: %s\n' '{}' +
So, other than differences caused by using different shell execution environments, a standard utility that is implemented as a built-in and the non-built-in version of that standard utility are both required to behave as the standard specifies. But, if a non-standard utility is found in PATH before the standard utility's location in PATH, the non-standard utility must be invoked rather than the built-in. For instance, if the shell includes a built-in printf utility (which most shells do), PATH is initialized using:
PATH="$HOME/bin:$(command -p getconf PATH)"
and <tt>$HOME/bin/printf</tt> is an executable file containing:
command -p printf 'In %s with args:\n' "${0##*/}" >&2
command -p printf '\t%s\n' "$@" >&2
command -V printf >&2
command -Vp printf >&2
command -p printf "$@"
then the command:
printf '%s %s\n' HOME "$HOME" PATH "$PATH"
should produce output similar to:
In printf with args:
    %s %s\n
    HOME
    /Users/dwc
    PATH
    /Users/dwc/bin:/usr/bin:/bin:/usr/sbin:/sbin
printf is a tracked alias for /Users/dwc/bin/printf
printf is a shell builtin
HOME /Users/dwc
PATH /Users/dwc/bin:/usr/bin:/bin:/usr/sbin:/sbin
The current version of the Korn shell installs built-ins into the shell using a builtin utility that allows the built-in to be associated with the pathname of the non-built-in version of that utility. (Unfortunately, some implementations that use ksh93 as their standard sh utility do not make use of this feature and install built-ins for standard utilities that are not associated with a PATH search. And, most other shells incorrectly always use a built-in utility if one is installed, even when it should be overridden by a PATH search that should find the non-standard version of a utility with the name of that built-in.) Some other shells use a <percent-sign> character in a directory pathname in PATH to indicate one or more directories that should be used when processing PATH to determine when non-intrinsic standard utilities should be found. The POSIX.1-201x revision of the standard allows either of these methods to be used to install built-ins that meet the requirements stated in XCU subclause 2.9.1.1 by making the behavior of the built-in path search implementation-defined when a <percent-sign> character is found in PATH.
On page 3674 lines 125546-125552 (XRAT C.1.7 rationale for XCU 1.6 Built-In Utilities), change:
All of these utilities can be exec-ed. There is no requirement that these utilities are actually built into the shell itself, but many shells need the capability to do so because XCU Section 2.9.1.1 (on page 2339) requires that they be found prior to the PATH search. The shell could satisfy its requirements by keeping a list of the names and directly accessing the file-system versions regardless of PATH. Providing all of the required functionality for those such as cd or read would be more difficult.

There were originally three justifications for allowing the omission of exec-able versions:
to:
Other than the special built-in utilities, there is no requirement to build utilities into the shell itself. However, many shells implement certain utilities as regular built-ins for the following reasons:
  • To improve performance, especially for frequently used lightweight utilities (such as test, true, and false).
  • To eliminate the need for some sort of interprocess communication between the shell and those utilities that read or modifiy the shell's execution environment (such as cd).
  • To make it easier to satisfy the command search and execution requirements in XCU Section 2.9.1.1 (on page 2339) for intrinsic utilities. Intrinsic utilities must be found prior to the PATH search. The shell could satisfy this requirement by keeping a list of the intrinsic utility pathnames and directly accessing the file-system versions regardless of PATH, but these utilities usually need to read or modify the shell's execution environment anyway.
With the exception of the intrinsic utilities, all regular built-in utilities are subject to the PATH search and can be overridden by a specially crafted PATH environment variable.

All of the regular built-in utilities can be exec-ed. There were originally three justifications for allowing the omission of exec-able versions:
On page 3675 between lines 125572 and 125573, insert a new section heading:
C.1.8 Intrinsic Utilities
On page 3675 line 125573, change:
There were varying reasons for including utilities in the table of built-ins:
to:
There were varying reasons for including utilities in the table of intrinsic utilities:
On page 3675 line 125580, change:
cd, getopts, newgrp, read, umask, wait
to:
cd, getopts, hash, read, [XSI]type, ulimit,[/XSI] umask, wait
Remove page 3675 lines 125599-125602 (true, false).

On page 3675 after line 125602 insert a new paragraph:
The following utilities are frequently implemented as intrinsic (and built-in) utilities. Future versions of this standard might not allow these utilities, or any other standard utility not in Table 1-5 in XCU 1.7 on page 2318, to be intrinsic; implementations are encouraged to implement these as non-intrinsic utilities instead (but still built-in if they were previously built-in).
  • [
  • echo
  • false
  • newgrp
  • printf
  • pwd
  • test
  • true
On page 3694 lines 126360-126366 (rationale for XCU 2.9.1.1), change:
The sequence selected for the Shell and Utilities volume of POSIX.1-2008 acknowledges that special built-ins cannot be overridden, but gives the programmer full control over which versions of other utilities are executed. It provides a means of suppressing function lookup (via the command utility) for the user's own functions and ensures that any regular built-ins or functions provided by the implementation are under the control of the path search. The mechanisms for associating built-ins or functions with executable files in the path are not specified by the Shell and Utilities volume of POSIX.1-2008, [...]
to:
The sequence selected for the Shell and Utilities volume of POSIX.1-2008 acknowledges that special built-ins cannot be overridden, but gives the programmer full control over which versions of other utilities are executed (with some exceptions). It provides a means of suppressing function lookup (via the command utility) for the user's own functions and, with the exception of the intrinsic utilities (see XCU Section 1.7), ensures that any regular built-ins or functions provided by the implementation are under the control of the path search. The mechanisms for associating non-intrinsic built-ins or functions with executable files in the path are not specified by the Shell and Utilities volume of POSIX.1-2008, [...]
For each intrinsic utility in table 1-5 on page 2318, please add the following paragraph at the beginning of the utility's APPLICATION USAGE section:
This utility is required to be intrinsic. See section 1.7 on page 2318 for details.


(0002496)
rhansen (manager)
2014-12-12 05:20
edited on: 2014-12-12 05:56

Note: 0002495 is a combination of Note: 0002453 and Note: 0002477 with the following changes:
  • The new XCU section 1.7 was changed to remove some intrinsic utilities and add others:
    • remove false, true, newgrp, and pwd as these do not need to read/modify the shell's execution environment, and it is not important that they preempt the PATH search
    • add hash, type, and ulimit as these do need to read/modify the shell's execution environment and they are commonly implemented as intrinsic utilities
  • in rationale, mention utilities that have traditionally been intrinsic (true, false, newgrp, pwd, test, echo, printf) and warn that a future version might not allow them to be intrinsic
  • rationale cleanups
  • minor wording fixes

The new proposed changes allow current behavior (built-ins that are executed before a PATH search are now considered to be intrinsic utilities) but discourages the behavior.

Non-intrinsic built-ins are subject to a PATH search but the mechanism for associating a built-in with a directory is still unspecified. This gap is unfortunate, but I don't think any particular behavior is pervasive enough to standardize. And standardizing a particular behavior will be easier once we figure out a good way to reserve a namespace and/or some of the commonly used command names (like 'builtin').

(0002512)
wpollock (reporter)
2014-12-29 04:30
edited on: 2015-01-08 16:55

In addition to the changes noted and approved, the description of each (required to be) intrinsic utility needs to be updated to reflect that change, including a See Also reference to the new section 3.201 and/or 1.7.

I believe the two references to section 1.17 in the new text (with different page numbers: 2318 and 2319) are typos for section 1.7 on whatever page that ends up being after all the new material is added.

An additional change to consider: lines 23446-ff (confstr), change:

can be used as a value of the PATH environment variable that accesses all of the standard utilities of POSIX.1-2008, if the return value is less than or equal to sizeof (buf ).

To:

can be used as a value of the PATH environment variable that accesses all of the standard utilities of POSIX.1-2008, except the intrinsic utilities (see XCU Section 1.7 on page 2318), if the return value is less than or equal to sizeof (buf).

(0002518)
Don Cragun (manager)
2015-01-08 16:27
edited on: 2015-01-08 16:55

In response to Note: 0002512: The intrinsic utilities still have to be available as stand-alone exec-able utilities. Therefore the change to lines 23446-23447 is not needed. The fact that a utility is intrinsic does not affect the behavior of the utility, so we don't see the need to add a note to the DESCRIPTION section of the intrinsic utilities, but we have requested that the editor note in the APPLICATION USAGE that these utilities are now intrinsic.

(0002519)
rhansen (manager)
2015-01-08 16:55

I edited Note: 0002495 and made the following changes:
  • change 1.17 to 1.7 (typo fix)
  • added instructions to say that the utility is intrinsic in the utility's APPLICATION USAGE section (see the last two lines of the bugnote)
(0002522)
eblake (manager)
2015-01-09 17:45

Note: 0001485 of 0000654 mentions some implications of variable assignment in the same command as a function call; while I didn't spot any overlapping edits between that bug and this, we may want to consider specifying whether being intrinsic can/should affect whether variable assignments persist after a function call that implements a standard utility.
(0002556)
ajosey (manager)
2015-02-20 12:14

Interpretation proposed 20 Feb 2015
(0002597)
ajosey (manager)
2015-03-23 12:08

Interpretation approved 23rd March 2015
(0003228)
kre (reporter)
2016-05-15 06:03

I observe that section 2.14 (Special Built-In Utilities) says ...

    The term "built-in" implies that the shell can execute the utility directly
    and does not need to search for it.

And I wonder how that definition applies to the question in this issue, and
to its resolution.
(0003233)
rhansen (manager)
2016-05-26 15:39

@kre: I think it was an oversight that we didn't change that text when we resolved this bug report. Would you mind filing a new bug report about this?
(0004803)
geoffclare (manager)
2020-03-27 17:22
edited on: 2020-03-31 16:40

In applying this bug, I had to leave off the shading on type and ulimit in Table 1-5, as it is not possible to add shading to text within a table that is in column order. I don't see this as a problem since their utility pages show that they are part of the XSI option. (I also left off the shading in C.1.8, where I do not believe it is needed since it is non-normative text.)


- Issue History
Date Modified Username Field Change
2014-07-02 07:29 philip-guenther New Issue
2014-07-02 07:29 philip-guenther Name => Philip Guenther
2014-07-02 07:29 philip-guenther Organization => OpenBSD
2014-07-02 07:29 philip-guenther Section => XCU 2.9.1.1
2014-07-02 07:29 philip-guenther Page Number => 2339
2014-07-02 07:29 philip-guenther Line Number => 74397-74398
2014-12-01 06:39 Don Cragun Note Added: 0002453
2014-12-01 06:46 Don Cragun Note Edited: 0002453
2014-12-01 06:53 Don Cragun Note Edited: 0002453
2014-12-01 06:57 Don Cragun Note Edited: 0002453
2014-12-01 07:01 Don Cragun Interp Status => ---
2014-12-01 07:01 Don Cragun Final Accepted Text => See Note: 0002453.
2014-12-01 07:01 Don Cragun Tag Attached: issue8
2014-12-01 07:17 wpollock Note Added: 0002454
2014-12-01 07:34 Don Cragun Note Edited: 0002453
2014-12-01 07:36 Don Cragun Note Added: 0002455
2014-12-04 07:56 Don Cragun Note Edited: 0002453
2014-12-04 07:56 Don Cragun Note Edited: 0002455
2014-12-04 08:17 Don Cragun Note Edited: 0002453
2014-12-04 08:18 Don Cragun Note Edited: 0002453
2014-12-04 08:21 Don Cragun Note Edited: 0002455
2014-12-04 08:23 Don Cragun Note Edited: 0002453
2014-12-04 08:24 Don Cragun Note Edited: 0002455
2014-12-04 16:18 Don Cragun Note Edited: 0002453
2014-12-04 16:33 Don Cragun Note Edited: 0002453
2014-12-05 08:37 rhansen Note Added: 0002477
2014-12-05 08:39 rhansen Note Edited: 0002477
2014-12-05 08:42 rhansen Note Edited: 0002477
2014-12-05 08:43 rhansen Note Edited: 0002477
2014-12-05 14:03 chet_ramey Note Added: 0002479
2014-12-06 22:51 Don Cragun Note Added: 0002487
2014-12-06 22:54 Don Cragun Note Edited: 0002487
2014-12-06 23:01 Don Cragun Note Edited: 0002487
2014-12-06 23:01 Don Cragun Note View State: public: 2487
2014-12-08 15:03 Don Cragun Note Edited: 0002487
2014-12-11 16:21 nick Relationship added related to 0000901
2014-12-11 17:05 rhansen Note Edited: 0002477
2014-12-11 17:07 rhansen Note Edited: 0002477
2014-12-11 17:08 rhansen Note Edited: 0002477
2014-12-11 17:09 rhansen Note Edited: 0002477
2014-12-11 17:41 rhansen Note Edited: 0002477
2014-12-12 03:09 rhansen Note Added: 0002495
2014-12-12 03:26 rhansen Note Edited: 0002495
2014-12-12 05:20 rhansen Note Added: 0002496
2014-12-12 05:56 rhansen Note Edited: 0002496
2014-12-18 16:09 rhansen Note Edited: 0002495
2014-12-18 16:25 rhansen Note Edited: 0002495
2014-12-18 16:27 rhansen Note Edited: 0002495
2014-12-18 16:36 rhansen Note Edited: 0002495
2014-12-18 16:37 rhansen Note Edited: 0002495
2014-12-18 16:38 rhansen Interp Status --- => Pending
2014-12-18 16:38 rhansen Final Accepted Text See Note: 0002453. => See Note: 0002495
2014-12-18 16:38 rhansen Status New => Interpretation Required
2014-12-18 16:38 rhansen Resolution Open => Accepted As Marked
2014-12-29 04:30 wpollock Note Added: 0002512
2015-01-08 16:08 rhansen Note Edited: 0002495
2015-01-08 16:27 Don Cragun Note Added: 0002518
2015-01-08 16:28 Don Cragun Note Edited: 0002518
2015-01-08 16:35 Don Cragun Note Edited: 0002518
2015-01-08 16:50 rhansen Note Edited: 0002495
2015-01-08 16:51 Don Cragun Note Edited: 0002518
2015-01-08 16:55 rhansen Note Added: 0002519
2015-01-08 16:55 rhansen Note Edited: 0002518
2015-01-08 16:55 rhansen Note Edited: 0002512
2015-01-09 17:43 eblake Relationship added related to 0000654
2015-01-09 17:45 eblake Note Added: 0002522
2015-02-20 12:14 ajosey Interp Status Pending => Proposed
2015-02-20 12:14 ajosey Note Added: 0002556
2015-03-23 12:08 ajosey Interp Status Proposed => Approved
2015-03-23 12:08 ajosey Note Added: 0002597
2016-01-13 15:52 eblake Relationship added related to 0000705
2016-05-15 06:03 kre Note Added: 0003228
2016-05-26 15:39 rhansen Note Added: 0003233
2020-03-27 17:22 geoffclare Note Added: 0004803
2020-03-27 17:22 geoffclare Status Interpretation Required => Applied
2020-03-31 16:40 geoffclare Note Edited: 0004803
2020-08-14 08:47 geoffclare Relationship added has duplicate 0001389
2021-01-21 18:11 rhansen Relationship added related to 0001445


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker