Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001389 [1003.1(2016/18)/Issue7+TC2] Shell and Utilities Objection Error 2020-08-13 19:52 2020-08-20 15:31
Reporter kre View Status public  
Assigned To
Priority normal Resolution Open  
Status New  
Name Robert Elz
Organization
User Reference
Section 2.9.1.1 1.c (2.9.1.4 1.c in Issue8 Draft 1), 2.9.1.1 1.d, 2.9.1.1 1.e.i.a
Page Number 2367 (2335 in Issue8 draft 1)
Line Number 75564-73 75577-78 (74159-64 74168-71 in Issue8 draft 1)
Interp Status ---
Final Accepted Text
Summary 0001389: Shell command search procedure is incorrect
Description POSIX currently specifies that normal built-in standard utilities be
invoked only when the PATH search locates that utility.
 
There are all kinds of problems with this, but the most important
is that it does not document the standard behaviour (rather it
documents what it seems some people would like the standard behaviour
to be).

I set PATH to a list of directories that does not include /bin
(which, on my system, is where the standard utilities ls and test
are to be found - for completeness here, printf is in /usr/bin which
remained in $PATH, though the results below do not show that).
(I am not showing the value of my PATH with /bin removed, as what's
there isn't actually relevant for anything here).

Then I ran the following script, which tests every (posixish) shell
I have access to that I could remember I have...

for SHELL in /bin/sh /bin/ksh bash 'bash -o posix' fbsh dash bosh yash \
        pdksh mksh ksh93 zsh 'zsh --emulate sh'
do
        $SHELL -c '
                printf "\n\nTesting %s : \n" '"'$SHELL'"'
                ls /bin/test
                /bin/ls /bin/test
                IFS=:
                for p in $PATH
                do
                        [ -x $p/test ] && echo found test in $p
                done
                test x = x || echo weird
        '
done >/tmp/results 2>&1

It needs full pathnames for the standard NetBSD shells /bin/sh and /bin/ksh
as /bin isn't in PATH any more - /bin/ksh and pdksh are more or less the same
thing, though probably have slightly different sets of bugs fixed.
fbsh is the FreeBSD shell, the rest I think are well known. I don't
have a copy of ksh88.


In that the initial printf is just for documentation, the two ls
lines that follow are to show first that "ls" is not found (as /bin
is not in $PATH, and second that /bin/test exists - that is, that
is where the test utility would normally be found). From those
two lines I would expect (in every shell) to see some kind of error
about "ls" not being found, and then "/bin/test" as the output
from the 2nd /bin/ls command (and indeed, that is what happens,
as shown below).

IFS is set to break apart $PATH, and the for loop simply
verifies that there is no other "test" command anywhere else
that might be invoked. If there is another test, the printf
would tell us where it was located, if there isn't, then there
will be no output. Obviously all shells should do the same thing
here, and they do, none finds a "test" anywhere else.

Note that there is actually a possible issue here with the '['
command (which also lives in /bin) so a shell might complain that
'[' is not found. None do.

The final line is the important one, if a shell implemented the
specification in 2.9.1.1 1.c and 2.9.1.1 1.e.i.a "correctly", there
should be a "test not found" error (in some format or other,
just like the "ls not found" error) here, as the built-in version
of the standard utility test is only supposed to be located if test
is found in $PATH (which we have just seen it will not be).
The "|| echo weird" is just in case some non-standard test were
to be invoked, and there "echo" (which also lives in /bin) could
also result in "command not found" - this we don't know, as there
is no attempt made to execute that "echo" command, the test
command (in every shell I tested - I had hoped that in one or
two might not have test built in) is a built-in command, and
is executed (IMO correctly) here (and operates correctly,
returning a status of 0, so the || part is never attempted).

The results showing all of this appear bust below. To me this
is clear evidence that the current specification is not specifying
the standard behaviour, and should be fixed.


Testing /bin/sh :
ls: not found
/bin/test


Testing /bin/ksh :
/bin/ksh: ls: not found
/bin/test


Testing bash :
bash: ls: command not found
/bin/test


Testing bash -o posix :
bash: ls: command not found
/bin/test


Testing fbsh :
fbsh: ls: not found
/bin/test


Testing dash :
dash: 1: ls: not found
/bin/test


Testing bosh :
bosh: ls: not found
/bin/test


Testing yash :
yash: no such command `ls'
/bin/test


Testing pdksh :
pdksh: ls: not found
/bin/test


Testing mksh :
mksh: ls: not found
/bin/test


Testing ksh93 :
ksh93: ls: not found [No such file or directory]
/bin/test


Testing zsh :
zsh:1: command not found: ls
/bin/test


Testing zsh --emulate sh :
zsh:1: command not found: ls
/bin/test



Desired Action First, if considering Issue8 draft 1, all references below to
section 2.9.1.1 should be interpreted as references to section 2.9.1.4

In section 2.9.1.1 1.c (page 2367) delete the sentences:
If the implementation has provided a standard utility
in the form of a function, it shall not be recognized at this point.
It shall be invoked in conjunction with the path search in step 1e.


Replace the entire text of section 2.9.1.1 1.d (which differs dramatically
between Issue 7 and Issue 8 draft 1, so the old text is not quoted
here), (page 2367) with:
If the command name matches the name of a utility built
into the shell, that utility shall be invoked.


In section 2.9.1.1 1.e replace the entire first part of the text of
sub-section i (including the entire contents of sub-sub-sections 'a' and 'b')
down to, but not including the paragraph that begins
Once a utility has been searched for and found

The text to be removed is not included here, as it differs between
Issue 7 and Issue 8 draft 1. The replacement text should be:
If the search is successful the shell shall execute the
utility as described in Section 2.9.1.6.

[This is using the section numbering for the reference as it appears
in Issue 8 draft 1, where 2.9.1.6 is the section entitled:
Non-built-in Utility Execution.]

These changes will make the standard match what shells actually
implement, rather than some pipe dream about what they should implement.
Tags No tags attached.
Attached Files

- Relationships
duplicate of 0000854Applied 1003.1(2013)/Issue7+TC1 requirement for additional built-in utilities to be searched for via $PATH was not and is not existing practice 
related to 0001390New 1003.1(2016/18)/Issue7+TC2 No method to invoke a non-standard built-in utility 
related to 0001391New 1003.1(2016/18)/Issue7+TC2 Unclear language in specification of utilities implemented as functions 

-  Notes
(0004920)
geoffclare (manager)
2020-08-14 08:47

Although it goes into more detail, this is effectively a repeat of bug 0000854 and should be closed as a duplicate.
(0004924)
joerg (reporter)
2020-08-14 10:39
edited on: 2020-08-19 10:36

Just in case this is not known, ksh93 implements PATH search for builtin commands and this was a requirement for ksh93 integration into OpenSolaris in 2008 after a long and heaty discussion. The reason for that requirement was to retain POSIX compliance even though basic standard utilities are built into the standard shell.

If a shell does not implement PATH search for additional builtins, we would need to forbid shells to add more builtins than the minimal set.

ksh93 added many additional builtins like cat, chown, cmp, ... and POSIX grants that a user that if he has private version of system utilities, the shell needs to use them if they are first in PATH.

If a shell adds more builtins than currently documented, this would no longer work as historically builtin are always found first.

As a consequence, we created a list of tainted utilities (see 0000854) that already may be builtins.

If you like to add more builtins from the list of standard utilities, you either need to implement PATH search for these builtins, or you are forbidden to implement these utilities as builtins.

In ksh93, builtin commands are managed with the builtin named "builtin".

Call builtin for a list....e.g.

builtin
:
.
[
/bin/cat
/bin/chown
/bin/cksum
/bin/cmp
/bin/comm
/bin/cut
/bin/dirname
/bin/fold
/bin/getconf
/bin/head
/bin/join
/bin/logname
/bin/mkdir
/bin/mkfifo
/bin/mktemp
/bin/paste
/bin/pathchk
/bin/rev
/bin/rmdir
/bin/sum
/bin/sync
/bin/tail
/bin/tee
/bin/tty
/bin/uniq
/bin/wc
/bin/xgrep
/sbin/sync
/usr/ast/bin/basename
/usr/ast/bin/cat
/usr/ast/bin/chgrp
/usr/ast/bin/chmod
/usr/ast/bin/chown
/usr/ast/bin/cksum
/usr/ast/bin/cmp
/usr/ast/bin/comm
/usr/ast/bin/cp
/usr/ast/bin/cut
/usr/ast/bin/date
/usr/ast/bin/dirname
/usr/ast/bin/egrep
/usr/ast/bin/expr
/usr/ast/bin/fds
/usr/ast/bin/fgrep
/usr/ast/bin/fmt
/usr/ast/bin/fold
/usr/ast/bin/grep
/usr/ast/bin/head
/usr/ast/bin/id
/usr/ast/bin/join
/usr/ast/bin/ln
/usr/ast/bin/logname
/usr/ast/bin/md5sum
/usr/ast/bin/mkdir
/usr/ast/bin/mkfifo
/usr/ast/bin/mktemp
/usr/ast/bin/mv
/usr/ast/bin/paste
/usr/ast/bin/pathchk
/usr/ast/bin/readlink
/usr/ast/bin/rev
/usr/ast/bin/rm
/usr/ast/bin/rmdir
/usr/ast/bin/stty
/usr/ast/bin/sum
/usr/ast/bin/sync
/usr/ast/bin/tail
/usr/ast/bin/tee
/usr/ast/bin/tty
/usr/ast/bin/uname
/usr/ast/bin/uniq
/usr/ast/bin/wc
/usr/ast/bin/xgrep
/usr/sbin/sync
/usr/xpg4/bin/basename
/usr/xpg4/bin/egrep
/usr/xpg4/bin/fgrep
/usr/xpg4/bin/grep
/usr/xpg4/bin/id
/usr/xpg4/bin/rm
/usr/xpg4/bin/tail
/usr/xpg6/bin/expr
alarm
alias
bg
break
builtin
cd
command
....

Note that the builtins bound to /usr/ast/bin are not fully POSIX compliant and there fore are not bound to a path that appears in the default PATH.

If you do not care about POSIX compliance or if you do not like to add more builtins to your shell, you may do what ever you like. If you however like to add more builtins, you need to follow long existing rules about PATH search.

(0004934)
McDutchie (reporter)
2020-08-20 15:31

ksh93 does not follow the POSIX rules either, and never has done:

(1) ksh93 has several builtins (including builtin aliases, something POSIX doesn't allow either), not bound to any path search, that are "not allowed" by this rule, i.e.: which are neither a standard utility, nor in the list of names reserved as "unspecified" in XCU 2.9.1.4(b). They are: alarm, compound, enum, float, functions, integer, nameref, r, redirect.

(2) Several POSIX standard regular builtins are *never* subject to a path search in ksh93, although POSIX specifies that they must be. These are: [, echo, printf, sleep, test, type, ulimit.

What's more, path-bound builtins in ksh93 behave exactly like external commands. The only difference is their path doesn't actually exist in the file system. But that is a mere implementation detail. POSIX has no business specifying or regulating implementation details.

So, if this path search rule exists because of ksh93, then it is based on a fundamental misunderstanding of what ksh93 does.

In the real world, there are zero shells that actually manage to follow this rule. Not even the deliberately pedantic 'yash -o posix' fully manages, though it tries hard. Bash, Busybox ash, dash, FreeBSD sh, NetBSD sh, mksh, pdksh (OpenBSD sh), and zsh all ignore this rule completely. So does yash in its sane default mode.

This entire path search rule for builtins should quite simply be removed. I suppose it won't do much damage if POSIX decides to keep it; I'm confident the overwhelming majority of shells will continue to ignore it completely. The real damage is to POSIX's reputation as a standard to be taken seriously.

- Issue History
Date Modified Username Field Change
2020-08-13 19:52 kre New Issue
2020-08-13 19:52 kre Name => Your Name Here
2020-08-13 19:52 kre Section => (section number or name, can be interface name)
2020-08-13 19:52 kre Page Number => (page or range of pages)
2020-08-13 19:52 kre Line Number => 75564-73 75577-78 (74159-64 74168-71 in Issue8 draft 1)
2020-08-13 21:17 Don Cragun Name Your Name Here => Robert Elz
2020-08-13 21:17 Don Cragun Interp Status => ---
2020-08-13 21:17 Don Cragun Relationship added related to 0001390
2020-08-13 21:18 Don Cragun Relationship added related to 0001391
2020-08-13 21:24 Don Cragun Severity Editorial => Objection
2020-08-13 21:24 Don Cragun Type Clarification Requested => Error
2020-08-13 21:30 Don Cragun Section (section number or name, can be interface name) => 2.9.1.1 1.c (2.9.1.4 1.c in Issue8 Draft 1), 2.9.1.1 1.d, 2.9.1.1 1.e.i.a
2020-08-13 21:30 Don Cragun Page Number (page or range of pages) => 2367 (2335 in Issue8 draft 1)
2020-08-14 08:47 geoffclare Note Added: 0004920
2020-08-14 08:47 geoffclare Relationship added duplicate of 0000854
2020-08-14 10:39 joerg Note Added: 0004924
2020-08-14 10:40 joerg Note Edited: 0004924
2020-08-14 10:40 joerg Note Edited: 0004924
2020-08-14 10:41 joerg Note Edited: 0004924
2020-08-19 10:36 joerg Note Edited: 0004924
2020-08-20 15:31 McDutchie Note Added: 0004934


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker