0001457: Add readlink(1), realpath(1) utility

Notes
(0005700) geoffclare (manager) 2022-02-22 15:58 edited on: 2022-02-24 09:34	Suggested changes... On page 3194 insert a new page for readlink: NAME readlink - display target of symbolic link on standard output SYNOPSIS readlink [-fn] file DESCRIPTION If the -f option is not specified and the file operand names a symbolic link, the readlink utility shall not follow the symbolic link when resolving file and shall write the contents of the symbolic link to standard output. If the -f option is not specified and file does not refer to a symbolic link, readlink shall write a diagnostic message to standard error and exit with non-zero status. If the -f option is specified: If the file operand names a symbolic link, readlink shall write to standard output the absolute pathname that would be returned by a call to the realpath() function with the pathname as its first argument. If the file operand names a file that is not a symbolic link, or if the path prefix of file resolves, with symbolic links followed, to an existing directory and the final component of file does not exist as a directory entry in that directory, readlink shall write to standard output the absolute pathname that would be returned by a call to the realpath() function with the path prefix of file as its first argument, followed by a <slash> character and the final component of file. If file is non-empty and only has one pathname component, it shall be treated as if it had a path prefix of "./". If file has more than one pathname component and the path prefix of file does not resolve, with symbolic links followed, to an existing directory, readlink shall write a diagnostic message to standard error and exit with non-zero status. If the realpath() function would return a null pointer, readlink shall write a diagnostic message to standard error and exit with non-zero status. If the -n option is not specified, the output to standard output in all cases described above shall be followed by a <newline> character. OPTIONS The readlink utility shall conform to [xref to XBD 12.2]. The following options shall be supported: -f Canonicalize the pathname specified by file, as specified in the DESCRIPTION section. -n Do not output a trailing <newline> character. OPERANDS file A pathname of a symbolic link to be read or, if the -f option is specified, a pathname to be canonicalized. STDIN Not used. INPUT FILES None. ENVIRONMENT VARIABLES The following environment variables shall affect the execution of readlink: LANG Provide a default value for the internationalization variables that are unset or null. (See [xref to XBD 8.2] for the precedence of internationalization variables used to determine the values of locale categories.) LC_ALL If set to a non-empty string value, override the values of all the other internationalization variables. LC_CTYPE Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments). LC_MESSAGES Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error. [XSI]NLSPATH Determine the location of message catalogs for the processing of LC_MESSAGES.[/XSI] ASYNCHRONOUS EVENTS Default. STDOUT See DESCRIPTION. STDERR The standard error shall be used only for diagnostic messages. OUTPUT FILES None. EXTENDED DESCRIPTION None. EXIT STATUS The following exit values shall be returned: 0 Successful completion. >0 An error occurred. CONSEQUENCES OF ERRORS Default. APPLICATION USAGE As the final component of file does not need to exist when the -f option is used, a separate existence check has to be performed in applications that need to canonicalize a pathname and also check that the pathname names an existing file. EXAMPLES None. RATIONALE The readlink utility was added because using <tt>ls -l</tt> to obtain the contents of a symbolic link is difficult if the output includes more than one occurrence of the string " -> ". Although the behavior of readlink with -f is specified by reference to the realpath() function, which is part of the XSI option, non-XSI implementations that do not support realpath() are nevertheless required to implement readlink in accordance with the requirements described in this standard for realpath(). FUTURE DIRECTIONS None. SEE ALSO ln, ls, pwd XBD Chapter 8, Section 12.2 XSH readlink(), realpath() CHANGE HISTORY First released in Issue 8. On page 2901 line 95844 section ln, add readlink to SEE ALSO.

(0005702) steffen (reporter) 2022-02-22 22:19	Re #5701 That is a pity. Then despites Geoff's awesome work on providing a fully fledged standard entry including -f based on refined realpath() usage, readlink should stay and -f alone be dropped. And i should apologize for not looking everywhere. (That -f of Net and FreeBSD is really strange.)

(0005703) geoffclare (manager) 2022-02-23 08:32 edited on: 2022-02-24 09:36	I noticed that the GNU coreutils man page for readlink says "Note realpath(1) is the preferred command to use for canonicalization functionality." So perhaps we should add a realpath utility in place of readlink -f.

(0005704) emaste (reporter) 2022-02-23 13:31	I added a comment about the -f option on FreeBSD, but deleted it moments later after realizing I was mistaken. FreeBSD documents stat and readlink in the same man page and the option summary is unclear about which options apply to which program. Here is an excerpt of the stat/readlink man page that describes only readlink: readlink [-fn] [file ...] When invoked as readlink, only the target of the symbolic link is printed. If the given argument is not a symbolic link and the -f option is not specified, readlink will print nothing and exit with an error. If the -f option is specified, the output is canonicalized by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths, and return the absolute pathname corresponding to file. In this case, the argument does not need to be a symbolic link. -n Do not force a newline to appear at the end of each piece of output. From NetBSD's man page: readlink [-fnqsv] [file ...] When invoked as readlink, only the target of the symbolic link is printed. If the given argument is not a symbolic link and the -f option is not specified, readlink will print nothing and exit with an error. If the -f option is specified, the output is canonicalized by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths, and return the absolute pathname corresponding to file. In this case, the argument does not need to be a symbolic link. -n Do not force a newline to appear at the end of each piece of output. -q Suppress failure messages if calls to stat(2) or lstat(2) fail. When run as readlink, error messages are automatically suppressed. -s ... When run as readlink, suppress error messages. -v Turn off quiet mode. And OpenBSD: readlink [-fn] file -f Canonicalize by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths and return the absolute pathname corresponding to file. The argument does not need to be a symbolic link. -n Do not print a trailing newline character.

(0005705) calestyo (reporter) 2022-02-23 13:38	Re: #5703 "Note realpath(1) is the preferred command to use for canonicalization functionality." => but just for the canonicalization, AFAIU, realpath cannot print the resolved link (i.e. it's destination)

(0005706) geoffclare (manager) 2022-02-24 09:28	Looking at the GNU coreutils man page for realpath(1), it seems that its default behaviour is the same as readlink -f in that all components except the last need to exist, but it has -e and -m options which alter the existence requirements. It also has a -L option which eliminates ".." components before following symlinks. I assume these features are the reason the readlink man page says realpath is preferred over readlink -f. Even though there is no longer any obstacle to standardising readlink -f, I think we should still consider adding realpath (alongside readlink without -f). Solaris 11.4 has the GNU coreutils realpath. Do BSD systems have a realpath utility?

(0005721) mirabilos (reporter) 2022-02-25 21:24	mksh has a realpath builtin, please don’t add an incompatible requirement if you do ;-) It takes an optional first argument "--" (skipped) and a mandatory second argument (file); no options, and any other arguments cause an error message as well. I’ve recently learnt about some differences between BSD and POSIX realpath on the C level (from reading the OpenSSH mailing list). The realpath in mksh is specified to allow referring to nōnexistant files in existing directories, which I understand is very much not a POSIX thing. Without that, though, I’d consider it good as useless (one can always add a test -e afterwards). https://github.com/MirBSD/mksh/blob/5d135a8ee38d84c5a4f6c175b9a831d5acbc641b/misc.c#L1886 [^] in case of interest. I understand other OSes have diverging realpath utilities. mksh has a mode where a utility can call the builtin or an external one depending on the existence of the latter and, optionally, presence of flags. The latter is implemented for realpath, so “realpath -h” calls coreutils’ realpath on my GNU system, but not on the BSD system without an external realpath utility. This is a compromise. “readlink” is important. I’d argue for standardised readlink without and with -f but not adding realpath (unless the latter can be done in a way that doesn’t blow up (remember the story of cat(1) that went out and came back waving flags) mksh or, worse, make its implementation incompatible).

(0005898) geoffclare (manager) 2022-07-21 10:41 edited on: 2022-07-21 10:57	New suggested changes... On page 3194 insert new pages for readlink and realpath: NAME readlink - display the contents of a symbolic link SYNOPSIS readlink [-n] file DESCRIPTION If the file operand names a symbolic link, the readlink utility shall not follow the symbolic link when resolving file and shall write the contents of the symbolic link to standard output. If the -n option is not specified, the output to standard output shall be followed by a <newline> character. If file does not name a symbolic link, readlink shall write a diagnostic message to standard error and exit with non-zero status. OPTIONS The readlink utility shall conform to [xref to XBD 12.2]. The following option shall be supported: -n Do not output a trailing <newline> character. OPERANDS file A pathname of a symbolic link to be read. STDIN Not used. INPUT FILES None. ENVIRONMENT VARIABLES The following environment variables shall affect the execution of readlink: LANG Provide a default value for the internationalization variables that are unset or null. (See [xref to XBD 8.2] for the precedence of internationalization variables used to determine the values of locale categories.) LC_ALL If set to a non-empty string value, override the values of all the other internationalization variables. LC_CTYPE Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments). LC_MESSAGES Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error. [XSI]NLSPATH Determine the location of messages objects and message catalogs.[/XSI] ASYNCHRONOUS EVENTS Default. STDOUT See DESCRIPTION. STDERR The standard error shall be used only for diagnostic messages. OUTPUT FILES None. EXTENDED DESCRIPTION None. EXIT STATUS The following exit values shall be returned: 0 Successful completion. >0 An error occurred. CONSEQUENCES OF ERRORS Default. APPLICATION USAGE None. EXAMPLES None. RATIONALE The readlink utility was added because using <tt>ls -l</tt> to obtain the contents of a symbolic link is difficult if the output includes more than one occurrence of the string " -> ". The -f option found in many implementations was not included, as the realpath utility provides equivalent functionality with a choice of behaviors. FUTURE DIRECTIONS None. SEE ALSO ln, ls, pwd, realpath XBD Chapter 8, Section 12.2 XSH readlink() CHANGE HISTORY First released in Issue 8. NAME realpath - resolve a pathname SYNOPSIS realpath [-E\|-e] file DESCRIPTION The realpath utility shall canonicalize the pathname specified by the file operand as follows: If a call to the realpath() function with the specified pathname as its first argument would succeed, the canonicalized pathname shall be the pathname that would be returned by that realpath() call. Otherwise: If the -e option is specified, the canonicalization shall fail. If the -E option is specified, then if a call to the realpath() function with the specified pathname as its first argument would encounter an error condition other than [ENOENT], the canonicalization shall fail; if the call would encounter an [ENOENT] error, realpath shall expand all symbolic links that would be encountered in an attempt to resolve the specified pathname using the algorithm specified in [xref to XBD 4.13 Pathname Resolution], except that any trailing <slash> characters that are not also leading <slash> characters shall be ignored. If this expansion succeeds and the path prefix of the expanded pathname resolves to an existing directory, the canonicalized pathname shall be the expanded pathname. In all other cases, the canonicalization shall fail. If the expanded pathname is not empty, does not begin with a <slash>, and has exactly one pathname component, it shall be treated as if it had a path prefix of "./". If no options are specified, realpath shall canonicalize the specified pathname in an unspecified manner such that the resulting absolute pathname does not contain any components that refer to files of type symbolic link and does not contain any components that are dot or dot-dot. Upon successful canonicalization, realpath shall write the canonicalized pathname, followed by a <newline> character, to standard output. If canonicalization fails, or the canonicalized pathname is empty, nothing shall be written to standard output, a diagnostic message shall be written to standard error, and realpath shall exit with non-zero status. OPTIONS The realpath utility shall conform to [xref to XBD 12.2]. The following options shall be supported: -E Do not treat it as an error if attempting to resolve the last component of the canonicalized form of the file operand results in an [ENOENT] error condition. -e Treat it as an error if attempting to resolve the last component of the canonicalized form of the file operand results in an [ENOENT] error condition. Specifying more than one of the mutually-exclusive options -E and -e shall not be considered an error. The last option specified shall determine the behavior of the utility. OPERANDS file A pathname to be canonicalized. STDIN Not used. INPUT FILES None. ENVIRONMENT VARIABLES The following environment variables shall affect the execution of realpath: LANG Provide a default value for the internationalization variables that are unset or null. (See [xref to XBD 8.2] for the precedence of internationalization variables used to determine the values of locale categories.) LC_ALL If set to a non-empty string value, override the values of all the other internationalization variables. LC_CTYPE Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments). LC_MESSAGES Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error. [XSI]NLSPATH Determine the location of messages objects and message catalogs.[/XSI] ASYNCHRONOUS EVENTS Default. STDOUT See DESCRIPTION. STDERR The standard error shall be used only for diagnostic messages. OUTPUT FILES None. EXTENDED DESCRIPTION None. EXIT STATUS The following exit values shall be returned: 0 Successful completion. >0 An error occurred. CONSEQUENCES OF ERRORS Default. APPLICATION USAGE If neither the -e nor the -E option is specified, some implementations behave as if -e had been specified and others as if -E had been specified, but there are also implementations where the behavior differs from both of these. For example, the mksh shell has an internal implementation of realpath that canonicalizes /dir/regular_file/.. to /dir, whereas the realpath() function would return an [ENOTDIR] error in this case. Portable applications should always specify either -e or -E. EXAMPLES None. RATIONALE The realpath utility was added in preference to a -f option found in some implementations of the readlink utility because it allows the application to specify whether or not a missing final component is to be treated as an error. The behavior with the -E option when file does not resolve (with symbolic links followed) to an existing file is not the same as simply calling realpath() with the path prefix of the file operand and writing the resulting pathname, a <slash>, and the last component of file to standard output. For example, if /tmp/nofile does not exist, and file is A/B where A is an existing directory and B is a symbolic link to /tmp/nofile, realpath with -E will output /tmp/nofile, but if B is a symbolic link to /tmp/nofile/foo, realpath with -E will treat this as an error. In both cases <tt>realpath("A/B")</tt> would fail with errno set to [ENOENT]. Even though <tt>realpath("A")</tt> would succeed, in neither case is anything ending /B the result. Trailing <slash> characters (that follow a non-<slash>) are handled differently with -E than with -e. With -e they are handled as for the realpath() function. With -E they are sometimes effectively ignored, and they are never included in the output. For example, if /tmp/nofile does not exist and /tmp/regfile is an existing regular file: $ realpath -E /tmp/nofile/ /tmp/nofile $ realpath -E /tmp/regfile/ realpath: /tmp/regfile/: Not a directory Although the behavior of the realpath utility is specified by reference to the realpath() function, which is part of the XSI option, non-XSI implementations that do not support realpath() are nevertheless required to implement realpath in accordance with the requirements described in this standard for realpath(). FUTURE DIRECTIONS None. SEE ALSO ln, ls, pwd, readlink XBD Chapter 8, Section 12.2 XSH Section 2.3, realpath() CHANGE HISTORY First released in Issue 8. On page 2901 line 95844 section ln, add readlink and realpath to SEE ALSO. On page 2932 line 96992 section ls, add readlink to SEE ALSO.

(0005899) geoffclare (manager) 2022-07-21 10:45 edited on: 2022-07-21 10:47	My thanks to kre for helping to ensure that my proposed realpath page correctly describes the behaviour both of his new implementation for NetBSD and of the GNU coreutils version (once they add a no-op -E option). Update: actually, -E won't quite be a no-op, as it will need to turn off -e.

(0005900) kre (reporter) 2022-07-21 11:15	The NetBSD code (standard BSD licence, so available to everyone for anything, close enough) which implements Note: 0005898 is available at http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.bin/realpath/?only_with_tag=MAIN [^] just follow the links for the source file, the man page, or the Makefile and then click the download link on the most recent revision of what that gives you (depending on your browser, some kind of modified click might be needed to save the file, otherwise it will just display in the browser). (https:// should work as well, if you prefer). This should be fairly easy to build on any BSD derived platform (may need some minor updates to deal with <sys/cdefs.h> and the __RCSID() macro). This version originated at FreeBSD, but there's not a lot of that code remaining.

(0005901) kre (reporter) 2022-07-21 11:45	Also, wrt the now long deleted note#5701, I totally understand how that could have happened - the combined stat(1)+readlink(1) man page was a total mess, and almost impossible to decipher. At NetBSD we have now split it into two entirely separate pages, stat(1) and readlink(1) - the two (conceptually) are totally unrelated utilities. It just happens that they are the same binary, which looks at argv[0]. readlink (as it exists in BSD) is just stat with a particular set of options (which depends upon the options to readlink) - readlink could as easily have been a script that calls stat - except that for some uses it is used a lot, and adding a sh invocation to every call would be costly. But that note 5701 interpreted that messy man page incorrectly doesn't affect the conclusion to omit readlink -f from the standard. The coreutils readlink -f is (effectively) realpath -E (as specified here) and the BSD one is realpath -e - they are not compatible, and adding more options to readlink to "fix" that would be a sub-optimal solution (what's more, the coreutils version could not be implemented as a call to stat, no matter what options are given stat(1) never returns data about files that don't exist (it would have to invent that, and that's not going to happen.)

(0005921) geoffclare (manager) 2022-07-29 14:16	The readlink and realpath additions have been made in the Issue8NewAPIs branch in gitlab, based on Note: 0005898.

(0006079) geoffclare (manager) 2022-11-21 16:12	Make the changes from "Additional APIs for Issue 8, Part 2" (Austin/1273).

(0006219) eblake (manager) 2023-03-22 12:55	Updating title to reflect agreed-on action

(0006221) eblake (manager) 2023-03-22 14:31	GNU coreutils readlink has -q (--quiet) and -v (--verbose) options, and currently defaults to -q, which gives a non-zero status for a non-symlink but with no diagnostic message. This is in conflict with the proposed wording that mandates that readlink output a diagnostic. Should we amend this proposal to either standardize -v/-q, or to permit coreutils' current behavior? I've also opened a bug with the coreutils folks to see if they are willing to switch their default (perhaps only when POSIXLY_CORRECT is set?): https://lists.gnu.org/archive/html/bug-coreutils/2023-03/msg00035.html [^]

Issue History
Date Modified	Username	Field	Change
2021-03-09 21:21	steffen	New Issue
2021-03-09 21:21	steffen	Status	New => Under Review
2021-03-09 21:21	steffen	Assigned To	=> ajosey
2021-03-09 21:21	steffen	Name	=> Steffen Nurpmeso
2021-03-09 21:21	steffen	Section	=> Vol. 3: Shell and Utilities.
2021-03-09 21:21	steffen	Page Number	=> (page or range of pages)
2021-03-09 21:21	steffen	Line Number	=> (Line or range of lines)
2021-03-10 04:11	emaste	Issue Monitored: emaste
2022-02-22 15:58	geoffclare	Note Added: 0005700
2022-02-22 16:01	geoffclare	Note Edited: 0005700
2022-02-22 16:09	geoffclare	Note Edited: 0005700
2022-02-22 16:11	geoffclare	Project	1003.1(2008)/Issue 7 => 1003.1(2016/18)/Issue7+TC2
2022-02-22 16:13	geoffclare	Note Edited: 0005700
2022-02-22 21:39	emaste	Note Added: 0005701
2022-02-22 21:41	emaste	Note Deleted: 0005701
2022-02-22 22:19	steffen	Note Added: 0005702
2022-02-22 22:34	calestyo	Issue Monitored: calestyo
2022-02-23 08:32	geoffclare	Note Added: 0005703
2022-02-23 13:31	emaste	Note Added: 0005704
2022-02-23 13:38	calestyo	Note Added: 0005705
2022-02-24 09:28	geoffclare	Note Added: 0005706
2022-02-24 09:32	geoffclare	Note Edited: 0005700
2022-02-24 09:34	geoffclare	Note Edited: 0005700
2022-02-24 09:36	geoffclare	Note Edited: 0005703
2022-02-25 21:24	mirabilos	Note Added: 0005721
2022-07-21 10:41	geoffclare	Note Added: 0005898
2022-07-21 10:42	geoffclare	Note Edited: 0005898
2022-07-21 10:45	geoffclare	Note Added: 0005899
2022-07-21 10:47	geoffclare	Note Edited: 0005899
2022-07-21 10:57	geoffclare	Note Edited: 0005898
2022-07-21 11:15	kre	Note Added: 0005900
2022-07-21 11:45	kre	Note Added: 0005901
2022-07-29 14:16	geoffclare	Note Added: 0005921
2022-11-21 16:12	geoffclare	Note Added: 0006079
2022-11-21 16:13	geoffclare	Interp Status	=> ---
2022-11-21 16:13	geoffclare	Final Accepted Text	=> Note: 0006079
2022-11-21 16:13	geoffclare	Status	Under Review => Resolved
2022-11-21 16:13	geoffclare	Resolution	Open => Accepted As Marked
2022-11-21 16:14	geoffclare	Tag Attached: issue8
2022-11-24 10:09	geoffclare	Status	Resolved => Applied
2023-03-22 12:55	eblake	Note Added: 0006219
2023-03-22 12:55	eblake	Summary	Add readlink(1) utility => Add readlink(1), realpath(1) utility
2023-03-22 14:31	eblake	Note Added: 0006221
2024-06-11 09:08	agadmin	Status	Applied => Closed

Relationships

Aardvark Mark IV