Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000267 [1003.1(2004)/Issue 6] Shell and Utilities Objection Clarification Requested 2010-06-25 16:19 2020-02-06 10:02
Reporter Don Cragun View Status public  
Assigned To ajosey
Priority normal Resolution Reopened  
Status Under Review  
Name Don Cragun
Organization
User Reference
Section time
Page Number 916-919
Line Number 35505-35633
Interp Status ---
Final Accepted Text WAS Note: 0000442 and Note: 0001692
Summary 0000267: time (keyword)
Description  The rationale on XCU P918, L35624-35630 notes that time is a
 reserved word in ksh and says that the POSIX definition of time
 is intended to allow time to be implemented as a stand-alone
 utility or as a reserved word in the shell. Furthermore, the
 normative text on P916, L35522-35527 make the behavior of time
 unspecified when the utility operand is a pipeline.

 But, when time is a keyword, output redirection does not behave
 as specified by the standard. In particular, the command:
         time utility 2> timeout
 is required by the standard to write the timing statistics from
 invoking utility in a file named timeout. However, in ksh88,
 ksh93, bash, and zsh the file timeout will be empty unless the
 invocation of utility writes data to stderr. The stderr from
 the time command itself in these shells will be sent to stderr
 of the shell itself; not to the stderr output of the utility
 being timed.

 Furthermore, the standard behavior of time would be
 significantly improved (allowing timing of pipelines and timing
 of compound commands (e.g., for, while, and until loops; case
 statements; if statements) as well as timing individual invoked
 utilities if the standard really made time a reserved word.

 Since the rationale says that the intent is that time can be a
 shell reserved word, but the stderr section of the description
 doesn't allow time to be treated as a reserved word, I request
 a formal interpretation allowing implementation that have made
 time a reserved word to conform even though they do not perform
 stderr redirections as described by the current standard.
Desired Action  Issue an interpretation saying the standard is ambiguous on
 this issue, thereby allowing time to be a stand-alone utility
 or a reserved word.

 In the next revision of the standard make time a reserved word.
Tags No tags attached.
Attached Files

- Relationships
related to 0000789Resolvedajosey 1003.1(2008)/Issue 7 Add set -o pipefail 
has duplicate 0001051Closed 1003.1(2013)/Issue7+TC1 stderr redirection of the "time" utility 
related to 0000723Closed 1003.1(2013)/Issue7+TC1 time is not allowed to write error messages to stderr 

-  Notes
(0000437)
geoffclare (manager)
2010-06-25 16:20

Copied from xcubug2.txt ERN 183
(0000442)
geoffclare (manager)
2010-06-26 17:52

Interpretation response
------------------------
The standard is unclear on this issue, and no conformance distinction can be made between alternative implementations based on this. This is being referred to the sponsor.

Rationale:
-------------
None.

Notes to the Editor (not part of this interpretation):
-------------------------------------------------------
David Korn joined the call especially for this topic.
David agreed to look at propose some wording, the new words
will require both a time keyword and a time utility
for the next revision.

The time keyword will apply when time is not quoted as long as the first
operand to time does not begin with a minus sign, otherwise the time
utility will be used

----
The changes are as follows:
1. In section 2.4, add "time" to the list of reserved words on P2301,
    L72478-72481 maintaining alphabetic order.

2. In section 2.4 add the following after P2301, L72487:
        "however, the word \fBtime\fP shall not be
         recognized as a reserved word if the first
         character of the next token is '-' (<hyphen>)."
    indented to the same level as P2301, L72482-72483.

3. Add a new item to the list in section 2.5.3 after P2304, L72644:
    TIMEFORMAT The output format to be used to write timing
                        information produced by the shell for
                        a timed pipeline (see
                        Section 2.9.2.5, on page XXX) or a shelltime
                        pipeline (see Section 2.9.2.4, on page XXX) and
                        by the \fItime\fP utility (see time, on page
                        XXX) when the -p option is not specified. The
                        '%' character introduces a format sequence that
                        shall be expanded as follows:

                        %% A literal %.
                        %[p][l]R The elapsed time in seconds.
                        %[p][l]U The User CPU time, equivalent to
                                        the sum of the tms_utime and
                                        tms_cutime fields returned by
                                        the times() function defined in
                                        [xref to times()], in seconds.
                        %[p][l]S The System CPU time, equivalent
                                        to the sum of the tms_stime and
                                        tms_cstime fields returned by
                                        the times() function, in seconds.
                        %P The CPU percentage, computed as
                                        (U + S) / R, or zero if R is zero.

                        The optional p is a decimal digit specifying
                        the precision, the number of digits after the
                        decimal point. A value of 0 shall not display
                        the decimal point. The optional l specifies a
                        longer format, including hours if greater than
                        zero. For %P, the output format shall be:
                                "%.2f",
                                   (<user time> + <system time>) / <real time>

                        In the following three formats, <time> is the
                        <real time>, <user time>, or <system time> for
                        %R, %U, or %S, respectively.

                        For %R, %U, and %S, if l is not specified, the
                        output format shall be:
                                "%.

f", <time>

                        In the POSIX locale, if l is specified and the
                        time is greater than or equal to one hour, the
                        output format shall be:
                                "%dh%dm%02d%.

fs", (int)<time>/3600,
                                        ((int)<time>%3600)/60,
                                        ((int)<time>%60),
                                        <time>-(int)<time>

                        In the POSIX locale, if l is specified and the
                        time is less than one hour, the
                        output format shall be:
                                "%dm%02d%.

fs", ((int)<time>)/60,
                                        ((int)<time>)%60),
                                        <time>-(int)<time>

[[Note: the editors might prefer to use an italic p rather than

.]]
                        In the above three formats,

in the output
                        format represents the optional p specified
                        after the % in TIMEFORMAT. If the optional
                        p is not specified, the precision used
                        shall be sufficiently precise to accommodate
                        the size of the clock tick on the system (for
                        example, if there were 60 clock ticks per
                        second, at least two digits shall follow the
                        radix character). Unless p is specified to be
                        0, the number of digits following the radix
                        character shall be no less than one, even if
                        this always results in a trailing zero. If p
                        is specified to be 0, the radix character shall
                        not be written.

                        All other characters in TIMEFORMAT shall be output
                        without change and a trailing newline shall be added.

4. Change the entire section 2.9.2 Pipelines on P2318-2319,
    L73174-73191 to:
    \fB2.9.2 Pipelines\fP

                A \fIpipeline\fP is an optional '!' reserved word
                followed by a sequence of zero or more commands
                separated by the control operator '|'. A pipeline
                not beginning with '!' shall contain one or more
                commands. When a pipeline contains two or more
                commands, the standard
                output of all but the last command shall be connected
                to the standard input of the next command.

    \fI2.9.2.1 Unadorned Pipeline\fP

                The format for an unadorned pipeline is:

                command [ | pipeline ]

                The standard output of command shall be connected to
                the standard input of the first command in pipeline.
                The standard input, standard output, or both of a
                command shall be considered to be assigned by the
                pipeline before any redirection specified by
                redirection operators that are part of the command (see
                Section 2.7, on page XXX). If the pipeline is not in
                the background (see Section 2.9.3.1, on page XXX), the
                shell shall wait for the last command specified in the
                pipeline to complete, and may also wait for all
                commands to complete.

                \fBExit Status\fP

                The exit status of an unadorned pipeline shall be the
                exit status of the last element of the pipeline.

    \fI2.9.2.2 False Pipeline\fP

                The format for a false pipeline is:
                !

                A false pipeline shall have no effect other than
                setting its exit status.

                Exit Status\fP

                The exit status of a false pipeline shall be 1.

                \fBExamples\fP
                The pipeline:
                \fCW!\fP

                is roughly equivalent to the command:
                \fCWfalse\fP

    \fI2.9.2.3 Negated Pipeline\fP

                The format for a negated pipeline is:

                ! pipeline

                A negated pipeline shall be processed in the same way
                as the given pipeline, except that the exit status of
                the negated pipeline shall be the logical NOT of the
                exit status of the given pipeline.


                \fBExit Status\fP

                If the last element of the pipline returns zero, the
                exit status of the negated pipeline shall be 1; if the
                last element of the pipeline returns greater than zero,
                the exit status of the negated pipeline shall be zero.

                \fBExamples\fP

                The pipeline:

                \fCW! !\fP

                is roughly equivalent to the pipeline:

                \fCWtrue\fP

    \fI2.9.2.4 Shelltime Pipeline\fP

                The format for a shelltime pipeline is:

                \fBtime\fP

                If a pipeline is the \fBtime\fP reserved word, the time
                consumed by the current shell and all completed
                commands run by the current shell shall be output to
                standard error. If set, the TIMEFORMAT
                shell variable (see Section 2.5.3, on page XXX)
                determines the format of the time information.
                Otherwise, in the POSIX locale the output format shall
                be as if the TIMEFORMAT shell variable were set to a
                string containing a concatenation of the string "user",
                a <tab> character, the string "%2lU", a <newline>
                character, the string "sys", a <tab> character, and the
                string "%2lS".

                \fBExit Status\fP

                The exit status of a shelltime pipeline shall be 0.

                \fBExamples\fP

                Enclosing a shelltime pipeline in a brace group allows
                the output to be redirected:

                \fCW{ time;} 2>time_output\fP

                This is necessary because if redirections follow the
                \fBtime\fP reserved word the result is a timed
                pipeline instead of a shelltime pipeline.

    \fI2.9.2.5 Timed Pipeline\fP

                The format for a timed pipeline is:

                \fBtime\fP pipeline

                If a pipeline is preceded by the \fBtime\fP reserved
                word, the time consumed by all elements of the pipeline
                shall be output to standard error. If set, the
                TIMEFORMAT shell variable determines the format
                of the time information. Otherwise, in the POSIX locale
                the output format
                shall be as if the TIMEFORMAT shell variable were
                set to a string containing a concatenation of a
                <newline> character, the string "real", a <tab>
                character, the string "%2lR", a <newline> character,
                the string "user", a <tab> character, the string
                "%2lU", a <newline> character, the string "sys", a
                <tab> character, and the string "%2lS".

                \fBExit Status\fP

                The exit status of a timed pipeline shall be the exit
                status of the specified pipeline.

                \fBExamples\fP

                Enclosing a timed pipeline in a brace group allows
                the timing output to be redirected along with the
                standard error from the pipeline:

                \fCW{ time utility;} 2>utility_stderr+time_output\fP

                Without the brace group, only the standard error of
                the pipeline would be redirected and the timing
                output would go to the shell's standard error.

5. In section 2.9.3.1 on P2319, L73225, change:
        command1 & [command2 & ... ]
    to:
        command1 & [command2 & ] ...

6. In section 2.9.5 on P2324, L73379, change:
        fname() compound-command[io-redirect ...]
   to:
        fname() compound-command [io-redirect ...]

7. In section 2.10.2, change the text of the first paragraph of Rule 1
    on P2325, L73433-73435 from:
        When the \fBTOKEN\fP is exactly a reserved word, the token
        identifier for that reserved word shall result. Otherwise, the
        token \fBWORD\fP shall be returned. Also, if the parser is in
        any state where only a reserved word could be the next correct
        token, proceed as above.
    to:
        When the \fBTOKEN\fP is exactly a reserved word, the token
        identifier for that reserved word shall result unless:
        1. the \fBTOKEN\fP is not in a state where a reserved word is
            allowed as described in Section 2.4., or
        2. the \fBTOKEN\fP is \fBtime\fP and the first character of the
            next \fBTOKEN\fP is a '-' character.
        Otherwise, the token \fBWORD\fP shall be returned.

8. In section 2.10.2, add the token Time by changing P2327,
    L73514-73515 from:
        %token Lbrace Rbrace Bang
        /* '{' '}' '!' */
    to:
        %token Lbrace Rbrace Bang Time
        /* '{' '}' '!' 'time' */

9. In section 2.10.2, change the grammar for pipeline on P2327,
    L73533-73535 from:
        pipeline : pipe_sequence
                        | Bang pipe_sequence
                        ;
    to:
        pipeline : pipe_sequence
                        | Bang linebreak
                        | Bang pipeline
                        | Time linebreak
                        | Time pipeline
                        ;

[[Item 10 now merged into 14.]]

11. Delete the last full paragraph of the DESCRIPTION section of the
    time utility:
        When time is used as part of a pipeline, the times reported are
        unspecified, except when it is the sole command within a
        grouping command (see Section 2.9.4.1, on page 2321) in that
        pipeline. For example, the commands on the left are
        unspecified; those on the right report on utilities a and c,
        respectively:

        time a | b | c {time a; } | b | c
        a | b | time c a | b | (time c)
    from P3232, L107784-107789.

12. Add the following description of the TIMEFORMAT environment variable
    in the ENVIRONMENT VARIABLES section after P3233, L107822:
        TIMEFORMAT
                Determine the time statistics to be written and the
                format used when writing the statistics. The format of
                the TIMEFORMAT environment variable is as specified in
                Shell Variables (see section 2.5.3, on page XXX). The
                TIMEFORMAT environment variable shall be ignored if the
                -p option is specified on the command line.

13. Change the entire STDERR section on P3233, L107828-107837 (note that
    this incorporates the change specified in xcubug2.txt for ERN #184):
        Timing statistics shall be written to standard error.

        If -p is specified, the following format shall be used in the
        POSIX locale:

        "real %f\enuser %f\ensys %f\en", <real seconds>, <user seconds>,
                <system seconds>

        where each floating-point number shall be expressed in
        seconds. The precision used may be less than the default six
        digits of %f, but shall be sufficiently precise to accommodate
        the size of the clock tick on the system (for example, if there
        were 60 clock ticks per second, at least two digits shall
        follow the radix character). The number of digits following
        the radix character shall be no less than one, even if this
        always results in a trailing zero. The implementation may
        append white space and additional information following the
        format shown here, terminated by a <newline>.

        If the -p option is not specified and the TIMEFORMAT
        environment variable is not set, the information specified
        above shall be written, but the format and precision are
        unspecified.

        If the -p option is not specified and the TIMEFORMAT
        environment variable is set, the information specified by the
        TIMEFORMAT environment variable shall determine the statistics
        written and the format used.

        If the -p option is specified or the TIMEFORMAT environment
        variable is set, the implementation may also output a single
        empty line before the specified output.

14. In the APPLICATION USAGE section for time, add the following new
    paragraphs after P3234, L107860:
        Since \fBtime\fP is a reserved word in the shell command
        language, to invoke the \fItime\fP utility from the shell,
        either quote the word "time" or invoke it with an option or the
        "--" end of options indicator to prevent the shell from
        invoking the actions associated with the \fBtime\fP reserved
        word.

        Note that TIMEFORMAT needs to be an environment variable
        (that is, it needs to be exported) to have any effect on the
        \fItime\fP utility, but only needs to be a shell variable
        (it does not need to be exported) to affect \fBtime\fP
        reserved word output.

15. Replace the entire contents of the EXAMPLES section of the time
    utility on P3234, L107862-107866 with:
        The following examples use the \fBtime\fP reserved word or the
        \fItime\fP utility to time one or more elements of a pipeline:
        1. The \fBtime\fP reserved word reports the time used by
            utilities \fBa\fP, \fBb\fP, and \fBc\fP.
                \fCWtime a | b | c\fP

        2. The \fItime\fP utility reports the time used by utility
            \fBa\fP.
                \fCW"time" a | b | c\fP

        3. The \fBtime\fP reserved word reports the time used by
            utility \fBb\fP:
                \fCWa | { time b;} | c\fP

        4. The \fItime\fP utility reports the time used by utility
            \fBb\fP:
                \fCWa | time -p b | c\fP

        5. The \fBtime\fP reserved word reports the time used by
            utilities \fBb\fP and \fBc\fP:
                \fCWa | time b | c\fP

        6. Use the \fBtime\fP reserved word to capture standard error output
            from a utility in one file and the timing statistics for
            the execution of that utility in another file:
                \fCW{ time utility 2>utility_stderr;} 2>time_output\fP

        7. Use the \fItime\fP utility to capture standard error output
            from a utility and the timing statistics from the execution
            of that utility in a file:
                \fCWtime -- utility 2>time_output+utility_stderr\fP
            or
                \fCW2>time_output+utility_stderr time utility\fP

16. In the RATIONALE section for time, change the two bullets on P3234,
    L107871-107875 from: (Note that this change is fixing editorial
    glitches; not the intent of the rationale.)
        * The underlying CPU definitions from the System Interfaces
          volume of POSIX.1-2008 are vague, so the numeric output could
          not be compared accurately between systems or even between
          invocations.
        * The creation of portable benchmark programs was outside the
          scope this volume of POSIX.1-2008.
    to:
        * The underlying CPU definitions from ISO POSIX.1-1990 were
          vague, so the numeric output could not be compared accurately
          between systems or even between invocations.
        * The creation of portable benchmark programs was outside the
          scope of ISO POSIX.2-1993.

17. In the RATIONALE section for time, change the paragraph on P3234,
    L107882-107885 from:
        However, \fItime\fP does fit in the scope of user portability.
        Human judgement can be applied to the analysis of the output,
        and it could be very useful in hands-on debugging of
        applications or in providing subjective measures of system
        performance. Hence it has been included in this volume of
        POSIX.1-200x.
    to:
        However, \fItime\fP did fit in the scope of the user portability
        option. Human judgement can be applied to the analysis of the
        output, and it could be very useful in hands-on debugging of
        applications or in providing subjective measures of system
        performance. Hence it was included in the User Portability
        Utilities option of ISO POSIX.2-1993. It was moved from the
        User Portability Utilities option into the base in this
        standard.

18. In the RATIONALE section for time, change the last two paragraphs
    of the RATIONALE on P3234, L107883-107892 from:
        In the KornShell, \fItime\fP (sic) is a shell reserved word
        that can be used to time an entire pipeline, rather than just a
        simple command. The POSIX definition has been worded to allow
        this implementation. Consideration was given to invalidating
        this approach because of the historical model from the C shell
        and System V shell. However, since the System V \fItime\fP
        utility historically has not produced accurate results in
        pipeline timing (because the constituent processes are not all
        owned by the same parent process, as allowed by POSIX), it did
        not seem worthwhile to break historical KornShell usage.

        The term utility is used, rather than command, to highlight the
        fact that shell compound commands, pipelines, special
        built-ins, and so on, cannot be used directly. However,
        utility includes user application programs and shell scripts,
        not just the standard utilities.
    to:
        In the KornShell, \fBtime\fP is a shell reserved word that can
        be used to time an entire pipeline, rather than just a simple
        command. In versions of this standard before POSIX.1-201x, the
        description of the time utility was intended to be worded to
        allow this implementation. Consideration was given to
        invalidating this approach because of the historical model from
        the C shell and System V shell. However, since the System V
        \fItime\fP utility historically did not produce accurate
        results in pipeline timing (because the constituent processes
        are not all owned by the same parent process, as allowed by
        POSIX), it did not seem worthwhile to break historical
        KornShell usage. But, due to differences in file redirections
        between utilities and shell reserved words, the redirection
        requirements were ambiguous.

        The term utility is used in the description of the \fItime\fP
        utility, rather than command, to highlight the fact that shell
        compound commands, pipelines, special built-ins, and so on,
        cannot be used directly. However, utility includes user
        application programs and shell scripts, not just the standard
        utilities.

        This issue of the standard requires both a \fBtime\fP reserved
        word (see Section 2.9.2.5, on page XXX) that provides timing
        results for compound commands, pipelines, special built-ins,
        and so on) and the \fItime\fP utility described here that times
        one \fIutility\fP.

        The TIMEFORMAT shell variable was invented to control output
        from the \fBtime\fP reserved word (since reserved words don't
        accept options such as the \ftime\fP utility's -p option).
        For consistency, this standard requires that TIMEFORMAT (as
        an environment variable) also controls the output from the
        \fPtime\fP utility, when invoked without -p. If TIMEFORMAT
        is not set and -p is not used, the output format is unspecified.

(0000493)
ajosey (manager)
2010-07-30 08:19

Comments/objections on the proposed interpretation are due by COB Aug 31 2010
(0001692)
geoffclare (manager)
2013-08-01 15:13
edited on: 2013-08-01 15:15

Extra note to the editor:

Re-apply the changes from 0000723 to the STDERR section after the changes for this bug.

(0003199)
stephane (reporter)
2016-05-05 22:18
edited on: 2016-05-06 07:46

I'm seeing this bug for the first time 6 years later and have got a few
comments and objections.

The approved changes introduce radical chances to the grammar that are
unrelated to the time utility or keyword, that would break most shell
implementations and are uncalled for IMO.

The change now allows and specifies

  true | ! true

which breaks bash, yash, zsh and dash implementations. It would also
complicate/confuse the specification of "pipefail" (bug 789), and would be a hinderance for shells that implement a $pipestatus or $PIPESTATUS array like zsh and bash.

ksh -c ': | ! false | true' returns false, that is the ! applied to the
pipeline to the right of it. I can't think of any reason why one may
want to use "cmd1 | ! cmd2 | cmd3". With a pipefail option, that could
make sense if the "!" applied to only one command (to cmd2), otherwise
it doesn't (to me).

It also contradicts:

 > The exit status of an unadorned pipeline shall be the
 > exit status of the last element of the pipeline.

The "!" false pipeline is uncalled for. It's redundant with the "false"
utility (or "! true" or "test"...). I could understand a "!" that is
short for [ "$?" -ne 0 ], as in cmd; if !; then...; fi ("if not"), but
"!" alone meaning "false" is not particularly intuitive. The current
version of the spec makes it unspecified, I can't see a good reason to
change it. It also breaks dash, mksh and yash. Even ksh93 fails on
things like "! &" (it also fails on "> file &" btw which looks like a
conformance bug)

! ! cmd breaks dash and zsh and is not useful except to change a
non-zero $? to 1 (which you can do with [ "$?" -eq 0 ]).

What are changes 5 and 6 doing here? Are they related to the "time"
utility? Are they only about formatting?

The "time" pipeline is redundant with the "times" utility

The grammar section is not consistent with the textual sections above.
The textual sections made a point of introducing different sorts of
pipelines beside the simple "pipeline" (changed to allow "!" in front of
any command in the pipeline), but the grammar lumps everything together
allowing "! time time ! time"...

time -- cmd
time -p cmd
< redir time cmd
TIMEFORMAT=x time cmd
"time" cmd

All would disable the "time" keyword and mean the time utility is called
instead.

time "-p" cmd
or
time ${POSIXLY_CORRECT+-p} cmd
or
time < redir -p cmd

wouldn't though and would try to run the "-p" command. It may be worth
pointing out especially considering that's not backward compatible with the
current version of the spec (which at the moment says they should run the time utility and produce a "portable"/"parsable"/"posix" output).

As a history note, The TIMEFORMAT variable was added to bash in 1996
(2.0), copied by ksh93 in 2003 (ksh93o) but it looks like zsh was the
first Bourne-like shell to introduce it, though it is called TIMEFMT
there.

I don't know why bash decided to call it TIMEFORMAT instead. Why it used
%R instead of %E makes even less sense to me (especially considering
that GNU time used %R for something else like tcsh long before that). I suppose it's to match the "real" label output by historical implementations as in the POSIX format (the man pages in historical implementations of time(1) have always refered to "elapsed time" (since Unix v4 which apparently was the first to provide it), but the code seems to always have output "real" (at least since Unix v5)).

The feature probably originated in tcsh in the late 80s. Maybe bash and
zsh copied it independently from there. In tcsh:

set time = (10 "%U %E %P...")
time cmd

(cmds that take more than 10 seconds to run would be timed
automatically).


Already in 2.00.03 in 1991, zsh had for TIMEFMT:

  %U CPU seconds spent in user mode.
  %S CPU seconds spent in kernel mode.
  %E Elapsed time in seconds.
  %P The CPU percentage, computed as (%U+%S)/%E.
  %W Number of times the process was swapped.
  %X The average amount in (shared) text space used in Kbytes.
  %D The average amount in (unshared) data/stack space used in Kbytes.
  %K The total space used (%X+%D) in Kbytes.
  %M The maximum memory the process had in use at any time in Kbytes.
  %F The number of major page faults (page needed to be brought from disk).
  %R The number of minor page faults.
  %I The number of input operations.
  %O The number of output operations.
  %r The number of socket messages received.
  %s The number of socket messages sent.
  %k The number of signals received.
  %w Number of voluntary context switches (waits).
  %c Number of involuntary context switches.

Using getrusage(). Like tcsh. GNU time also recognises those for its
"-f" (at least as far back as 1990) and acknowledges the tcsh heritage
in its doc. Looks like Joerg copied it as well for his Bourne shell
derivative (and uses %E for elapsed time as well not %R).

The alternate long form in zsh is with %*U instead of %lU.

Bash supports %E as an undocumented alias for %R. I'd rather the spec
settled for %E as most implementations support it, and %R means
something else in zsh, shily-sh, tcsh, GNU time -f.

The new text should leave the behaviour unspecified
for %<anything-other-than-PESU> to allow extensions like those.

Also, in:

time cmd1 | cmd2

zsh times both cmd1 and cmd2 independently which is a lot more useful.
It would be good if the spec at least allowed that behaviour.

Some references to "decimal point" in the new text for TIMEFORMAT should
be changed to "radix character" for consistency.

A note that the radix character should be based on locale would be good.

The exit status of a timed pipeline or of the time utility when time
fails to write the statistics to stderr may need to be clarified as
well (GNU time returns with $?=1 there).

Now, IMO, if the "time" keyword is introduced, the grammar should be:

pipeline = pipesequence
          | ! pipesequence
      | time pipesequence
      | time ! pipesequence

No ! or time alone, no ! !. Behaviour unspecified in cmd1 | time cmd2 or
cmd1 | ! cmd2. Use cmd1 | "time" cmd2 to use the "time" utility or cmd1
| { time cmd2 | cmd3; } to time cmd2 | cmd3.

I'd rather we avoid the next token starting with "-" nonsense. But looks
like we can't do much about it as "time -p" is already out of the bag
even if the current specification is broken. We could deprecate it
though for the "time" keyword (that is introduce it only as a transition
to avoid breaking existing scripts).

Keep the "-p" option for the "time" utility (as then the current spec is
not broken), but encourage people to use $TIMEFORMAT for the "time"
keyword instead. And quote "time" to disable the keyword.

In APPLICATION USAGE, some note that:

  TIMEFORMAT=%U time cmd1 | cmd2

actually invokes the time utility and thus times cmd1 only.
Use TIMEFORMAT=%U; time cmd1 | cmd2 to invoke the keyword.

Maybe an example usage to retrieve the statistics of a pipeline:

{
  stats=$(
    exec 2>&1 >&3 3>&-
    TIMEFORMAT='user=%U sys=%S elapsed=%E pcpu=%P'
    time { cmd1 | cmd2; } 2>&4 4>&-
  )
} 3>&1 4>&2
cmd2_exit_status=$?
eval "$stats"

(0003202)
joerg (reporter)
2016-05-09 10:06

@Note: 0003199

First a hint for readability: If you like to refer to the current
versions of the Bourne Shell and do this in a list of filesystem
names, that contain names like ksh93, bash, mksh, ... please use
the name "bosh".

I created the name "bosh" together with David Korn on a
OpenSolaris core developers meeting in 2008 on the beach in
Santa Cruz during a social game. "bosh" is used since then as
an alternate name that allows to specify the shell in an
unabiguous way.

Now to your other statements:

  true | ! true

works with ksh88, ksh93, bosh and mksh.

It however does not work with bash, dash, yash, zsh. I would
not really care about yash and zsh that to not really try to be
POSIX compliant, but it may make sense to send a bug report to
the bash and dash people as this definitely is a bug because
assing support to "!" into the Bourne Shell was a small and easy
modification, and lead to the expected behavior.

Could you please explain what your point is when discussing

  cmd | ! cmd2

This is a straightforward command that fits the shell syntax and
there is no need to discuss whether it makes sense. A shell
implementation that fails to pare it correctly may have other problems
in it's parser as well.

BTW: ! ! fails with the same shells as cmd | ! cmd2, so this seems
to be just another problem with their parser that should be fixed.

"times" does not make "time" redundant. It satisfies a slightly
different purpose.

Timing commands inside csh work since a really long time. The
code was already present October 9 1980 when csh was put under
control of SCCS. You are mistaken with "tcsh in the late 1980s".
The idea of inline timing only when a certain CPU time limit
is reached however seems to be from tcsh in the late 1980s.

The Bourne Shell uses the same letters as used in csh in 1980
already and this is why it uses %E for the elapsed time.

Given that bosh offers more features with TIMEFORMAT than the
other shells, I am OK with standardizing it only as long as this
does not break the bosh extensions.
(0003209)
stephane (reporter)
2016-05-11 11:38

There's also the question of *when* the TIMEFORMAT variable should/can/may/may not be set.

In both bash and ksh93, it's the value of that variable at the time the statistics are printed that matters:

$ bash -c 'TIMEFORMAT=x; time { TIMEFORMAT=y; time true; }'
y
y
$ ksh -c 'TIMEFORMAT=x; time { TIMEFORMAT=y; time true; }'
y
y
$ ksh -c 'TIMEFORMAT=x; time { TIMEFORMAT=y; time echo z | read TIMEFORMAT; }'
z
z

Something to bear in mind when timing a function that may set TIMEFORMAT, or use

time { cmd; TIMEFORMAT=myformat; }

instead of

TIMEFORMAT=myformat; time cmd

to be on the safe side.
(0003210)
joerg (reporter)
2016-05-11 11:57
edited on: 2016-05-11 13:43

@ Note: 0003209

You did not mention that the third example when used with bash prints
an "y".

But this may be a result of when bash "waits" for the pipeline or how
bash implements "time" in general.

ksh93 and bosh both implement "time" to time the whole pipeline but
they lookup the "TIMEFORMAT" value while in the function that prints
the timing results.

If you like a different behavior, the shell interpreter would need to
prefetch the value of "TIMEFORMAT" before running the pipeline and then
pass this string (that may have been free()d from a value change
operation already) to the time printing routine.

(0003211)
stephane (reporter)
2016-05-11 12:31
edited on: 2016-05-11 13:42

I feel like the comments made by Jilles at http://thread.gmane.org/gmane.comp.standards.posix.austin.general/2424/focus=2434 [^] in 2010 should be included here.

The text as proposed doesn't cover for the various behaviour of shells in:

# helper function that runs a busy loop for a given amount of seconds. Here
# using GNU timeout, adapt to your system.
busysleep() { timeout "$1" wc /dev/zero; }

busysleep 1 & time sleep 2 # accounting busysleep's CPU time
                           # in bash, ksh93, not mksh nor zsh

time { busysleep 1 & sleep 2; } # mksh not accounting busysleep's CPU time

busysleep 1 & (time sleep 2) # or
busysleep 1 & time (sleep 2) # in those cases bash (still not ksh93 as ksh93
                             # doesn't run subshells in child processes) no
                             # longer accounts busysleep's CPU time

busysleep 2 & time sleep 1
# or
time { busysleep 2 & sleep 1; } # busysleep's usage not accounted
                                # in any shell since busysleep has not
                                # returned by the time the report is printed

busysleep 1 & time wait # busysleep's CPU time accounted even in mksh

In zsh, "time" doesn't work for things that don't run in a child process (compound commands, functions, builtins...) for which you need a subshell

  time (compound or builtin)

time (pipeline)

works reliably in that instance (times all waited-for processes started by the pipeline) in all but ksh93. In bash, redirecting the command substitution also has the side effect of redirecting the timing output, so for portability, you'd need:

time ({ pipeline; } > redirs)

That also avoids the problem of where to define TIMEFORMAT.

All in all, it's not very different from

time sh -c 'pipeline > redirs'

as per the current spec (except that it also times sh's initialisation, command line parsing and cleanup).

(0003212)
joerg (reporter)
2016-05-11 12:47
edited on: 2016-05-11 12:49

@ Note: 0003211

The "problem" with "busysleep" may be influenced by two facts:

- if a shell uses the non-standard wait3(), it could retrieve
  resource usage information separately.

- if a shell uses the standard waitid() + getrusage(), the
  results depend on when the shell waits for the background
  process.

It seems that trying to standardize a solution for this problem
would either need a new POSIX interface: waitid5() - with integrated
pointer to a struct rusage or would go beyond the accepted influence
in the implementation details for POSIX.

For timing a pipeline, even wait3() or waitid5() would need extra
computations and extra job tracing in the shell.

(0003213)
stephane (reporter)
2016-05-11 13:14

@ Note: 0003212

From my guess by stracing mksh, mksh works around the problem by doing something like:

While waiting for a timed pipeline, upon reception of a SIGCHLD, it does a getrusage(children) before doing the waitpid() and another getrusage(children) after and if the process that just terminated was not started as part of that pipeline, its time (which we can determine by substracting the times for those 2 getrusages) is substracted in the final calculation.

Obviously, calling the "wait" utility affects that. And it gets complicated when you nest time keywords.

But, for the purpose of this specification, it's not as much *how* shells should do it, but *what* they should do. What CPU time should be included.

IMO, that should be the cpu time of all waited-for processes started by that pipeline which no implementation does in all cases, though mksh is the closest.
(0003214)
joerg (reporter)
2016-05-11 13:28
edited on: 2016-05-11 13:28

@ Note: 0003213

Then ksh93 should also work "correctly" as ksh93 waits for background
processes triggered by SIGCHLD.

What really happens is this: shells call getrusage() for self and children
before starting a pipeline and do the same after waiting for that pipeline.

If there is a finished background program in between, this does not affect
the difference that is computed for the longer running pipeline.

BTW: If you like to judge on the correctness of shell implementations,
you would need to check whether they correctly sum up all resources
for children's children or whether they only sum up the resources for
their direct children.

(0004771)
geoffclare (manager)
2020-02-06 10:01

I'm reopening this bug based on the notes added in 2016, discussion on the mailing list following those notes, and the fact that bug 0000789 has since been resolved with changes to 2.9.2 Pipelines (adding pipefail) which clash with those described here.

- Issue History
Date Modified Username Field Change
2010-06-25 16:19 geoffclare New Issue
2010-06-25 16:19 geoffclare Status New => Under Review
2010-06-25 16:19 geoffclare Assigned To => ajosey
2010-06-25 16:19 geoffclare Name => Don Cragun
2010-06-25 16:19 geoffclare Section => time
2010-06-25 16:19 geoffclare Page Number => 916-919
2010-06-25 16:19 geoffclare Line Number => 35505-35633
2010-06-25 16:19 geoffclare Interp Status => ---
2010-06-25 16:20 geoffclare Note Added: 0000437
2010-06-25 16:20 geoffclare Reporter geoffclare => Don Cragun
2010-06-26 17:52 geoffclare Note Added: 0000442
2010-06-26 17:54 geoffclare Interp Status --- => Pending
2010-06-26 17:54 geoffclare Final Accepted Text => Note: 0000442
2010-06-26 17:54 geoffclare Status Under Review => Interpretation Required
2010-06-26 17:54 geoffclare Resolution Open => Accepted As Marked
2010-07-30 08:19 ajosey Interp Status Pending => Proposed
2010-07-30 08:19 ajosey Note Added: 0000493
2010-09-03 16:38 ajosey Interp Status Proposed => Approved
2013-07-16 15:20 geoffclare Relationship added related to 0000723
2013-08-01 15:13 geoffclare Note Added: 0001692
2013-08-01 15:14 geoffclare Final Accepted Text Note: 0000442 => Note: 0000442 and Note: 0001692
2013-08-01 15:15 geoffclare Note Edited: 0001692
2016-05-04 15:10 geoffclare Relationship added related to 0001051
2016-05-04 18:21 shware_systems Tag Attached: issue8
2016-05-05 22:18 stephane Note Added: 0003199
2016-05-05 22:19 stephane Note Edited: 0003199
2016-05-06 07:46 stephane Note Edited: 0003199
2016-05-09 10:06 joerg Note Added: 0003202
2016-05-11 11:38 stephane Note Added: 0003209
2016-05-11 11:57 joerg Note Added: 0003210
2016-05-11 11:58 joerg Note Edited: 0003210
2016-05-11 11:59 joerg Note Edited: 0003210
2016-05-11 12:31 stephane Note Added: 0003211
2016-05-11 12:47 joerg Note Added: 0003212
2016-05-11 12:47 joerg Note Edited: 0003212
2016-05-11 12:49 joerg Note Edited: 0003212
2016-05-11 13:14 stephane Note Added: 0003213
2016-05-11 13:28 joerg Note Added: 0003214
2016-05-11 13:28 joerg Note Edited: 0003214
2016-05-11 13:42 stephane Note Edited: 0003211
2016-05-11 13:43 joerg Note Edited: 0003210
2017-06-08 15:08 geoffclare Relationship replaced has duplicate 0001051
2020-02-06 10:01 geoffclare Interp Status Approved => ---
2020-02-06 10:01 geoffclare Final Accepted Text Note: 0000442 and Note: 0001692 => WAS Note: 0000442 and Note: 0001692
2020-02-06 10:01 geoffclare Note Added: 0004771
2020-02-06 10:01 geoffclare Status Interpretation Required => Under Review
2020-02-06 10:01 geoffclare Resolution Accepted As Marked => Reopened
2020-02-06 10:01 geoffclare Tag Detached: issue8
2020-02-06 10:02 geoffclare Relationship added related to 0000789


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker