Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0000770 [1003.1(2013)/Issue7+TC1] Shell and Utilities Objection Enhancement Request 2013-10-14 17:33 2013-10-18 15:29
Reporter dwheeler View Status public  
Assigned To ajosey
Priority normal Resolution Open  
Status Under Review  
Name David A. Wheeler
Organization
User Reference
Section make
Page Number 2939-2953
Line Number 97219-97804
Interp Status ---
Final Accepted Text
Summary 0000770: Add support for .ONESHELL in make
Description It is sometimes preferable that all the commands in target rules be executed by a single invocation of the shell. Here are some reasons:
1. This can improve performance when there are many command lines, by avoiding the creation of extra processes.
2. The resulting commands tend to be simpler and easier to understand, since there is no more need to insert backslashes and “;” to work around current make’s limitations. This is especially true when there are if..else..fi, case...esac, “cd” commands, or here documents.
3. If alternative shells are supported (e.g., SHELL), it can be hard to use some languages (like Python) without a way to combine all lines into one command. (http://austingroupbugs.net/view.php?id=769). [^] This is not required for one-shell mode to be useful.

POSIX currently notes that there are advantages to a one-shell mode, and already recommends using “.ONESHELL” to implement it. Lines 97797- 97804 say: “The default in some advanced versions of make is to group all the command lines for a target and execute them using a single shell invocation; the System V method is to pass each line individually to a separate shell. The single-shell method has the advantages in performance and the lack of a requirement for many continued lines. However, converting to this newer method has caused portability problems with many historical makefiles, so the behavior with the POSIX makefile is specified to be the same as that of System V. It is suggested that the special target .ONESHELL be used as an implementation extension to achieve the single-shell grouping for a target or group of targets.”

Since users who do not include “.ONESHELL” in their makefiles get the traditional (System V) behavior, adding .ONESHELL support does not cause problems in makefiles without .ONESHELL.

GNU make supports the POSIX-recommended .ONESHELL, as documented in: “http://www.gnu.org/software/make/manual/html_node/One-Shell.html [^]”. “If the .ONESHELL special target appears anywhere in the makefile then all recipe lines for each target will be provided to a single invocation of the shell. Newlines between recipe lines will be preserved… If .ONESHELL is provided, then only the first line of the recipe will be checked for the special prefix characters (‘@’, ‘-’, and ‘+’)... As a special feature, if SHELL is determined to be a POSIX-style shell, the special prefix characters in “internal” recipe lines will removed before the recipe is processed. This feature is intended to allow existing makefiles to add the .ONESHELL special target and still run properly without extensive modifications. Since the special prefix characters are not legal at the beginning of a line in a POSIX shell script this is not a loss in functionality.”

The *BSD makes do not directly support .ONESHELL, but they do turn on a one-shell mode if the “-j” parameter is given, as described here: http://www.freebsd.org/cgi/man.cgi?query=make&sektion=1. [^]

This capability has advantages and has a number of implementations (with variances in how to enable it). I recommend that POSIX finally standardize the previous POSIX suggestion.
Desired Action Before line 97219, add:
“.ONESHELL Run all the commands for a given target using a single invocation of the shell. This strips off the leading tabs on commands, and preserves newlines between the commands. Only the first line of the recipe will be checked for the special prefix characters (‘@’, ‘-’, and ‘+’).”

If SHELL support is added, then also add after this: “If the SHELL is determined to be a POSIX-style shell (e.g., if it is unset, the special value "sh", or the special value "/bin/sh"), then the special prefix characters (‘@’, ‘-’, and ‘+’) in following command lines MAY be removed before the recipe is processed.”

On line 97187, add: “If the special target .ONESHELL is included, the sequence of commands is treated as if it were a single command; newlines are preserved.”

Also, delete lines 97797- 97804, .ONESHELL would now be included.
Tags No tags attached.
Attached Files

- Relationships
related to 0000769Closedajosey Add support for SHELL in make 

-  Notes
(0001906)
joerg (reporter)
2013-10-15 16:16
edited on: 2013-10-15 16:16

The performance problem no longer exists, as SunPro make and smake
already execute simple commands without using the shell. Smake in
addition implements the "echo" command inline and thus allows to speed
up the common case of commands like:

     @echo COMPILING $@; cc ....

So if we like to standardize something new, it should give real benefits.

Your current proposal does not look like a clever idea, as it does
not allow the user to configure the behavior on a per command base.

Let me for this reason propose an alternate method as a base for
discussions:

If a command line starts with the character '~', the next line is
expected to be concatenated with this line after adding a newline
to the current line. If a line is found that does not start with '~',
the accumulated command block is executed by the shell.

(0001907)
brkorb (reporter)
2013-10-15 17:22

The issue is not only the overhead of exec-ing shell, but also the writing of the script itself:

       for f in $(LIST) ; do
       something $$f
       done

versus:

       for f in $(LIST) ; do \
       something $$f ; \
       done

Expressive ability is part of the reason. And validly so. It is a "real benefit". "~" continuations are ugly.
(0001908)
dwheeler (reporter)
2013-10-15 18:06

I agree with brkorb. Clean, simple expressive ability is a "real benefit", and validly so. "~" continuations are ugly, in contrast, and typical implementations don't do that. Many makes, including GNU make and *BSD make, do this already.

joerg: "Your current proposal does not look like a clever idea, as it does not allow the user to configure the behavior on a per command base."

It doesn't need to be *clever* idea to be standardized. I do agree that this approach as written doesn't allow per-command configuration, but it's not clear to me that there's much call for that.

If there *is* a call for per-command configuration, I think it'd be better to handle that differently. E.G., once ".ONESHELL" is used, all following command lines could be marked as being done with one shell until some other command (maybe ".ENDONESHELL"). If we did that, I think it should be nestable to support include files, i.e., ".ENDONESHELL" would restore the PREVIOUS status of .ONESHELL. If you don't want to end it, then a single .ONESHELL would work as expected, as it'd continue through the whole file.

In the end, though, this standards body will determine if there's a need for per-command configuration. I'm skeptical, but if that's what is needed to get it passed, then great!

Thoughts?
(0001911)
dwheeler (reporter)
2013-10-15 19:04

Oh, an addendum. If we *wanted* to support ".ENDONESHELL", and be backwards-compatible, we could say that if there is *no* .ENDONESHELL, and there *is* .ONESHELL, then *all* commands are considered as ".ONESHELL". This would be trivial to implement; include a marker with each commandset as a flag ("was this in .ONESHELL?), and a separate flag to indicate if .ONESHELL is global.

But this is only if the standards committee things it's important to support per-command configuration. In the past, people have not bothered, so it's not clear there's a need. I'd be happy with or without that capability.
(0001915)
joerg (reporter)
2013-10-16 09:35

As mentioned before, the overhead of execing the shell may have been
a problem 20 years ago, but it currently is not. This is because modern
OS have become fast with exec. In addition, the fact that some make
implementations (SunPro make and smake) avoid using the shell for simple
commands (in special smake that also inlines "echo" avoids most shell calls).
BTW: on Solaris this saves aprox. 2% of the typical time for a larger project
and on the slow Win-DOS, this saves aprox. 30% of the execution time. In
other words, for UNIX, the overhead is neglible.

If you don't like to use '~' for every line of a single shell block, we
could also standardize to use '~' for the first line of a command block
in order to make a whole block be executed by a single shell. This of
course has a big advantage on the .ONESHELL proposal as it would be less
ugly and as it makes it easier to read and understand Makefiles.

David: regarding the fact that I verified that BSD make does _not_
implement .ONESHELL, I like you to verify your other claims.

Regarding .ENDONESHELL, this does not fit nicely into the current make
practice. There however already _is_ something with a comparable
goal in smake since a longer time: .SPACE_IN_NAMES is used in
smake to allow to implement auto-dependencies for Win-DOS.

Lists if filenames that may contain spaces, are between:

.SPACE_IN_NAMES: true
...
.SPACE_IN_NAMES:

brackets.
(0001917)
dwheeler (reporter)
2013-10-16 12:43

Here's a reply to Note 1915:

"the overhead of execing the shell may have been a problem 20 years ago, but it currently is not... on the slow Win-DOS, this saves aprox. 30% of the execution time." Even if we accepted this claim (intentionally leaving Win-DOS users, including Cygwin users, to their terrible fate), that ignores points 2 and 3. Namely, the "resulting commands tend to be simpler and easier to understand" and "alternative shells". I think point 2, "simpler and easier", is a particularly good argument; if..then..else, case...esac, cd, and especially here documents are impacted by the current approach. Makefiles would be MUCH cleaner if we didn't need to infest them with "\" and ";".

"If you don't like to use '~' for every line of a single shell block, we
could also standardize to use '~' for the first line of a command block
in order to make a whole block be executed by a single shell. This of
course has a big advantage on the .ONESHELL proposal as it would be less
ugly and as it makes it easier to read and understand Makefiles."

This is not an insane idea. I agree that one advantage of this approach is that it'd make it easier to do selectively, and doing it only 1/command is better than every command. However, I see a lot of problems with this:
1. No one does this, to my knowledge. In contrast, GNU make already implements .ONESHELL, and BSD has a similar makefile-wide mode (though on the BSDs it's invoked in a different way).
2. POSIX already recommended .ONESHELL, which means that other make implementations may also implement this.
3. I think "~" is MORE ugly, not less, than .ONESHELL. You'd have to add this to every command for it to be used.
4. This would conflict with commands beginning with "~", e.g., "~/bin/my-subst ....". You could change "~" to "$$(HOME)", but I don't like creating easy opportunities for subtle errors through name conflicts. The "~" is a perfectly legal character on a shell command line.

That said, if the committee agrees to add "~" instead, I could live with that. I want the *capability*,


"Regarding the fact that I verified that BSD make does _not_
implement .ONESHELL, I like you to verify your other claims."

Um, what? I didn't make that claim, and I already gave citations. I think you misunderstood me, or you're replying to some other comment. But just to be clear, I'll repeat some of the points.

I never claimed that BSD make implements .ONESHELL, so I don't understand why you said I did. I said it had a similar mode. And, I already gave a citation. Here is what I said above (repeated): "The *BSD makes do not directly support .ONESHELL, but they do turn on a one-shell mode if the “-j” parameter is given, as described here: http://www.freebsd.org/cgi/man.cgi?query=make&sektion=1." [^] Note the URL as a citation.

I also gave a citation for GNU make's .ONESHELL implementation; here it is again: http://www.gnu.org/software/make/manual/html_node/One-Shell.html [^]

I also cited were POSIX (2013) currently notes the advantages of a one-shell mode and recommends .ONESHELL, namely, "Lines 97797- 97804 say....".


"Regarding .ENDONESHELL ....". I'm not really fond of .ENDONESHELL either. Current practice is to simply support ".ONESHELL" and use that as a global setting. I was just looking for a way to support common practice *and* your desire to enable scoping of it. If .ONESHELL is just added, with its typical global scope, that'd be fine with me.
(0001920)
joerg (reporter)
2013-10-16 13:28

Just if you don't remember, you wrote: "Many makes, including GNU make and *BSD make, do this already.", so you claimed that BSD make implements .ONESHELL.
This is why I checked the recent source and why I asked again after I could
not see the support.
(0001921)
dwheeler (reporter)
2013-10-16 14:24

In note 0001908 I did say: "Many makes, including GNU make and *BSD make, do this already", but the "this" antecedent was a "Clean, simple expressive ability", not the .ONESHELL syntax. Also, I *did* start this discussion with a description that explicitly said, "The *BSD makes do not directly support .ONESHELL, but they do turn on a one-shell mode if the “-j” parameter is given, as described here: http://www.freebsd.org/cgi/man.cgi?query=make&sektion=1." [^]

Anyway, glad that confusion's been cleared up. Now we can get back to the discussion about supporting a one-shell mode in make, and a standard syntax for achieving it.
(0001925)
joerg (reporter)
2013-10-17 10:07

I could live with .ONESHELL if this is not the only way to configure
this feature. I don't like concepts that are too coarse in order to
be usable for the forseable future, so if there is also an official
way to configure the feature at least on a per target command list
base, I am happy.

As mentioned before, with a typical modern make implementation,
there is usually no need for such a feature. Even worse, I am not
sure whether it would be wise to avoid shell calls with multi-line
command blocks that need to be processed by a single instance. So
enabling this feature could be counter productive. I however know
hat the current standard does not allow to have sed scripts inline
to makefiles.
(0001931)
dwheeler (reporter)
2013-10-17 22:46

Great, sounds like we're making progress! I certainly don't *object* to providing finer control when it's desired.

I don't like the leading "~" syntax, for reasons I described above. Perhaps we could borrow your ".SPACE_IN_NAMES:" approach, in the sense that ".ONESHELL:" could be followed by an optional parameter. If "true" follows, then the following commands are run with oneshell. If "false", then the following commands are NOT run with oneshell. If omitted, then every command in the makefile is run with oneshell (for backwards compatibility). ("previous" could be added to restore the previous value.) Result:
.ONESHELL: false
(STUFF)
.ONESHELL: true

I'm kind-of thinking out loud here. I'm flexible about the syntax details; the key is that it has to be something that people are willing to implement and use.
(0001932)
joerg (reporter)
2013-10-18 10:37
edited on: 2013-10-18 16:25

You are proposing something new here that does not refer to existing
practice.

We currently have two types of special targets in make.

- Simple ones like .SILENT: or .POSIX: that only allow to switch on a specific
  feature

- Structured ones like .SUFFIXES: that allow to clear a specific behavior
  if they appear without "dependencies".

(0001933)
dwheeler (reporter)
2013-10-18 15:29

(Replying to #1932): That's true, that would be something new. Do you have alternatives to suggest?

For purposes of the POSIX standard, I think supporting just ".ONESHELL:" as a global setting would be just fine. That syntax is ALREADY implemented by at least one "make" (GNU make), and I suspect it's implemented by others too. If people want to add a per-command setting, excellent. But existing implementations that support this capability always do it globally (GNU make with .ONESHELL, BSD make with options like -j), not per-command, which suggests to me that per-command setting of this capability is not considered a strong need.

- Issue History
Date Modified Username Field Change
2013-10-14 17:33 dwheeler New Issue
2013-10-14 17:33 dwheeler Status New => Under Review
2013-10-14 17:33 dwheeler Assigned To => ajosey
2013-10-14 17:33 dwheeler Name => David A. Wheeler
2013-10-14 17:33 dwheeler Section => make
2013-10-14 17:33 dwheeler Page Number => 2939-2953
2013-10-14 17:33 dwheeler Line Number => 97219-97804
2013-10-14 18:22 Don Cragun Project 1003.1(2008)/Issue 7 => 1003.1(2013)/Issue7+TC1
2013-10-15 16:16 joerg Note Added: 0001906
2013-10-15 16:16 joerg Note Edited: 0001906
2013-10-15 17:22 brkorb Note Added: 0001907
2013-10-15 18:06 dwheeler Note Added: 0001908
2013-10-15 19:04 dwheeler Note Added: 0001911
2013-10-16 09:35 joerg Note Added: 0001915
2013-10-16 12:43 dwheeler Note Added: 0001917
2013-10-16 13:28 joerg Note Added: 0001920
2013-10-16 14:24 dwheeler Note Added: 0001921
2013-10-17 10:07 joerg Note Added: 0001925
2013-10-17 22:46 dwheeler Note Added: 0001931
2013-10-18 10:37 joerg Note Added: 0001932
2013-10-18 15:29 dwheeler Note Added: 0001933
2013-10-18 16:25 joerg Note Edited: 0001932
2013-11-14 16:56 eblake Relationship added related to 0000769


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker