Austin Group Defect Tracker

Aardvark Mark IV


Viewing Issue Simple Details Jump to Notes ] Issue History ] Print ]
ID Category Severity Type Date Submitted Last Update
0001422 [Issue 8 drafts] Shell and Utilities Objection Error 2020-11-18 13:49 2021-03-25 14:24
Reporter joerg View Status public  
Assigned To
Priority normal Resolution Accepted As Marked  
Status Applied   Product Version Draft 1.1
Name Jörg Schilling
Organization
User Reference
Section
Page Number 2894-2895
Line Number 97277-97282
Final Accepted Text Note: 0005265
Summary 0001422: The current definition for the += assingment is in conflict with existing behavior of make and unpredictable/error-prone
Description The += operator has been introduced in January 1986 by SunPro Make.
It is defined to behave the same way as the = operator (to use delayed
expansion) regardless of how the assigned macro has been used before.

The current text in the standard does not seem to be compatible to long
existing behavior and it would be higly error-prone in complex makefiles
in special if these makefiles make use of the include directive.

Complex makefiles that make use of the include directive usually have
previous macro assignments in files from global make-include directories.
The author of a specific makefile thus cannot easily know whether there
was a previous = or ::= assinment for a specific macro since this would
require to manually scan many include files. Even worse: Which makefiles
are included is not easy to predic in object oriented makefile systems
that include different makefiles depending on the content of make
macros. As a result, the current standard text would cause
unpredictable results of the whole.

It seems to be better and more flexible to introduce a new operator +:= or
:+= for immediate expansion of the string to append, or to document this
work-around using the helper macro VAR2:

VAR2 := $(VAR3)
VAR = some text
VAR += $(VAR2)

instead of

VAR = some text
VAR += $(VAR3)

to enforce quasi-immediate expansion of the appended text.

In order to avoid an incompatible change for gmake users, we could
permit gmake to continue to support its deviating bahavior for the
non-standad := operator and require the POSIX ::= operator to behave
in a predictable way as proposed in Desired Action below.

P.S.:

Immediate expansion cannot be seen as completely new behavior because
SunPro Make in January 1986 already introduced immediate assignment in
contrast to delayed assignment for this:

VAR :sh= command

which is an immediate command substitution that calls the shell once
at parse time and this construct has been in use since then (using
the echo command) to emulate the behavior that recently has been
standardized for ::=

This problem was discovered while implementing ::= for SunPro Make
Desired Action Change the text in line 97277-97282 to:

  If the macro named by string1 exists, then a <space> character followed
  by the unevaluated string2 shall be appended to the value currently
  assigned to the value of the macro named by string.



To make the standard compatible to the existing behavior.

Discuss the proposal for +:= or :+= and for the standard and write
a related proposal, or document the workaround mentioned above.
Tags issue8
Attached Files

- Relationships

-  Notes
(0005257)
geoffclare (manager)
2021-03-05 16:11

This request makes no sense and seems to be the result of misunderstanding how immediate-expansion macros work.

If the wording is changed so that += always appends an unexpanded value, as requested, then it would be impossible to use any macros on the right hand side of += when the left hand side is an immediate-expansion macro, because they will never be expanded. See lines 97290-97297 (emphasis added):
Macro expansions in string2 of macro definition lines shall be performed according to the form used to define the macro. Immediate-expansion forms (including appending to an existing immediate-expansion macro) shall be expanded in the macro definition line, and shall not be re-expanded when used in a rule or command. Delayed-expansion forms (including appending to an existing delayed-expansion macro, and conditional assignment to a macro not previously existing) shall be expanded when the macro identified by string1 is expanded in a rule or command, and the result of the expansion shall be scanned for further macros.


Regarding the point made in the bug description, "The author of a specific makefile thus cannot easily know whether there was a previous = or ::= assignment for a specific macro since this would require to manually scan many include files", makefile authors can easily avoid this situation by adopting a naming convention for macros that distinguishes the two types of macro. For example, use uppercase and underscore for delayed-expansion macros and something involving at least one lowercase character for immediate-expansion macros. It may be worth adding something to APPLICATION USAGE recommending the use of a naming convention to distinguish the macro types.
(0005258)
psmith (developer)
2021-03-05 17:07
edited on: 2021-03-05 17:17

GNU make's implementation of += has always been that the RHS is treated in the same way (immediate or deferred expansion) as the type of the variable being appended to. This can't be changed as that would be a disastrous backward-compatibility break.

I am not excited about the idea of having ::= work differently from := in this specific way, plus as Geoff mentions I don't see how this can actually work.

Put another way, I think that ::= is much closer to GNU make's := than it is to SunPro make's :sh= and that its behavior should follow GNU make's implementation.

(0005259)
joerg (reporter)
2021-03-05 18:30

Re: Note: 0005257

This seems to be a missunderstanding.

1) += always was a delayed (textual) append, the same way = is a delayed (textual) assignment. This always was independent from whether the macro does not exist before using +=, whether it has been assigned via = delayed assingment or via the immediate assignment "macro :sh=command" before.The current POSIX text breaks that rule that exists since Janary 1986.

2) using $(macro) in a make command or a right hand side of a dependency list always expands the content of that macro. If you use the "immediate epand" feature "macro :sh= command", this just immediately assigns the output of "command" into the content of the make macro "macro". If that output contains further make macro references, these references are expanded when $(macro) is finally referenced.

If I understand your text correctly and you expect a macro that has been assigned with := before should not undergo the usual final macro expansion step when used in a make command or LHS of a dependency, this is in conflict with existing make behavior and it would introduce a desastrous inconsistency.
(0005260)
psmith (developer)
2021-03-05 19:18
edited on: 2021-03-05 19:38

The behavior of += with respect to = (delayed) assignment is as you describe it: it appends the new text to the end of the existing text without any expansion and expansion happens when the variable is used.

Regarding "macro :sh=command", that is not part of the POSIX standard so you can decide how it works with respect to += as you like, I suppose.

Regarding the ::= assignment: this is intended to work (as I understand it) the same way that ":=" works in GNU make. That is, when the variable assignment is made the RHS is expanded immediately, and after that the value is treated as a static string and is no longer expanded when the variable is used. If a "+=" is made to a variable that was created with "::=", the RHS is expanded first, then the result of that expansion is appended to the variable.

I think it's misleading to consider "::=" to be related to Sun's ":sh="; they are different and unrelated to each other. The goal of introducing a new operator like "::=" is exactly because the behavior we want there is not like existing make behavior; so it is not an inconsistency that they are different.

Allowing the value of a "::=" assignment to be re-expanded again when the variable is used would be quite confusing. For example:

    foo ::= $$hello

    all: ; @echo '$(foo)'

would print 'ello', instead of the desired '$hello'.

(0005262)
joerg (reporter)
2021-03-08 15:32
edited on: 2021-03-08 15:42

The following make implementations support +=

SunPro Make This was the first make implementation (1986) to support +=
            All macro references left to : and in commands are always expanded.

gmake in an incompatible way since 1989

AIX make supports += and explicitly mentions that all macro references
            left to : and in commands are always expanded in it's man page

smake supports += since 1996 compatible to SunPro Make
            All macro references left to : and in commands are always expanded.

The problem is that gmake is incompatible to SunPro Make, AIX make and smake and POSIX should standardize commonly supported features. This does not apply to what gmake implements with :=.

As mentioned before: the main problem with the gmake := implementation is that it leads to unpredictable behavior in huge makefile systems that make use of "include". For this reason, a method should be supported that is compatible to what people expect (the SunPro behavior) and that results in predictable behavior.

A possible way to support predictable behavior could be to define a :::= that expands the right hand side but does not lead to a macro that behaves incompatible to previous make behaviot.

Another useful predictable method could be a +:= operator that immediately expands the right side of the += opration.

(0005263)
psmith (developer)
2021-03-08 18:56
edited on: 2021-03-08 18:58

Can you explain what is incompatible about GNU make's use of +=? As far as I understand it, GNU make's use of += is the same as all other versions of make _when applied to standard make variables_ (created with "="). Put another way, if we posit a makefile that adheres completely to traditional POSIX make plus the extra "+=" operator only (and not "::=") then there is no incompatibility across implementations. If you disagree please explain.

The question being raised here is what the behavior should be for += when applied to the new operator, "::=". Since this is a new operator that doesn't exist in other versions of make, it cannot be the case that its behavior is incompatible with other versions of make.

I just want to make clear that comparing "::=" to either SunPro ":=" or SunPro ":sh=" is not right; those are very different operators and have no relationship to "::="; using the way they work as examples for "::=" isn't appropriate IMO. No version of the POSIX standard makes any statement about how the ":=" operator should work.

It seems there are two issues being discussed WRT "::=".

The first is whether the value of such macros should be re-expanded when they are used, as the ":=" operator is defined in BSD make for example, or whether they should not be, as for the GNU make version of ":=" for example.

This difference was already raised in issue #330 comment ~577. The resolution of that issue chose the GNU make method, where the expansion happens only once. So for example given this makefile:

  one = $$one
  two ::= $$two
  all: ; @echo '$(one) $(two)'

you would get "$one $two" (as with GNU make ":=") not "$one wo" (as with BSD make ":="). Of course I'm biased but to me this behavior is more reasonable and what users would expect. It is confusing and unexpected to have values assigned with "=" to be evaluated one time and values assigned with "::=" to be evaluated twice.

The other issue is what the behavior should be for "+=" when applied to a variable set with "::=".

If we use the current behavior of "::=" as defined by issue #330 then it's clear that the RHS of the "+=" must be expanded immediately on assignment and the result appended to the value. In fact it is wrong to do anything else, otherwise, as Geoff points out above, the added text will never be expanded. This is the current definition of the behavior as accepted in issue #330.

If the behavior of "::=" were to be changed to allow for the BSD make version in addition to the GNU make version, then either form of "+=" (one where the expansion happens immediately and one where expansion is delayed) could work.

I personally am not a fan of that change since I think allowing both methods adds confusion and makes it harder to write portable makefiles.

(0005264)
joerg (reporter)
2021-03-11 15:50
edited on: 2021-03-11 15:51

The BSD version of make I have at home (from 4/2020) implements the
same behavior for := and += as smake. += does not expand the right side
and macro content is always expanded at use time.

Make on AIX supports += and explicitly documents that macros are always
expanded at use time. I currently have no access and thus cannot say
whether there is also support fpr :=.

So we have two make implementations that implement the full featureset
but behave identical in this area and different from GNU make.

It has however been reported that a ubuntu binary for bmake seems to have
been changed.

Yesterday, I worked on smake (I started with smake because I know smake
since 36 years and with SunPro Make only since 4 years) and tried to
implement the :::= and +:= proposal. The code is ready now and will be
published next week. So this is no longer invention.

I also checked whether it would be possible to disable macro expansion at
use time on request. This is at least hard to implement since macro name
expansion still needs to be done.

(0005265)
geoffclare (manager)
2021-03-11 16:26

On page 2903 line 97634,97638,97638 section make (APPLICATION USAGE) change "IMMED" to "Immed".

On page 2903 line 97641 section make (APPLICATION USAGE), add a new paragraph:
Because the behavior of the <tt>+=</tt> assignment differs depending on whether the macro being appended to is a delayed-expansion macro or an immediate-expansion macro, it is recommended that when both macro types are used in a set of multiple makefiles and include files, a naming convention is adopted to distinguish the two macro types. This avoids any confusion about whether <tt>+=</tt> will append the expanded or unexpanded value. A suitable convention might be to name delayed-expansion macros using uppercase and underscore characters and immediate-expansion macros using a name that contains at least one lowercase character.
(0005267)
dmitry_goncharov (reporter)
2021-03-11 22:07

> Make on AIX supports += and explicitly documents that macros are always
expanded at use time.

The man page for aix make says
"Variable substitution occurs at two distinct times, depending on where the variable is being used. Variables in dependency lines are expanded as the line is read. Variables in shell commands are expanded when the shell command is executed."

$ cat makefile
what = hello
msg += $(what)
all:
        @echo msg=$(msg)


what=bye
$ /usr/bin/make
msg=bye
$
$ cat makefile2
what = hello
msg += $(what)
all: $(msg)
        @echo msg=$(msg)


what=bye
$ /usr/bin/make -f makefile2
make: 1254-002 Cannot find a rule to create target hello from dependencies.
Stop.
$

> I currently have no access and thus cannot say
whether there is also support fpr :=.

No support for :=.
(0005268)
psmith (developer)
2021-03-12 19:18

Regarding implementation details for POSIX "::=", the straightforward way to implement it is to have a bit associated with the variable stating whether it was assigned via "::=" or not; when the variable value is to be expanded you check that bit and if it's set then instead of expanding the content of the variable you copy the content into the result verbatim. To implement "+=" you check if the bit is set and if so, you expand the RHS and append it to the current value.
(0005291)
joerg (reporter)
2021-03-19 14:34

Re: Note: 0005260 A make compliant way to avoid the problem you mentioned in that bugnote is to use:

foo ::= $$$$hello
all: ; @echo '$(foo)'


Re: Note: 0005263 A make compliant way to avoid the problem you mentioned in that bugnote is to use:

one = $$one
two ::= $$$$two
all: ; @echo '$(one) $(two)'


Re: Note: 0005268 Whom do you like to help with that note? The remark misses the most important problem:

A full blown make implementation needs to be able to update targets (e.g. arguments from include directives) while parsing Makefiles. This makes it hard to switch off expansion in a way similar to gnu make. The only way I see in e.g. smake without throwing all 40 year old code structures away is to introduce a global variable to enforce expansion when in a parser related part of the code. This enforces ugly code :-(

This is the background why I propose to add :::= for an expanding assignment without creating a different macro type and a +:= append operator that always expands the right side.
(0005297)
psmith (developer)
2021-03-22 14:48

Certainly there are workarounds. The argument raised against the behavior of "::=" as described in the standard is that it is inconsistent with other variables during expansion, but the only way to make it consistent on the expansion side is to make it inconsistent during assignment, as you show with these workarounds. My belief is that this would be a worse situation. YMMV.

I don't understand the comments about "updating targets (e.g. arguments from include directives)". To implement the "::=" operator as defined in the POSIX standard, there's no special handling of targets, included files, or anything else. Any implementation of make must have a mapping between variable names and variable values. To implement "::=" what's needed is to add a boolean flag to each value saying whether it was assigned with "::=" or not. Then during expansion (any expansion, at any time) if that flag is not set the value is expanded normally. If the flag is set then the value is copied as-is without expansion. During "+=" the variable being assigned is checked and if the flag is set then the RHS is expanded first, then appended to the existing value. Otherwise the unexpanded RHS is added to the value. That's all that needs to happen; I'm not familiar with the internal structure of smake but I don't see how this could require throwing out existing data structures; the only change to data structures is the one additional flag per variable.

A quick look in the smake code suggests that all that's needed is a new flag value, such as F_NOEXPAND or something, that can be added to flags in struct obj. No changes to data structures required.

I have no comment about a putative ":::=" since I don't understand exactly what it does or what it is for, except to say that I don't see a need for yet another assignment operator in make; certainly no GNU make users have suggested to me that there's a problem that needs to be solved.

- Issue History
Date Modified Username Field Change
2020-11-18 13:49 joerg New Issue
2020-11-18 13:49 joerg Name => Jörg Schilling
2020-11-18 13:49 joerg Page Number => 2894-2895
2020-11-18 13:49 joerg Line Number => 97277-97282
2021-03-05 16:11 geoffclare Note Added: 0005257
2021-03-05 17:07 psmith Note Added: 0005258
2021-03-05 17:10 psmith Note Edited: 0005258
2021-03-05 17:11 psmith Note Edited: 0005258
2021-03-05 17:17 psmith Note Edited: 0005258
2021-03-05 18:30 joerg Note Added: 0005259
2021-03-05 19:18 psmith Note Added: 0005260
2021-03-05 19:18 psmith Note Added: 0005261
2021-03-05 19:18 psmith Note Deleted: 0005261
2021-03-05 19:38 psmith Note Edited: 0005260
2021-03-08 15:32 joerg Note Added: 0005262
2021-03-08 15:42 joerg Note Edited: 0005262
2021-03-08 18:56 psmith Note Added: 0005263
2021-03-08 18:58 psmith Note Edited: 0005263
2021-03-11 15:50 joerg Note Added: 0005264
2021-03-11 15:51 joerg Note Edited: 0005264
2021-03-11 16:26 geoffclare Note Added: 0005265
2021-03-11 16:26 geoffclare Final Accepted Text => Note: 0005265
2021-03-11 16:26 geoffclare Status New => Resolved
2021-03-11 16:26 geoffclare Resolution Open => Accepted As Marked
2021-03-11 16:27 geoffclare Tag Attached: issue8
2021-03-11 22:07 dmitry_goncharov Note Added: 0005267
2021-03-12 19:18 psmith Note Added: 0005268
2021-03-19 14:34 joerg Note Added: 0005291
2021-03-22 14:48 psmith Note Added: 0005297
2021-03-25 14:24 geoffclare Status Resolved => Applied


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker