0001631: Default archive rules are incompatible with parallel processing.

Notes
(0006124) kre (reporter) 2023-01-21 21:48	I don't know make implementations, or whether any does this, but another option might be to require that when running sub-builds in parallel (whatever the correct terminology is for that) that no two such parallel sub-builds can be building the same target, simultaneously. To me that seems like a sane rule for all parallel builds, regardless of whether there are archives (or anything similar) involved.

(0006125) rillig (reporter) 2023-01-21 22:10	@dmitry_goncharov Could you please elaborate why the archiving is the problem with these rules? As far as I understand these rules, the same problem occurs when two competing files named `source.c` and `source.f` are compiled to `.o` files. Since the inference "shall be searched in the order defined", `source.c` is chosen by both parallel makes, and `source.f` is effectively ignored. Or are you talking about a different scenario in which one of the makefiles has deactivated the `.c.o` rule? But then, compilation would be as broken as archiving. Or did I miss something?

(0006126) dmitry_goncharov (reporter) 2023-01-22 02:08	> but another option might be to require that when running sub-builds in parallel (whatever the correct terminology is for that) that no two such parallel sub-builds can be building the same target, simultaneously. This looks a reasonable idea to me. Should be relatively easy to implement in top make alone. However, this is more difficult when submake is involved. One submake does not know which target another submake or top make is building. One technique could be to use a form of file locking on the target (as long as the target is an actual file in the fs). However, this is not cheap.

(0006127) dmitry_goncharov (reporter) 2023-01-22 03:02 edited on: 2023-01-22 03:13	> Could you please elaborate why the archiving is the problem with these rules? Let us assume we have hello.c and bye.c and have to build libhello.a which contains hello.o and bye.o. With these rules make will first compile hello.c and bye.c in parallel. Once an object file (either hello.o or bye.o) is ready make will fork-exec an instance of ar to insert that object file to libhello.a. Which possibly results in these two instances of ar running in parallel.

(0006128) psmith (developer) 2023-01-22 18:42	My preference would be to change the default rules to remove the $(AR) from the individual compilation, and add a new default rule to build the archive from object files using the "$?" (only updated files). But I'm not sure this can be easily done in the standard version of make, that doesn't have the complexity of intermediate files.

(0006131) steffen (reporter) 2023-01-23 21:59	My fault. Personally i think the standard inference rules are very much overcome, especially anything regarding SCCS that has been deprecated by life (except maybe for some private use cases). (And Jörg is dead to improve his SCCS.) The syntax regarding archives i have never seen used in real life. I would go with what the GNU make manual says, "either write some dedicated rules, or do not use -j for archives" (more or less cited correctly). So maybe a sentence of warning?

(0006132) steffen (reporter) 2023-01-23 22:02	P.S.: BSD make and smake (from Jörg) do not even get it right. BSD make also does not support the standard-documented way to output inference rules. But eg lib: lib(t1.o) lib(t2.o) only works with GNU make. (I have no Solaris to test.) And GNU make may fail for -j, as it documents (in its info manual). #?0\|kent:x$ bmake `lib' is up to date. #?0\|kent:x$ smake smake: Can't find any source for 'lib(t1.o)'. smake: Couldn't make 'lib'. #?1\|kent:x$ make gcc -O1 -g -c -o t1.o t1.c ar -rv lib t1.o ar: creating lib a - t1.o gcc -O1 -g -c -o t2.o t2.c ar -rv lib t2.o a - t2.o ok rm t2.o t1.o

(0006133) steffen (reporter) 2023-01-23 22:04	P.P.S: even with "lib.a" it gets worse: #?0\|kent:x$ smake smake: Can't find any source for 'lib.a(t1.o)'. smake: Couldn't make 'lib.a'. #?1\|kent:x$ bmake ok #?0\|kent:x$ ll total 12K -rw-r----- 1 steffen steffen 61 Jan 23 22:26 t1.c -rw-r----- 1 steffen steffen 61 Jan 23 22:26 t2.c drwxrwxrwt 10 root root 240 Jan 23 23:01 ../ drwxr-x--- 2 steffen steffen 100 Jan 23 23:02 ./ -rw-r----- 1 steffen steffen 41 Jan 23 23:02 makefile #?0\|kent:x$ make gcc -O1 -g -c -o t1.o t1.c ar -rv lib.a t1.o ar: creating lib.a a - t1.o gcc -O1 -g -c -o t2.o t2.c ar -rv lib.a t2.o a - t2.o ok rm t2.o t1.o #?0\|kent:x$ ll total 20K -rw-r----- 1 steffen steffen 61 Jan 23 22:26 t1.c -rw-r----- 1 steffen steffen 61 Jan 23 22:26 t2.c -rw-r----- 1 steffen steffen 41 Jan 23 23:02 makefile drwxrwxrwt 10 root root 240 Jan 23 23:02 ../ -rw-r----- 1 steffen steffen 7218 Jan 23 23:02 lib.a drwxr-x--- 2 steffen steffen 120 Jan 23 23:02 ./ #?0\|kent:x$ (But i have problems grasping inference rules, maybe it is my fault. I only used them very superficially.)

(0006134) hvd (reporter) 2023-01-23 22:11	> Let us assume we have hello.c and bye.c and have to build libhello.a which contains hello.o and bye.o. The predefined rules do not support that even without -j, do they? The predefined .c.a rule supports building hello.a from hello.c, using hello.o as an intermediate file. That's it. If you want a different name, such as libhello.a, or if you want to include multiple object files, you need custom rules, and it's then your own responsibility to making them parallel-build-safe. Whether the predefined rule should be kept or not, the parallel processing should have no impact on that. For the very limited scenarios that the built-in rules handle, the parallel processing should not be an issue.

(0006135) dmitry_goncharov (reporter) 2023-01-24 02:24	> The predefined rules do not support that even without -j, do they? ... If you want a different name, such as libhello.a, or if you want to include multiple object files, you need custom rules, and it's then your own responsibility to making them parallel-build-safe. You'll need to tell make that libhello.a depends on hello.o and bye.o. But, the default rules, along with their default recipes, can still be used to build libhello.a. $ ls bye.c hello.c makefile $ cat makefile libhello.a: libhello.a(hello.o) libhello.a(bye.o) $ make cc -c -o hello.o hello.c ar -rv libhello.a hello.o ar: creating libhello.a a - hello.o cc -c -o bye.o bye.c ar -rv libhello.a bye.o a - bye.o rm hello.o bye.o $ ar tv libhello.a rw-r--r-- 1001/1004 952 Jan 23 21:06 2023 hello.o rw-r--r-- 1001/1004 952 Jan 23 21:06 2023 bye.o > Whether the predefined rule should be kept or not, the parallel processing should have no impact on that. People use parallel processing a lot. With these rules we expect every user to read the manual and notice the warning and avoid parallel processing.

(0006136) hvd (reporter) 2023-01-24 09:56	> But, the default rules, along with their default recipes, can still be used to build libhello.a. Thanks for spelling it out. > My preference would be to change the default rules to remove the $(AR) from the individual compilation, and add a new default rule to build the archive from object files using the "$?" (only updated files). But I'm not sure this can be easily done in the standard version of make, that doesn't have the complexity of intermediate files. Right, they complicate things. It would be easy if the .o files did not need to be removed. Then, the default .c.o rule could be used to make the object file, and a rule to build the archive from the .o files could use $? to get the updated object files (as opposed to the updated source files). But the .o files getting removed complicates things. kre's idea to require that no two rules for the same target run in parallel seems like a good idea, but prevents hello.c and bye.c from being compiled in parallel. With the current rule, compiling them in parallel is unsafe as there is no way to not also cause the $(AR) commands to run in parallel, which then causes issues. If there is a way to separate the commands into separate rules, like with the referenced concept of intermediate files, that would become safe. Yet another option would be to leave make as is, but require ar to use file locking to allow concurrent invocations, where a second invocation blocks until a first one completes. This is probably a bad idea, but mentioning it for completeness.

(0006137) psmith (developer) 2023-01-24 18:55	If the .o files don't need to be removed then you don't need any special handling for archives at all. You can just write standard rules and you don't need the magical libfoo.a(foo.o) syntax or the special behaviors of the .a suffix rules, because you can just base things on the timestamp of the archive itself compared to the timestamp of the .o files. The only way this extra behavior is needed or useful is when you want to update the archive _without_ having to rebuild all the object files (like you do, for example, when you link a program).

(0006152) shware_systems (reporter) 2023-02-16 17:05	An alternative is possibly a change to ar requirements: that it check if an existing archive is already open for processing by another instance or application and then pause until it is capable of gaining such exclusive use. It is unknown whether any current implementation does this checking, however, but it would make use of -j with this rule plausible as the accesses and object adds would be self serializing.

(0006153) steffen (reporter) 2023-02-16 20:41	re @6152 You mean this would be the right time to standarIdize flock(1)? I use it often flock [options] file\|directory command [arguments] flock [options] file\|directory -c command flock [options] number DESCRIPTION This utility manages flock(2) locks from within shell scripts or from the command line. The first and second of the above forms wrap the lock around the execution of a command, in a manner similar to su(1) or newgrp(1). They lock a specified file or directory, which is created (assuming appropriate permissions) if it does not already exist. By default, if the lock cannot be immediately acquired, flock waits until the lock is available. The third form uses an open file by its file descriptor number. See the examples below for how that can be used.

(0006169) nick (manager) 2023-02-23 17:18 edited on: 2023-02-23 17:22	Add to APPLICATION USAGE (D2.1 P2952, after line 99106): It is important to be careful when using parallel execution (the -j switch) and archives. If multiple $(AR) commands run at the same time on the same archive file, they will not know about each other and can corrupt the file. If the -j option is used, it is necessary to use .WAIT in between archive member prerequisites to prevent this (see EXAMPLES). On D2.1 page 2953 line 99164 section make (EXAMPLES), change: lib: lib(file1.o) lib(file2.o) lib(file3.o) @echo lib is now up-to-date The .c.a rule is used to make file1.o, file2.o, and file3.o and insert them into lib. to: lib.a: lib.a(file1.o) .WAIT lib.a(file2.o) .WAIT lib.a(file3.o) @echo lib is now up-to-date The .c.a rule is used to make file1.o, file2.o, and file3.o and insert them into lib.a.

Issue History
Date Modified	Username	Field	Change
2023-01-21 18:45	dmitry_goncharov	New Issue
2023-01-21 18:45	dmitry_goncharov	Name	=> Dmitry Goncharov
2023-01-21 18:45	dmitry_goncharov	Section	=> make
2023-01-21 18:45	dmitry_goncharov	Page Number	=> https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/nframe.html [^]
2023-01-21 18:45	dmitry_goncharov	Line Number	=> 0
2023-01-21 21:48	kre	Note Added: 0006124
2023-01-21 22:10	rillig	Note Added: 0006125
2023-01-22 02:08	dmitry_goncharov	Note Added: 0006126
2023-01-22 03:02	dmitry_goncharov	Note Added: 0006127
2023-01-22 03:13	dmitry_goncharov	Note Edited: 0006127
2023-01-22 03:13	dmitry_goncharov	Note Edited: 0006127
2023-01-22 18:42	psmith	Note Added: 0006128
2023-01-23 21:59	steffen	Note Added: 0006131
2023-01-23 22:02	steffen	Note Added: 0006132
2023-01-23 22:04	steffen	Note Added: 0006133
2023-01-23 22:11	hvd	Note Added: 0006134
2023-01-24 02:24	dmitry_goncharov	Note Added: 0006135
2023-01-24 09:56	hvd	Note Added: 0006136
2023-01-24 18:55	psmith	Note Added: 0006137
2023-02-16 17:05	shware_systems	Note Added: 0006152
2023-02-16 20:41	steffen	Note Added: 0006153
2023-02-23 17:18	nick	Note Added: 0006169
2023-02-23 17:19	nick	Note Added: 0006170
2023-02-23 17:19	nick	Note Deleted: 0006170
2023-02-23 17:20	nick	Final Accepted Text	=> See Note: 0006169
2023-02-23 17:20	nick	Status	New => Resolved
2023-02-23 17:20	nick	Resolution	Open => Accepted As Marked
2023-02-23 17:22	nick	Note Edited: 0006169
2023-02-23 17:22	nick	Tag Attached: issue8
2023-03-10 10:53	geoffclare	Status	Resolved => Applied
2024-06-11 09:12	agadmin	Status	Applied => Closed

Relationships

Aardvark Mark IV