0001812: Support xargs -P 0 - Austin Group Defect Tracker

Notes
(0006659) geoffclare (manager) 2024-02-16 10:28	The wording "as many processes as possible" is problematic. An implementation could conform to this by just forking until it gets EAGAIN, which would not be good. Slightly better would be "an implementation-defined maximum number of processes", but this would still allow the same poor implementation: it would just need to document that number as being CHILD_MAX. In Issue 8 we have sysconf(SC_NPROCESSORS_ONLN) so perhaps we should involve that somehow, but we would need to assess whether existing implementations would conform to the new requirement. The best time to make a decision on this would probably be after we begin work on Issue 9, rather than now.

(0006661) kre (reporter) 2024-02-16 11:39	I agree with Note: 0006659 (had intended to say much the same thing), other than the possible use of SC_NPROCESSORS_ONLN probably (maybe that's better than nothing, but probably not). While that value has its uses, apportioning cpus to tasks isn't one of them, as not all cpus in a system are created equal any more - some are more suited for one kind of task, and others for different activities (which doesn't even begin to address whether the peculiar "hyperthreading" cpus count for any of this). I certainly agree with deferring a decision on this, and all other issue 9 matters (probably even Issue 8 TC1 matters) until after issue 8 is published (no reason why discussions cannot happen leading to a potential resolution however).

(0006667) dwheeler (reporter) 2024-02-19 19:58	In my experience, the most common value for xargs -P is 0, as that basically says "please maximize throughput" in a practically portable way. It is hard to formally specify though! I think a better approach would be to attempt to describe, from a human perspective, what is being requested. How about this, instead of: > If the value of maxprocs is 0, run as many processes as possible (this is often the number of processors available). how about this: > If the value of maxprocs is 0, the system heuristically attempts to minimize total runtime by executing many processes (e.g., by executing the number of processes equal to the number of processors).

(0006668) steffen (reporter) 2024-02-19 21:28	In my very humble opinion in this respect i want to point out that many programs use many different approaches to this. Reasonable i personally find zstd: If # is 0, attempt to detect and use the number of physical CPU cores. In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX, which is either 64 in 32-bit mode, or 256 for 64-bit environments. This modifier does nothing if zstd is compiled without multithread support. Very unreasonable certain make(1) implementation where 0 means, if i recall correctly, to unfold the entire tree, leading to hundreds of parallel invocations. (Again: iirc.) I think the standard, who extensively documented the make(1) -j parameter with even an implementation guideline, should possibly add an entry to "3. Definitions", and point to the according entry for xarg(1)s -P (0). I also want to point out that the sysctl's and such used to do the necessary "calculations" often return false numbers. This is true in particular for containers ("bound namespaces" more), where there are even open bug reports (Linux glibc; FreeBSD) which are not resolved because also readers of this list chose to go this route, continuing to actively report false numbers in order not to have a need to adjust a possibly large number of test cases in their test framework. (I am too lazy to search for the numbers now; but would on request.) Ie. The standardized and available possibilities to actually get a notion of the number of available CPU threads does not reflect actual reality. For xargs(1) in particular, also make(1), the number of tools actually spawned is anyway most likely much greater than the number given to -P or -j.

(0006669) hvd (reporter) 2024-02-19 22:44	Note also that in current implementations where xargs -P 0 is supported (GNU, FreeBSD), there is no limit based on anything like number of processors, and for some commands, it may not be meaningful to have such a limit because the processes are in no way CPU bound. If xargs -P 0 is currently used like that, and its behaviour is changed to start imposing a limit based on number of processors, the obvious question that a user would ask would be how to restore the old behaviour, and in my opinion a rationale should be provided to explain what such a user can do. From my reading of what is proposed here, such a user would simply be out of luck and should keep using an old version of xargs. I hope such a user can get a better alternative instead.

(0006671) gabravier (reporter) 2024-02-21 00:41	I can confirm that all current implementations I can check that have `-P0` don't limit themselves based on the number of processors - i.e. all of GNU findutils, FreeBSD, illumos, busybox and toybox. A simple of these, Busybox, does have a limitation, though, but this doesn't seem to have anything to do with the amount of processors on a machine: Busybox limits the amount of concurrent invocations to 100 with a code comment indicating the rationale as simply being "let's not go crazy high".

(0006751) geoffclare (manager) 2024-04-15 16:29	After making the page 3601 line 123232 change from bug 1801, change: If the value of maxprocs is non-positive, the behavior is unspecified. to: If the value of maxprocs is zero, the system heuristically attempts to minimize total runtime by executing multiple invocations of utility without waiting for previous executions to complete (e.g., by choosing the number of such invocations based on the number of available processors). If the value of maxprocs is negative, the behavior is unspecified.

(0006771) mohd_akram (reporter) 2024-04-19 20:49	FYI, I didn't add this in the other issue despite wanting it because of what hvd and gabravier mentioned. The option does not seem to behave reasonably in current implementations and IMO shouldn't be included because of that. This description is misleading because the current behavior is to maximize the number of processes without regard to runtime or anything else which can lead to hundreds of processes that slow down the entire OS. A user who is simply looking for a "parallelize this" flag can use `-P$(getconf NPROCESSORS_ONLN)`.

Issue History
Date Modified	Username	Field	Change
2024-02-15 20:53	dwheeler	New Issue
2024-02-15 20:53	dwheeler	Status	New => Under Review
2024-02-15 20:53	dwheeler	Assigned To	=> ajosey
2024-02-15 20:53	dwheeler	Name	=> David A. Wheeler
2024-02-15 20:53	dwheeler	Organization	=> The Linux Foundation
2024-02-15 20:53	dwheeler	Section	=> xargs
2024-02-15 20:53	dwheeler	Page Number	=> 3600-3601
2024-02-15 20:53	dwheeler	Line Number	=> 123252
2024-02-16 10:28	geoffclare	Note Added: 0006659
2024-02-16 11:39	kre	Note Added: 0006661
2024-02-19 19:58	dwheeler	Note Added: 0006667
2024-02-19 21:28	steffen	Note Added: 0006668
2024-02-19 22:44	hvd	Note Added: 0006669
2024-02-21 00:41	gabravier	Note Added: 0006671
2024-03-01 11:48	geoffclare	Project	1003.1(2008)/Issue 7 => Issue 8 drafts
2024-04-15 16:04	geoffclare	Relationship added	related to 0001801
2024-04-15 16:29	geoffclare	Note Added: 0006751
2024-04-15 16:30	geoffclare	Final Accepted Text	=> Note: 0006751
2024-04-15 16:30	geoffclare	Status	Under Review => Resolved
2024-04-15 16:30	geoffclare	Resolution	Open => Accepted As Marked
2024-04-15 16:31	geoffclare	Tag Attached: issue9
2024-04-19 20:49	mohd_akram	Note Added: 0006771
2024-08-15 19:56	salewski	Issue Monitored: salewski

Aardvark Mark IV