Anonymous | Login | 2024-05-12 22:51 UTC |
Main | My View | View Issues | Change Log | Docs |
Viewing Issue Simple Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||||||
ID | Category | Severity | Type | Date Submitted | Last Update | ||||||||
0001812 | [Issue 8 drafts] Shell and Utilities | Editorial | Enhancement Request | 2024-02-15 20:53 | 2024-04-19 20:49 | ||||||||
Reporter | dwheeler | View Status | public | ||||||||||
Assigned To | ajosey | ||||||||||||
Priority | normal | Resolution | Accepted As Marked | ||||||||||
Status | Resolved | Product Version | |||||||||||
Name | David A. Wheeler | ||||||||||||
Organization | The Linux Foundation | ||||||||||||
User Reference | |||||||||||||
Section | xargs | ||||||||||||
Page Number | 3600-3601 | ||||||||||||
Line Number | 123252 | ||||||||||||
Final Accepted Text | Note: 0006751 | ||||||||||||
Summary | 0001812: Support xargs -P 0 | ||||||||||||
Description |
Issue https://www.austingroupbugs.net/view.php?id=1801 [^] modifies xargs to support "-P". This is good, since *many* computers have multiple processors available. However, the proposed text doesn't define "-P 0". I don't think I've ever used an argument *other* than 0. The meaning of "0" seems to be widely agreed on as well, that is, "use as many as you can". Having a standard way to say "parallelize as much as you can" is extremely useful. |
||||||||||||
Desired Action |
Whenever this is eventually added, instead of: -P maxprocs Parallel mode: execute at most maxprocs invocations of utility concurrently. If the value of maxprocs is non-positive, the behavior is unspecified. Say something like this: -P maxprocs Parallel mode: execute at most maxprocs invocations of utility concurrently. If the value of maxprocs is 0, run as many processes as possible (this is often the number of processors available). If the value of maxprocs is negative, the behavior is unspecified. |
||||||||||||
Tags | issue9 | ||||||||||||
Attached Files | |||||||||||||
|
![]() |
||||||
|
![]() |
|
(0006659) geoffclare (manager) 2024-02-16 10:28 |
The wording "as many processes as possible" is problematic. An implementation could conform to this by just forking until it gets EAGAIN, which would not be good. Slightly better would be "an implementation-defined maximum number of processes", but this would still allow the same poor implementation: it would just need to document that number as being CHILD_MAX. In Issue 8 we have sysconf(SC_NPROCESSORS_ONLN) so perhaps we should involve that somehow, but we would need to assess whether existing implementations would conform to the new requirement. The best time to make a decision on this would probably be after we begin work on Issue 9, rather than now. |
(0006661) kre (reporter) 2024-02-16 11:39 |
I agree with Note: 0006659 (had intended to say much the same thing), other than the possible use of SC_NPROCESSORS_ONLN probably (maybe that's better than nothing, but probably not). While that value has its uses, apportioning cpus to tasks isn't one of them, as not all cpus in a system are created equal any more - some are more suited for one kind of task, and others for different activities (which doesn't even begin to address whether the peculiar "hyperthreading" cpus count for any of this). I certainly agree with deferring a decision on this, and all other issue 9 matters (probably even Issue 8 TC1 matters) until after issue 8 is published (no reason why discussions cannot happen leading to a potential resolution however). |
(0006667) dwheeler (reporter) 2024-02-19 19:58 |
In my experience, the most common value for xargs -P is *0*, as that basically says "please maximize throughput" in a practically portable way. It *is* hard to formally specify though! I think a better approach would be to attempt to describe, from a *human* perspective, what is being requested. How about this, instead of: > If the value of maxprocs is 0, run as many processes as possible (this is often the number of processors available). how about this: > If the value of maxprocs is 0, the system heuristically attempts to minimize total runtime by executing many processes (e.g., by executing the number of processes equal to the number of processors). |
(0006668) steffen (reporter) 2024-02-19 21:28 |
In my very humble opinion in this respect i want to point out that many programs use many different approaches to this. Reasonable i personally find zstd: If # is 0, attempt to detect and use the number of physical CPU cores. In all cases, the nb of threads is capped to ZSTDMT_NBWORKERS_MAX, which is either 64 in 32-bit mode, or 256 for 64-bit environments. This modifier does nothing if zstd is compiled without multithread support. Very unreasonable certain make(1) implementation where 0 means, if i recall correctly, to unfold the entire tree, leading to hundreds of parallel invocations. (Again: iirc.) I think the standard, who extensively documented the make(1) -j parameter with even an implementation guideline, should possibly add an entry to "3. Definitions", and point to the according entry for xarg(1)s -P (0). I also want to point out that the sysctl's and such used to do the necessary "calculations" often return false numbers. This is true in particular for containers ("bound namespaces" more), where there are even open bug reports (Linux glibc; FreeBSD) which are not resolved because also readers of this list chose to go this route, continuing to actively report false numbers in order not to have a need to adjust a possibly large number of test cases in their test framework. (I am too lazy to search for the numbers now; but would on request.) Ie. The standardized and available possibilities to actually get a notion of the number of available CPU threads does not reflect actual reality. For xargs(1) in particular, also make(1), the number of tools actually spawned is anyway most likely much greater than the number given to -P or -j. |
(0006669) hvd (reporter) 2024-02-19 22:44 |
Note also that in current implementations where xargs -P 0 is supported (GNU, FreeBSD), there is no limit based on anything like number of processors, and for some commands, it may not be meaningful to have such a limit because the processes are in no way CPU bound. If xargs -P 0 is currently used like that, and its behaviour is changed to start imposing a limit based on number of processors, the obvious question that a user would ask would be how to restore the old behaviour, and in my opinion a rationale should be provided to explain what such a user can do. From my reading of what is proposed here, such a user would simply be out of luck and should keep using an old version of xargs. I hope such a user can get a better alternative instead. |
(0006671) gabravier (reporter) 2024-02-21 00:41 |
I can confirm that all current implementations I can check that have `-P0` don't limit themselves based on the number of processors - i.e. all of GNU findutils, FreeBSD, illumos, busybox and toybox. A simple of these, Busybox, does have a limitation, though, but this doesn't seem to have anything to do with the amount of processors on a machine: Busybox limits the amount of concurrent invocations to 100 with a code comment indicating the rationale as simply being "let's not go crazy high". |
(0006751) geoffclare (manager) 2024-04-15 16:29 |
After making the page 3601 line 123232 change from bug 1801, change:If the value of maxprocs is non-positive, the behavior is unspecified. to: If the value of maxprocs is zero, the system heuristically attempts to minimize total runtime by executing multiple invocations of utility without waiting for previous executions to complete (e.g., by choosing the number of such invocations based on the number of available processors). |
(0006771) mohd_akram (reporter) 2024-04-19 20:49 |
FYI, I didn't add this in the other issue despite wanting it because of what hvd and gabravier mentioned. The option does not seem to behave reasonably in current implementations and IMO shouldn't be included because of that. This description is misleading because the current behavior is to maximize the number of processes without regard to runtime or anything else which can lead to hundreds of processes that slow down the entire OS. A user who is simply looking for a "parallelize this" flag can use `-P$(getconf NPROCESSORS_ONLN)`. |
![]() |
|||
Date Modified | Username | Field | Change |
2024-02-15 20:53 | dwheeler | New Issue | |
2024-02-15 20:53 | dwheeler | Status | New => Under Review |
2024-02-15 20:53 | dwheeler | Assigned To | => ajosey |
2024-02-15 20:53 | dwheeler | Name | => David A. Wheeler |
2024-02-15 20:53 | dwheeler | Organization | => The Linux Foundation |
2024-02-15 20:53 | dwheeler | Section | => xargs |
2024-02-15 20:53 | dwheeler | Page Number | => 3600-3601 |
2024-02-15 20:53 | dwheeler | Line Number | => 123252 |
2024-02-16 10:28 | geoffclare | Note Added: 0006659 | |
2024-02-16 11:39 | kre | Note Added: 0006661 | |
2024-02-19 19:58 | dwheeler | Note Added: 0006667 | |
2024-02-19 21:28 | steffen | Note Added: 0006668 | |
2024-02-19 22:44 | hvd | Note Added: 0006669 | |
2024-02-21 00:41 | gabravier | Note Added: 0006671 | |
2024-03-01 11:48 | geoffclare | Project | 1003.1(2008)/Issue 7 => Issue 8 drafts |
2024-04-15 16:04 | geoffclare | Relationship added | related to 0001801 |
2024-04-15 16:29 | geoffclare | Note Added: 0006751 | |
2024-04-15 16:30 | geoffclare | Final Accepted Text | => Note: 0006751 |
2024-04-15 16:30 | geoffclare | Status | Under Review => Resolved |
2024-04-15 16:30 | geoffclare | Resolution | Open => Accepted As Marked |
2024-04-15 16:31 | geoffclare | Tag Attached: issue9 | |
2024-04-19 20:49 | mohd_akram | Note Added: 0006771 |
Mantis 1.1.6[^] Copyright © 2000 - 2008 Mantis Group |