0000248: Fix the numerous filename/pathname processing errors in specification examples

Austin Group Defect Tracker

Aardvark Mark IV

Viewing Issue Simple Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Category

Severity

Type

Date Submitted

Last Update

0000248

[1003.1(2008)/Issue 7] Shell and Utilities

Objection

Error

2010-04-30 20:20

2011-07-06 23:59

Reporter

dwheeler

View Status

public

Assigned To

ajosey

Priority

normal

Resolution

Open

Status

Under Review

Name

David A. Wheeler

Organization

User Reference

Section

Shell & Utilities

Page Number

2789,3003,3260,3385,and more

Line Number

91004,99054-99063,108778,113329-113334,and more

Interp Status

---

Final Accepted Text

Summary

0000248: Fix the numerous filename/pathname processing errors in specification examples

Description

Many examples in the specification fail, sometimes spectacularly, when given filenames or pathnames permitted by the specification. The specification currently permits filenames to include newline, return, tab, escape, leading "-", double-quotes, single-quotes, ampersand, asterisk, and other "interesting" characters. However, many of the specification's examples will fail to work correctly when such filenames are present.

In a few cases, there's a note that the example presumes that some of these will not occur, but there is no standard way to *ensure* that such filenames cannot occur. In addition, many attackers have learned to *intentionally* create such filenames, since there is no way to prevent them in general. Thus, multi-user systems, or systems that receive external data (e.g., via USB sticks and network drives) are still at risk if they used these examples. This incorrect filename handling can be the basis of security problems, as noted in:
http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html [^]
It's worth noting that this is not a new problem. Line 82265 noted that the specification has been changed before "to remove the unreliable use of find | xargs".

The examples will need to be examined for patterns such as these:

# WRONG. Fails if newlines/tabs/space/backslash/apostrophe/double-quote/
# ampersand in filenames:
find . -type f | xargs COMMAND

# WRONG. Fails if a file in the current directory begins with "-" or
# if there are no files in the current directory:
COMMAND *
for file in * ; do
COMMAND "$file"
done

Please either:
1. Fix the various errors. Below are some of them, to get started. It's concerning that even the experts who developed this specification (and they're definitely experts!) made these mistakes. I believe this shows that it is too difficult to correctly use systems while conforming to this specification.
2. Change the rules on filenames to forbid certain filenames. In particular, forbidding control characters in filenames (at least bytes 1-31 and maybe 127) would completely eliminate a number of the problems. In addition, forbidding leading "-" would solve some more (though not as many as the control-characters one).

Desired Action

Page 2789, Line 91004:
Change:
find . −type f | xargs hash
To:
find . -type f -exec hash {} \;
Rationale:
This example tries to explain why hash won't work in this case,
but the example should only have *that* error, not *other* errors as well.

Page 3003, line 99054-99063:
Remove this entire example.
I believe it is not possible to do what this example is attempting to do
while using standard POSIX utilities, at least by using pax.
This example attempts to determine if the filenames in a pax archive
are valid, but filenames can include newline, and pax does not have any
way to disambiguate these cases (e.g., by having a -0 option) in its output.
POSIX tools simply aren't capable of doing accurate filename checking
in this case, at least not without re-implementing (say, in C) a significant
portion of pax.

Page 3260, line 108778:
Replace:
find . −type f | xargs type
with:
find . −type f -exec type {} \;

Page 3385, lines 113329-113334:
Remove this example, or change it to work properly
(and that may not be easy).
Although it documents that it will not work if
filenames contain newline, there's no way to be guaranteed of that.
It also presumes that the filenames don't have {}, which isn't guaranteed either. If this was executed on a system where a filename DID contain a newline,
or where the source/target directories contained {},
*disaster* could ensue, and examples should not recommend
techniques that might be disastrous.
Most of the other examples on page 3385 should arguably be
removed as well, since there's no mechanism for guaranteeing that
filenames do not include newline (if xargs -0 is added, then they
can probably be easily repaired and kept).

Notes
(0000885) Don Cragun (manager) 2011-07-06 23:59	We agree that there are problems in the examples listed. The fixes for these issues will be delayed until 0000251 is resolved. The resolution to 0000251 is expected to make many of these examples much easier to implement reliably using standard features.

Issue History
Date Modified	Username	Field	Change
2010-04-30 20:20	dwheeler	New Issue
2010-04-30 20:20	dwheeler	Status	New => Under Review
2010-04-30 20:20	dwheeler	Assigned To	=> ajosey
2010-04-30 20:20	dwheeler	Name	=> David A. Wheeler
2010-04-30 20:20	dwheeler	Section	=> Shell & Utilities
2010-04-30 20:20	dwheeler	Page Number	=> 2789,3003,3260,3385,and more
2010-04-30 20:20	dwheeler	Line Number	=> 91004,99054-99063,108778,113329-113334,and more
2011-07-06 23:59	Don Cragun	Note Added: 0000885

Relationships