So... the main reason to use Unicode in general, and UTF-8 specifically, is that it's the common denominator of a lot of weird stuff you'd see in the wild.
For example, most Unix platforms allow filenames to be arbitrary sets of bytes, while Windows lets filenames be UCS-2 (i.e. invalid surrogates are supported). Also, both Unix and Windows have some notion of a "local encoding" (LC_ALL etc on Unix, codepages on Windows).
The common denominator, the Schelling point [1], of all of these weird systems is Unicode. Without prior coordination, you can generally assume that other participants in your system would try to use Unicode, and probably with the UTF-8 encoding.
Checking at the boundaries of your program that your inputs are valid Unicode/UTF-8 leads to (a) good error messages when they aren't, and (b) not having to deal with jank internally.
Why would "the path name you entered is not valid Unicode/UTF-8" be a good error message, if the path actually exists on the system?
Also, what does "jank" mean here? What do you gain by treating file names as Unicode instead of byte sequences, for the majority of programs that don't even need to display the name, except perhaps in logs?
The way I see it, Unicode is only relevant for displaying strings to humans, or for taking input from humans directly. For virtually all other purposes, strings should be treated as byte sequences internally, regardless of whether they were intended to be UTF-8 or something else. For example, if you're reading a JSON document and looking for a hardcoded key, there's no reason whatsoever to represent the JSON or the key as Unicode. The key is a sequence of bytes, the JSON objects have sequences of bytes as keys. The fact that JSON usually prefers UTF-8 is of relatively little relevance.
> Why would "the path name you entered is not valid Unicode/UTF-8" be a good error message, if the path actually exists on the system?
Some programs have to cope with arbitrary file names. (So yes, cp shouldn't require filenames to be UTF-8.) The vast majority don't.
I maintain a Rust crate called camino [1], the readme for which outlines the general philosophy. The fact is that simply enforcing that the file paths you deal with are always UTF-8 greatly simplifies a lot of code.
> Also, what does "jank" mean here? What do you gain by treating file names as Unicode instead of byte sequences, for the majority of programs that don't even need to display the name, except perhaps in logs?
If you ever have filenames in a text file, how do you match them up with filenames on disk? If you try to support the full space of filenames that can possibly exist on a platform, there is no general, cross-platform solution for doing so. (This is known as the "makefile problem", and if you find the right wiki page you'll see a large table exploring all the possibilities and their tradeoffs.) And if you start pulling that thread, you'll unravel a very large number of problems trying to handle non-Unicode output in reality.
But you can simply cut the knot by restricting filenames to Unicode, and most programs should do that.
For example, I work at Oxide. Why would any of our services want any internal filenames to be anything but UTF-8 (or really ASCII)? Trying to support weird filenames is unnecessary complexity. So we just use camino.
I would say loads of Unix tools have a much bigger problem with files that contain whitespace and especially newlines then they do with the character encoding. Similarly, you can usefully process a lot of file types if you can safely assume only the encoding of special characters for that format, like {",[ and newline for JSON.
This is why I don't get what you mean by "text file" in this context. Obviously it's hard, if not impossible, to meaningfully interpret any part of a random text file as a file name, regardless of encoding. But if you have a text file in some known structured format, it shouldn't be a significant problem at all, as long as you know the encoding of those special characters and have some basic conventions. In particular, the agreement could be that the filenames will be represented as raw bytes except for format-specific escapes (like escaping " in JSON or > in XML), then the file name part need not even fully match the intended encoding of the rest of the file. It's true though that it's not very easy to work with a byte array that has different encodings in different parts.
On the other hand, I fully agree that it's a good idea to restrict things to simple sunsets of characters if you can get away with it. I just don't think that restring to "all of Unicode" is particularly useful. Restricting to a subset of ASCII or even just to the BMP does have meaningful advantages, if it's an option for a particular domain.
> Restricting to a subset of ASCII or even just to the BMP does have meaningful advantages, if it's an option for a particular domain.
This is definitely appropriate in some cases, but for Rust specifically gets in the way a lot. For example, &camino::Utf8Path and &str have transparent conversions both ways, in a way that users rely on heavily (passing in a string into a function that takes an AsRef<Path> or AsRef<Utf8Path> is extremely common). If you introduced, say, AsciiPath, there would be a lot more friction -- you couldn't just pass in an arbitrary string and treat it as an AsciiPath.
Again, Schelling point -- without prior coordination you can assume that folks are using strings.
Because there can be a serde 2 or proptest 2 in the future, but camino's API surface is relatively small and pretty rigid so there will never be a camino 2.
If camino's MSRV was more modern (1.60 I think?) Rust I'd remove the `serde` and `proptest` features entirely via the `dep:` syntax. (Come to think of it, it may be worth bumping the MSRV for that! Would want to look at some data, and maybe in a few months -- camino deliberately has an ancient MSRV as a foundational crate.)
> The way I see it, Unicode is only relevant for displaying strings to humans, or for taking input from humans directly. For virtually all other purposes, strings should be treated as byte sequences internally, regardless of whether they were intended to be UTF-8 or something else. For example, if you're reading a JSON document and looking for a hardcoded key, there's no reason whatsoever to represent the JSON or the key as Unicode. The key is a sequence of bytes, the JSON objects have sequences of bytes as keys. The fact that JSON usually prefers UTF-8 is of relatively little relevance.
This is true (although Unicode is not the best character set, but that is a separate issue), although in the case of JSON, being treated as Unicode is relevant because of the escape codes that can be used in JSON string literals (although this does not make it necessary to validate UTF-8; it only makes it necessary to encode UTF-8 when an escape code is encountered).
Furthermore, when displaying text only for writing to a file, or to a terminal which is assumed to already have the correct character encoding (if you do not need to deal with alignment and stuff like that), you do not need to worry about UTF-8, and in fact is better that you don't; then it will use the same character encoding that it already is and will already be correct, whether it is UTF-8 or not (and you can avoid unnecessaily wasting time with validating UTF-8). (A program might require though that it is valid ASCII or extended ASCII (so e.g. UTF-16 will not work), but shouldn't need to care what the non-ASCII bytes mean.)
Unfortunately, some programming languages make it difficult.
For example, most Unix platforms allow filenames to be arbitrary sets of bytes, while Windows lets filenames be UCS-2 (i.e. invalid surrogates are supported). Also, both Unix and Windows have some notion of a "local encoding" (LC_ALL etc on Unix, codepages on Windows).
The common denominator, the Schelling point [1], of all of these weird systems is Unicode. Without prior coordination, you can generally assume that other participants in your system would try to use Unicode, and probably with the UTF-8 encoding.
Checking at the boundaries of your program that your inputs are valid Unicode/UTF-8 leads to (a) good error messages when they aren't, and (b) not having to deal with jank internally.
[1] https://en.wikipedia.org/wiki/Focal_point_(game_theory)