BStr and BString transparent wrapper types for probably-human-readable text

[joshtriplett]

Proposal

Problem statement

Support users working with byte vectors/slices that are usually human-readable and usually UTF-8, but aren't guaranteed to be. Provide a common set of vocabulary types that users can use, without requiring them to go to the crates ecosystem.

Motivating examples or use cases

If you need to work with "usually but not always human-readable data", and you use Vec<u8> or &[u8], and you print those, you get very long strings like [97, 98, 99, ..., rather than "abc...". A Vec<u8> doesn't tell the type system that you generally expect human-readable text.

Having separate types allows having different trait impls, for purposes such as Debug, serialize/deserialize traits, database query traits, etc.

The extremely popular bstr crate, with numerous downloads and numerous crates depending on it, provides evidence of the desirability of these types.

Embedding a BStr or BString in a data structure provides type-level information that the user expects it to be human readable rather than arbitrary bytes.

A few of the many potential use cases:

• Handling strings from stdin/stdout of a (potentially remote) command, which are probably human-readable but they might have human-supplied non-UTF-8 that should be emitted as-is.
• Implementations of tools like git, which need portable paths (not the paths of the local OS) that are always bytes.
• Implementations of text processing tools that assume the data is human-readable.

Solution sketch

// In core::bstr
#[repr(transparent)]
pub struct BStr(pub [u8]);

impl Debug for BStr { ... }
impl Display for BStr { ... }
impl Deref for BStr { type Target = [u8]; ... }
impl DerefMut for BStr { ... }
// Other trait impls from bstr, including From impls

// In alloc::bstr
#[repr(transparent)]
pub struct BString(pub Vec<u8>);

impl Debug for BString { ... }
impl Display for BString { ... }
impl Deref for BString { type Target = Vec<u8>; ... }
impl DerefMut for BString { ... }
// Other trait impls from bstr, including From impls

This ACP is not proposing to add any inherent methods. The bstr types have no inherent methods except new, and this ACP is proposing to make the types transparent wrappers instead.

#499 provides the relevant methods, and those methods will be accessible via Deref/DerefMut. This is similar to the approach bstr uses.

Alternatives

The primary alternative would be to continue using a crate from crates.io. I would propose that these types are simple enough and fundamental enough that they should be in the standard library as "vocabulary types".

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[pitaj]

One thing that I think would be an important part of this API:

impl [u8] {
    fn bstr(&self) -> &Bstr;
    fn bstr_mut(&mut self) -> &mut Bstr;
}

[tgross35]

I would be quite happy to have a string type that is easier to edit without reparsing, especially when resources are tight or when an allocator isn't available.

For related API, It would be nice if we could make constructing a Box<BStr> easier. I think with bstr you currently need to go &[u8] -> Box<[u8]> -> Box<BStr> -> (optional) Rc<BStr>, which is pretty inconvenient.

[joshtriplett]

@pitaj I would hope to satisfy that with .into(). If there's a reason .into() won't work, no objections to adding methods like those, though I'd add an as_ prefix.

@tgross35 Makes sense. I'd like to try to get the types themselves in first, but that may make sense as a followup.

[ChrisDenton]

into is great when it just works but annoying when it can't magically infer the type. You can still make it work but it gets awkward.

Also into isn't const.

[joshtriplett]

@ChrisDenton Wouldn't BStr(xyz) work in const? But, of course, that doesn't work nicely when chaining. Zero objections to adding a const-capable as_bstr and as_bstr_mut, but again I wouldn't want to do that as part of any initial proposal.

[tgross35]

This could also improve my old ACP that I never finished up, #287. Instead of taking &[u8] those methods could take &BStr.

[Amanieu]

We discussed this in today's @rust-lang/libs-api meeting.

@the8472 asked for an unresolved question for what the behavior of the Display impl should be: matching Debug by using escapes, or using the Unicode replacement character.

Other than that there was a consensus to accept this.