[WIP] Add support for custom allocator for String

[zachs18]

Roadmap for allocator support.

API Change Proposal: rust-lang/libs-team#101 [ Accepted ]

A subset/bring-to-current of #79500 utilizing a workaround(?) to the issue (#79499) that was blocking it.
(Note: I previously tried rebasing that (2 year old) PR onto current master, which was not my best idea).

---

Blocked on:

• Add Allocator support for String libs-team#101
• Tracking Issue for RFC 213: Default Type Parameter Fallback #27336 , "defaults affect inference" problem in general e.g. Constructing a simple HashSet<char> requires RandomState type annotation #101319
• (diagnostics, maybe blocking) Apparent duplicated diagnostic note for type mismatch involving type with defaulted type parameter. #101992

Todo/Unresolved Questions:

• Fix UI tests. Done? (see my question below) (see Apparent duplicated diagnostic note for type mismatch involving type with defaulted type parameter. #101992)
• Gdb/Lldb pretty-printing of String<A>.
• Should From<String<A>> for Rc<str>/Arc be removed from this PR? (Corresponding impl does not exist for Vec/Rc<[T]>, and also it may preclude a future From<String<A>> for Rc<str, A>) removed
• Should impl StructuralEq for String<_> be re-added? This PR removes it (because it no longer derive(Eq)), but it may not be observable because String: !StructuralPartialEq
  • Also for FromUtf8Error<_>, but that derived both PartialEq and Eq so it had both StructuralPartialEq and StructuralEq.
  • (Future: This will also will apply to CString, which derives PartialEq and Eq.)
• impl<A: Allocator> From<String<A>> to Box<str, A> breaks orphan rules? (because Box and str are fundamental?)
• Add better ways to make a new non-empty String<A>?
  • fn String<A>::from_str_in(s: &str, alloc: A) -> String<A>?
  • Likewise fn str::to_string_in(alloc: A) -> String<A> (parallels slice::to_vec_in)
  • Or maybe add to_string_in to ToString? (Probably a non-starter, since that would make ToString not object-safe, unless we put a where Self: Sized, which would preclude str)
• [ ] Other FIXMEs (implement or remove comments) (removed FIXME comments, APIs moved to future work)

Possible Future work

• How to handle from_utf8_lossy, which returns a non-allocator-aware Cow<str>?
• How to handle from_utf16(_lossy), which return String and Result<String, _>?
  • Could have them return String<A> where A: Default
  • Could add from_utf16(_lossy)_in, (Add support for custom allocator for (C)String #79500 only added from_utf16_in, not the lossy version)
• (If specialization like this is sound), From<Cow<str>> for String<A> where A: Default could be implemented as from_str_in(&*cow, A::default()) , and specialized for A = Global to the current implementation cow.into_owned().
• From<&String<A>> for String<A> where A: Clone or From<&String<A>> for String<B> where B: Default (I don't think both can exist).
• FromIterator<_> for String, FromStr for String, and From<&str-like> for String could be for String<A> where A: Default.
• impl const Default for String could be impl<A: _ + ~const Default> Default for String<A>, except that Global doesn't implement const Default currently (but I don't see any reason it couldn't).

[rust-highfive]

r? @Mark-Simulacrum

(rust-highfive has picked a reviewer for you, use r? to override)

[zachs18]

This PR changed a lot of the UI tests/diagnostics to have duplicated-ish note messages. I don't know enough about the diagnostics system to really understand why. Are these changes acceptable?

(Several similar changes, e.g. expected struct String, found type str in src/test/ui/issues/issue-53348.stderr, expected struct String, found unit type () in src/test/ui/block-result/consider-removing-last-semi.stderr)

[zachs18]

Note that #79500 had the same kind of duplicated-ish note messages, e.g..

[Mark-Simulacrum]

I also don't know the cause here, but it feels like it might be a larger bug -- maybe worth trying to reproduce on stable/nightly outside of the String type (e.g., have a struct and add a generic parameter to it) and see if that causes similar behavior on regular rustc compilers? If so, then you could file an issue for that.

If this is something specific to String that's probably worth poking at too.

[zachs18]

It does not appear to be specific to String; I have opened #101992 about it.

[zachs18]

Things that are the same as 79500:

The following were made allocator-aware (or added) in the same way as in 79500:

• alloc::str::from_boxed_utf8_unchecked
• alloc::string::String<A>
  • from_utf8(_unchecked), alloc::string::FromUtf8Error and all impls.
  • add from_raw_parts_in/into_raw_parts_with_alloc (from_raw_parts/into_raw_parts stay in String<Global>, unlike for Vec::into_raw_parts currently)
  • add new_in/with_capacity_in
  • into_bytes/as_mut_vec
  • split_off where A: Clone
  • drain, alloc::string::Drain and all impls,
  • Display/Debug/Hash
  • Clone where A: Clone
  • Extend<.*>/fmt::Write
  • Pattern
  • PartialEq to other string-like types (Cow<str>, str, &str)
  • Add(Assign)<&str>
  • Index(Mut)/Deref(Mut)
  • From<Box<str, A>> for String<A>
  • From<String<A>> for Vec<u8, A>

DIfferences from 79500 :

• This does not add String::from_utf16_in.
• This implements PartialEq and PartialOrd between Strings of different allocators.
• This manually implements PartialOrd, Eq, Ord (where 79500 derived these) to avoid an A: Eq (etc) bound.
  • This does not implement StructuralEq anymore.
  • (79500 made deriving work by implementing (Partial){Eq,Ord} for Global. This does not add or require those implementations.)
• This does not currently implement From<&String<A>> for String<A> where A: Clone in case we would want From<&String<A2>> for String<A1> where A1: Default.
• This does not currently implement From<String<A>> for Box<str, A> (because of orphan rules? A is an uncovered type parameter, because Box is fundamental?).

Additions from 79500:

• This adds String::allocator à la Vec::allocator
• This makes AsRef<str>/AsMut<str>/AsRef<[u8]> for String allocator-generic.
• This implements ToString for String<A> (using <str as ToOwned>::to_owned instead of the current <String as ToOwned>::to_owned, which clones the String)

[rustbot]

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

• Stabilizing library features
• Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
• Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
• Changing public documentation in ways that create new stability guarantees
• Changing observable runtime behavior of library APIs

[zachs18]

It passes the checks now, so I'm marking this as ready for review. Feedback is appreciated! (especially on the unresolved questions in the Todos).

(CC'ing people from 79500)
cc @Amanieu , @TimDiekmann

[zachs18]

(I think this is right?)
@rustbot label +T-libs-api -T-libs
(I don't know if this needs an API change proposal since it has a tracking issue?) ACP: rust-lang/libs-team#101

[Mark-Simulacrum]

I sort of feel like we shouldn't provide these APIs and instead encourage String::from(Vec::with_capacity_in(....)), since they do add noise to documentation and most users aren't using them. But that can be handled at stabilization time, I suppose, and is a broader question about everything being allocator-generic.

[Mark-Simulacrum]

I don't think we should add this comment; there's many APIs which could do this (even on Vec) and it doesn't seem worth leaving comments strewn around std for that future possibility.

[zachs18]

Yeah, I had intended to remove or implement most of the FIXMEs based on feedback before the PR gets merged. I'll just remove the ones that don't currently exist for Vec for this PR.

[Mark-Simulacrum]

A few notes:

String no longer implementing StructuralEq

I think this is not observable on stable, though I'm not entirely confident. The primary area where it matters that I know of is pattern matching with a const value, but that requires StructuralPartialEq too.

FromUtf8Error<_>, but that derived both PartialEq and Eq so it had both StructuralPartialEq and StructuralEq.

In order to check whether this is a breaking change we need to know whether any const context on stable can get the value -- I think the answer may be no, but I'm not confident.

---

Also left some comments inline, going to @rustbot author for now but please mark it as ready for review if the ACP goes through and you have addressed comments (or posted back with questions).

[bors]

☔ The latest upstream changes (presumably #136448) made this pull request unmergeable. Please resolve the merge conflicts.

[rust-log-analyzer]

The job x86_64-gnu-llvm-18 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#21 exporting to docker image format
#21 sending tarball 29.2s done
#21 DONE 35.1s
##[endgroup]
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-18]
debug: `DISABLE_CI_RUSTC_IF_INCOMPATIBLE` configured.
---
sccache: Starting the server...
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-18', '--enable-llvm-link-shared', '--set', 'rust.randomize-layout=true', '--set', 'rust.thin-lto-import-instr-limit=10', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--set', 'rust.lld=false', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--enable-new-symbol-mangling']
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-18/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.randomize-layout := True
configure: rust.thin-lto-import-instr-limit := 10
---
  Downloaded boml v0.3.1
   Compiling boml v0.3.1
   Compiling y v0.1.0 (/checkout/compiler/rustc_codegen_gcc/build_system)
    Finished `release` profile [optimized] target(s) in 4.14s
     Running `/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-codegen/x86_64-unknown-linux-gnu/release/y test --use-system-gcc --use-backend gcc --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/cg_gcc --release --mini-tests --std-tests`
Using system GCC
warning: target feature `x87` must be enabled to ensure that the ABI of the current target can be implemented correctly
  |
  = note: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
  = note: for more information, see issue #116344 <https://github.com/rust-lang/rust/issues/116344>
---
Testing alias-4.js ... OK
Testing alias.js ... OK
Testing asrawfd.js ... OK
Testing basic.js ... FAILED
==> Result not found in 'returned': '{"path":"std::string::String","name":"add"}'
 {
-     "path": "std::string::String",
+     "path": "std::string::string",
-     "name": "add",
---
Testing field-is-unary-func.js ... OK
Testing filter-crate.js ... OK
Testing fn-forget.js ... OK
Testing from_u.js ... FAILED
==> Result not found in 'others': '{"path":"std::string::String","name":"from_utf8"}'
 {
-     "path": "std::string::String",
+     "path": "std::char",
-     "name": "from_utf8",
---
Testing reference-shrink.js ... OK
Testing regex.js ... OK
Testing return-based-sort.js ... OK
Testing return-specific-literal.js ... FAILED
==> Result not found in 'in_args': '{"path":"std::string::String","name":"ne"}'
 {
-     "path": "std::string::String",
+     "path": "std::ffi",
-     "name": "ne",
-     "name": "ne",
+     "name": "from",
 }
==> Result not found in 'returned': '{"path":"std::string::String","name":"add"}'
 {
-     "path": "std::string::String",
+     "path": "std::string::string",
-     "name": "add",
-     "name": "add",
+     "name": "new",
 }
Testing return-specific.js ... FAILED
==> Result not found in 'in_args': '{"path":"std::string::String","name":"ne"}'
 {
-     "path": "std::string::String",
+     "path": "std::ffi",
-     "name": "ne",
-     "name": "ne",
+     "name": "from",
 }
==> Result not found in 'returned': '{"path":"std::string::String","name":"add"}'
 {
-     "path": "std::string::String",
+     "path": "std::string::string",
-     "name": "add",
-     "name": "add",
+     "name": "new",
 }
Testing should-fail.js ... OK
Testing simd-type-signatures.js ... OK
Testing string-from_ut.js ... FAILED
==> Result not found in 'others': '{"path":"std::string::String","name":"from_utf8"}'
 {
-     "path": "std::string::String",
+     "path": "std::string::string",
     "name": "from_utf8",
     "name": "from_utf8",
 }
==> Result not found in 'others': '{"path":"std::string::String","name":"from_utf8"}'
 {
-     "path": "std::string::String",
+     "path": "std::string::string",
-     "name": "from_utf8",
-     "name": "from_utf8",
+     "name": "from_utf16",
 }
==> Result not found in 'others': '{"path":"std::string::String","name":"from_utf8_lossy"}'
 {
-     "path": "std::string::String",
+     "path": "std::string::string",
-     "name": "from_utf8_lossy",
-     "name": "from_utf8_lossy",
+     "name": "from_utf16be",
 }
==> Result not found in 'others': '{"path":"std::string::String","name":"from_utf16_lossy"}'
 {
-     "path": "std::string::String",
+     "path": "std::string::string",
-     "name": "from_utf16_lossy",
-     "name": "from_utf16_lossy",
+     "name": "from_utf16le",
 }
==> Result not found in 'others': '{"path":"std::string::String","name":"from_utf8_unchecked"}'
 {
-     "path": "std::string::String",
+     "path": "std::string::string",
-     "name": "from_utf8_unchecked",
---
Testing typed-query.js ... OK
Testing vec-new.js ... OK
Testing vec-type-signatures.js ... OK
Testing write.js ... OK
Command has failed. Rerun with -v to see more details.
  local time: Mon Feb  3 06:48:19 UTC 2025
  network time: Mon, 03 Feb 2025 06:48:19 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

[bors]

☔ The latest upstream changes (presumably #136507) made this pull request unmergeable. Please resolve the merge conflicts.

[zachs18]

I won't have time to work on this in the near future, if someone wants to take over. It might make sense to split this into multiple PRs (e.g. one moving String to a submodule with no semantic changes, and another actually introducing the allocator parameter)

[yanchith]

As someone who's been waiting for this for years (it would allow me to delete hundreds of lines of workaround code), I am really tempted to say I want to push this through. Realistically, I can imagine reserving a couple of weeks around the end of September. Will check back in then to see if someone else didn't already.