Adapt any string-like type to be used by {fmt} just like the standard…

[DanielaE]

… string types already supported. The adaption is totally non-intrusive.

Signed-off-by: Daniela Engert [email protected]

[DanielaE]

This feature enables using string types like Qt's QString/QStringRef/QStringView, MFC/ATL's CString, Xerces' XStr, and many more. Any strings or string-like types like views with stable references can be adapted to yield compatible fmt::basic_string_views. The adaption neither intrudes fmt::basic_string_view nor the adapted string type.

@vitaut Victor, what do you think about it?

[vitaut]

Interesting. Do you envision this to be used primarily for formatting arguments or format strings as well? If the former then it is already possible to support arbitrary string types via the extension API:

template <>
struct fmt::formatter<MyString> : fmt::formatter<string_view> {
  auto format(const MyString& s, format_context& ctx) {
    return formatter<string_view>::format({s.data(), s.size()}, ctx);
  }
}; 

Edit: fixed example.

[DanielaE]

I see the use case in both the format string and the formatting arguments. I am particularily interested in using QStrings (typically in the form of tr("to-be-translated-literal")) or Boost.Locale messages (the proxy objects returned by translate("to-be-translated-literal") or translate("singular form", "plural form")) as format strings to get rid of Boost.Format and the tedious manual conversions required there (not to speak of the other disadvantages) in our codebase. As a companion work item I've augmented Boost.Locale such that the message proxies can return std::*::basic_string_views with stable references in addition to the old API offering only the implicit conversion to std::basic_strings which would create dangling references within the bowels of fmt during the formatting process. I am also exploring to make fmt create such foreign strings as long as they offer std::back_insert_iterator<ForeignString> to be used by format_to, possibly wrapped into a convenience template.

At present I am still working on exploring the ease (or tediousness) of using the extension API of fmt which will be part of my presentation at Meeting C++ on behalf of evangelizing {fmt}. Adapting strings from other (possibly old and no longer maintained) parts or 3rd-party libraries is a major bonus in my point of view. I see fmt as a hub to mediate between all those strings or string-like types. If you don't mind, I'd like to contact you by email for some specific questions on possibly missing parts in the extension API and your rationale for leaving them out.

[vitaut]

I am particularily interested in using QStrings (typically in the form of tr("to-be-translated-literal")) or Boost.Locale messages (the proxy objects returned by translate("to-be-translated-literal") or translate("singular form", "plural form")) as format strings ...

Sounds reasonable. One question regarding the API: is there any particular reason to use specialization and not function overloading, possibly with ADL? Something along the lines of

template <typename Char>
fmt::basic_string_view<Char> to_string_view(const StringType<Char> &S) {
  return { S.data(), S.length() };
}

// and then enable_if on to_string_view(declval<StringType<Char>>()

At present I am still working on exploring the ease (or tediousness) of using the extension API of fmt which will be part of my presentation at Meeting C++ on behalf of evangelizing {fmt}.

That's awesome, thank you for doing this! Let me know if you have any ideas on improving the extension API.

If you don't mind, I'd like to contact you by email for some specific questions on possibly missing parts in the extension API and your rationale for leaving them out.

I don't mind at all and will be happy to answer any question you might have.

[DanielaE]

One question regarding the API: is there any particular reason to use specialization and not function overloading, possibly with ADL? Something along the lines of ...

I have multiple reasons for that decision:

• structural similarity to formatter extensions.
• structs/classes can carry additional information. In this case I use the char_type type member to advertise the char types that the adapted string type can provide with stable references. F.i. in the case of 'QString', taking basic_string_views are possible only with wchar_t (on Windows) and char16_t on all platforms because QString stores its data in uint16_t code units - only. Therefore QStrings force respective format contexts when given as format string. As a formatting argument it may be used directly with matching format contexts or else require the automatic generation and instantiation of a custom formatter (much like with types that have an operator<< specialization). Such a generated formatter must then hold on to the temporary that is returned by the conversion of a QString to the required output char type.
• keeping all things related to the string adaption together. I'm not sure if the current members are sufficient, in particular with proxy-like types and conversions yielding temporaries that then lead to dangling references further down in fmt.
• not littering other namespaces with fmt-related stuff.
• the possibility of string types in inline namespaces with evolving, changing-over-time namespaces where those types live in. For these types to be picked up by ADL, your proposed to_string_view function must be put into the very same namespace where the string type itself lives in. Users may not be able to put the overload into the correct, unstable namespace of that type. At least this is my understanding of the ADL mechanism. Specializing the proposed foreign_string_adapter class in namespace fmt makes the whole process completely agnostic of such namespace-voodoo.

[vitaut]

structural similarity to formatter extensions.

There were multiple suggestions to replace specialization with overloading there as well, most recently in #518 (comment), so I wouldn't necessarily recommend trying to mimic it.

structs/classes can carry additional information.

I think the character type can also be detected in the overload case from the return type of to_string_view.

I'm not sure if the current members are sufficient, in particular with proxy-like types and conversions yielding temporaries that then lead to dangling references further down in fmt.

The current approach is to try disallowing conversions that result in temporaries whenever possible.

not littering other namespaces with fmt-related stuff.

The only reason it lives in the fmt namespace is because there is no standard mechanism to recognize if some type is string-like. Being string-like is a property of a type and the mechanism to recognize it can reasonably live in the type's namespace.

the possibility of string types in inline namespaces with evolving, changing-over-time namespaces where those types live in.

AFAICS ADL works with inline namespaces too: https://godbolt.org/z/k5QOnP

All that said, I don't feel strongly about it and see good arguments both ways.

[DanielaE]

All that said, I don't feel strongly about it and see good arguments both ways.

Right. At the end of the day, you are the master chief of {fmt}, so you decide about the direction of development. 😄 My strong expectation as user of a library is as much ease-of-use as reasonably possible to implement without putting too much burden on the developers of the library. In our team we prefer less typing over more typing (the latter usually a result of the notion of making anything explicit that can be made explicit) wherever possible without introducing nasty sideeffects. Therefore I prefer solutions being welcoming to as many string-like types as possible to be digested by {fmt} without requiring additional hoops by users when they call call fmt::functions.

[vitaut]

ADL-based approach seems slightly simpler so I'd recommend going with it unless there are some technical obstacles to applying it here. (For example, in the extension API the parse function doesn't take the object to be formatted which makes ADL problematic.)

[DanielaE]

As it turned out, using ADL detection given the unlimited zoo of class types rather than partially specializing a well-known template class is incredibly difficult to implement (at least with my limited capabilities in metaprogramming). My very first attempt worked very well with VS2017, VS2015, the clang versions from your Travis setup, and gcc 8.1 in my mingw environment. All other compilers (i.e. VS2013 and gcc at least up to version 6) were failing like crazy. I suspect that they all suffer from the same problem (at least the failures being identical suggest that): they go a bridge to far with accepting conversions during overload resolution. After days of despair I finally came up with some kind of hack to prevent these old compilers from even attempting the search for to_string_view for some explicitly marked fmt-internal types. If you want to experience this weird behavior yourself, simply replace the computed typename internal::accept_as_external_string<S>::type by std::true_type and run the test-suite with one of the mentioned compilers.
If you happen to have a better solution for that problem I'm all ears. I've completely exhausted my pool of ideas.

[vitaut]

Looks good, thanks! Just one comment re to_string_view namespace. Sorry you had to go through all the hoops to implement the ADL-based approach. I thought it would be easier but, as it is often the case, legacy compilers are a pain. I'll see if we can simplify this a bit, but that's not critical; the most important thing is that the user code looks simple.

[vitaut]

Now that to_string_view is an extension point, it should live in the fmt namespace. This will also eliminate the need for using internal::to_string_view and, likely, the internal2 namespace.

[DanielaE]

Hi Victor!
Actually, the fact that the built-in to_string_view sits in a sibling namespace to internal2 is the gist of the magic that makes the whole kaboodle work in the first place! In fact, the ADL detection would work much easier without all of this crazy stuff if the detection class could sit in a sibling namespace of fmt itself (which it can't). As soon as you move internal::to_string_view and the internal2 detection logic into the same namespace (or the detection logic into a child namespace such as internal), the ADL search would pick up the built-in to_string_view as possibly viable candidate again if a compiler finds some acceptable conversion path. The whole point of the ADL logic is to find only as exact matches as possible without competing overloads.

Did you happen to run a quick test on your machine with your suggested changes? In particular with some of the mentioned problematic compilers?

[DanielaE]

Ok, I've run a quick check on my Visual Studio here. Removing all occurences of using internal::to_string_view' and relocation to_string_viewfrom namespacefmt::internalto namespacefmt` made no difference on VS2017, but VS2013 (and possibly all of the old gcc versions) is failing again (in this case when compiling format.cc) :

1>------ Build started: Project: fmt, Configuration: Debug x64 ------
1>format.cc
1>E:\fmt\vs\fmt/format-inl.h(236): error C2783: 'std::basic_string<Char,std::char_traits<Char>,std::allocator<_Other>> fmt::v5::vformat(const S &,fmt::v5::basic_format_args<buffer_context<Char>::type>)' : could not deduce template argument for 'Char'
1>        e:\fmt\vs\fmt\core.h(1498) : see declaration of 'fmt::v5::vformat'
1>E:\fmt\vs\fmt/format-inl.h(236): error C2660: 'fmt::v5::format_system_error' : function does not take 2 arguments
1>E:\fmt\vs\fmt/format-inl.h(910): error C2752: 'fmt::v5::internal::format_string_traits<fmt::v5::wstring_view,void>' : more than one partial specialization matches the template argument list
1>        e:\fmt\vs\fmt\core.h(511): could be 'fmt::v5::internal::format_string_traits<S,std::enable_if<std::is_base_of<fmt::v5::basic_string_view<S::char_type>,S>::value,void>::type>'
1>        e:\fmt\vs\fmt\core.h(562): or       'fmt::v5::internal::format_string_traits<S,std::enable_if<fmt::v5::internal2::has_to_string_view<S>::value,void>::type>'
1>Done building project "fmt.vcxproj" -- FAILED.

[DanielaE]

Even explicitly knocking-out fmt::basic_string_view and derivatives by explicitly specializing accept_as_external_string<T> to std::false_type, and removing specializations of fmt::to_string_view based on this predicate doesn't help.

It looks like my metaprogramming-fu isn't powerful enough.

[vitaut]

Did you happen to run a quick test on your machine with your suggested changes? In particular with some of the mentioned problematic compilers?

Hmm, it does break the legacy compilers. There is also some redundancy between to_string_view and format_string_traits. Let me take a look in more details and report back.

[DanielaE]

This disambiguation is required for string-like types like boost::locale::basic_message<Char> which have implicit conversion operators to a type that a fmt::basic_string_viewcan be constructed from. I've noticed this problem during today's new tests with {fmt} and translation. I'll probably amend the last commit by a test case for such situations later this day.

[DanielaE]

Test updated to catch implicit conversions.

[vitaut]

I've made to_string_view in the fmt namespace work on legacy compilers: 6e2b16a. The implementation is a bit messy but it works. The key was to get rid of format_string_traits which duplicated the ADL-based extension machinery and define to_string_view for all string-like types of interest. Could you incorporate this into your PR?

[DanielaE]

Do you prefer it this way or shall I squash those five commits into one?

[vitaut]

Do you prefer it this way or shall I squash those five commits into one?

Definitely squashed. Also internal2 should probably go away. Thanks!

[DanielaE]

It looks like the automated test are no longer triggered. I've run the tests for MSVC2013/2015/2017 here locally on my machine and triggered the Travis PR test manually from my GitHub repo: test results. All of them passed.

[vitaut]

Merged with minor tweaks (mostly stylistic to improve consistency with the rest of the project) in 2c81c85. Thanks for your work!

[DanielaE]

Great - thanks a lot!
Does there happen to be some kind of standing document about your stylistic preferences? I'm asking because I still haven't figured out what these might be - at least not in a consistent manner. E.g. is it 2 chars worth of indentation here or is it 4? I've switched my code bases completely over to clang-format. No more discussions with myself 😄

[vitaut]

It's basically Google style with snake_case (for standardization reasons) and exceptions: https://github.com/fmtlib/fmt/blob/master/CONTRIBUTING.rst . I should probably switch to clang-format as well.

[cruisercoder]

I'm a bit unclear from this thread on these two cases:

• writing ptr/length in an extension (formatter<string_view>::format({s.data(), s.size()}...?)
• supporting custom string types through to_string_view (how?)

[vitaut]

If you have a string type MyString, you can make it work with fmt as follows:

template <>
struct fmt::formatter<MyString> : fmt::formatter<string_view> {
  auto format(const MyString& s, format_context& ctx) {
    return formatter<string_view>::format({s.data(), s.size()}, ctx);
  }
}; 

You don't need to use to_string_view.