[API Proposal]: Add Enumerable.Concat & Flatten overloads

[TonyValenti]

Background and Motivation

When working with different APIs, often times it increases code readability to aggregate results and then linearize them. For example, I could write the code:

Command.NameInfo.OnlineAccounts = new List<OnlineAccount>();
Command.NameInfo.OnlineAccounts.AddRange(Input.WebAddresses.Select(x => CreateWebAddress(x)));
Command.NameInfo.OnlineAccounts.AddRange(Input.EmailAddresses.Select(x => CreateEmailAddress(x)));

or I could write:

Command.NameInfo.OnlineAccounts = new[] {
    Input.WebAddresses.Select(x => CreateWebAddress(x)),
    Input.EmailAddresses.Select(x => CreateEmailAddress(x))
}.Concat().ToList();

The second is much more clear and concise.

Proposed API

namespace System.Linq 
{
    public static class Enumerable 
    {
        public static IEnumerable<T> Concat<T>(this IEnumerable<TSource> first, IEnumerable<TSource> second);
+        public static IEnumerable<T> Concat<T>(params IEnumerable<T>[] sources);
+        public static IEnumerable<T> Concat<T>(this IEnumerable<IEnumerable<T>> sources);
    }
}

Usage Examples

var source = Enumerable.Range(0, 2);

Enumerable.Concat(source, source, source, source); // 0, 1, 0, 1, 0, 1, 0, 1
Enumerable.Repeat(source, 3).Concat(); // 0, 1, 0, 1, 0, 1

Alternative Designs

Just use SelectMany(x => x)

Risks

None other than System.Linq size concerns.

[ghost]

Tagging subscribers to this area: @eiriktsarpalis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

When working with different APIs, often times it increases code readability to aggregate results and then linearize them. For example, I could write the code:

Command.NameInfo.OnlineAccounts = new List<OnlineAccount>();
Command.NameInfo.OnlineAccounts.AddRange(Input.WebAddresses.Select(x => CreateWebAddress(x)));
Command.NameInfo.OnlineAccounts.AddRange(Input.EmailAddresses.Select(x => CreateEmailAddress(x)));

or I could write:

            Command.NameInfo.OnlineAccounts = new[] {
                Input.WebAddresses.Select(x => CreateWebAddress(x)),
                Input.EmailAddresses.Select(x => CreateEmailAddress(x))
            }.SelectMany().ToList();

The second is much more clear and concise.

Proposed API

public static IEnumerable<T> SelectMany<T>(this IEnumerable<IEnumerable<T>> This) {
            return This.SelectMany(x => x);
        }

        public static IEnumerable<T> Concat<T>(this IEnumerable<IEnumerable<T>> This) {
            var ret = Enumerable.Empty<T>();

            foreach (var item in This) {
                if(item == null) {
                    throw new NullReferenceException();
                }

                if(ret == Enumerable.Empty<T>()) {
                    ret = item;
                } else {
                    ret = Enumerable.Concat(ret, item);
                }

            }

            return ret;
        }

Usage Examples

            Command.NameInfo.OnlineAccounts = new[] {
                Input.WebAddresses.Select(x => CreateWebAddress(x)),
                Input.EmailAddresses.Select(x => CreateEmailAddress(x))
            }.SelectMany().ToList();

Alternative Designs

Just use SelectMany(x => x)

Risks

None

Author:	TonyValenti
Assignees:	-
Labels:	api-suggestion, area-System.Linq, untriaged
Milestone:	-

[eiriktsarpalis]

I'm assuming SelectMany() and Concat() are just alternative names for the same proposed method? Why not just stick with one? I personally would prefer exposing Concat() as an overload, since F# uses the same name for its equivalent method.

[svick]

Note that another way to write this code is to use the existing overload of Concat:

Command.NameInfo.OnlineAccounts = 
    Input.WebAddresses.Select(x => CreateWebAddress(x))
        .Concat(Input.EmailAddresses.Select(x => CreateEmailAddress(x)))
        .ToList();

Or:

Command.NameInfo.OnlineAccounts = Enumerable.Concat(
    Input.WebAddresses.Select(x => CreateWebAddress(x)),
    Input.EmailAddresses.Select(x => CreateEmailAddress(x))
).ToList();

But you do have to choose between the fluent assymetric form and the non-fluent symmetric form. So I think the proposed overload of Concat would still be a good addition.

[eiriktsarpalis]

Agreed, @TonyValenti if you could update the OP following the API proposal template we can consider this proposal for .NET 7.

[danielchalmers]

I often make a Flatten<T>(this IEnumerable<T> l)=>l.SelectMany(x=>x) extension so I would love to see this in the BCL!

[eiriktsarpalis]

@danielchalmers please consider upvoting the original post, it provides a good signal on whether a particular proposal is popular!

[eiriktsarpalis]

Hi @TonyValenti, before I can mark this ready for API review would it be possible to update the original post so that it follows our API review template?

[TonyValenti]

@eiriktsarpalis - Maybe I'm missing something, but I think it does. This is the format, right?
https://github.com/dotnet/runtime/issues/new?assignees=&labels=api-suggestion&template=02_api_proposal.yml&title=%5BAPI+Proposal%5D%3A+

[eiriktsarpalis]

A few issues with the original text I would like to call out:

• Proposal contains two different methods for the same functionality: SelectMany and Concat. One needs to be removed, I would recommend keeping Concat.
• The proposed API is missing namespace and declaring class for the proposed methods.
• The proposed API contains implementation, which is not needed from the API review perspective.

The API review document contains links to a few API proposal issues, if you're looking for real-world examples.

[TonyValenti]

@eiriktsarpalis Thanks! I just updated it. I left the implementation just because I felt that added more clarity. I also added a params overload for Concat that will make joining multiple enumerables easier.

[eiriktsarpalis]

@TonyValenti I've updated your proposal slightly to better match the template. The reason we want this is to allow the API review committee quickly read and understand what is being proposed.

[TonyValenti]

Great! Thanks so much!

[bartonjs]

Video

• We changed the signature of the params overload to Concat to support extension invocation and look like an easier overload.
• We renamed the Concat(IEnumerable<IEnumerable<T>>) to Flatten to better describe its use.

namespace System.Linq 
{
    public static partial class Enumerable 
    {
        public static IEnumerable<TSource> Concat<TSource>(
            this IEnumerable<TSource> first,
             IEnumerable<TSource> second,
             params IEnumerable<TSource>[] rest);


        public static IEnumerable<TSource> Flatten<TSource>(this IEnumerable<IEnumerable<TSource>> sources);
    }

    public static partial class Queryable
    {
        public static IQueryable<TSource> Concat<TSource>(
            this IQueryable<TSource> source1,
             IEnumerable<TSource> source2,
             params IEnumerable<TSource>[] rest);


        public static IQueryable<TSource> Flatten<TSource>(this IQueryable<IEnumerable<TSource>> sources);
    }
}

[svick]

@bartonjs Why does the params overload have the second parameter? Couldn't it be just:

public static IEnumerable<TSource> Concat<TSource>(
    this IEnumerable<TSource> first,
    params IEnumerable<TSource>[] rest);

[eiriktsarpalis]

@svick we wanted to prevent concatenation with zero parameters, i.e. first.Concat(). Technically speaking it's a valid operation but it doesn't make a lot of sense.

[eiriktsarpalis]

I just realized that I completely forgot about IQueryable overloads:

namespace System.Linq 
{
    public static partial class Queryable
    {
        public static IQueryable<TSource> Concat<TSource>(
            this IQueryable<TSource> source1,
             IEnumerable<TSource> source2,
             params IEnumerable<TSource>[] rest);


        public static IQueryable<TSource> Flatten<TSource>(this IQueryable<IEnumerable<TSource>> sources);
    }
}

@bartonjs should we be reviewing those seperately?

[stephentoub]

Did we discuss params Span at all? Assuming that shows up in C# 11, would we still use params arrays here, would we add both, or would we say as a general rule we prefer spans moving forward?

[bartonjs]

@bartonjs should we be reviewing those seperately?

My recollection is we have a blanket policy of "oh, right, make queryable look the same as enumerable"

[bartonjs]

Did we discuss params Span at all? Assuming that shows up in C# 11, would we still use params arrays here, would we add both, or would we say as a general rule we prefer spans moving forward?

We did. Partly it'll end up depending on what it looks like for those methods to be called by VB/F#. If it Just Works, then @terrajobst suggested we'd go back and look at new-in-7 API that we could remove the redundancies from. If it doesn't just work, then we'll probably add them in parallel anywhere other than low-level types.

[ghost]

This issue has been marked with the api-needs-work label. This may suggest that the proposal requires further refinement before it can be considered for API review. Please refer to our API review guidelines for a detailed description of the process.

When ready to submit an amended proposal, please ensure that the original post in this issue has been updated, following the API proposal template and examples as provided in the guidelines.

[eiriktsarpalis]

Changed to api-needs-work due to feedback in #61230. cc @stephentoub

[TahirAhmadov]

@bartonjs did you consider the solution I proposed here ?

[LeaFrock]

No progress for .NET 8? Does IQueryable block it?

[JBrown526]

It would be nice to have a Flatten() or parameterless SelectMany() implementation at least, as the current usage of nestedCollections.SelectMany(x => x);` never seems particularly intuitive or meaningful for me

[julealgon]

or I could write:

Command.NameInfo.OnlineAccounts = new[] {
    Input.WebAddresses.Select(x => CreateWebAddress(x)),
    Input.EmailAddresses.Select(x => CreateEmailAddress(x))
}.Concat().ToList();

With collection literals and the spread operator, I assume this example would be completely nullified since you can just do:

Command.NameInfo.OnlineAccounts = [
    ..Input.WebAddresses.Select(x => CreateWebAddress(x)),
    ..Input.EmailAddresses.Select(x => CreateEmailAddress(x))
];

Or am I missing something?