Feature/add listagg macro

[graciegoheen]

This is a:

• bug fix PR with no breaking changes — please ensure the base branch is main
• new functionality — please ensure the base branch is the latest dev/ branch
• a breaking change — please ensure the base branch is the latest dev/ branch

Description & motivation

Create a cross database compatible listagg macro.

Checklist

• I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
  • BigQuery
  • Postgres
  • Redshift
  • Snowflake
• I followed guidelines to ensure that my changes will work on "non-core" adapters by:
  • dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
  • using the limit_zero() macro in place of the literal string: limit 0
  • using dbt_utils.type_* macros instead of explicit datatypes (e.g. dbt_utils.type_timestamp() instead of TIMESTAMP
• I have updated the README.md (if applicable)
• I have added tests & descriptions to my models (and macros if applicable)
• I have added an entry to CHANGELOG.md

[graciegoheen]

Following the example in datediff macro - why is this necessary?

[joellabes]

Good question! Some background:

• The Redshift adapter inherits from Postgres, so will try using Postgres-specific code before reverting to the default (read more)
• Postgres' datediff implementation is very limited and needs some ridiculous hacks.
• Redshift actually has a good, sensible datediff implementation of its own instead of just behaving the same way as Postgres, so we're skipping the Postgres implementation in favour of the base implementation.

This is the right thing to do here as well - Redshift uses listagg, not string_agg, so we should use the default implementation instead of Postgres'.

[joellabes]

This is looking slick @graciegoheen! Thanks for getting the ball rolling so quickly.

There's a handful of ergonomic comments (and, ironically, a request for less documentation 😬 ).

I'd also love to see some more tests which exercise more of the permutations of arguments. For example, one that doesn't provide an order by or limit clause, one that uses distinct once it's added... you can really go down a rabbithole of permutations because they explode exponentially as you add more.

But at a minimum I would say that each argument should be exercised at least once, along with a super basic version that passes no optional values.

For the version without an order by clause, you might have to use a different seed which has multiple copies of the same column value all the way through so that you don't get caught up in the nondeterministic sorting; I don't know. Fortunately the integration tests will let you know if that's a problem!

[joellabes]

Good question! Some background:

• The Redshift adapter inherits from Postgres, so will try using Postgres-specific code before reverting to the default (read more)
• Postgres' datediff implementation is very limited and needs some ridiculous hacks.
• Redshift actually has a good, sensible datediff implementation of its own instead of just behaving the same way as Postgres, so we're skipping the Postgres implementation in favour of the base implementation.

This is the right thing to do here as well - Redshift uses listagg, not string_agg, so we should use the default implementation instead of Postgres'.

[joellabes]

Looks like support for DISTINCT is missing here

[graciegoheen]

I believe DISTINCT is just part of the measure input. So if you wanted to use this macro with DISTINCT, you would write: {{ dbt_utils.listagg('DISTINCT string_text', "','", "order by order_col") }}.

[joellabes]

For optional arguments (order_by, limit), it'd be a good idea to set them to none so that the end user doesn't have to add that in themselves on every invocation. Otherwise I think that the macro will complain that it has four arguments and only two have been provided.

[graciegoheen]

Ah ok - out of curiosity, any reason none is better than an empty string here?

[joellabes]

Not all of your implementations support limit_clause - for the cross-db macros it's important that they all do the same thing, because package maintainers use them to get consistent behaviour across all the different adapters.

There's a couple of options, depending on how committed you're feeling (either option is OK! This is why open source is great):

• Remove the limit clause altogether - if and when someone else comes along who wants to add support, they can do it separately.
• Look into a way to get equivalent behaviour by using other patterns. For example this sort of thing: https://stackoverflow.com/questions/27405150/postgresql-string-agg-with-limited-number-of-elements (you could stick with the native code unless a limit clause is provided, and then switch to the alternative approach).

[graciegoheen]

alright! got this working (redshift was a beast) with the caveat that if there are instances of delimiter_text within your measure, you cannot include a limit_num

[joellabes]

The Redshift implementation makes my brain hurt, but it looks right and there's tests that validate its output so I'm on board! Good work 🥳

Last nitpicks I promise and then let's get this shipped! 🚢

[joellabes]

I like that all of these are in the one model, but I'm wondering if we can move away from magic numbers, and instead show what the goal of the version is. Something like comma_ordered, comma_ordered_limited, comma_unordered, distinct_comma, no_params...

Also, just noticed that all of these use , as their delimiter, which is also the default value, so again it'd be good to test it with something different to make sure that it's doing what we ask it to instead of being correct by accident.

It'd be particularly worthwhile to use a multi-character/whitespace-using delimiter for a bit of novelty, e.g. , or something weird like _|_

[graciegoheen]

Love these ideas! Thanks for all of the feedback :)

[graciegoheen]

Note " , " won't work as a delimiter if you have whitespace in your measure column. This is in-line with the note that "if there are instances of delimiter_text within your measure, you cannot include a limit_num".

[graciegoheen]

Working on making this handle special characters

[joellabes]

@graciegoheen incredible! I'm sorry this turned into such a messy contribution!

A gold star for you ⭐ ⭐ ⭐ ⭐ ⭐

The base branch was changed.