messaging: support vt_message_cols to limit cols

[derekperkins]

Description

this enables users to set an allow-list of columns that will be loaded into memory and streamed to subscribers when they call stream * from tbl

Related Issue(s)

#9666

Checklist

• Should this PR be backported? No
• Tests were added
• Documentation was added or is not required

[mattlord]

@derekperkins I apologize for the delay, I will be reviewing this today.

[derekperkins]

Great, thanks. I'll clean it up and get it passing tests again. I've been using this as our custom build branch trying to solve #9936. We've had this running in prod for 6 weeks or so and it has been working great.

[mattlord]

Looks good. Thank you for working on this! This will make it feasible to have much larger message cache sizes.

I had some questions/comments/nits mostly related to naming but you can make the final call on the naming aspects. Naming things is hard, and as long as we update the docs and explain the field well that's what matters most. Speaking of that, I assume you'll have a docs PR ready to review as well? Let me know. That has gotten much easier now that we have versioned docs and I can offer help if needed.

[rohit-nayak-ps]

aside from minor nits, functionality lgtm

[derekperkins]

Review comments are all addressed now in the last commit in the last 2 commits. This is rebased on top of #10132. Once that is fixed and merged, I'll rebase again to get the tests passing

[mattlord]

Review comments are all addressed now in the last commit in the last 2 commits. This is rebased on top of #10132. Once that is fixed and merged, I'll rebase again to get the tests passing

@derekperkins Should be good now, test wise. I am still working on the locking related stuff in the other PR but the tests should be done.

[derekperkins]

@mattlord I think this should be good to go. I ran into some rebasing issues when switching from being rebased on #10132 to main, not realizing you had squashed. In the end, it was easiest for me to just squash mine too, vs leaving the separate review commit. I addressed all the review items, via name changes and comments. May be worth a quick once over again, but no major functionality changes.

[mattlord]

Looks great! So much easier to read for me. Thank you for taking the time to make those changes, add comments, etc. It's not only easier to read, but more resilient and easier to use as well. ❤️

I had one minor question about the table structure used — hoping we can base it on what we now have in the docs — but that's it.

Thanks again!

[mattlord]

Oh, one last thing... will you be opening a website PR as well so that we add details on this new configuration parameter to the v14 docs here: https://github.com/vitessio/website/blob/prod/content/en/docs/14.0/reference/features/messaging.md and here: https://github.com/vitessio/website/blob/prod/content/zh/docs/14.0/reference/messaging.md ?

I can help with that (creation or review), just let me know.

[mattlord]

Friendly ping on this 🙂

[mattlord]

@derekperkins ping on this. We're quickly running out of time to get this into 14.0 (RC coming soon).

[derekperkins]

Working on the docs PR now