Avoid public constructors when called with columns to avoid unnecessary validation

[mroeschke]

Description

This PR continues an effort to avoid some public constructors when passing a column(s) to avoid unnecessary validation

Maybe we should consider disallowing public constructors to accept columns all-together, but I suspect some RAPIDS libraries are passing columns to public constructors

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[vyasr]

I suspect some RAPIDS libraries are passing columns to public constructors

I'd be fine with disallowing it as long as we tested and updated downstream libraries for this beforehand. devcontainers would be a good way to do this.

[vyasr]

/merge