DO NOT MERGE: Idea on Redacting of the Sensitive Options

[adam-christian-software]

This is just a POC to show possibility of redacting sensitive options. This will not be merged.

@adutra & @adnanhemani - Take a look at my commit 456c0e5 and see whether this satisfies @adnanhemani's criteria of having the ability to do redaction.

[adutra]

The approach is very interesting imho. Thanks for the proposal!

A few general remarks:

1. Redacting fields from JSON is tricky, we must be sure to replace the field with a valid value for the field type. Granted, most fields we'll redact will be of string type. null is probably a safer choice but even then, you could break the JSON schema if the field is non-nullable.
2. I wonder if the redaction rules should be made configurable. Some users may want more stuff redacted, some others may not want anything redacted (e.g. some use cases may require access to the table location). We should imo strive to be the least prescriptive possible, providing sane defaults for everyone to start with.
3. We may explore the possibility to use dedicated libraries for redaction, e.g. Phileas – although, probably overkill as a first implementation.
4. The redaction system may be considerably simpler to implement if flattened event types are implemented: https://lists.apache.org/thread/swn73976xj1lx8tkmcxpfvc2gv6znn2h.

[adnanhemani]

Took a quick look, this looks great @adam-christian-software! I think we are good to unblock #2962 - I will make a comment on there and look for another review.

Agreed on most points that @adutra said above. A few questions/followups:

• Regarding redactions in strings: Aren't we transforming all POJOs into strings for the matter of this transformation? In that case, isn't a string okay?
• Regarding configurable redaction rules: This would be interesting indeed - but how would some user do this? They would have to write new Java code and re-compile Polaris for this? Or were you thinking of something else?
• Agreed on the flattening making this all easier. Do we know if @olsoloviov is working on it, as per the last email on the ML thread?

[olsoloviov]

• Agreed on the flattening making this all easier. Do we know if @olsoloviov is working on it, as per the last email on the ML thread?

@adnanhemani, I am not working on that one yet, planned to do it after multiple listeners. I will take a look to see if it makes sense to reassess priorities