[SPARK-40687][SQL] Support data masking built-in function 'mask'

[vinodkc]

What changes were proposed in this pull request?

This PR supports data masking built-in Function mask, which returns a masked version of input string.
By default, upper case letters will be converted to "X", lower case letters will be converted to "x" and numbers will be converted to "n".

For example mask("abcd-EFGH-8765-4321") results in xxxx-XXXX-nnnn-nnnn will be able to override the characters used in the mask by supplying additional arguments: the second argument controls the mask character for upper case letters, the third argument for lower case letters and the fourth argument for numbers. For example, mask("abcd-EFGH-8765-4321", "U", "l", "#") should result in llll-UUUU-####-####

Examples:

 > SELECT mask('abcd-EFGH-8765-4321');
   xxxx-XXXX-nnnn-nnnn
 > SELECT mask('abcd-EFGH-8765-4321', 'Q');
   xxxx-QQQQ-nnnn-nnnn
 > SELECT mask('AbCD123-@$#', 'Q','q');
   QqQQnnn-@$#
 > SELECT mask('AbCD123-@$#');
   XxXXnnn-@$#
 > SELECT mask('AbCD123-@$#', 'Q');
   QxQQnnn-@$#
 > SELECT mask('AbCD123-@$#', 'Q','q');
   QqQQnnn-@$#
 > SELECT mask('AbCD123-@$#', 'Q','q', 'd');
   QqQQddd-@$#
 > SELECT mask('AbCD123-@$#', 'Q','q', 'd', 'o');
   QqQQdddoooo
 > SELECT mask('AbCD123-@$#', -1, 'q', 'd', 'o');
   AqCDdddoooo
 > SELECT mask('AbCD123-@$#', -1,-1, 'd', 'o');
   AbCDdddoooo
 > SELECT mask('AbCD123-@$#', -1,-1, -1, 'o');
   AbCD123oooo
 > SELECT mask(NULL, -1,-1, -1, 'o');
   NULL
 > SELECT mask(NULL);
   NULL
 > SELECT mask('AbCD123-@$#', -1, -1, -1, -1);
   AbCD123-@$#

Why are the changes needed?

To support data masking built-in function mask, which returns a masked version of the input string
Ref : Data masking functions

Does this PR introduce any user-facing change?

Yes, added a new build-in function named 'mask'

How was this patch tested?

Added test cases

[HyukjinKwon]

cc @dtenedor FYI. seems like it's sort of similar with #38101 somehow.

[vinodkc]

@HyukjinKwon , this PR is a generic approach to mask the string based on the arguments. This mask function can be applied to any string value and it does not expect a pattern on the input string. Apache Hive mask function has the same logic.
Eg:
Arguments:
* input - string value to mask. Supported types: STRING, VARCHAR, CHAR
* upperChar - character to replace upper-case characters with. Specify -1 to retain the original character. Default value: 'X'
* lowerChar - character to replace lower-case characters with. Specify -1 to retain the original character. Default value: 'x'
* digitChar - character to replace digit characters with. Specify -1 to retain the original character. Default value: 'n'
* otherChar - character to replace all other characters with. Specify -1 to retain the original character. Default value: -1

[melin]

Reference snowflake: https://docs.snowflake.com/en/user-guide/security-column-ddm-use.html

[vinodkc]

Reference snowflake: https://docs.snowflake.com/en/user-guide/security-column-ddm-use.html

@melin
Thanks to the reference from Snoflake, which describes dynamic data masking based on users/roles. As Spark does not support access control/roles, so we cannot implement dynamic data masking based on users/roles.

This PR implements the same data masking functionality as Hive

• mask(string str[, string upper[, string lower[, string number]]])

https://issues.apache.org/jira/browse/SPARK-40686

• mask_first_n(string str[, int n])
• mask_last_n(string str[, int n]
• mask_show_first_n(string str[, int n])
• mask_show_last_n(string str[, int n])
  Ref : https://cwiki.apache.org/confluence/display/hive/languagemanual+udf#LanguageManualUDF-DataMaskingFunctions

[dtenedor]

cc @dtenedor FYI. seems like it's sort of similar with #38101 somehow.

@HyukjinKwon @gengliangwang @vinodkc Yes, it looks like there is an overlap between this work in https://issues.apache.org/jira/browse/SPARK-40686, and https://issues.apache.org/jira/browse/SPARK-40623 which I created to track implementing masking functions in Spark. We should probably dedup these into one effort. @vinodkc do you want to take this one and I can close mine as a dup? I can then help with the review if needed.

[vinodkc]

@dtenedor , yes, please close yours as a dup. I appreciate your help in reviewing this PR and on top of this change, I'm planning to add additional built-in mask functions supported in Hive

[dtenedor]

@vinodkc sounds good! Thanks for working on this 👍

[dtenedor]

Mostly looks good. The test coverage is thorough, and the implementation is relatively simple because there are no error cases. I tried to think of test ideas.

[vinodkc]

Hi @dtenedor , @HyukjinKwon @gengliangwang
Review comments are resolved, can you please check

[dtenedor]

This change LGTM now

[vinodkc]

@HyukjinKwon , @dtenedor , Can you please check this PR?

[dtenedor]

@vinodkc Yes, I said the change LGTM :) sadly I am unable to merge this PR on my own though. We will need @HyukjinKwon to merge it.

[gengliangwang]

@vinodkc this one LGTM overall. I will merge it after my comments are addressed.

[vinodkc]

@gengliangwang, Review comments are addressed.

[gengliangwang]

@vinodkc Thanks for the work! I am focusing on other tasks this week.
Merging this one to the master branch now.

[srielau]

@vinodkc Can you reason why you needed -1 here rather than using e.g. NULL?
AFAIK using a value from another type domain altogether for an argument is quite "unique".
Would you object to changing -1 to NULL?
@gengliangwang @dtenedor

Note that I found the docs here:
https://cwiki.apache.org/confluence/display/hive/languagemanual+udf#LanguageManualUDF-DataMaskingFunctions
But they do NOT describe -1 (or NULL) behavior

[vinodkc]

@srielau , Thanks for checking this. There is no specific reason other than following same approach as in Hive documentation: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMask.java#L41
If still need to change to NULL I can raise a new PR. @gengliangwang @dtenedor , please share your recommendations too

[srielau]

@srielau , Thanks for checking this. There is no specific reason other than following same approach as in Hive documentation: https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMask.java#L41 If still need to change to NULL I can raise a new PR. @gengliangwang @dtenedor , please share your recommendations too

Yes please

[dtenedor]

If still need to change to NULL I can raise a new PR. @gengliangwang @dtenedor , please share your recommendations too

Yes, changing this to NULL would make the behavior more standard. I think it is OK to diverge from Hive in this case. 👍

[vinodkc]

Thanks for the suggestions , I'll raise a new PR to change -1 to NULL

[dtenedor]

Thanks so much for your work, this data masking functionality will be very useful!

[whutpencil]

@vinodkc In the source code of hive, the types supported by the mask have this annotation：

value - value to mask. Supported types: TINYINT, SMALLINT, INT, BIGINT, STRING, VARCHAR, CHAR, DATE

You only implemented the String type, do you need to further add parameter types?