Is pre-compile pattern string in regexp_match operation

[zhuliquan]

Is your feature request related to a problem or challenge?

I noticed below code:

datafusion/datafusion/physical-expr/src/expressions/binary.rs

Lines 537 to 564 in 6a4a280

	RegexMatch => binary_string_array_flag_op_scalar!(
	array,
	scalar,
	regexp_is_match,
	false,
	false
	),
	RegexIMatch => binary_string_array_flag_op_scalar!(
	array,
	scalar,
	regexp_is_match,
	false,
	true
	),
	RegexNotMatch => binary_string_array_flag_op_scalar!(
	array,
	scalar,
	regexp_is_match,
	true,
	false
	),
	RegexNotIMatch => binary_string_array_flag_op_scalar!(
	array,
	scalar,
	regexp_is_match,
	true,
	true
	),

This looks like every time record_batch is evaluated, it will execute the compiled pattern string and use the compiled results to match arrow-array

Describe the solution you'd like

when building binary physical expr , we can pre-compile pattern string if op is regex_match

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

Hi @zhuliquan -- I believe you are correct that the regexp is re-compiled for each batch. This is mentioned as one of the use cases for #8051

I think it would be a nice improvement to look into pre-compiling the regexp somehow, though last time I checked the only benchmark we have (one of the clickbench queries) compiling the regexp was not a significant consumer of time

[zhuliquan]

Hi @zhuliquan -- I believe you are correct that the regexp is re-compiled for each batch. This is mentioned as one of the use cases for #8051

I think it would be a nice improvement to look into pre-compiling the regexp somehow, though last time I checked the only benchmark we have (one of the clickbench queries) compiling the regexp was not a significant consumer of time

I think should pre-compiled when create binary-physical-expr, instead of evaluating record_batch.

datafusion/datafusion/physical-expr/src/expressions/binary.rs

Lines 60 to 66 in 57280e4

	pub fn new(
	left: Arc<dyn PhysicalExpr>,
	op: Operator,
	right: Arc<dyn PhysicalExpr>,
	) -> Self {
	Self { left, op, right }
	}

[alamb]

I think should pre-compiled when create binary-physical-expr, instead of evaluating record_batch.

I think this makes sense to me. Thank you

One thing to look into might be to replace the regexp match operator with a function (and have the function's implementation store the precompiled regexp)

One tricky part might be serialization (in datafusion-proto). We can probably handle it via an extension codec or something.