Add bloom_filter_agg Spark aggregate function

[jinchengchenghh]

This function is used in Spark Runtime Filters: apache/spark#35789

https://docs.google.com/document/d/16IEuyLeQlubQkH8YuVuXWKo2-grVIoDJqQpHZrE7q04/edit#heading=h.4v65wq7vzy4q

BloomFilter implementation in Velox is different from Spark, hence, serialized BloomFilter is different.

Velox has memory limit for contiguous memory buffer, hence BloomFilter capacity is less than in Spark when numBits is large. See #4713 (comment)

Spark allows for changing the defaults while Velox does not.

See also #3342

Fixes #3694

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	36b31e5
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/647ed270c4a3d00008e58334

[XinShuoWang]

From my understanding:

1. The expected result is at least 36 bytes, not just "\u0004".
2. You can use return StringView(pointer_to_data, data_size); instead of return "\u0004"; to avoid the risk of truncation.Enhance BloomFilter to serialize and memory track #3861 (comment)
3. Please reconfirm the correctness of the expected result.

[jinchengchenghh]

Thanks, I try to fix as you say, but failed by #4028 (comment).
Can you give me more suggestions?

[jinchengchenghh]

Spark fuzzer test will raise the exception, can you help to fix this? @duanmeng

/mnt/DP_disk1/code/velox/build/velox/exec/tests# ./spark_aggregation_fuzzer_test
E0414 09:32:32.838361 1385171 Exceptions.h:68] Line: /mnt/DP_disk1/code/velox/velox/functions/sparksql/aggregates/BloomFilterAggAggregate.cpp:237, Function:setConstantArgument, Expression: vec.isConstantMapping() originalEstimatedNumItems argument must be constant for all input rows, Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: originalEstimatedNumItems argument must be constant for all input rows
Retriable: False
Expression: vec.isConstantMapping()
Function: setConstantArgument
File: /mnt/DP_disk1/code/velox/velox/functions/sparksql/aggregates/BloomFilterAggAggregate.cpp
Line: 237
Stack trace:
# 0
# 1

[jinchengchenghh]

Can you help review this one? @mbasmanova Thanks!

[mbasmanova]

@jinchengchenghh

#4633 is fixed. Would you re-enable fuzzer test?

Curious how is this function used in Spark to reduce the amount of shuffle data. Is there something I can read about this?

[mbasmanova]

Why is this change? Let's remove. Feel free to open a separate PR with this change along if necessary.

[mbasmanova]

@jinchengchenghh Some comments.

[mbasmanova]

naming: struct members do not have trailing underscore

[mbasmanova]

make this method const

[mbasmanova]

make this method const

[mbasmanova]

Why is this check? Would you add a test case where some of the input data is null?

[jinchengchenghh]

Spark bloomfilter aggregate test only tests the empty input. https://github.com/apache/spark/blob/branch-3.3/sql/core/src/test/scala/org/apache/spark/sql/BloomFilterAggregateQuerySuite.scala#L196
I add the test:

  test("Test that bloom_filter_agg errors null") {
    spark.sql(
      """
        |SELECT bloom_filter_agg(null)"""
        .stripMargin)
  }

It will throw exception:

[DATATYPE_MISMATCH.BLOOM_FILTER_WRONG_TYPE] Cannot resolve "bloom_filter_agg(NULL, 1000000, 8388608)" due to data type mismatch: Input to function `bloom_filter_agg` should have been "BINARY" followed by value with "BIGINT", but it's ["VOID", "BIGINT", "BIGINT"].; line 2 pos 7;

In spark, its first argument is xxhash(table_col), so it won't be null.

Velox BloomFilter accepts uint64_t while xxhash() returns int64_t, so we need to use folly to hash twice.

[mbasmanova]

In spark, its first argument is xxhash(table_col), so it won't be null.

I think it is totally possible that table_col is null for some or all rows.

SELECT bloom_filter_agg(null)

Try changing this to something like this:

SELECT bloom_filter_agg(cast(null as varbinary))

[jinchengchenghh]

I change the test to

  test("Test that bloom_filter_agg errors null") {
    spark.sql(
      """
        |SELECT bloom_filter_agg(cast(null as binary))"""
        .stripMargin)
  }

Different exception:

[DATATYPE_MISMATCH.BLOOM_FILTER_WRONG_TYPE] Cannot resolve "bloom_filter_agg(CAST(NULL AS BINARY), 1000000, 8388608)" due to data type mismatch: Input to function `bloom_filter_agg` should have been "BINARY" followed by value with "BIGINT", but it's ["BINARY", "BIGINT", "BIGINT"].; line 2 pos 7;
'Aggregate [unresolvedalias(bloom_filter_agg(cast(null as binary), 1000000, 8388608, 0, 0), None)]
+- OneRowRelation

This is spark internal function, it is revoked by the planner, for the case table_col is null, it will be
bloom_filter_agg(xxhash64(null)), and xxhash64(null) is 42

[mbasmanova]

This is spark internal function, it is revoked by the planner, for the case table_col is null, it will be
bloom_filter_agg(xxhash64(null)), and xxhash64(null) is 42

Interesting. So the input to bloom_filter_agg is not a value, but a hash of the value. Let's clarify this in the documentation. What's the type if input? Is it VARBINARY or BIGINT?

[jinchengchenghh]

 val rowCount = filterCreationSidePlan.stats.rowCount
    val bloomFilterAgg = new BloomFilterAggregate(new XxHash64(Seq(filterCreationSideExp)), rowCount.get.longValue)

It is BIGINT

[mbasmanova]

It would be more efficient to compute total bytes needed for the whole result and call getBufferWithSpace once.

[mbasmanova]

naming: kDefaultExpe...

Use 1'000'000 for readability

[mbasmanova]

Should the documentation match?

[duanmeng]

@jinchengchenghh

#4633 is fixed. Would you re-enable fuzzer test?

Curious how is this function used in Spark to reduce the amount of shuffle data. Is there something I can read about this?

@mbasmanova Spark Runtime Filters, apache/spark#35789, https://docs.google.com/document/d/16IEuyLeQlubQkH8YuVuXWKo2-grVIoDJqQpHZrE7q04/edit#heading=h.4v65wq7vzy4q

[jinchengchenghh]

@mbasmanova In short, a big table A join with small table B with filter, it will generate the bloomfilter for table B after filter, then broadcast this bloomfilter to A side, and use might_contain to reduce the shuffle data of table A

[jinchengchenghh]

Spark aggregate fuzzer tests passed.

[jinchengchenghh]

I receive a core dump, but I don't think it is caused by my PR

I20230419 03:23:37.675078 303327 ExpressionVerifier.cpp:88] Executing expression: round(bitwise_and(remainder(unaryminus(bitwise_or(add(multiply("c0",round(14,pmod(520542265,bitwise_and("c1","c2")))),unaryminus(add(subtract(round("c3"),bitwise_or(58,44)),"c4"))),69)),unaryminus(bitwise_and("c5",add(round(round(bitwise_or(26,add("c6","c7")),add("c8","c9")),"c10"),add(bitwise_and(26,"c11"),"c12"))))),bitwise_and("c13",abs(round(add("c14",bitwise_and(43,64)),shiftright(subtract(shiftleft(subtract(1529826847,length("c15")),"c16"),length(upper(sha2(VPl+[xY#8IL#&+*X@/TzpX2i2Vi'\9_`|nird}Lqx-^cFtq.O[!'Rs,"c17")))),851317482))))))
I20230419 03:23:37.675148 303327 ExpressionVerifier.cpp:31] 18 vectors as input:
I20230419 03:23:37.675158 303327 ExpressionVerifier.cpp:33] 	[DICTIONARY TINYINT: 100 elements, 8 nulls], [CONSTANT TINYINT: 100 elements, 48]
I20230419 03:23:37.675171 303327 ExpressionVerifier.cpp:33] 	[DICTIONARY INTEGER: 100 elements, 9 nulls], [FLAT INTEGER: 100 elements, 15 nulls]
I20230419 03:23:37.675184 303327 ExpressionVerifier.cpp:33] 	[FLAT INTEGER: 100 elements, 14 nulls]
I20230419 03:23:37.675194 303327 ExpressionVerifier.cpp:33] 	[FLAT TINYINT: 100 elements, 13 nulls]
I20230419 03:23:37.675204 303327 ExpressionVerifier.cpp:33] 	[DICTIONARY TINYINT: 100 elements, 11 nulls], [FLAT TINYINT: 100 elements, 9 nulls]
I20230419 03:23:37.675215 303327 ExpressionVerifier.cpp:33] 	[DICTIONARY TINYINT: 100 elements, 14 nulls], [FLAT TINYINT: 100 elements, 14 nulls]
I20230419 03:23:37.675226 303327 ExpressionVerifier.cpp:33] 	[DICTIONARY TINYINT: 100 elements, 7 nulls], [DICTIONARY TINYINT: 100 elements, 10 nulls], [FLAT TINYINT: 100 elements, 12 nulls]
I20230419 03:23:37.675240 303327 ExpressionVerifier.cpp:33] 	[DICTIONARY TINYINT: 100 elements, 10 nulls], [DICTIONARY TINYINT: 100 elements, 5 nulls], [DICTIONARY TINYINT: 100 elements, 10 nulls], [CONSTANT TINYINT: 100 elements, 51]
I20230419 03:23:37.675254 303327 ExpressionVerifier.cpp:33] 	[DICTIONARY INTEGER: 100 elements, 12 nulls], [DICTIONARY INTEGER: 100 elements, 7 nulls], [DICTIONARY INTEGER: 100 elements, 12 nulls], [DICTIONARY INTEGER: 100 elements, 11 nulls], [FLAT INTEGER: 100 elements, 9 nulls]
E20230419 03:23:37.675282 303327 Exceptions.h:68] Line: ../.././velox/vector/SimpleVector.h:150, Function:toString, Expression: index < length_ (34 vs. 29) Vector index should be less than length., Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (34 vs. 29) Vector index should be less than length.
Retriable: False
Expression: index < length_
Function: toString
File: ../.././velox/vector/SimpleVector.h
Line: 150
Stack trace:
# 0  _ZN8facebook5velox7process10StackTraceC1Ei
# 1  _ZN8facebook5velox14VeloxException5State4makeIZNS1_C4EPKcmS5_St17basic_string_viewIcSt11char_traitsIcEES9_S9_S9_bNS1_4TypeES9_EUlRT_E_EESt10shared_ptrIKS2_ESA_SB_
# 2  _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 3  _ZN8facebook5velox17VeloxRuntimeErrorC2EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bS7_
# 4  _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 5  _ZNK8facebook5velox12SimpleVectorIiE8toStringB5cxx11Ei
# 6  _ZNK8facebook5velox14ConstantVectorIiE8toStringB5cxx11Ei
# 7  _ZNK8facebook5velox14ConstantVectorIiE15toSummaryStringB5cxx11Ev
# 8  _ZNK8facebook5velox10BaseVector8toStringB5cxx11Eb
# 9  _ZN8facebook5velox4test12_GLOBAL__N_112logRowVectorERKSt10shared_ptrINS0_9RowVectorEE
# 10 _ZN8facebook5velox4test18ExpressionVerifier6verifyERKSt10shared_ptrIKNS0_4core10ITypedExprEERKS3_INS0_9RowVectorEEOS3_INS0_10BaseVectorEEbSt6vectorIjSaIjEE
# 11 _ZN8facebook5velox4test16ExpressionFuzzer2goEv
# 12 _ZN8facebook5velox4test16expressionFuzzerESt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt6vectorIPKNS0_4exec17FunctionSignatureESaISD_EESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SF_EEEm
# 13 _ZN12FuzzerRunner3runERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmRKSt13unordered_setIS5_St4hashIS5_ESt8equal_toIS5_ESaIS5_EES7_
# 14 main
# 15 __libc_start_main
# 16 _start

*** Aborted at 1681874617 (Unix time, try 'date -d @1681874617') ***
*** Signal 6 (SIGABRT) (0x4a0df) received by PID 303327 (pthread TID 0x7f1d3f1052c0) (linux TID 303327) (maybe from PID 303327, UID 0) (code: -6), stack trace: ***
(error retrieving stack trace)
/bin/bash: line 8: 303327 Aborted                 (core dumped) _build/debug/velox/expression/tests/spark_expression_fuzzer_test --seed ${RANDOM} --duration_sec 60 --logtostderr=1 --minloglevel=0 --repro_persist_path=/tmp/spark_fuzzer_repro

Exited with code exit status 134

[mbasmanova]

I receive a core dump, but I don't think it is caused by my PR

Looks like this is tracked in #4652

[mbasmanova]

@jinchengchenghh Some follow up comments.

[mbasmanova]

nit: this can be done once before the loop

[mbasmanova]

This variable is not used. Let's remove.

[jinchengchenghh]

It is used in following flatResult->setNoCopy(i, serialized);

[mbasmanova]

Any particular reason this cannot be part of accumulator->serializedSize()? It could return zero in this case.

[jinchengchenghh]

accumulator->serializedSize() never return 0, it is

  uint32_t serializedSize() const {
    return 1 /* version */
        + 4 /* number of bits */
        + bits_.size() * 8;
  }

[mbasmanova]

nit: perhaps, getPreAllocatedBufferSize -> getTotalSize

[mbasmanova]

nit: should we add a method for this for consistency?

accumulator->initialized()

[mbasmanova]

do not abbreviate: currentValue and newValue

[mbasmanova]

This function is small and used only only. Consider folding this logic into the caller for readability.

[mbasmanova]

Do we need both originalEstimatedNumItems_ and estimatedNumItems_ member variables? Looks like just one would be sufficient.

[jinchengchenghh]

Yes, because constant originalEstimatedNumItems_ is the value in Vector, and it will compare with max value to get estimatedNumItems_ which is used in function

[mbasmanova]

Are you saying that estimatedNumItems_ can be lower than originalEstimatedNumItems_ if input value is too large?

[jinchengchenghh]

Yes

[mbasmanova]

Use VELOX_ASSERT_THROW

[mbasmanova]

nit: perhaps,

Creates bloom filter from values of 'x' and returns it serialized into VARBINARY.

``estimatedNumItems`` provides an estimate of the number of unique values of ``x``. Value is capped at 716,800.

``numBits`` specifies max capacity of the bloom filter, which allows to trade accuracy for memory.

[jinchengchenghh]

Change the number of unique values to the number of values, this is spark intend meaning.

[mbasmanova]

@jinchengchenghh I'm not sure I understand why would we want to specify the estimate of the total number of input values. Would you clarify?

[jinchengchenghh]

new BloomFilterImpl(optimalNumOfHashFunctions(expectedNumItems, numBits), numBits);
  /**
   * Computes the optimal k (number of hashes per item inserted in Bloom filter), given the
   * expected insertions and total number of bits in the Bloom filter.
   *
   * See http://en.wikipedia.org/wiki/File:Bloom_filter_fp_probability.svg for the formula.
   *
   * @param n expected insertions (must be positive)
   * @param m total number of bits in Bloom filter (must be positive)
   */
  private static int optimalNumOfHashFunctions(long n, long m) {
    // (m / n) * log(2), but avoid truncation due to division!
    return Math.max(1, (int) Math.round((double) m / n * Math.log(2)));
  }

The spark BloomFilter implementation use this value to compute number of hash functions, it has a optimal value according to the theory. Velox implementation does not use variable number of hash function, it uses constant 4

[mbasmanova]

@jinchengchenghh Are you saying Velox's implementation doesn't use estimatedNumItems arguments? If so, should we remove it? Otherwise, should we document that this argument is not used and remove logic for capping its value and initializing estimatedNumItems_ member variable.

[jinchengchenghh]

For the final solution, I may implement native Spark BloomFilter in Velox, it will have better performance, we can switch to it then.

And if we not specify numBits, we will use estimatedNumItems to estimate numBits, BloomFilter implemenation doesn't use estimatedNumItems argument, but used in bloom_filter_agg

[mbasmanova]

Got it. Let's clarify all this in the documentation. It is not obvious.

[jinchengchenghh]

Do we need to also document the difference between spark, or just clarify the usage in Velox?

[mbasmanova]

Let's document the difference with Spark as well. By default, the assumption is that Velox functions match semantics of the original engine.

[jinchengchenghh]

Ok, updated

[mbasmanova]

from values of hashed value 'x'

It is a bit cryptic. Perhaps, rename x to hash and say something like

.. spark:function:: bloom_filter_agg(hash, estimatedNumItems, numBits) -> varbinary

    Creates bloom filter from input hashes and returns it serialized into VARBINARY. The caller is expected to apply xxhash64 function to input data before calling bloom_filter_agg.

For example, 

     bloom_filter_agg(xxhash64(x), 1000000, 1024)

[mbasmanova]

typos:

In Spark implementation, ``estimatedNumItems`` and ``numBits`` are used to decide the number of hash functions and bloom filter capacity.

In Velox implementation, ``estimatedNumItems`` is not used.

[mbasmanova]

This can be removed.

[mbasmanova]

Let's remove since it is not used.

[mbasmanova]

typos:

In Spark,  the value of``numBits`` is automatically capped at 67,108,864.

In Velxo, the value of``numBits`` is automatically capped at 716,800.

[mbasmanova]

Let's also mention that x / hash cannot be null.

[mbasmanova]

First argument value

Users may not understand what this refers to. How about,

First argument of bloom_filter_agg cannot be null

However, !decodedRaw_.mayHaveNulls() is a very strong check. It may return false even if there are no nulls.

[jinchengchenghh]

I'm a bit confused about It may return false even if there are no nulls.
In my mind, I understand it is supposed to be return false if there are no nulls, return false or true when there are nulls or not, so it is determined there is no nulls.

Other codes obey this rule.
https://github.com/facebookincubator/velox/blob/main/velox/connectors/hive/HiveConnector.cpp#L633
Scalar function document:
https://facebookincubator.github.io/velox/develop/scalar-functions.html

bool mayHaveNulls() : constant time check on the underlying vector nullity. When it returns false, there are definitely no nulls, a true does not guarantee null existence.

[mbasmanova]

Can you check this comment?

[mbasmanova]

Same.

[mbasmanova]

Same.

[mbasmanova]

@jinchengchenghh Thank you for iterating on this PR. There are a lot of tricky details to get right. Some follow-up comments.

[mbasmanova]

Would you generate the docs locally and verify they get formatted nicely. It seems to me that we need some new lines or something around the example.

[jinchengchenghh]

Sorry for the document, I don't know how to confirm the document before.
Now I know how to convert it to html and use Vscode to preview, I will check the html format. Thanks for your kindly review.

[mbasmanova]

typo: Velxo -> Velox

is automatically capped at 716,800.

Let's update PR description to explain where this limitation comes from. CC: @xiaoxmeng

[mbasmanova]

x -> hash

[mbasmanova]

This description needs to be revised.

A version of ``bloom_filter_agg`` that uses ``numBits`` computed as ``estimatedNumItems`` * 8.

``estimatedNumItems`` provides an estimate of the number of values of <TBD: fill in; y seems wrong>.

Value of ``estimatedNumItems`` is capped at 4,000,000.

Does 4M cap come from Spark? If so, let's clarify

Value of ``estimatedNumItems`` is capped at 4,000,000 like to match Spark's implementation.

[jinchengchenghh]

This is Spark implementation https://github.com/apache/spark/blob/branch-3.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/BloomFilterAggregate.scala#L58

It matches Spark.

[mbasmanova]

It looks like defaults are configurable in Spark. Should these be configurable in Velox as well?

  val RUNTIME_BLOOM_FILTER_EXPECTED_NUM_ITEMS =
    buildConf("spark.sql.optimizer.runtime.bloomFilter.expectedNumItems")
      .doc("The default number of expected items for the runtime bloomfilter")
      .version("3.3.0")
      .longConf
      .createWithDefault(1000000L)

  val RUNTIME_BLOOM_FILTER_MAX_NUM_ITEMS =
    buildConf("spark.sql.optimizer.runtime.bloomFilter.maxNumItems")
      .doc("The max allowed number of expected items for the runtime bloom filter")
      .version("3.3.0")
      .longConf
      .createWithDefault(4000000L)


  val RUNTIME_BLOOM_FILTER_NUM_BITS =
    buildConf("spark.sql.optimizer.runtime.bloomFilter.numBits")
      .doc("The default number of bits to use for the runtime bloom filter")
      .version("3.3.0")
      .longConf
      .createWithDefault(8388608L)

  val RUNTIME_BLOOM_FILTER_MAX_NUM_BITS =
    buildConf("spark.sql.optimizer.runtime.bloomFilter.maxNumBits")
      .doc("The max number of bits to use for the runtime bloom filter")
      .version("3.3.0")
      .longConf
      .createWithDefault(67108864L)

[jinchengchenghh]

Now we cannot get configuration when we implement aggregate functions, because we cannot get queryContext config here.
If we need this feature, we should reserve config in GroupingSet when we initialize it in HashAggregation.cpp.
Now just Spiller::Config exists in GroupingSet.
And we need to add a new argument config or context to functions such as addRawInput, it will change all the aggregate function input arguments.
Or we can reserve the config in Aggregate, but I don't suggest this way, it change the code less but Aggregate is created from FunctionRegistry factory, we cannot use constructor to create it with config.

If you think it is needed, I can help to implement it in another PR.

[mbasmanova]

Thank you for pointing this out. Let's create a GitHub issue to explain this use case and discuss how best to implement it. For this PR, let's just mention in the documentation that Spark allows for changing the defaults, but Velox does not.

[mbasmanova]

Same.

A version of ``bloom_filter_agg`` that uses 8,000,000 as ``numBits``.

Would you confirm that this matches Spark's implementation?

[mbasmanova]

This works, but is quite a bit of code and easy to get wrong. For example, by forgetting to call buffer->setSize or forgetting to add buffer_.size() when initializing bufferPtr.

Consider introducing new method:

char* rawBuffer = flatResult->getRawStringBufferWithSpace(totalSize);

This method would return the pointer to the first "writable" byte and update the size of the 'buffer' to include totalSize.

[mbasmanova]

Please, check this comment.

[mbasmanova]

Same.

[mbasmanova]

Would this happen if we run masked aggregation and all rows for a given groups are masked out? Would you add a test case to verify this code path?

[jinchengchenghh]

Can you explain a bit more about BloomFilterAggAggregateTest.emptyInput.
Current test can cover this path, And gluten unit test can run into this path too.

[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from BloomFilterAggAggregateTest
[ RUN      ] BloomFilterAggAggregateTest.basic
not init
[       OK ] BloomFilterAggAggregateTest.basic (40 ms)
[ RUN      ] BloomFilterAggAggregateTest.bloomFilterAggArgument
not init
not init
[       OK ] BloomFilterAggAggregateTest.bloomFilterAggArgument (160 ms)
[ RUN      ] BloomFilterAggAggregateTest.emptyInput
not init
not init
not init
not init
not init
not init
not init
not init
not init
not init
[       OK ] BloomFilterAggAggregateTest.emptyInput (30 ms)

[mbasmanova]

do not abbreviate: vec -> vector or decoded

[jinchengchenghh]

This failure happens again, can you help check it? @mbasmanova

I20230508 01:38:13.277568 301220 AggregationFuzzer.cpp:452] Persisted input: /tmp/spark_aggregate_fuzzer_repro/velox_vector_MHUn0c and plan: /tmp/spark_aggregate_fuzzer_repro/velox_plan_tWDyIv
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: estimatedNumItems argument must be constant for all input rows
Retriable: False
Expression: vector.isConstantMapping()
Function: setConstantArgument
File: ../../velox/functions/sparksql/aggregates/BloomFilterAggAggregate.cpp
Line: 251

[mbasmanova]

CC: @kagamiori @laithsakka @bikramSingh91 @kgpai

[jinchengchenghh]

Do you have further comments? @mbasmanova

[mbasmanova]

@jinchengchenghh The CI is red. Would you rebase the PR and make sure CI is green?

[jinchengchenghh]

The CI passed @mbasmanova

[facebook-github-bot]

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mbasmanova]

linter pointed out that this semi-colon is not needed; let's remove

I'm also seeing that this method is lacking documentation and tests. Would you submit a separate PR to introduce this method, document it clearly and add a test?

[mbasmanova]

A few linter warnings:

[mbasmanova]

A few linter warnings:

[mbasmanova]

A few linter warnings:

[jinchengchenghh]

Fixed the linter warning, can it be imported? @mbasmanova

[facebook-github-bot]

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@mbasmanova merged this pull request in 86137eb.

[conbench-facebook]

Conbench analyzed the 1 benchmark run on commit 86137eb2.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.