Firehose support for multiple filter and rollups in a single instance

[dpajin-kentik]

What are you trying to achieve?

Combining multiple sets of filter/rollup combinations to a single instance of firehose caused unexpected data outputs. Due to this it was necessary to tee flow from one instance of firehose to a second instance allowing each instance to utilize it's own filter set and output rollups as required.

The use of two instances effectively doubles the expected CPU/Memory required to implement the solution
If each firehose instance is limited to only a single --filter then any additional rollups that might require a slightly different filter set would require yet another instance to be tee'd and instantiated increasing the footprint further

The request here is to implement a single firehose instance that can take multiple sets of filter criteria and matching rollup criteria optimizing for resources to limit the required compute footprint required.

What is your current work around? (optional)

tee and spawn a new instance of firehose for each --filter --rollup pair required for reporting purposes

cc @kentik-rbarnes please provide examples of sets filter + rollup that can be used for testing.

[i3149]

Want to allow something like this:

    -filters "string,custom_str.dst_network_bndry,==,external" 
         -rollups s_sum,sum_bytes_out,in_bytes,custom_str.src_subscriber_id
         -rollups s_sum,sum_bytes_out,in_bytes,custom_str.output_provider
    -filters "string,custom_str.src_subscriber_id,!=,''" 
         -rollups s_sum,sum_bytes_out,in_bytes,custom_str.output_provider

Where the indented rollups are calculated after each distinct filter set.

[i3149]

Thoughts on this syntax to link filters and rollups:

   -filters "string,custom_str.dst_network_bndry,==,external,sum_external"
        -rollups s_sum,sum_external,in_bytes,custom_str.src_subscriber_id

This is adding an optional 3rd argument to filter definitions, which is a rollup_name. If there's a rollup_name defined, this rollup will not get applied globally but rather only to values before the named filter is calculated.

rollup_name is the 2rd argument in the rollup definition, after the type value. Here it is sum_external and it can be any string.

This works equally well for flags and using the config file.

[kentik-rbarnes]

how will this work with AND and OR expressions? Do we need to apply the 3rd rollup_name to all instances?

-rollups s_sum,sum_bytes_in,in_bytes,custom_str.input_provider,device_name,input_int_desc,input_int_alias
-filters "string,custom_str.src_network_bndry,==,external and string,custom_str.src_connect_type,==,free_pni and string,custom_str.dst_subscriber_id,!=,''"

becomes
-filters "string,custom_str.src_network_bndry,==,external,sum_bytes_in and string,custom_str.src_connect_type,==,free_pni,sum_bytes_in and string,custom_str.dst_subscriber_id,!=,'',sum_bytes_in"