Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mep): fix header filter error #2938

Merged
merged 2 commits into from
Jul 14, 2022
Merged

Conversation

onewland
Copy link
Contributor

@onewland onewland commented Jul 14, 2022

I mistakenly thought that the KafkaHeaderFilter was passing matches but it was actually dropping them, so we were dropping all MEP messages (which is fine, it's pre-prod).

This fixes the issue

@onewland onewland requested a review from a team as a code owner July 14, 2022 14:58
snuba/datasets/message_filters.py Outdated Show resolved Hide resolved
snuba/datasets/message_filters.py Outdated Show resolved Hide resolved
Copy link
Member

@evanh evanh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why wasn't this caught with an existing test?

@onewland
Copy link
Contributor Author

onewland commented Jul 14, 2022

Why wasn't this caught with an existing test?

If you look at the original PR #2929, it felt more like a config than a functionality change. Any test would basically be mimicking the tests in test_message_filters

@fpacifici
Copy link
Contributor

Still regarding the test. Did we already create the end to end test for this dataset? That should have caught it

@codecov-commenter
Copy link

Codecov Report

Merging #2938 (e669605) into master (5d9755c) will decrease coverage by 0.05%.
The diff coverage is 100.00%.

❗ Current head e669605 differs from pull request most recent head a2b5d2b. Consider uploading reports for the commit a2b5d2b to get more accurate results

@@            Coverage Diff             @@
##           master    #2938      +/-   ##
==========================================
- Coverage   92.93%   92.87%   -0.06%     
==========================================
  Files         635      635              
  Lines       29069    29081      +12     
==========================================
- Hits        27016    27010       -6     
- Misses       2053     2071      +18     
Impacted Files Coverage Δ
snuba/datasets/message_filters.py 95.83% <100.00%> (+1.23%) ⬆️
snuba/datasets/storages/generic_metrics.py 96.15% <100.00%> (ø)
tests/datasets/test_message_filters.py 100.00% <100.00%> (ø)
snuba/settings/settings_distributed.py 0.00% <0.00%> (-100.00%) ⬇️
snuba/settings/settings_test_distributed.py 0.00% <0.00%> (-100.00%) ⬇️
...ts/0010_groupedmessages_onpremise_compatibility.py 95.55% <0.00%> (-4.45%) ⬇️
snuba/optimize.py 87.50% <0.00%> (-3.75%) ⬇️
snuba/migrations/table_engines.py 95.50% <0.00%> (-3.38%) ⬇️
...nsactions_onpremise_fix_orderby_and_partitionby.py 81.33% <0.00%> (-2.67%) ⬇️
tests/migrations/test_runner_individual.py 91.20% <0.00%> (-2.20%) ⬇️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d9755c...a2b5d2b. Read the comment docs.

@onewland
Copy link
Contributor Author

onewland commented Jul 14, 2022

Still regarding the test. Did we already create the end to end test for this dataset? That should have caught it

The way we write e2e tests (at least since I've joined) directly invokes process_message on the payload (which in production comes after the prefilter):

https://github.com/getsentry/snuba/blob/master/tests/test_generic_metrics_api.py#L114-L136

so we can't apply a header-based pre-filter with this mechanism. To do that I think we'd need to build an actual consumer /processing strategy factory (e.g. https://github.com/getsentry/snuba/blob/master/snuba/consumers/consumer_builder.py#L221-L265)

@onewland onewland merged commit 512e892 into master Jul 14, 2022
@onewland onewland deleted the fix-generic-metrics-filter branch July 14, 2022 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants