23.1.0

Various fixes & improvements

• feat(DC): Add simple Dataset configs (#3586) by @rahul-kumar-saini
• bug(DC): Add config built storage keys to all_storages dictionary in factory (#3502) by @enochtangg
• bump search_issues storage occurrence_type_id int size (#3593) by @enochtangg
• feat: Change settings.TOPIC_PARTITION_COUNTS to use logical topic name (#3589) by @lynnagara
• feat: Arroyo 2.4.0 (#3585) by @lynnagara
• feat(admin): implement basic admin user roles (#3522) by @MeredithAnya
• fix: Change DefaultNoneColumnMapper to use a normal set (#3580) by @evanh
• update migrations doc to reflect new style (#3584) by @dbanda
• fix(DC): Bring EntityContainsColumnsValidator back to config entities (#3579) by @volokluev
• feat(slices): Make Subscription Scheduler filter by slice ID (#3338) by @ayirr7
• Revert "fix(DC): Bring EntityContainsColumnsValidator back to config entities (#3540)" (#3578) by @volokluev
• Pick up self-hosted CI bugfix (#3577) by @chadwhitacre
• fix(DC): Bring EntityContainsColumnsValidator back to config entities (#3540) by @volokluev
• feat(oncall): add the request_id to the query breadcrumbs (#3561) by @volokluev
• feat(DC): Add Storage selector support for Entity configuration (#3545) by @enochtangg
• feat(issue-platform): increase occurrence_type_id column size from UInt8 to UInt16 (#3575) by @barkbarkimashark
• feat(issue-platform): process extra fields from event (#3571) by @barkbarkimashark
• docs(schema): why do we promote tags? (#3574) by @onewland
• Remove sorting experiment on sql conditions (#3567) by @enochtangg
• ref: Add docstring about topic_name vs get_physical_topic_name. (#3568) by @lynnagara
• feat(ops): allow disabled entities for storage sets that are not in use (#3572) by @onewland
• test(consumer): Add preliminary unit tests to ConsumerBuilder (#3389) by @ayirr7
• docs(slicing): Add mega-cluster and basic consumer setup (#3433) by @ayirr7
• feat(consumer): Remove parallel collect option (#3555) by @lynnagara

Plus 39 more

22.12.0

Various fixes & improvements

• give coverage xml files different paths to prevent overwrites (#3499) by @dbanda
• ref: Remove redundant filter (#3520) by @lynnagara
• ref(settings): Delete unused settings (#3518) by @lynnagara
• ref: Remove the mock batch writer (#3515) by @lynnagara
• feat(DC): ReplacerProcessor Registered + JSON Schema'd + Built (#3505) by @rahul-kumar-saini
• ref: Rewrite Snuba replacer to not use batch processing strategy (#3512) by @lynnagara
• ref(replacements): Move ProjectsQueryFlags out of errors_replacer (#3504) by @rahul-kumar-saini
• Revert "ref: Rewrite the KafkaConsumerWithCommitLog as a strategy (#3506)" (#3510) by @lynnagara
• ref: Rewrite the KafkaConsumerWithCommitLog as a strategy (#3506) by @lynnagara
• capture migration output on stdout (#3508) by @dbanda
• fix(admin): bind log to configure it before using (#3503) by @dbanda
• ref(replacements): Refactor Schema out of ReplacerProcessor init (#3496) by @rahul-kumar-saini
• ref(admin): minor migrations audit log fixes (#3492) by @MeredithAnya
• Add test migration (#3488) by @dbanda
• Rename optimize.py to test_optimize.py (#3498) by @dbanda
• ref(admin): split started/completed audit migrations (#3493) by @MeredithAnya
• ref(migrations): add copy-tables script (#3241) by @MeredithAnya
• add admin UI build to docker image (#3476) by @dbanda
• feat: Arroyo 2.2.0 (#3494) by @lynnagara
• feat(DC): Discover Storage as a Config (#3480) by @rahul-kumar-saini
• feat(CDC): Dynamic Row Processors (#3483) by @rahul-kumar-saini
• Increase optimize redis TTL (#3484) by @dbanda
• move discover mappers with all the other mappers (#3491) by @volokluev
• ref(abuse) Add more predefined querylog queries (#3489) by @evanh

Plus 72 more

22.11.0

Various fixes & improvements

• feat(migrations): add migration order validator SNS-1831 (#3353) by @dbanda
• feat(api-abuse): Add ability to throttle threads for an entire referrer (#3376) by @volokluev
• add bytes_scanned column to querylog storage (#3381) by @volokluev
• MIGRATION: add bytes_scanned to querylog (#3360) by @volokluev
• feat(slices): Make consumers "slice-aware" (#3259) by @ayirr7
• use new checkout action to fix GHA warnings (#3377) by @volokluev
• feat(DC): More inner types for Array (#3367) by @rahul-kumar-saini
• add bytes scanned to the querylog consumer (#3375) by @volokluev
• feat(replays): Add additional parsing safety to lists and uuids (#3369) by @cmanallen
• fix(init): uWSGI Snuba API processes not initializing Snuba (#3370) by @rahul-kumar-saini
• feat(replays): Add additional parsing safety to integer and datetime column types (#3352) by @cmanallen
• Explain the ordering flags in MIGRATIONS.md (#3365) by @dbanda
• fix(admin): Allow ARRAY JOIN with clickhouse_queries in admin (#3366) by @volokluev
• ref(admin): get_migration_group_policies (#3364) by @MeredithAnya
• Pause optimize on large merges. attempt 2 (#3339) by @dbanda
• Tag keys and values may not be null (#3357) by @cmanallen
• fix(rate_limit): Set explicit TTL for set of open requests [SNS-1864] (#3362) by @untitaker
• fix(logging): Incorrect Snuba init instrumentation (#3361) by @rahul-kumar-saini
• feat(DC): Profiles Storage as Config (#3359) by @rahul-kumar-saini
• clean up project id filtering (#3355) by @volokluev
• remove logging line that is not helpful (#3356) by @volokluev
• feat(replays): Add additional safety to tags processor (#3354) by @cmanallen
• feat(api): Snuba healthcheck observability (#3334) by @rahul-kumar-saini
• add useful test distributed command to make file (#3349) by @dbanda

Plus 68 more

22.10.0

Various fixes & improvements

• build: Actually increase the right CI timeout (#3264) by @lynnagara
• feat: Always pass a commit policy to Arroyo stream processor (#3257) by @lynnagara
• fix(gen-metrics): distributed table name wrong in config (#3260) by @onewland
• feat(TDC): Make custom function processors accessible from config (#3256) by @volokluev
• ref: Small settings cleanup (#3251) by @lynnagara
• fix(TDC) Make ConditionChecker a RegisteredClass (#3248) by @evanh
• ref(mdc): Remove dead code (#3252) by @lynnagara
• fix(JWT): stop trying to get JWT audience dynamically, hard-code in config (#3255) by @onewland
• ref: Remove filtering of error and transaction messages (#3249) by @lynnagara
• ref: Remove default mapping of "transaction" topic to "events" (#3250) by @lynnagara
• ref: Use message.position_to_commit from arroyo (#3246) by @lynnagara
• cleanup(mdc): remove flag for enabling/disabling datasets from config (#3224) by @onewland
• ref(admin): add migration groups list endpoint (#3231) by @dbanda
• cleanup(transactions): remove transactions_ro from the codebase (#3244) by @volokluev
• fix(MDC): Incorrect column name in Gen Metrics (#3245) by @rahul-kumar-saini
• build: Arroyo 1.0.7 (#3243) by @lynnagara
• build: Increase timeout minutes in CI (#3242) by @lynnagara
• ref(admin): add migration_groups API (#3227) by @MeredithAnya
• fix: Remove filtering on offset for transactions subscriptions (#3239) by @lynnagara
• build: Remove old transaction consumer from Freight (#3237) by @lynnagara
• ref(ci): fix set-output / set-state deprecation (#3240) by @asottile-sentry
• fix(TDC) Make QuerySplitStrategy a RegisteredClass (#3238) by @evanh
• feat(transactions): app_start_type reading/writing V2 (#3209) by @philipphofmann
• fix(grouping): Group clickhouse errors by referrer (#3121) by @untitaker

Plus 76 more

22.9.0

Various fixes & improvements

• feat(slicing): add partition mapping (#3135) by @onewland
• fix(MDC): Validate required fields and that no additional fields are added (#3137) by @rahul-kumar-saini
• fix(MDC): Fix to ensure schema is in the correct place (#3136) by @evanh
• feat(MDC): Validate configs in CI (#3128) by @rahul-kumar-saini
• fix e2e tests for dogfood self-hosted changes (#3134) by @hubertdeng123
• ref(MDC): Represent migrations in configuration (#3071) by @evanh
• feat(transactions): Add app_start_type migration (#3124) by @philipphofmann
• feat(upgrade): Similar query processors for both error storages (#3105) by @nikhars
• ref(arroyo): Fix arroyo imports (#3132) by @lynnagara
• fix(settings): Remove requirement of CDC and events to be on same cluster (#3126) by @nikhars
• feat(discover): add group_ids to discover (#3104) by @udameli
• ref(MDC): EntityKey enum to class (#3109) by @rahul-kumar-saini
• feat(mdc): Load entity subscriptions from config (#3107) by @enochtangg
• config(redis) set reinitialize_steps from settings (#3125) by @onewland
• ref(MDC): consolidate query processors into one folder (#3098) by @volokluev
• ref(EntityKey): Rename GROUPEDMESSAGES to GROUPEDMESSAGE (#3123) by @rahul-kumar-saini
• feat(MDC): Generic Metrics Dataset loaded from Config (#3108) by @rahul-kumar-saini
• ref: use internal pypi for prebuilt packages (#3100) by @asottile-sentry
• feat: Add initial documentation for Dataset configuration (#3089) by @evanh
• fix(mdc): use entity name for mapping when type is not sufficient (#3110) by @onewland
• feat(migrations): Update ddl changes script with new path to snuba_migrations (#3103) by @lynnagara
• ref(MDC): StorageKey enum to class (#3096) by @rahul-kumar-saini
• feat(mdc) add configuration for generic metrics distributions (#3102) by @onewland
• feat(mdc): Make translators, validators, and mappers registered classes (#3099) by @volokluev

Plus 32 more

22.8.0

Various fixes & improvements

• feat: Sort raw sql fields and conditions [Experiment] (#2988) by @enochtangg
• Revert "ref(MDC): cleanup dataset factory, formalize factory pattern (#3051)" (#3052) by @volokluev
• ref(MDC): cleanup dataset factory, formalize factory pattern (#3051) by @volokluev
• feat(MDC): Add Storage yaml parsing and schema (#3046) by @rahul-kumar-saini
• config(datasets) - create a PluggableEntity class (#3050) by @onewland
• Generate abstract_column_set from replays schema (#3041) by @cmanallen
• ref: remove future (unused python2 porting library) (#3040) by @asottile-sentry
• cleanup: remove unused query processor PatternReplacer (#3038) by @onewland
• feat(replays): Add urls, user_agent, and replay_start_timestamp fields and processor (#3023) by @cmanallen
• feat(replays): Add urls, user_agent, and replay_start_timestamp migration (#3021) by @cmanallen
• feat(datasets): Add configs directory structure (#3034) by @rahul-kumar-saini
• fix(replacements): Limit size of excluded groups set [INC-190] (#3027) by @untitaker
• fix(inc): Introduce denylist that stops attempting consistent queries per project [INC-190] (#3026) by @untitaker
• Allow us to disable the global rate limiter (#3025) by @fpacifici
• build: Bump confluent sentry-python to 1.9.0 (#3022) by @andriisoldatenko
• config(ds): get rid of existing logic to move to dynamic sampling (#3018) by @onewland
• fix(discover): add missing transaction_source to TRANSACTIONS_COLUMNS (#3020) by @andriisoldatenko
• config(mep): add table rate limit and tuple unaliaser to sets/distributions (#3019) by @onewland
• Remove replays from the skipped migrations group (#3017) by @cmanallen
• feat(rollout): Make transaction query processors same (#3006) by @nikhars
• feat(replays): Enable URL processing in the consumer (#3004) by @cmanallen
• feat(replays): add error_ids column (#3001) by @JoshFerge
• fix(admin): Fix deletion of configs with / in them (#3016) by @nikhars
• config(mep): add schedulers/executors to snuba-stable freight (#3015) by @onewland

Plus 56 more

22.7.0

Various fixes & improvements

• feat(mep): support time bucketing in queries (#2937) by @onewland
• add new subscription-related generic-metrics topics to settings/validation (#2935) by @onewland
• feat(mep): fix header filter error (#2938) by @onewland
• build(deps): bump jsonschema from 4.6.0 to 4.7.1 (#2926) by @dependabot
• ref: Remove temporary subscriptions rollout code (#2934) by @lynnagara
• ref: Fix TypedDict import (#2933) by @lynnagara
• feat(mep): add commit log config to generic metrics storages (#2931) by @onewland
• build(deps): bump @types/react-dom from 18.0.5 to 18.0.6 in /snuba/admin (#2924) by @dependabot
• build(deps): bump urllib3 from 1.26.9 to 1.26.10 (#2922) by @dependabot
• feat(mep): do kafka-header filtering to avoid JSON overhead for generic-metrics (#2929) by @onewland
• build(deps): bump @types/react from 18.0.14 to 18.0.15 in /snuba/admin (#2923) by @dependabot
• feat(mep): add commit log topics config (#2928) by @onewland
• feat(profiling): Add functions consumer and dataset (#2894) by @Zylphrex
• build(deps): bump redis from 4.3.3 to 4.3.4 (#2905) by @dependabot
• build(deps): bump python-rapidjson from 1.6 to 1.8 (#2920) by @dependabot
• ref(subscriptions): Remove legacy tick consumer (#2838) by @lynnagara
• chore(deps): Bump sentry-sdk to 1.7.0 to enable baggage propagation (#2927) by @sl0thentr0py
• feat(mep): subscript fix for multiple value arrays (#2914) by @onewland
• fix: Update Flask (#2880) by @evanh
• feat(replacements) Reimplement bypass properly (#2917) by @fpacifici
• fix(build): remove atomicwrites from requirements.txt (#2919) by @onewland
• feat(replacements) Script to execute batches of replacements (#2916) by @fpacifici
• fix(profiling): Disable vertical merge algorithm (#2912) by @phacops
• feat(replays): add replays dataset to dataset factory (#2913) by @JoshFerge

Plus 41 more

22.6.0

Various fixes & improvements

• ref(attribution): Split RequestSettings into QuerySetings and attribution (#2808) by @volokluev
• feat: Print the ClickHouse host, port and version if invalid version (#2816) by @lynnagara
• enforce retention_days is int type (#2809) by @MeredithAnya
• fix: Update to latest version of Redis (4.3.3) (#2801) by @evanh
• Update clickhouse-driver from 0.2.2 to 0.2.4 (#2805) by @evanh
• fix: Update frontend packages (#2802) by @evanh
• feat(replays): initial replays clickhouse migration (#2681) by @JoshFerge
• feat: Bump arroyo (#2796) by @lynnagara
• feat(metrics) - raw input table and materialized view for generic metrics sets (#2793) by @onewland
• feat: Remove legacy events subscriptions consumer from Freight (#2798) by @lynnagara
• feat: Don't pass next_offset to message (#2792) by @lynnagara
• fix(rate-limit): Don't override SerializableException constructor (#2797) by @nikhars
• feat(subscriptions): Allow scheduling watermark mode to be overridden (#2791) by @lynnagara
• refactor(metrics): Undo Optional in value processors #2794 (#2794) by @rahul-kumar-saini
• Revert "feat(rate-limit): Add rate limit metrics (#2784)" (#2795) by @nikhars
• ref(subscriptions): Executor created within strategy (#2762) by @MeredithAnya
• feat(rate-limit): Add rate limit metrics (#2784) by @nikhars
• build(deps): bump typescript from 4.7.2 to 4.7.3 in /snuba/admin (#2785) by @dependabot
• build(deps): bump @types/react from 18.0.9 to 18.0.10 in /snuba/admin (#2778) by @dependabot
• feat(subscriptions): Remove legacy transactions subscriptions from Freight (#2728) by @lynnagara
• feat(metrics): create generic sets aggregate table + indices (#2782) by @onewland
• refactor(dlq): Rename policy "closure" to "creator" (#2743) by @rahul-kumar-saini
• chore(tiger): Remove comparing results (#2787) by @nikhars
• Revert "fix(tiger): Implement tupleElement function in snuba (#2687)" (#2786) by @volokluev

Plus 59 more

22.5.0

Various fixes & improvements

• fix(tiger): don't call the explicit tuple function in clickhouse (#2699) by @volokluev
• fix(dlq): Metrics ignored messages should be Sentry Errors (#2698) by @rahul-kumar-saini
• fix: Allow arbitrary AppIDs (#2692) by @evanh
• dlq(metrics): Metrics produce policy + basic validation replaced with DLQ stuff (#2614) by @rahul-kumar-saini
• feat(devserver): Force latest offset in devserver (#2573) by @mitsuhiko
• fix(querylog): fix the kafka config, add test for producer (#2695) by @volokluev
• fix kafka config option (#2694) by @volokluev
• fix(querylog): compress, allow larger messages on the querylog producer (#2682) by @volokluev
• ref: try building the arm64 image using linux builders (#2688) by @asottile-sentry
• fix(tiger): Implement tupleElement function in snuba (#2687) by @volokluev
• fix: Support array as well as tuple in condition (#2686) by @evanh
• feat(arroyo): Upgrade Arroyo for new Parallel Transform Step (#2679) by @rahul-kumar-saini
• feat(subscriptions): Deploy transactions scheduler and executor (#2690) by @lynnagara
• Bring in CLA Lite (#2667) by @chadwhitacre
• feat(subscriptions): Record a metric on subscription creation and deletion (#2689) by @lynnagara
• feat(subscriptions): Combined scheduler and executor (#2666) by @volokluev
• chore(tiger): Raise a different timeout exception from tiger clusters (#2680) by @nikhars
• feat(subscriptions): Remove verifier from Freight (#2684) by @lynnagara
• feat(subscriptions): Debug producer shutdown (#2685) by @lynnagara
• feat(subscriptions): Add logging for skipped subscriptions (#2678) by @lynnagara
• ref(dev-tooling): Enable commit tracking attempt II (#2677) by @MeredithAnya
• feat(subscriptions): Add config to skip stale subscriptions in executor (#2676) by @lynnagara
• Skip old subscriptions (#2675) by @fpacifici
• Add logging (#2674) by @fpacifici

Plus 57 more

22.4.0

Various fixes & improvements

• feat(subscriptions): Temporarily run errors in global mode (#2616) by @lynnagara
• fix(subscriptions): Fix InvalidRangeError when rebalancing occurs (#2611) by @lynnagara
• fix(subscriptions): Fix off by one calculation in verifier (#2610) by @lynnagara
• feat(dlq): Introduce DLQ to Metrics with Ignore policy. (#2585) by @rahul-kumar-saini
• fix(query): Avoid race in pipeline delegator (#2608) by @nikhars
• fix: Match only migration files in scripts/ddl-changes.py (#2613) by @asottile-sentry
• feat: Bump arroyo to 0.0.16 (#2606) by @lynnagara
• feat(subscriptions): Count the off by one results separately in the verifier (#2607) by @lynnagara
• fix: Fix Clickhouse version check (#2584) by @evanh
• feat(subscriptions): Ignore stale subscriptions (#2596) by @lynnagara
• feat(subscriptions): Compare the subscription request as well in the verifier (#2600) by @lynnagara
• Revert "fix(query): Avoid race in pipeline delegator (#2597)" (#2601) by @nikhars
• fix(tiger writes): Disable ignore write errors (#2598) by @nikhars
• fix(query): Avoid race in pipeline delegator (#2597) by @nikhars
• feat: Record librdkafka buffer size from the subscriptions executor (#2599) by @lynnagara
• feat(subscriptions): Log non matching results to Sentry (#2595) by @lynnagara
• fix(subscription-verifier): Totals is not always present in result (#2594) by @lynnagara
• feat(subscriptions): Record more metrics from the subscriptions verifier (#2590) by @lynnagara
• feat(subscriptions): Record stale message topic data (#2589) by @lynnagara
• feat(admin): Added SnQL to SQL conversion (#2572) by @rahul-kumar-saini
• fix(consumers): Allow bypassing header based routing for small percentage of messages (#2233) by @nikhars
• ref(devserver): update metrics-consumer command (#2588) by @MeredithAnya
• OPS-1608: add events-subscriptions-verifier to freight config (#2592) by @mwarkentin
• fix(metrics): Remove bucket storages from devserver (#2591) by @jjbayer

Plus 75 more