Skip to content

feat(clickhouseexporter): Per pipeline JSON support#46553

Merged
songy23 merged 6 commits into
open-telemetry:mainfrom
Garbett1:per-pipeline-json-support
Mar 5, 2026
Merged

feat(clickhouseexporter): Per pipeline JSON support#46553
songy23 merged 6 commits into
open-telemetry:mainfrom
Garbett1:per-pipeline-json-support

Conversation

@Garbett1

@Garbett1 Garbett1 commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Description

Currently, JSON support for ClickHouse is added via a feature flag, so it's turned off or on for all ClickHouse pipelines.

This may not be desirable.

At this moment, there are a few features where Map might still win out, compared to JSON (at least in the medium term)

This shifts the config down to the pipeline level, but also still keeps the feature flag around for now. The behaviour is:

  • feature flag enabled == always JSON (parity with current)
  • feature flag disabled && config.use_json == JSON
  • feature flag disabled && config.!use_json (default) == Map

Testing

Added many tests in the PR!

Documentation

@Garbett1 Garbett1 requested a review from a team as a code owner March 2, 2026 15:24
@Garbett1 Garbett1 requested a review from crobert-1 March 2, 2026 15:24
@linux-foundation-easycla

linux-foundation-easycla Bot commented Mar 2, 2026

Copy link
Copy Markdown

CLA Signed

The committers listed above are authorized under a signed CLA.

@github-actions github-actions Bot added the first-time contributor PRs made by new contributors label Mar 2, 2026
@github-actions

github-actions Bot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Welcome, contributor! Thank you for your contribution to opentelemetry-collector-contrib.

Important reminders:

A maintainer will review your pull request soon. Thank you for helping make OpenTelemetry better!

@Garbett1

Garbett1 commented Mar 2, 2026

Copy link
Copy Markdown
Contributor Author

Well, I'm definitely not a first time contributor 😅 Looks like my CLA has dropped off though.

// testLogsCombinedExporter verifies that Map-based and JSON-based logs exporters
// can operate side-by-side against the same ClickHouse instance.
func testLogsCombinedExporter(t *testing.T, endpoint string, mapBody bool) {
overrideJSONStringSetting := func(config *Config) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is aping the other tests, debated extracting into helpers, but decided against it.

@paulojmdias

Copy link
Copy Markdown
Member

If you're proposing control the JSON support using config instead of feature gate, I don't think the feature gate is needed at all. Otherwise, the user will need to really do two actions to ensure the JSON is supported.

You may also need to add enable_json_type=1 to your endpoint query string parameters.

Looking into the README.md also shouldn't be enough, adding enable_json_type=0 to the ones you want to disable?

Just sharing my thoughts, but let's see what codeowners say about it.

@Garbett1

Garbett1 commented Mar 2, 2026

Copy link
Copy Markdown
Contributor Author

If you're proposing control the JSON support using config instead of feature gate, I don't think the feature gate is needed at all. Otherwise, the user will need to really do two actions to ensure the JSON is supported.

You may also need to add enable_json_type=1 to your endpoint query string parameters.

Looking into the README.md also shouldn't be enough, adding enable_json_type=0 to the ones you want to disable?

Just sharing my thoughts, but let's see what codeowners say about it.

On enable_json_type=1
This is ClickHouse side config on the query param, and doesn't really interfere with the map or not. TBH, it's a bit outdated, as I think all LTS supported versions of CH have this enabled these days.

I don't think the feature gate is needed at all

I agree, but dropping it is not backwards compatible. If we shifted to the config and ignored the feature flag, this would break prod workloads as soon as you updated.

IMO, the config approach I suggest above gives a more graceful way to update. My proposed end state would be that it's controlled only by config.

@paulojmdias

Copy link
Copy Markdown
Member

I agree, but dropping it is not backwards compatible. If we shifted to the config and ignored the feature flag, this would break prod workloads as soon as you updated.

Feature gate is in Alpha stage and not enabled by default at all. You could mark it as deprecated in favor of a config option and open an issue to remove the feature gate code related to it later.

@Garbett1

Garbett1 commented Mar 2, 2026

Copy link
Copy Markdown
Contributor Author

You could mark it as deprecated in favor of a config option and open an issue to remove the feature gate code related to it later.

That's a good shout for the issue, will make a follow up.

In terms of the feature flag being in alpha, I know of a few places that are currently relying on the feature gate, so felt it would be better to do it in two phases (maybe warn at startup)

@SpencerTorres SpencerTorres left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent PR! It's indeed hard to share pipelines with/without JSON, so this is a reasonable feature. I agree with the other review comments, we should deprecate the feature flag. Originally this was added under a feature flag because Map was supposed to be deprecated with JSON becoming the primary pipeline, we didn't want to maintain both implementations. Considering we have both implementations now I suppose there's not much harm in adding this config option. Everything looks good, my only nit is I would prefer json over use_json, but that's not a blocker.

Thanks for the PR!

@Garbett1

Garbett1 commented Mar 5, 2026

Copy link
Copy Markdown
Contributor Author

Just reading the code on the StageDeprecated.

It will throw an error if anyone sets it, not sure if we want a grace period of 1 release before marking that (so if your binary version and config aren't synced up, you don't immediately error)

Happy to defer to how you want to handle it as maintainers.

I mis-read the code! Ran a test, and it worked as we'd hope.

@Garbett1

Garbett1 commented Mar 5, 2026

Copy link
Copy Markdown
Contributor Author

@SpencerTorres , do you want to approve it with workflow perms so it can run the tests?

@Garbett1 Garbett1 requested a review from SpencerTorres March 5, 2026 11:01
@paulojmdias

Copy link
Copy Markdown
Member

/workflow-approve

@paulojmdias

Copy link
Copy Markdown
Member

/workflow-approve

@paulojmdias

Copy link
Copy Markdown
Member

/workflow-approve

@crobert-1 crobert-1 added the ready to merge Code review completed; ready to merge by maintainers label Mar 5, 2026
@songy23 songy23 merged commit a4bd57e into open-telemetry:main Mar 5, 2026
191 checks passed
antonio-mazzini pushed a commit to antonio-mazzini/opentelemetry-collector-contrib that referenced this pull request Mar 5, 2026
…6553)

#### Description
Currently, JSON support for ClickHouse is added via a feature flag, so
it's turned off or on for _all_ ClickHouse pipelines.

This may not be desirable.

At this moment, there are a few features where Map might still win out,
compared to JSON (at least in the medium term)

This shifts the config down to the pipeline level, but also still keeps
the feature flag around for now. The behaviour is:
* feature flag enabled == always JSON (parity with current)
* feature flag disabled && config.use_json == JSON
* feature flag disabled && config.!use_json (default) == Map

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Added many tests in the PR!

<!--Describe the documentation added.-->
#### Documentation
antonio-mazzini pushed a commit to antonio-mazzini/opentelemetry-collector-contrib that referenced this pull request Mar 5, 2026
…6553)

#### Description
Currently, JSON support for ClickHouse is added via a feature flag, so
it's turned off or on for _all_ ClickHouse pipelines.

This may not be desirable.

At this moment, there are a few features where Map might still win out,
compared to JSON (at least in the medium term)

This shifts the config down to the pipeline level, but also still keeps
the feature flag around for now. The behaviour is:
* feature flag enabled == always JSON (parity with current)
* feature flag disabled && config.use_json == JSON
* feature flag disabled && config.!use_json (default) == Map


<!--Describe what testing was performed and which tests were added.-->
#### Testing
Added many tests in the PR!

<!--Describe the documentation added.-->
#### Documentation
antonio-mazzini pushed a commit to antonio-mazzini/opentelemetry-collector-contrib that referenced this pull request Mar 5, 2026
…6553)

#### Description
Currently, JSON support for ClickHouse is added via a feature flag, so
it's turned off or on for _all_ ClickHouse pipelines.

This may not be desirable.

At this moment, there are a few features where Map might still win out,
compared to JSON (at least in the medium term)

This shifts the config down to the pipeline level, but also still keeps
the feature flag around for now. The behaviour is:
* feature flag enabled == always JSON (parity with current)
* feature flag disabled && config.use_json == JSON
* feature flag disabled && config.!use_json (default) == Map

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Added many tests in the PR!

<!--Describe the documentation added.-->
#### Documentation
avleentwilio pushed a commit to avleentwilio/opentelemetry-collector-contrib that referenced this pull request Apr 1, 2026
…6553)

#### Description
Currently, JSON support for ClickHouse is added via a feature flag, so
it's turned off or on for _all_ ClickHouse pipelines.

This may not be desirable.

At this moment, there are a few features where Map might still win out,
compared to JSON (at least in the medium term)

This shifts the config down to the pipeline level, but also still keeps
the feature flag around for now. The behaviour is:
* feature flag enabled == always JSON (parity with current)
* feature flag disabled && config.use_json == JSON
* feature flag disabled && config.!use_json (default) == Map


<!--Describe what testing was performed and which tests were added.-->
#### Testing
Added many tests in the PR!

<!--Describe the documentation added.-->
#### Documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exporter/clickhouse first-time contributor PRs made by new contributors ready to merge Code review completed; ready to merge by maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants