feat!: obfuscation for mysql2, dalli and postgresql as default option for db_statement

[xuan-cao-swi]

Description

Based on opentelemetry-specification and discussion, change the db.statement default sql to sanitized sql

Test

Tested with basic test suite (e.g. rake test)

[ericmustin]

Obfuscation adds a significant performance penalty.

We should either omit or include by default. The spec says to omit now if I understand correctly?

[arielvalentin]

Obfuscation adds a significant performance penalty.

We should either omit or include by default. The spec says to omit now if I understand correctly?

@ericmustin The spec says nothing about the defaults client-side sanitization, AFAIK, other than:

[2]: Should be collected by default only if there is sanitization that excludes sensitive information.

I interpret that to mean you should only collect it if the SQL is sanitized. If I extrapolate a bit, then it should never record any unsanitized SQL at all.

There are some open issues about db statement sanitization in general:

• Add db.statement sanitization/masking examples semantic-conventions#708
• db.statement sanitization default behavior opentelemetry-specification#3104
• Introduce common configuration option for sanitisation of db.statement. semantic-conventions#705

W/R/T db.operation it says:

[3]: When setting this to an SQL keyword, it is not recommended to attempt any client-side parsing of db.statement just to get this property, but it should be set if the operation name is provided by the library being instrumented. If the SQL statement has an ambiguous operation, or performs more than one operation, this value may be omitted.

Given this information I for one prefer to sanitize by default and omit if the driver does not have a sanitizer implemented.

[ericmustin]

The appropriate place to do sanitization is in the collector, or more generally out of process. If your interpretation of the spec is correct(and I think it is), then that would effectively exclude the possibility of doing sql sanitization in the collector because the spec would dictate it can never be exported without being sanitized. This seems flawed to me.

Additionally we know real examples of the "obfuscate" config option introducing a performance regression in applications at large enterprise organizations which use ruby. Setting it as the default would seem like a poor choice.

I would prefer omit as the default if we're left with a choice btwn omit and "conditionally obfuscate depending on the particulars of driver options", as I think users will find the lack of standard confusing.

[arielvalentin]

The appropriate place to do sanitization is in the collector, or more generally out of process. If your interpretation of the spec is correct(and I think it is), then that would effectively exclude the possibility of doing sql sanitization in the collector because the spec would dictate it can never be exported without being sanitized. This seems flawed to me.

Additionally we know real examples of the "obfuscate" config option introducing a performance regression in applications at large enterprise organizations which use ruby. Setting it as the default would seem like a poor choice.

I would prefer omit as the default if we're left with a choice btwn omit and "conditionally obfuscate depending on the particulars of driver options", as I think users will find the lack of standard confusing.

Would it make the most sense to discussion in the spec issues?

I think the challenge here is there are no OOTB SQL sanitizers in contrib. If there were, then I would totally be onboard with leaving include/raw as the default, but since there isn't I still advocate that we obfucate/sanitize by default.

[cheempz]

Regarding

The appropriate place to do sanitization is in the collector, or more generally out of process.

that disagrees with a recent separate discussion, which is the case with our product--telemetry is exported directly to our platform and doesn't go though a collector.

OTel JS and Java obfuscate/sanitize by default, so there's a bit of precedence despite the much needed spec clarification/configuration naming being discussed :)

[ericmustin]

Indeed, I disagree with the spec discussion. If I'm not
Mistaken, Both github and Shopify have introduced regressions into production by enabling obfuscation in app ootb. Perhaps benchmarking would be useful here, but im dubious of defaulting to obfuscation. The chief concern of observability instrumentation should be minimizing the observer effect as much as possible, and I feel we're disregarding that for what's effectively product/vendor developer experience preferences

[arielvalentin]

For context, at GH we've accepted the trade off so we obfuscate by default. It's slow but we mitigate it with the character limit setting.

[ericmustin]

just circling back, if others feel obfuscate is the appropriate default, then I won't stand in the way here.

[arielvalentin]

Is this all of the data store drivers? Or do the others already have obfuscate by default?

[xuan-cao-swi]

Is this all of the data store drivers? Or do the others already have obfuscate by default?

AFAIK, I think these three gems are the last to set obfuscate to default

[ericmustin]

So it goes

[arielvalentin]

@xuan-cao-swi just one more favor to ask before I merge.

Please update the commit messages to use conventional commits:

https://github.com/open-telemetry/opentelemetry-ruby-contrib/blob/main/CONTRIBUTING.md#use-conventional-commit-messages

This is something that I think requires a Minor release, so please using the BREAKING CHANGE format in the message, e.g.

feat!: Set db.statement option to obfuscate by default

[xuan-cao-swi]

@xuan-cao-swi just one more favor to ask before I merge.

Please update the commit messages to use conventional commits:

https://github.com/open-telemetry/opentelemetry-ruby-contrib/blob/main/CONTRIBUTING.md#use-conventional-commit-messages

This is something that I think requires a Minor release, so please using the BREAKING CHANGE format in the message, e.g.

feat!: Set db.statement option to obfuscate by default

Changed!