OpenTelemetry Protocol with Apache Arrow Exporter component#31996
Conversation
Minor changes: - Lint fixes to pass checks in opentelemetry-collector-contrib, drafted in open-telemetry/opentelemetry-collector-contrib#31996 - Recent work in the collector core component.ValidateConfig code makes it unnecessary for the Config struct in this repo to define its own Validate() method, since the core code uses reflection to do the same. This removes a few lines of code. - Since the OTel collector is in the middle of renaming a number of structs from Settings to Config, I renamed ArrowSettings to ArrowConfig.
… message at stream lifetime (#170) Fixes #160. Fixes a test-flake discovered while preparing open-telemetry/opentelemetry-collector-contrib#31996 For the test flake, rewrite the test. The test was relying on the behavior of a single stream at shutdown, whereas the test wanted to test the seamlessness of stream restart. The new test covers the intended condition better -- it runs a short lifetime stream and runs the test long enough for 5 stream lifetimes, then checks for no observed logs. I carried out manual testing of the shutdown process to help with #160. The problem was that the exporter receives an error from both its writer and its reader goroutines, and it logs both. There is new, consistent handling of the EOF and Canceled error states, which are very similar. EOF is what happens when the server responds to the client's CloseSend(). Canceled is gRPC-internally generated. Both exporter and receiver have similar error-logging logic now. Both consider EOF and Canceled the same condition, and will Debug-log a "shutdown" message when this happens. All other error conditions are logged as errors. This removes the former use of the StatusCode called Canceled, it wasn't necessary after switching the code to use a Status-wrapped Canceled code instead. If receivers are updated ahead of exporters, good. If exporters are upgraded ahead of receivers, they will log spurious errors at stream shutdown (but as #160 shows they were already issuing spurious logs). The code remains defined in the protocol, as it is mapped to the gRPC code space, and remains handled by the code as an error condition. --------- Co-authored-by: Laurent Quérel <laurent.querel@gmail.com>
|
The component will need a |
|
I will re-open this. |
…tor-contrib into jmacd/arrow_exporter
…#190) This addresses the test flake occasionally seen in this repository. There was a channel being set to `nil` inappropriately, and uses of a direct call to channel close() when the more appropriate pattern is to cancel a context. This fixes the test flakes and lint problems from the upstream repository in open-telemetry/opentelemetry-collector-contrib#31996.
…-collector-contrib into jmacd/arrow_exporter
|
@codeboten All generated tests have been re-enabled, I wasn't aware of that mechanism. Nice. |
| change_type: new_component | ||
|
|
||
| # The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver) | ||
| component: OpenTelemetry Protocol with Apache Arrow Exporter |
There was a problem hiding this comment.
| component: OpenTelemetry Protocol with Apache Arrow Exporter | |
| component: exporter/otelarrow |
| cfg component.Config, | ||
| ) (exporter.Traces, error) { | ||
| exp, err := newExporter(cfg, set, createArrowTracesStream) | ||
| oce, err := newExporter(cfg, set, createArrowTracesStream) |
There was a problem hiding this comment.
Not clear why this variable was renamed - what is oce stand for? 🤔
There was a problem hiding this comment.
The OTLP-general stuff up two levels was derived from the core otlpexporter and still has some irregular names as a result. I've tried to fix them as a I notice them. I fixed them in the skeleton for this component, but copied the old names by mistake when adding code from the otel-arrow repo. Will fix!
| } | ||
|
|
||
| func (e *baseExporter) helperOptions() []exporterhelper.Option { | ||
| func (oce *baseExporter) helperOptions() []exporterhelper.Option { |
There was a problem hiding this comment.
Not clear why this variable was renamed - what is oce stand for? 🤔
|
|
||
| // Prioritizer is a policy name for how load is distributed | ||
| // across streams. | ||
| Prioritizer arrow.PrioritizerName `mapstructure:"prioritizer"` |
There was a problem hiding this comment.
This config option is not described in the README. It's not a big issue since the component is in development stability.
| // Exporter exports OpenTelemetry Protocol with Apache Arrow protocol | ||
| // data for a specific signal. One of these structs is created per | ||
| // baseExporter, in the top-level module, when Arrow is enabled. | ||
| // Exporter is 1:1 with exporter, isolates arrow-specific |
There was a problem hiding this comment.
Not sure what this means "Exporter is 1:1 with exporter" 😅
EDIT: Oh I get it now: this internal exporter is otel-arrow specific, while the exporter in exporter/otelarrowexporter/otelarrow.go file supports both regular OTLP and dispatches to this file for otel-arrow.
| @@ -0,0 +1,74 @@ | |||
| // Code generated by MockGen. DO NOT EDIT. | |||
There was a problem hiding this comment.
Not sure about this generated code. Shouldn't this have its go:generate clause? When should this be re-generated? How can the CI confirm the generated code hasn't been manually modified?
There was a problem hiding this comment.
Oh .. yes this maybe deserves a generate clause. I tend to not want to ask everyone to install this tool, since it shouldn't have to change unless the test framework changes or the gRPC interface breaks. (LOL I guess that's not unlikely).
In the upstream repo, I have a makefile rule: https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter/internal/arrow/Makefile
The use of Uber/Google's Gomock framework maybe also stands out w/ the CONTRIBUTING doc recommending testify, but I have found it much more capable of testing asynchronous code.
| } else if sent { | ||
| return nil | ||
| } | ||
| req := ptraceotlp.NewExportRequestFromTraces(td) |
There was a problem hiding this comment.
Please help me understand - shouldn't we use the DisableDowngrade configuration option here? This code seems to always downgrade to OTLP if could not send with arrow. Am I missing something?
There was a problem hiding this comment.
The design of this component calls for it to fall back to standard OTLP when Arrow is unavailable.
This is why we copied from the core OTLP exporter to begin with (and the receiver, likewise) and earlier in the project we decided to sever the history, as it wasn't helping us.
If the DisableDowngrade option is set, the arrowSendAndWait() call above will always return err != nil or sent == true to avoid falling through. When the Arrow code decides to downgrade, it stops trying and quickly returns sent=false && err = nil.
Having implemented all of this, in hindsight, I'm not sure I would go this way again. The downgrade motion can be confusing in its own right, and the code is a lot more complex from having the option to downgrade (and there are some corner cases I'm not happy about). Still, it's designed to make the upgrade painless for users, and I can see how it will help them.
| req := ptraceotlp.NewExportRequestFromTraces(td) | |
| req := ptraceotlp.NewExportRequestFromTraces(td)and |
|
Since this component is in development stability, I'm happy to merge this request and have the above remarks resolved in separate pull requests, if needed. |
jmacd
left a comment
There was a problem hiding this comment.
Will send a PR with responses.
| } else if sent { | ||
| return nil | ||
| } | ||
| req := ptraceotlp.NewExportRequestFromTraces(td) |
There was a problem hiding this comment.
The design of this component calls for it to fall back to standard OTLP when Arrow is unavailable.
This is why we copied from the core OTLP exporter to begin with (and the receiver, likewise) and earlier in the project we decided to sever the history, as it wasn't helping us.
If the DisableDowngrade option is set, the arrowSendAndWait() call above will always return err != nil or sent == true to avoid falling through. When the Arrow code decides to downgrade, it stops trying and quickly returns sent=false && err = nil.
Having implemented all of this, in hindsight, I'm not sure I would go this way again. The downgrade motion can be confusing in its own right, and the code is a lot more complex from having the option to downgrade (and there are some corner cases I'm not happy about). Still, it's designed to make the upgrade painless for users, and I can see how it will help them.
| req := ptraceotlp.NewExportRequestFromTraces(td) | |
| req := ptraceotlp.NewExportRequestFromTraces(td)and |
| cfg component.Config, | ||
| ) (exporter.Traces, error) { | ||
| exp, err := newExporter(cfg, set, createArrowTracesStream) | ||
| oce, err := newExporter(cfg, set, createArrowTracesStream) |
There was a problem hiding this comment.
The OTLP-general stuff up two levels was derived from the core otlpexporter and still has some irregular names as a result. I've tried to fix them as a I notice them. I fixed them in the skeleton for this component, but copied the old names by mistake when adding code from the otel-arrow repo. Will fix!
**Description:** Apply partial fixes from late-arriving review feedback from @andrzej-stencel **Link to tracking Issue:** #31996 **Testing:** n/a **Documentation:** New documentation requested, added.
* Update module google.golang.org/api to v0.178.0 (#32927)
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[google.golang.org/api](https://github.com/googleapis/google-api-go-client)
| `v0.177.0` -> `v0.178.0` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>googleapis/google-api-go-client
(google.golang.org/api)</summary>
###
[`v0.178.0`](https://github.com/googleapis/google-api-go-client/releases/tag/v0.178.0)
[Compare
Source](https://github.com/googleapis/google-api-go-client/compare/v0.177.0...v0.178.0)
##### Features
- **all:** Auto-regenerate discovery clients
([#​2561](https://github.com/googleapis/google-api-go-client/issues/2561))
([2d22d11](https://github.com/googleapis/google-api-go-client/commit/2d22d11df9643a4fad0f9e952d7a92a419370122))
- **all:** Auto-regenerate discovery clients
([#​2564](https://github.com/googleapis/google-api-go-client/issues/2564))
([b313e4b](https://github.com/googleapis/google-api-go-client/commit/b313e4bd70e601fd7a2a931f168fb1dece980e75))
- **all:** Auto-regenerate discovery clients
([#​2565](https://github.com/googleapis/google-api-go-client/issues/2565))
([0843d21](https://github.com/googleapis/google-api-go-client/commit/0843d217048b2e713c0d273b95b33afb99926a8c))
- **all:** Auto-regenerate discovery clients
([#​2567](https://github.com/googleapis/google-api-go-client/issues/2567))
([76b27f1](https://github.com/googleapis/google-api-go-client/commit/76b27f162032649ddb3cb3f06ed24c7333b3fa66))
- **all:** Auto-regenerate discovery clients
([#​2568](https://github.com/googleapis/google-api-go-client/issues/2568))
([d922e3b](https://github.com/googleapis/google-api-go-client/commit/d922e3b559ce5832941390b4f9bf91210e3f6579))
- **all:** Auto-regenerate discovery clients
([#​2570](https://github.com/googleapis/google-api-go-client/issues/2570))
([f2da582](https://github.com/googleapis/google-api-go-client/commit/f2da582c9f6aab240d44c8ebd2dcc43f5096f896))
- **all:** Auto-regenerate discovery clients
([#​2571](https://github.com/googleapis/google-api-go-client/issues/2571))
([0c976dc](https://github.com/googleapis/google-api-go-client/commit/0c976dcc8d1d653f2284ce273493e6714a6d4b2a))
- **gen:** Add internaloption.EnableNewAuthLibrary
([#​2519](https://github.com/googleapis/google-api-go-client/issues/2519))
([8c74bb8](https://github.com/googleapis/google-api-go-client/commit/8c74bb83e2bc27188154c506e63a3e0f3a042f55))
- **google-api-go-client:** Add x-goog-api-version header
([#​2563](https://github.com/googleapis/google-api-go-client/issues/2563))
([fe54ffd](https://github.com/googleapis/google-api-go-client/commit/fe54ffd92359506fca1ffd70dc647db0ab9a903c))
##### Documentation
- Update commit style in CONTRIBUTING
([#​2566](https://github.com/googleapis/google-api-go-client/issues/2566))
([5e44215](https://github.com/googleapis/google-api-go-client/commit/5e44215df618fcafd5f6c1bbe259062cddd32f1a))
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNDAuMTAiLCJ1cGRhdGVkSW5WZXIiOiIzNy4zNDAuMTAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbImRlcGVuZGVuY2llcyIsInJlbm92YXRlYm90Il19-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
* chore: add constants for prometheus translation (#32830)
**Description:**
This is a cleanup to consolidate the constants used for prometheus
translation in a single place.
---------
Co-authored-by: Curtis Robert <crobert@splunk.com>
* sumologicexporter!: change metrics behavior (#32737)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
* remove suppport for carbon2 and graphite
* add support for otlp format
* do not support metadata attributes
* do not support source headers
* set otlp as default format
This PR reduces size of #32315
**Link to tracking Issue:** #31479
**Testing:**
- unit tests
- manual tests
**Documentation:**
- Readme
---------
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Co-authored-by: Adam Boguszewski <aboguszewski@sumologic.com>
* Update module github.com/aws/aws-sdk-go to v1.52.4 (#32928)
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) |
`v1.52.3` -> `v1.52.4` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>aws/aws-sdk-go (github.com/aws/aws-sdk-go)</summary>
###
[`v1.52.4`](https://github.com/aws/aws-sdk-go/blob/HEAD/CHANGELOG.md#Release-v1524-2024-05-07)
[Compare
Source](https://github.com/aws/aws-sdk-go/compare/v1.52.3...v1.52.4)
\===
##### Service Client Updates
- `service/b2bi`: Updates service documentation
- `service/budgets`: Updates service API and documentation
- This release adds tag support for budgets and budget actions.
- `service/resiliencehub`: Updates service API, documentation, and
paginators
- `service/route53profiles`: Updates service API and documentation
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNDAuMTAiLCJ1cGRhdGVkSW5WZXIiOiIzNy4zNDAuMTAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbImRlcGVuZGVuY2llcyIsInJlbm92YXRlYm90Il19-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
* [chore][receiver/sqlserver] Update documentation for darwin and linux OS (#32878)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
With the enhancement in
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/30297,
the SQL Server receiver can now run on MacOS and Linux, since it no
longer solely relies on Windows. The difference in functionality has
(hopefully) been documented clearly between the different platforms, so
the unsupported warning can be removed. I missed this in my previous
PRs.
* Update module github.com/grafana/loki/pkg/push to v0.0.0-20240507085123-772616cd8f5c (#32897)
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [github.com/grafana/loki/pkg/push](https://github.com/grafana/loki)
| `v0.0.0-20240506154431-a772ed705c65` ->
`v0.0.0-20240507085123-772616cd8f5c` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Configuration
📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNDAuMTAiLCJ1cGRhdGVkSW5WZXIiOiIzNy4zNDAuMTAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbImRlcGVuZGVuY2llcyIsInJlbm92YXRlYm90Il19-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
* chore(deps): update dependency go to v1.22.3 (#32621)
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [go](https://go.dev/) ([source](https://github.com/golang/go)) |
toolchain | minor | `1.21.9` -> `1.22.3` |
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>golang/go (go)</summary>
###
[`v1.22.3`](https://github.com/golang/go/compare/go1.22.2...go1.22.3)
###
[`v1.22.2`](https://github.com/golang/go/compare/go1.22.1...go1.22.2)
###
[`v1.22.1`](https://github.com/golang/go/compare/go1.22.0...go1.22.1)
###
[`v1.22.0`](https://github.com/golang/go/compare/go1.21.7...go1.22rc1)
###
[`v1.21.10`](https://github.com/golang/go/compare/go1.21.9...go1.21.10)
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zMTMuMSIsInVwZGF0ZWRJblZlciI6IjM3LjM0MC4xMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiZGVwZW5kZW5jaWVzIiwicmVub3ZhdGVib3QiXX0=-->
---------
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
Co-authored-by: Juraci Paixão Kröhling <juraci@kroehling.de>
* [receiver/haproxy] Unit test timing out (#32940)
I wasn't able to reproduce the exact situation from CI, but the tests I
changed weren't completing at all here. Looking closely, it looks like
they never closed the stream, something the first test does. After
closing it, the tests start passing locally for me. While I'm not
confident this will fix the CI flaky failures, this does make the test
work locally for me.
Fixes #32877
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com>
* [connector/servicegraph] Remove use of host.GetExporters (#32902)
Fixes #31628
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
---------
Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>
Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com>
* [exporter/loadbalancing] Improve the performance when merging traces belonging to the same backend (#32032)
**Description:** no need to reimplement that in an extremely
allocation-inefficient fashion.
I'm actually not sure why mergeTraces() and mergeMetrics() need to exist
in the first place; all the other exporters coupled with the batch
processor work just fine, not sure why loadbalancing would be special.
https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/30141
seems to imply they were implemented to improve performance, but I don't
really understand why batch processor would not have been sufficient for
that improvement originally.
benchmarks before:
```
goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter
BenchmarkMergeTraces_X100-8 50214 23507 ns/op
BenchmarkMergeTraces_X500-8 10000 113952 ns/op
BenchmarkMergeTraces_X1000-8 5208 226062 ns/op
BenchmarkMergeMetrics_X100-8 64933 18540 ns/op
BenchmarkMergeMetrics_X500-8 12885 91418 ns/op
BenchmarkMergeMetrics_X1000-8 6590 184584 ns/op
PASS
ok github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter 9.783s
```
and after:
```
goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter
BenchmarkMergeTraces_X100-8 295886529 3.836 ns/op
BenchmarkMergeTraces_X500-8 309865370 3.833 ns/op
BenchmarkMergeTraces_X1000-8 310739948 3.800 ns/op
BenchmarkMergeMetrics_X100-8 315567813 3.841 ns/op
BenchmarkMergeMetrics_X500-8 310341650 3.849 ns/op
BenchmarkMergeMetrics_X1000-8 314292003 3.830 ns/op
PASS
ok github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter 10.733s
```
**Link to tracking Issue:** n/a
**Testing:** unit tests pass & cpu time for our collectors using
loadbalancingexporter (12 replicas, total of 25k-40k spans/sec) went
from 800ms-1400ms/sec down to <40msec/sec.
**Documentation:** none
---------
Co-authored-by: Juraci Paixão Kröhling <juraci.github@kroehling.de>
* Move Aneurysm9 to emeritus status (#32943)
I have been unable to provide this position the bandwidth that it
deserves and it is time to formalize recognition of that fact.
Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>
* [receiver/googlecloudpubsubreceiver] Fix memory leak during shutdown (#32361)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
This PR contains the following changes:
1. Add `Close` call to the receiver's GRPC client. Without this,
goroutines were being leaked on shutdown.
2. Change `grpc.Dial` -> `grpc.NewClient`. They offer the same
functionality, but `Dial` is being deprecated in favor of `NewClient`.
3. Enable `goleak` checks on this receiver to help ensure no goroutines
are being leaked.
4. Change a couple `Assert.Nil` calls to `Assert.NoError`. The output of
`NoError` includes the error message if hit, `Nil` simply includes the
object's address, i.e. `&status.Error{s:(*status.Status)(0xc00007e158)}`
**Link to tracking Issue:** <Issue number if applicable>
#30438
**Testing:** <Describe what testing was performed and which tests were
added.>
All existing tests are passing, as well as added goleak check.
* [chore] Add some docs to readme regarding file exporter (#32855)
Admittedly I tested this via docker-compose. I can expand to having a
full docker-compose file with telemetry gen if that's preferred.
---------
Co-authored-by: Alex Boten <223565+codeboten@users.noreply.github.com>
* [receiver/vcenter] Fixes Cluster Resource Attributes for Datastore Resource (#32687)
**Description:** <Describe what has changed.>
Removed the `vcenter.cluster.name` resource attribute from all Datastore
resources.
**Link to tracking Issue:** <Issue number if applicable>
#32674
**Testing:** <Describe what testing was performed and which tests were
added.>
Unit/integration tests updated and tested. Local environment tested.
**Documentation:** <Describe the documentation added.>
New documentation generated based on the metadata.
---------
Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
Co-authored-by: Curtis Robert <crobert@splunk.com>
* [receiver/vcenter] Adds New Packet Dropped Rate Metric for VMs (#32930)
**Description:** <Describe what has changed.>
Adds new default disabled (with Warning log for default enabled on next
release) metric `vcenter.vm.network.packet.drop.rate` for Virtual
Machines.
This metric makes use of the `droppedRx` and `droppedTx` Network
performance metrics detailed
[here](https://vdc-repo.vmware.com/vmwb-repository/dcr-public/d1902b0e-d479-46bf-8ac9-cee0e31e8ec0/07ce8dbd-db48-4261-9b8f-c6d3ad8ba472/network_counters.html)
for Virtual machines. This would use the same metric attributes as the
other VM packet metrics and closely match
`vcenter.vm.network.packet.rate` in every other way.
**Link to tracking Issue:** <Issue number if applicable>
#32929
**Testing:** <Describe what testing was performed and which tests were
added.>
Unit/integration tests updated and tested. Local environment tested.
**Documentation:** <Describe the documentation added.>
New documentation generated based on the metadata.
* fix(deps): update module github.com/open-telemetry/otel-arrow to v0.22.0 (#32105)
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[github.com/open-telemetry/otel-arrow](https://github.com/open-telemetry/otel-arrow)
| `v0.18.0` -> `v0.22.0` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>open-telemetry/otel-arrow
(github.com/open-telemetry/otel-arrow)</summary>
###
[`v0.22.0`](https://github.com/open-telemetry/otel-arrow/releases/tag/v0.22.0)
[Compare
Source](https://github.com/open-telemetry/otel-arrow/compare/v0.21.0...v0.22.0)
Includes
[#​178](https://github.com/open-telemetry/otel-arrow/issues/178).
[CHANGELOG](https://github.com/open-telemetry/otel-arrow/blob/main/CHANGELOG.md)
###
[`v0.21.0`](https://github.com/open-telemetry/otel-arrow/releases/tag/v0.21.0)
[Compare
Source](https://github.com/open-telemetry/otel-arrow/compare/v0.20.0...v0.21.0)
See the
[CHANGELOG](https://github.com/open-telemetry/otel-arrow/blob/main/CHANGELOG.md).
###
[`v0.20.0`](https://github.com/open-telemetry/otel-arrow/releases/tag/v0.20.0)
[Compare
Source](https://github.com/open-telemetry/otel-arrow/compare/v0.19.0...v0.20.0)
##### What's Changed
- Backport lint fixes from OTel-Collector-Contrib PR 31996 by
[@​jmacd](https://github.com/jmacd) in
[https://github.com/open-telemetry/otel-arrow/pull/163](https://github.com/open-telemetry/otel-arrow/pull/163)
- Upgrade collector to v0.97.0 by
[@​moh-osman3](https://github.com/moh-osman3) in
[https://github.com/open-telemetry/otel-arrow/pull/164](https://github.com/open-telemetry/otel-arrow/pull/164)
**Full Changelog**:
https://github.com/open-telemetry/otel-arrow/compare/v0.19.0...v0.20.0
###
[`v0.19.0`](https://github.com/open-telemetry/otel-arrow/releases/tag/v0.19.0)
[Compare
Source](https://github.com/open-telemetry/otel-arrow/compare/v0.18.0...v0.19.0)
See
[CHANGELOG.md](https://github.com/open-telemetry/otel-arrow/blob/main/CHANGELOG.md#0190---2024-03-26)
for release notes.
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yNjkuMiIsInVwZGF0ZWRJblZlciI6IjM3LjM0MC4xMCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
* [remotetapprocessor] use 'time/rate' to limit traffic (#32481)
bug: The remotetapprocessor `limit` configure doesn't work.
how to fix: use `time/rate` to limit traffic.
Resolves
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/32385
---------
Co-authored-by: Andrzej Stencel <astencel@sumologic.com>
* [receiver/vcenter] Switches Over Metadata Configs Waiting for v0.100.0 Release (#32913)
**Description:** <Describe what has changed.>
A number of configurations were disabled by default and had warnings
that they were going to be enabled in v0.101.0 (1 metric had a warning
that it was going to be removed).
Now that v0.100.0 has been release, I have removed all of these
warnings, and made the modifications that the warnings "warned" about. I
have also updated the tests to reflect this.
**Link to tracking Issue:** <Issue number if applicable>
#32803 #32805 #32821 #32531 #32557
**Testing:** <Describe what testing was performed and which tests were
added.>
Unit/integration tests updated and tested. Local environment tested.
**Documentation:** <Describe the documentation added.>
New documentation generated based on the metadata.
* [chore][fileconsumer] Skip flaky TestFlushPeriodEOF on windows (#32946)
See
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/32715
This also adds a bit more debugging info for other tests which fail on
the same expectation, since it's not very obvious what was expected vs
actually found.
* [chore][CI/CD][arm] Trigger arm runs on label (#32955)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
I found in
https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/32948
that the label `Run ARM` has been added, but the `build-and-test-arm /
arm-unittest-matrix (pull_request) ` workflow is still skipped. This is
because the `label` action does not trigger a retry.
From
[documentation](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request):
```
if no activity types are specified, the workflow runs when a pull request is opened or reopened or when the head branch of the pull request is updated.
```
We need to specify that labelling issues should trigger the workflow to
check to see if it needs to run again. I've copied the added section
from the Windows workflow. I also added that we should only run on PRs
against `main`.
**Testing:**
This PR shows it's working as it should now. Arm test was [originally
skipped](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/9009218559/job/24753003216?pr=32955),
but after adding the label, tests [have
started](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/9009223570/job/24753017935?pr=32955)
* [chore][receiver/splunkenterprise] Add header to README (#32956)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
The readme for the Splunk Enterprise receiver does not currently have
the autogenerated header. This was missing because `mdatagen` requires
`<!-- status autogenerated section -->` and `<!-- end autogenerated
section -->` to know where to insert the generated data.
* [exporter/elasticsearch] Replace go-elasticsearch BulkIndexer with go-docappender (#32359)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
Replace go-elasticsearch BulkIndexer with go-docappender BulkIndexer for
Flush function in preparation for reliability fixes. Maintain similar
interface and implementation to go-elasticsearch BulkIndexer.
Further changes to expose individual `docappender.BulkIndexer` instances
are needed down the road but it is out of the scope of this PR.
Implications of this change:
- flush timeout is now enforced on client side
- oversize payload special handling is now removed
- go-docappender uses bulk request filterPath which means bulk response
is smaller, less JSON parsing and lower CPU usage
- document level retry debug logging is removed as retries are done
transparently
~~Blocked by #32585~~
**Link to tracking Issue:** <Issue number if applicable> Fixes #32378
**Testing:** Integration test is passing
---------
Co-authored-by: Vishal Raj <vishal.raj@elastic.co>
* fix(test): Skip flaky test around forcing collector re-registration until the root cause is confirmed (#32937)
**Description:** Remove flaky test around forcing collector
re-registration until the root cause is confirmed
**Link to tracking Issue:**
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/32785
**Testing:** Unit tests
* [chore][receiver/splunkenterprise] Splunkent wire component (#32795)
**Description:** Graduate splunkenterprise receiver component to alpha
**Link to tracking Issue:**
**Testing:** Performed `make otelcontribcol` and ran resulting binary
with the following config:
```yaml
extensions:
basicauth/indexer:
client_auth:
username: admin
password: securityFirst
basicauth/cluster_master:
client_auth:
username: admin
password: securityFirst
receivers:
splunkenterprise:
indexer:
auth:
authenticator: basicauth/indexer
endpoint: "https://localhost:8089/"
timeout: 45s
cluster_master:
auth:
authenticator: basicauth/cluster_master
endpoint: "https://localhost:8089/"
timeout: 45s
exporters:
otlp:
endpoint: 127.0.0.1:8000
service:
extensions: [basicauth/indexer, basicauth/cluster_master]
pipelines:
metrics:
receivers: [splunkenterprise]
exporters: [otlp]
```
and received the following output:
```
sh ~> ./otelcontribcol_linux_amd64 --config=file:config.yaml
2024-05-08T17:34:33.032-0500 info service@v0.100.0/service.go:102 Setting up own telemetry...
2024-05-08T17:34:33.032-0500 info service@v0.100.0/telemetry.go:103 Serving metrics {"address": ":8888", "level": "Normal"}
2024-05-08T17:34:33.032-0500 info receiver@v0.100.0/receiver.go:310 Development component. May change in the future. {"kind": "receiver", "name": "splunkenterprise", "data_type": "metrics"}
2024-05-08T17:34:33.033-0500 info service@v0.100.0/service.go:169 Starting otelcontribcol... {"Version": "0.100.0-dev", "NumCPU": 16}
2024-05-08T17:34:33.033-0500 info extensions/extensions.go:34 Starting extensions...
2024-05-08T17:34:33.033-0500 info extensions/extensions.go:37 Extension is starting... {"kind": "extension", "name": "basicauth/cluster_master"}
2024-05-08T17:34:33.033-0500 info extensions/extensions.go:52 Extension started. {"kind": "extension", "name": "basicauth/cluster_master"}
2024-05-08T17:34:33.033-0500 info extensions/extensions.go:37 Extension is starting... {"kind": "extension", "name": "basicauth/indexer"}
2024-05-08T17:34:33.033-0500 info extensions/extensions.go:52 Extension started. {"kind": "extension", "name": "basicauth/indexer"}
2024-05-08T17:34:33.033-0500 info service@v0.100.0/service.go:195 Everything is ready. Begin running and processing data.
```
indicating that the collector was able to successfully start with the
component configured.
**Documentation:** Documentation was updated to indicate change in
status from development to alpha
---------
Co-authored-by: Curtis Robert <crobert@splunk.com>
* [pkg/ottl] Added support for timezone in Time converter (#32479)
**Description:**
Added support for default timezone in Time converter.
Timezone is optional and can be specified as so: `Time("2023-05-26
12:34:56", "%Y-%m-%d %H:%M:%S", "America/New_York")`
**Link to tracking Issue:** #32140
**Testing:** Unit tests added
**Documentation:** Documentation in ottl/Readme updated
---------
Co-authored-by: Tyler Helmuth <12352919+TylerHelmuth@users.noreply.github.com>
Co-authored-by: Evan Bradley <11745660+evan-bradley@users.noreply.github.com>
* [processor/transform] Add common where clause (#31491)
**Description:** Add global conditions with where clause
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
**Link to tracking Issue:** Fixes #27830
**Testing:** Unit tests
**Documentation:** TODO
~~The main objective is to extend the `ContextStatements` struct by
adding a new `Conditions` parameter. By introducing `Conditions` to
`ContextStatements`, we can now apply a global condition to all related
statements in `WithStatementSequenceGlobalConditions` function.~~
Thanks in advance for your feedback! If this changes will be fine, I
will add common where clause into another context `span`, `metrics`.
* Instantiate ID in pkg/stanza/adapter tests (#32966)
**Description:** <Describe what has changed.>
In https://github.com/open-telemetry/opentelemetry-collector/pull/10069,
I am making Type an interface. This means the zero value of Type will be
nil - which will cause this test to fail. Initializing ID instead of
relying on the zero value fixes this
---------
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
Co-authored-by: Bogdan Drutu <bogdandrutu@gmail.com>
* Add connector usage to the testbed (#32881)
**Description:** Added a new component to the testbed called
DataConnectors which allows connectors to be added to the testbed
config. Also, added a sample connector correctness test using the
routingconnector as an example of the usage.
**Link to tracking Issue:** #30165
**Testing:** Sample correctness test using routingconnector.
**Documentation:** Will update the testbed README with new addition.
---------
Co-authored-by: Bryan Aguilar <bryaag@amazon.com>
* [chore][CONTRIBUTING.md] Update adding component directions (#32957)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
This is two main changes:
1. Remove `goleak` section. It's now added by default by mdatagen,
there's nothing required of users here.
2. Add information and reformat the `Last steps` section of adding a new
component.
- Move the directions to be bullet points
- Add `make generate` to make sure the component's README is updated
properly
- Explicitly point out that stability and distribution needs to be
updated in `metadata.yaml`
- Add step for adding the component to the releases repo. My
understanding is that if a component is `Alpha`, it should be included
in the release, so I've made that a noted part of the steps here. (Let
me know if we should word this in a more "optional" way.)
---------
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
* [opampextension]: Move custom message interface to separate module (#32951)
**Description:** <Describe what has changed.>
* Breaks our the custom message interface to a separate module, so other
components can use the interface without needing to import the
`opampextension` module in its entirety.
We could temporarily alias the old methods if we'd like, but I think
that the CustomMessage stuff has been so short lived that, in addition
to the alpha status of the opampextension component, it feels justified
to just skip the deprecation process and move it to a new module.
**Link to tracking Issue:** Closes #32950
**Testing:**
* Covered by existing unit tests
**Documentation:**
* Added more documentation on usage in the new module.
* Modified opampextension docs to point to the new module.
* [chore] Add contrib confmap providers to otelcontribcol (#32916)
**Description:**
Contrib hosts two confmap providers which aren't in the local binary. We
should add them so we can test them with a live system.
cc @Aneurysm9 @driverpt @atoulme
---------
Co-authored-by: Evan Bradley <evan-bradley@users.noreply.github.com>
* chore: remote write exporter retry on 429 (#31924)
**Description:** <Describe what has changed.>
This PR adds an option to retry the remote write requests when the
receiving backend responds with 429 http status code,
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
**Link to tracking Issue:** #31032
**Testing:** <Describe what testing was performed and which tests were
added.>
Added tests covering the case.
**Documentation:** <Describe the documentation added.>
Not sure what's the pattern for documenting feature flags.
---------
Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>
Co-authored-by: Pablo Baeyens <pbaeyens31+github@gmail.com>
Co-authored-by: David Ashpole <dashpole@google.com>
* [chore] vcenterreceiver Adds Accidentally Removed Unit Test Configs/Results (#32987)
**Description:** <Describe what has changed.>
In this
[PR](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/32913),
I accidentally removed all the enabled configs from the 2nd set of unit
tests. This was incorrect as there are still 4 of them that are
currently disabled by default. This is rectifying that by adding those
back in.
**Link to tracking Issue:** <Issue number if applicable>
**Testing:** <Describe what testing was performed and which tests were
added.>
This change is for the unit tests only
**Documentation:** <Describe the documentation added.>
NA
* OpenTelemetry Protocol with Apache Arrow Exporter component (#31996)
**Description:**
This is the same code as OTel-Arrow at
https://github.com/open-telemetry/otel-arrow/releases/tag/v0.23.0 (plus
[backported lint and test
fixes](https://github.com/open-telemetry/otel-arrow/commit/0910113d46454c80881db840e21f25485dce2499)).
Only import statements change here, to match the host repository.
**Link to tracking Issue:** #26491
**Testing:** Test coverage is approximately 90%.
**Documentation:** I double-checked and the existing README had only a
few updates needed.
* [exporter/sumologic] change logs behavior (#32939)
**Description:**
* set OTLP as default format
* add support for OTLP format
* do not support metadata attributes
* do not support source headers
**Link to tracking Issue:** #32315
**Testing:**
* unit tests
**Documentation:**
* inline comments
* readme
---------
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
* Fix diff to upstream
* [chore][exporter/rabbitmq] Add missing code owner (#32984)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
@atoulme [volunteered to be the
sponsor](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/28891#issuecomment-1981402878)
of this component, so I believe he should be listed as a code owner.
From
[CONTRIBUTING.md](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-new-components):
```
A sponsor is an approver who will be in charge of being the official reviewer of the code and become a code owner for the component.
```
**Link to tracking Issue:** <Issue number if applicable>
#28891
* [chore][receiver/sqlserver] Document Windows-only metrics (#32944)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
The referenced metrics are being gathered from Windows perf counters,
thus they're currently only available on Windows. With the introduction
of direct connection and other supported OSs, I think it would be good
to be clear to users about metric availability.
* [receiver/splunkehecreceiver] Align acking behavior with that of Splu… (#32996)
**Description:**
- Make the channelID header case-insensitive
- Make hecreceiver endpoints able to extract channelID from query params
**Link to tracking Issue:** https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/32995
* [receiver/vcenter] Collection Time Performance Enhancement (#32991)
**Description:** <Describe what has changed.>
There were already some improvements made as far as how networks calls
were made centered around Virtual Machines. This allowed collection
times to decrease from ~90s to ~27s in an environment with 1 Cluster, 2
Hosts, & 280 VMs.
Making similar changes for all resource types helped to further decrease
collection times. Now collection time has decreased from ~27s to <~3s
for the same environment.
Here's a general list of the changes made:
- Now makes all network calls (per datacenter) first and stores returned
data.
- Processes this data afterwards to convert to OTEL resources/metrics
(refactored to new file).
- Moves all metric recording to metrics.go to keep consistent.
- Moves all resource builder creation to resources.go to keep
consistent.
- Updates/fixes tests.
**Link to tracking Issue:** <Issue number if applicable>
#31837 Although this issue prescribes a solution to the problem
(goroutines) which ended up not being necessary
**Testing:** <Describe what testing was performed and which tests were
added.>
Unit Tests & Integration Tests Passing as well as Manual Testing in
Local Environments
**Documentation:** <Describe the documentation added.>
N/A
* [exporter/sumologic]: add sticky session (#33011)
**Description:**
Adds support for sticky session in order to better support AWS LB. This
code is moved from Sumo Logic repository
**Link to tracking Issue:**
#32315
**Testing:**
Tested manually
**Documentation:**
N/A
---------
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
* [receiver/sqlserver] Add more metrics (#32932)
**Description:**
This change adds a query, scraper, and some more metrics for data from
the performance counter SQL server table. The query itself is mostly
taken from Telegraf's SQL server plugin. This also reuses the existing
metric `sqlserver.lock.wait.rate`, so now it will be available both from
Windows performance counters, as well as other OSs when directly
connecting to the SQL server instance.
**Naming and format feedback on new metrics would be greatly
appreciated.**
**Link to tracking Issue:** #29865
**Testing:**
Added tests for the query itself, as well as the new scraper.
Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
* deltatocumulative: exponential histograms (#32030)
**Description:** Implements accumulation of exponential histograms by
adding bucket-per-bucket.
- [x] Align bucket offset to the smaller one
- [x] Merge buckets by adding up each buckets count
- [x] Widen zero buckets so they are the same
- [x] Adjust scale to the lowest one
**Link to tracking Issue:**
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/30705
**Testing:** Extensive tests have been added to the `internal/data`
package
**Documentation:** not needed
* [pkg/stanza] Add container operator parser (#32594)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
This PR implements the new container logs parser as it was proposed at
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/31959.
**Link to tracking Issue:** <Issue number if applicable>
https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/31959
**Testing:** <Describe what testing was performed and which tests were
added.>
Added unit tests. Providing manual testing steps as well:
### How to test this manually
1. Using the following config file:
```yaml
receivers:
filelog:
start_at: end
include_file_name: false
include_file_path: true
include:
- /var/log/pods/*/*/*.log
operators:
- id: container-parser
type: container
output: m1
- type: move
id: m1
from: attributes.k8s.pod.name
to: attributes.val
- id: some
type: add
field: attributes.key2.key_in
value: val2
exporters:
debug:
verbosity: detailed
service:
pipelines:
logs:
receivers: [filelog]
exporters: [debug]
processors: []
```
2. Start the collector:
`./bin/otelcontribcol_linux_amd64 --config
~/otelcol/container_parser/config.yaml`
3. Use the following bash script to create some logs:
```bash
#! /bin/bash
echo '2024-04-13T07:59:37.505201169-05:00 stdout P This is a very very long crio line th' >> /var/log/pods/kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler43/1.log
echo '{"log":"INFO: log line here","stream":"stdout","time":"2029-03-30T08:31:20.545192187Z"}' >> /var/log/pods/kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log
echo '2024-04-13T07:59:37.505201169-05:00 stdout F at is awesome! crio is awesome!' >> /var/log/pods/kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler43/1.log
echo '2021-06-22T10:27:25.813799277Z stdout P some containerd log th' >> /var/log/pods/kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log
echo '{"log":"INFO: another log line here","stream":"stdout","time":"2029-03-30T08:31:20.545192187Z"}' >> /var/log/pods/kube-controller-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d6/kube-controller/1.log
echo '2021-06-22T10:27:25.813799277Z stdout F at is super awesome! Containerd is awesome' >> /var/log/pods/kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log
echo '2024-04-13T07:59:37.505201169-05:00 stdout F standalone crio line which is awesome!' >> /var/log/pods/kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler43/1.log
echo '2021-06-22T10:27:25.813799277Z stdout F standalone containerd line that is super awesome!' >> /var/log/pods/kube-scheduler-kind-control-plane_49cc7c1fd3702c40b2686ea7486091d3/kube-scheduler44/1.log
```
4. Run the above as a bash script to verify any parallel processing.
Verify that the output is correct.
### Test manually on k8s
1. `make docker-otelcontribcol && docker tag otelcontribcol
otelcontribcol-dev:0.0.1 && kind load docker-image
otelcontribcol-dev:0.0.1`
2. Install using the following helm values file:
```yaml
mode: daemonset
presets:
logsCollection:
enabled: true
image:
repository: otelcontribcol-dev
tag: "0.0.1"
pullPolicy: IfNotPresent
command:
name: otelcontribcol
config:
exporters:
debug:
verbosity: detailed
receivers:
filelog:
start_at: end
include_file_name: false
include_file_path: true
exclude:
- /var/log/pods/default_daemonset-opentelemetry-collector*_*/opentelemetry-collector/*.log
include:
- /var/log/pods/*/*/*.log
operators:
- id: container-parser
type: container
output: some
- id: some
type: add
field: attributes.key2.key_in
value: val2
service:
pipelines:
logs:
receivers: [filelog]
processors: [batch]
exporters: [debug]
```
3. Check collector's output to verify the logs are parsed properly:
```console
2024-05-10T07:52:02.307Z info LogsExporter {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 2}
2024-05-10T07:52:02.307Z info ResourceLog #0
Resource SchemaURL:
ScopeLogs #0
ScopeLogs SchemaURL:
InstrumentationScope
LogRecord #0
ObservedTimestamp: 2024-05-10 07:52:02.046236071 +0000 UTC
Timestamp: 2024-05-10 07:52:01.92533954 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str(otel logs at 07:52:01)
Attributes:
-> log: Map({"iostream":"stdout"})
-> time: Str(2024-05-10T07:52:01.92533954Z)
-> k8s: Map({"container":{"name":"busybox","restart_count":"0"},"namespace":{"name":"default"},"pod":{"name":"daemonset-logs-6f6mn","uid":"1069e46b-03b2-4532-a71f-aaec06c0197b"}})
-> logtag: Str(F)
-> key2: Map({"key_in":"val2"})
-> log.file.path: Str(/var/log/pods/default_daemonset-logs-6f6mn_1069e46b-03b2-4532-a71f-aaec06c0197b/busybox/0.log)
Trace ID:
Span ID:
Flags: 0
LogRecord #1
ObservedTimestamp: 2024-05-10 07:52:02.046411602 +0000 UTC
Timestamp: 2024-05-10 07:52:02.027386192 +0000 UTC
SeverityText:
SeverityNumber: Unspecified(0)
Body: Str(otel logs at 07:52:02)
Attributes:
-> log.file.path: Str(/var/log/pods/default_daemonset-logs-6f6mn_1069e46b-03b2-4532-a71f-aaec06c0197b/busybox/0.log)
-> time: Str(2024-05-10T07:52:02.027386192Z)
-> log: Map({"iostream":"stdout"})
-> logtag: Str(F)
-> k8s: Map({"container":{"name":"busybox","restart_count":"0"},"namespace":{"name":"default"},"pod":{"name":"daemonset-logs-6f6mn","uid":"1069e46b-03b2-4532-a71f-aaec06c0197b"}})
-> key2: Map({"key_in":"val2"})
Trace ID:
Span ID:
Flags: 0
...
```
**Documentation:** <Describe the documentation added.> Added
Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
* fix(deps): update module google.golang.org/genproto/googleapis/api to v0.0.0-20240513163218-0867130af1f8 (#33039)
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[google.golang.org/genproto/googleapis/api](https://github.com/googleapis/go-genproto)
| `v0.0.0-20240506185236-b8a5c65736ae` ->
`v0.0.0-20240513163218-0867130af1f8` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Configuration
📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNTEuMiIsInVwZGF0ZWRJblZlciI6IjM3LjM1MS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZWJvdCJdfQ==-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
* [chore][receiver/sqlserver] Update documentation for lock wait rate metric (#33023)
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
As a result of
https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/32932
the metric `sqlserver.lock.wait.rate` is now available on Windows, or
when directly connecting to a SQL server instance. Since this metric is
always available, it no longer needs extended documentation about
availability.
* [exporter/datadog] Actually gzip host metadata (#32992)
**Description:** We were sending a payload and setting the
`Content-Encoding: gzip` HTTP header while actually sending uncompressed
data. This has been happening for at least a couple years.
This fixes the bug by actually gzipping the payload
* [chore][pkg/stanza] Add syslog octet counting test cases with whitespace scenarios (#32832)
Resolves #31477
---------
Co-authored-by: Tiffany Hrabusa <30397949+tiffany76@users.noreply.github.com>
Co-authored-by: Curtis Robert <crobert@splunk.com>
* fix(deps): update module github.com/grafana/loki/pkg/push to v0.0.0-20240514073905-7cc9a9386a8f (#33031)
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [github.com/grafana/loki/pkg/push](https://github.com/grafana/loki)
| `v0.0.0-20240507085123-772616cd8f5c` ->
`v0.0.0-20240514073905-7cc9a9386a8f` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Configuration
📅 **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any
time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNTEuMiIsInVwZGF0ZWRJblZlciI6IjM3LjM1MS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiLCJyZW5vdmF0ZWJvdCJdfQ==-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
* fix(deps): update module github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common to v1.0.919 (#33036)
[](https://renovatebot.com)
This PR contains the following updates:
| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
|
[github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common](https://github.com/tencentcloud/tencentcloud-sdk-go)
| `v1.0.914` -> `v1.0.919` |
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
[](https://docs.renovatebot.com/merge-confidence/)
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>tencentcloud/tencentcloud-sdk-go
(github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/common)</summary>
###
[`v1.0.919`](https://github.com/tencentcloud/tencentcloud-sdk-go/blob/HEAD/CHANGELOG.md#Release-v10919)
[Compare
Source](https://github.com/tencentcloud/tencentcloud-sdk-go/compare/v1.0.918...v1.0.919)
#### 负载均衡(clb) 版本:2018-03-17
##### 第 105 次发布
发布时间:2024-05-14 01:28:52
本次发布包含了以下内容:
改善已有的文档。
修改数据结构:
-
[TargetGroupAssociation](https://cloud.tencent.com/document/api/214/30694#TargetGroupAssociation)
- 新增成员:Weight
#### TDSQL-C MySQL 版(cynosdb) 版本:2019-01-07
##### 第 89 次发布
发布时间:2024-05-14 01:39:01
本次发布包含了以下内容:
改善已有的文档。
修改接口:
-
[CreateCLSDelivery](https://cloud.tencent.com/document/api/1003/106079)
- 新增入参:InstanceId, CLSInfoList
- 新增出参:TaskId
-
[DeleteCLSDelivery](https://cloud.tencent.com/document/api/1003/106078)
- 新增入参:InstanceId, CLSTopicIds
- 新增出参:TaskId
-
[DescribeInstanceCLSLogDelivery](https://cloud.tencent.com/document/api/1003/106077)
- 新增入参:InstanceId
- 新增出参:TotalCount, InstanceCLSDeliveryInfos
- [StartCLSDelivery](https://cloud.tencent.com/document/api/1003/106076)
- 新增入参:InstanceId, CLSTopicIds
- 新增出参:TaskId
- [StopCLSDelivery](https://cloud.tencent.com/document/api/1003/106075)
- 新增入参:InstanceId, CLSTopicIds
- 新增出参:TaskId
新增数据结构:
- [CLSInfo](https://cloud.tencent.com/document/api/1003/48097#CLSInfo)
-
[InstanceCLSDeliveryInfo](https://cloud.tencent.com/document/api/1003/48097#InstanceCLSDeliveryInfo)
#### 数据安全治理中心(dsgc) 版本:2019-07-23
##### 第 11 次发布
发布时间:2024-05-14 01:46:03
本次发布包含了以下内容:
改善已有的文档。
修改数据结构:
- [AKSKLeak](https://cloud.tencent.com/document/api/1087/96844#AKSKLeak)
- <font color="#dd0000">**修改成员**:</font>AK, SK, URL
-
[AssetCosDetail](https://cloud.tencent.com/document/api/1087/96844#AssetCosDetail)
- <font color="#dd0000">**修改成员**:</font>Bucket, DataType, FileNums,
SensitiveFileNums, DistributionData, MatchedNum
-
[DSPACosMetaDataInfo](https://cloud.tencent.com/document/api/1087/96844#DSPACosMetaDataInfo)
- <font color="#dd0000">**修改成员**:</font>BindStatus, Storage
-
[DbRelationStatusItem](https://cloud.tencent.com/document/api/1087/96844#DbRelationStatusItem)
- <font color="#dd0000">**修改成员**:</font>DbName, BindStatus, ValidStatus
-
[DbTaskResult](https://cloud.tencent.com/document/api/1087/96844#DbTaskResult)
- <font color="#dd0000">**修改成员**:</font>Result, ResultDescription,
ErrDescription, ResourceId, DbName
-
[ErrDescription](https://cloud.tencent.com/document/api/1087/96844#ErrDescription)
- <font color="#dd0000">**修改成员**:</font>ErrCode, ErrMessage
-
[SuggestRiskLevelMatrixItem](https://cloud.tencent.com/document/api/1087/96844#SuggestRiskLevelMatrixItem)
- <font color="#dd0000">**修改成员**:</font>SensitiveLevel,
VulnerabilityLevel, RiskName, RiskScore
#### 弹性 MapReduce(emr) 版本:2019-01-03
##### 第 63 次发布
发布时间:2024-05-14 01:50:50
本次发布包含了以下内容:
改善已有的文档。
修改数据结构:
-
[ImpalaQuery](https://cloud.tencent.com/document/api/589/33981#ImpalaQuery)
- 新增成员:BackendsCount, FragmentInstancesCount, RemainingFragmentCount
#### Elasticsearch Service(es) 版本:2018-04-16
##### 第 58 次发布
发布时间:2024-05-14 01:51:30
本次发布包含了以下内容:
改善已有的文档。
修改数据结构:
-
[ServerlessSpace](https://cloud.tencent.com/document/api/845/30634#ServerlessSpace)
- 新增成员:KibanaLanguage
#### 文字识别(ocr) 版本:2018-11-19
##### 第 135 次发布
发布时间:2024-05-14 02:17:52
本次发布包含了以下内容:
改善已有的文档。
修改接口:
- [HKIDCardOCR](https://cloud.tencent.com/document/api/866/46919)
- <font color="#dd0000">**修改入参**:</font>DetectFake
- 新增出参:WarnCardInfos
- [MLIDCardOCR](https://cloud.tencent.com/document/api/866/37656)
- 新增出参:WarnCardInfos
-
[RecognizeIndonesiaIDCardOCR](https://cloud.tencent.com/document/api/866/75195)
- 新增出参:WarnCardInfos
-
[RecognizeThaiIDCardOCR](https://cloud.tencent.com/document/api/866/48475)
- 新增出参:WarnCardInfos
#### TI-ONE 训练平台(tione) 版本:2021-11-11
##### 第 59 次发布
发布时间:2024-05-14 02:40:26
本次发布包含了以下内容:
改善已有的文档。
新增数据结构:
-
[CBSConfig](https://cloud.tencent.com/document/api/851/75051#CBSConfig)
修改数据结构:
-
[DataConfig](https://cloud.tencent.com/document/api/851/75051#DataConfig)
- 新增成员:CBSSource
#### TI-ONE 训练平台(tione) 版本:2019-10-22
##### 第 13 次发布
发布时间:2024-05-14 02:40:07
本次发布包含了以下内容:
改善已有的文档。
<font color="#dd0000">**删除接口**:</font>
- UpdateNotebookLifecycleScript
#### 实时音视频(trtc) 版本:2019-07-22
##### 第 73 次发布
发布时间:2024-05-14 02:45:49
本次发布包含了以下内容:
改善已有的文档。
修改数据结构:
-
[McuWaterMarkText](https://cloud.tencent.com/document/api/647/44055#McuWaterMarkText)
- 新增成员:Font
#### 微服务引擎(tse) 版本:2020-12-07
##### 第 66 次发布
发布时间:2024-05-14 02:46:25
本次发布包含了以下内容:
改善已有的文档。
修改数据结构:
-
[CloudNativeAPIGatewayNode](https://cloud.tencent.com/document/api/1364/54942#CloudNativeAPIGatewayNode)
- 新增成员:Weight, IsDefaultWeight
- <font color="#dd0000">**修改成员**:</font>ZoneId, Zone, GroupId,
GroupName, Status
-
[NativeGatewayServerGroup](https://cloud.tencent.com/document/api/1364/54942#NativeGatewayServerGroup)
…
Description:
This is the same code as OTel-Arrow at https://github.com/open-telemetry/otel-arrow/releases/tag/v0.23.0 (plus backported lint and test fixes). Only import statements change here, to match the host repository.
Link to tracking Issue: #26491
Testing: Test coverage is approximately 90%.
Documentation: I double-checked and the existing README had only a few updates needed.