Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement(opentelemetry source): Add HTTP server and integration test #13798

Merged
merged 17 commits into from
Aug 7, 2022

Conversation

spencergilbert
Copy link
Contributor

Closes: #13551

Technically this is breaking change to the configuration, but since the source hasn't been released on a stable tag I didn't mark the PR as such.

This doesn't implement JSON support, but could be enhanced to in the future as that becomes stable:

Ref: OTLP specification for requests and responses.

@spencergilbert spencergilbert added ci-condition: integration tests enable Run integration tests on this PR source: opentelemetry Anything `opentelemetry` source related labels Aug 2, 2022
@spencergilbert spencergilbert self-assigned this Aug 2, 2022
@netlify
Copy link

netlify bot commented Aug 2, 2022

Deploy Preview for vector-project ready!

Name Link
🔨 Latest commit fb47c16
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/62f02b0aa7c4e20008fdb345
😎 Deploy Preview https://deploy-preview-13798--vector-project.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions github-actions bot added domain: ci Anything related to Vector's CI environment domain: sources Anything related to the Vector's sources labels Aug 2, 2022
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Just left a couple of questions below.

src/sources/opentelemetry/tests.rs Show resolved Hide resolved
src/sources/opentelemetry/http.rs Outdated Show resolved Hide resolved
src/sources/opentelemetry/mod.rs Show resolved Hide resolved
@github-actions

This comment was marked as outdated.

@github-actions github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Aug 2, 2022
@github-actions
Copy link

github-actions bot commented Aug 4, 2022

Soak Test Results

Baseline: 4b3fe0a
Comparison: bf0f45a
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
splunk_hec_route_s3 712.9KiB 3.75 100.00% 18.58MiB 2.31MiB 48.03KiB 0 0.124111 19.27MiB 2.15MiB 44.84KiB 0 0.11128 False False
http_text_to_http_json 1.27MiB 3.36 100.00% 37.67MiB 901.06KiB 18.39KiB 0 0.0233538 38.94MiB 881.99KiB 18.0KiB 0 0.0221168 False False
socket_to_socket_blackhole 475.17KiB 2.08 100.00% 22.28MiB 443.91KiB 9.06KiB 0 0.0194514 22.75MiB 630.28KiB 12.87KiB 0 0.0270547 False False
syslog_loki 246.14KiB 1.72 100.00% 13.96MiB 472.14KiB 9.67KiB 0 0.0330177 14.2MiB 840.67KiB 17.09KiB 0 0.0577949 False False
http_pipelines_blackhole_acks 10.73KiB 0.89 100.00% 1.18MiB 107.09KiB 2.18KiB 0 0.0886779 1.19MiB 66.42KiB 1.35KiB 0 0.0545111 False False
datadog_agent_remap_blackhole 430.08KiB 0.69 99.96% 61.06MiB 4.63MiB 96.37KiB 0 0.0757577 61.48MiB 3.49MiB 72.87KiB 0 0.0567869 False False
syslog_log2metric_humio_metrics 48.68KiB 0.37 99.98% 12.77MiB 329.9KiB 6.74KiB 0 0.0252173 12.82MiB 544.26KiB 11.08KiB 0 0.0414485 False False
splunk_hec_to_splunk_hec_logs_noack 20.45KiB 0.08 90.25% 23.82MiB 504.27KiB 10.29KiB 0 0.0206721 23.84MiB 333.43KiB 6.81KiB 0 0.0136573 False False
enterprise_http_to_http -839.23B -0 9.11% 23.85MiB 246.73KiB 5.04KiB 0 0.0101025 23.84MiB 248.98KiB 5.09KiB 0 0.0101947 False False
splunk_hec_to_splunk_hec_logs_acks -2.46KiB -0.01 8.44% 23.76MiB 807.56KiB 16.43KiB 0 0.0331795 23.76MiB 803.5KiB 16.35KiB 0 0.0330162 False False
splunk_hec_indexer_ack_blackhole -5.45KiB -0.02 17.10% 23.75MiB 865.06KiB 17.6KiB 0 0.0355556 23.75MiB 889.04KiB 18.08KiB 0 0.0365495 False False
file_to_blackhole -57.47KiB -0.06 40.11% 95.34MiB 3.6MiB 74.72KiB 0 0.0377991 95.28MiB 3.83MiB 79.72KiB 0 0.0402187 False False
datadog_agent_remap_datadog_logs_acks -57.11KiB -0.09 37.25% 63.59MiB 3.37MiB 70.37KiB 0 0.052955 63.54MiB 4.53MiB 94.32KiB 0 0.0713023 False False
http_to_http_json -53.16KiB -0.22 99.98% 23.85MiB 340.11KiB 6.94KiB 0 0.013925 23.79MiB 606.96KiB 12.37KiB 0 0.0249049 False False
http_to_http_noack -77.58KiB -0.32 99.94% 23.84MiB 407.8KiB 8.34KiB 0 0.0167036 23.76MiB 1.01MiB 21.06KiB 0 0.042484 False False
http_pipelines_blackhole -5.29KiB -0.32 98.06% 1.63MiB 10.58KiB 221.36B 0 0.0063324 1.63MiB 110.39KiB 2.25KiB 0 0.0662844 False False
fluent_elasticsearch -315.15KiB -0.39 100.00% 79.47MiB 55.46KiB 1.12KiB 0 0.000681325 79.17MiB 3.56MiB 73.3KiB 0 0.045023 False False
syslog_humio_logs -98.23KiB -0.57 100.00% 16.83MiB 612.36KiB 12.5KiB 0 0.0355278 16.73MiB 481.09KiB 9.84KiB 0 0.0280719 False False
datadog_agent_remap_datadog_logs -415.77KiB -0.64 100.00% 63.32MiB 266.9KiB 5.46KiB 0 0.00411521 62.92MiB 3.97MiB 82.62KiB 0 0.0630155 False False
http_pipelines_no_grok_blackhole -77.16KiB -0.69 99.89% 10.99MiB 315.13KiB 6.43KiB 0 0.0279844 10.92MiB 1.1MiB 22.81KiB 0 0.100317 False False
datadog_agent_remap_blackhole_acks -609.85KiB -0.91 100.00% 65.33MiB 4.57MiB 95.2KiB 0 0.0699742 64.74MiB 3.06MiB 63.89KiB 0 0.0471882 False False
syslog_splunk_hec_logs -221.54KiB -1.27 100.00% 17.01MiB 782.79KiB 15.93KiB 0 0.0449396 16.79MiB 691.46KiB 14.09KiB 0 0.0402074 False False
syslog_regex_logs2metric_ddmetrics -196.09KiB -1.51 100.00% 12.71MiB 632.2KiB 12.88KiB 0 0.0485511 12.52MiB 684.82KiB 13.95KiB 0 0.0533962 False False
syslog_log2metric_splunk_hec_metrics -269.09KiB -1.51 100.00% 17.45MiB 664.02KiB 13.52KiB 0 0.0371508 17.19MiB 729.66KiB 14.85KiB 0 0.0414471 False False
http_to_http_acks -514.56KiB -2.7 97.50% 18.59MiB 7.4MiB 154.76KiB 0 0.398222 18.09MiB 8.11MiB 169.4KiB 0 0.448563 True True

@spencergilbert spencergilbert requested a review from jszwedko August 4, 2022 17:05
@github-actions
Copy link

github-actions bot commented Aug 4, 2022

Soak Test Results

Baseline: 2c768b0
Comparison: f6698d0
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json 1.27MiB 3.36 100.00% 37.9MiB 861.31KiB 17.58KiB 0 0.0221893 39.17MiB 765.65KiB 15.64KiB 0 0.0190833 False False
splunk_hec_route_s3 553.53KiB 2.91 100.00% 18.59MiB 2.29MiB 47.65KiB 0 0.123093 19.13MiB 2.17MiB 45.5KiB 0 0.113617 False False
syslog_loki 289.6KiB 1.97 100.00% 14.34MiB 249.95KiB 5.12KiB 0 0.0170224 14.62MiB 731.34KiB 14.87KiB 0 0.0488422 False False
socket_to_socket_blackhole 393.29KiB 1.71 100.00% 22.46MiB 451.63KiB 9.22KiB 0 0.0196322 22.85MiB 538.24KiB 10.99KiB 0 0.0230034 False False
http_to_http_acks 148.39KiB 0.8 45.63% 18.15MiB 8.49MiB 177.63KiB 0 0.467938 18.29MiB 8.03MiB 167.81KiB 0 0.438922 True True
syslog_log2metric_humio_metrics 63.15KiB 0.48 100.00% 12.84MiB 195.37KiB 3.99KiB 0 0.0148562 12.9MiB 496.14KiB 10.1KiB 0 0.0375467 False False
http_pipelines_blackhole_acks 3.1KiB 0.27 70.56% 1.12MiB 116.67KiB 2.37KiB 0 0.101535 1.12MiB 86.77KiB 1.77KiB 0 0.0753103 False False
splunk_hec_indexer_ack_blackhole 16.29KiB 0.07 46.31% 23.74MiB 939.46KiB 19.11KiB 0 0.0386351 23.76MiB 893.75KiB 18.19KiB 0 0.0367306 False False
datadog_agent_remap_datadog_logs_acks 37.04KiB 0.06 25.94% 63.0MiB 3.13MiB 65.32KiB 0 0.049595 63.04MiB 4.36MiB 90.82KiB 0 0.0691846 False False
splunk_hec_to_splunk_hec_logs_noack 9.41KiB 0.04 60.17% 23.83MiB 434.56KiB 8.87KiB 0 0.017806 23.84MiB 330.39KiB 6.74KiB 0 0.0135326 False False
enterprise_http_to_http -1.26KiB -0.01 14.04% 23.85MiB 244.54KiB 4.99KiB 0 0.0100123 23.85MiB 247.39KiB 5.06KiB 0 0.0101292 False False
splunk_hec_to_splunk_hec_logs_acks -10.81KiB -0.04 34.32% 23.76MiB 829.95KiB 16.88KiB 0 0.0341067 23.75MiB 861.06KiB 17.51KiB 0 0.0354008 False False
file_to_blackhole -130.91KiB -0.13 84.03% 95.36MiB 2.65MiB 54.83KiB 0 0.0277324 95.23MiB 3.62MiB 75.22KiB 0 0.0380022 False False
http_to_http_json -41.69KiB -0.17 99.82% 23.85MiB 342.18KiB 6.99KiB 0 0.0140101 23.81MiB 556.14KiB 11.35KiB 0 0.0228093 False False
http_pipelines_no_grok_blackhole -45.5KiB -0.4 96.21% 11.07MiB 173.68KiB 3.55KiB 0 0.0153167 11.03MiB 1.04MiB 21.62KiB 0 0.0941103 False False
http_to_http_noack -105.26KiB -0.43 100.00% 23.84MiB 403.25KiB 8.24KiB 0 0.0165167 23.73MiB 1.15MiB 23.95KiB 0 0.0484067 False False
http_pipelines_blackhole -7.92KiB -0.49 99.43% 1.58MiB 66.96KiB 1.37KiB 0 0.0415095 1.57MiB 123.54KiB 2.52KiB 0 0.0769597 False False
fluent_elasticsearch -399.33KiB -0.49 100.00% 79.47MiB 52.14KiB 1.05KiB 0 0.000640572 79.08MiB 4.4MiB 90.44KiB 0 0.0556615 False False
datadog_agent_remap_datadog_logs -531.27KiB -0.84 100.00% 61.95MiB 1.61MiB 33.66KiB 0 0.0259272 61.43MiB 4.27MiB 88.9KiB 0 0.0694672 False False
datadog_agent_remap_blackhole -576.22KiB -0.9 100.00% 62.46MiB 4.71MiB 98.22KiB 0 0.0754644 61.9MiB 3.99MiB 83.23KiB 0 0.0644325 False False
syslog_humio_logs -180.25KiB -1.05 100.00% 16.74MiB 540.97KiB 11.04KiB 0 0.0315561 16.56MiB 499.4KiB 10.22KiB 0 0.029441 False False
syslog_regex_logs2metric_ddmetrics -187.12KiB -1.49 100.00% 12.27MiB 710.47KiB 14.48KiB 0 0.0565561 12.08MiB 636.74KiB 12.98KiB 0 0.0514528 False False
syslog_log2metric_splunk_hec_metrics -358.96KiB -1.91 100.00% 18.33MiB 818.98KiB 16.69KiB 0 0.0436207 17.98MiB 931.33KiB 18.96KiB 0 0.0505721 False False
syslog_splunk_hec_logs -330.43KiB -1.97 100.00% 16.4MiB 835.81KiB 17.02KiB 0 0.0497512 16.08MiB 742.1KiB 15.14KiB 0 0.0450598 False False
datadog_agent_remap_blackhole_acks -1.53MiB -2.49 100.00% 61.49MiB 5.33MiB 110.93KiB 0 0.0866137 59.96MiB 4.0MiB 83.75KiB 0 0.0667652 False False

Signed-off-by: Jesse Szwedko <[email protected]>
@spencergilbert
Copy link
Contributor Author

That's certainly a cleaner way of CUEing 😆 - thanks @jszwedko

Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Changes around internal events look good to me. Just one question below. I refactored the cue a bit in a9e8e08 . I just did it directly since I wasn't sure exactly what it would look like or if it'd be possible.

src/sources/opentelemetry/http.rs Outdated Show resolved Hide resolved
src/sources/opentelemetry/http.rs Outdated Show resolved Hide resolved
@github-actions
Copy link

github-actions bot commented Aug 4, 2022

Soak Test Results

Baseline: c402e6e
Comparison: a9e8e08
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
socket_to_socket_blackhole 964.44KiB 4.2 100.00% 22.45MiB 110.45KiB 2.25KiB 0 0.00480404 23.39MiB 173.32KiB 3.54KiB 0 0.00723494 False False
http_text_to_http_json 1.41MiB 3.72 100.00% 37.92MiB 817.68KiB 16.69KiB 0 0.0210542 39.33MiB 724.75KiB 14.8KiB 0 0.0179924 False False
splunk_hec_route_s3 407.37KiB 2.27 100.00% 17.49MiB 2.32MiB 48.26KiB 0 0.132376 17.89MiB 2.27MiB 47.45KiB 0 0.126784 False False
http_to_http_acks 265.61KiB 1.45 73.03% 17.87MiB 8.35MiB 174.64KiB 0 0.467399 18.13MiB 7.93MiB 165.52KiB 0 0.437219 True True
http_pipelines_blackhole_acks 15.96KiB 1.33 100.00% 1.17MiB 108.95KiB 2.22KiB 0 0.0910956 1.18MiB 66.36KiB 1.35KiB 0 0.0547525 False False
datadog_agent_remap_datadog_logs_acks 335.2KiB 0.53 98.95% 61.81MiB 3.95MiB 82.52KiB 0 0.0639153 62.14MiB 4.88MiB 101.63KiB 0 0.0785626 False False
datadog_agent_remap_blackhole 217.97KiB 0.36 90.27% 58.36MiB 4.69MiB 97.9KiB 0 0.0804258 58.57MiB 4.2MiB 87.71KiB 0 0.0717642 False False
syslog_log2metric_humio_metrics 27.09KiB 0.2 98.53% 12.94MiB 273.8KiB 5.59KiB 0 0.0206512 12.97MiB 470.68KiB 9.59KiB 0 0.0354284 False False
splunk_hec_to_splunk_hec_logs_noack 15.54KiB 0.06 82.60% 23.82MiB 450.89KiB 9.2KiB 0 0.0184793 23.84MiB 332.18KiB 6.78KiB 0 0.0136055 False False
enterprise_http_to_http -1.54KiB -0.01 16.58% 23.85MiB 252.84KiB 5.16KiB 0 0.0103525 23.84MiB 257.14KiB 5.26KiB 0 0.0105292 False False
splunk_hec_indexer_ack_blackhole -8.59KiB -0.04 26.72% 23.75MiB 851.25KiB 17.31KiB 0 0.0349926 23.74MiB 898.61KiB 18.27KiB 0 0.0369525 False False
syslog_regex_logs2metric_ddmetrics -5.05KiB -0.04 22.63% 12.78MiB 652.32KiB 13.28KiB 0 0.0498347 12.78MiB 562.95KiB 11.48KiB 0 0.0430237 False False
file_to_blackhole -65.25KiB -0.07 45.36% 95.34MiB 3.34MiB 69.14KiB 0 0.0349775 95.28MiB 4.0MiB 83.17KiB 0 0.0419894 False False
http_to_http_json -28.14KiB -0.12 98.08% 23.85MiB 338.69KiB 6.91KiB 0 0.0138664 23.82MiB 480.76KiB 9.83KiB 0 0.0197054 False False
splunk_hec_to_splunk_hec_logs_acks -28.5KiB -0.12 75.90% 23.77MiB 790.26KiB 16.08KiB 0 0.0324652 23.74MiB 895.73KiB 18.21KiB 0 0.0368409 False False
http_to_http_noack -96.0KiB -0.39 99.99% 23.84MiB 407.99KiB 8.35KiB 0 0.0167115 23.74MiB 1.11MiB 23.07KiB 0 0.0465981 False False
syslog_splunk_hec_logs -76.52KiB -0.45 99.97% 16.54MiB 771.99KiB 15.72KiB 0 0.0455745 16.46MiB 681.79KiB 13.9KiB 0 0.0404323 False False
syslog_humio_logs -95.11KiB -0.55 100.00% 17.03MiB 265.63KiB 5.42KiB 0 0.0152304 16.94MiB 115.2KiB 2.36KiB 0 0.00664174 False False
fluent_elasticsearch -444.43KiB -0.55 100.00% 79.47MiB 54.12KiB 1.09KiB 0 0.000664846 79.04MiB 4.41MiB 90.54KiB 0 0.0557747 False False
datadog_agent_remap_datadog_logs -389.42KiB -0.62 99.91% 61.22MiB 2.69MiB 56.4KiB 0 0.0438942 60.84MiB 4.91MiB 102.2KiB 0 0.080642 False False
datadog_agent_remap_blackhole_acks -459.48KiB -0.75 99.99% 59.76MiB 4.44MiB 92.51KiB 0 0.0743159 59.31MiB 3.35MiB 70.0KiB 0 0.0564379 False False
http_pipelines_blackhole -12.13KiB -0.76 100.00% 1.56MiB 60.67KiB 1.24KiB 0 0.0379821 1.55MiB 128.66KiB 2.62KiB 0 0.0811618 False False
http_pipelines_no_grok_blackhole -131.41KiB -1.13 100.00% 11.33MiB 268.0KiB 5.47KiB 0 0.0230858 11.21MiB 1.11MiB 23.18KiB 0 0.0993497 False False
syslog_log2metric_splunk_hec_metrics -227.9KiB -1.2 100.00% 18.58MiB 585.08KiB 11.93KiB 0 0.0307524 18.35MiB 874.24KiB 17.78KiB 0 0.0465083 False False
syslog_loki -414.42KiB -2.7 100.00% 14.98MiB 278.61KiB 5.71KiB 0 0.018157 14.58MiB 737.71KiB 15.0KiB 0 0.0494118 False False

Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥳

@spencergilbert spencergilbert enabled auto-merge (squash) August 5, 2022 19:25
src/sources/opentelemetry/http.rs Show resolved Hide resolved
@github-actions
Copy link

github-actions bot commented Aug 5, 2022

Soak Test Results

Baseline: c402e6e
Comparison: 24e25aa
Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
socket_to_socket_blackhole 579.02KiB 2.64 100.00% 21.43MiB 306.18KiB 6.25KiB 0 0.0139521 21.99MiB 440.35KiB 8.99KiB 0 0.0195498 False False
splunk_hec_route_s3 362.9KiB 1.95 100.00% 18.17MiB 2.36MiB 49.1KiB 0 0.129743 18.52MiB 2.26MiB 47.34KiB 0 0.122153 False False
http_text_to_http_json 601.86KiB 1.55 100.00% 37.81MiB 830.36KiB 16.95KiB 0 0.0214438 38.39MiB 812.86KiB 16.6KiB 0 0.0206707 False False
datadog_agent_remap_blackhole 642.92KiB 1.01 100.00% 62.01MiB 4.37MiB 91.19KiB 0 0.0705369 62.63MiB 3.78MiB 78.85KiB 0 0.060342 False False
syslog_log2metric_humio_metrics 81.37KiB 0.61 100.00% 12.94MiB 343.88KiB 7.01KiB 0 0.0259537 13.02MiB 543.03KiB 11.05KiB 0 0.0407337 False False
http_to_http_acks 93.13KiB 0.51 28.93% 17.86MiB 8.75MiB 182.81KiB 0 0.489564 17.95MiB 8.25MiB 172.03KiB 0 0.459702 True True
datadog_agent_remap_blackhole_acks 232.09KiB 0.35 96.41% 64.18MiB 4.43MiB 92.28KiB 0 0.0690283 64.41MiB 2.91MiB 60.96KiB 0 0.0452398 False False
splunk_hec_to_splunk_hec_logs_acks 23.78KiB 0.1 66.88% 23.74MiB 897.37KiB 18.25KiB 0 0.0369061 23.76MiB 800.86KiB 16.3KiB 0 0.0329047 False False
splunk_hec_to_splunk_hec_logs_noack 19.79KiB 0.08 89.48% 23.82MiB 498.61KiB 10.18KiB 0 0.0204387 23.84MiB 330.63KiB 6.75KiB 0 0.0135419 False False
enterprise_http_to_http -4.57KiB -0.02 46.25% 23.85MiB 257.36KiB 5.25KiB 0 0.0105368 23.84MiB 255.73KiB 5.24KiB 0 0.0104719 False False
splunk_hec_indexer_ack_blackhole -3.91KiB -0.02 12.03% 23.75MiB 892.55KiB 18.15KiB 0 0.0366957 23.74MiB 904.49KiB 18.39KiB 0 0.0371926 False False
syslog_regex_logs2metric_ddmetrics -5.3KiB -0.04 23.83% 12.7MiB 614.96KiB 12.53KiB 0 0.0472892 12.69MiB 597.73KiB 12.18KiB 0 0.0459828 False False
file_to_blackhole -49.48KiB -0.05 56.99% 95.38MiB 2.0MiB 41.36KiB 0 0.0209146 95.33MiB 2.27MiB 47.12KiB 0 0.0237657 False False
datadog_agent_remap_datadog_logs_acks -39.45KiB -0.06 25.56% 62.95MiB 3.74MiB 78.1KiB 0 0.0594436 62.91MiB 4.44MiB 92.42KiB 0 0.0705579 False False
http_to_http_json -36.97KiB -0.15 99.55% 23.85MiB 344.3KiB 7.03KiB 0 0.0140973 23.81MiB 535.05KiB 10.93KiB 0 0.0219409 False False
fluent_elasticsearch -125.31KiB -0.15 100.00% 79.47MiB 53.68KiB 1.08KiB 0 0.000659507 79.35MiB 1.09MiB 22.49KiB 0 0.0137839 False False
http_pipelines_blackhole -3.26KiB -0.2 97.10% 1.56MiB 27.16KiB 568.44B 0 0.0170386 1.55MiB 67.86KiB 1.38KiB 0 0.0426553 False False
syslog_splunk_hec_logs -45.87KiB -0.27 93.68% 16.53MiB 924.93KiB 18.82KiB 0 0.0546475 16.48MiB 782.69KiB 15.97KiB 0 0.0463692 False False
http_to_http_noack -83.84KiB -0.34 99.92% 23.83MiB 514.1KiB 10.51KiB 0 0.0210655 23.75MiB 1.08MiB 22.6KiB 0 0.0456412 False False
syslog_humio_logs -63.19KiB -0.36 100.00% 17.2MiB 150.71KiB 3.08KiB 0 0.00855454 17.14MiB 125.4KiB 2.57KiB 0 0.00714349 False False
syslog_log2metric_splunk_hec_metrics -72.47KiB -0.4 98.81% 17.7MiB 940.87KiB 19.19KiB 0 0.0518964 17.63MiB 1.03MiB 21.47KiB 0 0.0583831 False False
http_pipelines_blackhole_acks -6.03KiB -0.5 97.75% 1.18MiB 106.69KiB 2.17KiB 0 0.0881409 1.18MiB 73.87KiB 1.51KiB 0 0.0613334 False False
datadog_agent_remap_datadog_logs -489.5KiB -0.77 100.00% 62.01MiB 1002.03KiB 20.51KiB 0 0.0157769 61.53MiB 4.27MiB 88.93KiB 0 0.0693999 False False
http_pipelines_no_grok_blackhole -101.63KiB -0.89 100.00% 11.18MiB 235.83KiB 4.81KiB 0 0.0205876 11.08MiB 1.06MiB 22.15KiB 0 0.0959047 False False
syslog_loki -486.46KiB -3.31 100.00% 14.37MiB 299.07KiB 6.13KiB 0 0.0203185 13.9MiB 695.1KiB 14.13KiB 0 0.0488392 False False

@spencergilbert spencergilbert merged commit 4611307 into master Aug 7, 2022
@spencergilbert spencergilbert deleted the spencer/otel-logs-http-source branch August 7, 2022 22:26
jdrouet pushed a commit that referenced this pull request Aug 22, 2022
…st (#13798)

* add stubs for http otel source

Signed-off-by: Spencer Gilbert <[email protected]>

* reorganize otel code

Signed-off-by: Spencer Gilbert <[email protected]>

* implement custom warp server

* reject bodies are rpc.status encoded protobuf

* add integration tests, pass protocol into logs filter

* emit http bytes after decompression

* +fix typo in HttpBytesReceived

* +emit HttpEventsReceived instead of just EventsReceived

* +based on offline discussions, revert event change

* +update cue for new config shape

* +fix example config for docs

* +remove client metadata key, not supported

* +cue fmt

* Refactor cue

Signed-off-by: Jesse Szwedko <[email protected]>

* +extract rejection handler, specify all gRPC status.codes

Co-authored-by: Jesse Szwedko <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-condition: integration tests enable Run integration tests on this PR domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation domain: sources Anything related to the Vector's sources source: opentelemetry Anything `opentelemetry` source related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support ingesting OpenTelemetry logs
3 participants