feat: migrate to async_nats client

[makarchuk]

Implements migration from nats::asynk client to more modern official client async_nats

Closes #15534

[netlify]

✅ Deploy Preview for vrl-playground ready!

Name	Link
🔨 Latest commit	83ab34c
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/63a5f240dcb1010008bafde8
😎 Deploy Preview	https://deploy-preview-15533--vrl-playground.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[netlify]

✅ Deploy Preview for vector-project ready!

Name	Link
🔨 Latest commit	83ab34c
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/63a5f2405a0994000837d153
😎 Deploy Preview	https://deploy-preview-15533--vector-project.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions]

Regression Test Results

Run ID: fa37f44d-fd16-4727-ad0e-c18288530361
Baseline: add1141
Comparison: 68592e1
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
syslog_regex_logs2metric_ddmetrics	74.96KiB/CPU-s	2.03	100.00%	3.61MiB/CPU-s	422.62KiB/CPU-s	5.46KiB/CPU-s	0.0	0.114459	3.68MiB/CPU-s	427.01KiB/CPU-s	5.51KiB/CPU-s	0.0	0.113346	True	False
datadog_agent_remap_blackhole	471.2KiB/CPU-s	1.49	100.00%	30.89MiB/CPU-s	1.79MiB/CPU-s	23.64KiB/CPU-s	0.0	0.057869	31.35MiB/CPU-s	998.78KiB/CPU-s	12.89KiB/CPU-s	0.0	0.031107	False	False
syslog_loki	90.85KiB/CPU-s	1.08	100.00%	8.18MiB/CPU-s	308.41KiB/CPU-s	3.98KiB/CPU-s	0.0	0.036817	8.27MiB/CPU-s	178.81KiB/CPU-s	2.31KiB/CPU-s	0.0	0.021117	False	False
http_to_http_acks	25.71KiB/CPU-s	0.48	37.61%	5.26MiB/CPU-s	2.78MiB/CPU-s	36.71KiB/CPU-s	0.0	0.528487	5.28MiB/CPU-s	2.83MiB/CPU-s	37.46KiB/CPU-s	0.0	0.536528	True	False
syslog_log2metric_splunk_hec_metrics	41.62KiB/CPU-s	0.45	100.00%	8.97MiB/CPU-s	514.52KiB/CPU-s	6.64KiB/CPU-s	0.0	0.055998	9.01MiB/CPU-s	178.86KiB/CPU-s	2.31KiB/CPU-s	0.0	0.019379	False	False
enterprise_http_to_http	12.26KiB/CPU-s	0.09	98.17%	13.61MiB/CPU-s	353.19KiB/CPU-s	4.56KiB/CPU-s	0.0	0.025341	13.62MiB/CPU-s	192.6KiB/CPU-s	2.49KiB/CPU-s	0.0	0.013807	False	False
splunk_hec_to_splunk_hec_logs_noack	5.41KiB/CPU-s	0.04	83.50%	13.62MiB/CPU-s	233.81KiB/CPU-s	3.02KiB/CPU-s	0.0	0.016763	13.63MiB/CPU-s	190.87KiB/CPU-s	2.46KiB/CPU-s	0.0	0.013679	False	False
fluent_elasticsearch	608.09B/CPU-s	0.0	72.77%	45.41MiB/CPU-s	29.74KiB/CPU-s	388.87B/CPU-s	0.0	0.00064	45.41MiB/CPU-s	30.17KiB/CPU-s	394.37B/CPU-s	0.0	0.000649	False	False
splunk_hec_indexer_ack_blackhole	-467.55B/CPU-s	-0.0	8.06%	13.62MiB/CPU-s	243.24KiB/CPU-s	3.14KiB/CPU-s	0.0	0.017442	13.62MiB/CPU-s	251.42KiB/CPU-s	3.24KiB/CPU-s	0.0	0.018029	False	False
http_to_http_noack	-1.94KiB/CPU-s	-0.01	30.35%	13.62MiB/CPU-s	258.4KiB/CPU-s	3.34KiB/CPU-s	0.0	0.018529	13.62MiB/CPU-s	286.1KiB/CPU-s	3.69KiB/CPU-s	0.0	0.020518	False	False
file_to_blackhole	-4.07KiB/CPU-s	-0.01	15.06%	54.49MiB/CPU-s	1.12MiB/CPU-s	14.82KiB/CPU-s	0.0	0.020584	54.49MiB/CPU-s	1.17MiB/CPU-s	15.5KiB/CPU-s	0.0	0.021551	False	False
splunk_hec_to_splunk_hec_logs_acks	-2.44KiB/CPU-s	-0.02	30.07%	13.62MiB/CPU-s	350.43KiB/CPU-s	4.52KiB/CPU-s	0.0	0.025125	13.62MiB/CPU-s	342.34KiB/CPU-s	4.42KiB/CPU-s	0.0	0.02455	False	False
http_to_http_json	-10.82KiB/CPU-s	-0.08	98.05%	13.61MiB/CPU-s	241.93KiB/CPU-s	3.12KiB/CPU-s	0.0	0.017357	13.6MiB/CPU-s	265.31KiB/CPU-s	3.42KiB/CPU-s	0.0	0.01905	False	False
datadog_agent_remap_datadog_logs	-81.94KiB/CPU-s	-0.24	99.97%	33.68MiB/CPU-s	1.2MiB/CPU-s	15.88KiB/CPU-s	0.0	0.035692	33.6MiB/CPU-s	1.23MiB/CPU-s	16.24KiB/CPU-s	0.0	0.03658	False	False
syslog_log2metric_humio_metrics	-24.48KiB/CPU-s	-0.4	100.00%	5.97MiB/CPU-s	140.42KiB/CPU-s	1.81KiB/CPU-s	0.0	0.022961	5.95MiB/CPU-s	338.97KiB/CPU-s	4.38KiB/CPU-s	0.0	0.055649	False	False
syslog_splunk_hec_logs	-47.5KiB/CPU-s	-0.53	100.00%	8.71MiB/CPU-s	176.38KiB/CPU-s	2.28KiB/CPU-s	0.0	0.019783	8.66MiB/CPU-s	214.67KiB/CPU-s	2.77KiB/CPU-s	0.0	0.024206	False	False
http_text_to_http_json	-143.58KiB/CPU-s	-0.58	100.00%	24.03MiB/CPU-s	608.64KiB/CPU-s	7.86KiB/CPU-s	0.0	0.02473	23.89MiB/CPU-s	550.71KiB/CPU-s	7.11KiB/CPU-s	0.0	0.022508	False	False
datadog_agent_remap_datadog_logs_acks	-254.08KiB/CPU-s	-0.73	100.00%	33.83MiB/CPU-s	999.42KiB/CPU-s	12.89KiB/CPU-s	0.0	0.028847	33.58MiB/CPU-s	926.18KiB/CPU-s	11.95KiB/CPU-s	0.0	0.02693	False	False
splunk_hec_route_s3	-88.23KiB/CPU-s	-0.73	100.00%	11.85MiB/CPU-s	514.64KiB/CPU-s	6.64KiB/CPU-s	0.0	0.04241	11.76MiB/CPU-s	503.05KiB/CPU-s	6.49KiB/CPU-s	0.0	0.041758	False	False
http_pipelines_no_grok_blackhole	-53.18KiB/CPU-s	-0.79	100.00%	6.6MiB/CPU-s	28.92KiB/CPU-s	382.3B/CPU-s	0.0	0.004276	6.55MiB/CPU-s	27.9KiB/CPU-s	368.94B/CPU-s	0.0	0.004159	False	False
syslog_humio_logs	-78.57KiB/CPU-s	-0.87	100.00%	8.85MiB/CPU-s	214.38KiB/CPU-s	2.77KiB/CPU-s	0.0	0.023649	8.78MiB/CPU-s	92.11KiB/CPU-s	1.19KiB/CPU-s	0.0	0.01025	False	False
http_pipelines_blackhole	-24.97KiB/CPU-s	-0.96	100.00%	2.54MiB/CPU-s	113.01KiB/CPU-s	1.46KiB/CPU-s	0.0	0.04339	2.52MiB/CPU-s	72.03KiB/CPU-s	952.36B/CPU-s	0.0	0.027924	False	False
otlp_grpc_to_blackhole	-12.12KiB/CPU-s	-1.25	100.00%	971.95KiB/CPU-s	33.05KiB/CPU-s	436.93B/CPU-s	0.0	0.034002	959.83KiB/CPU-s	46.55KiB/CPU-s	615.06B/CPU-s	0.0	0.048498	False	False
http_pipelines_blackhole_acks	-26.31KiB/CPU-s	-1.37	100.00%	1.87MiB/CPU-s	62.44KiB/CPU-s	825.53B/CPU-s	0.0	0.032608	1.84MiB/CPU-s	77.17KiB/CPU-s	1019.93B/CPU-s	0.0	0.040859	False	False
socket_to_socket_blackhole	-193.67KiB/CPU-s	-1.47	100.00%	12.84MiB/CPU-s	201.2KiB/CPU-s	2.6KiB/CPU-s	0.0	0.015296	12.66MiB/CPU-s	133.37KiB/CPU-s	1.72KiB/CPU-s	0.0	0.010291	False	False
otlp_http_to_blackhole	-26.19KiB/CPU-s	-1.78	100.00%	1.44MiB/CPU-s	131.52KiB/CPU-s	1.7KiB/CPU-s	0.0	0.089224	1.41MiB/CPU-s	142.49KiB/CPU-s	1.84KiB/CPU-s	0.0	0.098414	False	False
datadog_agent_remap_blackhole_acks	-604.84KiB/CPU-s	-1.83	100.00%	32.36MiB/CPU-s	411.96KiB/CPU-s	5.32KiB/CPU-s	0.0	0.012433	31.77MiB/CPU-s	883.73KiB/CPU-s	11.41KiB/CPU-s	0.0	0.027166	False	False

[StephenWakely]

Nice. Thanks for doing this.

The issue you mentioned around buffering should be fine, from what I can tell the async library isn't doing any buffering so it shouldn't be a problem.

There are a couple of CI errors that need fixing. The integration tests have some compile errors. nats.rs needs to be formatted correctly, which you can do by running the file through rustfmt.

[github-actions]

Regression Test Results

Run ID: 6e988203-128f-4e9c-bba8-b162ec5e6d2c
Baseline: 3c9e41f
Comparison: 83ab34c
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their bytes_written_per_cpu_second performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5% change in mean bytes_written_per_cpu_second are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in bytes_written_per_cpu_second with confidence ≥ 90.00% and absolute Δ mean >= ±5%.

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline outlier %	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison outlier %	comparison CoV	erratic	declared erratic
http_text_to_http_json	912.05KiB/CPU-s	3.57	100.00%	24.92MiB/CPU-s	556.35KiB/CPU-s	7.18KiB/CPU-s	0.0	0.021802	25.81MiB/CPU-s	522.31KiB/CPU-s	6.74KiB/CPU-s	0.0	0.019762	False	False
datadog_agent_remap_blackhole_acks	959.21KiB/CPU-s	3.02	100.00%	31.02MiB/CPU-s	1.22MiB/CPU-s	16.13KiB/CPU-s	0.0	0.039327	31.95MiB/CPU-s	602.28KiB/CPU-s	7.77KiB/CPU-s	0.0	0.018405	False	False
datadog_agent_remap_datadog_logs_acks	330.66KiB/CPU-s	0.96	100.00%	33.6MiB/CPU-s	959.98KiB/CPU-s	12.39KiB/CPU-s	0.0	0.027896	33.93MiB/CPU-s	1.09MiB/CPU-s	14.4KiB/CPU-s	0.0	0.032115	False	False
datadog_agent_remap_datadog_logs	123.68KiB/CPU-s	0.36	100.00%	33.85MiB/CPU-s	1.1MiB/CPU-s	14.52KiB/CPU-s	0.0	0.032473	33.97MiB/CPU-s	1.21MiB/CPU-s	15.97KiB/CPU-s	0.0	0.035579	False	False
syslog_log2metric_humio_metrics	21.15KiB/CPU-s	0.35	100.00%	5.98MiB/CPU-s	145.42KiB/CPU-s	1.88KiB/CPU-s	0.0	0.023729	6.0MiB/CPU-s	106.76KiB/CPU-s	1.38KiB/CPU-s	0.0	0.017361	False	False
splunk_hec_route_s3	42.66KiB/CPU-s	0.35	100.00%	11.85MiB/CPU-s	538.16KiB/CPU-s	6.94KiB/CPU-s	0.0	0.044365	11.89MiB/CPU-s	540.28KiB/CPU-s	6.97KiB/CPU-s	0.0	0.044383	False	False
datadog_agent_remap_blackhole	87.9KiB/CPU-s	0.27	100.00%	31.32MiB/CPU-s	725.2KiB/CPU-s	9.36KiB/CPU-s	0.0	0.022613	31.4MiB/CPU-s	603.75KiB/CPU-s	7.79KiB/CPU-s	0.0	0.018775	False	False
otlp_http_to_blackhole	2.41KiB/CPU-s	0.15	74.34%	1.53MiB/CPU-s	111.6KiB/CPU-s	1.44KiB/CPU-s	0.0	0.07132	1.53MiB/CPU-s	120.6KiB/CPU-s	1.56KiB/CPU-s	0.0	0.076951	False	False
socket_to_socket_blackhole	16.75KiB/CPU-s	0.12	100.00%	13.62MiB/CPU-s	231.67KiB/CPU-s	2.99KiB/CPU-s	0.0	0.016606	13.64MiB/CPU-s	100.27KiB/CPU-s	1.29KiB/CPU-s	0.0	0.007178	False	False
enterprise_http_to_http	10.38KiB/CPU-s	0.07	96.30%	13.61MiB/CPU-s	333.38KiB/CPU-s	4.3KiB/CPU-s	0.0	0.023916	13.62MiB/CPU-s	193.83KiB/CPU-s	2.5KiB/CPU-s	0.0	0.013895	False	False
splunk_hec_to_splunk_hec_logs_noack	7.95KiB/CPU-s	0.06	94.76%	13.62MiB/CPU-s	253.16KiB/CPU-s	3.27KiB/CPU-s	0.0	0.018153	13.63MiB/CPU-s	191.34KiB/CPU-s	2.47KiB/CPU-s	0.0	0.013712	False	False
syslog_log2metric_splunk_hec_metrics	4.65KiB/CPU-s	0.05	87.38%	9.12MiB/CPU-s	187.95KiB/CPU-s	2.43KiB/CPU-s	0.0	0.020127	9.12MiB/CPU-s	142.14KiB/CPU-s	1.84KiB/CPU-s	0.0	0.015214	False	False
http_to_http_noack	5.61KiB/CPU-s	0.04	65.10%	13.61MiB/CPU-s	347.16KiB/CPU-s	4.72KiB/CPU-s	0.0	0.024907	13.62MiB/CPU-s	285.06KiB/CPU-s	3.68KiB/CPU-s	0.0	0.020444	False	False
splunk_hec_to_splunk_hec_logs_acks	3.26KiB/CPU-s	0.02	40.82%	13.62MiB/CPU-s	340.12KiB/CPU-s	4.39KiB/CPU-s	0.0	0.024391	13.62MiB/CPU-s	325.79KiB/CPU-s	4.2KiB/CPU-s	0.0	0.023358	False	False
splunk_hec_indexer_ack_blackhole	943.62B/CPU-s	0.01	16.27%	13.62MiB/CPU-s	249.15KiB/CPU-s	3.21KiB/CPU-s	0.0	0.017867	13.62MiB/CPU-s	242.7KiB/CPU-s	3.13KiB/CPU-s	0.0	0.017403	False	False
http_to_http_json	431.71B/CPU-s	0.0	8.84%	13.62MiB/CPU-s	207.99KiB/CPU-s	2.68KiB/CPU-s	0.0	0.01491	13.62MiB/CPU-s	208.22KiB/CPU-s	2.69KiB/CPU-s	0.0	0.014926	False	False
fluent_elasticsearch	176.44B/CPU-s	0.0	24.80%	45.41MiB/CPU-s	29.95KiB/CPU-s	391.39B/CPU-s	0.0	0.000644	45.41MiB/CPU-s	30.46KiB/CPU-s	398.25B/CPU-s	0.0	0.000655	False	False
file_to_blackhole	-8.77KiB/CPU-s	-0.02	32.39%	54.5MiB/CPU-s	1.09MiB/CPU-s	14.35KiB/CPU-s	0.0	0.019931	54.49MiB/CPU-s	1.16MiB/CPU-s	15.31KiB/CPU-s	0.0	0.021288	False	False
http_to_http_acks	-7.73KiB/CPU-s	-0.14	11.86%	5.31MiB/CPU-s	2.79MiB/CPU-s	36.84KiB/CPU-s	0.0	0.525265	5.3MiB/CPU-s	2.76MiB/CPU-s	36.45KiB/CPU-s	0.0	0.520409	True	False
syslog_loki	-11.91KiB/CPU-s	-0.14	99.93%	8.43MiB/CPU-s	214.68KiB/CPU-s	2.77KiB/CPU-s	0.0	0.024859	8.42MiB/CPU-s	166.39KiB/CPU-s	2.15KiB/CPU-s	0.0	0.019294	False	False
syslog_splunk_hec_logs	-19.8KiB/CPU-s	-0.22	100.00%	8.77MiB/CPU-s	133.83KiB/CPU-s	1.73KiB/CPU-s	0.0	0.014907	8.75MiB/CPU-s	226.59KiB/CPU-s	2.92KiB/CPU-s	0.0	0.025294	False	False
syslog_humio_logs	-26.28KiB/CPU-s	-0.29	100.00%	8.75MiB/CPU-s	174.16KiB/CPU-s	2.25KiB/CPU-s	0.0	0.019447	8.72MiB/CPU-s	163.42KiB/CPU-s	2.11KiB/CPU-s	0.0	0.018301	False	False
otlp_grpc_to_blackhole	-4.42KiB/CPU-s	-0.42	100.00%	1.03MiB/CPU-s	42.24KiB/CPU-s	558.31B/CPU-s	0.0	0.040143	1.02MiB/CPU-s	50.06KiB/CPU-s	661.48B/CPU-s	0.0	0.047774	False	False
syslog_regex_logs2metric_ddmetrics	-31.19KiB/CPU-s	-0.81	99.99%	3.75MiB/CPU-s	419.44KiB/CPU-s	5.42KiB/CPU-s	0.0	0.109322	3.72MiB/CPU-s	434.11KiB/CPU-s	5.6KiB/CPU-s	0.0	0.114074	True	False

[StephenWakely]

Hi, I'm getting a bunch of integration test errors now, for example:

--- TRY 1 STDERR:        vector sinks::nats::integration_tests::nats_userpass_auth_valid ---
thread 'sinks::nats::integration_tests::nats_userpass_auth_valid' panicked at 'assertion failed: `(left == right)`
  left: `768`,
 right: `1000`', src/sinks/nats.rs:280:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Are they passing locally for you? You can run the tests by running make test-integration-nats from the vector project folder.

Unfortunately the integration tests weren't set to run in CI for this PR. I've just enabled it so we should now see the results on each push.

[jszwedko]

Didn't mean to close this.

[paolobarbolini]

Since this PR hasn't received any updates I'll try continuing it in #18165

[jszwedko]

Since this PR hasn't received any updates I'll try continuing it in #18165

That would be great, thank you!