Skip to content

Commit 2d1bf6b

Browse files
zmoogzmoog
authored
[azure logs] Add processor version options to Raw Events integration (v1) (#15900)
Add the processor version option to the Raw Events integration v1. With these new options, users can switch from processor v1 (current default) to the processor v2. This change anticipate the switch to v2 as default processor.
1 parent 221f106 commit 2d1bf6b

File tree

7 files changed

+155
-6
lines changed

7 files changed

+155
-6
lines changed

‎packages/azure/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
- version: "1.30.0"
2+
changes:
3+
- description: Add processor version options for the Raw Events (v1) integration.
4+
type: enhancement
5+
link: https://github.com/elastic/integrations/pull/15900
16
- version: "1.29.1"
27
changes:
38
- description: Update `time` formats parsing and fix `durationMs` parsing in signinlogs data stream.

‎packages/azure/data_stream/eventhub/agent/stream/stream.yml.hbs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,31 @@ storage_account: {{storage_account}}
1919
{{/if}}
2020
{{#if storage_account_key}}
2121
storage_account_key: {{storage_account_key}}
22+
storage_account_connection_string: DefaultEndpointsProtocol=https;AccountName={{storage_account}};AccountKey={{storage_account_key}};EndpointSuffix={{endpoint_suffix}}
2223
{{/if}}
2324
{{#if resource_manager_endpoint}}
2425
resource_manager_endpoint: {{resource_manager_endpoint}}
2526
{{/if}}
27+
28+
{{#if processor_version}}
29+
processor_version: {{processor_version}}
30+
{{/if}}
31+
{{#if migrate_checkpoint}}
32+
migrate_checkpoint: {{migrate_checkpoint}}
33+
{{/if}}
34+
{{#if processor_update_interval}}
35+
processor_update_interval: {{processor_update_interval}}
36+
{{/if}}
37+
{{#if processor_start_position}}
38+
processor_start_position: {{processor_start_position}}
39+
{{/if}}
40+
{{#if partition_receive_timeout}}
41+
partition_receive_timeout: {{partition_receive_timeout}}
42+
{{/if}}
43+
{{#if partition_receive_count}}
44+
partition_receive_count: {{partition_receive_count}}
45+
{{/if}}
46+
2647
data_stream:
2748
dataset: {{data_stream.dataset}}
2849
tags:

‎packages/azure/data_stream/eventhub/manifest.yml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,104 @@ streams:
7474
type: bool
7575
multi: false
7676
default: false
77+
78+
#
79+
# Processor v2 only settings
80+
#
81+
- name: processor_version
82+
type: select
83+
title: Processor version
84+
multi: false
85+
required: false
86+
show_user: false
87+
default: v1
88+
options:
89+
- text: v1
90+
value: v1
91+
- text: v2
92+
value: v2
93+
description: "The processor version that the integration should use. Possible values are `v1` and `v2` (preview). \nThe v2 event hub processor is in preview, so using the v1 processor is recommended for typical use cases.\nDefault is `v1`."
94+
- name: processor_update_interval
95+
type: text
96+
title: Processor update interval
97+
multi: false
98+
required: false
99+
show_user: false
100+
default: 10s
101+
description: >-
102+
(Processor v2 only) How often the processor should attempt to claim partitions.
103+
104+
Default is `10` seconds.
105+
- name: processor_start_position
106+
type: select
107+
title: Processor start position
108+
multi: false
109+
required: false
110+
show_user: false
111+
default: earliest
112+
options:
113+
- text: earliest
114+
value: earliest
115+
- text: latest
116+
value: latest
117+
description: >-
118+
(Processor v2 only) Controls from what position in the event hub the processor should start processing messages for all partitions.
119+
120+
Possible values are `earliest` and `latest`.
121+
122+
`earliest` starts processing messages from the last checkpoint, or the beginning of the event hub if no checkpoint is available.
123+
124+
`latest` starts processing messages from the the latest event in the event hub and continues to process new events as they arrive.
125+
126+
Default is `earliest`.
127+
- name: partition_receive_timeout
128+
type: text
129+
title: Partition receive timeout
130+
multi: false
131+
required: false
132+
show_user: false
133+
default: 5s
134+
description: >-
135+
(Processor v2 only) Maximum time to wait before processing the messages received from the event hub.
136+
137+
The partition consumer waits up to a "receive count" or a "receive timeout", whichever comes first.
138+
139+
Default is `5` seconds.
140+
- name: partition_receive_count
141+
type: text
142+
title: Partition receive count
143+
multi: false
144+
required: false
145+
show_user: false
146+
default: 100
147+
description: >-
148+
(Processor v2 only) Maximum number of messages from the event hub to wait for before processing them.
149+
150+
The partition consumer waits up to a "receive count" or a "receive timeout", whichever comes first.
151+
152+
Default is `100` messages.
153+
- name: migrate_checkpoint
154+
type: bool
155+
title: Migrate checkpoint information
156+
multi: false
157+
required: false
158+
show_user: false
159+
default: true
160+
description: >-
161+
(Processor v2 only) Flag to control if the processor should perform the checkpoint information migration from processor v1 to v2 at startup.
162+
163+
The checkpoint migration converts the checkpoint information from the v1 format to the v2 format.
164+
165+
Default is `false`, which means the processor will not perform the checkpoint migration.
166+
- name: endpoint_suffix
167+
type: text
168+
default: core.windows.net
169+
required: true
170+
title: Storage account endpoint suffix
171+
show_user: false
172+
description: >-
173+
(Processor v2 only) Override the default storage account endpoint suffix.
174+
77175
# Ensures agents have permissions to write data to `logs-*-*`
78176
elasticsearch:
79177
dynamic_dataset: true

‎packages/azure/data_stream/events/manifest.yml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,12 +81,17 @@ streams:
8181
multi: false
8282
default: false
8383
- name: processor_version
84-
type: text
84+
type: select
8585
title: Processor version
8686
multi: false
8787
required: false
8888
show_user: false
8989
default: v1
90+
options:
91+
- text: v1
92+
value: v1
93+
- text: v2
94+
value: v2
9095
description: "The processor version that the integration should use. Possible values are `v1` and `v2` (preview). \nThe v2 event hub processor is in preview, so using the v1 processor is recommended for typical use cases.\nDefault is `v1`."
9196
- name: processor_update_interval
9297
type: text
@@ -100,12 +105,17 @@ streams:
100105
101106
Default is `10` seconds.
102107
- name: processor_start_position
103-
type: text
108+
type: select
104109
title: Processor start position
105110
multi: false
106111
required: false
107112
show_user: false
108113
default: earliest
114+
options:
115+
- text: earliest
116+
value: earliest
117+
- text: latest
118+
value: latest
109119
description: >-
110120
(Processor v2 only) Controls from what position in the event hub the processor should start processing messages for all partitions.
111121

‎packages/azure/manifest.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
name: azure
22
title: Azure Logs
3-
version: "1.29.1"
3+
version: "1.30.0"
44
description: This Elastic integration collects logs from Azure
55
type: integration
66
icons:

‎packages/azure_logs/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "0.4.2"
3+
changes:
4+
- description: Switch `processor_version` and `processor_start_position` configuration options to `select` type (UI change).
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/15900
27
- version: "0.4.1"
38
changes:
49
- description: Add support to override `Endpoint Suffix` based on deployment environment.

‎packages/azure_logs/manifest.yml‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
format_version: 3.3.0
22
name: azure_logs
33
title: "Custom Azure Logs"
4-
version: "0.4.1"
4+
version: "0.4.2"
55
source:
66
license: Elastic-2.0
77
description: "Collect log events from Azure Event Hubs with Elastic Agent"
@@ -138,12 +138,17 @@ policy_templates:
138138
multi: false
139139
default: false
140140
- name: processor_version
141-
type: text
141+
type: select
142142
title: Processor version
143143
multi: false
144144
required: false
145145
show_user: false
146146
default: v1
147+
options:
148+
- text: v1
149+
value: v1
150+
- text: v2
151+
value: v2
147152
description: "The processor version that the integration should use. Possible values are `v1` and `v2` (preview). \nThe v2 event hub processor is in preview, so using the v1 processor is recommended for typical use cases.\nDefault is `v1`."
148153
- name: processor_update_interval
149154
type: text
@@ -157,12 +162,17 @@ policy_templates:
157162
158163
Default is `10` seconds.
159164
- name: processor_start_position
160-
type: text
165+
type: select
161166
title: Processor start position
162167
multi: false
163168
required: false
164169
show_user: false
165170
default: earliest
171+
options:
172+
- text: earliest
173+
value: earliest
174+
- text: latest
175+
value: latest
166176
description: >-
167177
(Processor v2 only) Controls from what position in the event hub the processor should start processing messages for all partitions.
168178

0 commit comments

Comments
 (0)