-
Notifications
You must be signed in to change notification settings - Fork 15
Producing and consuming messages tw review #197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 8 commits
1b3306c
696c701
e5f5470
9650fe4
9df5d82
4cfb611
a60db6e
71096ea
f7e9c0a
d6421eb
fab3b9d
d27cb5d
e8a013c
7720640
9a709c4
3310a23
8a6e6f8
c6f562f
df0602e
b57043d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,22 +1,33 @@ | ||
| Consumers should inherit from the **ApplicationConsumer**. You need to define a ```#consume``` method that will execute your business logic code against a batch of messages. | ||
| Karafka framework has a long-running server process responsible for fetching and consuming messages. Consumers should inherit from the **ApplicationConsumer**. You need to define a ```#consume``` method that will execute your business logic code against a batch of messages. Karafka fetches and consumes messages in batches by default. | ||
|
|
||
| Karafka fetches and consumes messages in batches by default. | ||
| ### Consuming Messages in Batches | ||
|
||
|
|
||
| ## Consuming Messages | ||
|
|
||
| Karafka framework has a long-running server process responsible for fetching and consuming messages. | ||
| Data fetched from Kafka is accessible using the `#messages` method. The returned object is an enumerable containing received data and additional information that can be useful during the processing. | ||
|
|
||
| To start the Karafka server process, use the following CLI command: | ||
| 1. To start the Karafka server process, use the following CLI command: | ||
|
|
||
| ```shell | ||
| bundle exec karafka server | ||
| ``` | ||
| 2. To access the message batch, use the `#messages` method: | ||
| ```ruby | ||
|
|
||
| class EventsConsumer < ApplicationConsumer | ||
| def consume | ||
| # Access the batch via messages method | ||
| batch = messages | ||
| end | ||
| end | ||
| ``` | ||
|
|
||
| ### In Batches | ||
| 3. Select one of two processing approaches based on your use case: | ||
|
|
||
| Data fetched from Kafka is accessible using the `#messages` method. The returned object is an enumerable containing received data and additional information that can be useful during the processing. | ||
| - Process each message one by one | ||
| - Process all payloads together to leverage batch database operations provided by many ORMs | ||
|
|
||
| 4. Access message payloads. | ||
|
|
||
| To access the payload of your messages, you can use the `#payload` method available for each received message: | ||
| For individual message iteration, use the `#payload` method available for each received message: | ||
|
|
||
| ```ruby | ||
| class EventsConsumer < ApplicationConsumer | ||
|
|
@@ -28,8 +39,8 @@ class EventsConsumer < ApplicationConsumer | |
| end | ||
| end | ||
| ``` | ||
|
|
||
| You can also access all the payloads together to elevate things like batch DB operations available for some of the ORMs: | ||
| For bulk operations, use the `#payloads` method to access all payloads at once: | ||
|
|
||
| ```ruby | ||
| class EventsConsumer < ApplicationConsumer | ||
|
|
@@ -40,11 +51,16 @@ class EventsConsumer < ApplicationConsumer | |
| end | ||
| ``` | ||
|
|
||
| ### One At a Time | ||
| ### Consuming Messages One At a Time | ||
|
|
||
| While we encourage you to process data in batches to elevate in-memory computation and many DBs batch APIs, you may want to process messages one at a time. | ||
| While batch processing is recommended to leverage in-memory computation and batch database operations provided by many ORMs, you may need to process messages individually for certain use cases. | ||
|
|
||
| You can achieve this by defining a base consumer with such a capability: | ||
| 1. To start the Karafka server process, use the following CLI command: | ||
|
|
||
| ```shell | ||
| bundle exec karafka server | ||
| ``` | ||
| 2. Define a reusable base consumer that handles the single-message iteration pattern: | ||
|
|
||
| ```ruby | ||
| class SingleMessageBaseConsumer < Karafka::BaseConsumer | ||
|
|
@@ -66,10 +82,17 @@ class Consumer < SingleMessageBaseConsumer | |
| end | ||
| end | ||
| ``` | ||
| **Result:** The `#consume_one` method will be called for each message in the batch, allowing you to process messages individually while maintaining the benefits of Karafka's batch fetching. | ||
|
|
||
| ### Accessing Topic Details | ||
|
||
|
|
||
| If, in any case, your logic is dependent on some routing details, you can access them from the consumer using the ```#topic``` method. You could use it, for example, in case you want to perform a different logic within a single consumer based on the topic from which your messages come: | ||
| If your logic depends on specific routing details, you can access them from the consumer, using the ```#topic``` method. | ||
|
|
||
| !!! example Use Case | ||
sabinaptas marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| You could use it, for example, when you want to perform a different logic within a single consumer based on the topic from which your messages come. | ||
|
|
||
| 1. To access the topic details, call the ```#topic``` method within the consume method: | ||
sabinaptas marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ```ruby | ||
| class UsersConsumer < ApplicationConsumer | ||
|
|
@@ -87,7 +110,7 @@ class UsersConsumer < ApplicationConsumer | |
| end | ||
| ``` | ||
|
|
||
| If you're interested in all the details that are stored in the topic, you can extract all of them at once, by using the ```#to_h``` method: | ||
| 2. To extract all the details that are stored in the topic at once, use the ```#to_h``` method: | ||
|
|
||
| ```ruby | ||
| class UsersConsumer < ApplicationConsumer | ||
|
|
@@ -97,9 +120,16 @@ class UsersConsumer < ApplicationConsumer | |
| end | ||
| ``` | ||
|
|
||
| ## Consuming From Earliest or Latest Offset | ||
| ## Setting Initial Offset Position | ||
|
|
||
| By default, Karafka starts consuming messages from the earliest available offset. Use this procedure to configure the initial offset position for your consumers. | ||
|
|
||
| Karafka, by default, will start consuming messages from the earliest it can reach. You can, however configure it to start consuming from the latest message by setting the `initial_offset` value as a default: | ||
| **To configure the initial offset globally:** | ||
|
|
||
| 1. Open your Karafka application configuration file. | ||
| 2. Set the `initial_offset` value in the setup block. | ||
|
|
||
| To start from the earliest offset (default behavior): | ||
|
|
||
| ```ruby | ||
| # This will start from the earliest (default) | ||
|
|
@@ -108,16 +138,23 @@ class KarafkaApp < Karafka::App | |
| config.initial_offset = 'earliest' | ||
| end | ||
| end | ||
| ``` | ||
|
|
||
| To start from the latest offset: | ||
| ```ruby | ||
| # This will make Karafka start consuming from the latest message on a given topic | ||
| class KarafkaApp < Karafka::App | ||
| setup do |config| | ||
| config.initial_offset = 'latest' | ||
| end | ||
| end | ||
| ``` | ||
| **Result:** All topics will use this offset position as the default. | ||
|
|
||
| **To configure the initial offset for specific topics:** | ||
|
|
||
| or on a per-topic basis: | ||
| 1. Open your Karafka routing configuration. | ||
| 2. Add the `initial_offset` setting to individual topic definitions: | ||
|
|
||
| ```ruby | ||
| class KarafkaApp < Karafka::App | ||
|
|
@@ -136,6 +173,7 @@ class KarafkaApp < Karafka::App | |
| end | ||
| end | ||
| ``` | ||
| **Result:** Each topic will use its configured offset position, overriding the global default. | ||
|
|
||
| !!! note | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # Best Practices | ||
|
|
||
| ## Documents structure and sources | ||
|
|
||
| 1. H1 and H2 headers in th TOC (Wiki) are rendered from the home.md file. Edit them directly in the home.md file. Renaming the titles in the navigation pane in the VS Code editor does nothing. | ||
sabinaptas marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| 2. Test step | ||
Uh oh!
There was an error while loading. Please reload this page.