on_kubernetes

Getting Started with PolarStreams on Kubernetes

You can install PolarStreams on Kubernetes using kubectl by using our kustomize base. You can follow our step by step guide to install on K8s.

After deploying PolarStreams on your Kubernetes cluster, you can start producing and consuming messages using a client library or directly invoking the PolarStreams's REST API. The following code snippets assume that the polar StatefulSet is deployed in a namespace with the name streams.

For the purpose of this guide, let's create a pod to produce and consume messages from PolarStreams that will get deleted after we exit the shell.

kubectl run sample-polar-client -it --image=alpine/curl --rm -- sh

From the sample pod in your K8s cluster, you can send a POST request to the REST API to produce a message:

curl -X POST -i -d '{"hello":"world"}' \
    -H "Content-Type: application/json" \
    "http://polar.streams:9251/v1/topic/my-topic/messages"

The PolarStreams service will route to a broker in a round-robin way when no partition key is specified. If you want to specify the partition key you can set it in the querystring, for example: http://polar.streams:9251/v1/topic/my-topic/messages?partitionKey=my-key

Consuming messages is also supported via client libraries and using the REST API. Consuming requires a certain request flow to support stateless HTTP clients and still provide ordering and delivery guarantees.

First, register a consumer in the cluster by setting your consumer identifier, the consumer group and the topics to subscribed to:

curl -X PUT \
    "http://polar.streams:9252/v1/consumer/register?consumerId=1&group=my-app&topic=my-topic"

Note that consumerId and group parameter values can be chosen freely by you, you only have to make sure those values are uniform across the different instances of your application. In most cases the application name is a good choice for consumer group name and the application instance id or a random uuid are suited for the consumerId value.

After registering, you can start polling from the brokers:

curl -i -X POST -H "Accept: application/json" \
    "http://polar.streams:9252/v1/consumer/poll?consumerId=1"

You can continue polling the brokers multiple times to consume data.

PolarStreams internally tracks the reader position of each consumer group (offset) in relationship to the topic and partition. The broker will automatically commit the previously read data when a new poll request is made from the same consumer.

If at any point in time you want to manually save the reader offset without having to retrieve more data, you can optionally send a commit request:

curl -i -X POST "http://polar.streams:9252/v1/consumer/commit?consumerId=1"

Once you finished consuming records from a topic, you can optionally send a goodbye request noting that the consumer instance will not continue reading. The broker will also attempt to manually commit the offset.

curl -X POST "http://polar.streams:9252/v1/consumer/goodbye?consumerId=1"

If a consumer does not send requests over a span of 2 minutes to a broker, the brokers will consider the consumer instance as inactive and it will not be included in the data assignment.

Read more about the API flow and guarantees on the REST API Documentation.