Skip to content
This repository has been archived by the owner on Jul 20, 2019. It is now read-only.

dsully/pykafka

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

28 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

pykafka

pykafka allows you to produce messages to the Kafka distributed publish/subscribe messaging service.

Requirements

You need to have access to your Kafka instance and be able to connect through TCP. You can obtain a copy and instructions on how to setup kafka at https://github.com/kafka-dev/kafka

Installation

pip install pykafka

Usage

Sending a simple message

import kafka

producer = kafka.producer.Producer('test')
message  = kafka.message.Message("Foo!")
producer.send(message)

Sending a sequence of messages

import kafka

producer = kafka.producer.Producer('test')
message1 = kafka.message.Message("Foo!")
message2 = kafka.message.Message("Bar!")
producer.send([message1, message2])

Batching a bunch of messages using a context manager.

import kafka
producer = kafka.producer.Producer('test')

with producer.batch() as messages:
  print "Batching a send of multiple messages.."
  messages.append(kafka.message.Message("first message to send")
  messages.append(kafka.message.Message("second message to send")
  • they will be sent all at once, after the context manager execution.

Consuming messages one by one

import kafka
consumer = kafka.consumer.Consumer('test')
messages = consumer.consume()

Consuming messages using a generator loop

import kafka

consumer = kafka.consumer.Consumer('test')

for message in consumer.loop():
  print message

Contact:

Please use the GitHub issues: https://github.com/dsully/pykafka/issues

About

Python API for the Kafka Message Queue

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages