Skip to content

[Event Hubs] Implementation of partition manage for durable store for checkpoints using blob storage#4996

Closed
ShivangiReja wants to merge 14 commits intoAzure:masterfrom
ShivangiReja:blob-partition-manager
Closed

[Event Hubs] Implementation of partition manage for durable store for checkpoints using blob storage#4996
ShivangiReja wants to merge 14 commits intoAzure:masterfrom
ShivangiReja:blob-partition-manager

Conversation

@ShivangiReja
Copy link
Member

This pr creates a separate plugin and implement partitionManager for durable store for checkpoints using Blob Storage.

TODO:

  • Update README
  • Add Sample
  • Update docs

@ShivangiReja ShivangiReja self-assigned this Sep 5, 2019
@ShivangiReja ShivangiReja added Client This issue points to a problem in the data-plane of the library. Event Hubs labels Sep 5, 2019
@chradek
Copy link
Contributor

chradek commented Sep 5, 2019

I wanted to make a couple high-level comments.

We aren't storing any checkpoint data in the body of a blob, we're storing it all as metadata. When you make a listBlobsFlat call, you can specify that you want the metadata returned on each blob as well:

await containerClient.listBlobsFlat({include: ['metadata']});

This lets us get the metadata for each blob without having to make individual calls to the Storage service for each blob.

Since we're storing the blobs with a known prefix, we can also include that to the listBlobsFlat call so we only get blobs that start with our prefix:

await containerClient.listBlobsFlat({
  include: ['metadata'],
  prefix: `${eventHubName}/${consumerGroupName}/`}
);

@ShivangiReja
Copy link
Member Author

Few things need to be updated, will create another PR for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Client This issue points to a problem in the data-plane of the library. Event Hubs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants