Provide a way to forward streams of data to meilisearch

[irevoire]

Hemlo,

I wanted to use the meilisearch-rust SDK this weekend and found out I was completely stuck because we don't provide any way to send documents that were not generated in our language.
For example, if I have a pipeline generating CSV file that I need to be sent to meilisearch, with the rust SDK, I need to parse the CSV myself to convert it to json and then send it meilisearch which is bad.

I made a PR on meilisearch-rust that adds this possibility, but I was wondering if it's the kind of thing we should provide to all SDKs; what do you think?

imo, we should at least provide a method to add+replace documents and one to add+update documents.
Both methods should take a stream of data with as few constraints on it as possible (the data could come from a file, a web request or anything), and the caller should specify the content type since the callee is not going to look into your data at all.
meilisearch/meilisearch-rust#417

[brunoocasali]

Hi @irevoire, thanks for your issue!

I am checking to see if I understood it correctly:

The idea is to have a function where we can receive a stream instead of having an already parsed/loaded in-memory string? Thinking about a use case could be:

File.foreach("log/production.log") { |line| client.index('logs').add_documents_from_stream(line) }

I would like to understand more possible real-world use cases for this feature. Please provide me with some examples. It will help me translate the importance of it to our other libraries.

[irevoire]

Hey @brunoocasali,

Right now, I am trying to integrate the meilisearch-rust SDK in mieli, and thus my use case was: people sending you anything; some data may be json/jsonlines/csv or even gzipped.

But in general, what I'm thinking about is any use-case where you have files to send to meilisearch instead of deserialized data.

The second thing is for anyone who wants to « hack » something and bypass the « official SDKs functions ».
ie currently, we don't support the gzipped format, if I want to send a huge batch of data to meilisearch I just have to give up entirely on the SDK.

[brunoocasali]

Hi @irevoire I'll submit this comment to the product team, so they prioritize the needs of this feature :)