Add command for dumping an Elasticsearch instance in the PaaS #188
Conversation
| sort: string[]; | ||
| }; | ||
|
|
||
| class PaasInit extends PaasKommand { |
There was a problem hiding this comment.
Yup, that's a copy-paste error! 😅
| ); | ||
|
|
||
| // Create the dump directory | ||
| await fs.mkdir(this.args.dumpDirectory, { recursive: true }); |
There was a problem hiding this comment.
does option { recursive true } works as mkdirp ?
There was a problem hiding this comment.
Yes, it works the same as mkdir -p
|
Why using multiple files instead of a single file in JSONL format? I see some potentials problems with the actual design:
Actually most of the issues you will encounter here were already discussed and solved for the index/collection export commands. You can have a look how it was done |
|
|
||
| // Finish the dump | ||
| try { | ||
| await this.finishDump(pitId); |
There was a problem hiding this comment.
What about the finishDump method if the dump crash in between two pages of documents, should it be called?
There was a problem hiding this comment.
Good catch, yes it should totally be called.
I didn't know about that file format, I will look at that 😄
You're right, I think I just forgot to make this limit configurable 😅
👍 |
|
🎉 This PR is included in version 1.0.0-beta.1 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
|
🎉 This PR is included in version 1.0.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
What does this PR do?
This adds the
paas:elasticsearch:dumpcommand that allows PaaS clients to easily dump the indexes and documents from an Elasticsearch instance of a PaaS application. This PR follows the addition of theapplication/storagecontroller in the PaaS console.This command asks for the PaaS environment name and application ID (a project name can also be specified), and a directory where to dump the Elasticsearch data.
The directory structure is automatically created.
First, the indexes are dumped using the
application/storage:getIndexesaction, and are stored in the<dump_dir>/indexes.jsonfile.Then, the documents are dumped usind the
application/storage:dumpDocumentsaction, and are stored in the<dump_dir>/documents.jsonlfile. This file is a Newline delimited JSON file.Because there could be a huge amount of documents totaling a big size for the response, the PaaS console only allows to dump at most a certain amount of documents at a time. Kourou makes multiple requests to the PaaS console until there are no more documents to dump to get them all. The amount of documents dumped can be set with the
batch-sizeflag, but keep in mind that the PaaS console may also limit the maximum amount of documents that can be returned at once.Finally, the dump session is cleanly terminated and the user is informed where to find their dump files.
Other changes
Types for Node.js were updated to Node 18.