[Documentation] Improve data export / import docs required for upgrade

[marvin-hansen]

Documentation

What you wanted to do

I wanted to update Dgraph, so I have read the manual and tried it out but to no avail.

What you actually did

1. SSH into alpha-0
2. curl http://localhost:8080/admin/export

Dgraph browser is available for running separately using the dgraph-ratel binary

WTF???
Do I have a backup, yes or no?

Why that wasn't great, with examples

1. Why is there no console output at all?

2. In the documentation, what does "Run Dgraph using new data directories." exactly means?

My reading is, I have to SSH into each node, copy the data folders (actually which?) and create empty ones, but is this actually correct?

Why isn't there a simple backup=/path/to/file script that actually makes sense?

What eventually happened:

From the documentation, I couldn't figure out how to do a data export, thus the upgrade process did not happen. This is not good, by any stretch of the imagination.

[marvin-hansen]

It would be great to add a bunch of admin task walk-through videos to the youtube channel because that would resolve this and related issues relatively fast and only requires adding some links to the videos to the docs. I think that's a clear win for everyone and helps people to get things done.

[MichelDiz]

What version are you using? (it is good to follow the template)
If you are using some beta release or something. For a moment, some admin endpoints were removed. And moved to GraphQL (so, in the future you have to use Graphql to do some operations). But, this was undone, as it was decided to deprecate it slowly.

The actual endpoint http://localhost:8080/admin/export returns an output like

{
    "code": "Success",
    "message": "Export completed."
}

In the documentation, what does "Run Dgraph using new data directories." exactly means?

Can you share the link?

Why isn't there a simple backup=/path/to/file script that actually makes sense?

Not sure what you mean, are you talking about a param to set an export path? (there is one). Dgraph exports to /mypath/export/*.

Export is a straightforward operation in Dgraph. Always worked. Maybe you are using the version without it.

Cheers.

[marvin-hansen]

Dgraph version : v2.0.0-rc1
Dgraph SHA-256 : 447ca408b0e56702568f25ceab89eb712090f5f19f06752b70b9a28ca1260cd6
Commit SHA-1 : a4af81a
Commit timestamp : 2020-02-14 17:05:06 -0800
Branch : HEAD

Same today.
curl http://localhost:8080/admin/export

Dgraph browser is available for running separately using the dgraph-ratel binary

moved to GraphQL (so, in the future you have to use Graphql to do some operations).

I guess I got one of those GraphQL mods. What is the backup endpoint for these?

Thanks

[sleto-it]

Hi @marvin-hansen, thanks for opening this ticket

I agree the documentation on the upgrade process needs improvement

We are committed to improve this and other parts of the documentation and improve, day after day, the experience of our users and customers

Many thanks,

[marvin-hansen]

@sleto-it

Thank you. I have to say, the 2.x version made substantial progress when compared to the 1.x version. Since I'm on the 2-rc1 version things work pretty well most of the time. I have vented my frustration with the 1.x a few times before, but I openly admit things are way better now and, especially the GraphQL documentation made big progress and things just work.

Just a simple question, would it be possible to use Helm update for "minor" updates, say from 2.0 to 2.0.1?

For major updates, say from 1.1 to 2.0, the current process is clearly the only viable option for the time being, but there are several minor updates in-between and having interruption-free minor updates is very much necessary and most welcome.

Helm works by pulling a new docker container and then goes round-robin to replace each node with the new version of the container. That only works if the underlying file format isn't changing and the container can shutdown & reboot gracefully.

Obviously, that can only work with minor fixes in minor revisions, but if so, it allows way better interruption-free operations while ensuring bug and security fixes get applied in a timely way.
Also, helm updates can be gracefully rolled back to the previous release in case something unexpected happens and, again, that would dramatically increase confidence in the update process.

I don't know enough about the internals of Dgraph to decide if or how it can be done but from an operational point of view that would give me much more assurance and that's very much welcome.

[MichelDiz]

@marvin-hansen, as you can see in the Changelogs https://github.com/dgraph-io/dgraph/blob/v2.0.0-rc1/CHANGELOG.md#200-rc1---2020-02-14 for the version you are using, those endpoints got removed. And the documentation for this is coming (it is already in master) #4739

The RC release doesn't come with Docs cuz we don't support RC documentations(Also beta and so on). For cases like this, you can always check master, some PR (waiting review) or wait until someone document it. But, usually, we have it documented before doing final releases.

Cheers.

[MichelDiz]

About the question related to Helm. You can use any tag available in Docker Hub. Using the command.

helm install my-release dgraph/dgraph --set image.tag=XXX

I'm not sure if an upgrade would succeed. But I believe it is this command you do.

# helm upgrade [RELEASE] [CHART] [flags]

helm upgrade my-release dgraph/dgraph --set image.tag=XXX

About the file format, it is something that Dgraph team is discussing internally. Discussing a strategy to not have too much-breaking changes consecutively. So that would help with minor upgrades.

[marvin-hansen]

@MichelDiz I tried a Helm upgrade yesterday, and it destroyed my dev installation completely...

Just saying, this needs serious work and a working strategy...

If I understand you correctly, the new way of doing admin tasks is sending an admin mutation to the GraphQL endpoint?

For data export, that would be:

mutation {
export(input: {format: "json"}) {
response {
code
message
}
}
}

Is there a dedicated GraphQL Admin endpoint that I cannot find the documentation?

For security reasons, I do not want to expose a "shutdown" or data export mutation to a client-facing API endpoint, so I would argue that separating admin from data remains paramount.

For all practicality, do I send the above mutation to the normal GraphQL endpoint,
or is there a dedicated admin endpoint?

[MichelDiz]

Is there a dedicated GraphQL Admin endpoint that I cannot find the documentation?

As I said, you won't find any docs for RCs or Beta 100% done. But, for this case, we have that documentation (the endpoint) ready in the Dgraph's graphql website. https://graphql.dgraph.io/docs/admin/ The changes in the main docs will come.

Just use some GraphQL UI of your preference (Like Insonmnia, graphql-playground, GraphiQL and so on) or Curl to /admin endpoint and run the admin mutations you need.

For security reasons, I do not want to expose a "shutdown" or data export mutation to a client-facing API endpoint, so I would argue that separating admin from data remains paramount.

You can always use the Whitelist https://docs.dgraph.io/deploy/#whitelist-admin-operations, put it behind a reverse proxy or something. Actually there's no difference between the new and old way of doing it - just the fact that it is moving from a simple HTTP endpoint to a GraphQL one. I think you have a ticket about it right? (the secure concerns) maybe we could add some admin/passwrd in addition to the existing ones.

But the easiest approach, for now, is protect it not exposing it directly. Or Whitelist.

For all practicality, do I send the above mutation to the normal GraphQL endpoint,
or is there a dedicated admin endpoint?

Send admin mutations to => http://localhost:8080/admin

Cheers.

[marvin-hansen]

Dang, Stupid me, I actually read the new GraphQL admin docs two days ago. Thanks, that's solved. http://skanakbangsa.com/about-us/

…

On Wed, Feb 26, 2020, 21:44 Michel Conrado ***@***.***> wrote: Is there a dedicated GraphQL Admin endpoint that I cannot find the documentation? As I said, you won't find any docs for RCs or Beta 100% done. But, for this case, we have that documentation (the endpoint) ready in the Dgraph's graphql website. https://graphql.dgraph.io/docs/admin/ The changes in the main docs will come. Just use some GraphQL UI of your preference (Like Insonmnia, graphql-playground, GraphiQL and so on) or Curl to /admin endpoint and run the admin mutations you need. For security reasons, I do not want to expose a "shutdown" or data export mutation to a client-facing API endpoint, so I would argue that separating admin from data remains paramount. You can always use the Whitelist https://docs.dgraph.io/deploy/#whitelist-admin-operations, put it behind a reverse proxy or something. Actually there's no difference between the new and old way of doing it - just the fact that it is moving from a simple HTTP endpoint to a GraphQL one. I think you have a ticket about it right? (the secure concerns) maybe we could add some admin/passwrd in addition to the existing ones. But the easiest approach, for now, is protect it not exposing it directly. Or Whitelist. For all practicality, do I send the above mutation to the normal GraphQL endpoint, or is there a dedicated admin endpoint? Send admin mutations to => http://localhost:8080/admin Cheers. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4839>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFYR7XHRDBI5H3IV333KJ4LRE4SHFANCNFSM4KZ7LIMA> .

[MichelDiz]

Okay, so the issue is done and clarified.

Closing,

Cheers.

PS. If any other topic needs further discussion. It needs its own ticket.

[sleto-it]

In the documentation, what does "Run Dgraph using new data directories." exactly means?

Some improvements to the upgrade section have been pushed, and are available at https://docs.dgraph.io/master/deploy/#upgrade-database

Thanks again for all the feedback. We will keep working to improve the documentation (including the admin tasks)

[marvin-hansen]

Thanks for the updated documentation!

…

On Sat, Feb 29, 2020, 14:08 sleto-it ***@***.***> wrote: Closed #4839 <#4839>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4839>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFYR7XEUFPD5VWKMSWC7FWLRFFOLDANCNFSM4KZ7LIMA> .