Provide ability to only read "data", rather than wrapped data.

[philsttr]

I'd like to be able to pull secrets out of vault in the same structure in which they are put in the vault (without a wrapper) i.e. I want to retrieve only the data for a particular path.

Currently, if I do this:

vault write secret/foo - <<EOF
{
     "user": "hello",
     "password": "world"
}
EOF

And then attempt to read the value back:

$ vault read -format=json secret/foo
{
        "lease_id": "",
        "lease_duration": 2592000,
        "renewable": false,
        "data": {
                "password": "world",
                "user": "hello"
        },
        "warnings": null
}

Notice that the data is wrapped in another json object.

I can retrieve individual fields within data by using the -field option (e.g. -field=user). But there's no way (that I have found) to retrieve only the data. i.e. I only want to retrieve

{
     "user": "hello",
     "password": "world"
}

I tried -field=data, but that results in Field data not present in secret, which makes sense, because data is not part of the actual secret. It's just part of the metadata.

The same request applies to the HTTP API as well.

It seems like this could be implemented as a new cli option and HTTP param.

I can see this being beneficial in two use cases

1. When apps don't care about metadata, and only care about the actual secret data
2. Until patch support is added (if it is added), updating a path that has many key/values can be performed with the following (without having to manually unwrap the contents at step 2
   • vault read > file,
   • modify file
   • vault write < file

[jefferai]

jq is your friend here... "jq '.data'" should do the trick!

[philsttr]

Yeah, I looked at jq, but that seems like a kludge. I'd rather not introduce another dependency.

This seems like a natural fit for first class support from vault, especially since vault already supports drilling into data with -field)

[philsttr]

Also, I'd love to read data in yaml format. I realize there's something like jq for yaml, but now, if I want some paths in json format, and other paths in yaml format (perhaps because they are read by different apps/tools), I have to introduce two dependencies.

[jefferai]

You can have the output printed as yaml via the -format=yaml option.

[philsttr]

Yeah, I'm aware of -format=yaml. The problem is that I don't want the vault metadata included in the output.

For example, I would only want the following output:

user: hello
password: world

(without lease_id and all the other vault metadata)

[bradwitte]

Also ran into this. We used to do:

vault read -field=data -format=json secret/foo
{
     "user": "hello",
     "password": "world"
}

No longer works 😢

[jefferai]

@bradwitte I can tell you with certainty that the behavior you're describing hasn't been the case for at least two years. :-)

[bradwitte]

Ha! I don't operate our vault installation so that is a bit out of my control. In any case, I worked around it by placing the secret in a field called "data". That way I didn't have to change the downstream automation.

vault write secret/foo [email protected]

Hope that helps someone.

[jefferai]

@bradwitte If you are actually running Vault prior to 0.5 (which is as far back as I tested) you definitely need to upgrade. We've had huge numbers of bug fixes, including a number of very important ones, plus security fixes since then.

[bradwitte]

Very odd - we definitely are not on a version that old. Being discrete about the field works fine, it just had me scratching my head for a bit. Thanks for looking into it.

[bradwitte]

Actually, looks like my workaround doesn't work:

vault read -format=json secret/foo
{
  "request_id": "b0fa5a39-1052-8773-f380-2b98d2b7d18a",
  "lease_id": "",
  "lease_duration": 2764800,
  "renewable": false,
  "data": {
    "data": "{\n     \"user\": \"hello\",\n     \"password\": \"world\"\n}\n"
  },
  "warnings": null
}

Note the nested "data". It is happily cramming the json object into that single field. Which is what I asked it to do. ;)

So I guess I need to add jq to my automation. Bummer.

[jefferai]

It's not clear to me what you were doing before. I wasn't able to reproduce with any version of the client.

[bradwitte]

Unfortunately I can't verify the versions since I have upgraded my client and haven't tracked what I was running or what the server was running in the past. I guess put me down as a 👍 for this request if it is ever reconsidered. Thanks again!

[elasticdog]

I've just hit this issue trying to store a json file that I'd like to get out in the exact same fashion. I also expected -format=json -field=data to dump things as mentioned above without the metadata, but that doesn't work. The workaround I'm doing is unfortunately duplicating the data in a raw named field:

vault write secret/example [email protected] @my-file.json

...and then reading out the original via:

vault read -field=raw secret/example

[jefferai]

So I think the reason I've been so confused is that, as far as I can tell (by going to a pre-0.9.2) version, this has never worked. So this isn't a bug report -- I thought it was detailing a functionality regression -- it's a feature request.

Tracking in #3987

[bradwitte]

Unless someone else can provide versions in which it did work that sounds fair to me.

[elasticdog]

Thank you @jefferai for the super quick response and resolution to support the special case! 😍

[jefferai]

NP!