Add support for age. #688

jimmycuadra · 2020-07-04T21:15:53Z

Implements support for age as I requested in #603. I'm not a Go programmer and was learning the sops code as I went, so there's likely a lot I missed here. But the basic encrypt and decrypt operations are working for me! I used #569 as a reference for how to do this, as suggested by @jvehent.

Example (this has been simplified in more recent commits. See #688 (comment)):

$ mkdir -p ~/.sops/age/
$ age-keygen | tee age.key
Public key: age1hhpw5e07syc00jwtwxs64e6nqv9exhmmjrlztcxkw9vgnjqev47qllp3ry
# created: 2020-07-04T13:55:01-07:00
# public key: age1hhpw5e07syc00jwtwxs64e6nqv9exhmmjrlztcxkw9vgnjqev47qllp3ry
AGE-SECRET-KEY-1ZWFCS8JEND5M8T7VYRW8VVWF2K7MLS6LQWDDG53NYHTJHNHTCYVQQTK7EG
$ mv age.key ~/.sops/age/age1hhpw5e07syc00jwtwxs64e6nqv9exhmmjrlztcxkw9vgnjqev47qllp3ry.key
$ echo 'foo: bar' > secret.yaml
$ sops --age age1hhpw5e07syc00jwtwxs64e6nqv9exhmmjrlztcxkw9vgnjqev47qllp3ry -e secret.yaml > secret.enc.yaml
$ sops -d secret.enc.yaml
foo: bar

Things that I know are missing:

~~Tests~~
~~Documentation~~
Support for scrypt keys which would allow for password-derived keys (only X25519 is implemented right now)
Support for SSH keys (this is likely bigger than I'm interested in working on myself)
~~Ability to customize where sops looks for private key material.~~
A subcommand to generate a new key and place it in ~/.sops/age, which would remove the need for the manual steps I took in the example above.

Which of these things would be necessary to get basic support merged? I can't guarantee a lot of time to work on this so I'm hoping to scope it down as small as possible. Alternatively, if someone wants to use my branch as a starting point and take over, that'd be great.

codecov-commenter · 2020-07-04T21:21:53Z

Codecov Report

Merging #688 into develop will decrease coverage by 3.22%.
The diff coverage is 10.12%.

@@             Coverage Diff             @@
##           develop     #688      +/-   ##
===========================================
- Coverage    36.74%   33.51%   -3.23%     
===========================================
  Files           22       23       +1     
  Lines         3217     3076     -141     
===========================================
- Hits          1182     1031     -151     
+ Misses        1916     1913       -3     
- Partials       119      132      +13

Impacted Files	Coverage Δ
config/config.go	`64.39% <0.00%> (-7.04%)`	⬇️
keyservice/keyservice.go	`0.00% <0.00%> (ø)`
keyservice/server.go	`4.06% <0.00%> (-1.21%)`	⬇️
stores/stores.go	`0.00% <0.00%> (ø)`
keyservice/keyservice.pb.go	`6.02% <7.02%> (+1.89%)`	⬆️
age/keysource.go	`34.78% <34.78%> (ø)`
pgp/keysource.go	`30.25% <0.00%> (-3.83%)`	⬇️
gcpkms/keysource.go	`21.87% <0.00%> (-2.81%)`	⬇️
aes/cipher.go	`65.34% <0.00%> (-1.95%)`	⬇️
stores/ini/store.go	`27.52% <0.00%> (-1.80%)`	⬇️
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 334be56...e9acafc. Read the comment docs.

age/keysource.go

keyservice/server.go

Mic92 · 2020-07-06T14:44:49Z

I have not tested but from a basic code review it looks good to me. I am not affiliated with the project but I assume that the maintainers also want to see some unit tests before they merge this.

autrilla · 2020-07-06T17:09:32Z

Thanks for the work :)

Two concerns:

As pointed out already, lack of tests
Key management: from a usability standpoint, this looks like it could be even worse than GPG to me. At least GPG has standardized locations to put the keys in. Right now, you're putting the keys in a sops-specific directory. Sharing keys with other people seems hard. I'm sure we get some security gains from AGE vs PGP, but still.

Mic92 · 2020-07-06T19:22:26Z

Thanks for the work :)

Two concerns:

* As pointed out already, lack of tests

* Key management: from a usability standpoint, this looks like it could be even worse than GPG to me. At least GPG has standardized locations to put the keys in. Right now, you're putting the keys in a sops-specific directory. Sharing keys with other people seems hard. I'm sure we get some security gains from AGE vs PGP, but still.

My main motivation for having this is an alternative for deploying sops on servers in none-cloud environments (no azure, no aws, no GCP) because key generation/provision with GPG is overly complex:

gnupg does fancy things with your tty, has weird exit codes when doing scripting
It has a complex database where as with age one can just deploy a single file
gnupg requires a long scripts with under-documented options that you have to steal from stackoverflow where in age its a oneliner
gnupg always spawns a daemon which has to be killed manually in your $configurationmanagement (chef, puppet, ansible in my case nixos)
gnupg can have a lot of dependencies i.e. gui libraries where as age would be just a library inside sops (No, the embedded pgp is not enough as it breaks with gpg 2.1 databases)

Key sharing is actually something I don't want to do in this case because unlike gpg keys those are machines rather than persons.

autrilla · 2020-07-06T19:47:29Z

I understand AGE has some benefits, and I'm unfortunately painfully aware of the issues with GPG.

Maybe documentation saying "put your AGE keys here" would be enough. I don't know if AGE has a standard location to put keys in. I want to have a better look at AGE before fully reviewing this PR regardless.

Mic92 · 2020-07-06T20:31:41Z

I understand AGE has some benefits, and I'm unfortunately painfully aware of the issues with GPG.

Maybe documentation saying "put your AGE keys here" would be enough. I don't know if AGE has a standard location to put keys in. I want to have a better look at AGE before fully reviewing this PR regardless.

On servers one could even use the ssh host key, which is already present on most Linux servers. Also hosters like github already expose ssh public keys, which can be used: https://github.com/FiloSottile.keys

jimmycuadra · 2020-07-07T04:37:32Z

The location of the private keys I chose is based on a similar pattern in the PR I was referencing, as noted in the description. The setup here is a little odd in that we don't save any sort of path or identifier for the private key in the encrypted file—it has to be derived from the public key somehow.

I suspect that age is intentionally agnostic on key management so that systems that use it can do whatever is most appropriate, but in our case it is unfortunate not to have a convention for where to find private keys that we could leverage. Perhaps @FiloSottile can share if there is any particular guidance on key management.

Also, I don't think age is an acronym, so it doesn't need to be capitalized. 😛

Mic92 · 2020-07-09T07:44:53Z

Seems to be a bit relevant: #692

jimmycuadra · 2020-07-18T11:05:37Z

Made some changes:

Added support for specifying the directory where sops will look for age key files with the SOPS_AGE_KEY_DIR environment variable. If not specified, defaults to what it was before (~/.sops/age/$PUBLIC_KEY.key).
Platform-agnostic path handling for the key directory.
Some basic tests for encrypting and decrypting with an age key. I'm not thrilled with setting the environment variable in the test to point at the test key I created (and assuming what $PWD will be). Again I'm not very familiar with Go idioms so I'll need some advice on how to do this better. Normally I'd have the function accept the key dir as an argument, but the decrypt function is part of an interface I can't change.

Mic92 · 2020-07-18T12:05:41Z

Made some changes:

* Added support for specifying the directory where sops will look for age key files with the `SOPS_AGE_KEY_DIR` environment variable. If not specified, defaults to what it was before (`~/.sops/age/$PUBLIC_KEY.key`).

* Platform-agnostic path handling for the key directory.

* Some basic tests for encrypting and decrypting with an age key. I'm not thrilled with setting the environment variable in the test to point at the test key I created (and assuming what `$PWD` will be). Again I'm not very familiar with Go idioms so I'll need some advice on how to do this better. Normally I'd have the function accept the key dir as an argument, but the decrypt function is part of an interface I can't change.

~~One question is there a good reason not to put ed25519 keys directly into environment variables? I assume they are as long as the actual fingerprints?~~

Nevermind these are private keys...

age/keysource.go

age/keysource_test.go

age/keysource.go

Mic92 · 2020-07-27T06:40:50Z

age/keysource.go

+	for scanner.Scan() {
+		line := scanner.Text()
+
+		if strings.HasPrefix(line, "AGE-SECRET-KEY") {
+			privateKey = line
+			break
+		}
+	}


It would be nice to just copy this function: https://github.com/FiloSottile/age/blob/189041b668629795593766bcb8d3f70ee248b842/cmd/age/parse.go#L36

Then it would not just support the age format but also ssh keys like age does.
Same goes for encrypting.

@FiloSottile How does making the linked code available via age's library sound? Ideally this logic wouldn't be duplicated in sops or it could get out of sync with what age does over time.

That would be the ideal case.

Mic92 · 2020-07-27T06:47:36Z

I think the next thing upstream would ask for is to provide a section in the README. I can also help out here.

jimmycuadra · 2020-07-28T10:34:27Z

Thanks for the review, @Mic92! That was very helpful. I fixed some of the smaller points, but I think it would make sense to see what Filippo says about opening up more functionality through the library before proceeding with the other changes.

Mic92 · 2020-07-30T14:54:04Z

Thanks for the review, @Mic92! That was very helpful. I fixed some of the smaller points, but I think it would make sense to see what Filippo says about opening up more functionality through the library before proceeding with the other changes.

That makes sense. Maybe we can fix this upstream.

FiloSottile · 2020-07-30T22:08:19Z

Hmm good question. The key files format is more part of the age CLI than the API, but maybe it should be exposed. Give me a couple days to think about it.

Mic92 · 2020-07-31T04:37:32Z

If fundamental parts like key location is not part of age api than will see inconsistency in the ecosystem on how age is integrated into applications.

jimmycuadra · 2020-07-31T12:34:54Z

I could see it being a separate lib that sits in between the current lib and the CLI, so that it's possible to use the base library just for encryption and decryption without anything more. The intermediate "storage" library could be consumed by both the CLI and third-party software like sops that wants to use the same conventions for storing keys as the CLI.

FiloSottile · 2020-07-31T13:13:07Z

If fundamental parts like key location is not part of age api than will see inconsistency in the ecosystem on how age is integrated into applications.

I don't know sops enough to tell you about your use case in particular, but I expect most application that use age to keep separate keys. age doesn't have the concept of a "user key" that is used for all applications that use age, and I don't want that kind of overloading. If you rotate your sops key you shouldn't have to worry about your password manager or backups at the same time.

What we might want to expose is the key file format, not the location.

Can I ask why the interest in supporting SSH keys? age supports them so one can encrypt to someone who doesn't have an age key without waiting a round-trip for them to generate it. Here it looks like you could just generate an age key?

Mic92 · 2020-07-31T16:56:14Z

If fundamental parts like key location is not part of age api than will see inconsistency in the ecosystem on how age is integrated into applications.

I don't know sops enough to tell you about your use case in particular, but I expect most application that use age to keep separate keys. age doesn't have the concept of a "user key" that is used for all applications that use age, and I don't want that kind of overloading. If you rotate your sops key you shouldn't have to worry about your password manager or backups at the same time.

The current workflow is that developers have a gnupg key that is distributed across the team (i.e. through public key server). On the server side there is some sort of key management service provided by some cloud provider. With age I hope to have a solution that lucks less than gnupg for users and also an alternative for non-cloud servers, which do not have access to an cloud-based vault.

What we might want to expose is the key file format, not the location.

Can I ask why the interest in supporting SSH keys? age supports them so one can encrypt to someone who doesn't have an age key without waiting a round-trip for them to generate it. Here it looks like you could just generate an age key?

If you have configuration shared in a team they are likely to have some sort of version control and hence you have public keys i.e. https://github.com/Mic92.keys This makes further key exchange unnecessary. The same is true for servers where configuration is deployed. Most of them already have ssh private host keys: i.e. /etc/ssh/ssh_host_ed25519_key This ssh key is already trusted and easy to obtain.

ajvb · 2020-07-31T23:08:09Z

If fundamental parts like key location is not part of age api than will see inconsistency in the ecosystem on how age is integrated into applications.

I don't know sops enough to tell you about your use case in particular, but I expect most application that use age to keep separate keys. age doesn't have the concept of a "user key" that is used for all applications that use age, and I don't want that kind of overloading. If you rotate your sops key you shouldn't have to worry about your password manager or backups at the same time.

What we might want to expose is the key file format, not the location.

I think that keeping separate keys for sops is what I would want us to recommend in most use-cases. At the same time, that doesn't mean that there cannot be a set default "age private key directory" where all of your keys are kept, even if it's 1 or more key per application.

jimmycuadra · 2020-08-01T05:56:37Z

age doesn't have the concept of a "user key" that is used for all applications that use age, and I don't want that kind of overloading.

What we might want to expose is the key file format, not the location.

This sounds good to me. Not having a direct equivalent to a PGP key is a good thing about age, and we shouldn't be trying to shoehorn that back in for sops usage. It makes sense to me to have a sops-specific directory to store age keys that are used specifically and only for the purpose of sops. It'd definitely be nice to have a standard key file format, though!

FiloSottile

Thank you all for working on this and for your patience. I added a section to the age docs about key management, which matches what you are doing and recommends not adding support for SSH keys at all.

https://pkg.go.dev/filippo.io/[email protected]#hdr-Key_management

I also added ParseIdentities to make it easier to parse the private key file.

age/keysource.go

Co-authored-by: Jörg Thalheim <[email protected]>

jimmycuadra · 2020-09-22T13:02:29Z

Updated to the latest age beta and made changes suggested by Filippo. Still looking for a sops maintainer with time for final review and hopefully merge!

autrilla · 2020-09-22T14:18:21Z

LGTM. I'll give it a couple of days to see if @ajvb has time to review, otherwise I'll just merge.

ajvb · 2020-09-22T21:05:38Z

🎉 This is really exciting! I should be able to get to a review sometime tomorrow.

ajvb · 2020-09-23T18:42:32Z

Thank you so much @jimmycuadra, @FiloSottile, @Mic92, and @colemickens - very exciting to get this in here.

dol · 2021-03-25T16:37:07Z

@jimmycuadra Thank you for the age integration. I'll will test it soon. ❤️

jimmycuadra mentioned this pull request Jul 4, 2020

age (https://age-encryption.org/) for encryption #603

Closed

Mic92 reviewed Jul 6, 2020

View reviewed changes

age/keysource.go Outdated Show resolved Hide resolved

Mic92 reviewed Jul 6, 2020

View reviewed changes

keyservice/server.go Outdated Show resolved Hide resolved

jvehent mentioned this pull request Jul 9, 2020

Adding support for ssh keys for encryption. #692

Open

ajvb self-requested a review July 15, 2020 18:41

Mic92 reviewed Jul 18, 2020

View reviewed changes

age/keysource.go Outdated Show resolved Hide resolved

Mic92 reviewed Jul 27, 2020

View reviewed changes

age/keysource.go Outdated Show resolved Hide resolved

Mic92 reviewed Jul 27, 2020

View reviewed changes

age/keysource_test.go Outdated Show resolved Hide resolved

Mic92 reviewed Jul 27, 2020

View reviewed changes

age/keysource.go Outdated Show resolved Hide resolved

Mic92 reviewed Jul 27, 2020

View reviewed changes

ajvb mentioned this pull request Sep 1, 2020

gpg.mozilla.org is broken (probably for good) #727

Open

FiloSottile approved these changes Sep 20, 2020

View reviewed changes

age/keysource.go Outdated Show resolved Hide resolved

age/keysource.go Outdated Show resolved Hide resolved

jimmycuadra and others added 16 commits September 21, 2020 12:47

Add support for age.

e8d0004

Use idiomatic style for assignment and error handling.

9ca46d0

Co-authored-by: Jörg Thalheim <[email protected]>

Allow age key dir to be set with SOPS_AGE_KEY_DIR and add tests.

9e4cbc9

Use user config dir instead of home dir as the root for age keys.

2741ab5

Determine age package path using current file rather than pwd.

d9b196c

Don't swallow potential errors from os.Stat.

5c171c8

Use a single keys.txt file for age private keys.

617db43

Try decrypting with all possible keys in the keyfile.

7f7ecbc

Document age usage.

ade5692

Reorder README sections and fix RST link.

a66a0a8

Use more concise style for constructing map.

6a6a936

Fix whitespace errors.

1dbea5d

Update go.mod/go.sum.

6068838

age: .sops.yaml support

50a89c8

age: MasterKeysFromRecipients: gracefully handle empty string

8f6271f

Update to age 1.0.0-beta5.

e9acafc

ajvb approved these changes Sep 23, 2020

View reviewed changes

ajvb merged commit 682bff4 into getsops:develop Sep 23, 2020

jimmycuadra deleted the age branch September 24, 2020 07:54

jtrakk mentioned this pull request Dec 17, 2020

Document age usage Mic92/sops-nix#53

Closed

ajvb mentioned this pull request Mar 24, 2021

v3.7.0 - develop -> master #838

Merged

hiddeco mentioned this pull request Mar 24, 2021

Update SOPS to v3.7.0 fluxcd/kustomize-controller#302

Closed

mrueg mentioned this pull request Nov 24, 2021

Investigate support for age as an alternative for gnupg onedr0p/cluster-template#152

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for age. #688

Add support for age. #688

jimmycuadra commented Jul 4, 2020 •

edited

Loading

codecov-commenter commented Jul 4, 2020 •

edited

Loading

Mic92 commented Jul 6, 2020

autrilla commented Jul 6, 2020

Mic92 commented Jul 6, 2020 •

edited

Loading

autrilla commented Jul 6, 2020

Mic92 commented Jul 6, 2020 •

edited

Loading

jimmycuadra commented Jul 7, 2020

Mic92 commented Jul 9, 2020

jimmycuadra commented Jul 18, 2020 •

edited

Loading

Mic92 commented Jul 18, 2020 •

edited

Loading

Mic92 Jul 27, 2020

jimmycuadra Jul 28, 2020

Mic92 Jul 28, 2020

Mic92 commented Jul 27, 2020

jimmycuadra commented Jul 28, 2020

Mic92 commented Jul 30, 2020 •

edited

Loading

FiloSottile commented Jul 30, 2020

Mic92 commented Jul 31, 2020

jimmycuadra commented Jul 31, 2020

FiloSottile commented Jul 31, 2020

Mic92 commented Jul 31, 2020 •

edited

Loading

ajvb commented Jul 31, 2020

jimmycuadra commented Aug 1, 2020

FiloSottile left a comment

jimmycuadra commented Sep 22, 2020

autrilla commented Sep 22, 2020

ajvb commented Sep 22, 2020

ajvb commented Sep 23, 2020

dol commented Mar 25, 2021

Add support for age. #688

Add support for age. #688

Conversation

jimmycuadra commented Jul 4, 2020 • edited Loading

codecov-commenter commented Jul 4, 2020 • edited Loading

Codecov Report

Mic92 commented Jul 6, 2020

autrilla commented Jul 6, 2020

Mic92 commented Jul 6, 2020 • edited Loading

autrilla commented Jul 6, 2020

Mic92 commented Jul 6, 2020 • edited Loading

jimmycuadra commented Jul 7, 2020

Mic92 commented Jul 9, 2020

jimmycuadra commented Jul 18, 2020 • edited Loading

Mic92 commented Jul 18, 2020 • edited Loading

Mic92 Jul 27, 2020

Choose a reason for hiding this comment

jimmycuadra Jul 28, 2020

Choose a reason for hiding this comment

Mic92 Jul 28, 2020

Choose a reason for hiding this comment

Mic92 commented Jul 27, 2020

jimmycuadra commented Jul 28, 2020

Mic92 commented Jul 30, 2020 • edited Loading

FiloSottile commented Jul 30, 2020

Mic92 commented Jul 31, 2020

jimmycuadra commented Jul 31, 2020

FiloSottile commented Jul 31, 2020

Mic92 commented Jul 31, 2020 • edited Loading

ajvb commented Jul 31, 2020

jimmycuadra commented Aug 1, 2020

FiloSottile left a comment

Choose a reason for hiding this comment

jimmycuadra commented Sep 22, 2020

autrilla commented Sep 22, 2020

ajvb commented Sep 22, 2020

ajvb commented Sep 23, 2020

dol commented Mar 25, 2021

jimmycuadra commented Jul 4, 2020 •

edited

Loading

codecov-commenter commented Jul 4, 2020 •

edited

Loading

Mic92 commented Jul 6, 2020 •

edited

Loading

Mic92 commented Jul 6, 2020 •

edited

Loading

jimmycuadra commented Jul 18, 2020 •

edited

Loading

Mic92 commented Jul 18, 2020 •

edited

Loading

Mic92 commented Jul 30, 2020 •

edited

Loading

Mic92 commented Jul 31, 2020 •

edited

Loading