Support gzipped user data

[tjkirch]

Issue number:

Fixes #1320. The goal is adding an easy way to handle larger amounts of user data, now that we support adding static pods, host container user data, and other settings that could potentially be long. (This isn't the only solution or the final one, but it's an easy use case.)

Description of changes:

778760a7 early-boot-config: add module to handle maybe-gzip inputs
7a558d9c early-boot-config: allow gzip compression of user data

Note: this is based on #1365 because I needed the model fix to properly run the vmware-dev variant.

Testing done:

aws-k8s-1.17

Started an instance with aws-cli's --user-data fileb://path argument pointing to a gzipped file with >16KiB of text uncompressed, saw those settings take effect (very long motd), and the instance was healthy.

Started the same AMI with a (smaller) uncompressed user data file and was still successful/healthy.

aws-dev

Used an uncompressed local user data file OK:

$ sudo sheltie
bash-5.0# mkdir /etc/early-boot-config
bash-5.0# echo -e '[settings]\nmotd = "woo"' > /etc/early-boot-config/user-data
bash-5.0# early-boot-config
bash-5.0# settings-committer
bash-5.0# thar-be-settings --all
bash-5.0# cat /etc/motd
woo

Then I did the same on a new instance, but outside of sheltie (so gzip is available) with echo -e '[settings]\nmotd = "compressed"' | gzip | sudo tee user-data. Still saw the motd update OK.

vmware-dev

I ran vmware-dev in qemu, and tested changing the motd with a plain uncompressed TOML file in a ISO user data; worked fine.

I tested with a compressed user-data file OK, confirming that it's compressed at /media/cdrom/user-data but uncompressed OK into settings.motd and the motd file.

Then I repeated the tests with an OVF file inside the ISO. Plain text was OK, I confirmed /media/cdrom/ovf_env.xml has the plain text base64, and the API/motd shows the text. Compressed was OK, I confirmed /media/cdrom/ovf_env.xml has the compressed text base64, and the API/motd shows uncompressed text.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[tjkirch]

^ This push clarifies the comment about file magic, and explicitly changes the gzip const length to 2, to represent the fact that it may differ in the future.

[tjkirch]

^ @bcressey gave the nice suggestion of a unit test that confirms giving part of the magic string works, in that it just gives back the input, it's not falsely recognized as gzip or anything. Added that in this push; test passes.

[tjkirch]

^ This push moves the main type to a struct, per @webern.

[zmrow]

🔬