Can't install / update gh due to expired GPG key?

[andyfeller]

• What's going to happen?
• What are we doing about it?
• What do you need to do about it?
  • Installed via apt
    • Docker build failing?
  • Installed via dnf
  • Obtained from a keyserver
  • Obtained using apt-key
• What are we going to do next?
• How did this happen?
• Final notes

What's going to happen?

On Friday 6th September at approximately 11:17 am UTC, the GPG key used to verify our .deb and .rpm package repository contents will expire.

This will impact apt update and dnf install / update usage, which will look something like the following:

W: GPG error: https://cli.github.com/packages stable InRelease: The following signatures were invalid: EXPKEYSIG 23F3D4EA75716059 GitHub CLI <[email protected]>

or

error: Verifying a signature using certificate 2C6106201985B60E6C7AC87323F3D4EA75716059 (GitHub CLI <[email protected]>):
  1. Certificate 23F3D4EA75716059 invalid: certificate is not alive
      because: The primary key is not live
      because: Expired on 2024-09-06T11:17:19Z
  2. Key 23F3D4EA75716059 invalid: key is not alive
      because: The primary key is not live
      because: Expired on 2024-09-06T11:17:19Z

If you have not previously installed the GitHub CLI via our package repositories, there should be no impact for you.

What are we doing about it?

We have extended the expiration dates on our key, which is available at https://cli.github.com/packages/githubcli-archive-keyring.gpg and on the following keyservers with the 2C6106201985B60E6C7AC87323F3D4EA75716059 ID:

1. keys.openpgp.org
2. keyserver.ubuntu.com

What do you need to do about it?

Important

This section will be updated as more information becomes available.

You will need to get this new key from one of the sources mentioned ☝️ above, which be achieved in multiple ways depending on your setup.

Installed via apt

If you followed our apt instructions, you should be able to run them again. Depending on when you ran these instructions, you may have placed the old keyring in a different location to the current instructions; the script below accounts for this difference by using the file location you previously used.

# Check for wget, if not installed, install it
(type -p wget >/dev/null || (sudo apt update && sudo apt-get install wget -y)) \
    && sudo mkdir -p -m 755 /etc/apt/keyrings

# Set keyring path based on existence of /usr/share/keyrings/githubcli-archive-keyring.gpg
# If it is in the old location, use that, otherwise always use the new location.
if [ -f /usr/share/keyrings/githubcli-archive-keyring.gpg ]; then
    keyring_path="/usr/share/keyrings/githubcli-archive-keyring.gpg"
else
    keyring_path="/etc/apt/keyrings/githubcli-archive-keyring.gpg"
fi

echo "replacing keyring at ${keyring_path}"

# Download and set up the keyring
wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee "$keyring_path" > /dev/null \
    && sudo chmod go+r "$keyring_path"

# Idempotently add the GitHub CLI repository as an apt source
echo "deb [arch=$(dpkg --print-architecture) signed-by=$keyring_path] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null

# Update the package lists, which should now pass
sudo apt update

The important line is:

wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee "$keyring_path" > /dev/null \
    && sudo chmod go+r "$keyring_path"

Which will fetch the new key and overwrite the expired one. From this point you should be able to apt update successfully, and apt upgrade gh successfully (once we have created a new release).

Docker build failing?

If your Docker build is failing as in aws/aws-codebuild-docker-images#739, most likely there is a layer in your image that previously added our package repository, and you have a later layer running apt update.

If you have control of the offending layer, you can re-build it so that it pulls the latest key. If you're searching for the layer, it most likely has a line that looks like wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null.

If you don't have control of the offending layer, you have two options:

You can add a new layer before the apt update that fetches the new key:

RUN wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
    && chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg

Or, if you aren't using gh and it just happens to be in the base image you are using, you can remove the repository so that apt update no longer tries to verify it:

sudo rm /etc/apt/sources.list.d/github-cli.list

Installed via dnf

If you followed our dnf instructions, you will need to remove the expired key from the rpm cache before downloading the new key.

Option 1: You are comfortable reinstalling gh

1. Find and remove the old key:

   sudo rpm -qa gpg-pubkey
   sudo rpm -e pubkey-xxxx-yyyy

   Under most circumstances we expect the key to be named gpg-pubkey-75716059-63172e8a but you can otherwise identify the correct key by checking whether a key's Packager is is "[email protected]" e.g.

   sudo rpm -qi gpg-pubkey-75716059-63172e8a
   Name        : gpg-pubkey
   Version     : 75716059
   Release     : 63172e8a
   Architecture: (none)
   Install Date: Thu 05 Sep 2024 02:39:41 PM EDT
   Group       : Public Keys
   Size        : 0
   License     : pubkey
   Signature   : (none)
   Source RPM  : (none)
   Build Date  : Tue 06 Sep 2022 07:27:06 AM EDT
   Build Host  : localhost
   Packager    : GitHub CLI <[email protected]>
   Summary     : GitHub CLI <[email protected]> public key
   ...
   

2. Removing and reinstalling gh:

   sudo dnf remove gh
   sudo dnf install gh

Option 2: You don't want to reinstall gh

In #6175 (comment), there was another suggested approach to download the keyring, create an .asc file from it, and import that into rpm:

1. Find and remove the old key like Option 1:

   sudo rpm -qa gpg-pubkey
   sudo rpm -e gpg-pubkey-xxxx-yyyy

2. Download new key and import into rpm keyring:

   curl -fsSL -o /var/tmp/githubcli-archive-keyring.gpg https://cli.github.com/packages/githubcli-archive-keyring.gpg
   gpg --keyring /var/tmp/githubcli-archive-keyring.gpg --no-default-keyring --export --armor > /var/tmp/githubcli-archive-keyring.asc
   sudo rpm --import /var/tmp/githubcli-archive-keyring.asc

Potential issue with Fedora

During our testing in Fedora 38, we encountered a sporadic error we believe will be fixed when a new release is created:

  1. Certificate invalid: policy violation
      because: No binding signature at time
  2. Certificate has no valid binding signature as of the signature's creation time, but is valid now.  The certificate has probably been stripped or minimized.

This appears to be due to Fedora 38 enforcing a strict policy about the contents of the public key. Some contents are stripped from the key when extending the expiration. We were only able to reproduce this error using the script above, and not when removing and reinstalling gh.

Again, we believe that this will be resolved when we do a release, and the rpm package repository contents are re-signed with the new, extended expiration private key.

More reading: rpm-software-management/rpm-sequoia#46

Obtained the key from a keyserver

You can fetch the new key by running:

gpg --keyserver keyserver.ubuntu.com --recv-keys 2C6106201985B60E6C7AC87323F3D4EA75716059

Obtained the key using apt-key

You can fetch the new key by running:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 2C6106201985B60E6C7AC87323F3D4EA75716059

What are we going to do next?

1. We want to work with our community to devise smoother package management experience

   We have created Ensure a smoother GPG key update process for September 2026 expiration #9572 to track improvements for how the GitHub CLI keys can be distributed for a smoother, native package management experience.

   If you or someone you know has experience with managing apt and rpm repository keys in OSS, we would love to hear from you on the aforementioned issue.

2. We will address issues raised by the community and GitHub support

   The information here is our best effort to account for the myriad ways that gh is installed, however it isn't definitive.

   If you run into any problems, please follow up here.

3. We are going to ensure future releases with new key work as expected

   The information captured here is to ensure past releases, which were signed by our key are still usable.

   We plan to cut a new release with the new key on Fri Sep 6th to ensure full confidence that future releases work seamlessly.

How did this happen?

Since the last time this key expired, the entire GitHub CLI team has changed, resulting in a loss of institutional knowledge. Unfortunately, the current team was unaware of the timebomb in this part of our release process.

Final Notes

Firstly, thanks to @kdarndt for raising awareness to these concerns in #9562!

Secondly, we want to apologise for the inconvenience this may have caused you. Hopefully through #9572 we can avoid it happening again.

Finally, thank you for your patience and understanding along with any constructive insights you can share.

[rrthomas]

Note that an earlier version of the instructions used the location /usr/share/keyrings instead of /etc/apt/keyrings in the sources.list.d file, so I had to update that to make it work with the above update instructions, and remove the old keyring file from /usr/share/keyrings.

Alternatively, one could of course download the updated key to /usr/share/keyrings, but we don't really want to pollute /usr with non-packaged files!

[andyfeller]

Note that an earlier version of the instructions used the location /usr/share/keyrings instead of /etc/apt/keyrings in the sources.list.d file, so I had to update that to make it work with the above update instructions, and remove the old keyring file from /usr/share/keyrings.

Alternatively, one could of course download the updated key to /usr/share/keyrings, but we don't really want to pollute /usr with non-packaged files!

@rrthomas : Great call out! 🙇 This change originated in #8693 back in early Feb 2024 for anyone wanting to see the older documentation changes and understand the motivations.

UPDATE: The instructions above have been enhanced to account for this older approach 👏

[dcaruso-pathpoint]

NB: if you can't run the apt update to install wget because of the problem we are all trying to solve you can use curl to

curl -sS https://cli.github.com/packages/githubcli-archive-keyring.gpg -o
/etc/apt/keyrings/githubcli-archive-keyring.gpg

[sozelfist]

I encountered an issue with my expired GPG key related to GitHub CLI today. Thankfully, @andyfeller's solutions worked for me.

[leifmadsen]

Slight issue in the Fedora instructions:

Current:

Option 1: You are comfortable reinstalling gh

Find and remove the old key:

rpm -qa gpg-pubkey
rpm -e pubkey-xxxx-yyyy

Suggest:

sudo rpm -e gpg-pubkey-xxxx-yyyy

Also in --> Option 2: You don't want to reinstall gh

[williammartin]

Thanks @leifmadsen, updated.

[leifmadsen]

Thanks @leifmadsen, updated.

Sorry, can you also adjust this?

sudo rpm -e pubkey-xxxx-yyyy

-->

sudo rpm -e gpg-pubkey-xxxx-yyyy

[venkatrahul-software-development]

Thanks for the help @andyfeller and @kdarndt. This worked like magic. I followed "Installed via apt" for my RaspberryPi

[Jmzp]

Thanks for the help @andyfeller

These two blocks solve the problem for me

if [ -f /usr/share/keyrings/githubcli-archive-keyring.gpg ]; then
    keyring_path="/usr/share/keyrings/githubcli-archive-keyring.gpg"
else
    keyring_path="/etc/apt/keyrings/githubcli-archive-keyring.gpg"
fi

# Download and set up the keyring
wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee "$keyring_path" > /dev/null \
    && sudo chmod go+r "$keyring_path"

[pirafrank]

Hi, I have come across this issue and adding up my 2c hoping they will be helpful as a one-stop solution for future users reading this thread.

Solution is a grab-and-go for Debian based distros.

keypath=$(cat /etc/apt/sources.list.d/github-cli.list | grep -oE '/[^]]*' | head -n1)
cat "$keypath"
sudo -E rm -rf "$keypath"
curl -sS https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /tmp/githubcli-archive-keyring.gpg
sudo mv /tmp/githubcli-archive-keyring.gpg /etc/apt/keyrings/githubcli-archive-keyring.gpg
sudo -E sed -i 's,/usr/share/keyrings,/etc/apt/keyrings,g' /etc/apt/sources.list.d/github-cli.list

[williammartin]

@pirafrank that looks good to me. Just making sure I understand correctly, this is equivalent to the script provided in the original issue description except that it will also update the signed-by to the new location (/etc/apt/keyrings)?

[pirafrank]

@williammartin yes, and I think it's a more straightforward solution (minus the wget check). If you got the new keypath it'll do nothing.

[tpalli]

In the dnf steps, option 1 (reinstall) will not work, it will keep complaining even if you remove the key from rpm. I was required to go with option 2 to fetch key and feed it for rpm and after this dnf installation worked. Problem is though that I did this already with last update and now I had to do this again ... will this be happening with each gh update?

[williammartin]

@tpalli thanks for letting us know. Option 1 worked for us during our investigation but it's good to know that there might be issues.

Problem is though that I did this already with last update and now I had to do this again ... will this be happening with each gh update?

I wonder if your problem with Option 1 relates to this. You should only need to do this once. After you get the new key and rpm is using that, things should work consistently. If it happens again, please drop by with logs of what you see and the steps you have taken. Cheers.

[Oaphi]

Can the issue be locked, please? At least temporarily as spammers seem to have got themselves a nice mailing service out of it with a wide reach via GH's email notifications going straight to subscribing users' inbox.

[williammartin]

Yes, sadly there is a lot of spam we have to combat. I'll lock this for the moment. If anyone has anything further to add please add it to our community discussion, or if you are experiencing issues with the instructions, create a new issue. Thanks.