Skip to content

OSDOCS-628 updating restricted network installation #16894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kalexand-rh wants to merge 3 commits into from
Closed

OSDOCS-628 updating restricted network installation #16894

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5de3002
OSDOCS-628 updating restricted network installation
kalexand-rh Sep 26, 2019
c7f88e4
updates per Trevor
kalexand-rh Nov 7, 2019
2e5060e
more updates per Trevor
kalexand-rh Nov 8, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions _topic_map.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,8 +181,8 @@ Topics:
File: updating-cluster-cli
- Name: Updating a cluster that includes RHEL compute machines
File: updating-cluster-rhel-compute
#- Name: Updating a disconnected cluster
# File: updating-disconnected-cluster
- Name: Updating a restricted network cluster
File: updating-restricted-network
# - Name: Troubleshooting an update
# File: updating-troubleshooting
---
Expand Down
4 changes: 2 additions & 2 deletions ...installing_restricted_networks/installing-restricted-networks-preparations.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ include::modules/cli-installing-cli.adoc[leveloffset=+2]

include::modules/installation-creating-mirror-registry.adoc[leveloffset=+1]

include::modules/installation-local-registry-pull-secret.adoc[leveloffset=+1]
//include::modules/installation-local-registry-pull-secret.adoc[leveloffset=+1]

//include::modules/installation-adding-registry-pull-secret.adoc[leveloffset=+1]
include::modules/installation-adding-registry-pull-secret.adoc[leveloffset=+1]

include::modules/installation-mirror-repository.adoc[leveloffset=+1]

Expand Down
37 changes: 21 additions & 16 deletions modules/installation-adding-registry-pull-secret.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,30 @@ restricted network.

Complete the following steps on the bastion host:

. Download your `registry.redhat.io` pull secret from the
link:https://cloud.redhat.com/openshift/install/pull-secret[Pull Secret] page on the {cloud-redhat-com} site.
. Generate the pull secret for your registry:
+
----
$ podman login --authfile ~/pullsecret_config.json <local_registry_host_name>:<local_registry_host_port> <1>
----
<1> For `<local_registry_host_name>`, specify the registry domain name
for your mirror registry, such as `registry.example.com`. For
`<local_registry_host_port>`, specify the port that your mirror registry uses to
serve content.
+
Provide your credentials for the mirror registry at the prompts.

. Generate the base64-encoded user name and password or token for your mirror
registry:
. View the pull secret that you created and record the pull secret value:
+
----
$ echo -n '<user_name>:<password>' | base64 -w0 <1>
# cat ~/pullsecret_config.json

BGVtbYk3ZHAtqXs=
{ "auths": { "<local_registry_host_name>:<local_registry_host_port>": { "auth": "ZHVtbXk6ZHVtbXk=" } } }
----
<1> For `<user_name>` and `<password>`, specify the user name and password that
you configured for your registry.

. Make a copy of your pull secret in JSON format:
. Download your pull secret from the
link:https://cloud.redhat.com/openshift/install/pull-secret[Pull Secret] page on the {cloud-redhat-com} site.

. Make a copy of the {cloud-redhat-com} link:https://cloud.redhat.com/openshift/install/pull-secret[Pull Secret] that you downloaded in JSON format:
+
----
$ cat ./pull-secret.text | jq . > <path>/<pull-secret-file><1>
Expand Down Expand Up @@ -64,22 +73,18 @@ The contents of the file resemble the following example:
}
----

. Edit the new file and add a section that describes your registry to it:
. Edit the {cloud-redhat-com} link:https://cloud.redhat.com/openshift/install/pull-secret[Pull Secret] file and add a section that describes your registry to it:
+
----
"auths": {
"auths": { <1>
...
"<local_registry_host_name>:<local_registry_host_port>": { <1>
"auth": "<credentials>", <2>
"email": "you@example.com"
},
...
----
<1> For `bastion_host_name`, specify the registry domain name
that you specified in your certificate, and for `<local_registry_host_port>`,
specify the port that your mirror registry uses to serve content.
<2> For `<credentials>`, specify the base64-encoded user name and password for
the mirror registry that you generated.
<1> Paste the contents of the `pullsecret_config.json` file that you created.
+
The file resembles the following example:
+
Expand Down
142 changes: 135 additions & 7 deletions modules/installation-mirror-repository.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,58 @@
// Module included in the following assemblies:
//
// * installing/installing_restricted_networks/installing-restricted-networks-preparations.adoc
// * updating/updating-restricted-network.adoc

ifeval::["{context}" == "installing-restricted-networks-preparations"]
:restricted:
:install:
endif::[]
ifeval::["{context}" == "updating-restricted-network"]
:restricted:
:update:
endif::[]

ifdef::install[]
[id="installation-mirror-repository_{context}"]
= Mirroring the {product-title} image repository

Mirror the {product-title} image repository to use during cluster installation
or upgrade.
Mirror the {product-title} image repository to use during cluster installation.

.Prerequisites

* You configured a mirror registry to use in your restricted network and
can access the certificate and credentials that you configured.
* You downloaded the pull secret from the
link:https://cloud.redhat.com/openshift/install/pull-secret[Pull Secret] page on the {cloud-redhat-com} site and modified it to include authentication to your mirror repository.
endif::install[]

ifdef::update[]
[id="update-mirror-image-repository_{context}"]
= Update the contents of the {product-title} image repository

Update the contents of the image repository that hosts the mirrored content that
you require for installing {product-title}. You must update the mirror registry
to update {product-title} to a new version.

.Prerequisites

* You have access to the mirror registry that you used to store the images that
you used to install {product-title}.
endif::update[]

.Procedure

Complete the following steps on the bastion host:

. Review the
link:https://access.redhat.com/downloads/content/290/[{product-title} downloads page]
to determine the version of {product-title} that you want to install.
to determine the version of {product-title} that you want to
ifdef::install[]
install.
endif::install[]
ifdef::update[]
update to.
endif::update[]

. Set the required environment variables:
+
Expand All @@ -34,7 +65,11 @@ $ export LOCAL_SECRET_JSON='<path_to_pull_secret>' <5>
$ export RELEASE_NAME="ocp-release" <6>
----
<1> For `<release_version>`, specify the version number of {product-title} to
install, such as `4.2.0`.
install, such as `4.2.1`.
ifdef::update[]
When you update {product-title}, you must specify a version number that is
higher than the version that is installed.
Comment thread
kalexand-rh marked this conversation as resolved.
Outdated
endif::update[]
<2> For `<local_registry_host_name>`, specify the registry domain name for your mirror
repository, and for `<local_registry_host_port>`, specify the port that it
serves content on.
Expand All @@ -57,11 +92,103 @@ $ oc adm -a ${LOCAL_SECRET_JSON} release mirror \
----
+
This command pulls the release information as a digest, and its output includes
the `imageContentSources` data that you require when you install your cluster.
text that resembles the following sample:

. Record the
ifdef::install[]
`imageContentSources`
endif::install[]
ifdef::update[]
`ImageContentSourcePolicy`
endif::update[]
section from the output of the previous
command. This information is required
ifdef::install[]
during {product-title} installation.
endif::install[]
ifdef::update[]
when you update your {product-title} cluster.
endif::update[]

. Build the signature for the content that you mirrored and verify that it is
signed by an official Red Hat key:
.. Set the digest for the version to
ifdef::install[]
install:
endif::install[]
ifdef::update[]
update to:
endif::update[]
+
----
$ DIGEST="$(oc adm release info quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} | sed -n 's/Pull From: .*@//p')" <1>
----
<1> For `<release_version>`, specify the version number of {product-title} to
that you mirrored content for.
Comment thread
kalexand-rh marked this conversation as resolved.
Outdated

.. Build a signature URI:
+
----
$ URI="https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/${DIGEST/:/=}/signature-<n>" <1>
----
<1> Specify the name of the signature to store. The signature name must be in the format of `signature-<n>`, where `<n>` is a number. If you have multiple signatures, the signature that includes the lowest number in its name is applied first, and other signatures are applied in increasing numerical order if one fails.

@wking wking Nov 8, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe:

<1> Specify the name of the signature to store. The signature name must be in the format of signature-<n>, where <n> is a number beginning with 1. If a signature is invalid, increment <n> until you find a valid signature or the requested signature-<n> does not exist.
Or some such. That adds "start at 1" and drop "is applied first", because this is more about "has anyone I trust signed this image?" and not about "who did something first?".

@kalexand-rh kalexand-rh Nov 8, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... If a signature is invalid, increment <n> until you find a valid signature or the requested signature-<n> does not exist.

Where do these other certificates come from? Is this a hidden step to say "If you get to $ gpg --no-default-keyring --keyring ./temp.keyring --verify <signature_name> and that doesn't work, then name the next one signature_n+1, and it'll just run through all the signature_n files in that directory until it finds a functioning one?

@wking wking Nov 8, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do these other certificates come from?

We don't have any at the moment, but may grow some during a future key-rotation or some such. In that case, we'd have signature-1 by the new key and signature-2 by the old key (order insignificant).

Is this a hidden step...

I was hoping that these docs made it explicit, but yeah "keep going until you find a satisfactory sig or run out of options" is the intended behavior. Hopefully we can teach oc to do this soon...

@kalexand-rh kalexand-rh Nov 8, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do these other certificates come from?

We don't have any at the moment, but may grow some during a future key-rotation or some such. In that case, we'd have signature-1 by the new key and signature-2 by the old key (order insignificant).

"We" meaning OCP devs or "we" meaning the person making these certs?

How and why would they have more than one, and why would they be checking multiples at this point? If I'm reading the process right, they generate a very specific certificate and then check it. Is it actually relevant that you can check multiples until you find a valid one?

Is this a hidden step...

I was hoping that these docs made it explicit, but yeah "keep going until you find a satisfactory sig or run out of options" is the intended behavior. Hopefully we can teach oc to do this soon...

That's what I'm trying to figure out. I'm not seeing a point in the current flow where a user would have a failed certificate or multiple certificates. There's not an explicit step to make another one. I'm trying to figure out if I need to another step around line 173 and what it would say or if this is a cool detail that's not helpful in this flow.

@wking wking Nov 8, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... or "we" meaning the person making these certs?

This one.

Is it actually relevant that you can check multiples until you find a valid one?

Not at the moment, but I'd be nervous if our manual-verification docs only supported signature-1 when all the software implementations were set up to walk available sigs. But maybe that would be fine and we'll have oc tooling for this by the time it matters?

I'm trying to figure out if I need to another step around line 173...

Yeah, if you think the wording here isn't enough, you'd want a "if the sig doesn't check out, increment signature-<index> , return to defining URI, and try again.

@kalexand-rh kalexand-rh Nov 8, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, if you think the wording here isn't enough, you'd want a "if the sig doesn't check out, increment signature-<index> , return to defining URI, and try again.

Yup. I'm adding a commit with another step. (If I took your suggestion as-is, I would expect to see a bug about the note not being clear enough and having to add the step later.) Thank you for working through this with me, and please let me know if you see more changes after the update.


.. Download the signature:
+
----
$ wget "${URI}"
----

.. Download the keys that were used to sign the signature:
+
----
$ wget https://www.redhat.com/security/data/f21541eb.txt
$ wget https://www.redhat.com/security/data/fd431d51.txt
----

.. Import the keys:
+
----
$ gpg --no-default-keyring --keyring ./temp.keyring --import ./f21541eb.txt
$ gpg --no-default-keyring --keyring ./temp.keyring --import ./fd431d51.txt
----

.. Verify the signature:
+
----
$ gpg --no-default-keyring --keyring ./temp.keyring --verify <signature_name> <1>

@wking wking Nov 8, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticed that this is checking for a valid sig, but not checking the signed content (so a single valid sig could be applied to multiple images). I'll get instructions for checking content when I get in.

@wking wking Nov 8, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, so what we want is --decrypt, not --verify.

$ gpg --no-default-keyring --keyring ./temp.keyring --decrypt signature-1 
{"critical": {"image": {"docker-manifest-digest": "sha256:a7e97365d16d8d920fedd3684b018b780337e069deb1dd8500e866c0d6110334"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.1.20"}}}gpg: Signature made Tue 15 Oct 2019 02:21:18 PM PDT using RSA key ID F21541EB
gpg: Good signature from "Red Hat, Inc. (beta key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: B08B 659E E86A F623 BC90  E8DB 938A 80CA F215 41EB

We want:

$ gpg --no-default-keyring --keyring ./temp.keyring --decrypt signature-1 | jq -r '.critical.image["docker-manifest-digest"]'
gpg: Signature made Tue 15 Oct 2019 02:21:18 PM PDT using RSA key ID F21541EB
gpg: Good signature from "Red Hat, Inc. (beta key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: B08B 659E E86A F623 BC90  E8DB 938A 80CA F215 41EB
sha256:a7e97365d16d8d920fedd3684b018b780337e069deb1dd8500e866c0d6110334

to match ${DIGEST}. Docs on the full structure here, altough I dunno how formal we want to get. I'll work on oc tooling for this next week so I'm ok cutting some corners in the docs here in the meantime, as long as we're confirming that the expected digest is in the signed content.

@mtrmac mtrmac Nov 8, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not even that.

This code, which resolves ${DIGEST} from the :${OCP_RELEASE} tag above, and then uses :${OCP_RELEASE} again in oc adm release mirror, is INSECURE and MUST NOT be used; the registry can maliciously modify the :${OCP_RELEASE} tag in the meantime. (Alternatively, if you trust the registry never to do that, and to always point at legitimate OCP images, you don’t need to bother with signatures, by definition.)

@mtrmac mtrmac Nov 8, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs on the full structure here, altough I dunno how formal we want to get. I'll work on oc tooling for this next week

Please use c/image/copy.Image, to do copying with signature verification integrated, if at all possible. Feel free to ping me anytime if you need help with that.

I don’t want copy&pasted partial implementations of that format all over OCP if I can help it.

gpg: Signature made Tue 24 Sep 2019 09:38:24 AM PDT using RSA key ID F21541EB
gpg: Good signature from "Red Hat, Inc. (beta key 2) <security@redhat.com>"
gpg: aka "Mark Cox Internal RSA 4096 test key <mjc@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: B08B 659E E86A F623 BC90 E8DB 938A 80CA F215 41EB
----
<1> Specify the name of the signature that you stored.

. Record the entire `imageContentSources` section from the output of the previous
command. The information about your mirrors is unique to your mirrored repository, and you must add the `imageContentSources` section to the `install-config.yaml` file during installation.
.. Confirm that the `Primary key fingerprint` value from the signature output is
is listed on the
link:https://access.redhat.com/security/team/key[Product Signing Keys]
page on the Red Hat Customer Portal.

. Mirror the repository:
+
----
$ oc adm -a ${LOCAL_SECRET_JSON} release mirror \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}
----
+
This command pulls the release information as a digest, and its output includes the
ifdef::install[]
`imageContentSources` data that you require when you install your cluster.
endif::install[]
ifdef::update[]
`ImageContentSourcePolicy` data that you require when you update your cluster.
endif::update[]

ifdef::install[]
. To create the installation program that is based on the content that you
mirrored, extract it and pin it to the release:
+
Expand All @@ -75,3 +202,4 @@ To ensure that you use the correct images for the version of {product-title}
that you selected, you must extract the installation program from the mirrored
content.
====
endif::install[]
Loading