rawResources failed to read Resources and secret truststore not found on getting started
#653
Comments
|
I ran Cheers! |
|
Forgive me as I've not run Kubernetes before, although I have done work with Docker in the past...
https://kubernetes.io/docs/concepts/configuration/secret/ Does that mean there is an issue with the I am able to get Cheers, |
|
Looks like some serious errors in the ... and also the |
|
Based on some hints here: https://stackoverflow.com/questions/59055244/kube8s-pod-unable-to-connect-to-scheduler The I got a few errors in the |
|
Following the instructions here for troubleshooting: https://github.com/ForgeRock/forgeops#troubleshooting-tips
... so it looks like the secrets are definitely not set up correctly. Cheers, |
|
Did you run:
bin/config.sh init
Check the output of the secrets pod. It looks like your secrets did not get
created.
…On Wed, Nov 4, 2020 at 3:41 PM Nigel Sheridan-Smith < ***@***.***> wrote:
Following the instructions here for troubleshooting:
https://github.com/ForgeRock/forgeops#troubleshooting-tips
kubectl describe pod am-7468ff6c8b-nj8lp gave me:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 41m default-scheduler Successfully assigned default/am-7468ff6c8b-nj8lp to minikube
Warning FailedMount 41m kubelet MountVolume.SetUp failed for volume "truststore" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 41m kubelet MountVolume.SetUp failed for volume "default-token-dpmmh" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 41m (x2 over 41m) kubelet MountVolume.SetUp failed for volume "amster-key" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 41m (x2 over 41m) kubelet MountVolume.SetUp failed for volume "am-secrets" : [secret "am-passwords" not found, failed to sync secret cache: timed out waiting for the condition]
Warning FailedMount 41m kubelet MountVolume.SetUp failed for volume "am-secrets" : [secret "am-passwords" not found, secret "am-keystore" not found]
Warning FailedMount 40m (x6 over 41m) kubelet MountVolume.SetUp failed for volume "amster-key" : secret "amster" not found
Warning FailedMount 40m (x5 over 41m) kubelet MountVolume.SetUp failed for volume "am-secrets" : secret "am-keystore" not found
Warning FailedMount 39m kubelet Unable to attach or mount volumes: unmounted volumes=[am-secrets amster-key], unattached volumes=[fbc default-token-dpmmh truststore am-secrets amster-key]: timed out waiting for the condition
Normal Pulled 39m kubelet Container image "am:fde36ae0e4bce97d06620f8e974fc6db1aaec941b2e1b2b25e5c0c54588c6d74" already present on machine
Normal Created 39m kubelet Created container fbc-init
Normal Started 39m kubelet Started container fbc-init
Normal Pulled 39m kubelet Container image "am:fde36ae0e4bce97d06620f8e974fc6db1aaec941b2e1b2b25e5c0c54588c6d74" already present on machine
Normal Created 39m kubelet Created container openam
Normal Started 39m kubelet Started container openam
Warning Unhealthy 36m (x4 over 37m) kubelet Readiness probe failed: Get "http://172.17.0.11:8080/am/json/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
... so it looks like the secrets are definitely not set up correctly.
Cheers,
Nigel
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#653 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADNEZHYREMQS3VW6UD2B3LSOHKDLANCNFSM4TJUDT4Q>
.
|
|
Looks like this is due to the Details of the secrets management:
|
Then I get this: After waiting a few minutes, the Admin page still doesn't appear (it times out a few times). Here's the key errors I can see: Cheers, |
|
The master branch requires installation of the secret agent operator -
which creates the required secrets. There is a script in the bin/
directory to install the operator.
…On Wed, Nov 4, 2020 at 5:53 PM Nigel Sheridan-Smith < ***@***.***> wrote:
minikube stop
minikube delete
git checkout 2020.08.07-ZucchiniRicotta.1
bin/config.sh -v 7.0 init
minikube start
skaffold dev
Then I get this:
forgeops % kubectl get pods
NAME READY STATUS RESTARTS AGE
admin-ui-545d6bccf5-t8vzd 1/1 Running 0 9m46s
am-6db56cf7c6-h5k98 1/1 Running 2 9m46s
amster-5fv7b 1/1 Running 0 43s
amster-rghpz 0/1 Error 0 9m45s
ds-cts-0 1/1 Running 0 9m45s
ds-idrepo-0 1/1 Running 0 9m45s
end-user-ui-85c655599d-tn9p6 1/1 Running 0 9m46s
forgeops-secrets-6pqqx 0/1 Completed 0 9m44s
idm-0 1/1 Running 1 9m45s
ldif-importer-lz7zc 0/1 Completed 0 9m44s
login-ui-8477c6b75c-trqsp 1/1 Running 0 9m46s
After waiting a few minutes, the Admin page still doesn't appear (it times
out a few times).
https://default.iam.example.com/am
Here's the key errors I can see:
kubectl logs am-6db56cf7c6-h5k98
...
Starting up OpenAM at Nov 5, 2020, 12:41:01 AM
{"timestamp":"2020-11-05T00:41:09.648Z","level":"WARN","thread":"main","mdc":{"transactionId":"418f6818-d461-47af-97ad-c2cd3facd3ea-1"},"logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=10, blocked=0, ***@***.***)' is no longer operational: Connect Error: UNUSED.IN.FBC","context":"default"}
{"timestamp":"2020-11-05T00:41:10.069Z","level":"ERROR","thread":"Thread-5","logger":"org.forgerock.openam.services.push.PushNotificationServiceConfigHelperFactory","message":"Unable to retrieve instance of the ServiceConfig for realm /.","context":"default"}
{"timestamp":"2020-11-05T00:41:10.116Z","level":"WARN","thread":"Thread-5","logger":"org.forgerock.openam.services.push.PushNotificationService","message":"Unable to read config for PushNotificationConfig in realm /","context":"default"}
...
Cheers,
Nigel
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#653 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADNEZEXKTF43DRZ4JY4WIDSOHZQHANCNFSM4TJUDT4Q>
.
|
|
I got the Kubernetes dashboard running, which makes life a bit easier to see what is going on: https://github.com/kubernetes/dashboard
Not sure what this means but there's some suggestions in this article: For the
Cheers, |
|
Seems to be an issue with the
There are also ingress errors such as |
|
Looks like https://github.com/csovant/forgeops/tree/bf28af969cbea80b59feb020b1b274d4f3da0646/helm#tls was the last commit to support Helm. csovant@471e93e removes the There was major changes in v7.0 for PR #540, documented here: Is Cheers, |
|
Nigel I think some of your issues may stem from reading documentation that does not match the branch/tag that you are trying to deploy. If you'd like to try the latest release, we have a "quickstart.sh" script that you can deploy on minikube. Caveat: There is a race condition that we sometimes see when creating the secrets. I had to kill the ds-cts-0 and ds-idrepo-0 pods to get them to correctly mount the secrets. We are working on a fix for that issue. Also - if you do want to view matching docs for release, we are now bunlding those up as a zip file as part of the release. This is a good way to ensure the doc you are reading matches the release. |
|
Nigel, to elaborate on what Warren said, consider using one of these releases: https://github.com/ForgeRock/forgeops/releases/tag/2020.10.28-AlSugoDiNoci - an unsupported interim release just issued last week. There's a zip file containing the docs for the release at https://github.com/ForgeRock/forgeops/releases/download/2020.10.28-AlSugoDiNoci/forgeops-docs.zip. https://github.com/ForgeRock/forgeops/releases/tag/2020.08.07-ZucchiniRicotta.1 - our official, supported release. Docs at https://backstage.forgerock.com/docs/forgeops/7/index-forgeops.html If you want to use the master branch, feel free to do so, but docs are not available for it. However, since the interim release is so new, almost everything in the doc for the interim release currently applies to the master branch. Some of our readmes have gotten a little stale; it might be a better approach to go with the published doc. Hope this helps get you up and running. |
|
Thanks @wstrange and @dgoldssfo, I think I missed one specific configuration required for Mac OS: ... which was in your DevOps guides (PDF, v6.5, and probably v7 as well). I'm still having trouble with some secrets but I feel confident I should be able to get it sorted today. Thanks for your help! Nigel |
|
Could you post your minikube start command here? Couple of gotcha - (1) the Docker driver on macOS does not support the Minikube ingress plugin, so you have to use a hypervisor (we suggest VBox but others will work, too); (2) we recommend --kubernetes-version=1.17.4; just haven't tested on later versions, and 1.17.4 is known to work. |
|
Also I'd suggest you make sure you delete EVERYTHING. e.g. Also you shouldn't have forgeops-secrets deployed at all. So make sure you're using the release git tag. |
|
Ok looking much better now... Switched to Ensure all the secrets are cleaned up and re-created by the Then in a separate window/console: Then I visited https://default.iam.example.com/am/ and the login page appears after I accept the SSL/TLS certificate! Thanks for all your tips and especially the gotchas. I'll see if I can get the newer (unofficial) release running too shortly... Best regards, |
|
I think the It looks like I can get into the Access Manager successfully, and I'll work on getting the Identity Gateway working now too - that was what I needed to test out (reverse proxy). Cheers, |
|
8Gb is the minimum. It's covered in the docs.
…On Thu, Nov 5, 2020, 18:39 Nigel Sheridan-Smith ***@***.***> wrote:
I think the minikube default of 4Gb of RAM is not enough, so I had to do
this when starting Kubernetes, prior to skaffold dev - due to some pods
being in the pending state:
minikube start --vm=true --memory 8196
It looks like I can get into the Access Manager successfully, and I'll
work on getting the Identity Gateway working now too - that was what I
needed to test out (reverse proxy).
Cheers,
Nigel
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#653 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANF4NO5BHIHT4KJVZWARKODSONOW3ANCNFSM4TJUDT4Q>
.
--
ForgeRock values your Privacy <https://www.forgerock.com/your-privacy>
|
|
Thanks @maxres-fr , found the more detailed documentation now: https://backstage.forgerock.com/docs/forgeops/7/devops-minikube-implementation-env.html#devops-implementation-env-cluster-mac-linux Cheers, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi there,
I came across a few issues following the instructions on
Mac OS 10.15.7withkubectlonv1.19.3:... failed with...
I solved this with
brew install kustomizeto getv3.8.6instead of the bundled2.0.3withkubectl. Here's some related issues:kubernetes-sigs/kustomize#1424
kubernetes/kubernetes#82905
kubernetes/kubernetes#89887
kubernetes/kubectl#818
kubernetes-sigs/kustomize#1500
kubernetes-sigs/kustomize#1647
kubernetes-sigs/kustomize#1648 (comment)
kubernetes-sigs/kustomize#776
kubernetes-sigs/kustomize#1556
Then I got stuck with...
... which I solved by running
bin/quickstart.sh.It's getting further along, but I'm now getting
trustStoreissues too.Any suggestions? I'll see if I can find the issue.
Cheers,
Nigel
The text was updated successfully, but these errors were encountered: