Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ca for local cluster "teleport.localdomain" missing signing keys, refusing to bootstrap backend, initialization failed #7853

Closed
flybyray opened this issue Aug 10, 2021 · 5 comments · Fixed by #8119
Closed

ca for local cluster "teleport.localdomain" missing signing keys, refusing to bootstrap backend, initialization failed #7853

flybyray opened this issue Aug 10, 2021 · 5 comments · Fixed by #8119
Assignees
@codingllama
Labels
bug

Comments

@flybyray
Copy link

flybyray commented Aug 10, 2021 •
edited
Loading

Bootstraping a previously backu config is not possible since version 5.

ERROR:

error: ca for local cluster "teleport.localdomain" missing signing keys, refusing to bootstrap backend, initialization failed

What you expected to happen:

with teleport version 4.4.10-1 no errors. the bootstrapping of previously backup config is possible. output would be:

DEBU [SQLITE]    Connected to: file:fresh/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:172
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:217
DEBU [KEYGEN]    SSH cert authority is going to pre-compute 25 keys. native/native.go:103
DEBU [PROC:1]    Using sqlite backend. service/service.go:2565
DEBU [SQLITE]    Connected to: file:fresh/backend/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:172
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:217
INFO [AUTH]      Applying 2 bootstrap resources (first initialization) auth/init.go:187
DEBU [AUTH]      Cluster configuration: ClusterName(teleport.localdomain). auth/init.go:279
INFO [AUTH]      Updating cluster configuration: StaticTokens([]). auth/init.go:285
INFO [AUTH]      Updating cluster configuration: AuthPreference(Type="local",SecondFactor="otp"). auth/init.go:291
INFO [AUTH]      Created namespace: "default". auth/init.go:298
INFO [AUTH]      Created default admin role: "admin". auth/init.go:307
DEBU [AUTH]      Migrations: skipping local cluster cert authority "teleport.localdomain". auth/init.go:906
INFO [AUTH]      Auth server is running periodic operations. auth/init.go:451
DEBU [AUTH]      Ticking with period: 15s. auth/auth.go:264
DEBU [PROC:1]    This server has local Auth server started, using it to add role to the cluster. service/connect.go:337
DEBU [KEYGEN]    Generated SSH host certificate for role Admin with principals: [6b4a5769-189d-4e38-91ba-501d5684ab45 teleport.localdomain]. native/native.go:230
INFO [CA]        Generating TLS certificate {0x4307f28 0xc0008020c0 CN=6b4a5769-189d-4e38-91ba-501d5684ab45.teleport.localdomain,O=Admin,POSTALCODE=null,STREET= 2031-08-08 07:12:00.3302921 +0000 UTC [teleport.localdomain *.teleport.cluster.local teleport.cluster.local]}. common_name:6b4a5769-189d-4e38-91ba-501d5684ab45.teleport.localdomain dns_names:[teleport.localdomain *.teleport.cluster.local teleport.cluster.local] locality:[] not_after:2031-08-08 07:12:00.3302921 +0000 UTC org:[Admin] org_unit:[] tlsca/ca.go:259
INFO [PROC]      Admin has obtained credentials to connect to cluster. service/connect.go:375
INFO [PROC:1]    The process has successfully wrote credentials and state of Admin to disk. service/connect.go:416
DEBU [PROC:1]    Connected client: Identity(Admin, cert(6b4a5769-189d-4e38-91ba-501d5684ab45.teleport.localdomain issued by teleport.localdomain:108640329696301173699781080095548616284),trust root(teleport.localdomain:108640329696301173699781080095548616284)) service/connect.go:80
DEBU [PROC:1]    Connected server: Identity(Admin, cert(6b4a5769-189d-4e38-91ba-501d5684ab45.teleport.localdomain issued by teleport.localdomain:108640329696301173699781080095548616284),trust root(teleport.localdomain:108640329696301173699781080095548616284)) service/connect.go:81
DEBU [PROC:1]    Creating in-memory backend for [auth]. service/service.go:1373
INFO [PROC:1]    Service auth is creating new listener on 0.0.0.0:3025. service/signals.go:214
INFO [AUTH:1]    Starting Auth service with PROXY protocol support. service/service.go:1150
DEBU [AUTH]      GRPC(SERVER): keep alive 1m0s count: 3. auth/grpcserver.go:735
DEBU [PROC]      Adding service to supervisor. service:auth.tls service/supervisor.go:180
DEBU [PROC]      Adding service to supervisor. service:auth.heartbeat.broadcast service/supervisor.go:180
WARN [AUTH:1]    Configuration setting auth_service/advertise_ip is not set. guessing 172.17.0.4:3025. service/service.go:1229
DEBU [AUTH:BEAT] Starting Auth heartbeat with announce period: 1m0s, keep-alive period 5m23.879691862s, poll period: 5s srv/heartbeat.go:130
DEBU [PROC]      Adding service to supervisor. service:auth.heartbeat service/supervisor.go:180
DEBU [PROC]      Adding service to supervisor. service:auth.shutdown service/supervisor.go:180
DEBU [PROC]      Adding service to supervisor. service:register.node service/supervisor.go:180
DEBU [PROC]      Adding service to supervisor. service:ssh.node service/supervisor.go:180
DEBU [PROC]      Adding service to supervisor. service:ssh.shutdown service/supervisor.go:180
WARN [PROC]      No TLS Keys provided, using self signed certificate. service/service.go:2712
WARN [PROC]      Generating self signed key and cert to fresh/webproxy_key.pem fresh/webproxy_cert.pem. service/service.go:2728
DEBU [PROC]      Adding service to supervisor. service:register.proxy service/supervisor.go:180
DEBU [PROC]      Adding service to supervisor. service:proxy.init service/supervisor.go:180
DEBU [PROC]      Adding service to supervisor. service:common.rotate service/supervisor.go:180
DEBU [PROC:1]    Service has started. service:register.node service/supervisor.go:241
DEBU [PROC:1]    Service has started. service:auth.heartbeat.broadcast service/supervisor.go:241
DEBU [PROC:1]    Service has started. service:register.proxy service/supervisor.go:241
DEBU [PROC:1]    Service has started. service:auth.tls service/supervisor.go:241
DEBU [PROC:1]    Service has started. service:ssh.shutdown service/supervisor.go:241
INFO [AUTH]      Auth service 4.4.10:v4.4.10-0-g710c3b0f2 is starting on 0.0.0.0:3025. utils/cli.go:174
DEBU [PROC:1]    Broadcasting event. event:AuthTLSReady service/supervisor.go:329
DEBU [PROC:1]    Service has started. service:common.rotate service/supervisor.go:241
DEBU [PROC:1]    Service has started. service:proxy.init service/supervisor.go:241
DEBU [PROC:1]    Service has started. service:auth.shutdown service/supervisor.go:241
DEBU [PROC:1]    Service has started. service:auth.heartbeat service/supervisor.go:241
DEBU [PROC:1]    No signal pipe to import, must be first Teleport process. service/service.go:708
DEBU [PROC:1]    Service has started. service:ssh.node service/supervisor.go:241
...

Reproduction Steps

  1. install a version with the bug since 5.0.0-1
yum install teleport-5.0.0-1
  1. run the following procedure. this is a minimal example as it might be described by the official documentation
rm -rf fresh fresh.yaml
mkdir fresh && cat > fresh.yaml << EOF
teleport:
  data_dir: fresh
EOF
teleport start --config ./fresh.yaml > /dev/null 2>&1 &

TELEPORT_PROCESS=$! \
&& sleep 5 \
&& tctl --config ./fresh.yaml get all --with-secrets > state-new.yaml \
&& kill -2 $TELEPORT_PROCESS \
&& rm -rf fresh fresh.yaml \
&& mkdir fresh && cat > fresh.yaml << EOF
teleport:
  data_dir: fresh
EOF

teleport start --debug --config ./fresh.yaml --bootstrap state-new.yaml &
TELEPORT_PROCESS=$! \
&& sleep 5 \
&& kill -2 $TELEPORT_PROCESS

Server Details

  • Teleport v5.0.0 git:v5.0.0-0-gac4971801 go1.15.5
  • CentOS Linux 7 (Core)
  • centos:7 container
  • Additional details:

Client Details

irrelevant

Debug Logs

Please include or attach debug logs, when appropriate. Obfuscate sensitive information!

  • Start Teleport with --debug flag (teleport --debug)
DEBU [SQLITE]    Connected to: file:fresh/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:173
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:218
DEBU [KEYGEN]    SSH cert authority is going to pre-compute 25 keys. native/native.go:104
DEBU [PROC:1]    Using sqlite backend. service/service.go:2913
DEBU [SQLITE]    Connected to: file:fresh/backend/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:173
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:218
INFO [AUTH]      Applying 3 bootstrap resources (first initialization) auth/init.go:188

ERROR REPORT:
Original Error: *trace.BadParameterError ca for local cluster "teleport.localdomain" missing signing keys
Stack Trace:
        /go/src/github.com/gravitational/teleport/lib/auth/init.go:532 github.com/gravitational/teleport/lib/auth.checkResourceConsistency
        /go/src/github.com/gravitational/teleport/lib/auth/init.go:188 github.com/gravitational/teleport/lib/auth.Init
        /go/src/github.com/gravitational/teleport/lib/service/service.go:1113 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthService
        /go/src/github.com/gravitational/teleport/lib/service/service.go:689 github.com/gravitational/teleport/lib/service.NewTeleport
        /go/src/github.com/gravitational/teleport/lib/service/service.go:423 github.com/gravitational/teleport/lib/service.newTeleportProcess
        /go/src/github.com/gravitational/teleport/lib/service/service.go:433 github.com/gravitational/teleport/lib/service.Run
        /go/src/github.com/gravitational/teleport/tool/teleport/common/teleport.go:197 github.com/gravitational/teleport/tool/teleport/common.OnStart
        /go/src/github.com/gravitational/teleport/tool/teleport/common/teleport.go:174 github.com/gravitational/teleport/tool/teleport/common.Run
        /go/src/github.com/gravitational/teleport/tool/teleport/main.go:29 main.main
        /opt/go/src/runtime/proc.go:213 runtime.main
        /opt/go/src/runtime/asm_amd64.s:1375 runtime.goexit
User Message: ca for local cluster "teleport.localdomain" missing signing keys, refusing to bootstrap backend, initialization failed
@flybyray flybyray added the bug label Aug 10, 2021
@codingllama codingllama mentioned this issue Sep 1, 2021
Merged
codingllama added a commit that referenced this issue Sep 2, 2021
@codingllama
Correctly validate JWT CA on bootstrap (#8119)
133ebcd 
Presently, teleport start --bootstrap state.yaml fails due to incorrect
handling of JWT CAs, even when the data is generated using
tctl get all --with-secrets.

Fixes #7853.

* Correctly validate JWT CertAuthorities on bootstrap
* Remove commented code
@codingllama
Copy link
Contributor

Closed on master, which means it'll take some time to appear in a future release. I'll backport this to Teleport 7 to speed things up - I expect it should appear on the next minor that goes out.

@codingllama
Copy link
Contributor

Backported to v7 by #8128.

@codingllama codingllama self-assigned this Sep 2, 2021
zmb3 pushed a commit that referenced this issue Sep 23, 2021
@codingllama @zmb3
Correctly validate JWT CA on bootstrap (#8119)
d69f6d6 
Presently, teleport start --bootstrap state.yaml fails due to incorrect
handling of JWT CAs, even when the data is generated using
tctl get all --with-secrets.

Fixes #7853.

* Correctly validate JWT CertAuthorities on bootstrap
* Remove commented code
@flybyray
Copy link
Author

flybyray commented Feb 7, 2022

Backported to v7 by #8128.

That means it was at that time available in the release v7.1.2 (rpm) .
Just had now the pleasure to try it out with that version.

The error changed now to:

ERROR: failed to sync CertAuthority key formats for CA(name=teleport.example.com, type=jwt): mis-matched SSH private(1) and public(0) key counts
backend bootstrap failed
initialization failed
mis-matched SSH private(1) and public(0) key counts

should i reopen this or should I create another ticket?

@codingllama
Copy link
Contributor

Hey @flybyray, I tried the repro steps we have above using Teleport v7.3.13 and it looked fine to me.

Could you try the latest 7.x and let me know if it works? If not, do let me know if there is anything different about the setup.

@flybyray
Copy link
Author

flybyray commented Feb 9, 2022

Hi @codingllama i did a workarround.
i just initialized a new cluster, exported the yaml with-secrets, extracted from this the jwt token, replaced the jwt in the bootstrap file and then it worked.
somehow the boostrap yaml exported with the previous version 6 had a wrong jwt exported which was not able to read by version 7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@codingllama codingllama

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Correctly validate JWT CA on bootstrap gravitational/teleport
2 participants
@flybyray @codingllama