vCluster connect fails silently when connecting with --server flag

[colinjlacy]

What happened?

When I run vcluster connect <vcluster-name> --server=<any-url> I get a positive response output, regardless of the actual result and whether or not a vCluster is even running at the URL that was entered.

This happens if there is a matching vCluster found in the current context even if the --server value points to the vCluster control plane.

(3.10.5) ➜  vcluster connect vcluster-dev -n default --server=https://google.com
05:36:31 done vCluster is up and running
05:36:31 done Switched active kube context to vcluster_vcluster-dev_default_kind-vcluster-host
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

I used Google just to prove a point, but the real problem comes in when trying to connect through a load balancer as explained here. I may be fooled into thinking that I'm working in a vCluster when I'm actually working in the host. That could very easily result in resource conflicts or deleting host resources entirely.

What did you expect to happen?

Some error message indicating that the vCluster could not be reached at the specified server endpoint.

How can we reproduce it (as minimally and precisely as possible)?

Set up a Kind cluster and install a vClsuter in the default namespace. I used Helm:

helm upgrade --install vcluster-dev vcluster --namespace default --repo https://charts.loft.sh  --repository-config=''

Once everything is up and running, run:

vcluster connect vcluster-dev -n default --server=https://google.com

You should see:

05:36:31 done vCluster is up and running
05:36:31 done Switched active kube context to vcluster_vcluster-dev_default_kind-vcluster-host
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

Now run kubectl config get-contexts, and you'll see that you're still using the host cluster context.

It's worth noting that if you delete the vCluster:

helm delete vcluster-dev

and then try to connect, the attempt will fail:

(3.10.5) ➜  vcluster connect vcluster-dev -n default --server=https://google.com
06:16:32 fatal couldn't find vcluster vcluster-dev

Anything else we need to know?

No response

Host cluster Kubernetes version

Client Version: v1.32.0
Kustomize Version: v5.5.0
Server Version: v1.31.2

vcluster version

vcluster version 0.22.1

VCluster Config

controlPlane:
advanced:
defaultImageRegistry: ""
globalMetadata:
annotations: {}
headlessService:
annotations: {}
labels: {}
serviceAccount:
annotations: {}
enabled: true
imagePullSecrets: []
labels: {}
name: ""
virtualScheduler:
enabled: false
workloadServiceAccount:
annotations: {}
enabled: true
imagePullSecrets: []
labels: {}
name: ""
backingStore:
database:
embedded:
enabled: false
external:
caFile: ""
certFile: ""
connector: ""
dataSource: ""
enabled: false
keyFile: ""
etcd:
deploy:
enabled: false
headlessService:
annotations: {}
service:
annotations: {}
enabled: true
statefulSet:
annotations: {}
enableServiceLinks: true
enabled: true
env: []
extraArgs: []
highAvailability:
replicas: 1
image:
registry: registry.k8s.io
repository: etcd
tag: 3.5.15-0
imagePullPolicy: ""
labels: {}
persistence:
addVolumeMounts: []
addVolumes: []
volumeClaim:
accessModes:
- ReadWriteOnce
enabled: true
retentionPolicy: Retain
size: 5Gi
storageClass: ""
volumeClaimTemplates: []
pods:
annotations: {}
labels: {}
resources:
requests:
cpu: 20m
memory: 150Mi
scheduling:
affinity: {}
nodeSelector: {}
podManagementPolicy: Parallel
priorityClassName: ""
tolerations: []
topologySpreadConstraints: []
security:
containerSecurityContext: {}
podSecurityContext: {}
embedded:
enabled: false
migrateFromDeployedEtcd: false
coredns:
deployment:
affinity: {}
annotations: {}
image: ""
labels: {}
nodeSelector: {}
pods:
annotations: {}
labels: {}
replicas: 1
resources:
limits:
cpu: 1000m
memory: 170Mi
requests:
cpu: 20m
memory: 64Mi
tolerations: []
topologySpreadConstraints:
- labelSelector:
matchLabels:
k8s-app: vcluster-kube-dns
maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
embedded: false
enabled: true
overwriteConfig: ""
overwriteManifests: ""
priorityClassName: ""
service:
annotations: {}
labels: {}
spec:
type: ClusterIP
distro:
k0s:
command: []
config: ""
enabled: false
extraArgs: []
image:
registry: ""
repository: k0sproject/k0s
tag: v1.30.2-k0s.0
imagePullPolicy: ""
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 40m
memory: 64Mi
securityContext: {}
k3s:
command: []
enabled: false
extraArgs: []
image:
registry: ""
repository: rancher/k3s
tag: v1.31.1-k3s1
imagePullPolicy: ""
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 40m
memory: 64Mi
securityContext: {}
k8s:
apiServer:
command: []
enabled: true
extraArgs: []
image:
registry: registry.k8s.io
repository: kube-apiserver
tag: v1.31.1
imagePullPolicy: ""
controllerManager:
command: []
enabled: true
extraArgs: []
image:
registry: registry.k8s.io
repository: kube-controller-manager
tag: v1.31.1
imagePullPolicy: ""
enabled: false
env: []
resources:
limits:
cpu: 100m
memory: 256Mi
requests:
cpu: 40m
memory: 64Mi
scheduler:
command: []
extraArgs: []
image:
registry: registry.k8s.io
repository: kube-scheduler
tag: v1.31.1
imagePullPolicy: ""
securityContext: {}
version: ""
ingress:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
enabled: false
host: my-host.com
labels: {}
pathType: ImplementationSpecific
spec:
tls: []
proxy:
bindAddress: 0.0.0.0
extraSANs: []
port: 8443
service:
annotations: {}
enabled: true
httpsNodePort: 0
kubeletNodePort: 0
labels: {}
spec:
type: ClusterIP
serviceMonitor:
annotations: {}
enabled: false
labels: {}
statefulSet:
annotations: {}
args: []
command: []
enableServiceLinks: true
env: []
highAvailability:
leaseDuration: 60
renewDeadline: 40
replicas: 1
retryPeriod: 15
image:
registry: ghcr.io
repository: loft-sh/vcluster-pro
tag: ""
imagePullPolicy: ""
labels: {}
persistence:
addVolumeMounts: []
addVolumes: []
binariesVolume:
- emptyDir: {}
name: binaries
dataVolume: []
volumeClaim:
accessModes:
- ReadWriteOnce
enabled: auto
retentionPolicy: Retain
size: 5Gi
storageClass: ""
volumeClaimTemplates: []
pods:
annotations: {}
labels: {}
probes:
livenessProbe:
enabled: true
readinessProbe:
enabled: true
startupProbe:
enabled: true
resources:
limits:
ephemeral-storage: 8Gi
memory: 2Gi
requests:
cpu: 200m
ephemeral-storage: 400Mi
memory: 256Mi
scheduling:
affinity: {}
nodeSelector: {}
podManagementPolicy: Parallel
priorityClassName: ""
tolerations: []
topologySpreadConstraints: []
security:
containerSecurityContext:
allowPrivilegeEscalation: false
runAsGroup: 0
runAsUser: 0
podSecurityContext: {}
workingDir: ""
experimental:
deploy:
host:
manifests: ""
manifestsTemplate: ""
vcluster:
helm: []
manifests: ""
manifestsTemplate: ""
genericSync:
clusterRole:
extraRules: []
role:
extraRules: []
isolatedControlPlane:
headless: false
multiNamespaceMode:
enabled: false
syncSettings:
disableSync: false
rewriteKubernetesService: false
setOwner: true
targetNamespace: ""
exportKubeConfig:
context: ""
insecure: false
secret:
name: ""
namespace: ""
server: ""
serviceAccount:
clusterRole: ""
name: ""
namespace: ""
external: {}
integrations:
certManager:
enabled: false
sync:
fromHost:
clusterIssuers:
enabled: true
selector:
labels: {}
toHost:
certificates:
enabled: true
issuers:
enabled: true
externalSecrets:
enabled: false
sync:
clusterStores:
enabled: false
selector:
labels: {}
externalSecrets:
enabled: true
stores:
enabled: false
webhook:
enabled: false
kubeVirt:
enabled: false
sync:
dataVolumes:
enabled: false
virtualMachineClones:
enabled: true
virtualMachineInstanceMigrations:
enabled: true
virtualMachineInstances:
enabled: true
virtualMachinePools:
enabled: true
virtualMachines:
enabled: true
webhook:
enabled: true
metricsServer:
enabled: false
nodes: true
pods: true
networking:
advanced:
clusterDomain: cluster.local
fallbackHostCluster: false
proxyKubelets:
byHostname: true
byIP: true
replicateServices:
fromHost: []
toHost: []
resolveDNS: []
plugins: {}
policies:
centralAdmission:
mutatingWebhooks: []
validatingWebhooks: []
limitRange:
annotations: {}
default:
cpu: "1"
ephemeral-storage: 8Gi
memory: 512Mi
defaultRequest:
cpu: 100m
ephemeral-storage: 3Gi
memory: 128Mi
enabled: auto
labels: {}
max: {}
min: {}
networkPolicy:
annotations: {}
enabled: false
fallbackDns: 8.8.8.8
labels: {}
outgoingConnections:
ipBlock:
cidr: 0.0.0.0/0
except:
- 100.64.0.0/10
- 127.0.0.0/8
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
platform: true
resourceQuota:
annotations: {}
enabled: auto
labels: {}
quota:
count/configmaps: 100
count/endpoints: 40
count/persistentvolumeclaims: 20
count/pods: 20
count/secrets: 100
count/services: 20
limits.cpu: 20
limits.ephemeral-storage: 160Gi
limits.memory: 40Gi
requests.cpu: 10
requests.ephemeral-storage: 60Gi
requests.memory: 20Gi
requests.storage: 100Gi
services.loadbalancers: 1
services.nodeports: 0
scopeSelector:
matchExpressions: []
scopes: []
rbac:
clusterRole:
enabled: auto
extraRules: []
overwriteRules: []
role:
enabled: true
extraRules: []
overwriteRules: []
sync:
fromHost:
csiDrivers:
enabled: auto
csiNodes:
enabled: auto
csiStorageCapacities:
enabled: auto
events:
enabled: true
ingressClasses:
enabled: false
nodes:
clearImageStatus: false
enabled: false
selector:
all: false
labels: {}
syncBackChanges: false
priorityClasses:
enabled: false
runtimeClasses:
enabled: false
storageClasses:
enabled: auto
volumeSnapshotClasses:
enabled: false
toHost:
configMaps:
all: false
enabled: true
endpoints:
enabled: true
ingresses:
enabled: false
networkPolicies:
enabled: false
persistentVolumeClaims:
enabled: true
persistentVolumes:
enabled: false
podDisruptionBudgets:
enabled: false
pods:
enabled: true
enforceTolerations: []
rewriteHosts:
enabled: true
initContainer:
image: library/alpine:3.20
resources:
limits:
cpu: 30m
memory: 64Mi
requests:
cpu: 30m
memory: 64Mi
translateImage: {}
useSecretsForSATokens: false
priorityClasses:
enabled: false
secrets:
all: false
enabled: true
serviceAccounts:
enabled: false
services:
enabled: true
storageClasses:
enabled: false
volumeSnapshotContents:
enabled: false
volumeSnapshots:
enabled: false
telemetry:
enabled: true

[kale-amruta]

@colinjlacy Thanks for raising the issue! As I understand the issue, I tried reproducing it, I do see that we are able to connect to the vcluster even when it does not exist on the endpoint but as you mentioned that the kubecontext still points to the host cluster context is not what I am seeing.

I am rather seeing the context set to the vcluster we are trying to connect to, its when we hit one of the kubectl commands to get/create the resources on the cluster where we start seeing the failure.

➜  vcluster git:(main) ✗  vcluster create my-vcluster -n my-vcluster --connect=false --expose    
17:50:31 info Create vcluster my-vcluster...
17:50:31 info execute command: helm upgrade my-vcluster /tmp/vcluster-0.22.1.tgz-4078691228 --create-namespace --kubeconfig /tmp/1234924392 --namespace my-vcluster --install --repository-config='' --values /tmp/3374424106
17:50:31 done Successfully created virtual cluster my-vcluster in namespace my-vcluster. 
- Use 'vcluster connect my-vcluster --namespace my-vcluster' to access the virtual cluster
- Use `vcluster connect my-vcluster --namespace my-vcluster -- kubectl get ns` to run a command directly within the vcluster

➜  vcluster git:(main) ✗ kubectl get svc -n my-vcluster
NAME                                   TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
kube-dns-x-kube-system-x-my-vcluster   ClusterIP      10.96.44.57    <none>        53/UDP,53/TCP,9153/TCP   58s
my-vcluster                            LoadBalancer   10.96.220.87   172.18.0.5    443:32327/TCP            81s
my-vcluster-headless                   ClusterIP      None           <none>        443/TCP                  81s
my-vcluster-node-kind-control-plane    ClusterIP      10.96.148.53   <none>        10250/TCP                58s

➜  vcluster git:(main) ✗ vcluster connect my-vcluster -n my-vcluster --server=google.com
17:53:46 done vCluster is up and running
17:53:46 done Switched active kube context to vcluster_my-vcluster_my-vcluster_kind-kind
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster

➜  vcluster git:(main) ✗ kubectl config get-contexts                           
CURRENT   NAME                                         CLUSTER                                      AUTHINFO                                     NAMESPACE
          kind-kind                                    kind-kind                                    kind-kind                                    
          kind-kind2                                   kind-kind2                                   kind-kind2                                   
*         vcluster_my-vcluster_my-vcluster_kind-kind   vcluster_my-vcluster_my-vcluster_kind-kind   vcluster_my-vcluster_my-vcluster_kind-kind   


➜  vcluster git:(main) ✗ kubectl get namespaces
E0115 18:00:57.467652  139030 memcache.go:265] couldn't get current server API group list: Get "https://google.com/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0115 18:00:57.571112  139030 memcache.go:265] couldn't get current server API group list: Get "https://google.com/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0115 18:00:57.673130  139030 memcache.go:265] couldn't get current server API group list: Get "https://google.com/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0115 18:00:57.757317  139030 memcache.go:265] couldn't get current server API group list: Get "https://google.com/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0115 18:00:57.887897  139030 memcache.go:265] couldn't get current server API group list: Get "https://google.com/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority