Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a target pod flag for linkerd diagnostics endpoints #9141

Closed
mateiidavid opened this issue Aug 12, 2022 · 0 comments · Fixed by #9200
Closed

Add a target pod flag for linkerd diagnostics endpoints #9141

mateiidavid opened this issue Aug 12, 2022 · 0 comments · Fixed by #9200

Comments

@mateiidavid
Copy link
Member

What problem are you trying to solve?

In order to debug service discovery communication, we can use linkerd diagnostics endpoints. The command effectively queries the same endpoint as the proxy, and will return the endpoints associated with a service. To communicate with the destination service, an external (out-of-cluster) client is created; this is done through a port-forward.

func NewExternalClient(ctx context.Context, controlPlaneNamespace string, kubeAPI *k8s.KubernetesAPI) (pb.DestinationClient, *grpc.ClientConn, error) {
portforward, err := k8s.NewPortForward(
ctx,
kubeAPI,
controlPlaneNamespace,
destinationDeployment,
"localhost",
0,
destinationPort,
false,
)
if err != nil {
return nil, nil, err
}
destinationAddress := portforward.AddressAndPort()
if err = portforward.Init(); err != nil {
return nil, nil, err
}
return NewClient(destinationAddress)
}

When dealing with multiple destination pods, however, it can be hard to target only a specific destination container. The port-forward will be random (and heavily dependent on the order of the pods as returned by the k8s api server).

for _, pod := range podList.Items {
if pod.Status.Phase == corev1.PodRunning {
grandparent, err := getDeploymentForPod(ctx, k8sAPI, pod)
if err != nil {
log.Warnf("Failed to get deploy for pod [%s]: %s", pod.Name, err)
continue
}
if grandparent == deployName {
podName = pod.Name
break
}
}
}

It would be useful to have a CLI flag that can target a specific pod; this way, multiple instances can be checked separately for errors.

How should the problem be solved?

Add a new flag to linkerd diagnostics endpoints (e.g --destination-pod). When a value is set for this flag, we can port-forward directly to the pod itself (if it exists), instead of fetching the list of pods under the deployment.

Any alternatives you've considered?

The alternative is to port-forward individual pods and use the destination client script. However, people that do not have access to the repository, or have not worked with it enough, may find this method more cumbersome to use.

How would users interact with this feature?

No response

Would you like to work on this feature?

No response

@adleong adleong added this to the stable-2.12.1 milestone Aug 18, 2022
@kleimkuhler kleimkuhler self-assigned this Aug 18, 2022
kleimkuhler added a commit that referenced this issue Aug 18, 2022
…mmand (#9200)

Closes #9141

This introduces the `--destination-pod` flag to the `linkerd diagnostics
endpoints` command which allows users to target a specific destination Pod when
there are multiple running in a cluster.

This can be useful for issues like #8956, where Linkerd HA is installed and
there seem to be stale endpoints in the destination service. Being able to run
this command and identity which destination Pod (if not all) have an incorrect
view of the cluster.

Signed-off-by: Kevin Leimkuhler <[email protected]>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 18, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants