Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions website/content/commands/troubleshoot/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
layout: commands
page_title: 'Commands: Troubleshoot'
description: |
The `consul troubleshoot` provides tools to troubleshoot Consul's service mesh configuration.
Comment thread
malizz marked this conversation as resolved.
Outdated
---

# Consul Troubleshooting

Command: `consul troubleshoot`

Use the `troubleshoot` command to diagnose Consul service mesh configuration or network issues.

## Usage

```text
Usage: consul troubleshoot <subcommand> [options]

# ...

Subcommands:
Comment thread
malizz marked this conversation as resolved.

Subcommands:

proxy Troubleshoots service mesh issues from the current envoy instance
upstreams Get upstream envoy identifiers for the current envoy instance
Comment thread
malizz marked this conversation as resolved.
Outdated
```

For more information, examples, and usage about a subcommand, click on the name
of the subcommand in the sidebar or one of the links below:

- [proxy](/consul/commands/troubleshoot/proxy)
- [upstreams](/consul/commands/troubleshoot/upstreams)
59 changes: 59 additions & 0 deletions website/content/commands/troubleshoot/proxy.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
layout: commands
page_title: 'Commands: Troubleshoot Proxy'
description: |
The `consul troubleshoot proxy` diagnoses Consul service mesh configuration and network issues to an upstream.
Comment thread
malizz marked this conversation as resolved.
Outdated
---

# Consul Troubleshoot Proxy

Command: `consul troubleshoot proxy`

Comment thread
malizz marked this conversation as resolved.
## Usage

Usage: `consul troubleshoot proxy (-upstream-ip <ip>|-upstream-envoy-id <envoy-id>) [options]
Comment thread
malizz marked this conversation as resolved.
Outdated

#### Command Options

- `-envoy-admin-endpoint=<value>` - Envoy Admin endpoint address for the local Envoy instance.
Defaults to `127.0.0.1:19000`.

- `-upstream-ip=<value>` - The IP address of the upstream transparent proxy.
Comment thread
malizz marked this conversation as resolved.
Outdated

- `-upstream-envoy-id=<value>` - The Envoy identifier of the upstream service to be troubleshooted.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#### Command Options
- `-envoy-admin-endpoint=<value>` - Envoy Admin endpoint address for the local Envoy instance.
Defaults to `127.0.0.1:19000`.
- `-upstream-ip=<value>` - The IP address of the upstream transparent proxy.
- `-upstream-envoy-id=<value>` - The Envoy identifier of the upstream service to be troubleshooted.
#### Command Options
- `-envoy-admin-endpoint=<value>` - Envoy admin endpoint address for the local Envoy instance.
Defaults to `127.0.0.1:19000`.
- `-upstream-ip=<value>` - The IP address of the upstream transparent proxy.
- `-upstream-envoy-id=<value>` - The Envoy identifier of the upstream service to troubleshoot.


## Examples

Troubleshoot Consul service mesh configuration and network issues from the current Envoy instance.

```shell-session hideClipboard
Comment thread
malizz marked this conversation as resolved.
Outdated
$ consul troubleshoot proxy -upstream-ip 10.4.6.160

==> Validation
✓ certificates are valid
✓ Envoy has 0 rejected configurations
✓ Envoy has detected 0 connection failure(s)
✓ listener for upstream "backend" found
✓ route for upstream "backend" found
✓ cluster "backend.default.dc1.internal..consul" for upstream "backend" found
✓ healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
✓ cluster "backend2.default.dc1.internal..consul" for upstream "backend" found
! no healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Examples
Troubleshoot Consul service mesh configuration and network issues from the current Envoy instance.
```shell-session hideClipboard
$ consul troubleshoot proxy -upstream-ip 10.4.6.160
==> Validation
✓ certificates are valid
✓ Envoy has 0 rejected configurations
✓ Envoy has detected 0 connection failure(s)
✓ listener for upstream "backend" found
✓ route for upstream "backend" found
✓ cluster "backend.default.dc1.internal..consul" for upstream "backend" found
✓ healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
✓ cluster "backend2.default.dc1.internal..consul" for upstream "backend" found
! no healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
```
## Examples
The following example troubleshoots the Consul service mesh configuration and network issues from the current Envoy instance.
```shell-session hideClipboard
$ consul troubleshoot proxy -upstream-ip 10.4.6.160
==> Validation
✓ certificates are valid
✓ Envoy has 0 rejected configurations
✓ Envoy has detected 0 connection failure(s)
✓ listener for upstream "backend" found
✓ route for upstream "backend" found
✓ cluster "backend.default.dc1.internal..consul" for upstream "backend" found
✓ healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
✓ cluster "backend2.default.dc1.internal..consul" for upstream "backend" found
! no healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
The backticks that end this code sample aren't loading in the preview, just flagging so you're aware.


Troubleshoot Consul service mesh configuration and network issues from the current Envoy instance to the given upstream Envoy identifier.

```shell-session
$ consul-k8s troubleshoot proxy -upstream-envoy-id db
```

```
==> Validation
✓ certificates are valid
✓ Envoy has 0 rejected configurations
✓ Envoy has detected 0 connection failure(s)
! no listener for upstream "db" found
! no route for upstream "backend" found
! no cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
! no healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Troubleshoot Consul service mesh configuration and network issues from the current Envoy instance to the given upstream Envoy identifier.
```shell-session
$ consul-k8s troubleshoot proxy -upstream-envoy-id db
```
```
==> Validation
✓ certificates are valid
✓ Envoy has 0 rejected configurations
✓ Envoy has detected 0 connection failure(s)
! no listener for upstream "db" found
! no route for upstream "backend" found
! no cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
! no healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
The following example troubleshoots the Consul service mesh configuration and network issues between the current Envoy instance and the given upstream Envoy identifier.
```shell-session
$ consul-k8s troubleshoot proxy -upstream-envoy-id db
==> Validation
 ✓ certificates are valid
 ✓ Envoy has 0 rejected configurations
 ✓ Envoy has detected 0 connection failure(s)
 ! no listener for upstream "db" found
 ! no route for upstream "backend" found
 ! no cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
 ! no healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found

The phrasing of "from the current Envoy instance to the given upstream Envoy identifier" was confusing to me. I understand it as troubleshooting issues between these two points. Is that description correct?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

```
36 changes: 36 additions & 0 deletions website/content/commands/troubleshoot/upstreams.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
layout: commands
page_title: 'Commands: Troubleshoot Upstreams'
description: |
The `consul troubleshoot upstream` lists the available upstreams in the Consul service mesh from the current service.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
description: |
The `consul troubleshoot upstream` lists the available upstreams in the Consul service mesh from the current service.
description: >-
The `consul troubleshoot upstream` lists the available upstreams in the Consul service mesh from the current service.

---

# Consul Troubleshoot Upstreams

Command: `consul troubleshoot upstreams`

Comment thread
malizz marked this conversation as resolved.
## Usage

Usage: `consul troubleshoot upstreams [options]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Usage: `consul troubleshoot upstreams [options]
Usage: `consul troubleshoot upstreams [options]`


#### Command Options

- `-envoy-admin-endpoint=<value>` - Envoy Admin endpoint address for the local Envoy instance.
Defaults to `127.0.0.1:19000`.

Comment on lines +16 to +22

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#### Command Options
- `-envoy-admin-endpoint=<value>` - Envoy Admin endpoint address for the local Envoy instance.
Defaults to `127.0.0.1:19000`.
#### Command Options
- `-envoy-admin-endpoint=<value>` - Envoy admin endpoint address for the local Envoy instance.
Defaults to `127.0.0.1:19000`.

## Examples

Display all transparent proxy upstreams in Consul service mesh from the current Envoy instance.
Comment thread
malizz marked this conversation as resolved.

```shell-session hideClipboard
$ consul troubleshoot upstreams

==> Upstreams (explicit upstreams only) (0)

==> Upstreams IPs (transparent proxy only) (1)
[10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]

If you don't see your upstream address or cluster for a transparent proxy upstream:
- Check intentions: Tproxy upstreams are configured based on intentions, make sure you have configured intentions to allow traffic to your upstream.
- You can also check that the right cluster is being dialed by running a DNS lookup for the upstream you are dialing (i.e dig backend.svc.consul). If the address you get from that is missing from the Upstream IPs your proxy may be misconfigured.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Examples
Display all transparent proxy upstreams in Consul service mesh from the current Envoy instance.
```shell-session hideClipboard
$ consul troubleshoot upstreams
==> Upstreams (explicit upstreams only) (0)
==> Upstreams IPs (transparent proxy only) (1)
[10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]
If you don't see your upstream address or cluster for a transparent proxy upstream:
- Check intentions: Tproxy upstreams are configured based on intentions, make sure you have configured intentions to allow traffic to your upstream.
- You can also check that the right cluster is being dialed by running a DNS lookup for the upstream you are dialing (i.e dig backend.svc.consul). If the address you get from that is missing from the Upstream IPs your proxy may be misconfigured.
## Examples
Display all transparent proxy upstreams in Consul service mesh from the current Envoy instance.
```shell-session hideClipboard
$ consul troubleshoot upstreams
==> Upstreams (explicit upstreams only) (0)
==> Upstreams IPs (transparent proxy only) (1)
[10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]
If you don't see your upstream address or cluster for a transparent proxy upstream:
- Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream.
- To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get in response is missing from the list of upstream IPs, your proxy may be misconfigured.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the output of the cli, I will send another PR to update it and reflect this change in the output.

```
110 changes: 110 additions & 0 deletions website/content/docs/k8s/k8s-cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ You can use the following commands with `consul-k8s`.
- [`proxy list`](#proxy-list): List all Pods running proxies managed by Consul.
- [`proxy read`](#proxy-read): Inspect the Envoy configuration for a given Pod.
- [`status`](#status): Check the status of a Consul installation on Kubernetes.
- [`troubleshoot`](#troubleshoot): Troubleshoot Consul service mesh and networking issues from a given pod.
- [`uninstall`](#uninstall): Uninstall Consul deployment.
- [`upgrade`](#upgrade): Upgrade Consul on Kubernetes from an existing installation.
- [`version`](#version): Print the version of the Consul on Kubernetes CLI.
Expand Down Expand Up @@ -490,6 +491,115 @@ $ consul-k8s status
✓ Consul clients healthy (3/3)
```

### `troubleshoot`

The `troubleshoot` command exposes two subcommand for troubleshooting Consul
service mesh and network issues from a given pod.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The `troubleshoot` command exposes two subcommand for troubleshooting Consul
service mesh and network issues from a given pod.
The `troubleshoot` command exposes two subcommands for troubleshooting Consul
service mesh and network issues from a given pod.


- [`troubleshoot upstreams`](#troubleshoot-upstreams): List all Envoy upstreams in Consul service mesh from the given pod.
- [`troubleshoot proxy`](#troubleshoot-proxy): Troubleshoot Consul service mesh configuration and network issues from the given pod to the given upstream.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [`troubleshoot proxy`](#troubleshoot-proxy): Troubleshoot Consul service mesh configuration and network issues from the given pod to the given upstream.
- [`troubleshoot proxy`](#troubleshoot-proxy): Troubleshoot Consul service mesh configuration and network issues between the given pod to the given upstream.


### `troubleshoot upstreams`

```shell-session
$ consul-k8s troubleshoot upstreams -pod <pod> <OPTIONS>
```

| Flag | Description | Default |
| ------------------------------------ | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| <nobr>`-namespace`, `-n`</nobr> | `String` The Kubernetes namespace to list proxies in. | Current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) namespace. |
| <nobr>`-envoy-admin-endpoint`</nobr> | `String` Envoy sidecar address and port | `127.0.0.1:19000` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change


#### Example Commands

Display all transparent proxy upstreams in Consul service mesh from the given pod.

```shell-session
$ consul-k8s troubleshoot upstreams -pod frontend-767ccfc8f9-6f6gx
```

```
==> Upstreams (explicit upstreams only) (0)

==> Upstreams IPs (transparent proxy only) (1)
[10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]

If you don't see your upstream address or cluster for a transparent proxy upstream:
- Check intentions: Tproxy upstreams are configured based on intentions, make sure you have configured intentions to allow traffic to your upstream.
- You can also check that the right cluster is being dialed by running a DNS lookup for the upstream you are dialing (i.e dig backend.svc.consul). If the address you get from that is missing from the Upstream IPs your proxy may be misconfigured.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Display all transparent proxy upstreams in Consul service mesh from the given pod.
```shell-session
$ consul-k8s troubleshoot upstreams -pod frontend-767ccfc8f9-6f6gx
```
```
==> Upstreams (explicit upstreams only) (0)
==> Upstreams IPs (transparent proxy only) (1)
[10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]
If you don't see your upstream address or cluster for a transparent proxy upstream:
- Check intentions: Tproxy upstreams are configured based on intentions, make sure you have configured intentions to allow traffic to your upstream.
- You can also check that the right cluster is being dialed by running a DNS lookup for the upstream you are dialing (i.e dig backend.svc.consul). If the address you get from that is missing from the Upstream IPs your proxy may be misconfigured.
The following example displays all transparent proxy upstreams in Consul service mesh from the given pod.
```shell-session
$ consul-k8s troubleshoot upstreams -pod frontend-767ccfc8f9-6f6gx
==> Upstreams (explicit upstreams only) (0)
==> Upstreams IPs (transparent proxy only) (1)
[10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]
```
If you cannot find the upstream address or cluster for a transparent proxy upstream:
- Check intentions: Tproxy upstreams are configured based on intentions. Make sure you configured intentions to allow traffic to your upstream.
- To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above.

```

Display all explicit upstreams in Consul service mesh from the given pod.
Comment thread
malizz marked this conversation as resolved.
Outdated

```shell-session
$ consul-k8s troubleshoot upstreams -pod client-767ccfc8f9-6f6gx
```

```
Comment thread
malizz marked this conversation as resolved.
Outdated
==> Upstreams (explicit upstreams only) (1)
server
counting

==> Upstreams IPs (transparent proxy only) (0)

If you don't see your upstream address or cluster for a transparent proxy upstream:
- Check intentions: Tproxy upstreams are configured based on intentions, make sure you have configured intentions to allow traffic to your upstream.
- You can also check that the right cluster is being dialed by running a DNS lookup for the upstream you are dialing (i.e dig backend.svc.consul). If the address you get from that is missing from the Upstream IPs your proxy may be misconfigured.
Comment thread
malizz marked this conversation as resolved.
Outdated
```

### `troubleshoot proxy`

```shell-session
$ consul-k8s troubleshoot proxy -pod <pod> -upstream-ip <ip> <OPTIONS>
$ consul-k8s troubleshoot proxy -pod <pod> -upstream-envoy-id <envoy-id> <OPTIONS>
```

| Flag | Description | Default |
| ------------------------------------ | ----------------------------------------------------------| ---------------------------------------------------------------------------------------------------------------------- |
| <nobr>`-namespace`, `-n`</nobr> | `String` The Kubernetes namespace to list proxies in. | Current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) namespace. |
| <nobr>`-envoy-admin-endpoint`</nobr> | `String` Envoy sidecar address and port | `127.0.0.1:19000` |
| <nobr>`-upstream-ip`</nobr> | `String` The IP address of the upstream transparent proxy | |
| <nobr>`-upstream-envoy-id`</nobr> | `String` The Envoy identifier of the upstream | |

#### Example Commands

Troubleshoot Consul service mesh configuration and network issues from the given pod to the given upstream IP.
Comment thread
malizz marked this conversation as resolved.
Outdated

```shell-session
$ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-ip 10.4.6.160
```

```
Comment thread
malizz marked this conversation as resolved.
Outdated
==> Validation
✓ certificates are valid
✓ Envoy has 0 rejected configurations
✓ Envoy has detected 0 connection failure(s)
✓ listener for upstream "backend" found
✓ route for upstream "backend" found
✓ cluster "backend.default.dc1.internal..consul" for upstream "backend" found
✓ healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
✓ cluster "backend2.default.dc1.internal..consul" for upstream "backend" found
! no healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
```

Troubleshoot Consul service mesh configuration and network issues from the given pod to the given upstream Envoy identifier.
Comment thread
malizz marked this conversation as resolved.
Outdated

```shell-session
$ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-envoy-id db
```

```
==> Validation
✓ certificates are valid
✓ Envoy has 0 rejected configurations
✓ Envoy has detected 0 connection failure(s)
! no listener for upstream "db" found
! no route for upstream "backend" found
! no cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
! no healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
```

### `uninstall`

The `uninstall` command removes Consul from Kubernetes.
Expand Down