new docs for consul and consul-k8s troubleshoot command

.changelog/16284.txt

@@ -0,0 +1,3 @@
+```release-note:feature
+cli: adds new CLI commands `consul troubleshoot upstreams` and `consul troubleshoot proxy` to troubleshoot Consul's service mesh configuration and network issues.
+```

website/content/commands/index.mdx

@@ -56,6 +56,7 @@ Available commands are:
     services       Interact with services
     snapshot       Saves, restores and inspects snapshots of Consul server state
     tls            Builtin helpers for creating CAs and certificates
+    troubleshoot   Provides tools to troubleshoot Consul's service mesh configuration
     validate       Validate config files/directories
     version        Prints the Consul version
     watch          Watch for changes in Consul

website/content/commands/troubleshoot/index.mdx

@@ -0,0 +1,31 @@
+---
+layout: commands
+page_title: 'Commands: Troubleshoot'
+description: >-
+  The `consul troubleshoot` command provides tools to troubleshoot Consul's service mesh configuration.
+---
+
+# Consul Troubleshooting
+
+Command: `consul troubleshoot`
+
+Use the `troubleshoot` command to diagnose Consul service mesh configuration or network issues.
+
+## Usage
+
+```text
+Usage: consul troubleshoot <subcommand> [options]
+
+  # ...
+
+Subcommands:
+
+    proxy        Troubleshoots service mesh issues from the current Envoy instance
+    upstreams    Gets upstream Envoy identifiers and IPs configured for the proxy
+```
+
+For more information, examples, and usage about a subcommand, click on the name
+of the subcommand in the sidebar or one of the links below:
+
+- [proxy](/consul/commands/troubleshoot/proxy)
+- [upstreams](/consul/commands/troubleshoot/upstreams)

website/content/commands/troubleshoot/proxy.mdx

@@ -0,0 +1,68 @@
+---
+layout: commands
+page_title: 'Commands: Troubleshoot Proxy'
+description: >-
+  The `consul troubleshoot proxy` diagnoses Consul service mesh configuration and network issues to an upstream.
+---
+
+# Consul Troubleshoot Proxy
+
+Command: `consul troubleshoot proxy`
+
+The `troubleshoot proxy` command diagnoses Consul service mesh configuration and network issues to an upstream.
+
+## Usage
+
+Usage: `consul troubleshoot proxy (-upstream-ip <ip>|-upstream-envoy-id <envoy-id>) [options]`
+This command requires `-envoy-admin-endpoint` or `-upstream-ip` to specify the upstream.
+
+#### Command Options
+
+- `-envoy-admin-endpoint=<value>` - Envoy admin endpoint address for the local Envoy instance.
+Defaults to `127.0.0.1:19000`.
+
+- `-upstream-ip=<value>` -  The IP address of the upstream service; Use when the upstream is reached via a transparent proxy DNS address (KubeDNS or Consul DNS)
+
+- `-upstream-envoy-id=<value>` - The Envoy identifier of the upstream service; Use when the upstream is explicitly configured on the downstream service.
+
+## Examples
+
+The following example illustrates how to troubleshoot Consul service mesh configuration and network issues to an upstream from a source proxy.
+
+  ```shell-session
+  $ consul troubleshoot proxy -upstream-ip 10.4.6.160
+
+  ==> Validation
+   ✓ Certificates are valid
+   ✓ Envoy has 0 rejected configurations
+   ✓ Envoy has detected 0 connection failure(s)
+   ✓ Listener for upstream "backend" found
+   ✓ Route for upstream "backend" found
+   ✓ Cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
+   ✓ Healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
+   ✓ Cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
+   ! No healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
+     -> Check that your upstream service is healthy and running
+     -> Check that your upstream service is registered with Consul
+     -> Check that the upstream proxy is healthy and running
+     -> If you are explicitly configuring upstreams, ensure the name of the upstream is correct
+  ```
+
+The following example troubleshoots Consul service mesh configuration and network issues from the local Envoy instance to the given upstream.
+
+  ```shell-session
+  $ consul-k8s troubleshoot proxy -upstream-envoy-id db
+
+  ==> Validation
+   ✓ Certificates are valid
+   ✓ Envoy has 0 rejected configurations
+   ✓ Envoy has detected 0 connection failure(s)
+   ✓ Listener for upstream "db" found
+   ✓ Route for upstream "db" found
+   ✓ Cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0.consul" for upstream "db" found
+   ✓ Healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0.consul" for upstream "db" found
+
+  If you are still experiencing issues, you can:
+  -> Check intentions to ensure the upstream allows traffic from this source
+  -> If using transparent proxy, ensure DNS resolution is to the same IP you have verified here
+  ```

website/content/commands/troubleshoot/upstreams.mdx

@@ -0,0 +1,37 @@
+---
+layout: commands
+page_title: 'Commands: Troubleshoot Upstreams'
+description: >-
+  The `consul troubleshoot upstreams` lists the available upstreams in the Consul service mesh from the current service.
+---
+
+# Consul Troubleshoot Upstreams
+
+Command: `consul troubleshoot upstreams`
+
+The `troubleshoot upstreams` lists the available upstreams in the Consul service mesh from the current service.
+
+## Usage
+
+Usage: `consul troubleshoot upstreams [options]`
+
+#### Command Options
+
+- `-envoy-admin-endpoint=<value>` - Envoy admin endpoint address for the local Envoy instance.
+Defaults to `127.0.0.1:19000`.
+
+## Examples
+
+Display all transparent proxy upstreams in Consul service mesh from the current Envoy instance.
+
+  ```shell-session
+  $ consul troubleshoot upstreams
+  ==> Upstreams (explicit upstreams only) (0)
+
+  ==> Upstreams IPs (transparent proxy only) (1)
+  [10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]
+
+  If you cannot find the upstream address or cluster for a transparent proxy upstream:
+  - Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream.
+  - To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured.
+  ```

website/content/docs/k8s/k8s-cli.mdx

@@ -33,6 +33,7 @@ You can use the following commands with `consul-k8s`.
       - [`proxy list`](#proxy-list): List all Pods running proxies managed by Consul.
       - [`proxy read`](#proxy-read): Inspect the Envoy configuration for a given Pod.
   - [`status`](#status): Check the status of a Consul installation on Kubernetes.
+  - [`troubleshoot`](#troubleshoot): Troubleshoot Consul service mesh and networking issues from a given pod.
   - [`uninstall`](#uninstall): Uninstall Consul deployment.
   - [`upgrade`](#upgrade): Upgrade Consul on Kubernetes from an existing installation.
   - [`version`](#version): Print the version of the Consul on Kubernetes CLI.
@@ -490,6 +491,106 @@ $ consul-k8s status
  ✓ Consul clients healthy (3/3)
 ```
 
+### `troubleshoot`
+
+The `troubleshoot` command exposes two subcommands for troubleshooting Consul
+service mesh and network issues from a given pod.
+
+- [`troubleshoot upstreams`](#troubleshoot-upstreams): List all Envoy upstreams in Consul service mesh from the given pod.
+- [`troubleshoot proxy`](#troubleshoot-proxy): Troubleshoot Consul service mesh configuration and network issues between the given pod and the given upstream.
+
+### `troubleshoot upstreams`
+
+```shell-session
+$ consul-k8s troubleshoot upstreams -pod <pod> <OPTIONS>
+```
+
+| Flag                                 | Description                                           | Default                                                                                                                |
+| ------------------------------------ | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
+| <nobr>`-namespace`, `-n`</nobr>      | `String` The Kubernetes namespace to list proxies in. | Current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) namespace. |
+| <nobr>`-envoy-admin-endpoint`</nobr> | `String` Envoy sidecar address and port               | `127.0.0.1:19000`                                                                                                      |
+
+#### Example Commands
+
+The following example displays all transparent proxy upstreams in Consul service mesh from the given pod.
+
+  ```shell-session
+  $ consul-k8s troubleshoot upstreams -pod frontend-767ccfc8f9-6f6gx
+
+  ==> Upstreams (explicit upstreams only) (0)
+
+  ==> Upstreams IPs (transparent proxy only) (1)
+  [10.4.6.160 240.0.0.3] true map[backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul]
+
+  If you cannot find the upstream address or cluster for a transparent proxy upstream:
+  - Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream.
+  - To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured.
+  ```
+
+The following example displays all explicit upstreams from the given pod in the Consul service mesh.
+
+  ```shell-session
+  $ consul-k8s troubleshoot upstreams -pod client-767ccfc8f9-6f6gx
+
+  ==> Upstreams (explicit upstreams only) (1)
+  server
+  counting
+
+  ==> Upstreams IPs (transparent proxy only) (0)
+
+  If you cannot find the upstream address or cluster for a transparent proxy upstream:
+  - Check intentions: Tproxy upstreams are configured based on intentions. Make sure you have configured intentions to allow traffic to your upstream.
+  - To check that the right cluster is being dialed, run a DNS lookup for the upstream you are dialing. For example, run `dig backend.svc.consul` to return the IP address for the `backend` service. If the address you get from that is missing from the upstream IPs, it means that your proxy may be misconfigured.
+  ```
+
+### `troubleshoot proxy`
+
+```shell-session
+$ consul-k8s troubleshoot proxy -pod <pod> -upstream-ip <ip> <OPTIONS>
+$ consul-k8s troubleshoot proxy -pod <pod> -upstream-envoy-id <envoy-id> <OPTIONS>
+```
+
+| Flag                                 | Description                                           | Default                                                                                                                |
+| ------------------------------------ | ----------------------------------------------------------| ---------------------------------------------------------------------------------------------------------------------- |
+| <nobr>`-namespace`, `-n`</nobr>      | `String` The Kubernetes namespace to list proxies in.     | Current [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) namespace. |
+| <nobr>`-envoy-admin-endpoint`</nobr> | `String` Envoy sidecar address and port                   | `127.0.0.1:19000`                                                                                                               |
+| <nobr>`-upstream-ip`</nobr>          | `String` The IP address of the upstream transparent proxy |                                                                                                                |
+| <nobr>`-upstream-envoy-id`</nobr>    | `String` The Envoy identifier of the upstream             |                                                                                                                |
+
+#### Example Commands
+
+The following example troubleshoots the Consul service mesh configuration and network issues between the given pod and the given upstream IP.
+
+  ```shell-session
+  $ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-ip 10.4.6.160
+
+  ==> Validation
+   ✓ certificates are valid
+   ✓ Envoy has 0 rejected configurations
+   ✓ Envoy has detected 0 connection failure(s)
+   ✓ listener for upstream "backend" found
+   ✓ route for upstream "backend" found
+   ✓ cluster "backend.default.dc1.internal..consul" for upstream "backend" found
+   ✓ healthy endpoints for cluster "backend.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
+   ✓ cluster "backend2.default.dc1.internal..consul" for upstream "backend" found
+   ! no healthy endpoints for cluster "backend2.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "backend" found
+  ```
+
+The following example troubleshoots the Consul service mesh configuration and network issues between the given pod and the given upstream.
+
+  ```shell-session
+  $ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-envoy-id db
+
+  ==> Validation
+   ✓ certificates are valid
+   ✓ Envoy has 0 rejected configurations
+   ✓ Envoy has detected 0 connection failure(s)
+   ! no listener for upstream "db" found
+   ! no route for upstream "backend" found
+   ! no cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
+   ! no healthy endpoints for cluster "db.default.dc1.internal.e08fa6d6-e91e-dfe0-f6e1-ba097a828e31.consul" for upstream "db" found
+  ```
+
 ### `uninstall`
 
 The `uninstall` command removes Consul from Kubernetes.

website/data/commands-nav-data.json

@@ -528,6 +528,23 @@
       }
     ]
   },
+  {
+    "title": "troubleshoot",
+    "routes": [
+      {
+        "title": "Overview",
+        "path": "troubleshoot"
+      },
+      {
+        "title": "upstreams",
+        "path": "troubleshoot/upstreams"
+      },
+      {
+        "title": "proxy",
+        "path": "troubleshoot/proxy"
+      }
+    ]
+  },
   {
     "title": "validate",
     "path": "validate"