Skip to content

VNet docs: Provide clear instructions for getting debug logs #56068

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ravicious merged 5 commits into from
Jun 26, 2025
Merged

VNet docs: Provide clear instructions for getting debug logs #56068

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ff3f7be
Add info about enabling debug logs in VNet
ravicious Jun 24, 2025
32c2c6d
Mention the diag report
ravicious Jun 24, 2025
c972e3a
VNet docs: Use info from diag report instead of grepping logs
ravicious Jun 24, 2025
f2a7f01
Add evtx to docs/cspell.json
ravicious Jun 25, 2025
5f6fa65
Mention `&& exit` in the docs
ravicious Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/cspell.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -492,6 +492,7 @@
"enzos",
"errcode",
"etcdctl",
"evtx",
"exadata",
"exadatadomain",
"examplecontainer",
Expand Down Expand Up @@ -1087,4 +1088,4 @@
"**/reference/operator-resources/**",
"**/includes/reference/code-blocks-no-cspell/**"
]
}
}
108 changes: 75 additions & 33 deletions docs/pages/connect-your-client/vnet.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,19 +125,14 @@ manually.

### Verifying that VNet receives DNS queries

Start VNet with `tsh vnet -d`. Look at `/var/log/vnet.log` and note the IPv6 and IPv4 CIDR range used by VNet.

```code
From tsh vnet -d:
INFO [VNET] Running Teleport VNet. ipv6_prefix:fd60:67ec:4325:: vnet/vnet.go:317

From /var/log/vnet.log:
INFO Setting an IP route for the VNet. netmask:100.64.0.0/10 vnet/osconfig_darwin.go:47
```
Open Teleport Connect. From the Connections panel in the top left, select VNet. Make sure VNet is
running, then select "Open Diag Report". Note the IPv6 prefix and the IPv4 CIDR range used by VNet.

Send a query for a TCP app available in your cluster, replacing <Var
name="tcp-app.teleport.example.com" /> with the name of your app:

<Tabs>
<TabItem label="macOS">
```code
$ dscacheutil -q host -a name <Var name="tcp-app.teleport.example.com" />
name: tcp-app.teleport.example.com
Expand All @@ -146,11 +141,24 @@ ipv6_address: fd60:67ec:4325::647a:547d
name: tcp-app.teleport.example.com
ip_address: 100.68.51.151
```
</TabItem>
<TabItem label="Windows">
```code
# In PowerShell.
$ Resolve-DnsName <Var name="tcp-app.teleport.example.com" />

Name Type TTL Section IPAddress
---- ---- --- ------- ---------
tcp-app.teleport.example.com AAAA 10 Answer fd60:67ec:4325::647a:547d
tcp-app.teleport.example.com A 10 Answer 100.68.51.151
```
</TabItem>
</Tabs>

The addresses reported by `dscacheutil` should belong to ranges reported by VNet above.
The returned addresses should belong to ranges listed in the VNet diag report.

Querying for anything other than an address of a TCP app should return the address belonging to the
Proxy Service.
Proxy Service. Using macOS as an example:

```code
$ dscacheutil -q host -a name dashboard.teleport.example.com
Expand All @@ -161,38 +169,47 @@ name: dashboard.teleport.example.com
ip_address: 93.184.215.14
```

Querying for both addresses should result in some output being emitted by `tsh vnet -d`.
Querying for any of those hostnames should result in some output being emitted in the debug logs of
VNet (see [Submitting an issue](#submitting-an-issue) on how to enable debug logs).

### Submitting an issue

When [submitting an
issue](https://github.com/gravitational/teleport/issues/new?assignees=&labels=bug,vnet&template=bug_report.md),
make sure to include VNet logs as well as [Teleport Connect
logs](teleport-connect.mdx#submitting-an-issue).
make sure to include a VNet diag report and debug logs from VNet and Teleport Connect.

To save a diag report to a file, open Teleport Connect. From the Connections panel in the top left
select VNet, then "Open Diag Report". In the new tab with the report that was opened click the "Save
Report to File" icon.

You can collect VNet logs using the instructions below:
To collect VNet and Teleport Connect logs use the instructions below:

<Tabs>
<TabItem label="macOS">
Logs from the VNet daemon are sent to [the unified logging system](https://support.apple.com/en-gb/guide/console/welcome/mac).
To enable debug logs in VNet, first stop Teleport Connect and then run the following command. It
enables debug logs just for the next invocation of VNet:

To stream logs:
```code
$ sudo launchctl debug system/com.gravitational.teleport.tsh.vnetd --environment TELEPORT_DEBUG=1
```

Next, start capturing logs from VNet into a file:

```code
$ log stream --predicate 'subsystem ENDSWITH ".vnetd"' --style syslog --level info
$ log stream --predicate 'subsystem ENDSWITH ".vnetd"' --style syslog --level debug > vnet.log
```

To dump logs captured so far to a file:
Then start Teleport Connect using the following command to enable debug logs for Teleport Connect:

```code
$ log show --predicate 'subsystem ENDSWITH ".vnetd"' --style syslog --info > vnet.log
$ open -a "Teleport Connect" --args --connect-debug
```

The logs can also be inspected in [the Console
app](https://support.apple.com/en-gb/guide/console/cnsl1012/1.1/mac/15.0). Info logs are not shown
by default, so make sure to select "Include Info Messages" from the Action menu.
Next, attempt to reproduce the issue with VNet.

At the moment it's not possible to enable debug logs in the VNet daemon.
To gather logs from Teleport Connect, from the app menu select Help → Open Logs Directory which
opens `~/Library/Application Support/Teleport Connect/logs` in Finder. Attach all files together
with `vnet.log` produced in the earlier step.

{/* TODO: DELETE IN 21.0.0 */}
Before version 18.0.0, VNet logs were saved in `/var/log/vnet.log`.
Expand All @@ -206,21 +223,46 @@ $ grep tsh /var/log/com.apple.xpc.launchd/launchd.log
```
</TabItem>
<TabItem label="Windows">
Logs are saved to a custom log in Event Log called Teleport. To browse them, open [Event
Viewer](https://learn.microsoft.com/en-us/shows/inside/event-viewer), select "Applications and
Services Logs" in the sidebar on the left and choose "Teleport".
To enable debug logs in VNet, first stop Teleport Connect. Then in the Start menu look for Command
Prompt and from the right click menu select Run as administrator. The following command enables
debug logs in VNet and immediately closes the admin command prompt to prevent you from starting
Teleport Connect as an admin by mistake.

To save them to a file, select "Save All Events As…" from the sidebar on the right.
```code
$ reg.exe ADD HKLM\SYSTEM\CurrentControlSet\Services\TeleportVNet /v Environment /t REG_MULTI_SZ /d TELEPORT_DEBUG=1 /f && exit
```

Alternatively, you can save them to a file with a PowerShell command:
Next, from the Start menu open the Run app. Execute the following to start Teleport Connect with
debug logs enabled:

```code
$ Get-WinEvent -LogName Teleport -FilterXPath "*[System[Provider[@Name='vnet']]]" -Oldest | Format-Table -Property TimeCreated,LevelDisplayName,Message -Wrap | Out-File vnet.log
$ "%PROGRAMFILES%\Teleport Connect\Teleport Connect.exe" --connect-debug
```

To enable debug logs, search for "Edit the system environment variables" in the Start Menu. Select
"Environment Variables…" and then add a new _system_ variable with the name `TELEPORT_DEBUG` and the
value set to `1`, then restart VNet.
Next, attempt to reproduce the issue with VNet.

Once that's done, execute the following command from the administrator Command Prompt to disable
debug logs in VNet:

```code
$ reg.exe DELETE HKLM\SYSTEM\CurrentControlSet\Services\TeleportVNet /v Environment /f
```

The last step is collecting the logs. Let's start with the VNet logs. From the Start menu, open Event Viewer.
From the sidebar on the left, select Event Viewer (Local) → Applications and Services Logs →
Teleport. From the sidebar on the right, select "Save All Events As…". Save the logs as .evtx file.
If Event Viewer asks about Display Information, choose "No display information".

To gather logs from Teleport Connect, press `Alt` while in the app, then select Help → Open Logs
Directory. This opens `C:\Users\%UserName%\AppData\Roaming\Teleport Connect\logs`. Attach all files
together with the .evtx file from the previous step.

Outside of submitting an issue, VNet logs can be quickly saved to a file with the following
PowerShell command. However, when submitting an issue please attach the .evtx file instead.

```code
$ Get-WinEvent -LogName Teleport -FilterXPath "*[System[Provider[@Name='vnet']]]" -Oldest | Format-Table -Property TimeCreated,LevelDisplayName,Message -Wrap | Out-File vnet.log
```

{/* TODO: DELETE IN 21.0.0 */}
Before version 18.0.0, VNet logs were saved in `C:\Program Files\Teleport Connect\resources\bin\logs.txt`.
Expand Down
Loading