Error 500 when visiting /machines endpoint

[spomata]

Hi, I managed to run the webui container in a rootless podman setup / reverse proxy via caddy (I can pass you the details so that they can be added to documentation).

On visiting the machines page, it crashes constantly. Pasting the output here:

[2023-03-15 12:41:53,102] ERROR in app: Exception on /machines [GET] Traceback (most recent call last): File "/app/.venv/lib/python3.11/site-packages/flask/app.py", line 2528, in wsgi_app response = self.full_dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/flask/app.py", line 1825, in full_dispatch_request rv = self.handle_user_exception(e) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/flask/app.py", line 1823, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/flask/app.py", line 1799, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/server.py", line 115, in decorated return view_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/server.py", line 180, in machines_page cards = renderer.render_machines_cards() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/renderer.py", line 425, in render_machines_cards content = content+str(sorted_machines[index]) ~~~~~~~~~~~~~~~^^^^^^^ KeyError: 0

[iFargle]

Do you have any machines added to Headscale currently?

[iFargle]

These are a bit weird to troubleshoot... I need to figure out how to get the futures threads to log to console or something. I'm pretty sure it's failing there somewhere -- The only thing I can think of right off the bat is no machines in Headscale. If that's the case, it should be a pretty easy fix.

[spomata]

Do you have any machines added to Headscale currently?

Yes there's about 40 machines added to the setup.
Let me know if any command issued on the container could help debugging.

[iFargle]

Another possibility:
Do you have any weird content (like emoji?) in machine names / tags?
It's failing to get one of these:

headscale-webui/renderer.py

Lines 369 to 390 in 110ee90

	'machines_card.html',
	given_name = machine["givenName"],
	machine_id = machine["id"],
	hostname = machine["name"],
	ns_name = machine["user"]["name"],
	ns_id = machine["user"]["id"],
	ns_created = machine["user"]["createdAt"],
	last_seen = str(last_seen_print),
	last_update = str(last_update_print),
	machine_ips = Markup(machine_ips),
	advertised_routes = Markup(routes),
	exit_node_badge = Markup(exit_node_badge),
	status_badge = Markup(status_badge),
	user_badge = Markup(user_badge),
	last_update_time = str(last_update_time),
	last_seen_time = str(last_seen_time),
	created_time = str(created_time),
	expiry_time = str(expiry_time),
	preauth_key = str(preauth_key),
	expiration_badge = Markup(expiration_badge),
	machine_tags = Markup(tags),
	)))

I think I know an ok-ish way of doing some troubleshooting. It will require a new build though.
When I get some time (hopefully tomorrow) i'll push a new build and have you do some testing

[spomata]

Ok.

At a glance, all tags have at most a ":" (colon) character, while machines are standard (although some phones have a whitespace in their name).

[spomata]

I was checking the code and noted that the section you pointed above is before the future threads execution, while in my log the "Finished futures" message is being printed correctly and the issue might lie around line 425.

[iFargle]

From experience it's usually caused by a failure in futures. You should see a log for each future that ran. It doesn't populate sorted_machines properly. I'm redoing my builds at the moment. I should have a build ready for you to test in the next few hours.

[iFargle]

Apologies for the delay. Pull :v0.5.4 and set the environment variable LOG_LEVEL to debug, it should give more detailed info on where it's failing! (It will also render the machines page much more slowly, as it doesn't do threading -- just runs through all of your machines serially)

[spomata]

Apologies for the delay. Pull :v0.5.4 and set the environment variable LOG_LEVEL to debug, it should give more detailed info on where it's failing! (It will also render the machines page much more slowly, as it doesn't do threading -- just runs through all of your machines serially)

No problem at all. Here's the log (spoiler: some date/time conversion from UTC gone wrong, maybe our headscale config is missing a parameter?)

[2023-03-17 11:49:59,500] ERROR in app: Exception on /machines [GET] Traceback (most recent call last): File "/app/.venv/lib/python3.11/site-packages/flask/app.py", line 2528, in wsgi_app response = self.full_dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/flask/app.py", line 1825, in full_dispatch_request rv = self.handle_user_exception(e) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/flask/app.py", line 1823, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/flask/app.py", line 1799, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/server.py", line 115, in decorated return view_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/server.py", line 180, in machines_page cards = renderer.render_machines_cards() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/renderer.py", line 411, in render_machines_cards for idx in iterable: thread_machine_content(machines_list["machines"][idx], machine_content, idx) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/renderer.py", line 339, in thread_machine_content expiry_local = expiry_parse.astimezone(timezone) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/.venv/lib/python3.11/site-packages/pytz/tzinfo.py", line 201, in fromutc return (dt + inf[0]).replace(tzinfo=self._tzinfos[inf]) ~~~^~~~~~~~ OverflowError: date value out of range

[iFargle]

What TZ do you have set?
It defaults to UTC if the "TZ" environment variable isn't set

[spomata]

I'm setting Europe/Madrid via the podman run command.

podman run -e TZ=Europe/Madrid ...

[iFargle]

Sweet -- I get the same error. Let me fiddle for a bit and I'll get back to you!

[iFargle]

I think I figured it out.
Try ghcr.io/ifargle/headscale-webui:testing

[iFargle]

So the problem was:
Your expiration times were likely 0001-01-01T00:00:00Z.
When it tried to take the Europe/Madrid TZ offset of -1, we entered negative time!
And PyTZ had no idea what to do.

Should be working now!

[spomata]

Yep, it works indeed!
Good to know, although I think those expiration times are pretty common for nodes which have been onboarded to headscale with an auth token.
All our non-human nodes in the headscale network have such feature.