Error: cannot convert to safe number

[xolott]

I get a spinning wheel when accessing Hubble UI. The developer console shows this error:

stream.ts:223 Unknown error:  Error: cannot convert to safe number
    at si.toNumber (pb-long.js:50:19)
    at t.value (observer_pb.ts:1340:56)
    at t.value (observer_pb.ts:1263:45)
    at t.value (status_pb.ts:126:54)
    at t.value (notifications_pb.ts:141:51)
    at t.value (ui_pb.ts:1609:52)
    at t.fromBinary (message-type.js:84:21)
    at control-stream.ts:42:50
    at emitter.ts:82:33
    at Set.forEach (<anonymous>)

Backend logs:

time="2024-11-19T23:48:20Z" level=info msg="GetEventsRequest parsed" channelId=85c23770ab166c36 component=APIServer handler=ServiceMapStream req="event_types:FLOWS event_types:SERVICE_STATE event_types:SERVICE_LINK_STATE blacklist:{flow_filter:{source_label:\"reserved:unknown\" source_label:\"reserved:host\" source_label:\"k8s:k8s-app=kube-dns\" source_label:\"reserved:remote-node\" source_label:\"k8s:app=prometheus\"}} blacklist:{flow_filter:{destination_label:\"reserved:unknown\" destination_label:\"reserved:host\" destination_label:\"reserved:remote-node\" destination_label:\"k8s:app=prometheus\"}} blacklist:{flow_filter:{destination_label:\"k8s:k8s-app=kube-dns\" destination_port:\"53\"}} blacklist:{flow_filter:{source_fqdn:\"*.cluster.local*\"}} blacklist:{flow_filter:{destination_fqdn:\"*.cluster.local*\"}} whitelist:{flow_filter:{reply:false}}"
time="2024-11-19T23:48:30Z" level=info msg="watcher is running" channelId=cd28d7afa5b20087 component=ControlStream.NSWatcher handler=ControlStream ns-kind=k8s
time="2024-11-19T23:48:30Z" level=info msg="running data fetch loop" StatusChecker=0xc00150cd80 channelId=cd28d7afa5b20087 component=StatusChecker handler=ControlStream
time="2024-11-19T23:48:31Z" level=info msg="GetEventsRequest parsed" channelId=cfbe3755efbbeceb component=APIServer handler=ServiceMapStream req="event_types:FLOWS event_types:SERVICE_STATE event_types:SERVICE_LINK_STATE blacklist:{flow_filter:{source_label:\"reserved:unknown\" source_label:\"reserved:host\" source_label:\"k8s:k8s-app=kube-dns\" source_label:\"reserved:remote-node\" source_label:\"k8s:app=prometheus\"}} blacklist:{flow_filter:{destination_label:\"reserved:unknown\" destination_label:\"reserved:host\" destination_label:\"reserved:remote-node\" destination_label:\"k8s:app=prometheus\"}} blacklist:{flow_filter:{destination_label:\"k8s:k8s-app=kube-dns\" destination_port:\"53\"}} blacklist:{flow_filter:{source_fqdn:\"*.cluster.local*\"}} blacklist:{flow_filter:{destination_fqdn:\"*.cluster.local*\"}} whitelist:{flow_filter:{reply:false}}"
time="2024-11-19T23:48:31Z" level=error msg="onTick failed" channelId=9dae0a20031fc572 component=StatusChecker error="context canceled" handler=ControlStream
time="2024-11-19T23:48:31Z" level=error msg="failed to send NSWatcher response" channelId=9dae0a20031fc572 component=APIServer error="channel closed" handler=ControlStream
time="2024-11-19T23:48:31Z" level=info msg="watcher is stopped\n" channelId=9dae0a20031fc572 component=ControlStream.NSWatcher handler=ControlStream ns-kind=k8s
time="2024-11-19T23:48:31Z" level=info msg="FlowStream has been stopped" FlowStream=0xc000466d00 component=FlowStream hubble-client=relay
time="2024-11-19T23:48:31Z" level=info msg="FlowStream has been stopped" FlowStream=0xc000466d00 component=FlowStream hubble-client=relay
time="2024-11-19T23:48:41Z" level=error msg="onTick failed" channelId=cd28d7afa5b20087 component=StatusChecker error="context canceled" handler=ControlStream
time="2024-11-19T23:48:41Z" level=info msg="FlowStream has been stopped" FlowStream=0xc001213880 component=FlowStream hubble-client=relay
time="2024-11-19T23:48:41Z" level=info msg="FlowStream has been stopped" FlowStream=0xc001213880 component=FlowStream hubble-client=relay
time="2024-11-19T23:48:41Z" level=error msg="failed to send NSWatcher response" channelId=cd28d7afa5b20087 component=APIServer error="channel closed" handler=ControlStream
time="2024-11-19T23:48:41Z" level=info msg="watcher is stopped\n" channelId=cd28d7afa5b20087 component=ControlStream.NSWatcher handler=ControlStream ns-kind=k8s
time="2024-11-19T23:49:41Z" level=info msg="FlowStream has been stopped" FlowStream=0xc00150de00 component=FlowStream hubble-client=relay
time="2024-11-19T23:49:41Z" level=info msg="FlowStream has been stopped" FlowStream=0xc00150de00 component=FlowStream hubble-client=relay
time="2024-11-19T23:49:43Z" level=info msg="GetEventsRequest parsed" channelId=cfbe3755efbbeceb component=APIServer handler=ServiceMapStream req=
time="2024-11-19T23:49:43Z" level=info msg="Requested events are empty, terminating..." channelId=cfbe3755efbbeceb component=APIServer handler=ServiceMapStream
time="2024-11-19T23:49:44Z" level=info msg="GetEventsRequest parsed" channelId=a093e43c6e7f8002 component=APIServer handler=ServiceMapStream req="event_types:FLOWS event_types:SERVICE_STATE event_types:SERVICE_LINK_STATE blacklist:{flow_filter:{source_label:\"reserved:unknown\" source_label:\"reserved:host\" source_label:\"k8s:k8s-app=kube-dns\" source_label:\"reserved:remote-node\" source_label:\"k8s:app=prometheus\"}} blacklist:{flow_filter:{destination_label:\"reserved:unknown\" destination_label:\"reserved:host\" destination_label:\"reserved:remote-node\" destination_label:\"k8s:app=prometheus\"}} blacklist:{flow_filter:{destination_label:\"k8s:k8s-app=kube-dns\" destination_port:\"53\"}} blacklist:{flow_filter:{source_fqdn:\"*.cluster.local*\"}} blacklist:{flow_filter:{destination_fqdn:\"*.cluster.local*\"}} whitelist:{flow_filter:{reply:false}}"
time="2024-11-19T23:51:10Z" level=info msg="watcher is running" channelId=23d724233950a9ce component=ControlStream.NSWatcher handler=ControlStream ns-kind=k8s
time="2024-11-19T23:51:10Z" level=info msg="running data fetch loop" StatusChecker=0xc001edc100 channelId=23d724233950a9ce component=StatusChecker handler=ControlStream
time="2024-11-19T23:51:10Z" level=info msg="GetEventsRequest parsed" channelId=d44f1a1fed135295 component=APIServer handler=ServiceMapStream req="event_types:FLOWS event_types:SERVICE_STATE event_types:SERVICE_LINK_STATE blacklist:{flow_filter:{source_label:\"reserved:unknown\" source_label:\"reserved:host\" source_label:\"k8s:k8s-app=kube-dns\" source_label:\"reserved:remote-node\" source_label:\"k8s:app=prometheus\"}} blacklist:{flow_filter:{destination_label:\"reserved:unknown\" destination_label:\"reserved:host\" destination_label:\"reserved:remote-node\" destination_label:\"k8s:app=prometheus\"}} blacklist:{flow_filter:{destination_label:\"k8s:k8s-app=kube-dns\" destination_port:\"53\"}} blacklist:{flow_filter:{source_fqdn:\"*.cluster.local*\"}} blacklist:{flow_filter:{destination_fqdn:\"*.cluster.local*\"}} whitelist:{flow_filter:{reply:false}}"
time="2024-11-19T23:51:21Z" level=error msg="onTick failed" channelId=23d724233950a9ce component=StatusChecker error="context canceled" handler=ControlStream
time="2024-11-19T23:51:21Z" level=info msg="FlowStream has been stopped" FlowStream=0xc000466d00 component=FlowStream hubble-client=relay
time="2024-11-19T23:51:21Z" level=info msg="FlowStream has been stopped" FlowStream=0xc000466d00 component=FlowStream hubble-client=relay
time="2024-11-19T23:51:21Z" level=error msg="failed to send NSWatcher response" channelId=23d724233950a9ce component=APIServer error="channel closed" handler=ControlStream
time="2024-11-19T23:51:21Z" level=info msg="watcher is stopped\n" channelId=23d724233950a9ce component=ControlStream.NSWatcher handler=ControlStream ns-kind=k8s

Hubble UI: v0.13.1
Hubble Relay: v1.15.7

[wgoyer]

Running into the same issue with cililum/hubble v1.16.1. Can be worked around by setting the query param on the url manually: https://<host>/?namespace=kube-system

[mattslane]

For us, it always seems to error on these lines in observer_pb.ts (1339 in the debugger) suggesting that some data is invalid.

case /* uint64 uptime_ns */ 6:
     message.uptimeNs = reader.uint64().toNumber();`

We're running karpenter on EKS and most of our nodes have version: v1.30.9-eks-5d632ec except for 1-2 nodes used to bootstrap the cluster which are on v1.30.4-eks-a737599 and have large uptimes (120d) etc. When I look at the value of message.name in the debugger it always corresponds to one of these old nodes.

When I recycle these old nodes, hubble starts working again suggesting that the uptime of the node doesn't convert nicely to a unint64.

However 120 days to Ns is roughly an e+16 and the max uint64 is roughly an e+19 I believe so this is likely an error in parsing the number somewhere.