nats cli : context deadline exceeded

[tilakraj9560]

Observed behavior

NATS CLI commands are failing with "context deadline exceeded" errors when attempting to interact with NATS JetStream server operations.

Its a standalone server. Looking at k8s pods, even the readiness probes fail.

Warning Unhealthy 3m44s (x5264 over 12h) kubelet Readiness probe failed: Get "http://<omitted>:8222/healthz?js-server-only=true": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

~ # nats stream ls
nats: error: could not list streams: context deadline exceeded
~ # nats stream info
? Select a Stream DLEventStream
nats: error: could not request Stream info: context deadline exceeded

Logs:

[7] 2025/09/10 05:21:23.500607 [DBG] 10.100.0.165:48248 - cid:471 - Delaying PING due to remote client data or ping 49s ago
[7] 2025/09/10 05:21:46.572032 [DBG] 10.100.0.165:44622 - cid:843323 - Client Ping Timer
[7] 2025/09/10 05:21:46.572052 [DBG] 10.100.0.165:44622 - cid:843323 - Delaying PING due to remote client data or ping 2s ago
[7] 2025/09/10 05:22:21.967505 [DBG] 10.100.0.161:54834 - cid:845421 - Client Ping Timer
[7] 2025/09/10 05:22:21.967530 [DBG] 10.100.0.161:54834 - cid:845421 - Delaying PING due to remote client data or ping 1s ago
[7] 2025/09/10 05:22:31.542021 [DBG] 10.100.0.162:47600 - cid:845505 - Client connection created
[7] 2025/09/10 05:22:31.686549 [DBG] 10.100.0.162:54126 - cid:845502 - Client Ping Timer
[7] 2025/09/10 05:22:33.142562 [DBG] 10.100.0.165:33492 - cid:845355 - Client Ping Timer
[7] 2025/09/10 05:22:33.142581 [DBG] 10.100.0.165:33492 - cid:845355 - Delaying PING due to remote client data or ping 1m36s ago
[7] 2025/09/10 05:22:33.858581 [DBG] 10.100.0.162:47600 - cid:845505 - Client Ping Timer
[7] 2025/09/10 05:22:34.790239 [DBG] 10.100.0.162:54126 - cid:845502 - Error flushing: writev tcp 10.100.0.155:4222->10.100.0.162:54126: writev: broken pipe
[7] 2025/09/10 05:22:34.790260 [DBG] 10.100.0.162:54126 - cid:845502 - Client connection closed: Write Error
[7] 2025/09/10 05:22:53.270424 [DBG] 10.100.0.132:39030 - cid:845503 - Client Ping Timer
[7] 2025/09/10 05:23:01.461325 [DBG] 10.100.0.132:51204 - cid:845504 - Client Ping Timer
[7] 2025/09/10 05:23:08.413105 [DBG] 10.100.0.132:39166 - cid:845506 - Client connection created
[7] 2025/09/10 05:23:10.720927 [DBG] 10.100.0.132:39166 - cid:845506 - Client Ping Timer
[7] 2025/09/10 05:23:23.501479 [DBG] 10.100.0.165:48248 - cid:471 - Client Ping Timer
[7] 2025/09/10 05:23:23.501500 [DBG] 10.100.0.165:48248 - cid:471 - Delaying PING due to remote client data or ping 49s ago
[7] 2025/09/10 05:23:46.572529 [DBG] 10.100.0.165:44622 - cid:843323 - Client Ping Timer
[7] 2025/09/10 05:23:46.572561 [DBG] 10.100.0.165:44622 - cid:843323 - Delaying PING due to remote client data or ping 2s ago
[7] 2025/09/10 05:24:21.968430 [DBG] 10.100.0.161:54834 - cid:845421 - Client Ping Timer
[7] 2025/09/10 05:24:21.968465 [DBG] 10.100.0.161:54834 - cid:845421 - Delaying PING due to remote client data or ping 1s ago
[7] 2025/09/10 05:24:33.143048 [DBG] 10.100.0.165:33492 - cid:845355 - Client Ping Timer
[7] 2025/09/10 05:24:33.143085 [DBG] 10.100.0.165:33492 - cid:845355 - Delaying PING due to remote client data or ping 1m36s ago
[7] 2025/09/10 05:24:33.859024 [DBG] 10.100.0.162:47600 - cid:845505 - Client Ping Timer
[7] 2025/09/10 05:24:33.859052 [DBG] 10.100.0.162:47600 - cid:845505 - Delaying PING due to remote client data or ping 1m39s ago
[7] 2025/09/10 05:24:53.270853 [DBG] 10.100.0.132:39030 - cid:845503 - Client Ping Timer
[7] 2025/09/10 05:24:53.270889 [DBG] 10.100.0.132:39030 - cid:845503 - Stale Client Connection - Closing
[7] 2025/09/10 05:24:53.270910 [DBG] 10.100.0.132:39030 - cid:845503 - Client connection closed: Stale Connection
[7] 2025/09/10 05:24:53.271025 [DBG] 10.100.0.132:39030 - cid:845503 - Error flushing: writev tcp 10.100.0.155:4222->10.100.0.132:39030: writev: broken pipe
[7] 2025/09/10 05:25:01.462163 [DBG] 10.100.0.132:51204 - cid:845504 - Client Ping Timer
[7] 2025/09/10 05:25:01.462183 [DBG] 10.100.0.132:51204 - cid:845504 - Stale Client Connection - Closing
[7] 2025/09/10 05:25:01.462205 [DBG] 10.100.0.132:51204 - cid:845504 - Client connection closed: Stale Connection
[7] 2025/09/10 05:25:01.462312 [DBG] 10.100.0.132:51204 - cid:845504 - Error flushing: writev tcp 10.100.0.155:4222->10.100.0.132:51204: writev: broken pipe
[7] 2025/09/10 05:25:10.721264 [DBG] 10.100.0.132:39166 - cid:845506 - Client Ping Timer
[7] 2025/09/10 05:25:23.502234 [DBG] 10.100.0.165:48248 - cid:471 - Client Ping Timer
[7] 2025/09/10 05:25:23.502252 [DBG] 10.100.0.165:48248 - cid:471 - Delaying PING due to remote client data or ping 49s ago
[7] 2025/09/10 05:25:46.572846 [DBG] 10.100.0.165:44622 - cid:843323 - Client Ping Timer
[7] 2025/09/10 05:25:46.572866 [DBG] 10.100.0.165:44622 - cid:843323 - Delaying PING due to remote client data or ping 2s ago
[7] 2025/09/10 05:26:21.969462 [DBG] 10.100.0.161:54834 - cid:845421 - Client Ping Timer
[7] 2025/09/10 05:26:21.969482 [DBG] 10.100.0.161:54834 - cid:845421 - Delaying PING due to remote client data or ping 1s ago
[7] 2025/09/10 05:26:33.143722 [DBG] 10.100.0.165:33492 - cid:845355 - Client Ping Timer
[7] 2025/09/10 05:26:33.143743 [DBG] 10.100.0.165:33492 - cid:845355 - Delaying PING due to remote client data or ping 1m36s ago
[7] 2025/09/10 05:26:33.860051 [DBG] 10.100.0.162:47600 - cid:845505 - Client Ping Timer
[7] 2025/09/10 05:27:10.722372 [DBG] 10.100.0.132:39166 - cid:845506 - Client Ping Timer
[7] 2025/09/10 05:27:10.722395 [DBG] 10.100.0.132:39166 - cid:845506 - Stale Client Connection - Closing
[7] 2025/09/10 05:27:10.722413 [DBG] 10.100.0.132:39166 - cid:845506 - Client connection closed: Stale Connection
[7] 2025/09/10 05:27:10.722502 [DBG] 10.100.0.132:39166 - cid:845506 - Error flushing: writev tcp 10.100.0.155:4222->10.100.0.132:39166: writev: broken pipe
[7] 2025/09/10 05:27:23.502855 [DBG] 10.100.0.165:48248 - cid:471 - Client Ping Timer
[7] 2025/09/10 05:27:23.502880 [DBG] 10.100.0.165:48248 - cid:471 - Delaying PING due to remote client data or ping 49s ago
[7] 2025/09/10 05:27:46.573156 [DBG] 10.100.0.165:44622 - cid:843323 - Client Ping Timer
[7] 2025/09/10 05:27:46.573174 [DBG] 10.100.0.165:44622 - cid:843323 - Delaying PING due to remote client data or ping 2s ago
[7] 2025/09/10 05:28:21.970105 [DBG] 10.100.0.161:54834 - cid:845421 - Client Ping Timer
[7] 2025/09/10 05:28:21.970138 [DBG] 10.100.0.161:54834 - cid:845421 - Delaying PING due to remote client data or ping 1s ago
[7] 2025/09/10 05:28:33.144567 [DBG] 10.100.0.165:33492 - cid:845355 - Client Ping Timer
[7] 2025/09/10 05:28:33.144586 [DBG] 10.100.0.165:33492 - cid:845355 - Delaying PING due to remote client data or ping 1m36s ago
[7] 2025/09/10 05:28:33.623774 [DBG] 10.100.0.162:37480 - cid:845507 - Client connection created
[7] 2025/09/10 05:28:33.861019 [DBG] 10.100.0.162:47600 - cid:845505 - Client Ping Timer
[7] 2025/09/10 05:28:35.543459 [DBG] 10.100.0.162:47600 - cid:845505 - Error flushing: writev tcp 10.100.0.155:4222->10.100.0.162:47600: writev: broken pipe
[7] 2025/09/10 05:28:35.543490 [DBG] 10.100.0.162:47600 - cid:845505 - Client connection closed: Write Error
[7] 2025/09/10 05:28:35.782397 [DBG] 10.100.0.162:37480 - cid:845507 - Client Ping Timer
[7] 2025/09/10 05:29:23.503515 [DBG] 10.100.0.165:48248 - cid:471 - Client Ping Timer
[7] 2025/09/10 05:29:23.503536 [DBG] 10.100.0.165:48248 - cid:471 - Delaying PING due to remote client data or ping 49s ago
[7] 2025/09/10 05:29:46.573371 [DBG] 10.100.0.165:44622 - cid:843323 - Client Ping Timer
[7] 2025/09/10 05:29:46.573397 [DBG] 10.100.0.165:44622 - cid:843323 - Delaying PING due to remote client data or ping 2s ago
[7] 2025/09/10 05:29:56.455776 [DBG] 10.100.0.132:56296 - cid:845508 - Client connection created
[7] 2025/09/10 05:29:58.659972 [DBG] 10.100.0.132:56296 - cid:845508 - Client Ping Timer
[7] 2025/09/10 05:30:21.970527 [DBG] 10.100.0.161:54834 - cid:845421 - Client Ping Timer
[7] 2025/09/10 05:30:33.145267 [DBG] 10.100.0.165:33492 - cid:845355 - Client Ping Timer
[7] 2025/09/10 05:30:33.145295 [DBG] 10.100.0.165:33492 - cid:845355 - Delaying PING due to remote client data or ping 1m36s ago
[7] 2025/09/10 05:30:35.783301 [DBG] 10.100.0.162:37480 - cid:845507 - Client Ping Timer
[7] 2025/09/10 05:30:35.783320 [DBG] 10.100.0.162:37480 - cid:845507 - Delaying PING due to remote client data or ping 1m40s ago
[7] 2025/09/10 05:30:37.272843 [DBG] 10.100.0.132:46654 - cid:845509 - Client connection created
[7] 2025/09/10 05:30:39.535839 [DBG] 10.100.0.132:46654 - cid:845509 - Client Ping Timer
[7] 2025/09/10 05:30:50.336656 [DBG] 10.100.0.132:35022 - cid:845510 - Client connection created
[7] 2025/09/10 05:30:52.542650 [DBG] 10.100.0.132:35022 - cid:845510 - Client Ping Timer
[7] 2025/09/10 05:30:58.659186 [DBG] 10.100.0.132:37404 - cid:845511 - Client connection created
[7] 2025/09/10 05:31:01.043017 [DBG] 10.100.0.132:37404 - cid:845511 - Client Ping Timer
[7] 2025/09/10 05:31:23.503842 [DBG] 10.100.0.165:48248 - cid:471 - Client Ping Timer
[7] 2025/09/10 05:31:23.503863 [DBG] 10.100.0.165:48248 - cid:471 - Delaying PING due to remote client data or ping 49s ago

conf file:

# NATS Clients Port
port: 4222

# PID file shared with configuration reloader.
pid_file: "/var/run/nats/nats.pid"

# System Account and Users Configuration

accounts: {
  SYS: {
    users: [
      {user: "<>", password: "<>"}
    ]
  }
}
system_account: SYS

###############
#             #
# Monitoring  #
#             #
###############
http: 8222
server_name:$POD_NAME
###################################
#                                 #
# NATS JetStream                  #
#                                 #
###################################
jetstream {
  store_dir: /var/lib/nats/store
}
debug: true
logtime: true
connect_error_reports: 3600
max_payload: 2MB/

Expected behavior

nats cli commands should run.

Server and client version

Server: 2.10.29-alpine
Client: v0.2.3

Host environment

No response

Steps to reproduce

No response

[neilalexander]

Is this a one-off or is it happening regularly? 2.10.29 is quite an old version now, can you please try with 2.11.9?

[tilakraj9560]

Hi @neilalexander , I’m not sure if I’ll be able to reproduce the issue on newer versions. However, it has been occurring consistently for over 12h, and the system has not been able to recover.

So if i do nats stream info, it does give me streams to select but once it is selected it times out. Not sure how that works.

All other operations to the server fails.

[neilalexander]

Would recommend you upgrade either way, if it solves the problem then great. If not then I can advise on which debugging info to help diagnose.

[sh1nkey]

I had the same error, fixed with updating server from 2.7.0 -> 2.11.9, thanks @neilalexander