Ingester fails initializing when WAL PVC is out of space

[Fresa]

We run Loki in simple scalable mode on Kubernetes with WAL enabled writing to PVCs. When a PVC becomes full it seems the corresponding loki-write instance bricks itself due to that the ingester no longer can initialize due to insufficient space on the PVC and hence cannot start to replay and flush the WAL to long term storage and start freeing up storage space on the PVC.

Startup logs from loki-write:

2024-12-03 14:48:43.807	
level=error ts=2024-12-03T13:48:43.80708689Z caller=log.go:216 msg="error running loki" err="open /var/loki/wal/00039919: no space left on device
error initialising module: ingester
github.com/grafana/dskit/modules.(*Manager).initModule
  /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
  /src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
  /src/loki/pkg/loki/loki.go:458
main.main
 /src/loki/cmd/loki/main.go:129
runtime.main
  /usr/local/go/src/runtime/proc.go:271
runtime.goexit
  /usr/local/go/src/runtime/asm_amd64.s:1695"
2024-12-03 14:48:43.807	
level=info ts=2024-12-03T13:48:43.800163536Z caller=head_manager.go:308 index-store=tsdb-2024-09-17 component=tsdb-head-manager msg="loaded wals by period" groups=1
2024-12-03 14:48:43.807	
level=info ts=2024-12-03T13:48:43.800103805Z caller=manager.go:86 index-store=tsdb-2024-09-17 component=tsdb-manager msg="loaded leftover local indices" err=null successful=true buckets=49 indices=30 failures=0
2024-12-03 14:48:43.807	
level=info ts=2024-12-03T13:48:43.796691475Z caller=head_manager.go:308 index-store=tsdb-2024-09-17 component=tsdb-head-manager msg="loaded wals by period" groups=0
2024-12-03 14:48:43.807	
level=info ts=2024-12-03T13:48:43.796371461Z caller=table_manager.go:136 index-store=tsdb-2024-09-17 msg="uploading tables"
2024-12-03 14:48:43.807	
level=info ts=2024-12-03T13:48:43.79630754Z caller=shipper.go:160 index-store=tsdb-2024-09-17 msg="starting index shipper in WO mode"

We had multiple problems leading to this particular issue, but I think the root problem started with an incorrectly configured ingester.wal.replay_memory_ceiling. It was set to default (4GB according to the docs) while the memory resource limit for loki-write was configured to 2GB. During a deployment log ingestion happen to be surging and when one of the loki-write instances was initializing it got OOM-killed during WAL replay, however ingestion continued on the same instance before this happened letting the WAL grow even further and eventually it ran out of storage space of the PVC. At this point we realized the misconfiguration of replay_memory_ceiling, adjusted it to 1.5GB, and redeployed. This is where the above log record starting to appear, causing the container to crash.

We ended up deleting the PVC to get the instance up and running again.

Config:

auth_enabled: false
chunk_store_config:
  chunk_cache_config:
    background:
      writeback_buffer: 500000
      writeback_goroutines: 1
      writeback_size_limit: 500MB
    default_validity: 0s
    memcached:
      batch_size: 4
      parallelism: 5
    memcached_client:
      addresses: dnssrvnoa+_memcached-client._tcp.loki-chunks-cache.loki.svc
      consistent_hash: true
      max_idle_conns: 72
      timeout: 2000ms
common:
  compactor_address: 'http://loki-backend:3100'
  path_prefix: /var/loki
  replication_factor: 3
  storage:
    azure:
      account_name: <redacted>
      container_name: chunks
      use_federated_token: true
      use_managed_identity: false
compactor:
  compaction_interval: 10m
  delete_request_store: azure
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  retention_enabled: true
frontend:
  scheduler_address: ""
  tail_proxy_url: ""
frontend_worker:
  scheduler_address: ""
index_gateway:
  mode: simple
ingester:
  chunk_encoding: snappy
limits_config:
  allow_structured_metadata: true
  ingestion_rate_strategy: local
  max_cache_freshness_per_query: 10m
  max_chunks_per_query: 300000
  otlp_config:
    resource_attributes:
      attributes_config:
      - action: index_label
        attributes:
        - namespace
        - container
        - pod
        - cluster
        - k8s.cronjob.name
        - k8s.job.name
        - k8s.daemonset.name
        - k8s.statefulset.name
        - k8s.deployment.name
        - k8s.replicaset.name
      ignore_defaults: true
  query_timeout: 300s
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 30d
  split_queries_by_interval: 15m
  volume_enabled: true
memberlist:
  join_members:
  - loki-memberlist
pattern_ingester:
  enabled: true
querier:
  max_concurrent: 4
query_range:
  align_queries_with_step: true
  cache_results: true
  results_cache:
    cache:
      background:
        writeback_buffer: 500000
        writeback_goroutines: 1
        writeback_size_limit: 500MB
      default_validity: 12h
      memcached_client:
        addresses: dnssrvnoa+_memcached-client._tcp.loki-results-cache.loki.svc
        consistent_hash: true
        timeout: 500ms
        update_interval: 1m
ruler:
  storage:
    azure:
      account_name: <redacted>
      container_name: ruler
      use_federated_token: true
      use_managed_identity: false
    type: azure
runtime_config:
  file: /etc/loki/runtime-config/runtime-config.yaml
schema_config:
  configs:
  - from: "2024-09-17"
    index:
      period: 24h
      prefix: loki_index_
    object_store: azure
    schema: v13
    store: tsdb
server:
  grpc_listen_port: 9095
  grpc_server_max_recv_msg_size: 8388608
  grpc_server_max_send_msg_size: 8388608
  http_listen_port: 3100
  http_server_read_timeout: 600s
  http_server_write_timeout: 600s
  log_level: info
storage_config:
  boltdb_shipper:
    index_gateway_client:
      server_address: dns+loki-backend-headless.loki.svc.cluster.local:9095
  hedging:
    at: 250ms
    max_per_second: 20
    up_to: 3
  tsdb_shipper:
    index_gateway_client:
      server_address: dns+loki-backend-headless.loki.svc.cluster.local:9095
tracing:
  enabled: true

Environment:

• Infrastructure: AKS / Kubernetes v1.29.4
• Deployment tool: Helm, ArgoCD
• loki-write: docker.io/grafana/loki:3.1.1
• Loki helm chart: 6.12.0 (https://github.com/grafana/loki/tree/main/production/helm/loki)