Elasticsearch Container keeps restarting

[DerSchwarzeJugo]

Hi, I am trying to set up a custom instance, with the maps of switzerland. I have created a custom config in the directory projects/switzerland. I have run the provided script after doing all the config stuff (created .env, elasticsearch.yml, pelias.json, docker-compose.yml). As a starting point i used "projects/austria" and then changed the necessary parts in the files pelias.json and .env

I have tested the API an am able to send and receive requests over the endpoint "/search"

Describe the bug
While creating the index it had some errors. When I try to create the index manually with "pelias elastic create" i run into this error

Elasticsearch ERROR: 2023-12-19T15:33:05Z
  Error: Request error, retrying
  GET http://elasticsearch:9200/ => getaddrinfo EAI_AGAIN elasticsearch
      at Log.error (/code/pelias/schema/node_modules/elasticsearch/src/lib/log.js:239:56)
      at checkRespForFailure (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:298:18)
      at HttpConnector.<anonymous> (/code/pelias/schema/node_modules/elasticsearch/src/lib/connectors/http.js:171:7)
      at ClientRequest.wrapper (/code/pelias/schema/node_modules/lodash/lodash.js:4991:19)
      at ClientRequest.emit (node:events:513:28)
      at Socket.socketErrorListener (node:_http_client:494:9)
      at Socket.emit (node:events:513:28)
      at emitErrorNT (node:internal/streams/destroy:157:8)
      at emitErrorCloseNT (node:internal/streams/destroy:122:3)
      at processTicksAndRejections (node:internal/process/task_queues:83:21)

Elasticsearch WARNING: 2023-12-19T15:33:05Z
  Unable to revive connection: http://elasticsearch:9200/

Elasticsearch WARNING: 2023-12-19T15:33:05Z
  No living connections

NoConnections [Error]: No Living connections
    at sendReqWithConnection (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:266:15)
    at next (/code/pelias/schema/node_modules/elasticsearch/src/lib/connection_pool.js:243:7)
    at processTicksAndRejections (node:internal/process/task_queues:78:11)
unsupported elasticsearch version. try: >=7.4.2

Log output of elastic in docker

pelias_elasticsearch  | uncaught exception in thread [main]
pelias_elasticsearch  | {"type": "server", "timestamp": "2023-12-19T15:29:58,444Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "pelias-dev", "node.name": "dca7d9aa3163", "message": "uncaught exception in thread [main]",
pelias_elasticsearch  | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
pelias_elasticsearch  | "at org.elasticsearch.node.Node.<init>(Node.java:1090) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "... 6 more",
pelias_elasticsearch  | "Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
pelias_elasticsearch  | "at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
pelias_elasticsearch  | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]",
pelias_elasticsearch  | "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
pelias_elasticsearch  | "at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398) ~[?:?]",
pelias_elasticsearch  | "at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]",
pelias_elasticsearch  | "at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]",
pelias_elasticsearch  | "at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?]",
pelias_elasticsearch  | "at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.node.Node.<init>(Node.java:427) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.1.jar:7.16.1]",
pelias_elasticsearch  | "... 6 more"] }
pelias_elasticsearch  | ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];
pelias_elasticsearch  | Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
pelias_elasticsearch  |         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
pelias_elasticsearch  |         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
pelias_elasticsearch  |         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
pelias_elasticsearch  |         at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:398)
pelias_elasticsearch  |         at java.base/java.nio.file.Files.createDirectory(Files.java:700)
pelias_elasticsearch  |         at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
pelias_elasticsearch  |         at java.base/java.nio.file.Files.createDirectories(Files.java:793)
pelias_elasticsearch  |         at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:300)
pelias_elasticsearch  |         at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:224)
pelias_elasticsearch  |         at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:298)
pelias_elasticsearch  |         at org.elasticsearch.node.Node.<init>(Node.java:427)
pelias_elasticsearch  |         at org.elasticsearch.node.Node.<init>(Node.java:309)
pelias_elasticsearch  |         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
pelias_elasticsearch  |         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
pelias_elasticsearch  |         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
pelias_elasticsearch  |         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166)
pelias_elasticsearch  |         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157)
pelias_elasticsearch  |         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
pelias_elasticsearch  |         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
pelias_elasticsearch  |         at org.elasticsearch.cli.Command.main(Command.java:77)
pelias_elasticsearch  |         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)
pelias_elasticsearch  |         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
pelias_elasticsearch  | For complete error details, refer to the log at /usr/share/elasticsearch/logs/pelias-dev.log

Steps to Reproduce
.env

COMPOSE_PROJECT_NAME=pelias
DATA_DIR=./data

docker-compose.yml

version: '3'
networks:
  default:
    driver: bridge
services:
  libpostal:
    image: pelias/libpostal-service
    container_name: pelias_libpostal
    user: "${DOCKER_USER}"
    restart: always
    ports: [ "127.0.0.1:4400:4400" ]
  schema:
    image: pelias/schema:master
    container_name: pelias_schema
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
  api:
    image: pelias/api:master
    container_name: pelias_api
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4000" ]
    ports: [ "0.0.0.0:4000:4000" ]
    volumes:
      - "./pelias.json:/code/pelias.json"
  placeholder:
    image: pelias/placeholder:master
    container_name: pelias_placeholder
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4100" ]
    ports: [ "127.0.0.1:4100:4100" ]
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
      - "./blacklist/:/data/blacklist"
  whosonfirst:
    image: pelias/whosonfirst:master
    container_name: pelias_whosonfirst
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  openstreetmap:
    image: pelias/openstreetmap:master
    container_name: pelias_openstreetmap
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  openaddresses:
    image: pelias/openaddresses:master
    container_name: pelias_openaddresses
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  csv-importer:
    image: pelias/csv-importer:master
    container_name: pelias_csv_importer
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
      - "./blacklist/:/data/blacklist"
  polylines:
    image: pelias/polylines:master
    container_name: pelias_polylines
    user: "${DOCKER_USER}"
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  interpolation:
    image: pelias/interpolation:master
    container_name: pelias_interpolation
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4300" ]
    ports: [ "127.0.0.1:4300:4300" ]
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  pip:
    image: pelias/pip-service:master
    container_name: pelias_pip-service
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4200" ]
    ports: [ "127.0.0.1:4200:4200" ]
    volumes:
      - "./pelias.json:/code/pelias.json"
      - "${DATA_DIR}:/data"
  elasticsearch:
    image: pelias/elasticsearch:7.16.1
    container_name: pelias_elasticsearch
    user: "${DOCKER_USER}"
    restart: always
    ports: [ "127.0.0.1:9200:9200", "127.0.0.1:9300:9300" ]
    volumes:
      - "./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro"
      - "${DATA_DIR}/elasticsearch:/usr/share/elasticsearch/data"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add: [ "IPC_LOCK" ]
    security_opt:
      - seccomp=unconfined

pelias.json

{
  "logger": {
    "level": "debug",
    "timestamp": false
  },
  "esclient": {
    "apiVersion": "7.5",
    "hosts": [
      { "host": "elasticsearch" }
    ]
  },
  "elasticsearch": {
    "settings": {
      "index": {
        "refresh_interval": "10s",
        "number_of_replicas": "0",
        "number_of_shards": "1"
      }
    }
  },
  "api": {
    "services": {
      "pip": { "url": "http://pip:4200" },
      "libpostal": { "url": "http://libpostal:4400" },
      "placeholder": { "url": "http://placeholder:4100" },
      "interpolation": { "url": "http://interpolation:4300" }
    }
  },
  "imports": {
    "adminLookup": {
      "enabled": true
    },
    "geonames": {
      "datapath": "/data/geonames",
      "countryCode": "CH"
    },
    "openstreetmap": {
      "download": [
        { "sourceURL": "http://download.geofabrik.de/europe/switzerland-latest.osm.pbf" }
      ],
      "leveldbpath": "/tmp",
      "datapath": "/data/openstreetmap",
      "import": [{
        "filename": "switzerland-latest.osm.pbf"
      }]
    },
    "openaddresses": {
      "datapath": "/data/openaddresses",
      "files": ["ch/countrywide.csv", "ch/aargau.csv", "ch/basel-land.csv", "ch/basel-stadt.csv", "ch/bern.csv", "ch/fribourg.csv", "ch/geneva.csv", "ch/glarus.csv", "ch/grisons.csv", "ch/luzern.csv", "ch/schaffhausen.csv", "ch/schwyz.csv", "ch/solothurn.csv", "ch/uri.csv", "ch/zurich.csv"]
    },
    "polyline": {
      "datapath": "/data/polylines",
      "files": [ "extract.0sv" ]
    },
    "whosonfirst": {
      "datapath": "/data/whosonfirst",
      "countryCode": "CH",
      "importPostalcodes": true,
      "importPlace": [ "85633051" ]
    }
  }
}

Expected behavior
Clean Install and created Index, working setup..

Environment (please complete the following information):
OS:
Ubuntu WSL, Windows 10

Docker:

Client: Docker Engine - Community
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:16 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker Compose version v2.23.0-desktop.1

[missinglink]

I haven't seen this before, the getaddrinfo EAI_AGAIN elasticsearch log entry indicates that the DNS lookup of the name 'elasticsearch' failed, however this DNS resolution is handled by Docker and has been the same for many years.

[missinglink]

The ElasticsearchException[failed to bind service] error seems to indicate that the port 9200 couldn't be bound, do you possibly have another service already bound on that port, such as another instance of elasticsearch running outside of Docker?

[DerSchwarzeJugo]

I haven't seen this before, the getaddrinfo EAI_AGAIN elasticsearch log entry indicates that the DNS lookup of the name 'elasticsearch' failed, however this DNS resolution is handled by Docker and has been the same for many years.

I have also never had any problems with Docker DNS, it usually worked as intended in the past.

The ElasticsearchException[failed to bind service] error seems to indicate that the port 9200 couldn't be bound, do you possibly have another service already bound on that port, such as another instance of elasticsearch running outside of Docker?

I do not have any other instance of elasticsearch running on this machine. Also if I check the output of netstat, I don't see any binded service on port 9200.

I see now that windows is not supported in the documentation requirements... does this also include when running over WSL?

[missinglink]

The DNS error may be a consequence of the failed bind.

None of the core team have ever tested on Windows, I'd suggest using Linux or Mac to test the config out to see if it's WSL specifically causing the issues.

I would also suggest digging through the WSL settings to see if Docker has all the permissions it needs to do things like bind to ports on the host machine.

You might want to also search through the issues on this repo to see if there are any other users running on Windows and their experiences.

[missinglink]

Just also a gentle reminder to use the issue templates, it saves a lot of time diagnosing issues when people are clear about their environment when posting the issue.

[DerSchwarzeJugo]

Well i found this Answer here where someone was able to set up with WSL.

I will try to setup the default portland-metro project, to see if I have maybe misconfigured something. If this does not work aswell, I have to find a solution to test it on Linux. Ist it Possible to run it inside a VM from Virtualbox maybe?

[DerSchwarzeJugo]

there seems to be some kind of weird behaviour.. after deleting all the contents of the data folder and running the build script again, the elasticsearch container does not restart anymore and has no errors.