Problems with elasticsearch logs #183

ElDudelino · 2020-03-29T08:21:32Z

I'm trying to setup pelias via the docker setup under debian buster. Everything installs fine until I run
pelias elastic start
The service comes up fine first, but then moves to restarting.

> pelias compose ps
        Name                      Command                 State      Ports
--------------------------------------------------------------------------
pelias_elasticsearch   /usr/local/bin/docker-entr ...   Restarting

There seems to be an issue with the logs:

> pelias compose logs elasticsearch
Attaching to pelias_elasticsearch
elasticsearch_1  | Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
elasticsearch_1  | output:
elasticsearch_1  | [0.018s][error][logging] Error opening log file 'logs/gc.log': Operation not permitted
elasticsearch_1  | [0.018s][error][logging] Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
elasticsearch_1  | error:
elasticsearch_1  | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elasticsearch_1  | Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
elasticsearch_1  | Error: Could not create the Java Virtual Machine.
elasticsearch_1  | Error: A fatal exception has occurred. Program will exit.
elasticsearch_1  |      at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
elasticsearch_1  |      at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
elasticsearch_1  |      at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
elasticsearch_1  |      at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)

Any idea how I can troubleshoot and fix this?

The text was updated successfully, but these errors were encountered:

missinglink · 2020-03-29T08:31:22Z

This sounds very familiar, we fixed this issue in the past via #150

Which elasticsearch version are you using?

missinglink · 2020-03-29T08:35:14Z

It should 'just work' but if it doesn't then you'll need to check that the directory+file logs/gc.log is writable (inside the docker container) by the docker user.

ElDudelino · 2020-03-29T08:57:24Z

This sounds very familiar, we fixed this issue in the past via #150

Which elasticsearch version are you using?

I downloaded pelias with
git clone https://github.com/pelias/docker.git
and set it up with
pelias compose pull

So I guess latest? How can I check the versions installed?

It should 'just work' but if it doesn't then you'll need to check that the directory+file logs/gc.log is writable (inside the docker container) by the docker user.

Since I'm quite new to this container stuff: how can I do this inside the container? Is the container filesystem located anywhere outside?

missinglink · 2020-03-29T09:12:34Z

It's a little complex so I can have a look for you ;)

You probably also cd into one of the project directories? Which one? Did you modify any files there, if so what changes did you make?

Please post the output of docker images

ElDudelino · 2020-03-29T09:22:42Z

I started with portland-metro (planet after I get it working).
I edited .env and changed the following lines:

DATA_DIR=/data/pelias/data
DOCKER_USER=1001:998

1001 is me, 998 is the docker group.

Docker was set up via the guidelines about installation from repository under
https://docs.docker.com/install/linux/docker-ce/debian/
and additional 'Manage Docker as a non-root use' steps from
https://docs.docker.com/install/linux/linux-postinstall/

docker run hello-world and docker run -it ubuntu bash run fine from my user.

The output is

krm@linux2:/data/pelias/code/docker/projects/portland-metro$ docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
pelias/schema              master              0936cdfdb588        33 hours ago        521MB
ubuntu                     latest              4e5021d210f6        8 days ago          64.2MB
pelias/whosonfirst         master              aa0ebd617460        9 days ago          539MB
pelias/interpolation       master              e7ee048f3599        9 days ago          3.12GB
pelias/openstreetmap       master              7a57f110c59f        9 days ago          652MB
pelias/fuzzy-tester        master              cf8babe50766        9 days ago          600MB
pelias/openaddresses       master              0ef17eb050fb        9 days ago          590MB
pelias/polylines           master              4183abe022e0        11 days ago         1.02GB
pelias/libpostal-service   latest              4c4af01d0797        2 weeks ago         3.19GB
pelias/csv-importer        master              7132ed1b09ca        2 weeks ago         589MB
pelias/pip-service         master              eff632051abd        2 weeks ago         590MB
pelias/transit             master              264b4f84a52e        2 weeks ago         594MB
pelias/placeholder         master              3c6b7e8019b1        3 weeks ago         552MB
pelias/api                 master              2ec3637d2ecb        7 weeks ago         562MB
pelias/elasticsearch       7.5.1               6f90d70df8ee        2 months ago        1.32GB
hello-world                latest              fce289e99eb9        15 months ago       1.84kB

ElDudelino · 2020-03-29T09:40:11Z

I started a shell for the container with
docker run -it pelias/elasticsearch:7.5.1 bash
and checked the logs folder:

[elasticsearch@07eea2c614f4 ~]$ ls -l
total 576
-rw-r--r--  1 elasticsearch root  13675 Dec 16 22:54 LICENSE.txt
-rw-r--r--  1 elasticsearch root 523209 Dec 16 23:01 NOTICE.txt
-rw-r--r--  1 elasticsearch root   8499 Dec 16 22:54 README.textile
drwxr-xr-x  2 elasticsearch root   4096 Dec 16 23:02 bin
drwxrwxrwx  5 elasticsearch root   4096 Jan  7 17:35 config
drwxrwxrwx  2 elasticsearch root   4096 Dec 16 23:02 data
drwxr-xr-x 17 elasticsearch root   4096 Dec 16 23:01 jdk
drwxr-xr-x  4 elasticsearch root   4096 Dec 16 23:01 lib
drwxrwxrwx  2 elasticsearch root   4096 Mar 29 09:35 logs
drwxr-xr-x 74 elasticsearch root   4096 Dec 16 23:01 modules
drwxr-xr-x  6 elasticsearch root   4096 Jan  7 17:35 plugins

The folder itself is empty.

Edit: the modification date was updated because I tried to add a gc.log with 777. No dice.

missinglink · 2020-03-29T11:29:22Z

hmm.. I'm still not able to reproduce the error, the following command works fine for me (server starts and is responsive):

note: it takes some time to fully start up ~30s

docker run --rm -it -u 1001:998 pelias/elasticsearch:7.5.1

I'm guessing that the same command fails for you?

missinglink · 2020-03-29T11:32:23Z

docker --version
Docker version 19.03.8, build afacb8b

docker images | grep '7\.5\.1'
pelias/elasticsearch  7.5.1 6f90d70df8ee 2 months ago 1.32GB

ElDudelino · 2020-03-29T11:37:06Z

Yes, it fails rather quickly:

krm@linux2:/data/pelias/code/docker/projects/portland-metro$ docker run --rm -it -u 1001:998 pelias/elasticsearch:7.5.1
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
output:
[0.019s][error][logging] Error opening log file 'logs/gc.log': Operation not permitted
[0.019s][error][logging] Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
error:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
        at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
        at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
        at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
        at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)

krm@linux2:/data/pelias/code/docker/projects/portland-metro$ docker --version
Docker version 19.03.8, build afacb8b7f0

missinglink · 2020-03-29T11:44:40Z

not sure what to say... there most be something else amiss on your end?

docker run --rm -it -u 1001:998 --entrypoint=bash pelias/elasticsearch:7.5.1 -c 'id > logs/gc.log; ls -lah logs/gc.log; cat logs/gc.log'
-rw-r--r-- 1 1001 998 28 Mar 29 11:46 logs/gc.log
uid=1001 gid=998 groups=998

ElDudelino · 2020-03-29T11:46:17Z

krm@linux2:/data/pelias/code/docker/projects/portland-metro$ docker run --rm -it -u 1001:998 --entrypoint=bash pelias/elasticsearch:7.5.1 -c 'ls -lah ./logs; id > logs/gc.log; cat logs/gc.log'
total 8.0K
drwxrwxrwx  2 elasticsearch root 4.0K Dec 16 23:01 .
drwxrwxrwx 29 elasticsearch root 4.0K Dec 16 23:02 ..
bash: logs/gc.log: Operation not permitted
cat: logs/gc.log: No such file or directory

missinglink · 2020-03-29T11:49:02Z

Try docker run --privileged ....

ElDudelino · 2020-03-29T11:50:45Z

Same deal. Plus an tty issue

krm@linux2:/data/pelias/code/docker/projects/portland-metro$ docker run --privileged --rm -it -u 1001:998 --entrypoint=bash pelias/elasticsearch:7.5.1 -c 'ls -lah ./logs; id > logs/gc.log; cat logs/gc.log'
total 8.0K
drwxrwxrwx  2 elasticsearch root 4.0K Dec 16 23:01 .
drwxrwxrwx 29 elasticsearch root 4.0K Dec 16 23:02 ..
bash: logs/gc.log: Operation not permitted
cat: logs/gc.log: No such file or directory
failed to resize tty, using default size

missinglink · 2020-03-29T11:52:48Z

I really don't know sorry, you're the only person who's reported this so, as unhelpful as it is, I'm 99% sure it's something to do with your environment or how you configured docker.

Is there anything else about your system that I should know about, such as unusual mount settings, network drives or layered virtualisation?

ElDudelino · 2020-03-29T11:59:30Z

It's a fresh virtual installation on a Windows server I think. I'm not sure about the virtual machine running this.

root@linux2:/data/pelias/code/docker/images/elasticsearch/7.5.1# uname -a
Linux linux2 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux

/data is a Netapp network drive.

root@linux2:/data/pelias/code/docker/images/elasticsearch/7.5.1# mount|grep data
x.x.x.x:/linux2 on /data type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=x.x.x.x,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=x.x.x.x)

IP address removed

missinglink · 2020-03-29T12:02:40Z

Hmm ok, it's Sunday morning and I'm volunteering my time, you said it was Debian Buster and now you're saying it's a Windows server?

It's most certainly some combination of these many layers of virtualization and the network drive that are causing the issue.

ElDudelino · 2020-03-29T12:06:31Z

It's Debian Buster on a virtual machine which is hosted on a Windows server.
I did not think that this virtualization could cause problems.
Guess I need to find a non-virtual setup since I have no clues how to troubleshoot this further.

Many thanks for your time and help!

missinglink · 2020-03-29T12:16:47Z

FYI #124

ElDudelino · 2020-03-29T12:28:30Z

Followup: omitting the DOCKER_USER did let me continue with the installation.

ElDudelino · 2020-03-30T08:01:15Z

Post mortem: it was my misconfiguration/misuse/misunderstanding of DOCKER_USER. I hadn't changed anything in the docker configuration, so the user elasticsearch inside the container is 1000. Which does not map to the correct user outside of the container. So currently the files and processes are run by 1000 outside too, which is the local account of another admin. Which isn't the intended configuration, but it works.
So Pelias with portland dataset is up and running in the virtual Debian now and can be accessed from other computers.
Thanks for helping me understand!

missinglink closed this as completed Mar 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problems with elasticsearch logs #183

Problems with elasticsearch logs #183

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

missinglink commented Mar 29, 2020 •

edited

Loading

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

ElDudelino commented Mar 29, 2020 •

edited

Loading

ElDudelino commented Mar 29, 2020 •

edited

Loading

missinglink commented Mar 29, 2020

missinglink commented Mar 29, 2020 •

edited

Loading

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020 •

edited

Loading

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020 •

edited

Loading

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

ElDudelino commented Mar 29, 2020

ElDudelino commented Mar 30, 2020

Problems with elasticsearch logs #183

Problems with elasticsearch logs #183

Comments

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

missinglink commented Mar 29, 2020 • edited Loading

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

ElDudelino commented Mar 29, 2020 • edited Loading

ElDudelino commented Mar 29, 2020 • edited Loading

missinglink commented Mar 29, 2020

missinglink commented Mar 29, 2020 • edited Loading

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020 • edited Loading

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020 • edited Loading

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

ElDudelino commented Mar 29, 2020

missinglink commented Mar 29, 2020

ElDudelino commented Mar 29, 2020

ElDudelino commented Mar 30, 2020

missinglink commented Mar 29, 2020 •

edited

Loading

ElDudelino commented Mar 29, 2020 •

edited

Loading

ElDudelino commented Mar 29, 2020 •

edited

Loading

missinglink commented Mar 29, 2020 •

edited

Loading

missinglink commented Mar 29, 2020 •

edited

Loading

missinglink commented Mar 29, 2020 •

edited

Loading