Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

5.7.10-fails to mount volume /var/log/mysql #146

Closed
asmoker opened this issue Feb 23, 2016 · 24 comments
Closed

5.7.10-fails to mount volume /var/log/mysql #146

asmoker opened this issue Feb 23, 2016 · 24 comments

Comments

@asmoker
Copy link

asmoker commented Feb 23, 2016

In oder to collect mysql error log and slow query log to logstash, I mount host volume /data/logs/mysql to the container /var/log/mysql:
My docker-compose.yml:

version: '2'
services:
mysql:
image: clkj.com/docker/mysql:5.7.10
container_name: mysql
restart: always
network_mode: "bridge"
privileged: true
environment:
- MYSQL_ROOT_PASSWORD=password
volumes:
- /data/mysql/var/lib/mysql:/var/lib/mysql
- ./conf:/etc/mysql/conf.d
- /data/logs/mysql:/var/log/mysql

and my conf/log.cnf:

[mysqld]
log-error = /var/log/mysql/error.log
slow_query_log = ON
slow_query_log_file = /var/log/mysql/slow.log
slow_launch_time = 2

but the container exited with code 0. The container runs up when I remove the - /data/logs/mysql:/var/log/mysql configuration;

OS: CentOS 7
MySQL: 5.7.10

@yosifkit
Copy link
Member

Probably need to chown or chmod the /data/logs/mysql/ directory so that the mysql user from the container can write to it.

@asmoker
Copy link
Author

asmoker commented Mar 22, 2016

@yosifkit Thanks for your reply.
I trid to chown and chmod the /data/logs/mysql/ directory, but that's no effect.
Beacause the dirctory was created by the mysql container, I think it has the permission to read and write the folder.

@ltangvald
Copy link
Collaborator

Anything from docker logs ?
Does it still happen if you disable slow_query_log (or just not map the config file)?
Could you try starting the container in interactive mode (-it and add a /bin/bash to the end of the run command) with the same mapping, then running myqld --initialize?

@asmoker
Copy link
Author

asmoker commented Mar 22, 2016

@ltangvald
If I don't map the config file, the container can run up normally.
So, if I don't map the config file, the container runs up and mysqld also runs up, it can provide database service normally.
If I map the config file, the container won't run up and status is Restarting (1) 1 seconds ago, and I can't use the command docker exec --it mysql /bin/bash to access the container.
Maybe there is a problem with my config file?
😶

@ltangvald
Copy link
Collaborator

You can run
docker logs containerid
to see if anything was printed to stdout in the container before it shut down.
Can you try mapping up the config file, but removing the slow log options?
Might be an issue with the config file, but it's also possible there's a bug in the server that's being uncovered because of the uncommon way the docker script initializes the database.
Just to be sure, could you try the same setup but with a 5.7.11 image?

@yosifkit
Copy link
Member

Still seems like a permission issue. I used your exact config and these are the steps I tried:

$ docker run -it --rm -v ~/dicker/tmp/my.cnf:/etc/mysql/conf.d/custom.cnf -v ~/dicker/tmp/mysql-logs/:/var/log/mysql/ --name mysql -e MYSQL_ROOT_PASSWORD=12345 mysql bash
root@1c384a908efa:/# /entrypoint.sh mysqld # try default command
Initializing database
root@1c384a908efa:/# mysqld --initialize # try to just init the database, no output :(
root@1c384a908efa:/# ls -al /var/lib/mysql/ # check contents of the directories
total 8
drwxr-xr-x  2 mysql mysql 4096 Mar 22 20:14 .
drwxr-xr-x 14 root  root  4096 Mar 17 22:24 ..
root@1c384a908efa:/# ls -al /var/log/mysql/
total 8
drwxr-sr-x 2 root 1000 4096 Mar 22 20:14 .
drwxr-xr-x 5 root root 4096 Mar 17 22:24 ..
root@1c384a908efa:/# exit
$ # ok, lets do a new container to be safe         
$ docker run -it --rm -v ~/dicker/tmp/my.cnf:/etc/mysql/conf.d/my.cnf -v ~/dicker/tmp/mysql-logs/:/var/log/mysql/ --name mysql -e MYSQL_ROOT_PASSWORD=12345 mysql bash
root@d27a8e03f5db:/# ls -al /var/log/mysql/ # still empty and owned by root and my group
total 8
drwxr-sr-x 2 root 1000 4096 Mar 22 19:52 .
drwxr-xr-x 5 root root 4096 Mar 17 22:24 ..
root@d27a8e03f5db:/# chown mysql:mysql /var/log/mysql/
root@d27a8e03f5db:/# /entrypoint.sh mysqld
Initializing database
Database initialized
MySQL init process in progress...
Warning: Unable to load '/usr/share/zoneinfo/Factory' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/posix/Factory' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/right/Factory' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.

/entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*


MySQL init process done. Ready for start up.

$ # looks like it is ready!
$ # from other terminal, let's test mysql client
$ docker exec -it mysql bash
root@d27a8e03f5db:/# mysql -p      
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.11-log MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select 1;
+---+
| 1 |
+---+
| 1 |
+---+
1 row in set (0.00 sec)

mysql> 

The mount point will be owned by root or whoever created the folder on the host. You should be able to just do chown 999:999 /data/logs/mysql and it will work. If you need access by your host user, then it should work to put the group as your user's group like 1000 (and chmoding so that group has read access to everything).

@asmoker
Copy link
Author

asmoker commented Mar 23, 2016

@yosifkit Thanks a lot.
That's really a permission issue. After I run chown 999:999 /data/logs/mysql, the container can run up and provide database service, and slow query log file was created in the dirctory /data/logs/mysql. Everything is OK.
What I confused before is that why the mysql container can't read and write the '/data/logs/mysql` even though the dirctory is created by the container.
However, Thanks a lot.
Please forgive my poor English.

@ltangvald
Copy link
Collaborator

The entrypoint script ensures the correct permission to datadir. Maybe we need to do something similar for logdir?

@yosifkit
Copy link
Member

@ltangvald can we use the same --vebose --help to get any log directories that they have configured?

@ltangvald
Copy link
Collaborator

Yeah, I think we can do it in the same way, though I haven't tested it.

olegabr added a commit to olegabr/mysql that referenced this issue Jul 25, 2016
@luffyke
Copy link

luffyke commented Jan 23, 2017

@yosifkit hi, please help to fix this issue as there are many users meet it on SO. thanks a lot.

@kiwifly
Copy link

kiwifly commented Mar 9, 2017

hi, I also meet this issue when I want persist /var/lib/mysql, and I fix this issue by using the following method~~

sudo docker volume create --name mysql_db
sudo docker run -d -v mysql_db:/var/lib/mysql <Container ID>
sudo docker volume inspect mysql_db

@mscislak
Copy link

Hi, I solved this issue by using the following command:

docker exec mysql chown mysql:root /var/log/mysql

@yu2003w
Copy link

yu2003w commented Jun 1, 2017

By executing chown 999:999 solved the problem.

@tianon
Copy link
Member

tianon commented Dec 26, 2017

Closing solved issue!

@tianon tianon closed this as completed Dec 26, 2017
@zaneclaes
Copy link

zaneclaes commented Oct 9, 2018

Still not working here. The docker image provides absolutely zero information about what's going wrong (it just exits with status 1), and there are no logs in Docker other than [Entrypoint] Initializing database. I cannot exec because the image won't boot, so that's not a solution. I tried doing a chown -R on the local machine, which also did not work. I can see the the timestamps were modified on the files, so clearly MySQL can in fact access the files. Out of desperation, I deleted the entire folder and let MySQL recreate it. The volume mounts and all the contents are created, and then it exits with the same status and no message.

@yosifkit
Copy link
Member

yosifkit commented Oct 9, 2018

@zaneclaes, The problem discussed in this issue is caused by user-supplied config to MySQL for logging to some directory and yet not giving the mysql user of the container (user id 999) access to the specified directory.

I alluded to using mysqld --vebose --help to get log directories (like we do for data dir), but I am hesitant to do so. If we add this for log-error and slow_query_log_file then we need to do it for every MySQL config that can specify a directory or file that MySQL should use and it seems unsustainable. I am leaning toward the answer that if a user specifies some other file or directory for MySQL to use (for logging or whatever), then they should ensure the mysql user can access it (or whatever --user they run the container as).

@Abadii
Copy link

Abadii commented Jan 9, 2020

In Docker without mount, I had to make the folders and touch the log files in the folder I want to use:

RUN mkdir -p /var/log/mysql && touch /var/log/mysql/error.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log

After performing a chown on that folder, everything worked well:

RUN chown -R mysql:mysql /var/log/mysql

@laurinkeithdavis
Copy link

@Abadii

Still can't get it to work.

@Abadii
Copy link

Abadii commented Mar 15, 2020

@Abadii

Still can't get it to work.

Can you provide your Dockerfile and the errors?

@laurinkeithdavis
Copy link

I just realized that your instructions were for without using a mount, and if I remove the mount, this does work. However, I still get access is denied when using a mount.

version: '3.7'

services:
     mysql:
          build: '.'
          container_name: intranet-mysql
          ports:
               - '3306:3306'
          volumes:
               - data:/var/lib/mysql
               - /docker/mysql01/conf.d:/etc/mysql/conf.d
               - /docker/mysql01/logs:/var/log/mysql
          environment:
               MYSQL_ROOT_PASSWORD: 'xxxxxxxxxx'
               MYSQL_DATABASE: 'pride'
               MYSQL_USER: 'xxxxxxxxxx'
               MYSQL_PASSWORD: 'xxxxxxxxxxxx'
volumes:
     data:
FROM mysql:5.7.28
RUN mkdir -p /var/log/mysql && touch /var/log/mysql/error.log /var/log/mysql/mysql.log \
	/var/log/mysql/mysql-slow.log
RUN chown -R mysql:mysql /var/log/mysql
intranet-mysql | 2020-03-15 14:25:12+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.28-1debian9 started.
intranet-mysql | 2020-03-15 14:25:12+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
intranet-mysql | 2020-03-15 14:25:12+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.28-1debian9 started.
intranet-mysql | 2020-03-15T14:25:13.089312-00:00 0 [ERROR] Could not open file '/var/log/mysql/error.log' for error logging: Permission denied
intranet-mysql | 2020-03-15T14:25:13.089390-00:00 0 [ERROR] Aborting

@Abadii
Copy link

Abadii commented Mar 15, 2020

I just realized that your instructions were for without using a mount, and if I remove the mount, this does work. However, I still get access is denied when using a mount.

version: '3.7'

services:
     mysql:
          build: '.'
          container_name: intranet-mysql
          ports:
               - '3306:3306'
          volumes:
               - data:/var/lib/mysql
               - /docker/mysql01/conf.d:/etc/mysql/conf.d
               - /docker/mysql01/logs:/var/log/mysql
          environment:
               MYSQL_ROOT_PASSWORD: 'xxxxxxxxxx'
               MYSQL_DATABASE: 'pride'
               MYSQL_USER: 'xxxxxxxxxx'
               MYSQL_PASSWORD: 'xxxxxxxxxxxx'
volumes:
     data:
FROM mysql:5.7.28
RUN mkdir -p /var/log/mysql && touch /var/log/mysql/error.log /var/log/mysql/mysql.log \
	/var/log/mysql/mysql-slow.log
RUN chown -R mysql:mysql /var/log/mysql
intranet-mysql | 2020-03-15 14:25:12+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.28-1debian9 started.
intranet-mysql | 2020-03-15 14:25:12+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
intranet-mysql | 2020-03-15 14:25:12+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.28-1debian9 started.
intranet-mysql | 2020-03-15T14:25:13.089312-00:00 0 [ERROR] Could not open file '/var/log/mysql/error.log' for error logging: Permission denied
intranet-mysql | 2020-03-15T14:25:13.089390-00:00 0 [ERROR] Aborting

I see that you are mounting the volumes so your local files are overwritten and mapped with your container. However, you have a permission issue to write and/or read on that file.
You could try to chown the files in your own host with the same userid:groupid of your container.

You could also try to use the below command in your CMD or Entrypoint:

chown -R mysql:mysql /var/log/mysql

This way, before starting mysql, you are setting the permissions for the folder you are mapping

@laurinkeithdavis
Copy link

@Abadii

But why would I have to do that? Why is this so hard? Just to be able to have log files written to a folder that is accessible from the host?

Also, why would I have to chown that folder from the Entrypoint, when I'm doing it from the dockerfile?

@Abadii
Copy link

Abadii commented Mar 15, 2020

@laurin1
The chown in your Dockerfile would have no effect if you are mounting the folder as volume. So evertying you have in that specific folder, will be overwritten with your local files that you have mounted.

Apparently after mouting, your container does not have the permission to edit the file. So you need to update the permissions of that file.

The volumes are ALWAYS mapped after your Docker build. That's why, chown in your Dockerfile doesn't have any effect if you mount that folder afterwards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.