Error while taking a backup of Mysql database

[muratyarali]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the installation guide https://github.com/kubeflow/manifests?tab=readme-ov-file ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.8

Describe your issue

I deployed kubeflow v1.8.0 in k8s cluster. I want to back up mysql and katib-mysql databases that are running in pods in kubeflow namespace.

katib-mysql-845496cf76-r668n                             1/1     Running    0               13d
mysql-7cb4fb95bf-gdmcb                                   2/2     Running    0               5h45m

I used mysqldump-secure-uploader tool for this purpose. mysqldump-secure-uploader creates cronjob and when we run this cronjob, a pod is running and we can get the backup.

$ k get cj -n kfl-dev-katib
NAME                        SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
mysqldump-secure-uploader   0 */12 * * *   False     0        112m            6d22h

$ k get cj -n kubeflow
NAME                        SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
mysqldump-secure-uploader   0 */12 * * *   False     0        113m            132m

I could back up katib-mysql without any problem by using mysqldump but can not mysql (in same namespace with katib-mysql).

$ k get po -n kfl-dev-katib
NAME                                       READY   STATUS      RESTARTS   AGE
mysqldump-secure-uploader-28715760-nnzth   0/1     Completed   0          25h

$ k get po -n kubeflow | grep mysqldump-secure
mysqldump-secure-uploader-manual-skrj4                   0/2     Init:1/3   0               72m

According to my examination, mysql db server is only connectable from the pods using serviceAccounts in this AuthorizationPolicy.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  annotations:
  labels:
    application-crd-id: kubeflow-pipelines
  name: mysql
  namespace: kubeflow
spec:
  rules:
  - from:
    - source:
        principals:
        - cluster.local/ns/kubeflow/sa/ml-pipeline
        - cluster.local/ns/kubeflow/sa/ml-pipeline-ui
        - cluster.local/ns/kubeflow/sa/ml-pipeline-persistenceagent
        - cluster.local/ns/kubeflow/sa/ml-pipeline-scheduledworkflow
        - cluster.local/ns/kubeflow/sa/ml-pipeline-viewer-crd-service-account
        - cluster.local/ns/kubeflow/sa/kubeflow-pipelines-cache
        - cluster.local/ns/kubeflow/sa/metadata-grpc-server
  selector:
    matchLabels:
      app: mysql

I created a test mysql client pod that uses ml-pipeline serviceAccount in kubeflow namespace and it can conenct to mysql db.

$ k exec -it mysql-client -n kubeflow -- bash -c "mysql -h mysql.kubeflow.svc.cluster.local -u root"

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 8.0.26 MySQL Community Server - GPL

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

and the logs of mysqldump container are here

$ k logs -n kfl-dev-katib mysqldump-secure-uploader-28713600-n52xf -c mysqldump
[INFO]  (OPT): Logging disabled
[DEBUG] (CFG): Using config: /etc/mysqldump-secure.conf
[DEBUG] (CFG): Destination dir: /var/mysqldump-secure
[DEBUG] (CFG): Using file Prefix: 2024-08-05_00-15__
[INFO]  (OPT): MySQL SSL connection disabled
[INFO]  (OPT): Compression enabled
[DEBUG] (OPT): Compression arguments: gzip -9 --stdout
[INFO]  (OPT): Encryption disabled
[INFO]  (OPT): Deletion disabled
[INFO]  (OPT): Nagios log disabled
[INFO]  (OPT): Info files enabled
[DEBUG] (SRV): MySQL server connection: katib-mysql.kubeflow.svc.cluster.local via TCP/IP
[DEBUG] (SRV): MySQL server connection: Using SSL (Cipher in use is ECDHE-RSA-AES128-GCM-SHA256)
[DEBUG] (SRV): MySQL server rep type:   master
[DEBUG] (SRV): MySQL server version:    8.0.29 MySQL Community Server - GPL
[DEBUG] (SRV): MySQL server hostname:   katib-mysql-845496cf76-r668n:3306
[DEBUG] (SQL): Retrieving list of databases...  5
[INFO]  (SQL): 1/5 Skipping: information_schema (DB is ignored)
[INFO]  (SQL): 2/5 Dumping:  katib (0.02 MB)  (compressed) (--single-transaction) (--skip-quick)  0 sec (0.00 MB)
[INFO]  (SQL): 3/5 Skipping: mysql (DB is ignored)
[INFO]  (SQL): 4/5 Skipping: performance_schema (DB is ignored)
[INFO]  (SQL): 5/5 Skipping: sys (DB is ignored)
[DEBUG] (RUN): Dumping finished (OK: 1 dbs, IGN: 4 dbs, ERR: 0, TOTAL: 5)
[DEBUG] (RUN): Took 1 seconds
[DEBUG] (RUN): Total size dumped: 0.02 MB
[OK]    Finished successfully

But when mysqldump container in mysqldump-secure-uploader pod tries to connect, it fails.

$ k exec -it mysqldump-secure-uploader-manual-skrj4 -c mysqldump -n kubeflow -- bash -c "mysql -h mysql.kubeflow.svc.cluster.local -u root"
<waiting for a long time and no answer>

and logs are here

$ k logs -n kfl-dev-mysql -c mysqldump mysqldump-secure-uploader-manual-9qlml -f
[INFO]  (OPT): Logging disabled
[DEBUG] (CFG): Using config: /etc/mysqldump-secure.conf
[DEBUG] (CFG): Destination dir: /var/mysqldump-secure
[DEBUG] (CFG): Using file Prefix: 2024-08-01_09-20__
[INFO]  (OPT): MySQL SSL connection disabled
[INFO]  (OPT): Compression enabled
[DEBUG] (OPT): Compression arguments: gzip -9 --stdout
[INFO]  (OPT): Encryption disabled
[INFO]  (OPT): Deletion disabled
[INFO]  (OPT): Nagios log disabled
[INFO]  (OPT): Info files enabled
[FATAL] (RUN): Cannot connect to mysql database.
[FATAL] (RUN): Via: /usr/bin/mysql --defaults-file=/etc/mysqldump-secure.cnf
[FATAL] (RUN): SQL: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 104   
[FATAL] (HLP): Fix credentials in: /etc/mysqldump-secure.cnf
[FATAL] Aborting

There are no NetworkPolicies in the related namespace and only one AuthorizationPolicy. I adhere to this AuthorizationPolicy, but still mysqldump can not connect to mysql server pod.

Do you know why mysqldump-secure-uploader pod can not connect to mysql database?

Steps to reproduce the issue

I added these to /etc/mysql/my.cnf configuration file in mysql server pod because this is suggested in some forums but it did not work.

[mysqld]
bind-address        = 0.0.0.0
wait_timeout        = 28800
net_read_timeout    = 120
max_allowed_packet  = 1073741824

I created one more AuthorizationPolicy that is allowing all the connections to MySQL server pod but it did not work too.

Put here any screenshots or videos (optional)

No response

[juliusvonkohout]

@muratyarali can we close this or do you want to post your findings here?

[juliusvonkohout]

closing due to inactivity.