Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Windmill Postgres error cannot create the DB correctly #3917

Closed
kikohs opened this issue Oct 15, 2024 · 11 comments · Fixed by #3878
Closed

[Bug]: Windmill Postgres error cannot create the DB correctly #3917

kikohs opened this issue Oct 15, 2024 · 11 comments · Fixed by #3878

Comments

@kikohs
Copy link

kikohs commented Oct 15, 2024

Error Message and Logs

The postgres container of windmill cannot create the database correctly:

2024-10-15T07:56:55.133805118Z The files belonging to this database system will be owned by user "postgres".
2024-10-15T07:56:55.133869142Z This user must also own the server process.
2024-10-15T07:56:55.133873390Z 
2024-10-15T07:56:55.133876035Z The database cluster will be initialized with locale "en_US.utf8".
2024-10-15T07:56:55.133878430Z The default database encoding has accordingly been set to "UTF8".
2024-10-15T07:56:55.133880695Z The default text search configuration will be set to "english".
2024-10-15T07:56:55.133882869Z 
2024-10-15T07:56:55.133884863Z Data page checksums are disabled.
2024-10-15T07:56:55.133886987Z 
2024-10-15T07:56:55.133888971Z fixing permissions on existing directory /var/lib/postgresql/data ... ok
2024-10-15T07:56:55.134168685Z creating subdirectories ... ok
2024-10-15T07:56:55.134422578Z selecting dynamic shared memory implementation ... posix
2024-10-15T07:56:55.150935801Z selecting default max_connections ... 100
2024-10-15T07:56:55.174263840Z selecting default shared_buffers ... 128MB
2024-10-15T07:56:55.216190116Z selecting default time zone ... Etc/UTC
2024-10-15T07:56:55.217596482Z creating configuration files ... ok
2024-10-15T07:56:55.349159955Z running bootstrap script ... ok
2024-10-15T07:56:55.695253700Z performing post-bootstrap initialization ... ok
2024-10-15T07:56:55.803697118Z initdb: warning: enabling "trust" authentication for local connections
2024-10-15T07:56:55.803722016Z initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
2024-10-15T07:56:55.803843052Z syncing data to disk ... ok
2024-10-15T07:56:55.803847661Z 
2024-10-15T07:56:55.803849895Z 
2024-10-15T07:56:55.803851909Z Success. You can now start the database server using:
2024-10-15T07:56:55.803854153Z 
2024-10-15T07:56:55.803856168Z     pg_ctl -D /var/lib/postgresql/data -l logfile start
2024-10-15T07:56:55.803858242Z 
2024-10-15T07:56:55.854662839Z waiting for server to start....2024-10-15 07:56:55.854 UTC [48] LOG:  starting PostgreSQL 16.4 (Debian 16.4-1.pgdg120+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-10-15T07:56:55.855835380Z 2024-10-15 07:56:55.855 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-10-15T07:56:55.859702532Z 2024-10-15 07:56:55.859 UTC [51] LOG:  database system was shut down at 2024-10-15 07:56:55 UTC
2024-10-15T07:56:55.864666659Z 2024-10-15 07:56:55.864 UTC [48] LOG:  database system is ready to accept connections
2024-10-15T07:56:55.932710040Z  done
2024-10-15T07:56:55.932741210Z server started
2024-10-15T07:56:56.074679224Z CREATE DATABASE
2024-10-15T07:56:56.075621646Z 
2024-10-15T07:56:56.075759995Z 
2024-10-15T07:56:56.076010063Z /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
2024-10-15T07:56:56.076022997Z 
2024-10-15T07:56:56.077354798Z waiting for server to shut down....2024-10-15 07:56:56.077 UTC [48] LOG:  received fast shutdown request
2024-10-15T07:56:56.078513461Z 2024-10-15 07:56:56.078 UTC [48] LOG:  aborting any active transactions
2024-10-15T07:56:56.080440150Z 2024-10-15 07:56:56.080 UTC [48] LOG:  background worker "logical replication launcher" (PID 54) exited with exit code 1
2024-10-15T07:56:56.080760553Z 2024-10-15 07:56:56.080 UTC [49] LOG:  shutting down
2024-10-15T07:56:56.081804584Z 2024-10-15 07:56:56.081 UTC [49] LOG:  checkpoint starting: shutdown immediate
2024-10-15T07:56:56.125090085Z 2024-10-15 07:56:56.124 UTC [49] LOG:  checkpoint complete: wrote 922 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.017 s, sync=0.024 s, total=0.045 s; sync files=301, longest=0.002 s, average=0.001 s; distance=4255 kB, estimate=4255 kB; lsn=0/1912108, redo lsn=0/1912108
2024-10-15T07:56:56.135249688Z 2024-10-15 07:56:56.135 UTC [48] LOG:  database system is shut down
2024-10-15T07:56:56.177602324Z  done
2024-10-15T07:56:56.177639957Z server stopped
2024-10-15T07:56:56.179304455Z 
2024-10-15T07:56:56.179328913Z PostgreSQL init process complete; ready for start up.
2024-10-15T07:56:56.179334213Z 
2024-10-15T07:56:56.207993081Z 2024-10-15 07:56:56.207 UTC [1] LOG:  starting PostgreSQL 16.4 (Debian 16.4-1.pgdg120+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-10-15T07:56:56.208134637Z 2024-10-15 07:56:56.208 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-10-15T07:56:56.208144045Z 2024-10-15 07:56:56.208 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-10-15T07:56:56.210838837Z 2024-10-15 07:56:56.210 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-10-15T07:56:56.216249432Z 2024-10-15 07:56:56.216 UTC [64] LOG:  database system was shut down at 2024-10-15 07:56:56 UTC
2024-10-15T07:56:56.222571649Z 2024-10-15 07:56:56.222 UTC [1] LOG:  database system is ready to accept connections
2024-10-15T07:57:06.008125917Z 2024-10-15 07:57:06.007 UTC [76] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.008139714Z 2024-10-15 07:57:06.007 UTC [76] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = 'fix_flow_versioning_2')
2024-10-15T07:57:06.010364081Z 2024-10-15 07:57:06.010 UTC [77] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.010374922Z 2024-10-15 07:57:06.010 UTC [77] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = $1)
2024-10-15T07:57:06.026815293Z 2024-10-15 07:57:06.026 UTC [80] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.026828809Z 2024-10-15 07:57:06.026 UTC [80] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = 'fix_flow_versioning_2')
2024-10-15T07:57:06.029100528Z 2024-10-15 07:57:06.027 UTC [82] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.029111850Z 2024-10-15 07:57:06.027 UTC [82] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = $1)
2024-10-15T07:57:06.029114595Z 2024-10-15 07:57:06.029 UTC [79] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.029117100Z 2024-10-15 07:57:06.029 UTC [79] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = 'fix_flow_versioning_2')
2024-10-15T07:57:06.030642447Z 2024-10-15 07:57:06.030 UTC [81] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.030668178Z 2024-10-15 07:57:06.030 UTC [81] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = $1)
2024-10-15T07:57:06.051515610Z 2024-10-15 07:57:06.047 UTC [85] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.051529288Z 2024-10-15 07:57:06.047 UTC [85] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = 'fix_flow_versioning_2')
2024-10-15T07:57:06.051532193Z 2024-10-15 07:57:06.049 UTC [86] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.051534928Z 2024-10-15 07:57:06.049 UTC [86] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = $1)
2024-10-15T07:57:06.069812963Z 2024-10-15 07:57:06.068 UTC [88] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.069828854Z 2024-10-15 07:57:06.068 UTC [88] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = 'fix_flow_versioning_2')
2024-10-15T07:57:06.072060134Z 2024-10-15 07:57:06.069 UTC [89] ERROR:  relation "windmill_migrations" does not exist at character 32
2024-10-15T07:57:06.072073060Z 2024-10-15 07:57:06.069 UTC [89] STATEMENT:  SELECT EXISTS(SELECT name FROM windmill_migrations WHERE name = $1)

Steps to Reproduce

  1. Install coolify from the UI, default config
  2. Inspect logs

Example Repository URL

No response

Coolify Version

v4.0.0-beta460

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubunto 24.04

Additional Information

No response

@kikohs kikohs added 🐛 Bug Reported issues that need to be reproduced by the team. 🔍 Triage Issues that need assessment and prioritization. labels Oct 15, 2024
@che0one
Copy link
Contributor

che0one commented Oct 15, 2024

https://github.com/coollabsio/coolify/blob/460b9929940351b6c848d47116ac2a0201755a60/templates/compose/windmill.yaml
first version of compose, before all commits related to healtcheck, have the same error but at least works...

@kikohs
Copy link
Author

kikohs commented Oct 15, 2024

Ok. I have a 404. Something wrong with TLS but i thought it might be the postgres thing..

@peaklabs-dev
Copy link
Member

Please use this updated compose and let me know if you still get this issue: https://github.com/coollabsio/coolify/blob/next/templates/compose/windmill.yaml

@peaklabs-dev peaklabs-dev removed the 🔍 Triage Issues that need assessment and prioritization. label Oct 15, 2024
@kikohs
Copy link
Author

kikohs commented Oct 15, 2024

it works now!
Thank you

@kikohs kikohs closed this as completed Oct 15, 2024
@github-actions github-actions bot removed the 🐛 Bug Reported issues that need to be reproduced by the team. label Oct 15, 2024
@kikohs
Copy link
Author

kikohs commented Oct 15, 2024

I can't run any jobs in any lang. It seems the workers are not detected

Screenshot 2024-10-15 171222

@kikohs kikohs reopened this Oct 15, 2024
@davidpp
Copy link

davidpp commented Oct 15, 2024

Just hit the same issues with workers, will let you know if I find the cause.

@kikohs
Copy link
Author

kikohs commented Oct 15, 2024

This docker-compose works for me now. I only need 1 worker and 1 native:

# documentation: https://www.windmill.dev/docs/

# slogan: Windmill is a developer platform to build production-grade multi-steps automations and internal apps.

# tags: windmill,workflow,automation,developer,platform

# logo: svgs/windmill.svg

# port: 8000

  

services:

  db:

    image: postgres:16

    shm_size: 1g

    restart: unless-stopped

    volumes:

      - db-data:/var/lib/postgresql/data

    environment:

      - POSTGRES_PASSWORD=$SERVICE_PASSWORD_POSTGRES

      - POSTGRES_DB=${POSTGRES_DB:-windmill}

    healthcheck:

      test: ["CMD-SHELL", "pg_isready -U postgres"]

      interval: 10s

      timeout: 5s

      retries: 5

  

  windmill-server:

    image: ghcr.io/windmill-labs/windmill:main

    restart: unless-stopped

    environment:

      - SERVICE_FQDN_WINDMILL_8000

      - DATABASE_URL=postgres://postgres:$SERVICE_PASSWORD_POSTGRES@db/windmill

      - MODE=server

      - BASE_URL=$SERVICE_FQDN_WINDMILL

    depends_on:

      db:

        condition: service_healthy

    volumes:

      - worker-logs:/tmp/windmill/logs

    healthcheck:

      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]

      interval: 30s

      timeout: 10s

      retries: 3

  

  windmill-worker:

    image: ghcr.io/windmill-labs/windmill:main

    restart: unless-stopped

    environment:

      - DATABASE_URL=postgres://postgres:$SERVICE_PASSWORD_POSTGRES@db/windmill

      - MODE=worker

      - WORKER_GROUP=default

    depends_on:

      db:

        condition: service_healthy

    volumes:

      - /var/run/docker.sock:/var/run/docker.sock

      - worker-dependency-cache:/tmp/windmill/cache

      - worker-logs:/tmp/windmill/logs

    healthcheck:

      test: ["CMD-SHELL", "exit 0"]

      interval: 30s

      timeout: 10s

      retries: 3

  

  windmill-worker-native:

    image: ghcr.io/windmill-labs/windmill:main

    restart: unless-stopped

    environment:

      - DATABASE_URL=postgres://postgres:$SERVICE_PASSWORD_POSTGRES@db/windmill

      - MODE=worker

      - WORKER_GROUP=native

      - NUM_WORKERS=8

      - SLEEP_QUEUE=200

    depends_on:

      db:

        condition: service_healthy

    volumes:

      - worker-logs:/tmp/windmill/logs

    healthcheck:

      test: ["CMD-SHELL", "exit 0"]

      interval: 30s

      timeout: 10s

      retries: 3

  

  lsp:

    image: ghcr.io/windmill-labs/windmill-lsp:latest

    restart: unless-stopped

    volumes:

      - lsp-cache:/root/.cache

    healthcheck:

      test: ["CMD-SHELL", "exit 0"]

      interval: 30s

      timeout: 10s

      retries: 3

  

volumes:

  db-data:

  worker-dependency-cache:

  worker-logs:

  lsp-cache:

@davidpp
Copy link

davidpp commented Oct 15, 2024

Yeah,

The issue was the MODE that is reading the variable, but variable is "server" and worker need "worker" mode.

services:
  db:
    image: 'postgres:16'
    shm_size: 1g
    volumes:
      - 'db-data:/var/lib/postgresql/data'
    environment:
      - POSTGRES_PASSWORD=$SERVICE_PASSWORD_POSTGRES
      - 'POSTGRES_DB=${POSTGRES_DB:-windmill}'
    healthcheck:
      test:
        - CMD-SHELL
        - 'pg_isready -U postgres'
      interval: 10s
      timeout: 5s
      retries: 5
  windmill-server:
    image: 'ghcr.io/windmill-labs/windmill:main'
    environment:
      - SERVICE_FQDN_WINDMILL_8000
      - 'DATABASE_URL=postgres://postgres:$SERVICE_PASSWORD_POSTGRES@db/windmill'
      - 'MODE=server'
      - BASE_URL=$SERVICE_FQDN_WINDMILL
    depends_on:
      db:
        condition: service_healthy
    volumes:
      - 'worker-logs:/tmp/windmill/logs'
    healthcheck:
      test:
        - CMD
        - curl
        - '-f'
        - 'http://localhost:8000/health'
      interval: 30s
      timeout: 10s
      retries: 3
  windmill-worker-1:
    image: 'ghcr.io/windmill-labs/windmill:main'
    environment:
      - 'DATABASE_URL=postgres://postgres:$SERVICE_PASSWORD_POSTGRES@db/windmill'
      - 'MODE=worker'
      - 'WORKER_GROUP=${WORKER_GROUP:-default}'
    depends_on:
      db:
        condition: service_healthy
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
      - 'worker-dependency-cache:/tmp/windmill/cache'
      - 'worker-logs:/tmp/windmill/logs'
    healthcheck:
      test:
        - CMD-SHELL
        - 'exit 0'
      interval: 30s
      timeout: 10s
      retries: 3
  windmill-worker-2:
    image: 'ghcr.io/windmill-labs/windmill:main'
    environment:
      - 'DATABASE_URL=postgres://postgres:$SERVICE_PASSWORD_POSTGRES@db/windmill'
      - 'MODE=worker'
      - 'WORKER_GROUP=${WORKER_GROUP:-default}'
    depends_on:
      db:
        condition: service_healthy
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
      - 'worker-dependency-cache:/tmp/windmill/cache'
      - 'worker-logs:/tmp/windmill/logs'
    healthcheck:
      test:
        - CMD-SHELL
        - 'exit 0'
      interval: 30s
      timeout: 10s
      retries: 3
  windmill-worker-3:
    image: 'ghcr.io/windmill-labs/windmill:main'
    environment:
      - 'DATABASE_URL=postgres://postgres:$SERVICE_PASSWORD_POSTGRES@db/windmill'
      - 'MODE=worker'
      - 'WORKER_GROUP=${WORKER_GROUP:-default}'
    depends_on:
      db:
        condition: service_healthy
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
      - 'worker-dependency-cache:/tmp/windmill/cache'
      - 'worker-logs:/tmp/windmill/logs'
    healthcheck:
      test:
        - CMD-SHELL
        - 'exit 0'
      interval: 30s
      timeout: 10s
      retries: 3
  windmill-worker-native:
    image: 'ghcr.io/windmill-labs/windmill:main'
    environment:
      - 'DATABASE_URL=postgres://postgres:$SERVICE_PASSWORD_POSTGRES@db/windmill'
      - 'MODE=worker'
      - 'WORKER_GROUP=native'
      - 'NUM_WORKERS=${NUM_WORKERS:-8}'
      - 'SLEEP_QUEUE=${SLEEP_QUEUE:-200}'
    depends_on:
      db:
        condition: service_healthy
    volumes:
      - 'worker-logs:/tmp/windmill/logs'
    healthcheck:
      test:
        - CMD-SHELL
        - 'exit 0'
      interval: 30s
      timeout: 10s
      retries: 3
  lsp:
    image: 'ghcr.io/windmill-labs/windmill-lsp:latest'
    volumes:
      - 'lsp-cache:/root/.cache'
    healthcheck:
      test:
        - CMD-SHELL
        - 'exit 0'
      interval: 30s
      timeout: 10s
      retries: 3

@rubenfiszel
Copy link

I created a PR (did not test) with more sane compose (founder of Windmill)

#3924

@peaklabs-dev peaklabs-dev self-assigned this Oct 17, 2024
@peaklabs-dev peaklabs-dev added 🐛 Bug Reported issues that need to be reproduced by the team. 🐞 Confirmed Bug Verified issues that have been reproduced by the team. ✅ Done Issues that are fixed and a PR is ready to be merged. labels Oct 17, 2024
@peaklabs-dev
Copy link
Member

I fixed some things again, could you please retest if it works now: https://github.com/coollabsio/coolify/blob/next/templates/compose/windmill.yaml

@kikohs
Copy link
Author

kikohs commented Oct 17, 2024

I managed to get it to work with a reduced number of workers yes.
Thanks

@kikohs kikohs closed this as completed Oct 17, 2024
@github-actions github-actions bot removed 🐛 Bug Reported issues that need to be reproduced by the team. 🐞 Confirmed Bug Verified issues that have been reproduced by the team. ✅ Done Issues that are fixed and a PR is ready to be merged. labels Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants