docker image / container runs fine with docker run, but not when deployed to Kubernetes (unprivileged)

[divStar]

Hello!
Thank you very much for such a handy tool.

I use the following Dockerfile to build the image:

FROM alpine:latest

# Install required packages
RUN apk add --no-cache curl postgresql-client bash diffutils

# Latest releases available at https://github.com/aptible/supercronic/releases
ENV SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/v0.2.2/supercronic-linux-amd64 \
    SUPERCRONIC=supercronic-linux-amd64 \
    SUPERCRONIC_SHA1SUM=2319da694833c7a147976b8e5f337cd83397d6be

RUN curl -fsSLO "$SUPERCRONIC_URL" \
 && echo "${SUPERCRONIC_SHA1SUM}  ${SUPERCRONIC}" | sha1sum -c - \
 && chmod +x "$SUPERCRONIC" \
 && mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" \
 && ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic

# Create a dummy query.sh
RUN mkdir /workdir
RUN echo "#!/bin/bash\necho 'Replace the /workdir/query.sh with your own by mounting it!'" > /workdir/query.sh
RUN chmod +x /workdir/query.sh

# Copy relevant certificate
COPY ca-certificates/BaltimoreCyberTrustRoot.crt.pem /certs/

# Define a default cron expression
ENV CRON_EXPRESSION "* * * * *"

# Define default log level
ENV LOG_LEVEL "2"

# Create a startup script that sets up the cron job
RUN echo -e "${CRON_EXPRESSION} /workdir/query.sh > /proc/1/fd/1 2>&1" > /crontab
RUN chmod -R 0777 /workdir

WORKDIR /workdir

# Start cron with the configured expression and run the startup script
#CMD ["/bin/sh", "-c", "./start.sh & crond -l $LOG_LEVEL -f"]
CMD sh -c "id && supercronic /crontab"

I run the image/container using the following command from CLI just fine:
docker run -t --init --user 0:0 --cap-drop ALL --security-opt no-new-privileges --read-only --group-add 1 --group-add 2 --group-add 3 --group-add 4 --group-add 6 --group-add 10 --group-add 11 --group-add 20 --group-add 26 --group-add 27 --rm -e CRON_EXPRESSION="* * * * *" -v /.../query.sh:/workdir/query.sh:ro .../cronic-image:1
It also works flawlessly if I do not group-add any groups at all and no matter what user I specify (even nobody:nogroup works).

The output from docker run is about something like this:

uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
INFO[2023-04-07T02:11:04Z] read crontab: /crontab                       
INFO[2023-04-07T02:12:00Z] starting                                      iteration=0 job.command="/workdir/query.sh > /proc/1/fd/1 2>&1" job.position=0 job.schedule="* * * * *"
 count 
-------
     3
(1 row)

INFO[2023-04-07T02:12:00Z] job succeeded                                 iteration=0 job.command="/workdir/query.sh > /proc/1/fd/1 2>&1" job.position=0 job.schedule="* * * * *"

However: when I deploy it to our Kubernetes cluster, I get the following output from ArgoCD:

uid=0(root) gid=0(root) groups=1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
time="2023-04-07T02:12:47Z" level=info msg="read crontab: /crontab"
time="2023-04-07T02:13:00Z" level=info msg=starting iteration=0 job.command="/workdir/query.sh > /proc/1/fd/1 2>&1" job.position=0 job.schedule="* * * * *"
/bin/sh: /workdir/query.sh: Permission denied
time="2023-04-07T02:13:00Z" level=error msg="error running command: exit status 126" iteration=0 job.command="/workdir/query.sh > /proc/1/fd/1 2>&1" job.position=0 job.schedule="* * * * *"

Note: in groups=... 0(root) is not included when deployed to the Kubernetes cluster, but is very much so if I run it locally using docker run....

I suspect, that the Kubernetes cluster is restricted and we are not supposed to run unprivileged images and containers, but I thought I had made the container work in an unprivileged manner.

Do you happen to have an idea why that is and how to fix it? I really I tried everything I could. Setting securityContext along with runAsUser: ... and runAsGroup, but nothing works. It seems supercronic doesn't have enough permissions to execute /workdir/query.sh, that I mount...

[turner-aptible]

Hi @divStar. Currently, Supercronic was developed for our platform which is primarily Docker-focused, so within the K8s landscape isn't something we developed for. After reviewing your specifics I think that this issue comes down to the fsGroup policy not being defined within the securityContext. Depending on the version you are running this could cause an issue if not defined. You can see the blog post for K8s that might be useful. I'd also revamp the dockerfile to be explicit with a new user that adds a user and group with their GUIDs to match your securityContext. I'd then make sure the permissions are set for that new user on the paths. Below is a snippet to add a user as non-root. There is an optional snippet for adding doas for sudo like functionality, but that was more for reference.

ARG USER=test-user
ARG UID=1000
ARG GID=$UID

RUN addgroup \
        --gid "$GID" \
        "$USER"

    && adduser \
        --disabled-password \
        --gecos "" \
        --home "$(pwd)" \
        --ingroup "$USER" \
        --no-create-home \
        --uid "$UID" \
        "$USER"
    # Optional add doas for sudo commands
    && apk add doas; \
    echo 'permit $USER as root' > /etc/doas.d/doas.conf

USER $USER

Hopefully, that helps you move a little further in your troubleshooting.

[divStar]

Since I had a bit of trouble finding a full Dockerfile in the beginning, here is a working example for an unprivileged Dockerfile with Supercronic in it:

FROM alpine:latest

# Install required packages
RUN apk add --no-cache curl postgresql-client bash diffutils

# Latest releases available at https://github.com/aptible/supercronic/releases
ENV SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/v0.2.2/supercronic-linux-amd64 \
    SUPERCRONIC=supercronic-linux-amd64 \
    SUPERCRONIC_SHA1SUM=2319da694833c7a147976b8e5f337cd83397d6be

RUN curl -fsSLO "$SUPERCRONIC_URL" \
 && echo "${SUPERCRONIC_SHA1SUM}  ${SUPERCRONIC}" | sha1sum -c - \
 && chmod +x "$SUPERCRONIC" \
 && mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" \
 && ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic

# Create a dummy query.sh
RUN mkdir /workdir
RUN echo "#!/bin/bash\necho 'Replace the /workdir/query.sh with your own by mounting it!'" > /workdir/query.sh
RUN chmod +x /workdir/query.sh

# Define a default cron expression
ENV CRON_EXPRESSION "* * * * *"

# Create a startup script that sets up the cron job
RUN echo -e "${CRON_EXPRESSION} /workdir/query.sh > /proc/1/fd/1 2>&1" > /crontab
RUN chmod -R 0777 /workdir

# Create user and group
ENV USER_ID=1337
ENV GROUP_ID=1337
ENV USER_NAME=appuser
ENV GROUP_NAME=appgroup

RUN addgroup -g $GROUP_ID $GROUP_NAME && \
    adduser -D --uid $USER_ID --ingroup $GROUP_NAME $USER_NAME

WORKDIR /workdir
USER appuser

# Start cron with the configured expression and run the startup script
#CMD ["/bin/sh", "-c", "./start.sh & crond -l $LOG_LEVEL -f"]
CMD sh -c "id && supercronic /crontab"

In my deployment.yaml I included this securityContext:

      securityContext:
        runAsUser: 1337
        runAsGroup: 1337
        fsGroup: 1337

I am not 100% sure which of the changes did the trick, but it works.

Also for those reading: you might want to add / remove particular packages and you might not want id to run as one of the CMD commands.

Thank you for your help :).

[divStar]

@turner-aptible I actually do have one more question.

RUN echo -e "${CRON_EXPRESSION} /workdir/query.sh > /proc/1/fd/1 2>&1" > /crontab does not work, because it uses the environment variable set at build time. What I want though is to change this expression when the image is run.

I suppose I could somehow make that line part of the CMD command, but it seems a bit much. Is there a way to have /crontab refer to an environment variable at runtime in order to know when to run?

My current solution is:

...
CMD sh -c "id && supercronic <(echo '$(cat "/crontab" | sed "s/\$CRON_EXPRESSION/$(echo "$CRON_EXPRESSION" | sed 's/[\\/]/\\\//')/")')"

This replaces the hardcoded $CRON_EXPRESSION in the /crontab file with the actual environment variable value on startup. It works, but is somewhat hard to read.