[Bug]: Can't login to Trow registry when secured with password

[jzaron]

Describe the bug

I deploy trow on a GKE cluster using Helm chart. When I set user/password to protect it, I can't docker login to list/push/pull images:

kuba@***:~$ docker login -u <correct user> -p <correct password> trow.*****
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: login attempt to https://trow.*****/v2/ failed with status: 401 Unauthorized

However, when I give wrong credentials an error is different:

kuba@***:~$ docker login -u <correct user> -p <incorrect password> trow.*****
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://trow.image-proxy-test.respo.tech/v2/": unauthorized: authentication required

I'm sure that the network setup (Service, Ingress, TLS cert) is fine - when I comment out user/password in values.yaml all works fine - I can list/push/pull images.

I also checked the K8s password Secret value and it's mount as a file in the Trow container - they are fine. The file's content is the same as the password I enter into values.yaml.

To Reproduce

GKE cluster version: 1.30.x
GKE node pools version: 1.30.x
trow chart version: 0.8.1
trow image tag: 0.6.4

trow is set up with Google Managed Certificates on GKE as in HELM_INSTALL.md

values.yaml content (password and domain masked):

# Default values for trow.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: ghcr.io/trow-registry/trow
  tag:
  pullPolicy: Always

trow:
  ## if using NodePort, this can be set to 127.0.0.1:XXXX
  ## Where "XXXX" is the value of service.nodePort
  domain: trow.*****
  user: user
  password: *****

  ## The Trow validation webhook validates which images are allowed to run on your cluster.
  ## Note: it runs AFTER the mutating webhook.
  validationWebhook:
    enabled: true
    failurePolicy: Ignore
    ## The validation webhook need to be configured.
    ## Note: it runs AFTER mutation webhooks.
    ## Example:
    config:
      ## Default action for unknown images (Deny or Allow)
      default: Allow
      ## List of allowed image URI prefixes
      allow:
        - 0.0.0.0/0
      deny: []
        # - 127.0.0.1/f/docker/nope/
  ## The proxyRegistries section allows you to configure Trow to act as a proxy for other registries.
  proxyRegistries:
    ## The Trow mutating webhook will automatically rewrite the pod image URLs to use the Trow proxy.
    webhook:
      enabled: true
      failurePolicy: Ignore
    config:
      # offline: false
      registries:
        - alias: docker
          host: registry-1.docker.io
      #   - alias: ghcr
      #     host: ghcr.io
      #     ignore_repos: ["trow-registry/trow"]
  ## For more info on log levels see https://docs.rs/tracing-subscriber/0.3.17/tracing_subscriber/filter/struct.EnvFilter.html
  logLevel: info

## Job to configure the webhooks
webhooks:
  ## It is possible to restrict webhook access to some namespaces, ref:
  ##   https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector
  # namespaceSelector:
  replicas: 2
  ## Config to create a TLS cert for the webhooks (required if webhooks are used)
  tls:
    ## Create the cert using certmanager
    certmanager:
      enabled: false
      rootCert:
        # default to be 5y
        duration: ""
      admissionCert:
        # default to be 1y
        duration: ""
      # issuerRef:
      #   name: "issuer"
      #   kind: "ClusterIssuer"
    ## Create a self-signed cert using kube-webhook-certgen to patch the webhooks (creates ClusterRole)
    patch:
      enabled: true
      image:
        image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.4@sha256:a9f03b34a3cbfbb26d103a14046ab2c5130a80c3d69d526ff8063d2b37b9fd3f
        pullPolicy: IfNotPresent
    ## Use an existing secret that contains tls.crt and tls.key
    existingSecretRef: ~

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

service:
  type: NodePort
  port: 8000
  ## Specify the nodePort value for the LoadBalancer and NodePort service types.
  nodePort:

ingress:
  enabled: true
  gke: true
  annotations:
        networking.gke.io/managed-certificates: trow-certificate
    # kubernetes.io/ingress.class: "gce"
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  hosts:
    - paths: ["/"]
      # use "none" to not set a host (otherwise defaults to trow.domain)
      host:
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources:
  limits:
    memory: 256Mi
  #   cpu: 100m
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

# run pods on dedicated, persistent nodes (other nodes are in autoscaling groups with min number of instances set to 0)
nodeSelector:
  *****.cluster.node.type: infra-services

tolerations:
  - key: components.gke.io/gke-managed-components
    operator: Exists
    effect: NoSchedule

affinity: {}

volumeClaim:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 40Gi

Expected behavior

I should be able to protect Trow registry with user and password and be able to log in with proper credentials.

Output/Logs

During failed docker login attempts (the ones with proper credentials) I can see in trow pod logs:

2025-01-09T22:33:46.906117Z  INFO request{method="GET" path="/v2/"}: trow::routes: done in 54.535µs
2025-01-09T22:33:47.012424Z  INFO request{method="GET" path="/login"}: trow::routes: done in 18.573506ms
2025-01-09T22:33:47.137624Z  WARN request{method="GET" path="/v2/"}: trow::response::trow_token: Failed to decode secret: Base64 error: Invalid byte 45, offset 8.
2025-01-09T22:33:47.137675Z  INFO request{method="GET" path="/v2/"}: trow::routes: done in 159.229µs

With improper credentials logs are different:

2025-01-09T22:37:10.743599Z  INFO request{method="GET" path="/v2/"}: trow::routes: done in 53.722µs
2025-01-09T22:37:10.846017Z  INFO request{method="GET" path="/login"}: trow::routes: done in 18.306034ms

Trow Info

trow chart version: trow-0.8.1
trow image tag: 0.6.4

Kubernetes

• GKE cluster version: 1.30.x
• GKE node pools version: 1.30.x
• Container Runtime: containerd://1.7.24

Additional context

Trow seems great and it covers my use case perfectly but I can't use it when exposed unprotected to the Internet as this would mean exposing company's closed code. It has to be somehow secured. Maybe it's possible to provide a setup when it's protected on networking level, but password protection seems easier (except I can't make it work 😅 ).

What's interesting is that I can get a proper JWT token with:

kuba@***:~$ curl -u <proper user>:<proper password> "https://trow.*****/login"
{"token": "eyJ0eX(...)"}

I wonder if building Trow image from the master branch and using it during deployment could solve this problem? Newest version is 0.8.1, whereas the newest available Docker image is from 0.6.4 version.

[awoimbee]

Hi, thanks for the report.

Locally it seems to work:

$ cargo run -- -U toto -P p4ss &
$ curl -i localhost:8000/login 
HTTP/1.1 401 Unauthorized
$ curl -iu toto:p4ss localhost:8000/login
HTTP/1.1 200 OK

I'll check if the helm chart is not rendering correctly.

---

Newest version is 0.8.1, whereas the newest available Docker image is from 0.6.4 version

Latest version of the helm chart is 0.8.1, latest version of trow is 0.6.4 (the helm release CI action creates confusing release names :/).

[jzaron]

Thanks for the answer.

The thing is that in my case this curl works - when I set user/password I get a token in response.

What's interesting is that I can get a proper JWT token with:

kuba@***:~$ curl -u <proper user>:<proper password> "https://trow.*****/login"
{"token": "eyJ0eX(...)"}

But docker login fails with Error response from daemon: login attempt to https://trow.*****/v2/ failed with status: 401 Unauthorized

About the Trow version - in Git there is the trow-0.8.1 tag, and there a lot of differences in Rust code between v0.6.4 tagged revision, also regarding auth (changes in src/response/trow_token.rs). But if I understand correctly the trow-* tags are only Helm chart releases, not Trow releases?

[awoimbee]

Ah sorry, indeed I can reproduce:

$ RUST_LOG=debug cargo run -- -U toto -P p4ss &
$ podman login --tls-verify=false -u toto -p p4ss 127.0.0.1:8000
Error: logging into "127.0.0.1:8000": invalid username/password

The error: trow::response::trow_token: Failed to decode secret: Base64 error: Invalid byte 45, offset 8..

---

About the Trow version - in Git there is the [...]

Yes trow-* tags are created by helm/chart-releaser-action.
The rust code has changed quite a bit because I'm refactoring a lot of code (getting close to completion, even though I don't spend a lot of time on Trow !).

[jzaron]

Thanks. So it seems there is an actual bug.

1. Is it more of a biggie, or an easy one?
2. Do you see some workaround for this?