URLs with special characters don't work in the presence of URL rewriting

[wknapik]

NGINX Ingress controller version:
0.26.1

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2019-05-01T04:14:38Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Debian 10 (quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1)
• Kernel (e.g. uname -a): Linux redacted 4.9.0-11-amd64 Basic structure  #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux (via kops)
• Install tools: kops 1.11.1
• Others:

What happened:

If urls are being rewritten for the given ingress, requests to urls with some special characters, such as | (%7c), get decoded before being passed upstream, producing an invalid path.

What you expected to happen:

I expected to be able to have any (correctly encoded) characters in urls.

How to reproduce it:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /site/$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-service
          servicePort: 8080
        path: /(/*|$)(.*)

Accessing https://example.com/whatever%7Cwhatever produces a proxied request to the decoded url /site/whatever|whatever, which is invalid and ends in a 400 response.

It appears there are only two ways out of it now (and possibly, as a result of this bug report, a third one in the future):

1. Create a custom nginx config.
2. Do the rewriting in lua.
3. Modify ingress-nginx to allow overriding the proxy_pass argument.

I haven't tried the first solution, but I don't want to do that anyway - maintaining the custom config is not something I want to do.

I will be trying the second option now (any hints would be appreciated!), hopefully this will work, but it would still be better to do it statically/declaratively in the generated config, than running custom lua code per request. If this does turn out to be a fix, I would suggest including it in official documentation.

The third option would be an annotation like

some.annotation.for.proxy_pass: http://upstream_balancer/site$request_uri

or

some.annotation.for.proxy_pass.path: /site$request_uri

That should achieve a rewrite without a rewrite and produce valid urls for the upstream service.

Anything else we need to know:

I tried dropping the regex altogether (matching on path /) and rewriting to /site$request_uri?, but then I end up with double encoding.

That's fixable for the query part of the url with something like:

nginx.ingress.kubernetes.io/server-snippet: |
    rewrite ^ /site$request_uri;
    rewrite ^([^?]*)(\?.*) $1 break;

but that still leaves the path part double-encoded, so %7c becomes %257c, which is not the url being requested.

This is unexpected and quite difficult to sort out. Ideally, ingress-nginx would just handle it automagically. If that's not an option, being able to override the proxy_pass argument with an annotation would probably be sufficient to make this work.

Relevant resources:

• https://trac.nginx.org/nginx/ticket/727
• https://stackoverflow.com/questions/28684300/

/kind bug

[ElvinEfendi]

For the above example if ingress-nginx generated following Nginx configuration maybe it would fix the problem (based on the SO answer):

    location /(/*|$)(.*) {
        rewrite ^ $request_uri;
        rewrite (?i)(/*|$)(.*) /site/$2 break;
        proxy_pass http://upstream_balancer/$uri;
    }

Currently ingress-nginx would generate:

    location /(/*|$)(.*) {
        rewrite (?i)(/*|$)(.*) /site/$2 break;
        proxy_pass http://upstream_balancer;
    }

It seems safe to do - but I'd have to experiment more with this. I think we have to be careful to not introduce an insecure default here.

@wknapik you can experiment with this by modifying

ingress-nginx/internal/ingress/controller/template/template.go

Line 480 in a082962

func buildProxyPass(host string, b interface{}, loc interface{}) string {

.

[wknapik]

ngx.req.set_uri("/site" .. ngx.var.uri) produces the same effect as the pure-nginx rewrite.

[wknapik]

@ElvinEfendi I need a quick fix for now, so modifying ingress-nginx is not really an option for me.

This appears to work, though more testing is needed:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      location / {
        set $proxy_upstream_name "<redacted>";
        proxy_pass http://upstream_balancer/site$request_uri;
      }
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-service
          servicePort: 8080
        path: /

I'm not sure why it breaks without the set $proxy_upstream_name line, should be inherited from the parent location...

This can be adapted for arbitrary rewrites (see: #5576 (comment)), not just adding a path prefix.

[aledbf]

Today this was reported https://trac.nginx.org/nginx/ticket/1808
Not sure the suggested rewrite is the right fix for all the cases

[wknapik]

@aledbf I responded in the nginx issue with the following:

You can do this for redirects:

location / {
   rewrite ^ $request_uri;
   rewrite /foo/(.*) /bar/$1;
   return 301 $uri;
}

and this for proxying

location / {
   rewrite ^ $request_uri;
   rewrite /foo/(.*) /bar/$1 break;
   proxy_pass http://127.0.0.1:8080$uri;
}

I still think the decoding that nginx does makes no sense, but maybe I'm missing something.

In any case, if you need a solution that doesn't involve modifying nginx, the above should work.

The second example is what we're discussing here, the first one is an adaptation of the same approach for redirects.

[wknapik]

In theory, if ingress-nginx was to adopt this approach, it would be a backwards-incompatible change, because rewrite ^ $request_uri; causes further rewrites to be performed on an encoded url.

In most cases, the regexes would not require any changes, but if either the matching, or the replacement involved special characters, the end user would have to take that into account. This might be risky not only on the functional level, but also in terms of security.

All this would require testing to confirm.

[aledbf]

In theory, if ingress-nginx was to adopt this approach, it would be a backwards-incompatible change, because rewrite ^ $request_uri; causes further rewrites to be performed on an encoded url.

Just in case, we are not going to introduce any breaking change to rewrites. Users still complain about the change introduced in 0.20.0.

[wknapik]

Just in case, we are not going to introduce any breaking change to rewrites. Users still complain about the change introduced in 0.20.0.

I don't think nginx offers any other way to deal with this at the moment.

IMO this has to be considered a bug in ingress-nginx, even if the root cause is on the nginx side.

There is nothing wrong in the original config from this issue and it still breaks on some characters in valid URLs.

[wknapik]

The best solution would be for nginx to encode paths before they're passed on - to upstream servers (via proxy_pass), to the client (via the location header in redirects), or anywhere else.

I can't figure out why anyone thought it would be a good idea to pass those paths in their decoded form.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[wknapik]

/remove-lifecycle stale

[DaleyKD]

We're currently trying to get something like this to work using annotations. It seems like people think this isn't that big of a deal, but for us, we simply want a REST endpoint where we can put a URL as part of the path.

e.g. - https://www.mysite.com/myapp/api/controller/https%3A%2F%2Fexample.com%2Fsites

I was able to get this to work in my local dev, which doesn't use ingress-controller, by doing the following:

location ~* "^/myapp(/|$)(.*)" {
    set $upstream myapp.web;
    rewrite ^ $request_uri;
    rewrite "(?i)/myapp(/|$)(.*)" /$2 break;
    return 400;
    proxy_pass http://$upstream$uri;
}

We can't seem to get this to work in our k8s environment, even with us trying to use some of these workarounds.

This is a snippet from our values file for k8s:

    nginx.ingress.kubernetes.io/configuration-snippet: |
      location ^/myapp(/|$)(.*) {
        set $proxy_upstream_name "myapp.web";
        rewrite ^ $request_uri;
        rewrite "(?i)/myapp(/|$)(.*)" /$2 break;
        return 400;
        proxy_pass http://upstream_balancer$uri;
      }
  path: /myapp(/|$)(.*)

I feel like we're missing something so simple...

[dpavelescu]

The scenario I'm looking to implement:
mydomain/myapp/someurlencoded => mydomain/someurlencoded
also added a question here: https://stackoverflow.com/questions/65378554/nginx-url-rewrite-without-url-decoding

1. Do you know if any native support is planned in the near future for this scenario as part of ingress nginx?
2. Do you know how such a scenario shall be configured in ngnix reverse proxy?

many thanks

[dlandtwing]

@ElvinEfendi I need a quick fix for now, so modifying ingress-nginx is not really an option for me.

This appears to work, though more testing is needed:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      location / {
        set $proxy_upstream_name "<redacted>";
        proxy_pass http://upstream_balancer/site$request_uri;
      }
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-service
          servicePort: 8080
        path: /

I'm not sure why it breaks without the set $proxy_upstream_name line, should be inherited from the parent location...

This can be adapted for arbitrary rewrites (see: #5576 (comment)), not just adding a path prefix.

If you're using load balancing with sticky sessions (i.e. nginx.ingress.kubernetes.io/affinity-mode: persistent), don't forget to add the following to your snippet
set $location_path "[your location path, see generated config]";
The $location_path, like $proxy_upstream_name, is not inherited from the parent location and is required for the lua balancer to work

[calumgoodchild]

I ran:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($request_uri ~ "^/example(/.*)") {
        proxy_pass http://upstream_balancer$1;
        break;
      }
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-service
          servicePort: 8080
        path: /example

It's a little messy as it's a conditional statement, I used this because I had multiple paths, with the same rule needing to be applied. Also it creates the same if statement in each rule which isn't ideal, but it works.

It removes the rewrite so URLs don't get decoded upstream.

I would like to see an annotation added as well, this fix isn't ideal and takes away from the cleanliness of yaml.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[wknapik]

/remove-lifecycle stale

[aeimer]

We are facing the same issue.
The Ingresses are looking like:

metadata:
  name: admin-service
  annotations:
    kubernetes.io/ingress.class: main
    nginx.ingress.kubernetes.io/proxy-read-timeout: '120'
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/x-forwarded-prefix: /admin-service
spec:
  tls:
    - hosts:
        - somehost.som
      secretName: somecert
  rules:
    - host: somehost.com
      http:
        paths:
          - path: /admin-service/(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: admin-service
                port:
                  name: http

This will expose our service under somehost.com/admin-service/foo/bar and send an http upstream request against admin-service.namespace/foo/bar instead of admin-service.namespace/admin-service/foo/bar (the intended behaviour of rewrite-target works with this simple path).

If the url has some mandatory to be encoded chars like somehost.com/admin-service/foo$bar in it, it will be transmitted like somehost.com%2Fadmin-service%2Ffoo%24bar the ingress will then decode it for the sake of rewriting and then call admin-service.namespace/foo$bar, but instead it should call admin-service.namespace%2Ffoo%24bar.
If the calling url will not be encoded again, the HTTP call fails.

Is my understanding correct?
Is there a chance that we will get this issue fixed?

[damiancurry]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[wknapik]

/remove-lifecycle stale

[kev24uk]

The conditional statement works for me too, thanks! Unfortunately it does stop the CORS rules working though...any ideas on how to solve that?

[yishaihl]

@kev24uk hey maybe you or someone can help me (busting my head on it for hours)?
i need to pass this source url to the backend: https://something.com/consult/Q6psbroRO_?version=pr-af3807d629af4ff16e9840c563ec35369e797267 and i want to pass /consult/Q6psbroRO_?version=pr-af3807d629af4ff16e9840c563ec35369e797267 what should be set in the conditional statement for $request_uri (if so) and for upstream_balancer$ (if so) and for "^/example(/.*)" ? thanks a lot for your help

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[wknapik]

/remove-lifecycle rotten

[iamNoah1]

@damiancurry friendly ping. Are you still working on that one?
/triage accepted
/priority important-longterm

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[aeimer]

/remove-lifecycle stale

[nikowitt]

Just encountered this issue as (having a Tomcat in backend). Unfortunately, the suggested workarounds fail due to kubernetes/kubernetes#126811 as now proxy-pass is on the blacklist.

Are there any other ideas to bypass this except for modifying backend to allow the characters?

[rayate2410]

I ran:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($request_uri ~ "^/example(/.*)") {
        proxy_pass http://upstream_balancer$1;
        break;
      }
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: example-service
          servicePort: 8080
        path: /example

It's a little messy as it's a conditional statement, I used this because I had multiple paths, with the same rule needing to be applied. Also it creates the same if statement in each rule which isn't ideal, but it works.
It removes the rewrite so URLs don't get decoded upstream.
I would like to see an annotation added as well, this fix isn't ideal and takes away from the cleanliness of yaml.

Great solution, I was finally able to make rabbitmqAdmin running correctly for vhost "/".

@brutus333 would you be able to share the ingress yaml for rabbitmq UI. I'm also facing the same issue with vhost "/"

[willemm]

To be fair, all bug fixes are "breaking changes".

Our teams are running into the same issue, so we're wondering how big of a security hole it is to allow config snippets with location and/or proxy_pass in them.

[willemm]

This almost works, and doesn't use any of the 'forbidden' snippet keywords:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^ $request_uri;
      reqrite ^([^?]*) $1;
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: example-service
            port:
              number: 8080
        path: /example(/|$)(.*)
        pathType: Prefix

I say almost, because it goes wrong is when you use query parameters, those get duplicated in the path.
Edit: By making the path match explicitly not match question marks, it filters out the query parameters and it works
Edit2: We can just add an extra rewrite that strips off the query parameters, which I find a neater solution.

[AlexanderMakarov]

@willemm it works but not as transparent as it expected - service receives encoded string, i.e. if path ends with k%7C then service receives it as k%7C but expected decoded k| (as with Traefik Ingress).

It may be simplified with only

    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^ $request_uri?;
spec:

about ? at the end there is a note in http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#rewrite

(also there is a typo in your snippet reqrite -> rewrite)

[AliAMQ]

I just wanted to share my case and see if it's the same issue. I also have it in StackOverflow here.

Here's my ingress:

And here is my endpoint:
GET /api/v2/products/<<product_name_path_variable>>/aggregate

It works fine with a path variable like androidphone:
https://something1.something2.com/microservice/api/v2/products/androidphone/aggregate

But it returns 404 for a path variable like android%2Fphone or android\/phone:
https://myhost.domain.com/myservice/api/v2/products/android%2Fphone/aggregate

Would there be any workaround like playing with the configuration-snippet or something?

[longwuyuan]

@wknapik There is no engaging and update on this issue and several changes have been done to the controller, including validation and hardening of paths.

I will close this for now due to inactivity, but please feel free to reopen after posting updates and check based on the recent release of ingrss-nginx controller.

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

@wknapik There is no engaging and update on this issue and several changes have been done to the controller, including validation and hardening of paths.

I will close this for now due to inactivity, but please feel free to reopen after posting updates and check based on the recent release of ingrss-nginx controller.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wknapik]

@longwuyuan "validation and hardening of paths" doesn't sound like it would change this. I no longer use ingress-nginx, so won't test.

[afrancoc2000]

I'm still facing this issue trying to use the rabbitMQ admin interface, I'm using the managed NGINX ingress with the application routing add-on for AKS, is running version 1.9.4

This is my ingress right now:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rabbitmq-hub-ingress
  labels:
    helm.sh/chart: rabbitmq-hub-0.1.0
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx-internal
  tls:
    - hosts:
        - "mycompany.com"
      secretName: keyvault-rabbitmq-hub-ingress
  rules:
    - http:
        paths:
          - path: /rabbitmq-hub-mgt(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: rabbitmq-hub
                port:
                  number: 15672
      host: "mycompany.com"

And I'm still facing the same issue:
Routes like this one resolve: https://mycompany.com/rabbitmq-hub-mgt/#/queues
but routes like this one fail: https://mycompany.com/rabbitmq-hub-mgt/#/queues/%2F/myqueue

I tried using this workaround

    nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($request_uri ~ "^/example(/.*)") {
        proxy_pass http://upstream_balancer$1;
        break;
      }

and this one

    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^ $request_uri;
      rewrite ^([^?]*) $1;

without luck

Any chance this can be fixed?

[longwuyuan]

@afrancoc2000 enable rewrite-logs annotation and maybe you will see that your request turns into https://mycompany.com/myqueue or something like that because you are rewriting to regex-group 2 after the root /.

Regardless of that, the rewrite annotation works because I tested it and the special-characters are dealt with as per whatever the standards are. If RabbitMQ breaks because of this rule, then I would question why you are not dedicating a host like rabbitmq.mycomaony.com, with path as just / and pathType as Prefix. That is sort of reasonable for a full-blown vendor provided app like RabbitMQ which may or may not have its own rewrites and redirections.

The original issue description is not relevant as per the post of the original creator of this issue. So kindly provide all the details like ;

• Your attempt without any paths or rewrites or regex for RabbitMQ
• If above test fails then show al lthe related info as well ask RabbitMQ support if they support path based routing for RabbitMQ
  • kubectl describe output for all related resources i.e. controller and RabbitMQ
  • Kubectl logs of the controller pod
  • Make sure rewrite-logs are enabled
  • Other such info
  • Maybe a test in a Kind or a Minikube cluster dedicated to just RabbitMQ as a deployment in the cluster

[afrancoc2000]

@longwuyuan we can dedicate a subdomain like rabbitmq.mycomaony.com, but the redirection should work. it looks like when using %2F the rewrite stops working.

Right now, we need these conversions:

https://mycompany.com/rabbitmq-hub-mgt/#/queues -> /#/queues
https://mycompany.com/rabbitmq-hub-mgt/#/queues/%2F/myqueue -> /#/queues/%2F/myqueue

The first one works perfectly but the second one fails with a 404 probably the wrong rewrite.

I don't see how to enable rewrite-logs using the azure add on, but I will check, what I can tell you now:

• Without rewrite using a subdomain https://rabbitmq-hub-mgt.mycompany/#/queues/%2F/myqueue is able to resolve correctly.
• Describe output for controller pod:

Name:                 nginx-internal-0-5f654fd544-g4lc5
Namespace:            app-routing-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      nginx-internal-0
Node:                 aks-default-39893812-vmss000000/10.123.0.6
Start Time:           Fri, 07 Jun 2024 21:03:29 +0000
Labels:               app=nginx-internal-0
                      app.kubernetes.io/component=ingress-controller
                      app.kubernetes.io/managed-by=aks-app-routing-operator
                      pod-template-hash=5f654fd544
Annotations:          prometheus.io/port: 10254
                      prometheus.io/scrape: true
Status:               Running
IP:                   192.168.0.201
IPs:
  IP:           192.168.0.201
Controlled By:  ReplicaSet/nginx-internal-0-5f654fd544
Containers:
  controller:
    Container ID:  containerd://ef38318222d4cf5f856c50aa23a063e92feb9db7ac6e30656f45fcb58a7694bf
    Image:         mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller:v1.9.4
    Image ID:      mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller@sha256:eb4f9048cc04636011bcbbadeb68cee57f7981ebd29f93edb56ce0d182cbcbf6
    Ports:         8080/TCP, 8443/TCP, 10254/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --ingress-class=nginx-internal
      --controller-class=approuting.kubernetes.azure.com/nginx-internal
      --election-id=nginx-internal-0
      --publish-service=$(POD_NAMESPACE)/nginx-internal-0
      --configmap=$(POD_NAMESPACE)/nginx-internal-0
      --enable-annotation-validation=true
      --http-port=8080
      --https-port=8443
    State:          Running
      Started:      Fri, 07 Jun 2024 21:03:30 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      500m
      memory:   127Mi
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=5s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-internal-0-5f654fd544-g4lc5 (v1:metadata.name)
      POD_NAMESPACE:  app-routing-system (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mhrks (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-mhrks:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    Burstable
Node-Selectors:               <none>
Tolerations:                  CriticalAddonsOnly op=Exists
                              node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                              node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:ScheduleAnyway when max skew 1 is exceeded for selector app=nginx-internal-0
Events:
  Type    Reason  Age                   From                      Message
  ----    ------  ----                  ----                      -------
  Normal  RELOAD  2m1s (x40 over 7d1h)  nginx-ingress-controller  NGINX reload triggered due to a change in configuration

• Logs for controller pod (just a couple of lines):

10.216.36.116 - default_user_NYJMWcntuEF8lUBN_Se [11/Jun/2024:20:20:25 +0000] "GET /rabbitmq-hub-mgt/api/exchanges/%2F/amq.direct?msg_rates_age=60&msg_rates_incr=5 HTTP/1.1" 404 49 "http://mycompany/rabbitmq-hub-mgt/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0" 672 0.001 [rabbitmq-hub-rabbitmq-hub-15672] [] 192.168.1.169:15672 49 0.001 404 fe38795b7f7876b31a43f5421aa8d69e
10.216.64.186 - default_user_NYJMWcntuEF8lUBN_Se [11/Jun/2024:20:20:30 +0000] "GET /rabbitmq-hub-mgt/api/users?page=1&page_size=100&name=&use_regex=false&pagination=true HTTP/1.1" 200 234 "http://mycompany/rabbitmq-hub-mgt/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0" 603 0.002 [rabbitmq-hub-rabbitmq-hub-15672] [] 192.168.1.169:15672 234 0.002 200 c3cb9cc1b8c06d6806a1d62e84bfcbd6
10.216.64.186 - default_user_NYJMWcntuEF8lUBN_Se [11/Jun/2024:20:20:30 +0000] "GET /rabbitmq-hub-mgt/api/permissions HTTP/1.1" 200 99 "http://mycompany/rabbitmq-hub-mgt/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0" 550 0.001 [rabbitmq-hub-rabbitmq-hub-15672] [] 192.168.1.169:15672 99 0.001 200 a2ef0b5eeb669f83792743204cdfe655
10.216.36.116 - default_user_NYJMWcntuEF8lUBN_Se [11/Jun/2024:20:20:31 +0000] "GET /rabbitmq-hub-mgt/api/exchanges/%23/amq.direct?msg_rates_age=60&msg_rates_incr=5 HTTP/1.1" 404 49 "http://mycompany/rabbitmq-hub-mgt/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0" 672 0.002 [rabbitmq-hub-rabbitmq-hub-15672] [] 192.168.1.169:15672 49 0.002 404 7663feffc6fd930ccec97911a9b7ad36

So, the problem comes in the rewrite with paths that include "%2F".

I will switch to use a subdomain, but I think this issue should remain open, as the problem persists.

[longwuyuan]

• Does the subdomain work with path set as / and pathType as Prefix ? If yes, then it indicates redirection from RabbitMQ webserver and highly likely that it does not like being homed in a path based routing ingress

• If the characters %2F are the root-cause then its likely that you are unaware of the hardening of the controller path regex https://kubernetes.github.io/ingress-nginx/faq/#validation-of-path

• Rewrite-log can be enabled with annotation https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#enable-rewrite-log

• Its not easy to interpret much unless the kubectl describe for all related resources (not just controller pod) is available for analysys. That includs ingress, service, pod etc etc. And the logs of hte controller-pod contianing the rewrite-logs. And not to mention a curl request with -v as test

[afrancoc2000]

Thank you these links help, I enabled the rewrite logs and this is an example of what I received:

2024/06/17 18:53:04 [notice] 166#166: *802979 "(?i)/rabbitmq-hub-mgt($|/)(.*)" matches "/rabbitmq-hub-mgt/api/queues/myqueue", client: XX.XX.XX.XX, server: mycompany.com, request: "GET /rabbitmq-hub-mgt/api/queues/%2F/myqueue?lengths_age=60&lengths_incr=5&msg_rates_age=60&msg_rates_incr=5&data_rates_age=60&data_rates_incr=5 HTTP/2.0", host: "mycompany.com", referrer: "https://mycompany.com/rabbitmq-hub-mgt/"
2024/06/17 18:53:04 [notice] 166#166: *802979 rewritten data: "/api/queues/myqueue", args: "lengths_age=60&lengths_incr=5&msg_rates_age=60&msg_rates_incr=5&data_rates_age=60&data_rates_incr=5", client: 10.216.32.169, server: mycompany.com, request: "GET /rabbitmq-hub-mgt/api/queues/%2F/myqueue?lengths_age=60&lengths_incr=5&msg_rates_age=60&msg_rates_incr=5&data_rates_age=60&data_rates_incr=5 HTTP/2.0", host: "mycompany.com", referrer: "https://mycompany.com/rabbitmq-hub-mgt/"
10.216.32.169 - default_user_XXXXX [17/Jun/2024:18:53:04 +0000] "GET /rabbitmq-hub-mgt/api/queues/%2F/myqueue?lengths_age=60&lengths_incr=5&msg_rates_age=60&msg_rates_incr=5&data_rates_age=60&data_rates_incr=5 HTTP/2.0" 404 49 "https://mycompany/rabbitmq-hub-mgt/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0" 145 0.001 [rabbitmq-hub-rabbitmq-hub-15672] [] 192.168.1.64:15672 49 0.001 404 XXXXXXXXXXXXX

It looks like it ignores and removes the %2F. changing Prefix to ImplementationSpecific didn't work either the problem persists.

Here are the describe logs from the ingress:

Name:             rabbitmq-hub-ingress
Labels:           app.kubernetes.io/instance=rabbitmq-hub
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=rabbitmq-hub
                  app.kubernetes.io/version=1.16.0
                  helm.sh/chart=rabbitmq-hub-0.1.0
Namespace:        rabbitmq-hub
Address:          XX.XX.XX.XX
Ingress Class:    nginx-internal
Default backend:  <default>
TLS:
  keyvault-rabbitmq-hub-ingress terminates mycompany.com
Rules:
  Host                                  Path  Backends
  ----                                  ----  --------
  mycompany.com
                                        /rabbitmq-hub-mgt($|/)(.*)   rabbitmq-hub:15672 (XX.XX.XX.XX:15672)
Annotations:                            kubernetes.azure.com/tls-cert-keyvault-uri: https://mykv/certificates/tlsCertificate
                                        meta.helm.sh/release-name: rabbitmq-hub
                                        meta.helm.sh/release-namespace: rabbitmq-hub
                                        nginx.ingress.kubernetes.io/enable-rewrite-log: true
                                        nginx.ingress.kubernetes.io/rewrite-target: /$2
                                        nginx.ingress.kubernetes.io/use-regex: true
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    37m (x2 over 14h)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    37m (x2 over 14h)  nginx-ingress-controller  Scheduled for sync

This is the curl

curl "https://mycompany.com/rabbitmq-hub-mgt/api/queues/%2F/myqueue?lengths_age=60&lengths_incr=5&msg_rates_age=60&msg_rates_incr=5&data_rates_age=60&data_rates_incr=5" \
  -H "accept: */*" \
  -H "authorization: Basic XXXXXXX" \
  -v
* Host mycompany.com:443 was resolved.
* IPv6: (none)
* IPv4: XX.XX.XX.XX
*   Trying XX.XX.XX.XX...
* Connected to mycompany.com (XX.XX.XX.XX) port 443
* schannel: disabled automatic use of client certificate
* using HTTP/1.x
> GET /rabbitmq-hub-mgt/api/queues/%2F/myqueue?lengths_age=60&lengths_incr=5&msg_rates_age=60&msg_rates_incr=5&data_rates_age=60&data_rates_incr=5 HTTP/1.1
> Host: mycompany.com
> User-Agent: curl/8.8.0
> accept: */*
> authorization: Basic XXXXXXXXXX
>
* Request completely sent off
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
< HTTP/1.1 404 Not Found
< Date: Mon, 17 Jun 2024 19:36:21 GMT
< Content-Type: application/json
< Content-Length: 49
< Connection: keep-alive
< content-security-policy: script-src 'self' 'unsafe-eval' 'unsafe-inline'; object-src 'self'
< vary: accept, accept-encoding, origin
< Strict-Transport-Security: max-age=31536000; includeSubDomains
<
{"error":"Object Not Found","reason":"Not Found"}* Connection #0 to host mycompany.com left intact

I don't see the need for the service describe, hope this helps, I'm moving to the subdomain implementation then.

Thanks