Support for YARP-based reverse proxy to use as ingress

[Kralizek]

In Tye we could define ingress nodes to route requests to the different services. From this comment, it seems that the idea is to leverage YARP.

It would be cool if we could have something built-in that could let me translate

ingress:
  - name: admin
    tags:
      - admin
    bindings:
      - port: 12000
        protocol: http
        name: http
    rules:
      - service: api
        path: /api/
        preservePath: false
      - service: admin-web
        path: /
        preservePath: true
  - name: app
    tags:
      - app
    bindings:
      - port: 12100
        protocol: http
        name: http
    rules:
      - service: api
        path: /api/
        preservePath: false
      - service: app-web
        path: /
        preservePath: true

into

var database = builder.AddPostgresContainer("database")
    .WithVolumeMount("../../data/database/data", "/var/lib/postgresql/data");

var apiService = builder.AddProject<Projects.API>("api")
    .WithReference(database);

var webAdmin = builder.AddProject<Projects.Admin_Web>("admin")
    .WithReference(apiService);

var webApp = builder.AddProject<Projects.App_Web>("app")
    .WithReference(apiService);

builder.AddIngress("app")
    .WithPath("/api/", apiService, preservePath: false)
    .WithPath("/", webApp, preservePath: true)
    .WithBinding(12100, "http");

builder.AddIngress("admin")
    .WithPath("/api/", apiService, preservePath: false)
    .WithPath("/", webAdmin, preservePath: true)
    .WithBinding(12000, "http");

builder.Build().Run();

[davidfowl]

I don’t think we want to build ingress into the app model. Given that we have YARP now, just make it one of your projects and configure it there with the service discovery extensibility. Take a look at eshop lite for an example.

https://github.com/dotnet/aspire/tree/main/samples/eShopLite/ApiGateway

[a-shoemaker]

I'm not exactly sure how everything works in Aspire yet, but seems like it would fit nicelly into the AppHost project as, for how we use it at least, it's a local dev thing. Only the configuration matters for taking to deployment. If not built in, guessing you can add your own stuff to the AppHost project perhaps?

(I feel how Dapr works or will work perhaps is relevant.)

[davidfowl]

Where are you deploying to? What extracts this configuration and does something in deployment? The advantage of making it a project with YARP is that it works like any of your other projects and is truly portable and doesn't need to be translated into some other form since it is your proxy project.

You can build an ingress resource that uses YARP under the covers in the apphost project itself (it can host another web server) but then you also need to support that mapping from ingress rules into whatever makes sense for your deployment.

I'll attempt to build a sample that you can use, but we won't be building this into the apphost, at least as of right now.

[a-shoemaker]

We have our own specification language (YAML configuration) and we dynamically create the tye.yaml for local dev or terraform / helm (values.yaml) for AKS deployment (we don't use Tye for deployments). So, it goes to K8 Ingress, we don't need/want our own C# project for an ingress component (it wouldn't be used in deployements). We utilize dapr with communication through the side cars. The ingress gives us the nice single static port to use with the path that matches the deployment and everything else uses "service discovery" (dynamic ports in local dev).

I would for sure vote this item up. To me it makes total sense to be integrated into Aspire, the manifest/tooling/dashboard, etc. This is potentially something that will keep us from using Aspire anytime soon. But this could also be due to me not fully understanding Aspire yet...

[Kralizek]

I understand what you suggest and I understand the complexity of the mapping/rewrite rules.

But now I need to pollute my src/ directory with a project and code that is only to facilitate local development.

Probably the example I posted isn't the best one because you dont really need the ingress in this case.

The other case I use the ingress is in a react+aspnet core app.

We deploy the react bits on a Azure Static Site and let access the backend with relative paths instead of having to sissle around with hostnames to be passed in the react build process.

This deployment solution really works well for us and being able to easily replicate this behavior locally was a huge thumb up for Tye.

On the other hand, being forced to roll out my own mapping project only for development purpose feels a bit of a let down for a "development inner cycle" tool.

[davidfowl]

So, it goes to K8 Ingress, we don't need/want our own C# project for an ingress component (it wouldn't be used in deployements)

This is the design of the YARP ingress. You make a project and you deploy it to K8s.

The app model is fully extensible C# you can build this yourself, in fact this is a feature. I'd urge you to attempt to build this if it's something you need and give us some feedback about how extensible the app model is for these situations. We're not going to build generic ingress mapping to various proxies. YARP based ingress for development only is a sweet spot.

That said, I look forward to customers extending the platform for their needs.

[Kralizek]

Hi @davidfowl
I'm trying to giving it a shot here: https://github.com/Kralizek/AspireIngress (see the AspireIngress.Resources project)

I could use some pointers on how to hook into the resource creation after it has been defined. I went through the repo looking at how projects and containers are created but I think I miss something.

Also, I can't find the source generator that takes care of creating the service metadata classes for the referenced projects.

[davidfowl]

I have a spike of this, I'll be able to share tomorrow hopefully.

[Kralizek]

Hi @davidfowl

Is there any documentation on how to communicate with the DCP (which I guess is the component that does all the magic here)?

I'm really stuck at understanding how the flow to get Aspire to create a custom resource is.

Thanks!

[davidfowl]

This wouldn't be a DCP resource. DCP is the kubernetes API and it supports containers and executables (https://github.com/dotnet/aspire/tree/main/src/Aspire.Hosting/Dcp/Model). The ingress resource would either need to run directly in the app host or be another process that runs as part of DCP orchestration. That's why it's so attractive making it a project.

[Kralizek]

That's why it's so attractive making it a project.

My idea was to create a "hidden" project. With just the YARP setup extrapolated from the annotations.

[davidfowl]

@Kralizek where is this hidden project?

[Kralizek]

Well, hooking into the pipeline of creation of resources/project is what I'm trying to understand :)

That's why I was looking for some pointers on what to look at.

[davidfowl]

After spending some time with this, I think it would be better to avoid running the proxy code in the hosting process and instead treat it like project or executable installed on the machine. So you can use something like nginx/Yarp/caddy/ etc etc as your proxy (but you need something configurable) and then you need to turn the ingress config into the native config of the proxy being used. YARP requires another project. Nginx works but you have to generate configuration files at runtime (not the end of the world I guess).

The benefit of using a project for YARP is that it's portable and you can deploy it to the environment.

The other part of this is describing the manifest resource for an ingress. That would preserve pretty much what your tye.yaml looks like but specifically for deployment. The harder part is the fact that ingress may not translate to anything outside of kubernetes, so azd up wouldn't understand this resource in some cases. If it was a container that was deployed with configuration, then it would work in those scenarios.

[rjygraham]

All, I put together a very crude implementation of using YARP with Aspire. This allows you to easily wire-up API project backends for YARP using Aspire semantics within Program.cs of the AppHost project.

My primary motivation in doing this is to be able to wire-up DevTunnels to all my microservice APIs for a mobile app to consume while in development.

Code can be found here:
https://github.com/rjygraham/AspireYarpTest

[mitchdenny]

Sooo there probably isn't anything for us to do here right? Or are we proposing introducing Ngrinx/Caddy resource types with extension methods to drive configuration? (noting that we don't have a YARP container we can just pick up and configure with EnvVars either.

[JohnGalt1717]

So I've looked at the eshop example and it's linking to source projectsd that look like they'll be nugets. When will these be published?

And It would be really nice to not get the configuration from settings, but to load them from service discovery. I.e. we have most of what we need save the subdirectory for wiring all of the endpoints to Yarp for a single endpoint in the host.

Does it make sense to have the host generate the yarp endpoint and have it just work or do something similar? It can of course be opt in and should have a project that just takes the WithReferences (change to WithServices that takes the sub path) and does the rest of the work for you.

If the dashboard is split out of host, then host itself could become the yarp endpoint. The only part that would have to be defined is certificate management and then in K8s, how to make the host the ingress so you don't have to use nginx or anything else, just yarp.

[davidfowl]

I took a stab at this with the latest capabilities of the app host and was able to build something like this https://github.com/davidfowl/AspireYarp.

This is only usable in development, it doesn't attempt to define a generic ingress resource. This is a YARP specific resource with YARP specific configuration.

PS: This is using the latest preview3 builds (daily builds).

Also, custom resources don't show up in the dashboard as yet (#436)

[Kralizek]

Beautiful!

[Kralizek]

Resurrecting this thread as I'm trying to use the work of the repo above on Preview 5.

The only change I needed to get to compile is this line as follows:

// var context = new EnvironmentCallbackContext(options.Value.Publisher!);

var executionContext = new DistributedApplicationExecutionContext(DistributedApplicationOperation.Run);
var context = new EnvironmentCallbackContext(executionContext);

This is how I use it in my AppHost project

builder.AddYarp("login")
    .Route("auth", auth, path: "/");

Now the dashboard shows the resource but doesn't add the link to the resource and doesn't report its running status.

When I navigate to the address from the logs, I get the following error message

info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 8.0.0-preview.5.24201.12+1b627b7d5d399d4f9118366e7611e11e56de4554
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: C:\Development\TD\IdentityManager\App\tools\AppHost
info: Aspire.Hosting.DistributedApplication[0]
      Now listening on: https://localhost:17079
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://127.0.0.1:50751
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Development\TD\IdentityManager\App\tools\AppHost
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application started. Press Ctrl+C to shut down.
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[9]
      Proxying to http://aspire.hosting.applicationmodel.endpointreference:0/ HTTP/2 RequestVersionOrLower 
warn: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      Request: An error was encountered before receiving a response.
      System.Net.Http.HttpRequestException: No such host is known. (aspire.hosting.applicationmodel.endpointreference:0)
       ---> System.Net.Sockets.SocketException (11001): No such host is known.
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)

Apparently, it tries to proxy the request to the address http://aspire.hosting.applicationmodel.endpointreference:0/ but I have no idea if it's correct for the Yarp-based service discovery system.

I guess it originates from this line:

[target.Resource.Name] = new() { Address = $"http://{target.Resource.Name}" }

Finally, the request path transformation isn't working as it throws an exception when starting the AppHost.

fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.InvalidOperationException: Unable to load or apply the proxy configuration.
       ---> System.AggregateException: The proxy config is invalid. (Unknown transform: )
       ---> System.ArgumentException: Unknown transform: 
         --- End of inner exception stack trace ---
         at Yarp.ReverseProxy.Management.ProxyConfigManager.ApplyConfigAsync(IReadOnlyList`1 routes, IReadOnlyList`1 clusters)
         at Yarp.ReverseProxy.Management.ProxyConfigManager.InitialLoadAsync()

[JohnGalt1717]

Everything was working with B5 but it is now broken.

builder.Services.AddReverseProxy().AddServiceDiscoveryDestinationResolver();

No longer resolves.

[davidfowl]

I'm updating the repo now!

[Kralizek]

Do you think the repo could be promoted to a maintained nuget package?

[JohnGalt1717]

Is the repro updated?

[davidfowl]

Yes it’s updated to preview 5

[Kralizek]

It's still giving me issues with the request transformation.

[davidfowl]

File an issue on the repo with instructions on how to reproduce the problem.

[JohnGalt1717]

I'm confused with how this is working now.

I currently have a Yarp project that uses config file routing setup, and those use ServiceDiscovery. That worked until P5 and now doesn't work because it's not resolving service discovery.

It looks like AddYarp is some sort of extension method now in the host that generates the yarp based on the services instead of having a separate project.

Is there any way to get the service discovery working like it used to with yarp?

The release notes tell you that ServiceDiscovery changed and then that takes you to a github issue that is frankly unintelligable as to what actually changed instead of just telling you what changed and how to fix it.

If not, ok, with putting it in the host, but it really needs to be promoted as requested as a full project for aspire and the aspir8 needs to be updated to translate that into deploy of the yarp ingress etc.

[Kralizek]

The release notes tell you that ServiceDiscovery changed and then that takes you to a github issue that is frankly unintelligable as to what actually changed instead of just telling you what changed and how to fix it.

I had the same feeling.

[davidfowl]

I currently have a Yarp project that uses config file routing setup, and those use ServiceDiscovery. That worked until P5 and now doesn't work because it's not resolving service discovery.

Is there any way to get the service discovery working like it used to with yarp?

I'll try to reproduce this normally.

It looks like AddYarp is some sort of extension method now in the host that generates the yarp based on the services instead of having a separate project.

My project is puts YARP in the apphost as a development time only dependency. They are different approaches.

[JohnGalt1717]

I got 1 endpoint to work and I don't know why. The rest don't.

Update, it was caching. It doesn't work either. The Service discovery isn't working (and because of the other issue with the token, I can't see the logs.)

[davidfowl]

OK I built a sample and it seems like its working. I created a new preview 6 project and added yarp and was able to forward a simple request to the page. The YARP project had this:

{
"ReverseProxy": {
  "Routes": {
    "fe": {
      "ClusterId": "fe",
      "Match": {
        "Path": "/fe/{**catch-all}"
      },
      "Transforms": [
        { "PathRemovePrefix": "/fe" }
      ]
    }
  },
  "Clusters": {
    "fe": {
      "Destinations": {
        "fe": {
          "Address": "https://webfrontend",
          "Host": "localhost"
        }
      }
    }
  }
}
}

Apphost:

var builder = DistributedApplication.CreateBuilder(args);

var cache = builder.AddRedis("cache");

var apiService = builder.AddProject<Projects.AspireApp47_ApiService>("apiservice");

var fe = builder.AddProject<Projects.AspireApp47_Web>("webfrontend")
    .WithExternalHttpEndpoints()
    .WithReference(cache)
    .WithReference(apiService);

builder.AddProject<Projects.WebApplication1>("gateway")
    .WithReference(apiService)
    .WithReference(fe);

builder.Build().Run();

YARP project:

var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

builder.Services.AddReverseProxy()
                .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"))
                .AddServiceDiscoveryDestinationResolver();

var app = builder.Build();

app.MapReverseProxy();

app.MapGet("/", () => "Hello World!");

app.Run();

I can confirm that YARP works with preview 6. Now we can focus on narrowing to what you might be running into in your existing project.

[JohnGalt1717]

Why is host required? And how would that change for k8s when we deploy?

It wasn't required before hence why I'm asking.

As for AddYarp, how would one bind that to https (it errors if I try and make it an https port) with the same?

We put our help endpoint on the yarp proxy (just swagger.js stuff) and then have it have endpoints on all of the microservices configured with their unique swagger.json in the drop down. Would the best approach if using AddYarp be to have another microservice for /help that just serves that instead of the proxy or is there a better way?

[davidfowl]

Why is host required? And how would that change for k8s when we deploy?

There's a bug with https where we set the original host name. The cert mismatch happens when running locally because the ASP.NET dev cert. I don't think this would be required if you use http nor would it matter if you're on k8s.

As for AddYarp, how would one bind that to https (it errors if I try and make it an https port) with the same?

I would have to write more code 😄

[davidfowl]

Would the best approach if using AddYarp be to have another microservice for /help that just serves that instead of the proxy or is there a better way?

AddYarp is only useful if you using it for development only. Is that what you're doing?

[JohnGalt1717]

Would the best approach if using AddYarp be to have another microservice for /help that just serves that instead of the proxy or is there a better way?

AddYarp is only useful if you using it for development only. Is that what you're doing?

My hope would be for Aspire to define the ingress using AdYarp which would dump to the exported data, which could then be used by aspir8 and others to spin up the Yarp ingress with the settings I've defined.

Sort of like it would be nice to be able to define secrets and certificates to use as well and those would dump and then aspir8 would map those to secrets and certificates in k8s or if you were deploying to Azure as an example those would map to your KeyVault equivalents.

[davidfowl]

My hope would be for Aspire to define the ingress using AdYarp which would dump to the exported data, which could then be used by aspir8 and others to spin up the Yarp ingress with the settings I've defined.

That doesn't exist, but that's what this issue is about. My project does not do this. Ingress is generic and doesn't match to YARP's configuration. It needs to be its own resource type that YARP can implement. This isn't the approach that the YARP resource takes.

Sort of like it would be nice to be able to define secrets and certificates to use as well and those would dump and then aspir8 would map those to secrets and certificates in k8s or if you were deploying to Azure as an example those would map to your KeyVault equivalents.

I like it but we're not close to having any end to end out of the box for this. Hopefully, it will be possible to build it from the primitives we offer.

[samsp-msft]

@davidfowl @mitchdenny - We should revisit the idea of being able to configure routes via the app-model and have the app-model spit out configuration for YARP as a result.
The deployment step would be interesting as it could convert that to a hosted YARP and remap the endpoints based on the deployment target and how service discovery happens in those systems.

microsoft/reverse-proxy#2525 is to drive having a docker image for YARP.

[davidfowl]

That’s the killer scenario @samsp-msft!

[Kralizek]

That would be really nice. I currently use the Yarp support from Aspirant and it works really nicely.

But I have few routes that are hidden in the configuration file.

  "ReverseProxy": {
    "Routes": {
      "backend": {
        "ClusterId": "backend",
        "Order": 1,
        "Match": {
          "Path": "/api/v1/{**remainder}"
        },
        "Transforms": [
          { "PathRemovePrefix": "/api/v1" }
        ]
      },
      "legacy-api": {
        "ClusterId": "legacy",
        "Order": 100,
        "Match": {
          "Path": "/api/{**remainder}"
        }
      },
      "legacy-services": {
        "ClusterId": "legacy",
        "Order": 101,
        "Match": {
          "Path": "/services/{**remainder}"
        }
      },
      "swagger": {
        "ClusterId": "backend",
        "Order": 900,
        "Match": {
          "Path": "/swagger/{**remainder}"
        }
      },
      "openapi": {
        "ClusterId": "backend",
        "Order": 901,
        "Match": {
          "Path": "/schema/{**remainder}"
        }
      },
      "frontend": {
        "ClusterId": "frontend",
        "Order": 1000,
        "Match": {
          "Path": "{**catchall}"
        }
      }
    },
    "Clusters": {
      "backend": {
        "Destinations": {
          "backend": {
            "Address": "http://backend"
          }
        }
      },
      "frontend": {
        "Destinations": {
          "frontend": {
            "Address": "http://frontend"
          }
        }
      },
      "legacy": {
        "Destinations": {
          "frontend": {
            "Address": "http://legacy"
          }
        }
      }
    }
  }

From one side, I see the value of not cluttering the AppHost Program file, but I also see the value of keeping these rules in the forefront.

[JohnGalt1717]

The trick here, and maybe this is something Aspire should be considering, is that you often will have different mappings.

As an example, in dev we have our accounts section go to the proxy/accounts, but in production we have it go to accounts.xyz.com

The reason for this is localhost.... which of course has other issues care of Google Chrome etc.

Ideally we'd be able to use accounts.local.dev or something that would work properly, but again, this is still environment specific. I.e. QA would be accounts.qa.xyz.com and Staging accounts.staging.xyz.com and Production accounts.xyz.com

The configuration file of course handles this nicely. In the case of relative path replace, this is easy because it doesn't care about the host, but there is this use case that pops up all of the time, especially if you're using OpenIDConnect that needs to be considered.

[davidfowl]

@JohnGalt1717 I don't want to couple these problems with the initial resource. We need to build something that works in local dev and can be deployed for basic scenarios. When we get it working then we need to talk about making local host name routing work and making certs more seamless for local dev, and how to make k8s ingress for YARP work when you deploy in that environment.

We need to sequence features, so we don't boil the ocean.

[JohnGalt1717]

100%. Just want to make sure that we don't forget that often, these routes are environment specific.

[plppp2001]

I don’t think we want to build ingress into the app model. Given that we have YARP now, just make it one of your projects and configure it there with the service discovery extensibility. Take a look at eshop lite for an example.

https://github.com/dotnet/aspire/tree/main/samples/eShopLite/ApiGateway

This link doesn't work....

[davidfowl]

https://github.com/dotnet/aspire/tree/13bcaa03819f5e342d96bdffed92a50c7175198d/playground/TestShop/ApiGateway

[samsp-msft]

Related is #6640 to have WithReplica load balancing for ingress.