-
Notifications
You must be signed in to change notification settings - Fork 52
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #26 from CommunityToolkit/ollama
Ollama
- Loading branch information
Showing
40 changed files
with
1,248 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # CommunityToolkit.Aspire.Hosting.Ollama | ||
|
|
||
| [](https://nuget.org/packages/CommunityToolkit.Aspire.Ollama/) | [>)](https://nuget.org/packages/CommunityToolkit.Aspire.Ollama/absoluteLatest) | ||
|
|
||
| ## Overview | ||
|
|
||
| An Aspire component leveraging the [Ollama](https://ollama.com) container with support for downloading a model on startup. | ||
|
|
||
| ## Usage | ||
|
|
||
| Use the static `AddOllama` method to add this container component to the application builder. | ||
|
|
||
| ```csharp | ||
| // The distributed application builder is created here | ||
| var ollama = builder.AddOllama("ollama"); | ||
|
|
||
| // The builder is used to build and run the app somewhere down here | ||
| ``` | ||
|
|
||
| ### Configuration | ||
|
|
||
| The AddOllama method has optional arguments to set the `name`, `port` and `modelName`. | ||
| The `name` is what gets displayed in the Aspire orchestration app against this component. | ||
| The `port` is provided randomly by Aspire. If for whatever reason you need a fixed port, you can set that here. | ||
| The `modelName` specifies what LLM to pull when it starts up. The default is `llama3`. You can also set this to null to prevent any models being pulled on startup - leaving you with a plain Ollama container to work with. | ||
|
|
||
| ## Downloading the LLM | ||
|
|
||
| When the Ollama container for this component first spins up, this component will download the LLM (llama3 unless otherwise specified). | ||
| The progress of this download will be displayed in the State column for this component on the Aspire orchestration app. | ||
| Important: Keep the Aspire orchestration app open until the download is complete, otherwise the download will be cancelled. | ||
| In the spirit of productivity, we recommend kicking off this process before heading for lunch. | ||
| This component binds a volume called "ollama" so that once the model is fully downloaded, it'll be available for subsequent runs. | ||
|
|
||
| ## Accessing the Ollama server from other Aspire components | ||
|
|
||
| You can pass the ollama component to other Aspire components in the usual way: | ||
|
|
||
| ```csharp | ||
| builder.AddMyComponent().WithReference(ollama); | ||
| ``` | ||
|
|
||
| Within that component (e.g. a web app), you can fetch the Ollama connection string from the application builder as follows. | ||
| Note that if you changed the name of the Ollama component via the `name` argument, then you'll need to use that here when specifying which connection string to get. | ||
|
|
||
| ```csharp | ||
| var connectionString = builder.Configuration.GetConnectionString("Ollama"); | ||
| ``` | ||
|
|
||
| You can then call any of the Ollama endpoints through this connection string. We recommend using the [OllamaSharp](https://www.nuget.org/packages/OllamaSharp) client to do this. |
21 changes: 21 additions & 0 deletions
21
...olkit.Aspire.Hosting.Ollama.AppHost/CommunityToolkit.Aspire.Hosting.Ollama.AppHost.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <OutputType>Exe</OutputType> | ||
| <TargetFramework>net8.0</TargetFramework> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <IsAspireHost>true</IsAspireHost> | ||
| <UserSecretsId>f39fb70f-21f3-4af9-89b4-3062ff4431e6</UserSecretsId> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\CommunityToolkit.Aspire.Hosting.Ollama.Web\CommunityToolkit.Aspire.Hosting.Ollama.Web.csproj" /> | ||
| <ProjectReference Include="..\..\..\src\CommunityToolkit.Aspire.Hosting.Ollama\CommunityToolkit.Aspire.Hosting.Ollama.csproj" IsAspireProjectResource="false" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Aspire.Hosting.AppHost" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
10 changes: 10 additions & 0 deletions
10
examples/ollama/CommunityToolkit.Aspire.Hosting.Ollama.AppHost/Program.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| var builder = DistributedApplication.CreateBuilder(args); | ||
|
|
||
| var ollama = builder.AddOllama("ollama", modelName: "phi3"); | ||
|
|
||
| builder.AddProject<Projects.CommunityToolkit_Aspire_Hosting_Ollama_Web>("webfrontend") | ||
| .WithExternalHttpEndpoints() | ||
| .WithReference(ollama) | ||
| .WithEnvironment("ollama:model", "phi3"); | ||
|
|
||
| builder.Build().Run(); |
29 changes: 29 additions & 0 deletions
29
...ples/ollama/CommunityToolkit.Aspire.Hosting.Ollama.AppHost/Properties/launchSettings.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| { | ||
| "$schema": "https://json.schemastore.org/launchsettings.json", | ||
| "profiles": { | ||
| "https": { | ||
| "commandName": "Project", | ||
| "dotnetRunMessages": true, | ||
| "launchBrowser": true, | ||
| "applicationUrl": "https://localhost:17117;http://localhost:15077", | ||
| "environmentVariables": { | ||
| "ASPNETCORE_ENVIRONMENT": "Development", | ||
| "DOTNET_ENVIRONMENT": "Development", | ||
| "DOTNET_DASHBOARD_OTLP_ENDPOINT_URL": "https://localhost:21075", | ||
| "DOTNET_RESOURCE_SERVICE_ENDPOINT_URL": "https://localhost:22200" | ||
| } | ||
| }, | ||
| "http": { | ||
| "commandName": "Project", | ||
| "dotnetRunMessages": true, | ||
| "launchBrowser": true, | ||
| "applicationUrl": "http://localhost:15077", | ||
| "environmentVariables": { | ||
| "ASPNETCORE_ENVIRONMENT": "Development", | ||
| "DOTNET_ENVIRONMENT": "Development", | ||
| "DOTNET_DASHBOARD_OTLP_ENDPOINT_URL": "http://localhost:19024", | ||
| "DOTNET_RESOURCE_SERVICE_ENDPOINT_URL": "http://localhost:20271" | ||
| } | ||
| } | ||
| } | ||
| } |
9 changes: 9 additions & 0 deletions
9
examples/ollama/CommunityToolkit.Aspire.Hosting.Ollama.AppHost/appsettings.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "Logging": { | ||
| "LogLevel": { | ||
| "Default": "Information", | ||
| "Microsoft.AspNetCore": "Warning", | ||
| "Aspire.Hosting.Dcp": "Warning" | ||
| } | ||
| } | ||
| } |
22 changes: 22 additions & 0 deletions
22
...ting.Ollama.ServiceDefaults/CommunityToolkit.Aspire.Hosting.Ollama.ServiceDefaults.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <TargetFramework>net8.0</TargetFramework> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <IsAspireSharedProject>true</IsAspireSharedProject> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <FrameworkReference Include="Microsoft.AspNetCore.App" /> | ||
|
|
||
| <PackageReference Include="Microsoft.Extensions.Http.Resilience" /> | ||
| <PackageReference Include="Microsoft.Extensions.ServiceDiscovery" /> | ||
| <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" /> | ||
| <PackageReference Include="OpenTelemetry.Extensions.Hosting" /> | ||
| <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" /> | ||
| <PackageReference Include="OpenTelemetry.Instrumentation.Http" /> | ||
| <PackageReference Include="OpenTelemetry.Instrumentation.Runtime" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
111 changes: 111 additions & 0 deletions
111
examples/ollama/CommunityToolkit.Aspire.Hosting.Ollama.ServiceDefaults/Extensions.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| using Microsoft.AspNetCore.Builder; | ||
| using Microsoft.AspNetCore.Diagnostics.HealthChecks; | ||
| using Microsoft.Extensions.DependencyInjection; | ||
| using Microsoft.Extensions.Diagnostics.HealthChecks; | ||
| using Microsoft.Extensions.Logging; | ||
| using OpenTelemetry; | ||
| using OpenTelemetry.Metrics; | ||
| using OpenTelemetry.Trace; | ||
|
|
||
| namespace Microsoft.Extensions.Hosting; | ||
|
|
||
| // Adds common .NET Aspire services: service discovery, resilience, health checks, and OpenTelemetry. | ||
| // This project should be referenced by each service project in your solution. | ||
| // To learn more about using this project, see https://aka.ms/dotnet/aspire/service-defaults | ||
| public static class Extensions | ||
| { | ||
| public static IHostApplicationBuilder AddServiceDefaults(this IHostApplicationBuilder builder) | ||
| { | ||
| builder.ConfigureOpenTelemetry(); | ||
|
|
||
| builder.AddDefaultHealthChecks(); | ||
|
|
||
| builder.Services.AddServiceDiscovery(); | ||
|
|
||
| builder.Services.ConfigureHttpClientDefaults(http => | ||
| { | ||
| // Turn on resilience by default | ||
| http.AddStandardResilienceHandler(); | ||
|
|
||
| // Turn on service discovery by default | ||
| http.AddServiceDiscovery(); | ||
| }); | ||
|
|
||
| return builder; | ||
| } | ||
|
|
||
| public static IHostApplicationBuilder ConfigureOpenTelemetry(this IHostApplicationBuilder builder) | ||
| { | ||
| builder.Logging.AddOpenTelemetry(logging => | ||
| { | ||
| logging.IncludeFormattedMessage = true; | ||
| logging.IncludeScopes = true; | ||
| }); | ||
|
|
||
| builder.Services.AddOpenTelemetry() | ||
| .WithMetrics(metrics => | ||
| { | ||
| metrics.AddAspNetCoreInstrumentation() | ||
| .AddHttpClientInstrumentation() | ||
| .AddRuntimeInstrumentation(); | ||
| }) | ||
| .WithTracing(tracing => | ||
| { | ||
| tracing.AddAspNetCoreInstrumentation() | ||
| // Uncomment the following line to enable gRPC instrumentation (requires the OpenTelemetry.Instrumentation.GrpcNetClient package) | ||
| //.AddGrpcClientInstrumentation() | ||
| .AddHttpClientInstrumentation(); | ||
| }); | ||
|
|
||
| builder.AddOpenTelemetryExporters(); | ||
|
|
||
| return builder; | ||
| } | ||
|
|
||
| private static IHostApplicationBuilder AddOpenTelemetryExporters(this IHostApplicationBuilder builder) | ||
| { | ||
| var useOtlpExporter = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]); | ||
|
|
||
| if (useOtlpExporter) | ||
| { | ||
| builder.Services.AddOpenTelemetry().UseOtlpExporter(); | ||
| } | ||
|
|
||
| // Uncomment the following lines to enable the Azure Monitor exporter (requires the Azure.Monitor.OpenTelemetry.AspNetCore package) | ||
| //if (!string.IsNullOrEmpty(builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"])) | ||
| //{ | ||
| // builder.Services.AddOpenTelemetry() | ||
| // .UseAzureMonitor(); | ||
| //} | ||
|
|
||
| return builder; | ||
| } | ||
|
|
||
| public static IHostApplicationBuilder AddDefaultHealthChecks(this IHostApplicationBuilder builder) | ||
| { | ||
| builder.Services.AddHealthChecks() | ||
| // Add a default liveness check to ensure app is responsive | ||
| .AddCheck("self", () => HealthCheckResult.Healthy(), ["live"]); | ||
|
|
||
| return builder; | ||
| } | ||
|
|
||
| public static WebApplication MapDefaultEndpoints(this WebApplication app) | ||
| { | ||
| // Adding health checks endpoints to applications in non-development environments has security implications. | ||
| // See https://aka.ms/dotnet/aspire/healthchecks for details before enabling these endpoints in non-development environments. | ||
| if (app.Environment.IsDevelopment()) | ||
| { | ||
| // All health checks must pass for app to be considered ready to accept traffic after starting | ||
| app.MapHealthChecks("/health"); | ||
|
|
||
| // Only health checks tagged with the "live" tag must pass for app to be considered alive | ||
| app.MapHealthChecks("/alive", new HealthCheckOptions | ||
| { | ||
| Predicate = r => r.Tags.Contains("live") | ||
| }); | ||
| } | ||
|
|
||
| return app; | ||
| } | ||
| } |
17 changes: 17 additions & 0 deletions
17
...munityToolkit.Aspire.Hosting.Ollama.Web/CommunityToolkit.Aspire.Hosting.Ollama.Web.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk.Web"> | ||
|
|
||
| <PropertyGroup> | ||
| <TargetFramework>net8.0</TargetFramework> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\CommunityToolkit.Aspire.Hosting.Ollama.ServiceDefaults\CommunityToolkit.Aspire.Hosting.Ollama.ServiceDefaults.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="OllamaSharp" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
Oops, something went wrong.