Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
.Net: SK Telemetry improvements (#1905)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> This PR contains changes to improve observability for Semantic Kernel users, including: - Logging: using already existing `ILogger` interface with some improvements described below. - Metering: implemented using `Meter` and `MeterListener` classes from `System.Diagnostics.Metrics` namespace. - Tracing: implemented using `Activity`, `ActivitySource` and `ActivityListener` classes from `System.Diagnostics` namespace. Added telemetry uses native .NET methods, which means that it's not dependent on specific telemetry tool. Current PR contains console application using Application Insights as example to show telemetry capabilities in Kernel. Changes include instrumentation for `SequentialPlanner` as a starting point and later there will be new PRs for other types of planners and more improvements for kernel telemetry. Also, there are updates for `LogLevel` handling across codebase to align with described log level purpose: https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.loglevel?view=dotnet-plat-ext-7.0 Particularly, there are changes for `LogLevel.Trace` to be used for sensitive data and most detailed messages, which should not be enabled in prodution environments. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Changes: - Marked `.WithLogger` method in KernelBuilder as obsolete and added new method `.WithLogging`. - Added new methods `.WithMetering` and `.WithTracing` in KernelBuilder. - Improved `LogLevel` handling across codebase to cover cases of logging sensitive data. - Implemented `InstrumentedSequentialPlanner` for instrumentation using decorator pattern. - Extended `InvokeAsync` method in `Plan` for instrumentation. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows SK Contribution Guidelines (https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) - [x] The code follows the .NET coding conventions (https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions) verified with `dotnet format` - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Shawn Callegari <[email protected]>
- Loading branch information
1 parent
f622bfe
commit 55dc794
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <TargetFramework>net6.0</TargetFramework> | ||
| <RollForward>LatestMajor</RollForward> | ||
| <OutputType>Exe</OutputType> | ||
| <LangVersion>10</LangVersion> | ||
| <Nullable>enable</Nullable> | ||
| <ImplicitUsings>disable</ImplicitUsings> | ||
| <IsPackable>false</IsPackable> | ||
| <!-- Suppress: "Declare types in namespaces", "Require ConfigureAwait" --> | ||
| <NoWarn>CA1050;CA1707;CA2007;VSTHRD111</NoWarn> | ||
| <UserSecretsId>5ee045b0-aea3-4f08-8d31-32d1a6f8fed0</UserSecretsId> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="Microsoft.ApplicationInsights.WorkerService" /> | ||
| <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" /> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\src\Extensions\Planning.SequentialPlanner\Planning.SequentialPlanner.csproj" /> | ||
| <ProjectReference Include="..\..\src\Connectors\Connectors.AI.OpenAI\Connectors.AI.OpenAI.csproj" /> | ||
| <ProjectReference Include="..\..\src\SemanticKernel\SemanticKernel.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| // Copyright (c) Microsoft. All rights reserved. | ||
|
|
||
| using System; | ||
| using System.Collections.Concurrent; | ||
| using System.Diagnostics; | ||
| using System.Diagnostics.Metrics; | ||
| using System.Threading.Tasks; | ||
| using Microsoft.ApplicationInsights; | ||
| using Microsoft.ApplicationInsights.DataContracts; | ||
| using Microsoft.ApplicationInsights.Extensibility; | ||
| using Microsoft.Extensions.DependencyInjection; | ||
| using Microsoft.Extensions.Logging; | ||
| using Microsoft.Extensions.Logging.ApplicationInsights; | ||
| using Microsoft.SemanticKernel; | ||
| using Microsoft.SemanticKernel.Planning; | ||
| using Microsoft.SemanticKernel.Planning.Sequential; | ||
|
|
||
| /// <summary> | ||
| /// Example of telemetry in Semantic Kernel using Application Insights within console application. | ||
| /// </summary> | ||
| public sealed class Program | ||
| { | ||
| /// <summary> | ||
| /// Log level to be used by <see cref="ILogger"/>. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// <see cref="LogLevel.Information"/> is set by default. <para /> | ||
| /// <see cref="LogLevel.Trace"/> will enable logging with more detailed information, including sensitive data. Should not be used in production. <para /> | ||
| /// </remarks> | ||
| private static LogLevel LogLevel = LogLevel.Information; | ||
|
|
||
| public static async Task Main() | ||
| { | ||
| var serviceProvider = GetServiceProvider(); | ||
|
|
||
| var telemetryClient = serviceProvider.GetRequiredService<TelemetryClient>(); | ||
| var logger = serviceProvider.GetRequiredService<ILogger<Program>>(); | ||
|
|
||
| using var meterListener = new MeterListener(); | ||
| using var activityListener = new ActivityListener(); | ||
|
|
||
| ConfigureMetering(meterListener, telemetryClient); | ||
| ConfigureTracing(activityListener, telemetryClient); | ||
|
|
||
| var kernel = GetKernel(logger); | ||
| var planner = GetPlanner(kernel, logger); | ||
|
|
||
| try | ||
| { | ||
| using var operation = telemetryClient.StartOperation<DependencyTelemetry>("ApplicationInsights.Example"); | ||
|
|
||
| Console.WriteLine("Operation/Trace ID:"); | ||
| Console.WriteLine(Activity.Current?.TraceId); | ||
|
|
||
| var plan = await planner.CreatePlanAsync("Write a poem about John Doe, then translate it into Italian."); | ||
|
|
||
| Console.WriteLine("Original plan:"); | ||
| Console.WriteLine(plan.ToPlanString()); | ||
|
|
||
| var result = await kernel.RunAsync(plan); | ||
|
|
||
| Console.WriteLine("Result:"); | ||
| Console.WriteLine(result.Result); | ||
| } | ||
| finally | ||
| { | ||
| // Explicitly call Flush() followed by sleep is required in console apps. | ||
| // This is to ensure that even if application terminates, telemetry is sent to the back-end. | ||
| telemetryClient.Flush(); | ||
| await Task.Delay(5000); | ||
| } | ||
| } | ||
|
|
||
| private static ServiceProvider GetServiceProvider() | ||
| { | ||
| var services = new ServiceCollection(); | ||
|
|
||
| ConfigureApplicationInsightsTelemetry(services); | ||
|
|
||
| return services.BuildServiceProvider(); | ||
| } | ||
|
|
||
| private static void ConfigureApplicationInsightsTelemetry(ServiceCollection services) | ||
| { | ||
| string instrumentationKey = Env.Var("ApplicationInsights__InstrumentationKey"); | ||
|
|
||
| services.AddLogging(loggingBuilder => | ||
| { | ||
| loggingBuilder.AddFilter<ApplicationInsightsLoggerProvider>(typeof(Program).FullName, LogLevel); | ||
| loggingBuilder.SetMinimumLevel(LogLevel); | ||
| }); | ||
|
|
||
| services.AddApplicationInsightsTelemetryWorkerService(options => | ||
| { | ||
| options.ConnectionString = $"InstrumentationKey={instrumentationKey}"; | ||
| }); | ||
| } | ||
|
|
||
| private static IKernel GetKernel(ILogger logger) | ||
| { | ||
| string folder = RepoFiles.SampleSkillsPath(); | ||
|
|
||
| var kernel = new KernelBuilder() | ||
| .WithLogger(logger) | ||
| .WithAzureChatCompletionService( | ||
| Env.Var("AzureOpenAI__ChatDeploymentName"), | ||
| Env.Var("AzureOpenAI__Endpoint"), | ||
| Env.Var("AzureOpenAI__ApiKey")) | ||
| .Build(); | ||
|
|
||
| kernel.ImportSemanticSkillFromDirectory(folder, "SummarizeSkill", "WriterSkill"); | ||
|
|
||
| return kernel; | ||
| } | ||
|
|
||
| private static ISequentialPlanner GetPlanner( | ||
| IKernel kernel, | ||
| ILogger logger, | ||
| int maxTokens = 1024) | ||
| { | ||
| var plannerConfig = new SequentialPlannerConfig { MaxTokens = maxTokens }; | ||
|
|
||
| return new SequentialPlanner(kernel, plannerConfig).WithInstrumentation(logger); | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// Example of metering configuration in Application Insights | ||
| /// using <see cref="MeterListener"/> to attach for <see cref="Meter"/> recordings. | ||
| /// </summary> | ||
| /// <param name="meterListener">Instance of <see cref="MeterListener"/> for metering configuration.</param> | ||
| /// <param name="telemetryClient">Instance of Application Insights <see cref="TelemetryClient"/>.</param> | ||
| private static void ConfigureMetering(MeterListener meterListener, TelemetryClient telemetryClient) | ||
| { | ||
| meterListener.InstrumentPublished = (instrument, listener) => | ||
| { | ||
| // Subscribe to all metrics in Semantic Kernel | ||
| if (instrument.Meter.Name.StartsWith("Microsoft.SemanticKernel", StringComparison.Ordinal)) | ||
| { | ||
| listener.EnableMeasurementEvents(instrument); | ||
| } | ||
| }; | ||
|
|
||
| MeasurementCallback<double> measurementCallback = (instrument, measurement, tags, state) => | ||
| { | ||
| telemetryClient.GetMetric(instrument.Name).TrackValue(measurement); | ||
| }; | ||
|
|
||
| meterListener.SetMeasurementEventCallback(measurementCallback); | ||
|
|
||
| meterListener.Start(); | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// Example of advanced distributed tracing configuration in Application Insights | ||
| /// using <see cref="ActivityListener"/> to attach for <see cref="Activity"/> events. | ||
| /// </summary> | ||
| /// <param name="activityListener">Instance of <see cref="ActivityListener"/> for tracing configuration.</param> | ||
| /// <param name="telemetryClient">Instance of Application Insights <see cref="TelemetryClient"/>.</param> | ||
| private static void ConfigureTracing(ActivityListener activityListener, TelemetryClient telemetryClient) | ||
| { | ||
| var operations = new ConcurrentDictionary<string, IOperationHolder<DependencyTelemetry>>(); | ||
|
|
||
| // For more detailed tracing we need to attach Activity entity to Application Insights operation manually. | ||
| Action<Activity> activityStarted = activity => | ||
| { | ||
| var operation = telemetryClient.StartOperation<DependencyTelemetry>(activity); | ||
| operation.Telemetry.Type = activity.Kind.ToString(); | ||
| operations.TryAdd(activity.TraceId.ToString(), operation); | ||
| }; | ||
|
|
||
| // We also need to manually stop Application Insights operation when Activity entity is stopped. | ||
| Action<Activity> activityStopped = activity => | ||
| { | ||
| if (operations.TryRemove(activity.TraceId.ToString(), out var operation)) | ||
| { | ||
| telemetryClient.StopOperation(operation); | ||
| } | ||
| }; | ||
|
|
||
| // Subscribe to all traces in Semantic Kernel | ||
| activityListener.ShouldListenTo = | ||
| activitySource => activitySource.Name.StartsWith("Microsoft.SemanticKernel", StringComparison.Ordinal); | ||
|
|
||
| activityListener.Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllData; | ||
| activityListener.SampleUsingParentId = (ref ActivityCreationOptions<string> _) => ActivitySamplingResult.AllData; | ||
| activityListener.ActivityStarted = activityStarted; | ||
| activityListener.ActivityStopped = activityStopped; | ||
|
|
||
| ActivitySource.AddActivityListener(activityListener); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| // Copyright (c) Microsoft. All rights reserved. | ||
|
|
||
| using System; | ||
| using Microsoft.Extensions.Configuration; | ||
|
|
||
| #pragma warning disable CA1812 // instantiated by AddUserSecrets | ||
| internal sealed class Env | ||
| #pragma warning restore CA1812 | ||
| { | ||
| /// <summary> | ||
| /// Simple helper used to load env vars and secrets like credentials, | ||
| /// to avoid hard coding them in the sample code | ||
| /// </summary> | ||
| /// <param name="name">Secret name / Env var name</param> | ||
| /// <returns>Value found in Secret Manager or Environment Variable</returns> | ||
| internal static string Var(string name) | ||
| { | ||
| var configuration = new ConfigurationBuilder() | ||
| .AddUserSecrets<Env>() | ||
| .Build(); | ||
|
|
||
| var value = configuration[name]; | ||
| if (!string.IsNullOrEmpty(value)) | ||
| { | ||
| return value; | ||
| } | ||
|
|
||
| value = Environment.GetEnvironmentVariable(name); | ||
| if (string.IsNullOrEmpty(value)) | ||
| { | ||
| throw new ArgumentException($"Secret / Env var not set: {name}"); | ||
| } | ||
|
|
||
| return value; | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| // Copyright (c) Microsoft. All rights reserved. | ||
|
|
||
| using System.IO; | ||
| using System.Reflection; | ||
|
|
||
| internal static class RepoFiles | ||
| { | ||
| /// <summary> | ||
| /// Scan the local folders from the repo, looking for "samples/skills" folder. | ||
| /// </summary> | ||
| /// <returns>The full path to samples/skills</returns> | ||
| public static string SampleSkillsPath() | ||
| { | ||
| const string Parent = "samples"; | ||
| const string Folder = "skills"; | ||
|
|
||
| bool SearchPath(string pathToFind, out string result, int maxAttempts = 10) | ||
| { | ||
| var currDir = Path.GetFullPath(Assembly.GetExecutingAssembly().Location); | ||
| bool found; | ||
| do | ||
| { | ||
| result = Path.Join(currDir, pathToFind); | ||
| found = Directory.Exists(result); | ||
| currDir = Path.GetFullPath(Path.Combine(currDir, "..")); | ||
| } while (maxAttempts-- > 0 && !found); | ||
|
|
||
| return found; | ||
| } | ||
|
|
||
| if (!SearchPath(Parent + Path.DirectorySeparatorChar + Folder, out string path) | ||
| && !SearchPath(Folder, out path)) | ||
| { | ||
| throw new DirectoryNotFoundException("Skills directory not found. The app needs the skills from the repo to work."); | ||
| } | ||
|
|
||
| return path; | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
This introduces dependency on .net 7.0 thus breaking azure functions deployment as well as any other products targeting .net 6 LTS (related issue #1793)