From 9069b8703ccf966496dbff259a0815bee7a489a3 Mon Sep 17 00:00:00 2001 From: Mike Wasson <3992422+MikeWasson@users.noreply.github.com> Date: Thu, 22 Aug 2019 10:47:18 -0700 Subject: [PATCH 1/2] Update Serverless title --- docs/reference-architectures/serverless/event-processing.md | 2 +- docs/reference-architectures/serverless/web-app.md | 2 +- docs/serverless/index.md | 4 ++-- docs/serverless/overview.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference-architectures/serverless/event-processing.md b/docs/reference-architectures/serverless/event-processing.md index 2a1913adf0b..0d3be0c6238 100644 --- a/docs/reference-architectures/serverless/event-processing.md +++ b/docs/reference-architectures/serverless/event-processing.md @@ -134,7 +134,7 @@ To deploy this reference architecture, view the [GitHub readme][readme]. ## Next steps -To learn more about the reference implementation, read [Show me the code: Serverless application with Azure Functions](../../serverless/index.md). +To learn more about the reference implementation, read [Code walkthrough: Serverless application with Azure Functions](../../serverless/index.md). diff --git a/docs/reference-architectures/serverless/web-app.md b/docs/reference-architectures/serverless/web-app.md index 1977a341942..c9739fe7682 100644 --- a/docs/reference-architectures/serverless/web-app.md +++ b/docs/reference-architectures/serverless/web-app.md @@ -255,7 +255,7 @@ To deploy the reference implementation for this architecture, see the [GitHub re ## Next steps -To learn more about the reference implementation, read [Show me the code: Serverless application with Azure Functions](../../serverless/index.md). +To learn more about the reference implementation, read [Code walkthrough: Serverless application with Azure Functions](../../serverless/index.md). diff --git a/docs/serverless/index.md b/docs/serverless/index.md index b4889581ca7..691bba75dd3 100644 --- a/docs/serverless/index.md +++ b/docs/serverless/index.md @@ -9,9 +9,9 @@ ms.service: architecture-center ms.subservice: reference-architecture --- -# Show me the code: Serverless application with Azure Functions +# Code walkthrough: Serverless application with Azure Functions -This article presents a code walk-through of a serverless application that uses [Azure Functions](/azure/azure-functions/). It describes the design decisions, implementation details, and some of the "gotchas" that you might encounter. +This article walks through the code for a serverless web application that uses [Azure Functions](/azure/azure-functions/). It describes the design decisions, implementation details, and some of the "gotchas" that you might encounter. ![GitHub logo](../_images/github.png) The source code for this application is available on [GitHub][github]. diff --git a/docs/serverless/overview.md b/docs/serverless/overview.md index 828fd517bd5..8e95485aac1 100644 --- a/docs/serverless/overview.md +++ b/docs/serverless/overview.md @@ -18,7 +18,7 @@ Serverless models abstract the underlying compute infrastructure. This allows de To explore serverless technologies in Azure, start with a serverless reference solution developed and tested by Microsoft. This two-part solution describes a hypothetical drone delivery system. Drones send in-flight status to the cloud, which stores these messages for later use. A web application allows users to retrieve these messages to get the latest status of these devices. - The code for this solution is available to download from [GitHub](https://github.com/mspnp/serverless-reference-implementation/tree/v0.1.0). -- The article [Show me the code: Serverless application with Azure Functions](index.md) walks you through this code, and explains why various choices were made. +- The article [Code walkthrough: Serverless application with Azure Functions](index.md) walks you through this code, and explains why various choices were made. Once you get a feel for how this reference solution works, proceed to learning the best practices and recommendations for developing similar serverless solutions: From d371e957872fd6c4f2eb45c9817300a1751f43b7 Mon Sep 17 00:00:00 2001 From: Mike Wasson <3992422+MikeWasson@users.noreply.github.com> Date: Thu, 22 Aug 2019 10:53:29 -0700 Subject: [PATCH 2/2] Fix up TOC node for Serverless --- .../serverless/event-processing.md | 2 +- .../serverless/web-app.md | 2 +- docs/serverless/code.md | 623 ++++++++++++++++++ docs/serverless/index.md | 619 +---------------- docs/serverless/overview.md | 34 - docs/toc.yml | 2 + 6 files changed, 642 insertions(+), 640 deletions(-) create mode 100644 docs/serverless/code.md delete mode 100644 docs/serverless/overview.md diff --git a/docs/reference-architectures/serverless/event-processing.md b/docs/reference-architectures/serverless/event-processing.md index 0d3be0c6238..3a0db839d7b 100644 --- a/docs/reference-architectures/serverless/event-processing.md +++ b/docs/reference-architectures/serverless/event-processing.md @@ -134,7 +134,7 @@ To deploy this reference architecture, view the [GitHub readme][readme]. ## Next steps -To learn more about the reference implementation, read [Code walkthrough: Serverless application with Azure Functions](../../serverless/index.md). +To learn more about the reference implementation, read [Code walkthrough: Serverless application with Azure Functions](../../serverless/code.md). diff --git a/docs/reference-architectures/serverless/web-app.md b/docs/reference-architectures/serverless/web-app.md index c9739fe7682..14999e223ba 100644 --- a/docs/reference-architectures/serverless/web-app.md +++ b/docs/reference-architectures/serverless/web-app.md @@ -255,7 +255,7 @@ To deploy the reference implementation for this architecture, see the [GitHub re ## Next steps -To learn more about the reference implementation, read [Code walkthrough: Serverless application with Azure Functions](../../serverless/index.md). +To learn more about the reference implementation, read [Code walkthrough: Serverless application with Azure Functions](../../serverless/code.md). diff --git a/docs/serverless/code.md b/docs/serverless/code.md new file mode 100644 index 00000000000..691bba75dd3 --- /dev/null +++ b/docs/serverless/code.md @@ -0,0 +1,623 @@ +--- +title: Implement a serverless application with Azure Functions +description: Code walk-through of a serverless application using Azure Functions +author: MikeWasson +ms.date: 06/13/2019 +ms.author: pnp +ms.topic: reference-architecture +ms.service: architecture-center +ms.subservice: reference-architecture +--- + +# Code walkthrough: Serverless application with Azure Functions + +This article walks through the code for a serverless web application that uses [Azure Functions](/azure/azure-functions/). It describes the design decisions, implementation details, and some of the "gotchas" that you might encounter. + +![GitHub logo](../_images/github.png) The source code for this application is available on [GitHub][github]. + +This article assumes a basic level of familiarity with the following technologies: + +- [Azure Functions](/azure/azure-functions/) +- [Azure Event Hubs](/azure/event-hubs/) +- [.NET Core](/dotnet/core/) + +You don't need to be an expert in Functions or Event Hubs, but you should understand their features at a high level. Here are some good resources to get started: + +- [An introduction to Azure Functions](/azure/azure-functions/functions-overview) +- [Features and terminology in Azure Event Hubs](/azure/event-hubs/event-hubs-features) + +## The scenario + +![Diagram of the functional blocks](./images/functional-diagram.png) + +Fabrikam manages a fleet of drones for a drone delivery service. The application consists of two main functional areas: + +- **Event ingestion**. During flight, drones send status messages to a cloud endpoint. The application ingests and processes these messages, and writes the results to a back-end database (Cosmos DB). The devices send messages in [protocol buffer](https://developers.google.com/protocol-buffers/) (protobuf) format. Protobuf is an efficient, self-describing serialization format. + + These messages contain partial updates. At a fixed interval, each drone sends a "key frame" message that contains all of the status fields. Between key frames, the status messages only include fields that changed since the last message. This behavior is typical of many IoT devices that need to conserve bandwidth and power. + +- **Web app**. A web application allows users to look up a device and query the device's last-known status. Users must sign into the application and authenticate with Azure Active Directory (Azure AD). The application only allows requests from users who are authorized to access the app. + +Here's a screenshot of the web app, showing the result of a query: + +![Screenshot of client app](./images/client-app.png) + +## Designing the application + +Fabrikam has decided to use Azure Functions to implement the application business logic. Azure Functions is an example of "Functions as a Service" (FaaS). In this computing model, a *function*"* is a piece of code that is deployed to the cloud and runs in a hosting environment. This hosting environment completely abstracts the servers that run the code. + +### Why choose a serverless approach? + +A serverless architecture with Functions is an example of an event-driven architecture. The function code is a triggered by some event that's external to the function — in this case, either a message from a drone, or an HTTP request from a client application. With a function app, you don't need to write any code for the trigger. You only write the code that runs in response to the trigger. That means you can focus on your business logic, rather than writing a lot of code to handle infrastructure concerns like messaging. + +There are also some operational advantages to using a serverless architecture: + +- There is no need to manage servers. +- Compute resources are allocated dynamically as needed. +- You are charged only for the compute resources used to execute your code. +- The compute resources scale on demand based on traffic. + +### Architecture + +The following diagram shows the high-level architecture of the application: + +![Architecture](./images/architecture.png) + +Event ingestion: + +1. Drone messages are ingested by Azure Event Hubs. +1. Event Hubs produces a stream of events that contain the message data. +1. These events trigger an Azure Functions app to process them. +1. The results are stored in Cosmos DB. + +Web app: + +1. Static files are served by CDN from Blob storage. +1. A user signs into the web app using Azure AD. +1. Azure API Management acts as a gateway that exposes a REST API endpoint. +1. HTTP requests from the client trigger an Azure Functions app that reads from Cosmos DB and returns the result. + +This application is based on two reference architectures, corresponding to the two functional blocks described above: + +- [Serverless event processing using Azure Functions](../reference-architectures/serverless/event-processing.md) +- [Serverless web application on Azure](../reference-architectures/serverless/web-app.md) + +You can read those articles to learn more about the high-level architecture, the Azure services that are used in the solution, and considerations for scalability, security, and reliability. + +## Drone telemetry function + +Let's start by looking at the function that processes drone messages from Event Hubs. The function is defined in a class named `RawTelemetryFunction`: + +```csharp +namespace DroneTelemetryFunctionApp +{ + public class RawTelemetryFunction + { + private readonly ITelemetryProcessor telemetryProcessor; + private readonly IStateChangeProcessor stateChangeProcessor; + private readonly TelemetryClient telemetryClient; + + public RawTelemetryFunction(ITelemetryProcessor telemetryProcessor, IStateChangeProcessor stateChangeProcessor, TelemetryClient telemetryClient) + { + this.telemetryProcessor = telemetryProcessor; + this.stateChangeProcessor = stateChangeProcessor; + this.telemetryClient = telemetryClient; + } + } + ... +} +``` + +This class has several dependencies, which are injected into the constructor using dependency injection: + +- The `ITelemetryProcessor` and `IStateChangeProcessor` interfaces define two helper objects. As we'll see, these objects do most of the work. + +- The [TelemetryClient](/dotnet/api/microsoft.applicationinsights.telemetryclient?view=azure-dotnet) is part of the Application Insights SDK. It is used to send custom application metrics to Application Insights. + +Later, we'll look at how to configure the dependency injection. For now, just assume these dependencies exist. + +## Configure the Event Hubs trigger + +The logic in the function is implemented as an asynchronous method named `RunAsync`. Here is the method signature: + +```csharp +[FunctionName("RawTelemetryFunction")] +[StorageAccount("DeadLetterStorage")] +public async Task RunAsync( + [EventHubTrigger("%EventHubName%", Connection = "EventHubConnection", ConsumerGroup ="%EventHubConsumerGroup%")]EventData[] messages, + [Queue("deadletterqueue")] IAsyncCollector deadLetterMessages, + ILogger logger) +{ + // implementation goes here +} +``` + +The method takes the following parameters: + +- `messages` is an array of event hub messages. +- `deadLetterMessages` is an Azure Storage Queue, used for storing dead letter messages. +- `logging` provides a logging interface, for writing application logs. These logs are sent to Azure Monitor. + +The `EventHubTrigger` attribute on the `messages` parameter configures the trigger. The properties of the attribute specify an event hub name, a connection string, and a [consumer group](/azure/event-hubs/event-hubs-features#event-consumers). (A *consumer group* is an isolated view of the Event Hubs event stream. This abstraction allows for multiple consumers of the same event hub.) + +Notice the percent signs (%) in some of the attribute properties. These indicate that the property specifies the name of an app setting, and the actual value is taken from that app setting at run time. Otherwise, without percent signs, the property gives the literal value. + +The `Connection` property is an exception. This property always specifies an app setting name, never a literal value, so the percent sign is not needed. The reason for this distinction is that a connection string is secret and should never be checked into source code. + +While the other two properties (event hub name and consumer group) are not sensitive data like a connection string, it's still better to put them into app settings, rather than hard coding. That way, they can be updated without recompiling the app. + +For more information about configuring this trigger, see [Azure Event Hubs bindings for Azure Functions](/azure/azure-functions/functions-bindings-event-hubs#trigger---attributes). + + +## Message processing logic + +Here's the implementation of the `RawTelemetryFunction.RunAsync` method that processes a batch of messages: + +```csharp +[FunctionName("RawTelemetryFunction")] +[StorageAccount("DeadLetterStorage")] +public async Task RunAsync( + [EventHubTrigger("%EventHubName%", Connection = "EventHubConnection", ConsumerGroup ="%EventHubConsumerGroup%")]EventData[] messages, + [Queue("deadletterqueue")] IAsyncCollector deadLetterMessages, + ILogger logger) +{ + telemetryClient.GetMetric("EventHubMessageBatchSize").TrackValue(messages.Length); + + foreach (var message in messages) + { + DeviceState deviceState = null; + + try + { + deviceState = telemetryProcessor.Deserialize(message.Body.Array, logger); + + try + { + await stateChangeProcessor.UpdateState(deviceState, logger); + } + catch (Exception ex) + { + logger.LogError(ex, "Error updating status document", deviceState); + await deadLetterMessages.AddAsync(new DeadLetterMessage { Exception = ex, EventData = message, DeviceState = deviceState }); + } + } + catch (Exception ex) + { + logger.LogError(ex, "Error deserializing message", message.SystemProperties.PartitionKey, message.SystemProperties.SequenceNumber); + await deadLetterMessages.AddAsync(new DeadLetterMessage { Exception = ex, EventData = message }); + } + } +} +``` + +When the function is invoked, the `messages` parameter contains an array of messages from the event hub. +Processing messages in batches will generally yield better performance than reading one message at a time. However, you have to make sure the function is resilient and handles failures and exceptions gracefully. Otherwise, if the function throws an unhandled exception in the middle of a batch, you might lose the remaining messages. This consideration is discussed in more detail in the section [Error handling](#error-handling). + +But if you ignore the exception handling, the processing logic for each message is simple: + +1. Call `ITelemetryProcessor.Deserialize` to deserialize the message that contains a device state change. +1. Call `IStateChangeProcessor.UpdateState` to process the state change. + +Let's look at these two methods in more detail, starting with the `Deserialize` method. + +### Deserialize method + +The `TelemetryProcess.Deserialize` method takes a byte array that contains the message payload. It deserializes this payload and returns a `DeviceState` object, which represents the state of a drone. The state may represent a partial update, containing just the delta from the last-known state. Therefore, the method needs to handle `null` fields in the deserialized payload. + +```csharp +public class TelemetryProcessor : ITelemetryProcessor +{ + private readonly ITelemetrySerializer serializer; + + public TelemetryProcessor(ITelemetrySerializer serializer) + { + this.serializer = serializer; + } + + public DeviceState Deserialize(byte[] payload, ILogger log) + { + DroneState restored = serializer.Deserialize(payload); + + log.LogInformation("Deserialize message for device ID {DeviceId}", restored.DeviceId); + + var deviceState = new DeviceState(); + deviceState.DeviceId = restored.DeviceId; + + if (restored.Battery != null) + { + deviceState.Battery = restored.Battery; + } + if (restored.FlightMode != null) + { + deviceState.FlightMode = (int)restored.FlightMode; + } + if (restored.Position != null) + { + deviceState.Latitude = restored.Position.Value.Latitude; + deviceState.Longitude = restored.Position.Value.Longitude; + deviceState.Altitude = restored.Position.Value.Altitude; + } + if (restored.Health != null) + { + deviceState.AccelerometerOK = restored.Health.Value.AccelerometerOK; + deviceState.GyrometerOK = restored.Health.Value.GyrometerOK; + deviceState.MagnetometerOK = restored.Health.Value.MagnetometerOK; + } + return deviceState; + } +} +``` + +This method uses another helper interface, `ITelemetrySerializer`, to deserialize the raw message. The results are then transformed into a [POCO](https://wikipedia.org/wiki/Plain_old_CLR_object) model that is easier to work with. This design helps to isolate the processing logic from the serialization implementation details. The `ITelemetrySerializer` interface is defined in a shared library, which is also used by the device simulator to generate simulated device events and send them to Event Hubs. + +```csharp +using System; + +namespace Serverless.Serialization +{ + public interface ITelemetrySerializer + { + T Deserialize(byte[] message); + + ArraySegment Serialize(T message); + } +} +``` + +### UpdateState method + +The `StateChangeProcessor.UpdateState` method applies the state changes. The last-known state for each drone is stored as a JSON document in Cosmos DB. Because the drones send partial updates, the application can't simply overwrite the document when it gets an update. Instead, it needs to fetch the previous state, merge the fields, and then perform an upsert operation. + +```csharp +public class StateChangeProcessor : IStateChangeProcessor +{ + private IDocumentClient client; + private readonly string cosmosDBDatabase; + private readonly string cosmosDBCollection; + + public StateChangeProcessor(IDocumentClient client, IOptions options) + { + this.client = client; + this.cosmosDBDatabase = options.Value.COSMOSDB_DATABASE_NAME; + this.cosmosDBCollection = options.Value.COSMOSDB_DATABASE_COL; + } + + public async Task> UpdateState(DeviceState source, ILogger log) + { + log.LogInformation("Processing change message for device ID {DeviceId}", source.DeviceId); + + DeviceState target = null; + + try + { + var response = await client.ReadDocumentAsync(UriFactory.CreateDocumentUri(cosmosDBDatabase, cosmosDBCollection, source.DeviceId), + new RequestOptions { PartitionKey = new PartitionKey(source.DeviceId) }); + + target = (DeviceState)(dynamic)response.Resource; + + // Merge properties + target.Battery = source.Battery ?? target.Battery; + target.FlightMode = source.FlightMode ?? target.FlightMode; + target.Latitude = source.Latitude ?? target.Latitude; + target.Longitude = source.Longitude ?? target.Longitude; + target.Altitude = source.Altitude ?? target.Altitude; + target.AccelerometerOK = source.AccelerometerOK ?? target.AccelerometerOK; + target.GyrometerOK = source.GyrometerOK ?? target.GyrometerOK; + target.MagnetometerOK = source.MagnetometerOK ?? target.MagnetometerOK; + } + catch (DocumentClientException ex) + { + if (ex.StatusCode == System.Net.HttpStatusCode.NotFound) + { + target = source; + } + } + + var collectionLink = UriFactory.CreateDocumentCollectionUri(cosmosDBDatabase, cosmosDBCollection); + return await client.UpsertDocumentAsync(collectionLink, target); + } +} +``` + +This code uses the `IDocumentClient` interface to fetch a document from Cosmos DB. If the document exists, the new state values are merged into the existing document. Otherwise, a new document is created. Both cases are handled by the `UpsertDocumentAsync` method. + +This code is optimized for the case where the document already exists and can be merged. On the first telemetry message from a given drone, the `ReadDocumentAsync` method will throw an exception, because there is no document for that drone. After the first message, the document will be available. + +Notice that this class uses dependency injection to inject the `IDocumentClient` for Cosmos DB and an `IOptions` with configuration settings. We'll see how to set up the dependency injection later. + +> [!NOTE] +> Azure Functions supports an output binding for Cosmos DB. This binding lets the function app write documents in Cosmos DB without any code. However, the output binding won't work for this particular scenario, because of the custom upsert logic that's needed. + +## Error handling + +As mentioned earlier, the `RawTelemetryFunction` function app processes a batch of messages in a loop. That means the function needs to handle any exceptions gracefully and continue processing the rest of the batch. Otherwise, messages might get dropped. + +If an exception is encountered when processing a message, the function puts the message onto a dead-letter queue: + +```csharp +catch (Exception ex) +{ + logger.LogError(ex, "Error deserializing message", message.SystemProperties.PartitionKey, message.SystemProperties.SequenceNumber); + await deadLetterMessages.AddAsync(new DeadLetterMessage { Exception = ex, EventData = message }); +} + ``` + +The dead-letter queue is defined using an [output binding](/azure/azure-functions/functions-bindings-storage-queue#output) to a storage queue: + +```csharp +[FunctionName("RawTelemetryFunction")] +[StorageAccount("DeadLetterStorage")] // App setting that holds the connection string +public async Task RunAsync( + [EventHubTrigger("%EventHubName%", Connection = "EventHubConnection", ConsumerGroup ="%EventHubConsumerGroup%")]EventData[] messages, + [Queue("deadletterqueue")] IAsyncCollector deadLetterMessages, // output binding + ILogger logger) + +``` + +Here the `Queue` attribute specifies the output binding, and the `StorageAccount` attribute specifies the name of an app setting that holds the connection string for the storage account. + +**Deployment tip:** In the Resource Manager template that creates the storage account, you can automatically populate an app setting with the connection string. The trick is to use the [listkeys](/azure/azure-resource-manager/resource-group-template-functions-resource#listkeys) function. + +Here is the section of the template that creates the storage account for the queue: + +```json + { + "name": "[variables('droneTelemetryDeadLetterStorageQueueAccountName')]", + "type": "Microsoft.Storage/storageAccounts", + "location": "[resourceGroup().location]", + "apiVersion": "2017-10-01", + "sku": { + "name": "[parameters('storageAccountType')]" + }, +``` + +Here is the section of the template that creates the function app. + +```json + + { + "apiVersion": "2015-08-01", + "type": "Microsoft.Web/sites", + "name": "[variables('droneTelemetryFunctionAppName')]", + "location": "[resourceGroup().location]", + "tags": { + "displayName": "Drone Telemetry Function App" + }, + "kind": "functionapp", + "dependsOn": [ + "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]", + ... + ], + "properties": { + "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]", + "siteConfig": { + "appSettings": [ + { + "name": "DeadLetterStorage", + "value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('droneTelemetryDeadLetterStorageQueueAccountName'), ';AccountKey=', listKeys(variables('droneTelemetryDeadLetterStorageQueueAccountId'),'2015-05-01-preview').key1)]" + }, + ... + +``` + +This defines an app setting named `DeadLetterStorage` whose value is populated using the `listkeys` function. It's important to make the function app resource depend on the storage account resource (see the `dependsOn` element). This guarantees that the storage account is created first and the connection string is available. + +## Setting up dependency injection + +The following code sets up dependency injection for the `RawTelemetryFunction` function: + +```csharp +[assembly: FunctionsStartup(typeof(DroneTelemetryFunctionApp.Startup))] + +namespace DroneTelemetryFunctionApp +{ + public class Startup : FunctionsStartup + { + public override void Configure(IFunctionsHostBuilder builder) + { + builder.Services.AddOptions() + .Configure((configSection, configuration) => + { + configuration.Bind(configSection); + }); + + builder.Services.AddTransient, TelemetrySerializer>(); + builder.Services.AddTransient(); + builder.Services.AddTransient(); + + builder.Services.AddSingleton(ctx => { + var config = ctx.GetService(); + var cosmosDBEndpoint = config.GetValue("CosmosDBEndpoint"); + var cosmosDBKey = config.GetValue("CosmosDBKey"); + return new DocumentClient(new Uri(cosmosDBEndpoint), cosmosDBKey); + }); + } + } +} +``` + +Azure Functions written for .NET can use the ASP.NET Core dependency injection framework. The basic idea is that you declare a startup method for your assembly. The method takes an `IFunctionsHostBuilder` interface, which is used to declare the dependencies for DI. You do this by calling `Add*` method on the `Services` object. When you add a dependency, you specify its lifetime: + +- *Transient* objects are created each time they're requested. +- *Scoped* objects are created once per function execution. +- *Singleton* objects are reused across function executions, within the lifetime of the function host. + +In this example, the `TelemetryProcessor` and `StateChangeProcessor` objects are declared as transient. This is appropriate for lightweight, stateless services. The `DocumentClient` class, on the other hand, should be a singleton for best performance. For more information, see [Performance tips for Azure Cosmos DB and .NET](/azure/cosmos-db/performance-tips#sdk-usage). + +If you refer back to the code for the [RawTelemetryFunction](#drone-telemetry-function), you'll see there another dependency that doesn't appear in DI setup code, namely the `TelemetryClient` class that is used to log application metrics. The Functions runtime automatically registers this class into the DI container, so you don't need to register it explicitly. + +For more information about DI in Azure Functions, see the following articles: + +- [Use dependency injection in .NET Azure Functions](/azure/azure-functions/functions-dotnet-dependency-injection) +- [Dependency injection in ASP.NET Core](/aspnet/core/fundamentals/dependency-injection) + +### Passing configuration settings in DI + +Sometimes an object must be initialized with some configuration values. Generally, these settings should come from app settings or (in the case of secrets) from Azure Key Vault. + +There are two examples in this application. First, the `DocumentClient` class takes a Cosmos DB service endpoint and key. For this object, the application registers a lambda that will be invoked by the DI container. This lambda uses the `IConfiguration` interface to read the configuration values: + +```csharp +builder.Services.AddSingleton(ctx => { + var config = ctx.GetService(); + var cosmosDBEndpoint = config.GetValue("CosmosDBEndpoint"); + var cosmosDBKey = config.GetValue("CosmosDBKey"); + return new DocumentClient(new Uri(cosmosDBEndpoint), cosmosDBKey); +}); +``` + +The second example is the `StateChangeProcessor` class. For this object, we use an approach called the [options pattern](/aspnet/core/fundamentals/configuration/options). Here's how it works: + +1. Define a class `T` that contains your configuration settings. In this case, the Cosmos DB database name and collection name. + + ```csharp + public class StateChangeProcessorOptions + { + public string COSMOSDB_DATABASE_NAME { get; set; } + public string COSMOSDB_DATABASE_COL { get; set; } + } + ``` + +1. Add the class `T` as an options class for DI. + + ```csharp + builder.Services.AddOptions() + .Configure((configSection, configuration) => + { + configuration.Bind(configSection); + }); + ``` + +1. In the constructor of the class that is being configured, include an `IOptions` parameter. + + ```csharp + public StateChangeProcessor(IDocumentClient client, IOptions options) + ``` + +The DI system will automatically populate the options class with configuration values and pass this to the constructor. + +There are several advantages of this approach: + +- Decouple the class from the source of the configuration values. +- Easily set up different configuration sources, such as environment variables or JSON configuration files. +- Simplify unit testing. +- Use a strongly typed options class, which is less error prone than just passing in scalar values. + +## GetStatus function + +The other Functions app in this solution implements a simple REST API to get the last-known status of a drone. +This function is defined in a class named `GetStatusFunction`. Here is the complete code for the function: + +```csharp +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Mvc; +using Microsoft.Azure.WebJobs; +using Microsoft.Azure.WebJobs.Extensions.Http; +using Microsoft.Extensions.Logging; +using System.Security.Claims; +using System.Threading.Tasks; + +namespace DroneStatusFunctionApp +{ + public static class GetStatusFunction + { + public const string GetDeviceStatusRoleName = "GetStatus"; + + [FunctionName("GetStatusFunction")] + public static IActionResult Run( + [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)]HttpRequest req, + [CosmosDB( + databaseName: "%COSMOSDB_DATABASE_NAME%", + collectionName: "%COSMOSDB_DATABASE_COL%", + ConnectionStringSetting = "COSMOSDB_CONNECTION_STRING", + Id = "{Query.deviceId}", + PartitionKey = "{Query.deviceId}")] dynamic deviceStatus, + ClaimsPrincipal principal, + ILogger log) + { + log.LogInformation("Processing GetStatus request."); + + if (!principal.IsAuthorizedByRoles(new[] { GetDeviceStatusRoleName }, log)) + { + return new UnauthorizedResult(); + } + + string deviceId = req.Query["deviceId"]; + if (deviceId == null) + { + return new BadRequestObjectResult("Missing DeviceId"); + } + + if (deviceStatus == null) + { + return new NotFoundResult(); + } + else + { + return new OkObjectResult(deviceStatus); + } + } + } +} +``` + +This function uses an HTTP trigger to process an HTTP GET request. The function uses a Cosmos DB input binding to fetch the requested document. One consideration is that this binding will run before the authorization logic is performed inside the function. If an unauthorized user requests a document, the function binding will still fetch the document. Then the authorization code will return a 401, so the user won't see the document. Whether this behavior is acceptable may depend on your requirements. For example, this approach might make it harder to audit data access for sensitive data. + +## Authentication and authorization + +The web app uses Azure AD to authenticate users. Because the app is a single-page application (SPA) running in the browser, the [implicit grant flow](/azure/active-directory/develop/v1-oauth2-implicit-grant-flow) is appropriate: + +1. The web app redirects the user to the identity provider (in this case, Azure AD). +1. The user enters their credentials. +1. The identity provider redirects back to the web app with an access token. +1. The web app sends a request to the web API and includes the access token in the Authorization header. + +![Implicit flow diagram](./images/implicit-flow.png) + +A Function application can be configured to authenticate users with zero code. For more information, see [Authentication and authorization in Azure App Service](/azure/app-service/overview-authentication-authorization). + +Authorization, on the other hand, generally requires some business logic. Azure AD supports *claims based authentication*. In this model, a user's identity is represented as a set of claims that come from the identity provider. A claim can be any piece of information about the user, such as their name or email address. + +The access token contains a subset of user claims. Among these are any application roles that the user is assigned to. + +The `principal` parameter of the function is a [ClaimsPrincipal](/dotnet/api/system.security.claims.claimsprincipal) object that contains the claims from the access token. Each claim is a key/value pair of claim type and claim value. The application uses these to authorize the request. + +The following extension method tests whether a `ClaimsPrincipal` object contains a set of roles. It returns `false` if any of the specified roles is missing. If this method returns false, the function returns HTTP 401 (Unauthorized). + +```csharp +namespace DroneStatusFunctionApp +{ + public static class ClaimsPrincipalAuthorizationExtensions + { + public static bool IsAuthorizedByRoles( + this ClaimsPrincipal principal, + string[] roles, + ILogger log) + { + var principalRoles = new HashSet(principal.Claims.Where(kvp => kvp.Type == "roles").Select(kvp => kvp.Value)); + var missingRoles = roles.Where(r => !principalRoles.Contains(r)).ToArray(); + if (missingRoles.Length > 0) + { + log.LogWarning("The principal does not have the required {roles}", string.Join(", ", missingRoles)); + return false; + } + + return true; + } + } +} +``` + +For more information about authentication and authorization in this application, see the [Security considerations](../reference-architectures/serverless/web-app.md#security-considerations) section of the reference architecture. + +## Next steps + +- View the source code on [GitHub][github]. + +- This application is an example of an event-driven architecture. Read more about the [Event-driven architecture style](../guide/architecture-styles/event-driven.md). + +- Azure Functions is just one of the compute options on Azure. For help with choosing a compute technology, see [Overview of Azure compute options](../guide/technology-choices/compute-overview.md). + + + +[github]: https://github.com/mspnp/serverless-reference-implementation/tree/v0.1.0 diff --git a/docs/serverless/index.md b/docs/serverless/index.md index 691bba75dd3..a47c294d2ed 100644 --- a/docs/serverless/index.md +++ b/docs/serverless/index.md @@ -1,623 +1,34 @@ --- -title: Implement a serverless application with Azure Functions -description: Code walk-through of a serverless application using Azure Functions -author: MikeWasson -ms.date: 06/13/2019 +title: Overview of serverless applications in Azure | Microsoft Docs +description: This article is a starting point to explore serverless architectures in Azure. +author: dsk-2015 +ms.date: 07/26/2019 ms.author: pnp ms.topic: reference-architecture ms.service: architecture-center ms.subservice: reference-architecture --- -# Code walkthrough: Serverless application with Azure Functions +# Building serverless applications on Azure -This article walks through the code for a serverless web application that uses [Azure Functions](/azure/azure-functions/). It describes the design decisions, implementation details, and some of the "gotchas" that you might encounter. +Serverless models abstract the underlying compute infrastructure. This allows developers to focus on business logic without needing extensive startup or maintenance cost to set up the solution. Serverless reduces the overall costs since you pay only for the duration the code was executed. This event-driven model is suitable for situations where some event triggers a defined action. For example, receiving an incoming device messages to store for later use, or a database update that needs some further processing. -![GitHub logo](../_images/github.png) The source code for this application is available on [GitHub][github]. +## Explore the recommendations -This article assumes a basic level of familiarity with the following technologies: +To explore serverless technologies in Azure, start with a serverless reference solution developed and tested by Microsoft. This two-part solution describes a hypothetical drone delivery system. Drones send in-flight status to the cloud, which stores these messages for later use. A web application allows users to retrieve these messages to get the latest status of these devices. -- [Azure Functions](/azure/azure-functions/) -- [Azure Event Hubs](/azure/event-hubs/) -- [.NET Core](/dotnet/core/) +- The code for this solution is available to download from [GitHub](https://github.com/mspnp/serverless-reference-implementation/tree/v0.1.0). +- The article [Code walkthrough: Serverless application with Azure Functions](./code.md) walks you through this code, and explains why various choices were made. -You don't need to be an expert in Functions or Event Hubs, but you should understand their features at a high level. Here are some good resources to get started: - -- [An introduction to Azure Functions](/azure/azure-functions/functions-overview) -- [Features and terminology in Azure Event Hubs](/azure/event-hubs/event-hubs-features) - -## The scenario - -![Diagram of the functional blocks](./images/functional-diagram.png) - -Fabrikam manages a fleet of drones for a drone delivery service. The application consists of two main functional areas: - -- **Event ingestion**. During flight, drones send status messages to a cloud endpoint. The application ingests and processes these messages, and writes the results to a back-end database (Cosmos DB). The devices send messages in [protocol buffer](https://developers.google.com/protocol-buffers/) (protobuf) format. Protobuf is an efficient, self-describing serialization format. - - These messages contain partial updates. At a fixed interval, each drone sends a "key frame" message that contains all of the status fields. Between key frames, the status messages only include fields that changed since the last message. This behavior is typical of many IoT devices that need to conserve bandwidth and power. - -- **Web app**. A web application allows users to look up a device and query the device's last-known status. Users must sign into the application and authenticate with Azure Active Directory (Azure AD). The application only allows requests from users who are authorized to access the app. - -Here's a screenshot of the web app, showing the result of a query: - -![Screenshot of client app](./images/client-app.png) - -## Designing the application - -Fabrikam has decided to use Azure Functions to implement the application business logic. Azure Functions is an example of "Functions as a Service" (FaaS). In this computing model, a *function*"* is a piece of code that is deployed to the cloud and runs in a hosting environment. This hosting environment completely abstracts the servers that run the code. - -### Why choose a serverless approach? - -A serverless architecture with Functions is an example of an event-driven architecture. The function code is a triggered by some event that's external to the function — in this case, either a message from a drone, or an HTTP request from a client application. With a function app, you don't need to write any code for the trigger. You only write the code that runs in response to the trigger. That means you can focus on your business logic, rather than writing a lot of code to handle infrastructure concerns like messaging. - -There are also some operational advantages to using a serverless architecture: - -- There is no need to manage servers. -- Compute resources are allocated dynamically as needed. -- You are charged only for the compute resources used to execute your code. -- The compute resources scale on demand based on traffic. - -### Architecture - -The following diagram shows the high-level architecture of the application: - -![Architecture](./images/architecture.png) - -Event ingestion: - -1. Drone messages are ingested by Azure Event Hubs. -1. Event Hubs produces a stream of events that contain the message data. -1. These events trigger an Azure Functions app to process them. -1. The results are stored in Cosmos DB. - -Web app: - -1. Static files are served by CDN from Blob storage. -1. A user signs into the web app using Azure AD. -1. Azure API Management acts as a gateway that exposes a REST API endpoint. -1. HTTP requests from the client trigger an Azure Functions app that reads from Cosmos DB and returns the result. - -This application is based on two reference architectures, corresponding to the two functional blocks described above: - -- [Serverless event processing using Azure Functions](../reference-architectures/serverless/event-processing.md) -- [Serverless web application on Azure](../reference-architectures/serverless/web-app.md) - -You can read those articles to learn more about the high-level architecture, the Azure services that are used in the solution, and considerations for scalability, security, and reliability. - -## Drone telemetry function - -Let's start by looking at the function that processes drone messages from Event Hubs. The function is defined in a class named `RawTelemetryFunction`: - -```csharp -namespace DroneTelemetryFunctionApp -{ - public class RawTelemetryFunction - { - private readonly ITelemetryProcessor telemetryProcessor; - private readonly IStateChangeProcessor stateChangeProcessor; - private readonly TelemetryClient telemetryClient; - - public RawTelemetryFunction(ITelemetryProcessor telemetryProcessor, IStateChangeProcessor stateChangeProcessor, TelemetryClient telemetryClient) - { - this.telemetryProcessor = telemetryProcessor; - this.stateChangeProcessor = stateChangeProcessor; - this.telemetryClient = telemetryClient; - } - } - ... -} -``` - -This class has several dependencies, which are injected into the constructor using dependency injection: - -- The `ITelemetryProcessor` and `IStateChangeProcessor` interfaces define two helper objects. As we'll see, these objects do most of the work. - -- The [TelemetryClient](/dotnet/api/microsoft.applicationinsights.telemetryclient?view=azure-dotnet) is part of the Application Insights SDK. It is used to send custom application metrics to Application Insights. - -Later, we'll look at how to configure the dependency injection. For now, just assume these dependencies exist. - -## Configure the Event Hubs trigger - -The logic in the function is implemented as an asynchronous method named `RunAsync`. Here is the method signature: - -```csharp -[FunctionName("RawTelemetryFunction")] -[StorageAccount("DeadLetterStorage")] -public async Task RunAsync( - [EventHubTrigger("%EventHubName%", Connection = "EventHubConnection", ConsumerGroup ="%EventHubConsumerGroup%")]EventData[] messages, - [Queue("deadletterqueue")] IAsyncCollector deadLetterMessages, - ILogger logger) -{ - // implementation goes here -} -``` - -The method takes the following parameters: - -- `messages` is an array of event hub messages. -- `deadLetterMessages` is an Azure Storage Queue, used for storing dead letter messages. -- `logging` provides a logging interface, for writing application logs. These logs are sent to Azure Monitor. - -The `EventHubTrigger` attribute on the `messages` parameter configures the trigger. The properties of the attribute specify an event hub name, a connection string, and a [consumer group](/azure/event-hubs/event-hubs-features#event-consumers). (A *consumer group* is an isolated view of the Event Hubs event stream. This abstraction allows for multiple consumers of the same event hub.) - -Notice the percent signs (%) in some of the attribute properties. These indicate that the property specifies the name of an app setting, and the actual value is taken from that app setting at run time. Otherwise, without percent signs, the property gives the literal value. - -The `Connection` property is an exception. This property always specifies an app setting name, never a literal value, so the percent sign is not needed. The reason for this distinction is that a connection string is secret and should never be checked into source code. - -While the other two properties (event hub name and consumer group) are not sensitive data like a connection string, it's still better to put them into app settings, rather than hard coding. That way, they can be updated without recompiling the app. - -For more information about configuring this trigger, see [Azure Event Hubs bindings for Azure Functions](/azure/azure-functions/functions-bindings-event-hubs#trigger---attributes). - - -## Message processing logic - -Here's the implementation of the `RawTelemetryFunction.RunAsync` method that processes a batch of messages: - -```csharp -[FunctionName("RawTelemetryFunction")] -[StorageAccount("DeadLetterStorage")] -public async Task RunAsync( - [EventHubTrigger("%EventHubName%", Connection = "EventHubConnection", ConsumerGroup ="%EventHubConsumerGroup%")]EventData[] messages, - [Queue("deadletterqueue")] IAsyncCollector deadLetterMessages, - ILogger logger) -{ - telemetryClient.GetMetric("EventHubMessageBatchSize").TrackValue(messages.Length); - - foreach (var message in messages) - { - DeviceState deviceState = null; - - try - { - deviceState = telemetryProcessor.Deserialize(message.Body.Array, logger); - - try - { - await stateChangeProcessor.UpdateState(deviceState, logger); - } - catch (Exception ex) - { - logger.LogError(ex, "Error updating status document", deviceState); - await deadLetterMessages.AddAsync(new DeadLetterMessage { Exception = ex, EventData = message, DeviceState = deviceState }); - } - } - catch (Exception ex) - { - logger.LogError(ex, "Error deserializing message", message.SystemProperties.PartitionKey, message.SystemProperties.SequenceNumber); - await deadLetterMessages.AddAsync(new DeadLetterMessage { Exception = ex, EventData = message }); - } - } -} -``` - -When the function is invoked, the `messages` parameter contains an array of messages from the event hub. -Processing messages in batches will generally yield better performance than reading one message at a time. However, you have to make sure the function is resilient and handles failures and exceptions gracefully. Otherwise, if the function throws an unhandled exception in the middle of a batch, you might lose the remaining messages. This consideration is discussed in more detail in the section [Error handling](#error-handling). - -But if you ignore the exception handling, the processing logic for each message is simple: - -1. Call `ITelemetryProcessor.Deserialize` to deserialize the message that contains a device state change. -1. Call `IStateChangeProcessor.UpdateState` to process the state change. - -Let's look at these two methods in more detail, starting with the `Deserialize` method. - -### Deserialize method - -The `TelemetryProcess.Deserialize` method takes a byte array that contains the message payload. It deserializes this payload and returns a `DeviceState` object, which represents the state of a drone. The state may represent a partial update, containing just the delta from the last-known state. Therefore, the method needs to handle `null` fields in the deserialized payload. - -```csharp -public class TelemetryProcessor : ITelemetryProcessor -{ - private readonly ITelemetrySerializer serializer; - - public TelemetryProcessor(ITelemetrySerializer serializer) - { - this.serializer = serializer; - } - - public DeviceState Deserialize(byte[] payload, ILogger log) - { - DroneState restored = serializer.Deserialize(payload); - - log.LogInformation("Deserialize message for device ID {DeviceId}", restored.DeviceId); - - var deviceState = new DeviceState(); - deviceState.DeviceId = restored.DeviceId; - - if (restored.Battery != null) - { - deviceState.Battery = restored.Battery; - } - if (restored.FlightMode != null) - { - deviceState.FlightMode = (int)restored.FlightMode; - } - if (restored.Position != null) - { - deviceState.Latitude = restored.Position.Value.Latitude; - deviceState.Longitude = restored.Position.Value.Longitude; - deviceState.Altitude = restored.Position.Value.Altitude; - } - if (restored.Health != null) - { - deviceState.AccelerometerOK = restored.Health.Value.AccelerometerOK; - deviceState.GyrometerOK = restored.Health.Value.GyrometerOK; - deviceState.MagnetometerOK = restored.Health.Value.MagnetometerOK; - } - return deviceState; - } -} -``` - -This method uses another helper interface, `ITelemetrySerializer`, to deserialize the raw message. The results are then transformed into a [POCO](https://wikipedia.org/wiki/Plain_old_CLR_object) model that is easier to work with. This design helps to isolate the processing logic from the serialization implementation details. The `ITelemetrySerializer` interface is defined in a shared library, which is also used by the device simulator to generate simulated device events and send them to Event Hubs. - -```csharp -using System; - -namespace Serverless.Serialization -{ - public interface ITelemetrySerializer - { - T Deserialize(byte[] message); - - ArraySegment Serialize(T message); - } -} -``` - -### UpdateState method - -The `StateChangeProcessor.UpdateState` method applies the state changes. The last-known state for each drone is stored as a JSON document in Cosmos DB. Because the drones send partial updates, the application can't simply overwrite the document when it gets an update. Instead, it needs to fetch the previous state, merge the fields, and then perform an upsert operation. - -```csharp -public class StateChangeProcessor : IStateChangeProcessor -{ - private IDocumentClient client; - private readonly string cosmosDBDatabase; - private readonly string cosmosDBCollection; - - public StateChangeProcessor(IDocumentClient client, IOptions options) - { - this.client = client; - this.cosmosDBDatabase = options.Value.COSMOSDB_DATABASE_NAME; - this.cosmosDBCollection = options.Value.COSMOSDB_DATABASE_COL; - } - - public async Task> UpdateState(DeviceState source, ILogger log) - { - log.LogInformation("Processing change message for device ID {DeviceId}", source.DeviceId); - - DeviceState target = null; - - try - { - var response = await client.ReadDocumentAsync(UriFactory.CreateDocumentUri(cosmosDBDatabase, cosmosDBCollection, source.DeviceId), - new RequestOptions { PartitionKey = new PartitionKey(source.DeviceId) }); - - target = (DeviceState)(dynamic)response.Resource; - - // Merge properties - target.Battery = source.Battery ?? target.Battery; - target.FlightMode = source.FlightMode ?? target.FlightMode; - target.Latitude = source.Latitude ?? target.Latitude; - target.Longitude = source.Longitude ?? target.Longitude; - target.Altitude = source.Altitude ?? target.Altitude; - target.AccelerometerOK = source.AccelerometerOK ?? target.AccelerometerOK; - target.GyrometerOK = source.GyrometerOK ?? target.GyrometerOK; - target.MagnetometerOK = source.MagnetometerOK ?? target.MagnetometerOK; - } - catch (DocumentClientException ex) - { - if (ex.StatusCode == System.Net.HttpStatusCode.NotFound) - { - target = source; - } - } - - var collectionLink = UriFactory.CreateDocumentCollectionUri(cosmosDBDatabase, cosmosDBCollection); - return await client.UpsertDocumentAsync(collectionLink, target); - } -} -``` - -This code uses the `IDocumentClient` interface to fetch a document from Cosmos DB. If the document exists, the new state values are merged into the existing document. Otherwise, a new document is created. Both cases are handled by the `UpsertDocumentAsync` method. - -This code is optimized for the case where the document already exists and can be merged. On the first telemetry message from a given drone, the `ReadDocumentAsync` method will throw an exception, because there is no document for that drone. After the first message, the document will be available. - -Notice that this class uses dependency injection to inject the `IDocumentClient` for Cosmos DB and an `IOptions` with configuration settings. We'll see how to set up the dependency injection later. - -> [!NOTE] -> Azure Functions supports an output binding for Cosmos DB. This binding lets the function app write documents in Cosmos DB without any code. However, the output binding won't work for this particular scenario, because of the custom upsert logic that's needed. - -## Error handling - -As mentioned earlier, the `RawTelemetryFunction` function app processes a batch of messages in a loop. That means the function needs to handle any exceptions gracefully and continue processing the rest of the batch. Otherwise, messages might get dropped. - -If an exception is encountered when processing a message, the function puts the message onto a dead-letter queue: - -```csharp -catch (Exception ex) -{ - logger.LogError(ex, "Error deserializing message", message.SystemProperties.PartitionKey, message.SystemProperties.SequenceNumber); - await deadLetterMessages.AddAsync(new DeadLetterMessage { Exception = ex, EventData = message }); -} - ``` - -The dead-letter queue is defined using an [output binding](/azure/azure-functions/functions-bindings-storage-queue#output) to a storage queue: - -```csharp -[FunctionName("RawTelemetryFunction")] -[StorageAccount("DeadLetterStorage")] // App setting that holds the connection string -public async Task RunAsync( - [EventHubTrigger("%EventHubName%", Connection = "EventHubConnection", ConsumerGroup ="%EventHubConsumerGroup%")]EventData[] messages, - [Queue("deadletterqueue")] IAsyncCollector deadLetterMessages, // output binding - ILogger logger) - -``` - -Here the `Queue` attribute specifies the output binding, and the `StorageAccount` attribute specifies the name of an app setting that holds the connection string for the storage account. - -**Deployment tip:** In the Resource Manager template that creates the storage account, you can automatically populate an app setting with the connection string. The trick is to use the [listkeys](/azure/azure-resource-manager/resource-group-template-functions-resource#listkeys) function. - -Here is the section of the template that creates the storage account for the queue: - -```json - { - "name": "[variables('droneTelemetryDeadLetterStorageQueueAccountName')]", - "type": "Microsoft.Storage/storageAccounts", - "location": "[resourceGroup().location]", - "apiVersion": "2017-10-01", - "sku": { - "name": "[parameters('storageAccountType')]" - }, -``` - -Here is the section of the template that creates the function app. - -```json - - { - "apiVersion": "2015-08-01", - "type": "Microsoft.Web/sites", - "name": "[variables('droneTelemetryFunctionAppName')]", - "location": "[resourceGroup().location]", - "tags": { - "displayName": "Drone Telemetry Function App" - }, - "kind": "functionapp", - "dependsOn": [ - "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]", - ... - ], - "properties": { - "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]", - "siteConfig": { - "appSettings": [ - { - "name": "DeadLetterStorage", - "value": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('droneTelemetryDeadLetterStorageQueueAccountName'), ';AccountKey=', listKeys(variables('droneTelemetryDeadLetterStorageQueueAccountId'),'2015-05-01-preview').key1)]" - }, - ... - -``` - -This defines an app setting named `DeadLetterStorage` whose value is populated using the `listkeys` function. It's important to make the function app resource depend on the storage account resource (see the `dependsOn` element). This guarantees that the storage account is created first and the connection string is available. - -## Setting up dependency injection - -The following code sets up dependency injection for the `RawTelemetryFunction` function: - -```csharp -[assembly: FunctionsStartup(typeof(DroneTelemetryFunctionApp.Startup))] - -namespace DroneTelemetryFunctionApp -{ - public class Startup : FunctionsStartup - { - public override void Configure(IFunctionsHostBuilder builder) - { - builder.Services.AddOptions() - .Configure((configSection, configuration) => - { - configuration.Bind(configSection); - }); - - builder.Services.AddTransient, TelemetrySerializer>(); - builder.Services.AddTransient(); - builder.Services.AddTransient(); - - builder.Services.AddSingleton(ctx => { - var config = ctx.GetService(); - var cosmosDBEndpoint = config.GetValue("CosmosDBEndpoint"); - var cosmosDBKey = config.GetValue("CosmosDBKey"); - return new DocumentClient(new Uri(cosmosDBEndpoint), cosmosDBKey); - }); - } - } -} -``` - -Azure Functions written for .NET can use the ASP.NET Core dependency injection framework. The basic idea is that you declare a startup method for your assembly. The method takes an `IFunctionsHostBuilder` interface, which is used to declare the dependencies for DI. You do this by calling `Add*` method on the `Services` object. When you add a dependency, you specify its lifetime: - -- *Transient* objects are created each time they're requested. -- *Scoped* objects are created once per function execution. -- *Singleton* objects are reused across function executions, within the lifetime of the function host. - -In this example, the `TelemetryProcessor` and `StateChangeProcessor` objects are declared as transient. This is appropriate for lightweight, stateless services. The `DocumentClient` class, on the other hand, should be a singleton for best performance. For more information, see [Performance tips for Azure Cosmos DB and .NET](/azure/cosmos-db/performance-tips#sdk-usage). - -If you refer back to the code for the [RawTelemetryFunction](#drone-telemetry-function), you'll see there another dependency that doesn't appear in DI setup code, namely the `TelemetryClient` class that is used to log application metrics. The Functions runtime automatically registers this class into the DI container, so you don't need to register it explicitly. - -For more information about DI in Azure Functions, see the following articles: - -- [Use dependency injection in .NET Azure Functions](/azure/azure-functions/functions-dotnet-dependency-injection) -- [Dependency injection in ASP.NET Core](/aspnet/core/fundamentals/dependency-injection) - -### Passing configuration settings in DI - -Sometimes an object must be initialized with some configuration values. Generally, these settings should come from app settings or (in the case of secrets) from Azure Key Vault. - -There are two examples in this application. First, the `DocumentClient` class takes a Cosmos DB service endpoint and key. For this object, the application registers a lambda that will be invoked by the DI container. This lambda uses the `IConfiguration` interface to read the configuration values: - -```csharp -builder.Services.AddSingleton(ctx => { - var config = ctx.GetService(); - var cosmosDBEndpoint = config.GetValue("CosmosDBEndpoint"); - var cosmosDBKey = config.GetValue("CosmosDBKey"); - return new DocumentClient(new Uri(cosmosDBEndpoint), cosmosDBKey); -}); -``` - -The second example is the `StateChangeProcessor` class. For this object, we use an approach called the [options pattern](/aspnet/core/fundamentals/configuration/options). Here's how it works: - -1. Define a class `T` that contains your configuration settings. In this case, the Cosmos DB database name and collection name. - - ```csharp - public class StateChangeProcessorOptions - { - public string COSMOSDB_DATABASE_NAME { get; set; } - public string COSMOSDB_DATABASE_COL { get; set; } - } - ``` - -1. Add the class `T` as an options class for DI. - - ```csharp - builder.Services.AddOptions() - .Configure((configSection, configuration) => - { - configuration.Bind(configSection); - }); - ``` - -1. In the constructor of the class that is being configured, include an `IOptions` parameter. - - ```csharp - public StateChangeProcessor(IDocumentClient client, IOptions options) - ``` - -The DI system will automatically populate the options class with configuration values and pass this to the constructor. - -There are several advantages of this approach: - -- Decouple the class from the source of the configuration values. -- Easily set up different configuration sources, such as environment variables or JSON configuration files. -- Simplify unit testing. -- Use a strongly typed options class, which is less error prone than just passing in scalar values. - -## GetStatus function - -The other Functions app in this solution implements a simple REST API to get the last-known status of a drone. -This function is defined in a class named `GetStatusFunction`. Here is the complete code for the function: - -```csharp -using Microsoft.AspNetCore.Http; -using Microsoft.AspNetCore.Mvc; -using Microsoft.Azure.WebJobs; -using Microsoft.Azure.WebJobs.Extensions.Http; -using Microsoft.Extensions.Logging; -using System.Security.Claims; -using System.Threading.Tasks; - -namespace DroneStatusFunctionApp -{ - public static class GetStatusFunction - { - public const string GetDeviceStatusRoleName = "GetStatus"; - - [FunctionName("GetStatusFunction")] - public static IActionResult Run( - [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)]HttpRequest req, - [CosmosDB( - databaseName: "%COSMOSDB_DATABASE_NAME%", - collectionName: "%COSMOSDB_DATABASE_COL%", - ConnectionStringSetting = "COSMOSDB_CONNECTION_STRING", - Id = "{Query.deviceId}", - PartitionKey = "{Query.deviceId}")] dynamic deviceStatus, - ClaimsPrincipal principal, - ILogger log) - { - log.LogInformation("Processing GetStatus request."); - - if (!principal.IsAuthorizedByRoles(new[] { GetDeviceStatusRoleName }, log)) - { - return new UnauthorizedResult(); - } - - string deviceId = req.Query["deviceId"]; - if (deviceId == null) - { - return new BadRequestObjectResult("Missing DeviceId"); - } - - if (deviceStatus == null) - { - return new NotFoundResult(); - } - else - { - return new OkObjectResult(deviceStatus); - } - } - } -} -``` - -This function uses an HTTP trigger to process an HTTP GET request. The function uses a Cosmos DB input binding to fetch the requested document. One consideration is that this binding will run before the authorization logic is performed inside the function. If an unauthorized user requests a document, the function binding will still fetch the document. Then the authorization code will return a 401, so the user won't see the document. Whether this behavior is acceptable may depend on your requirements. For example, this approach might make it harder to audit data access for sensitive data. - -## Authentication and authorization - -The web app uses Azure AD to authenticate users. Because the app is a single-page application (SPA) running in the browser, the [implicit grant flow](/azure/active-directory/develop/v1-oauth2-implicit-grant-flow) is appropriate: - -1. The web app redirects the user to the identity provider (in this case, Azure AD). -1. The user enters their credentials. -1. The identity provider redirects back to the web app with an access token. -1. The web app sends a request to the web API and includes the access token in the Authorization header. - -![Implicit flow diagram](./images/implicit-flow.png) - -A Function application can be configured to authenticate users with zero code. For more information, see [Authentication and authorization in Azure App Service](/azure/app-service/overview-authentication-authorization). - -Authorization, on the other hand, generally requires some business logic. Azure AD supports *claims based authentication*. In this model, a user's identity is represented as a set of claims that come from the identity provider. A claim can be any piece of information about the user, such as their name or email address. - -The access token contains a subset of user claims. Among these are any application roles that the user is assigned to. - -The `principal` parameter of the function is a [ClaimsPrincipal](/dotnet/api/system.security.claims.claimsprincipal) object that contains the claims from the access token. Each claim is a key/value pair of claim type and claim value. The application uses these to authorize the request. - -The following extension method tests whether a `ClaimsPrincipal` object contains a set of roles. It returns `false` if any of the specified roles is missing. If this method returns false, the function returns HTTP 401 (Unauthorized). - -```csharp -namespace DroneStatusFunctionApp -{ - public static class ClaimsPrincipalAuthorizationExtensions - { - public static bool IsAuthorizedByRoles( - this ClaimsPrincipal principal, - string[] roles, - ILogger log) - { - var principalRoles = new HashSet(principal.Claims.Where(kvp => kvp.Type == "roles").Select(kvp => kvp.Value)); - var missingRoles = roles.Where(r => !principalRoles.Contains(r)).ToArray(); - if (missingRoles.Length > 0) - { - log.LogWarning("The principal does not have the required {roles}", string.Join(", ", missingRoles)); - return false; - } - - return true; - } - } -} -``` - -For more information about authentication and authorization in this application, see the [Security considerations](../reference-architectures/serverless/web-app.md#security-considerations) section of the reference architecture. +Once you get a feel for how this reference solution works, proceed to learning the best practices and recommendations for developing similar serverless solutions: +- For developing a serverless *event ingestion* solution, refer to the reference-based guidance at [Serverless event processing using Azure Functions](../reference-architectures/serverless/event-processing.md). +- For developing a serverless *web application*, refer to the reference-based guidance at [Serverless web application on Azure](../reference-architectures/serverless/web-app.md). + ## Next steps -- View the source code on [GitHub][github]. +For in-depth discussion on developing serverless solutions on premises as well as in cloud, read [Serverless apps: Architecture, patterns, and Azure implementation](https://docs.microsoft.com/dotnet/standard/serverless-architecture/). -- This application is an example of an event-driven architecture. Read more about the [Event-driven architecture style](../guide/architecture-styles/event-driven.md). -- Azure Functions is just one of the compute options on Azure. For help with choosing a compute technology, see [Overview of Azure compute options](../guide/technology-choices/compute-overview.md). - -[github]: https://github.com/mspnp/serverless-reference-implementation/tree/v0.1.0 diff --git a/docs/serverless/overview.md b/docs/serverless/overview.md deleted file mode 100644 index 8e95485aac1..00000000000 --- a/docs/serverless/overview.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -title: Overview of serverless applications in Azure | Microsoft Docs -description: This article is a starting point to explore serverless architectures in Azure. -author: dsk-2015 -ms.date: 07/26/2019 -ms.author: pnp -ms.topic: reference-architecture -ms.service: architecture-center -ms.subservice: reference-architecture ---- - -# Serverless solution on Azure - -Serverless models abstract the underlying compute infrastructure. This allows developers to focus on business logic without needing extensive startup or maintenance cost to set up the solution. Serverless reduces the overall costs since you pay only for the duration the code was executed. This event-driven model is suitable for situations where some event triggers a defined action. For example, receiving an incoming device messages to store for later use, or a database update that needs some further processing. - -## Explore the recommendations - -To explore serverless technologies in Azure, start with a serverless reference solution developed and tested by Microsoft. This two-part solution describes a hypothetical drone delivery system. Drones send in-flight status to the cloud, which stores these messages for later use. A web application allows users to retrieve these messages to get the latest status of these devices. - -- The code for this solution is available to download from [GitHub](https://github.com/mspnp/serverless-reference-implementation/tree/v0.1.0). -- The article [Code walkthrough: Serverless application with Azure Functions](index.md) walks you through this code, and explains why various choices were made. - -Once you get a feel for how this reference solution works, proceed to learning the best practices and recommendations for developing similar serverless solutions: - -- For developing a serverless *event ingestion* solution, refer to the reference-based guidance at [Serverless event processing using Azure Functions](../reference-architectures/serverless/event-processing.md). -- For developing a serverless *web application*, refer to the reference-based guidance at [Serverless web application on Azure](../reference-architectures/serverless/web-app.md). - -## Next steps - -For in-depth discussion on developing serverless solutions on premises as well as in cloud, read [Serverless apps: Architecture, patterns, and Azure implementation](https://docs.microsoft.com/dotnet/standard/serverless-architecture/). - - - - diff --git a/docs/toc.yml b/docs/toc.yml index f432d0beb0d..65f9e61193f 100644 --- a/docs/toc.yml +++ b/docs/toc.yml @@ -617,6 +617,8 @@ items: items: - name: Overview href: serverless/index.md + - name: Code walkthrough + href: serverless/code.md - name: Reference architectures items: - name: Serverless event processing