Azure Config - Set Azure Key Vault secrets and Azure App Config settings to a struct
- Getting started
- Usage
This module is used to get secrets from an Azure Key Vault and settings from App Configuraion and set them into a struct. The idea of parsing
configuration values into a struct was inspired by env
.
It is not required to have a Key Vault if not parsing secrets, and it is not required to have an App Configuration if not parsing settings.
To mark a field in a struct to be populated by a secret set the struct tag secret
followed by the name
of the secret in Azure Key Vault, like so:
`secret:"<secret-name>"`
To mark a field i a struct to be populated by a setting set the struct tag setting
followed by the name
of the setting in Azure App Configuration, like so:
`setting:"<setting-name>"`
If the secret or setting does not exist the field will keep the value it had prior to the call to Parse
.
The secret and setting can be marked as required, this will make the call to Parse
return an error if the they
do not exist:
secret:"<secret-name>,required"
setting:"<setting-name>,required"
The error message contains all fields that have been marked as required that didn't have a value associated with them.
Note: Unexported fields will be ignored.
Note: If no struct tags are set for either secrets or settings the call to Parse
will be a no-op.
go get github.com/KarlGW/azcfg
- Go 1.18
- Azure Key Vault (if using secrets)
- Identity with at least read access to secrets in the target Key Vault
- Azure App Configuration (is using settings)
- Identity with at least read access to settings (key values) in the target App Configuration (if not using access key or connection string)
Using a system assigned managed identity as credential on an Azure service. For other authentication and credential methods see the sections Authentication and Credentials.
This scenario requires minimal configuration, as azcfg
automatically detects if the platform is configured
with a managed identity.
Set the environment variable AZCFG_KEYVAULT_NAME
to the name of the target Key Vault.
An alternative is to pass the option WithKeyVault
to the call to Parse
.
package main
import (
"context"
"fmt"
"time"
"github.com/KarlGW/azcfg"
)
type config struct {
Host string
Port int
Username string `secret:"username"`
Password string `secret:"password"`
Credential credential
}
type credential struct {
Key int `secret:"key"`
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
cfg := config{}
if err := azcfg.Parse(ctx, &cfg); err != nil {
// Handle error.
}
fmt.Printf("%+v\n", cfg)
}
{Host: Port:0 Username:username-from-keyvault Password:password-from-keyvault Credential:{Key:12345}}
Set the environment variable AZCFG_APPCONFIGURATION_NAME
to the name of the target App Configuration.
An alternative is to pass the option WithAppConfiguration
to the call to Parse
.
If the settings have a label associated with them, set the environment variable AZCFG_SETTINGS_LABEL
, or pass the option
WithSettingsLabel
to the call to Parse
. More information on handling labels (including multiple labels)
can be found in the section App Configuration setting labels.
package main
import (
"context"
"fmt"
"time"
"github.com/KarlGW/azcfg"
)
type config struct {
Host string
Port int
Username string `setting:"username"`
Password string `setting:"password"`
Credential credential
}
type credential struct {
Key int `setting:"key"`
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var cfg config
if err := azcfg.Parse(ctx, &cfg); err != nil {
// Handle error.
}
fmt.Printf("%+v\n", cfg)
}
{Host: Port:0 Username:username-from-appconfig Password:password-from-appconfig Credential:{Key:12345}}
- Set the environment variable
AZCFG_KEYVAULT_NAME
to the name of the target Key Vault. An alternative is to pass the optionWithKeyVault
to the call toParse
. - Set the environment variable
AZCFG_APPCONFIGURATION_NAME
to the name of the target App Configuration. An alternative is to pass the optionWithAppConfiguration
to the call toParse
.- If applicable set
AZCFG_SETTINGS_LABEL
, or use the optionWithSettingsLabel
, to target a specific label for the settings.
- If applicable set
package main
import (
"context"
"fmt"
"time"
"github.com/KarlGW/azcfg"
)
type config struct {
Host string
Port int
Username string `setting:"username"`
Password string `setting:"password"`
Credential credential
}
type credential struct {
Key int `secret:"key"`
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var cfg config
if err := azcfg.Parse(ctx, &cfg); err != nil {
// Handle error.
}
fmt.Printf("%+v\n", cfg)
}
{Host: Port:0 Username:username-from-appconfig Password:password-from-appconfig Credential:{Key:12345}}
Supported types
string
bool
uint
,uint8
,uint16
,uint32
,uint64
int
,int8
,int16
,int32
,int64
float32
,float64
complex64
,complex128
time.Duration
Note: Pointers to the above types are supported.
Slices are supported if the secret/setting are comma separated values (spaces are trimmed by the parser).
Strings
value1,value2,value3
Numbers
1,2,3
Configuration of the parser can be done with options provided to Parse
or to NewParser
and environment variables.
Options will override environment variables if provided.
Options can be set on Parse
or the parser with NewParser
. For the available options see Options.
Option functions are provided by the module for convenience, see Option.
Example with providing options with an anonymous function:
package main
import (
"context"
"time"
"github.com/KarlGW/azcfg"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
´
var cfg config
if err := azcfg.Parse(ctx, &cfg, func(o *Options) {
o.Credential = cred
o.KeyVault = "vault"
o.AppConfiguration = "appconfig"
o.Concurrency = 20
o.Timeout = 20 * time.Second
}); err != nil {
// Handle error.
}
}
These are the environment variables that are available to use to configure parsing.
AZCFG_CLOUD
- Target Cloud (Azure Puplic, Azure Government and Azure China).AZCFG_CONCURRENCY
- Concurrency limit for secret and setting retrieval.AZCFG_TIMEOUT
- Timeout for the underlying HTTP client.
AZCFG_TENANT_ID
- Tenant ID for service principal/app registration.AZCFG_CLIENT_ID
- Client/App ID for service principal/app registration or user assigned managed identity.AZCFG_CLIENT_SECRET
- Secret for service principal/app registration.AZCFG_CLIENT_CERTIFICATE
- PEM certificate encoded in Base64 for service principal/app registration.AZCFG_CLIENT_CERTIFICATE_PATH
- Path to PEM certificate for service principal/app registration.AZCFG_AZURE_CLI_CREDENTIAL
- Use Azure CLI credential.AZCFG_APPCONFIGURATION_ACCESS_KEY_ID
- Access key ID for App Configuration.AZCFG_APPCONFIGURATION_ACCESS_KEY_SECRET
- Access key secret for App Configuration.AZCFG_APPCONFIGURATION_CONNECTION_STRING
- Connection string for App Configuration.
More details on how to authenticate can be found in the Authentication section.
AZCFG_KEYVAULT_NAME
- Name of Key Vault containing secrets.AZCFG_SECRETS_VERSIONS
- Secret names and versions when requiring specific secret versions.
AZCFG_APPCONFIGURATION_NAME
- Name of App Configuration containing settings.AZCFG_SETTINGS_LABEL
- Label for the intended settings.AZCFG_SETTINGS_LABELS
- Setting names and labels when requiring specific labels for specific settings.
The default behaviour of Parse
is to ignore secrets and settings that does not exist and let the field contain it's original value.
To enforce fields to be set the option required
can be used.
type Example struct {
FieldA `secret:"field-a"`
FieldB `secret:"field-b,required"`
FieldC `setting:"field-c,required"`
}
If a required
secret or setting doesn't exist in the Key Vault an error will be returned. The error message contains all
fields that have been marked as required that didn't have a secret or setting associated with them.
An independent parser
can be created and passed around inside of the application.
package main
import (
"context"
"time"
"github.com/KarlGW/azcfg"
)
func main() {
parser, err := azcfg.NewParser()
if err != nil {
// Handle error.
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var cfg config
if err := parser.Parse(ctx, &cfg); err != nil {
// Handle error.
}
}
The constructor function NewParser
supports the same options as the module level Parse
function.
For supported options see Options
struct or list of function options in the Options section.
A struct can be set with values prior to parsing. This is useful if not all fields should be handled by the parser, or default values should
be set on the struct (in this case tag ,required
should not be set on the field).
If the values for fields that are tagged are retrived, they will overwrite the current values.
package main
import (
"context"
"fmt"
"time"
"github.com/KarlGW/azcfg"
)
type config struct {
Host string
Port int
Username string `secret:"username"`
Password string `secret:"password"`
Credential credential
}
type credential struct {
Key int `secret:"key"`
}
func main() {
cfg := config{
Host: "localhost",
Port: 8080
Username: os.Getenv("USERNAME")
Password: os.Getenv("PASSWORD")
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := azcfg.Parse(ctx, &cfg); err != nil {
// Handle error.
}
fmt.Printf("%+v\n", cfg)
}
Every call to Parse
or parser.Parse
requires a context.Context
. This is
the main way of setting timeouts. However, the internal clients
for fetching secrets and settings and their underlying HTTP client has
a default timeout of 30 seconds. This can be configured with setting
the Timeout
field on the Options
struct in an option function, or
using the dedicated option function, WithTimeout
.
Secrets in Key Vault have versions associated with them. By default the latest version is retrieved.
To target specific versions for specific secrets:
- Set the secret names with their associated versions to the environment variable
AZCFG_SECRETS_VERSIONS
with formatsecret1=version1,secret2=version2
. - Use the option function
WithSecretsVersions
and provide amap[string]string
with the secret name as key and version as value.
Settings in App Configuration can have labels associated with them.
To target a specific label for all settings:
- Set the label to the environment variable
AZCFG_SETTINGS_LABEL
. - Use the option function
WithSettingsLabel
.
To target speciefic settings with specific labels:
- Set the setting names with their associated labels to the environment variable
AZCFG_SETTINGS_LABELS
with format:setting1=label1,setting2=label2
. - Use the option function
WithSettingsLabels
and provide amap[string]string
with the setting name as key and label as value.
The module supports several ways of authenticating to Azure and get secrets from the target Key Vault and settings from the target App Configuration.
- Built-in credentials that supports Service Principal (Client Credentials with secret, certificate or an assertion), managed identity (system and user assigned) and Azure CLI. These provide in-memory caching of tokens to handle repeat calls.
- Credentials from
azidentity
with the submoduleauthopts
- Custom credential handling by implementing the
auth.Credential
interface.
For more information about option 2 and 3, see Credentials.
The built-in credential handling for managed identities have been tested on:
- Azure Functions
- Azure Container Apps
- Azure Container Instances
- Azure Virtual Machines
In addition to this it should work on:
- Azure App Services (since they make use of a similar mechanism for managed identities as Azure Functions)
Note: Sometimes it can take some time for the IMDS endpoint to start up on Azure Container instances, resulting in authentication failures. If these issues occur, either:
- Set the environment variable
AZCFG_MANAGED_IDENTITY_IMDS_DIAL_TIMEOUT
with a longer duration string, example:5s
. - Use the option
WithManagedIdentityIMDSDialTimeout
with atime.Duration
.
For more advanced scenarios for managed identities like Azure Stack or Service Fabric see the section
about using authopts
together with azidentity
.
For all authentication scenarios the following environment variables are used:
AZCFG_KEYVAULT_NAME
- Name of the Azure Key Vault (if using secrets).AZCFG_APPCONFIGURATION_NAME
- Name of the Azure App Configuration (if using settings).
Service Principal (client credentials with secret)
AZCFG_TENANT_ID
- Tenant ID of the service principal/application registration.AZCFG_CLIENT_ID
- Client ID (also called Application ID) of the service principal/application registration.AZCFG_CLIENT_SECRET
- Client Secret of the service principal/application registration.
Service Principal (client credentials with certificate)
AZCFG_TENANT_ID
- Tenant ID of the service principal/application registration.AZCFG_CLIENT_ID
- Client ID (also called Application ID) of the service principal/application registration.AZCFG_CLIENT_CERTIFICATE
- Base64 encoded certificate (PEM).AZCFG_CLIENT_CERTIFICATE_PATH
- Path to certificate (PEM).
Managed identity
AZCFG_CLIENT_ID
- (Optional) Client ID (also called Application ID) of the Managed Identity. Set if using a user assigned managed identity.
Azure CLI
AZCFG_AZURE_CLI_CREDENTIAL
Requires Azure CLI to be installed, and being logged in.
# Login to Azure.
az login
# Set subscription where Key Vault is provisioned.
az account set --subscription <subscription-id>
If more control is needed, such as custom environment variables or other means of getting the necessary values, options can be used. Provided options will override values set from environment variables.
Service Principal
// Client Secret.
azcfg.Parse(
ctx,
&cfg,
azcfg.WithClientSecretCredential(tenantID, clientID, clientSecret),
azcfg.WithKeyVault(vault),
)
// Client certificate.
azcfg.Parse(
ctx,
&cfg,
azcfg.WithClientCertificateCredential(tenantID, clientID, certificates, key),
azcfg.WithKeyVault(vault),
)
// Client assertion/federated credentials.
azcfg.Parse(
ctx,
&cfg,
azcfg.WithClientAssertionCredential(tenantID, clientID, func() (string, error) {
return "assertion", nil
}),
azcfg.WithKeyVault(vault),
)
Managed identity
// System assigned identity.
azcfg.Parse(ctx, &cfg, azcfg.WithManagedIdentity(), azcfg.WithKeyVault(vault))
// User assigned identity.
azcfg.Parse(ctx, &cfg, azcfg.WithManagedIdentity(), azcfg.WithClientID(clientID), azcfg.WithKeyVault(vault))
Azure CLI
azcfg.Parse(ctx, &cfg, azcfg.WithAzureCLICredential(), azcfg.WithKeyVault(vault))
Note: Azure CLI credentials are best suited for development.
Other
To use a credential provided from elsewhere, such as the azidentity
module see the section about
Credentials.
In addition to using a managed identity or a service principal (the recommended methods) to access an app configuration, it supports the use of an access key or connection string. If one of these are provided, they will take precedence over the identity credential.
Access key
Either use environment variables:
AZCFG_APPCONFIGURATION_ACCESS_KEY_ID
- ID of the access key.AZCFG_APPCONFIGURATION_ACCESS_KEY_SECRET
- Secret of the access key.
Or provide an option:
azcfg.Parse(ctx, &cfg, azcfg.WithAppConfigurationAccessKey(accessKeyID, accessKeySecret))
Connection string
The connection string has the following format:
Endpoint=https://{appConfiguration}.azconfig.io;Id={accessKeyID};Secret={accessKeySecret}
Either use an environment variable:
AZCFG_APPCONFIGURATION_CONNECTION_STRING
Or provide an option:
azcfg.Parse(ctx, &cfg, azcfg.WithAppConfigurationConnectionString(connectionString))
Custom credentials with token retrieval can be used using the option WithCredential
. They must satisfy the interface Credential
:
// Credential is the interface that wraps around method Token, Scope
// and SetScope.
type Credential interface {
Token(ctx context.Context, options ...TokenOption) (Token, error)
}
Note: It is up to the provided implementation to cache tokens if needed.
Since it is reasonable to assume that credentials retrieved with the help of the azidentity
module might need to be used, a submodule, authopts
is provided. This make it easier to reuse credentials from azidentity
.
Usage
go get github.com/KarlGW/azcfg/authopts
package main
import (
"context"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/KarlGW/azcfg"
"github.com/KarlGW/azcfg/authopts"
)
func main() {
cred, err := azidentity.NewDefaultAzureCredential(nil)
if err != nil {
// Handle error.
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var cfg config
if err := azcfg.Parse(ctx, &cfg, authopts.WithTokenCredential(cred)); err != nil {
// Handle error.
}
}
Note Like the default credential implementations, in-memory token caching is provided for repeat calls.
For additional information about how to use azidentity
, check its documentation.
By default the standard Azure (Public Cloud) is the target for all requests. By either setting an option or an environment variable others can be used.
The following are supported:
- Azure Public (the standard Azure cloud)
- Azure Government
- Azure China
With options
package main
import (
"context"
"time"
"github.com/KarlGW/azcfg"
"github.com/KarlGW/azcfg/azure/cloud"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var cfg Config
// Azure Public (is default, but for the case of the example).
if err := azcfg.Parse(ctx, &cfg, azcfg.WithCloud(cloud.AzurePublic)); err != nil {
// Handle error.
}
// Azure Government.
if err := azcfg.Parse(ctx, &cfg, azcfg.WithCloud(cloud.AzureGovernment)); err != nil {
// Handle error.
}
// Azure China.
if err := azcfg.Parse(ctx, &cfg, azcfg.WithCloud(cloud.AzureChina)); err != nil {
// Handle error.
}
// Parser.
parser, err := azcfg.NewParser(azcfg.WithCloud(cloud.AzurePublic))
if err != nil {
// Handle error.
}
}
With environment variable
Set the environment variable AZCFG_CLOUD
.
- Azure Public:
Azure
,Public
orAzurePublic
. - Azure Government:
Government
orAzureGovernment
. - Azure China:
China
orAzureChina
.
Note: Case insensitive.
The module have error types and sentinel errors.
Error
- Generic error that can contain multiple errors (including the sentinel errors of the module).RequiredFieldsError
- Error for required fields. Contains the required and missing fields.
These be handled with errors.As
.
Error
// Generic azcfg error.
var e *azcfg.Error
if errors.As(err, &e) {
for _, err := range e.Errors() {
if errors.Is(err, azcfg.ErrSecretRetrieval) {
// Handle secret retrieval error.
}
}
}
RequiredFieldsError
var e *azcfg.RequiredFieldsError
if errors.As(err, &e) {
// Range over required fields.
for _, required := range e.Required() {}
// Range over missing fields, required and not set.
for _, missing := range e.Missing() {}
}
ErrSetValue
ErrCredential
ErrSecretClient
ErrSecretRetrieval
ErrSettingClient
ErrSettingRetrieval
ErrIMDSEndpointUnavailable
ErrParseConnectionString
These can be handled with errors.Is
.