DSC meta configuration

[michaeltlombardi]

Summary of the new feature / enhancement

As a user, I want to define default behaviors/options for dsc so that I don't need to pass the options to every invocation or define a function to do so for me.

Right now, there's no persistent way for a user to control how dsc behaves except for the DSC_RESOURCE_PATH environmental variable. Users may want to set the default output format for DSC to pretty-json, or only use manifests that have been signed, or disable specific providers (see #274).

As the options for how dsc should behave expand, these needs will compound - consider the WhatIf scenario, whether to pre-check for permissions, etc.

Proposed technical implementation details (optional)

I haven't done much research on the available options for rust applications, but in an ideal world, we could borrow (a subset of) the functionality that viper has, supporting layered overrides, where each item in the following list takes precedence over the item below it:

• Explicitly defined flag option, like --format
• Environment variable value, like DSC_FORMAT
• Configuration file key value, like format
• Lookup from a key-value store, like consul or etcd (I think this is out of scope, included for completeness)
• Defined defaults in the code

Even if we only supported flags and environment variables, I think we'd have a more manageable UX. We could implement the configuration file handling later, if ever. With support for using environment variables as default options, users could use .env files as lightweight configuration, or set the variables in their CI jobs, or whatever makes sense for their context.

[anmenaga]

My vote is for a json configuration file sitting next to dsc.exe, similar to what PowerShell is using - it's proven to be a good solution and easy to implement from dev perspective.
We can even have a DSC resource to read/modify it, so that it won't require manual editing.

[michaeltlombardi]

I'm fine with either/or JSON or YAML, and we can readily schematize the configuration file for documenting/editing. I think a DSC Resource to read/modify it is perfectly sensible, could just be calling dsc meta get|set built into the CLI.

[SteveL-MSFT]

dsc should be a resource for its own config and as such the config file format can be either JSON or YAML

[theJasonHelmick]

DSC meta settings

This document describes potential settings for DSC's meta configuration. There are six
scenarios,including examples, organized into near-term and future priorities.

Note

The settings in the examples use snake-case and YAML. This is for readability in the proposal,
but actual naming should be more carefully considered. The example settings are illustrative, not
prescriptive.

• Resource discovery path - (P0)
• Tracing - (P0)
• Allowed configuration documents and operations - (P1)
• Allow listing resources and invocation operations - (P2)
• Require version pins in configurations - (P2)
• Require verified documents and resources - (P2)

Configuration file

The configuration file that contains the meta settings should be located with dsc.exe for portable
configurations, otherwise, should be located in the correct configuration location for that
operating system. File can be json or yaml and configured by a DSC resource (planned future).

File name json: dsc.settings.json
File name yaml: dsc.settings.yaml

P0 -----------------------

Resource discovery path

As a user, I want to pre-define the path DSC uses to look for resources and to control whether
that path can be overridden with the DSC_RESOURCE_PATH environment variable, so that I can have
more deterministic control over the resources invoked on my machine.

Users should be able to define an array of directories that DSC should search for non-built-in
resources¹. The settings should:

• Accept an array of directories instead of a path syntax to allow cross-platform path variable
  concatenation.
• Give users the option to append the PATH environment variable to the directories, ensuring that
  specifically included directories are searched for resources first and treated as canonical.
• Give users the option to define whether to allow overriding with the DSC_RESOURCE_PATH
  environment variable.

Example settings

With an empty configuration file

# settings.dsc.yaml

---
# Effective settings
resource_path:
  allow_env_override: true
  append_env_path:    true
  directories:        null

With directories defined

# settings.dsc.yaml
resource_path:
  directories:
    - '/ops/dsc/resources'
    - '/app/dsc'
---
# Effective settings
resource_path:
  allow_env_override: false
  append_env_path:    false
  directories:
    - '/ops/dsc/resources'
    - '/app/dsc'

With directories defined and PATH appended.

# settings.dsc.yaml
resource_path:
  append_env_path: true
  directories:
    - '/ops/dsc/resources'
    - '/app/dsc'
---
# Effective settings
resource_path:
  allow_env_override: false
  append_env_path:    true
  directories:
    - '/ops/dsc/resources'
    - '/app/dsc'

Tracing

As a user, I want to define a default trace level and format for DSC instead of always passing
the specified flags at runtime or using the DSC_TRACE_LEVEL environment variable.

Currently, the tracing defaults are built-in and can only be overridden by specifying the option flags on the root command or the DSC_TRACE_LEVEL environment variable (level only, not format). The settings should:

• Allow setting the trace level
• Allow setting the trace format
• Allow defining whether the settings can be overrridden.

Example settings

When tracing isn't specified in settings

# settings.dsc.yaml

---
# Effective settings
tracing:
  level:  warning
  format: default
  allow_override: true

When tracing.level is specified

# settings.dsc.yaml
tracing:
  level: info
---
# Effective settings
tracing:
  level: info
  format: default
  allow_override: true

When tracing is fully specified

# settings.dsc.yaml
tracing:
  level:  info
  format: json
  allow_override: false
---
# Effective settings
tracing:
  level:  info
  format: json
  allow_override: false

P1-P2 -----------------------

Allowed configuration documents and operations

As an infrastructure engineer, I want to limit DSC to only using approved configuration documents
so that we can limit issues and how the documents change our system.

As an infrastructure engineer, I want to limit the dsc config <operation> commands that can be
executed to ensure stronger control over how users can query and modify our systems.

The most coherent approach to this seems to be enabling users to define a list of allowed configuration documents with optional additional restrictions.

We could also define how users can send documents to DSC - the most reliable being allow-listed
document files with a known SHA and trusted signature, but we can allow users to configure whether
they allow arbitrary documents from stdin or as a JSON blob to an argument.

The settings should:

• Allow any configuration document from file or command input (stdin or argument) for any operation
  by default.

• Enable users to define an array of allowed documents by file path glob with optional constraints
  (like specifying the SHA of the document, or that the document is signed by a trusted persona).

• Enable users to define per-operation whether to allow documents from input and/or file. The value for each operation should be one of:

  • false - Forbid the operation command entirely.

  • true - Allow the operation command. When the settings also specify a list of allowed
    documents, only allow passing a configuration document by file path for set and what-if
    operations.

  • An array of values indicating how that operation command can receive a document:

    Array	Effect
    []	Forbid the command entirely.
    [from_document]	Forbid passing a configuration document from stdin or as an argument. Only allow using configuration documents saved as files.
    [from_input]	Forbid specifying the path to a configuration document saved as a file. Only allow using configuration documents passed from stdin or as an argument. (Unlikely to be used).
    [from_document, from_input]	Allow command regardless of how the configuration document is passed.

• When users define a list of allowed documents, forbid passing documents from input by default for
  the set command, including in what-if mode

• When passing a document by file path with a defined allow list, always raise an error for
  documents outside the allow list for set commands regardless of mode.

• When passing a document from stdin or argument with a defined allow list, raise a warning when
  the SHA of the document doesn't match the SHA of an allowed document.

• For configurations that include other configurations, the included configurations must also be
  allowed or DSC should raise an error (to prevent a malicious actor from altering an included
  configuration to inject misbehavior into a trusted document).

Example settings

When configuration isn't specified in settings

# settings.dsc.yaml

---
# Effective settings
configuration:
  operations:
    export: [from_input, from_document]
    get:    [from_input, from_document]
    test:   [from_input, from_document]
    whatIf: [from_input, from_document]
    set:    [from_input, from_document]

With an allow list for configuration documents

# settings.dsc.yaml
configuration:
  allow_documents:
    - /ops/dsc/configs/*          # simple string as file glob
    - path: /app/dsc/web.dsc.json # object with advanced options
      sha256: <sha>
---
# Effective settings
configuration:
  allow_documents:
    - path: /ops/dsc/configs/*   
    - path: /app/dsc/web.dsc.json
      sha256: <sha>
  operations:
    export: [from_input, from_document]
    get:    [from_input, from_document]
    test:   [from_input, from_document]
    whatIf: [from_document]
    set:    [from_document]

With an allow list and forbidding export

# settings.dsc.yaml
configuration:
  allow_documents:
    - /ops/dsc/configs/*          # simple string as file glob
    - path: /app/dsc/web.dsc.json # object with advanced options
      sha256: <sha>
  operations:
    export: false # convenience, could also be empty array
---
# Effective settings
configuration:
  allow_documents:
    - path: /ops/dsc/configs/*   
    - path: /app/dsc/web.dsc.json
      sha256: <sha>
  operations:
    export: []
    get:    [from_input, from_document]
    test:   [from_input, from_document]
    whatIf: [from_document]
    set:    [from_document]

With an allow list and boolean operation values

# settings.dsc.yaml
configuration:
  allow_documents:
    - /ops/dsc/configs/*          # simple string as file glob
    - path: /app/dsc/web.dsc.json # object with advanced options
      sha256: <sha>
  operations:
    export: false
    get:    true
    test:   true
    whatIf: true
    set:    true
---
# Effective settings
configuration:
  allow_documents:
    - path: /ops/dsc/configs/*   
    - path: /app/dsc/web.dsc.json
      sha256: <sha>
  operations:
    export: []
    get:    [from_input, from_document]
    test:   [from_input, from_document]
    whatIf: [from_document]
    set:    [from_document]

With boolean operation values and without an allow list

configuration:
  allow_documents:
    - /ops/dsc/configs/*          # simple string as file glob
    - path: /app/dsc/web.dsc.json # object with advanced options
      sha256: <sha>
  operations:
    export: []
    get:    true
    test:   true
    whatIf: true
    set:    true
---
# Effective settings
configuration:
  operations:
    export: []
    get:    [from_input, from_document]
    test:   [from_input, from_document]
    whatIf: [from_input, from_document]
    set:    [from_input, from_document]

With fully explicit settings

configuration:
  allow_documents:
    - /ops/dsc/configs/*          # simple string as file glob
    - path: /app/dsc/web.dsc.json # object with advanced options
      sha256: <sha>
  operations:
    export: []
    get:    [from_document]
    test:   [from_document]
    whatIf: [from_document]
    set:    [from_document]
---
# Effective settings
configuration:
  allow_documents:
    - path: /ops/dsc/configs/*   
    - path: /app/dsc/web.dsc.json
      sha256: <sha>
  operations:
    export: []
    get:    [from_document]
    test:   [from_document]
    whatIf: [from_document]
    set:    [from_document]

Allow listing resources and invocation operations

As an infrastructure engineer, I want to define an allow list for resources that can be invoked
or included in a configuration document.

As an infrastructure engineer, I want to limit the dsc resource <operation> commands that can be
executed to ensure stronger control over how users can query and modify our systems.

The most coherent approach to this seems to be enabling users to define a list of allowed resources
with optional additional restrictions. Users may prefer to define this list by path or by type or
both. This check should be independent of discovery, which uses the
resource_path settings.

For resource invocation, it probably makes the most sense to define a map of boolean values to
operations, true enabling direct invocation and false forbidding it for a given operation. For
simplicity, users should be able to specify true or false to allow/forbid all direct invocation
operations.

These settings should:

• Enable users to define whether to allow use of built-in resources (suspect this will always be
  true, but perhaps an org forbids them for some reason?)
• Enable users to specify file globs for the paths to allowed resources, with optional restrictions
  like SHA, signature, version range, etc.
• Enable users to specify name globs for the types of allowed resources, with optional restrictions.
• Enable users to specify which DSC resource invocation operations are allowed.
• Enable users to specify true or false for the setting that allows invocations to indicate
  whether to allow or forbid invocations in general.
• If the settings don't specifically define which invocation operations are allowed, forbid set
  and delete when the settings explicitly define any allowed configuration documents to prevent
  one-by-one changes to the system outside of the allowed configurations.
• If the settings define allowed invocation operations as true, still forbid set and delete
  when the settings also define a list of allowed configuration documents.

Example settings

With an empty configuration file

# settings.dsc.yaml

---
# Effective settings
resource:
  allow_built_in: true
  allow_invocation:
    get:    true
    test:   true
    set:    true
    export: true
    delete: true

With a list of allowed documents

# settings.dsc.yaml
configuration:
  allow_documents:
    - /ops/dsc/configs/*          # simple string as file glob
    - path: /app/dsc/web.dsc.json # object with advanced options
      sha256: <sha>
---
# Effective settings
configuration:
  allow_documents:
    - path: /ops/dsc/configs/*   
    - path: /app/dsc/web.dsc.json
      sha256: <sha>
  operations:
    export: [from_input, from_document]
    get:    [from_input, from_document]
    test:   [from_input, from_document]
    whatIf: [from_document]
    set:    [from_document]
resource:
  allow_built_in: true
  allow_invocation:
    get:    true
    test:   true
    set:    false
    export: true
    delete: false

With a list of allowed documents and allow all invocations

# settings.dsc.yaml
configuration:
  allow_documents:
    - /ops/dsc/configs/*          # simple string as file glob
    - path: /app/dsc/web.dsc.json # object with advanced options
      sha256: <sha>
resource:
  allow_invocation: true
---
# Effective settings
configuration:
  allow_documents:
    - path: /ops/dsc/configs/*   
    - path: /app/dsc/web.dsc.json
      sha256: <sha>
  operations:
    export: [from_input, from_document]
    get:    [from_input, from_document]
    test:   [from_input, from_document]
    whatIf: [from_document]
    set:    [from_document]
resource:
  allow_built_in: true
  allow_invocation:
    get:    true
    test:   true
    set:    false
    export: true
    delete: false

With all invocations forbidden

# settings.dsc.yaml
resource:
  allow_invocation: false
---
# Effective settings
resource:
  allow_built_in: true
  allow_invocation:
    get:    false
    test:   false
    set:    false
    export: false
    delete: false

With resources allowed by path

# settings.dsc.yaml
resource:
  allow_paths:
    - /ops/dsc/resources/*                             # simple string as file glob
    - path:   /app/dsc/resources/web.dsc.resource.json # object with advanced options
      sha256: <sha>
---
# Effective settings
resource:
  allow_built_in: true
  allow_paths:
    - path:   /ops/dsc/resources/*
    - path:   /app/dsc/resources/web.dsc.resource.json
      sha256: <sha>
  allow_invocation:
    get:    false
    test:   false
    set:    false
    export: false
    delete: false

With resources allowed by type

# settings.dsc.yaml
resource:
  allow_types:
      - Microsoft.SqlServer*    # Simple string as type glob
      - type: TSToy*            # object with advanced options
        require_signature: true
---
# Effective settings
resource:
  allow_built_in: true
  allow_types:
      - type: Microsoft.SqlServer*
      - type: TSToy*
        require_signature: true
  allow_invocation:
    get:    false
    test:   false
    set:    false
    export: false
    delete: false

With resources allowed by path and type

# settings.dsc.yaml
resource:
  allow_paths:
    - /ops/dsc/resources/*                             # simple string as file glob
    - path:   /app/dsc/resources/web.dsc.resource.json # object with advanced options
      sha256: <sha>
  allow_types:
      - Microsoft.SqlServer*    # Simple string as type glob
      - type: TSToy*            # object with advanced options
        require_signature: true
---
# Effective settings
resource:
  allow_built_in: true
  allow_paths:
    - path:   /ops/dsc/resources/*
    - path:   /app/dsc/resources/web.dsc.resource.json
      sha256: <sha>
  allow_types:
      - type: Microsoft.SqlServer*
      - type: TSToy*
        require_signature: true
  allow_invocation:
    get:    true
    test:   true
    set:    true
    export: true
    delete: true

resource:
  # Allow use of built-in resources, like `Microsoft.DSC/Group`
  allow_built_in: true # default
  # Define allowed resources by path glob
  allow_paths:
    # simple string as file glob
    - /ops/dsc/resources
    # object with advanced options
    - path:   /app/dsc/resources/web.dsc.resource.json
      sha256: <sha>
  # Define allowed resources by type glob
  allow_types:
      # Simle string as type glob
      - Microsoft.*
      # object with advanced options
      - type: Microsoft.SqlServer*
        require_signature: true
  allow_invocation:
    get:    true
    test:   true
    set:    true
    export: true
    delete: true

Require version pins in configurations

As an infrastructure engineer, I want to ensure that my configurations are as deterministic as
possible without relying entirely on manual verification by having DSC reject configuration
documents with resource instances that don't declare their version pin.

Currently, DSC doesn't support version pinning per resource instance or configuration. To help
users ensure deterministic configurations, we should have an opt-in setting that requires version
pins for resources in a configuration.

When DSC is set to require version pinning, it should fail the validation for any passed configuration
that includes one or more instances without a version pin. The error should collect all such instances
and report them in the error message, not just the first invalid resource instance.

Example settings

With an empty configuration file

# settings.dsc.yaml

---
# Effective settings
configuration:
  require_version_pinning: false

With the setting defined

# settings.dsc.yaml
configuration:
  require_version_pinning: true
---
# Effective settings
configuration:
  require_version_pinning: true

Require verified documents and resources

As an infrastructure engineer, I want to require all configuration documents and resources to be
signed by a trusted persona so that I can have increased trust in the resources that modify my
systems.

As an infrastructure engineer, I want to require all configuration documents and resources to
have a software bill of materials (SBOM) to comply with policies/regulations and have increased
confidence in the provenance of code that modifies my systems.

This requires answering questions about signing configurations and resources, and having a trust
model for DSC. This must be deferred until those domains are handled, but should be considered when
implementing them.

If signatures are required, any operation that includes an unsigned configuration document or
resource should raise an error indicating the document or resource isn't signed. Users should be
able to specify whether to require signatures for either or both, and to override on a per-document
level for explicitly allowed documents.

If SBOMs are required, any operation that includes a configuration or resource without one should
raise an error indicating what items are missing an SBOM. Users should be able to specify whether
to require SBOMs for either or both, and to override the setting on a per-resource level for
explicitly allowed resources.

The following code snippet is a gesture in this direction, but I don't have the familiarity with
signing or SBOMs to have a better idea at this time.

# Settings to define what DSC trusts for verifying signatures
verification:
  trusted_authorities:     # Define a list of authorities to use for verification
  trusted_personas:        # Define a subset of personas to trust
  require_signature: false # Define whether to require signatures for resources and documents
  require_sbom:      false # Define whether to require SBOMs for 
# Configuration settings
configuration:
  require_signature: true # Overrides verification.require_signature
  require_sbom:      true # Overrides verification.require_sbom
  allow_documents:
    # Allow all documents in this folder (require sbom and signature)
    - /ops/dsc/configs/*           
    # Allow all documents signed by this persona, regardless of path
    - signed_by: <trusted persona>  
    # Allow all documents in the folder & signed by the ops team
    - path: /ops/dsc/configs/*      
      signed_by: <ops team persona>
    # Allow this document without sbom, require signature
    - path: /temp/debug.dsc.config.yaml
      require_sbom: false
resource:
  require_signature: true # Overrides verification.require_signature
  require_sbom:      true # overrides verification.require_sbom
  allow_types:
    # Allow all resources in this namespace (require sbom and signature)
    - MyOrg*                       
    # Allow all resources signed by this persona, regardless of type
    - signed_by: <trusted persona> 
    # Allow all resources in this namespace signed by the ops team
    - type: MyOrg.Ops*             
      signed_by: <ops team persona>
    # Allow this resource without sbom, require signature
    - type: MyOrg.Debug/TelemetryTrace 

Remote discovery

Footnotes

1. DSC currently defines the DSC_RESOURCE_PATH environment variable. We should consider
   renaming this to DSC_DISCOVERY_PATH if we plan to support publishing configurations or
   supporting extensions (#), or accept that we'll need to define new
   DSC_CONFIGURATION_PATH/DSC_EXTENSION_PATH environment variables later, or that
   DSC_RESOURCE_PATH will also be used to find non-resource items. ↩

[SteveL-MSFT]

Per discussion for 3.0, we'll only implement the P0 items. Current thinking is to have a Microsoft.DSC/Config and Microsoft.DSC/Policy resources that handle scoping where Config is for the current user (the config file is stored in the same location as the cache) and Policy is for the instance of DSC (presumably can be installed system wide under a protected folder). Here Policy means the values cannot be overridden by the user while Config means it sets the default value, but can be overridden at runtime.

[michaeltlombardi]

I would recommend using Microsoft.DSC/Settings instead of Microsoft.DSC/Config to avoid further overloading the term config, if possible. The distinction between Policy and Settings is still intuitive to me.

[jambar42]

I like the idea of having a Settings file with each resource / configuration used as an override to the higher level Settings. This is similar to how it is done in Bicep as well. This would allow for more complex configurations where some resources are allowed to change state, while other potentially disruptive SET operations are disallowed.