Service Management in Bedrock

This document describes the workflow for deploying a set of services Bedrock workflows via the bedrock CLI tool.

High Level Overview

Confirm you have met the Requirements for Bedrock's automation.
Install and configure Bedrock CLI
Create and configure the required high level definition and manifest repositories
Initialize the Bedrock Application Repository
Add a service to the Bedrock Application Repository
Optional: Create a Service Revision

Notes:

Steps 1-4 typically only need to be done once. Multiple clusters may be configured to sync from the single Materialized Manifest Repositories, and multiple Project repositories can be pointed to the single High Level Definition Repository.
Step 5 can be repeated each time you need to onboard a service to your Bedrock automated infrastructure.
Step 6 can be run as many times as required to add a service revision to a Bedrock project.

An overview of how these different pieces fit together from an automation perspective:

Requirements

This guide assumes a few things as requirements to use this automation:

The application code and supporting repositories are hosted on Azure Devops.
- If starting from scratch, then first create a new Azure Devops Organization, then create a project.
The application will be packaged and run using container images hosted on Azure Container Registry
The user running bedrock has full access to the above resources.
The user is running the latest bedrock release.
The user has Azure CLI installed.
The user is running git version 2.22 or later.
Optional: If a user wishes to store helm charts in the application repositories, then all repositories (application, high level definition, materialized manifests) must be in the same Azure DevOps Organization AND Project.

Installing and Configuring Bedrock

bedrock is the Command Line Interface that provides automation around defining and operating Kubernetes clusters with Bedrock principles.

Setup Bedrock

Download the latest version of bedrock from the releases page and add it to your PATH.

To setup a local configuration:

Generate a Personal Access Token
Create a bedrock config file
Initialize bedrock

Generate Personal Access Token

Generate a new Personal Access Token (PAT) to grant bedrock permissions in the Azure Devops Project. Please grant PAT the following permissions:

Build (Read & execute)
Code (Read, write, & manage)
Variable Groups (Read, create, & manage)

For help, follow the guide.

Create Bedrock config file

Create a copy of bedrock-config.yaml from the starter template with the appropriate values for the azure_devops section.

Note: This bedrock-config.yaml should not be commited anywhere, as it contains sensitive credentials.

Initialize Bedrock

Run bedrock init -f <bedrock-config.yaml> where <bedrock-config.yaml> the path to the configuation file.

Note: When running bedrock init -f <bedrock-config.yaml>, bedrock will copy the values from the config file and store it into local memory elsewhere. If you wish to utilize bedrock with another project or target, then you must rerun bedrock init with another configuration first OR, you may overwrite each commands via flags.

Repositories

Our next step is to create or onboard the repositories that support the deployment of our services:

The high level definition repository
The materialized manifest repository
The application source code repository

High Level Definition Repository

This repository holds the Bedrock High Level Deployment Definition (HLD) and associated configurations.

This HLD is processed via fabrikate in Azure Devops on each change to generate Kubernetes YAML manifests that are applied to the Kubernetes cluster by Flux.

Initializing the High Level Definition Repository

Create a repository in the given AzDO project.
Edit your Bedrock config to point to this repo (if you haven't already done this).
Clone the repository.
Initialize via bedrock, this will add the fabrikate traefik2 as the initial sample component. This can be overridden via optional flags.
```
bedrock hld init --git-push
```

NOTE bedrock hld command documentation can be found here.

Materialized Manifests Repository

This repository holds all the materialized kubernetes manifests that should be deployed to a cluster. If a cluster has been deployed via Bedrock's Terraform templates, then flux should be configured to point to this repository and will deploy all manifests in this repository to the cluster periodically.

Initializing the Materialized Manifests Repository

Create a repository in the given AzDO project.
Edit your Bedrock config to point to this repo (if you haven't already done this).
Clone the repository.

Add a simple README to the repository

touch README.md
echo "This is the Flux Manifest Repository." >> README.md
git add -A
git commit -m "Initializing Materialized Manifests repository with a README."
git push -u origin --all

Deploy Manifest Generation Pipeline

Deploy a manifest generation pipeline between the high level definition repo and the materialized manifests repo. Assuming you have configured bedrock, you can run this without flag parameters from your HLD repo root:

$ bedrock hld install-manifest-pipeline

Application Repositories

These repositories hold the application code, its associated Dockerfile(s), and helm deployment charts.

Additionally, these repositories can hold one (single application) or more (monorepository) applications depending on your development methodology. Typically, each repository shold be configured with a "hld-lifecycle" Azure DevOps pipeline that will add all managed applications inside the repository to the High Level Definition Repository. Additionally, each application inside the repository should also have an associated Azure DevOps multi-stage pipeline that both builds and deploys the latest Docker image to Azure Container Registry and updates the associated configuation in the HLD repository with the latest image tag.

-TBD Section on packages directory and manging monorepositories vs single application repositories

Initializing an Application Repository

Create a repository in the given AzDO project.
Clone the repository.

Initialize the project via bedrock

$ bedrock project init
$ git add -A
$ git commit -m "Initializing application repository."
$ git push -u origin --all

Create a variable group via bedrock. If you are using a repo per service source control strategy, you only need to do this once.
```
$ export VARIABLE_GROUP_NAME=<my-vg-name>
$ bedrock project create-variable-group $VARIABLE_GROUP_NAME -r $ACR_NAME -u $SP_APP_ID -t $SP_TENANT -p $SP_PASS
$ git add -A
$ git commit -m "Adding Project Variable Group."
$ git push -u origin --all
```
where ACR_NAME is the name of the Azure Container Registry where the Docker Images will be served from and SP_APP_ID, SP_PASS, and, SP_TENANT are an associated Service Principal's ID, Password, and Tenant, that have Read and Write access to the ACR.
Deploy the lifecycle pipeline (optional flag parameters can be used if bedrock was not intialized)
```
$ bedrock project install-lifecycle-pipeline --org-name $ORG_NAME --devops-project $DEVOPS_PROJECT --repo-url $SERVICE_REPO_URL --repo-name $SERVICE_NAME
```
where ORG_NAME is the name of your Azure Devops org, DEVOPS_PROJECT is the name of your Azure Devops project, SERVICE_REPO_URL is the url that you used to clone your service from Azure Devops, and SERVICE_NAME is the name of the service.

Note: If you are using a repo per service source control strategy you should run install-lifecycle-pipeline once for each repo.

NOTE bedrock project command documentation can be found here.

Adding a Service to a Application Repository

Clone the repository.
Create the service via bedrock, there are optional parameters that should be used to configure the service and its associated helm charts (other optional flag parameters can be used if bedrock was not intialized)
```
SERVICE_NAME=<my-new-service-name>
bedrock service create $SERVICE_NAME . ...
git add -A
git commit -m "Adding $SERVICE_NAME to the repository."
git push -u origin --all
```

Deploy the service's multistage build pipeline via bedrock (optional flag parameters can be used if bedrock was not intialized)

bedrock service install-build-pipeline . -n $SERVICE_NAME-build-pipeline -o $ORG_NAME -r $SERVICE_NAME -u $SERVICE_REPO_URL -d $DEVOPS_PROJECT

Review and accept the pull request to add the service to the high level definition in Azure Devops.

NOTE bedrock service command documentation can be found here.

Helm Configuration for Bedrock CLI

bedrock service create allows a user to configure a service a number of ways with a backing helm chart.

Presently, there are are a number of options for bedrock service create documented below:

  -c, --helm-chart-chart <helm-chart>                         bedrock helm chart name. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")
  -r, --helm-chart-repository <helm-repository>               bedrock helm chart repository. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")

  -g, --helm-config-git <helm-git>                            bedrock helm chart configuration git repository. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")
  -b, --helm-config-branch <helm-branch>                      bedrock custom helm chart configuration branch. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")
  -p, --helm-config-path <helm-path>                          bedrock custom helm chart configuration path. --helm-chart-* and --helm-config-* are exclusive; you may only use one. (default: "")
  --service-build-vg <variable-group>                         existing azure devops variable groups
  may use multiple. (default: "")
  --service-build-variables <variables>                       existing variables from azure devops variable groups
  may use multiple. (default: "")

As noted by the the documentation text, helm-chart-* and helm-config-* are both mutually exclusive configurations: you can only use one set of configurations or the other.

This section intends on documenting the various use cases for both sets of mutually exclusive configurations.

Helm Charts in a well-known Helm Repository

A Helm Repository is a well known set of helm charts conforming to the helm repository guidelines. Perhaps the best known helm repository is the community run helm charts repository.

As an bedrock user, if you would like to incorporate helm charts from a well known public repository, you may simply run bedrock the following helm-chart arguments:

bedrock service create nginx my-nginx-service --helm-chart-chart stable/nginx --helm-chart-repository github.com/helm/charts

Helm Charts in a distinct Git Repository from Application Sources in the same Azure DevOps Project

If your Helm Charts are in their own distinct Git Repository in the same Azure DevOps project, you can use the helm-config arguments to configure bedrock:

bedrock service create fabrikam path/to/fabrikam \
  --helm-config-git https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts
  --helm-config-branch master \
  --helm-path /charts/fabrikam

The above invocation presumes that the helm chart repository configured for bedrock is different from the application repository configured for bedrock usage.

The helm-config-git parameter must not contain the username portion of a url. If you retrieve the URL from Azure DevOps's "Clone Repository" UI, it will automatically have the username filled for an HTTPs clone ie:

https://fabrikam@dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts

Ensure that the you remove the fabrikam@ portion of the URL when passing parameters to bedrock service create --helm-config-git:

https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts

When you invoke bedrock service create with --helm-config argments, there is a 4th optional argument that can be provided --helm-config-access-token-variable. This configuration option is the name of the environment variable containing a Personal Access Token to access the git repository in helm config git. In this scenario, however, both git repositories are in the same Azure DevOps project, fabrikam-project, so it is un-necessary to provide --helm-config-access-token-variable, as the Personal Access Token utilized by the pipelines, ACCESS_TOKEN_SECRET, will be the same for both application repository and helm chart repository. When --helm-config-access-token-variable is not provided, the repository defined by --helm-config-git will automatically use ACCESS_TOKEN_SECRET. The following scenario covers a situation in which you will want to configure the --helm-config-access-token-variable to something custom.

Helm Charts in a distinct Git Repository from Application Sources in a different Azure DevOps Project

If your Helm Charts are in their own distinct Git Repository in a different Azure DevOps project, you can use still use the helm-config arguments to configure bedrock, but must also provide another option, --helm-config-access-token-variable. This configuration option is the name of the environment variable containing the Personal Access Token to access the git repository in helm-config-git:

bedrock service create fabrikam path/to/fabrikam \
  --helm-config-git https://dev.azure.com/fabrikam/fabrikam-helm-charts-project/_git/fabrikam-helm-charts \
  --helm-config-branch master \
  --helm-path /charts/fabrikam \
  --helm-config-access-token-variable FABRIKAM_HELM_CHARTS_REPO_PAT

The helm-config-git parameter must not contain the username portion of a url. If you retrieve the URL from Azure DevOps's "Clone Repository" UI, it will automatically have the username filled for an HTTPs clone ie:

https://fabrikam@dev.azure.com/fabrikam/fabrikam-helm-charts-project/_git/fabrikam-helm-charts

Ensure that the you remove the fabrikam@ portion of the URL when passing parameters to bedrock service create --helm-config-git:

https://dev.azure.com/fabrikam/fabrikam-helm-charts-project/_git/fabrikam-helm-charts

Note the different Azure DevOps URL for the helm charts project in Azure DevOps and the addition of the --helm-config-access-token-variable parameter.

The --helm-config-access-token-variable parameter configures how an access.yaml file is written to the HLD for fabrikate to consume when rendering helm charts. For more information on authenticating with private git repositories when rendering helm charts, please refer to fabrikate's documentation.

When the bedrock.yaml is committed, all the pipelines created, and the HLD repository is populated, you must add the environment variable, FABRIKAM_HELM_CHARTS_REPO_PAT to the HLD to Materialized pipeline as a pipeline variable. To do so, find the HLD to Materialized pipeline in the pipelines view on Azure DevOps, select it, select Variables, then select New Variable:

Helm Charts in the same repository as the application

If your Helm Charts are intended to be placed adjacent to your application source (no distinct git repository), you may still use the helm-config arguments to configure bedrock.

If you presume that the fabrikam-app repository is the same repository as the repository of your application sources, then the following invocation will allow a user to configure a service with helm charts tracked in the same repository.

bedrock service create
  --helm-config-git https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-app \
  --helm-config-branch master \
  --helm-path /charts/fabrikam

The helm-config-git parameter must not contain the username portion of a url. If you retrieve the URL from Azure DevOps's "Clone Repository" UI, it will automatically have the username filled for an HTTPs clone ie:

https://fabrikam@dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts

Ensure that the you remove the fabrikam@ portion of the URL when passing parameters to bedrock service create --helm-config-git:

https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-helm-charts

Passing Variables as Dockerfile Build Arguments

If you want to pass in build arguments during the Dockerfile build process, you can inject them using the --service-build-vg and --service-build-variables arguments, which will take in multiple variable groups and variables, respectively.

bedrock service create
  --helm-config-git https://dev.azure.com/fabrikam/fabrikam-project/_git/fabrikam-app \
  --helm-config-branch master \
  --helm-path /charts/fabrikam \
  --service-build-vg bedrock-vg,fabrikam-vg \
  --service-build-variables FOO,BAR

In this example, variables FOO and BAR exist in either variable groups bedrock-vg or fabrikam-vg, and will be passed in as build arguments for the Dockerfile.

NOTE: It is important to understand that Azure DevOps will have the last variable group in the pipeline yaml take precedence. This means that if you have the same variable defined in both variable groups, the variable will take the value from the last variable group appended to the pipeline yaml file.

Creating a Service Revision

Create and checkout a new git branch

git branch <my-new-feature-branch>
git checkout <my-new-feature-branch>

Make code changes and commit

echo "# My New Added File" >> myNewFile.md
git add myNewFile.md
git commit -m "Adding my new file"
git push --set-upstream origin <my-new-feature-branch>

Create Service Revision via bedrock (optional flag parameters can be used if bedrock was not intialized)

bedrock service create-revision . -n $SERVICE_NAME-build-pipeline -o $ORG_NAME -r $SERVICE_NAME -u $SERVICE_REPO_URL -d $DEVOPS_PROJECT

NOTE bedrock service command documentation can be found here.

Helm Charts

To have bedrock's build pipelines work properly, an application needs an associated Helm chart with specific variables.

See the guide to building helm charts with bedrock for more details

Requirements

TBD

Sample Helm Chart

This repository bundles a sample helm chart. Please refer to the guide for building helm charts and the sample helm chart for details.

Variable Groups

TBD

Done to hold secure credentials and secrets.

Pipelines

TBD

Application build & update (1 per application)
HLD lifecycle (adds applications to HLD repo)
HLD to Manifests (generates manifests via fabrikate and places manifests into flux's source repo)

Upgrade your existing generated pipelines to get latest changes

When there's a new bedrock-cli release that introduces changes to generated pipelines that you would like to take advantage of, follow the steps below to update your existing pipelines.

To update your service repository pipelines:

Download the latest release of bedrock-cli.
Navigate to the locally cloned service source code repository for the branch you're working on
Rename hld-lifecycle.yaml to hld-lifecycle.yaml.backup
Run bedrock project init
Compare the newly created hld-lifecycle.yaml to hld-lifecycle.yaml.backup in case there are any custom changes you'd like to keep in the new pipeline.
cd into your services and for each service, rename build-update-hld.yaml to build-update-hld.yaml.backup
Run bedrock service create for each of your services, for eg. bedrock service create fabrikam.acme.frontend fabrikam.acme.frontend -d services -p chart. It should create a new build-update-hld.yaml and you can compare it against your build-update-hld.yaml.backup to restore customizations (if any). It should skip generation of Dockerfile if an existing one is found.
Make sure your variable group(s) are added correctly in these newly generated files. You can also run bedrock project append-variable-group to add a variable group.
View all the changes, stage, and commit them to remote repository.
Navigate to your pipelines on the browser and make sure pipelines execute successfully.
Run steps 7, 8, 9 and 10 for all your services

To update your HLD repository pipelines:

Head over to the HLD repository to update the manifest-generation pipeline.
Rename manifest-generation.yaml to manifest-generation.yaml.backup.
Run bedrock hld init. It should create a new manifest-generation.yaml and you can compare it against your manifest-generation.yaml.backup to restore customizations (if any). It should skip generation of component.yaml if an existing one is found.
Make sure your variable group(s) are added correctly in this newly generated yaml file. You can also run bedrock hld append-variable-group to add a variable group to this pipeline yaml.
View all the changes, stage, and commit them to remote repository.
Navigate to your pipelines on the browser and make sure pipelines execute successfully.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

project-service-management-guide.md

project-service-management-guide.md

Service Management in Bedrock

High Level Overview

Requirements

Installing and Configuring Bedrock

Setup Bedrock

Generate Personal Access Token

Create Bedrock config file

Initialize Bedrock

Repositories

High Level Definition Repository

Initializing the High Level Definition Repository

Materialized Manifests Repository

Initializing the Materialized Manifests Repository

Deploy Manifest Generation Pipeline

Application Repositories

Initializing an Application Repository

Adding a Service to a Application Repository

Helm Configuration for Bedrock CLI

Helm Charts in a well-known Helm Repository

Helm Charts in a distinct Git Repository from Application Sources in the same Azure DevOps Project

Helm Charts in a distinct Git Repository from Application Sources in a different Azure DevOps Project

Helm Charts in the same repository as the application

Passing Variables as Dockerfile Build Arguments

Creating a Service Revision

Helm Charts

Requirements

Sample Helm Chart

Variable Groups

Pipelines

Upgrade your existing generated pipelines to get latest changes

To update your service repository pipelines:

To update your HLD repository pipelines:

Files

project-service-management-guide.md

Latest commit

History

project-service-management-guide.md

File metadata and controls

Service Management in Bedrock

High Level Overview

Requirements

Installing and Configuring Bedrock

Setup Bedrock

Generate Personal Access Token

Create Bedrock config file

Initialize Bedrock

Repositories

High Level Definition Repository

Initializing the High Level Definition Repository

Materialized Manifests Repository

Initializing the Materialized Manifests Repository

Deploy Manifest Generation Pipeline

Application Repositories

Initializing an Application Repository

Adding a Service to a Application Repository

Helm Configuration for Bedrock CLI

Helm Charts in a well-known Helm Repository

Helm Charts in a distinct Git Repository from Application Sources in the same Azure DevOps Project

Helm Charts in a distinct Git Repository from Application Sources in a different Azure DevOps Project

Helm Charts in the same repository as the application

Passing Variables as Dockerfile Build Arguments

Creating a Service Revision

Helm Charts

Requirements

Sample Helm Chart

Variable Groups

Pipelines

Upgrade your existing generated pipelines to get latest changes

To update your service repository pipelines:

To update your HLD repository pipelines: