feat: toggleable backstage

Author: johanneswuerbach

.github/workflows/ci.yaml

@@ -20,7 +20,7 @@ jobs:
       - name: Install terraform-docs
         run: |
           WORK_DIR=$(mktemp -d)
-          curl -Lo ${WORK_DIR}/terraform-docs.tar.gz https://github.com/terraform-docs/terraform-docs/releases/download/v0.16.0/terraform-docs-v0.16.0-$(uname)-amd64.tar.gz
+          curl -Lo ${WORK_DIR}/terraform-docs.tar.gz https://github.com/terraform-docs/terraform-docs/releases/download/v0.18.0/terraform-docs-v0.18.0-$(uname)-amd64.tar.gz
           cd ${WORK_DIR}
           tar -xzf terraform-docs.tar.gz
           chmod +x terraform-docs
@@ -31,11 +31,11 @@ jobs:
       - name: Check git diff is clean (all files generated should be committed)
         run: git diff --exit-code
 
-      - name: Terraform Format Check
-        run: make fmt-check
-
-      - name: Stub Github App credentials (required for validation)
-        run: cd ./examples/with-backstage && STUB_FILE=1 node create-gh-app/index.js
+      - uses: terraform-linters/setup-tflint@v4
+        with:
+          tflint_version: v0.51.1
+      - name: Terraform Lint
+        run: make lint
 
       - name: Terraform Validate
         run: make validate

.tflint.hcl

@@ -0,0 +1,10 @@
+plugin "terraform" {
+  enabled = true
+  preset  = "recommended"
+}
+
+plugin "google" {
+  enabled = true
+  version = "0.29.0"
+  source  = "github.com/terraform-linters/tflint-ruleset-google"
+}

Makefile

@@ -1,14 +1,18 @@
 TF_DIRS = $(patsubst %/main.tf, %, $(shell find . -type d -name .terraform -prune -o -name 'main.tf' -print))
 VALIDATE_TF_DIRS = $(addprefix validate-,$(TF_DIRS))
+LINT_TF_DIRS = $(addprefix lint-,$(TF_DIRS))
+DOCS_TF_DIRS = $(addprefix docs-,$(TF_DIRS))
+
+# Generate docs for a terraform directories
+$(DOCS_TF_DIRS): docs-%:
+	@echo "Docs $*"
+	terraform-docs --config docs/.terraform-docs.yaml $*
+	terraform-docs --config docs/.terraform-docs-example.yaml $*
 
 # Generate docs
 .PHONY: docs
-docs:
-	# terraform-docs --lockfile=false ./modules/base
-	terraform-docs --config docs/.terraform-docs.yaml .
-	terraform-docs --config docs/.terraform-docs-example.yaml .
-	terraform-docs --config docs/.terraform-docs.yaml ./examples/with-backstage
-	terraform-docs --config docs/.terraform-docs-example.yaml ./examples/with-backstage
+docs: $(DOCS_TF_DIRS)
+	@echo "All docs generated"
 
 # Format all terraform files
 fmt:
@@ -27,3 +31,16 @@ $(VALIDATE_TF_DIRS): validate-%:
 # Validate all terraform directories
 validate: $(VALIDATE_TF_DIRS)
 	@echo "All validated"
+
+# Lint a terraform directories
+$(LINT_TF_DIRS): lint-%:
+	@echo "Lint $*"
+	tflint --config "$(PWD)/.tflint.hcl" --chdir="$*"
+
+# Initialize tflint
+lint-init:
+	tflint --init
+
+# Lint all terraform directories
+lint: lint-init $(LINT_TF_DIRS) fmt-check
+	@echo "All linted"

README.md

@@ -1,4 +1,5 @@
 # Humanitec Google Cloud Reference Architecture Implementation
+>
 > TL;DR
 >
 >Skip the theory? Go [here](#how-to-spin-up-your-humanitec-google-cloud-reference-architecture-implementation) to spin up your Humanitec Google Cloud Reference Architecture Implementation.
@@ -19,33 +20,34 @@ When McKinsey originally [published the reference architecture](https://youtu.be
 
 ![Google Cloud reference architecture Humanitec](docs/images/Reference-Architecture-Google-Cloud-Humanitec.png)
 
-### Developer Control Plane 
+### Developer Control Plane
 
 This plane is the primary configuration layer and interaction point for the platform users. It harbors the following components:
 
 * A **Version Control System**. GitHub is a prominent example, but this can be any system that contains two types of repositories:
-    * Application Source Code
-    * Platform Source Code, e.g. using Terraform
+  * Application Source Code
+  * Platform Source Code, e.g. using Terraform
 * **Workload specifications**. The reference architecture uses [Score](https://developer.humanitec.com/score/overview/).
 * A **portal** for developers to interact with. It can be the Humanitec Portal, but you might also use [Backstage](https://backstage.io/) or any other portal on the market.
 
-### Integration and Delivery Plane 
+### Integration and Delivery Plane
 
 This plane is about building and storing the image, creating app and infra configs from the abstractions provided by the developers, and deploying the final state. It’s where the domains of developers and platform engineers meet.
 
 This plane usually contains four different tools:
+
 * A **CI pipeline**. It can be Github Actions or any CI tooling on the market.
 * The **image registry** holding your container images. Again, this can be any registry on the market.
 * An **orchestrator** which in our example, is the Humanitec Platform Orchestrator.
 * The **CD system**, which can be the Platform Orchestrator’s deployment pipeline capabilities — an external system triggered by the Orchestrator using a webhook, or a setup in tandem with GitOps operators like ArgoCD.
 
-### Monitoring and Logging Plane 
+### Monitoring and Logging Plane
 
 The integration of monitoring and logging systems varies greatly depending on the system. This plane however is not a focus of the reference architecture.
 
-### Security Plane 
+### Security Plane
 
-The security plane of the reference architecture is focused on the secrets management system. The secrets manager stores configuration information such as database passwords, API keys, or TLS certificates needed by an Application at runtime. It allows the Platform Orchestrator to reference the secrets and inject them into the Workloads dynamically. You can learn more about secrets management and integration with other secrets management [here](https://developer.humanitec.com/platform-orchestrator/security/overview). 
+The security plane of the reference architecture is focused on the secrets management system. The secrets manager stores configuration information such as database passwords, API keys, or TLS certificates needed by an Application at runtime. It allows the Platform Orchestrator to reference the secrets and inject them into the Workloads dynamically. You can learn more about secrets management and integration with other secrets management [here](https://developer.humanitec.com/platform-orchestrator/security/overview).
 
 The reference architecture sample implementations use the secrets store attached to the Humanitec SaaS system.
 
@@ -54,22 +56,19 @@ The reference architecture sample implementations use the secrets store attached
 This plane is where the actual infrastructure exists including clusters, databases, storage, or DNS services. The configuration of the Resources is managed by the Platform Orchestrator which dynamically creates app and infrastructure configurations with every deployment and creates, updates, or deletes dependent Resources as required.
 
 ## How to spin up your Humanitec Google Cloud Reference Architecture Implementation
-This repo contains an implementation of part of the Humanitec Reference Architecture for an Internal Developer Platform.
-
-To install an implementation containing add-ons, follow the separate README. We currently feature these add-ons:
 
-* [Base layer plus Backstage](examples/with-backstage/)
+This repo contains an implementation of part of the Humanitec Reference Architecture for an Internal Developer Platform, including Backstage as optional Portal solution.
 
 This repo covers the base layer of the implementation for Google Cloud (GCP).
 
 By default, the following will be provisioned:
 
-- VPC
-- GKE Autopilot Cluster
-- Google Service Account to access the cluster
-- Ingress NGINX in the cluster
-- Resource Definitions in Humanitec for:
-  - Kubernetes Cluster
+* VPC
+* GKE Autopilot Cluster
+* Google Service Account to access the cluster
+* Ingress NGINX in the cluster
+* Resource Definitions in Humanitec for:
+  * Kubernetes Cluster
 
 ### Prerequisites
 
@@ -99,13 +98,13 @@ This reference architecture implementation uses Terraform. You will need to do t
 
    For example:
 
-   ```
+   ```shell
    export HUMANITEC_TOKEN="my-humanitec-api-token"
    ```
 
 5. Run terraform:
 
-   ```
+   ```shell
    terraform init
    terraform plan
    terraform apply
@@ -130,46 +129,90 @@ There are many other optional inputs that can be set. The full list is described
 Check for the existence of key elements of the reference architecture. This is a subset of all elements only. For a complete list of what was installed, review the Terraform code.
 
 1. Set the `HUMANITEC_ORG` environment variable to the ID of your Humanitec Organization (must be all lowercase):
-```bash
-export HUMANITEC_ORG="my-humanitec-org"   
-```
+
+   ```bash
+   export HUMANITEC_ORG="my-humanitec-org"   
+   ```
 
 2. Verify the existence of the Resource Definition for the GKE cluster in your Humanitec Organization:
-```bash
-curl -s https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs/htc-ref-arch-cluster \
-  --header "Authorization: Bearer ${HUMANITEC_TOKEN}" \
-  | jq .id,.type
-```
-This should output:
-```bash
-"htc-ref-arch-cluster"
-"k8s-cluster"
-```
+
+   ```bash
+   curl -s https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs/htc-ref-arch-cluster \
+     --header "Authorization: Bearer ${HUMANITEC_TOKEN}" \
+     | jq .id,.type
+   ```
+
+   This should output:
+
+   ```bash
+   "htc-ref-arch-cluster"
+   "k8s-cluster"
+   ```
 
 3. Verify the existence of the newly created GKE cluster:
 
-```bash
-gcloud container clusters list --filter "name=htc-ref-arch-cluster"
-```
-This should output cluster data like this:
-```bash
-NAME                  LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE    NODE_VERSION    NUM_NODES  STATUS
-htc-ref-arch-cluster  <your-region>  xx.xx.xx-gke.xx xx.xx.xx.xx     n2d-standard-4  xx.xx.xx-gke.xx 3          RUNNING
-```
+   ```bash
+   gcloud container clusters list --filter "name=htc-ref-arch-cluster"
+   ```
+
+   This should output cluster data like this:
+
+   ```bash
+   NAME                  LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE    NODE_VERSION    NUM_NODES  STATUS
+   htc-ref-arch-cluster  <your-region>  xx.xx.xx-gke.xx xx.xx.xx.xx     n2d-standard-4  xx.xx.xx-gke.xx 3          RUNNING
+   ```
+
+### Enable a portal (optional)
+
+#### Portal Prerequisites
+
+Backstage requires a GitHub connection, which in turn needs:
+
+* A GitHub organization and permission to create new repositories in it. Go to <https://github.com/account/organizations/new> to create a new org (the "Free" option is fine). Note: is has to be an organization, a free account is not sufficient.
+* Create a classic github personal access token with `repo`, `workflow`, `delete_repo` and `admin:org` scope [here](https://github.com/settings/tokens).
+* Set the `GITHUB_TOKEN` environment variable to your token.
+
+  ```shell
+  export GITHUB_TOKEN="my-github-token"
+  ```
+
+* Set the `GITHUB_ORG_ID` environment variable to your GitHub organization ID.
+
+  ```shell
+  export GITHUB_ORG_ID="my-github-org-id"
+  ```
+
+* Install the GitHub App for Backstage into your GitHub organization
+  * Run `docker run --rm -it -e GITHUB_ORG_ID -v $(pwd):/pwd -p 127.0.0.1:3000:3000 ghcr.io/humanitec-architecture/create-gh-app` ([image source](https://github.com/humanitec-architecture/create-gh-app/)) and follow the instructions:
+    * “All repositories” ~> Install
+    * “Okay, […] was installed on the […] account.” ~> You can close the window and server.
+
+#### Portal Usage
+
+* Enable `with_backstage` inside your `terraform.tfvars` and configure the additional variables that a required for Backstage.
+* Perform another `terraform apply`
+
+#### Verify portal setup
+
+* [Fetch the DNS entry](https://developer.humanitec.com/score/getting-started/get-dns/) of the Humanitec Application `backstage`, Environment `development`.
+* Open the host in your browser.
+* Click the "Create" button and scaffold your first application.
 
 ### Cleaning up
 
 Once you are finished with the reference architecture, you can remove all provisioned infrastructure and the resource definitions created in Humanitec with the following:
 
-1. Ensure you are (still) logged in with `gcloud`.
+1. Delete all Humanitec Applications scaffolded using the Portal, if you used one, but not the `backstage` app itself.
+
+2. Ensure you are (still) logged in with `gcloud`.
 
-2. Ensure you still have the `HUMANITEC_TOKEN` environment variable set to an appropriate Humanitec API token with the `Administrator` role on the Humanitec Organization.
+3. Ensure you still have the `HUMANITEC_TOKEN` environment variable set to an appropriate Humanitec API token with the `Administrator` role on the Humanitec Organization.
 
    You can verify this in the UI if you log in with an Administrator user and go to Resource Management, and check the "Usage" of each resource definition with the prefix set in `humanitec_prefix` - by default this is `htc-ref-arch-`.
 
-3. Run terraform:
+4. Run terraform:
 
-   ```
+   ```shell
    terraform destroy
    ```
 
@@ -180,31 +223,63 @@ Once you are finished with the reference architecture, you can remove all provis
 
 | Name | Version |
 |------|---------|
+| terraform | >= 1.0.0 |
 | terraform | >= 1.3.0 |
+| github | ~> 5.38 |
+| google | ~> 5.1 |
+| helm | ~> 2.12 |
+| humanitec | ~> 1.0 |
+| kubernetes | ~> 2.25 |
+| random | ~> 3.5 |
+
+### Providers
+
+| Name | Version |
+|------|---------|
 | google | ~> 5.1 |
 | humanitec | ~> 1.0 |
+| local | n/a |
+| random | ~> 3.5 |
 
 ### Modules
 
 | Name | Source | Version |
 |------|--------|---------|
 | base | ./modules/base | n/a |
+| github | ./modules/github | n/a |
+| github\_app | github.com/humanitec-architecture/shared-terraform-modules | v2024-06-12//modules/github-app |
+| portal\_backstage | ./modules/portal-backstage | n/a |
+
+### Resources
+
+| Name | Type |
+|------|------|
+| [google_storage_bucket.terraform_state](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket) | resource |
+| [humanitec_service_user_token.deployer](https://registry.terraform.io/providers/humanitec/humanitec/latest/docs/resources/service_user_token) | resource |
+| [humanitec_user.deployer](https://registry.terraform.io/providers/humanitec/humanitec/latest/docs/resources/user) | resource |
+| [local_file.terraform_backend_config](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
+| [random_id.bucket_prefix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |
 
 ### Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
-| humanitec\_org\_id | ID of the Humanitec Organization to associate resources with. | `string` | n/a | yes |
+| gar\_repository\_location | Location of the Google Artifact Registry repository, | `string` | n/a | yes |
 | project\_id | GCP Project ID to provision resources in. | `string` | n/a | yes |
 | region | GCP Region to provision resources in. | `string` | n/a | yes |
 | environment | The environment to associate the reference architecture with. | `string` | `null` | no |
 | environment\_type | The environment type to associate the reference architecture with. | `string` | `"development"` | no |
+| gar\_repository\_id | Google Artifact Registry repository ID. | `string` | `"htc-ref-arch"` | no |
+| github\_org\_id | GitHub org id (required for Backstage) | `string` | `null` | no |
+| humanitec\_org\_id | Humanitec Organization ID (required for Backstage) | `string` | `null` | no |
 | humanitec\_prefix | A prefix that will be attached to all IDs created in Humanitec. | `string` | `"htc-ref-arch-"` | no |
+| with\_backstage | Deploy Backstage | `bool` | `false` | no |
 <!-- END_TF_DOCS -->
 
 ## Learn more
 
 Expand your knowledge by heading over to our learning path, and discover how to:
+
 * Deploy the Humanitec reference architecture using a cloud provider of your choice
 * Deploy and manage Applications using the Humanitec Platform Orchestrator and Score
 * Provision additional Resources and connect to them
@@ -227,5 +302,3 @@ Expand your knowledge by heading over to our learning path, and discover how to:
 * [Resource management theory](https://developer.humanitec.com/training/master-your-internal-developer-platform/resource-management-theory/)
 * [Tutorial: Provision a Redis cluster on AWS using Terraform](https://developer.humanitec.com/training/master-your-internal-developer-platform/provision-redis-aws/)
 * [Tutorial: Update Resource Definitions for related Applications](https://developer.humanitec.com/training/master-your-internal-developer-platform/update-resource-definitions-for-related-applications/)
-
-