Skip to content

[bitnami/airflow] adds support for retrieving the password for external Postgres and Redis from an existing secret #1917

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sameersbn merged 12 commits into from
Feb 20, 2020
Merged

[bitnami/airflow] adds support for retrieving the password for external Postgres and Redis from an existing secret #1917

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a00fdac
use existing secret to read external redis and postgres passwords from
chgl Feb 12, 2020
88695b8
bumped chart version
chgl Feb 12, 2020
0d01f3d
Merge remote-tracking branch 'upstream/master'
chgl Feb 12, 2020
2e36e90
Merge remote-tracking branch 'upstream/master'
chgl Feb 13, 2020
a9dfb7d
Merge branch 'master' of https://github.com/bitnami/charts
chgl Feb 15, 2020
2ca88f1
implemented airflow.redis.secretName and airflow.postgresql.secretNam…
chgl Feb 16, 2020
03e26fc
renamed 'db-password' key to 'postgresql-password'
chgl Feb 16, 2020
dddb050
use airflow.secretName template to retrieve the app credentials in NO…
chgl Feb 16, 2020
305946b
combined conditional check for secret creation
chgl Feb 17, 2020
799d054
Merge remote-tracking branch 'upstream/master'
chgl Feb 17, 2020
d44178d
Merge branch 'master' into master
chgl Feb 17, 2020
2cd77f0
Merge branch 'master' into master
Feb 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion bitnami/airflow/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v1
name: airflow
version: 4.3.1
version: 4.4.0
appVersion: 1.10.9
description: Apache Airflow is a platform to programmatically author, schedule and monitor workflows.
keywords:
Expand Down
41 changes: 39 additions & 2 deletions bitnami/airflow/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ The command removes all the Kubernetes components associated with the chart and

## Parameters

The following tables lists the configurable parameters of the Kafka chart and their default values.
The following tables lists the configurable parameters of the Airflow chart and their default values.

| Parameter | Description | Default |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
Expand Down Expand Up @@ -106,7 +106,7 @@ The following tables lists the configurable parameters of the Kafka chart and th
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.port` | Airflow Web port | `8080` |
| `service.nodePort` | Kubernetes Service nodePort | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Kafka Service | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Airflow Service | `nil` |
| `service.annotations` | Service annotations | `` |
| `ingress.enabled` | Enable ingress controller resource | `false` |
| `ingress.certManager` | Add annotations for cert-manager | `false` |
Expand Down Expand Up @@ -144,11 +144,13 @@ The following tables lists the configurable parameters of the Kafka chart and th
| `externalDatabase.password` | External PostgreSQL password | `nil` |
| `externalDatabase.database` | External PostgreSQL database name | `nil` |
| `externalDatabase.port` | External PostgreSQL port | `nil` |
| `externalDatabase.existingSecret` | Name of an existing secret containing the postgres password ('db-password' key) | `nil` |
| `redis.enabled` | Switch to enable or disable the Redis helm chart | `true` |
| `externalRedis.host` | External Redis host | `nil` |
| `externalRedis.port` | External Redis port | `nil` |
| `externalRedis.password` | External Redis password | `nil` |
| `externalRedis.username` | External Redis username (not required on most Redis implementations) | `nil` |
| `externalRedis.existingSecret` | Name of an existing secret containing the Redis password ('redis-password' key) | `nil` |
| `metrics.enabled` | Start a side-car prometheus exporter | `false` |
| `metrics.image.registry` | Airflow exporter image registry | `docker.io` |
| `metrics.image.repository` | Airflow exporter image name | `bitnami/airflow-exporter` |
Expand Down Expand Up @@ -256,6 +258,41 @@ airflow.clonePluginsFromGit.branch=v1.0.9-branch
airflow.clonePluginsFromGit.path=plugins
```

### Existing Secrets

You can use an existing secret to configure your Airflow auth, external Postgres, and extern Redis passwords:

```console
postgresql.enabled=false
externalDatabase.host=my.external.postgres.host
externalDatabase.user=bn_airflow
externalDatabase.database=bitnami_airflow
externalDatabase.existingSecret=all-my-secrets

redis.enabled=false
externalRedis.host=my.external.redis.host
externalRedis.existingSecret=all-my-secrets

airflow.auth.existingSecret=all-my-secrets
```

The expected secret resource looks as follows:

```yaml
apiVersion: v1
kind: Secret
metadata:
name: all-my-secrets
type: Opaque
data:
airflow-password: "Smo1QTJLdGxXMg=="
airflow-fernetKey: "YVRZeVJVWnlXbU4wY1dOalVrdE1SV3cxWWtKeFIzWkVRVTVrVjNaTFR6WT0="
db-password: "cG9zdGdyZXMK"
Comment thread
chgl marked this conversation as resolved.
Outdated
redis-password: "cmVkaXMK"
```

This is useful if you plan on using [Bitnami's sealed secrets](https://github.com/bitnami-labs/sealed-secrets) to manage your passwords.

## Persistence

The Bitnami Airflow chart relies on the PostgreSQL chart persistence. This means that Airflow does not persist anything.
Expand Down
6 changes: 6 additions & 0 deletions bitnami/airflow/templates/deployment-scheduler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,9 @@ spec:
{{- if .Values.postgresql.enabled }}
name: {{ template "airflow.postgresql.fullname" . }}
key: postgresql-password
{{- else if .Values.externalDatabase.existingSecret}}
name: {{ .Values.externalDatabase.existingSecret }}
key: db-password
Comment thread
chgl marked this conversation as resolved.
Outdated
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
Expand Down Expand Up @@ -166,6 +169,9 @@ spec:
{{- if .Values.redis.enabled }}
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
{{- else if .Values.externalRedis.existingSecret}}
name: {{ .Values.externalRedis.existingSecret }}
key: redis-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
key: redis-password
Expand Down
6 changes: 6 additions & 0 deletions bitnami/airflow/templates/deployment-web.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,9 @@ spec:
{{- if .Values.postgresql.enabled }}
name: {{ template "airflow.postgresql.fullname" . }}
key: postgresql-password
{{- else if .Values.externalDatabase.existingSecret}}
name: {{ .Values.externalDatabase.existingSecret }}
key: db-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
Expand Down Expand Up @@ -166,6 +169,9 @@ spec:
{{- if .Values.redis.enabled }}
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
{{- else if .Values.externalRedis.existingSecret}}
name: {{ .Values.externalRedis.existingSecret }}
key: redis-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
key: redis-password
Expand Down
3 changes: 3 additions & 0 deletions bitnami/airflow/templates/metrics-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ spec:
{{- if .Values.postgresql.enabled }}
name: {{ template "airflow.postgresql.fullname" . }}
key: postgresql-password
{{- else if .Values.externalDatabase.existingSecret}}
name: {{ .Values.externalDatabase.existingSecret }}
key: db-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
Expand Down
2 changes: 2 additions & 0 deletions bitnami/airflow/templates/secret-externaldb.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{{- if not .Values.postgresql.enabled }}
{{- if not .Values.externalDatabase.existingSecret }}
Comment thread
chgl marked this conversation as resolved.
Outdated
apiVersion: v1
kind: Secret
metadata:
Expand All @@ -11,4 +12,5 @@ metadata:
type: Opaque
data:
db-password: {{ .Values.externalDatabase.password | b64enc | quote }}
{{- end }}
{{- end }}
2 changes: 2 additions & 0 deletions bitnami/airflow/templates/secret-externalredis.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
{{- if not .Values.redis.enabled }}
{{- if not .Values.externalRedis.existingSecret }}
Comment thread
chgl marked this conversation as resolved.
Outdated
apiVersion: v1
kind: Secret
metadata:
Expand All @@ -11,4 +12,5 @@ metadata:
type: Opaque
data:
redis-password: {{ .Values.externalRedis.password | b64enc | quote }}
{{- end }}
{{- end }}
6 changes: 6 additions & 0 deletions bitnami/airflow/templates/statefulset-worker.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,9 @@ spec:
{{- if .Values.postgresql.enabled }}
name: {{ template "airflow.postgresql.fullname" . }}
key: postgresql-password
{{- else if .Values.externalDatabase.existingSecret}}
name: {{ .Values.externalDatabase.existingSecret }}
key: db-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
Expand Down Expand Up @@ -168,6 +171,9 @@ spec:
{{- if .Values.redis.enabled }}
name: {{ template "airflow.redis.secretName" . }}
key: redis-password
{{- else if .Values.externalRedis.existingSecret}}
name: {{ .Values.externalRedis.existingSecret }}
key: redis-password
{{- else }}
name: {{ printf "%s-%s" .Release.Name "externalredis" }}
key: redis-password
Expand Down
4 changes: 4 additions & 0 deletions bitnami/airflow/values-production.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,8 @@ externalDatabase:
database: bitnami_airflow
## Database port number
port: 5432
## Name of an existing secret resource containing the Postgres password in a 'db-password' key
existingSecret: ""

##
## Redis chart configuration
Expand All @@ -354,6 +356,8 @@ externalRedis:
## and it should be enough with the password
##
# username: ""
## Name of an existing secret resource containing the Redis password in a 'redis-password' key
existingSecret: ""

## Prometheus Exporter / Metrics
##
Expand Down
5 changes: 5 additions & 0 deletions bitnami/airflow/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,8 @@ externalDatabase:
user: bn_airflow
## Database password
password: ""
## Name of an existing secret resource containing the DB password in a 'db-password' key
existingSecret: ""
## Database name
database: bitnami_airflow
## Database port number
Expand All @@ -349,6 +351,9 @@ externalRedis:
## Redis password
##
password: ""
## Name of an existing secret resource containing the Redis password in a 'redis-password' key
##
existingSecret: ""
## Redis username
## Most Redis implementnations do not require a username to authenticate
## and it should be enough with the password
Expand Down