Skip to content

[bitnami/airflow] separate resources #1951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Mauraza merged 10 commits into from
Feb 20, 2020
Merged

[bitnami/airflow] separate resources #1951

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ea93518
Add separate resources for worker, web and scheduler for airflow
Feb 19, 2020
8e89191
update chart version
Feb 19, 2020
ba1f2bd
change worker replicas to 1
Feb 19, 2020
957e8ee
change version, in readme set the default values, values.yaml and val…
Feb 19, 2020
fffb41c
Add if in resources and separate limits and requests
Feb 19, 2020
0e85557
Fix typpo in readme
Feb 19, 2020
3cd5220
Merge branch 'master' into airflow/separate_resources
Feb 20, 2020
128a5d4
Delete the old resources, delete a blank line in values-production an…
Feb 20, 2020
42f882b
Changes in values-production and values
Feb 20, 2020
21d7f7c
Merge branch 'airflow/separate_resources' of https://github.com/bitna…
Feb 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion bitnami/airflow/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v1
name: airflow
version: 4.3.3
version: 4.3.4
Comment thread
Mauraza marked this conversation as resolved.
Outdated
appVersion: 1.10.9
description: Apache Airflow is a platform to programmatically author, schedule and monitor workflows.
keywords:
Expand Down
6 changes: 4 additions & 2 deletions bitnami/airflow/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,20 +93,23 @@ The following tables lists the configurable parameters of the Airflow chart and
| `airflow.baseUrl` | URL used to access to airflow web ui | `nil` |
| `airflow.worker.port` | Airflow Worker port | `8793` |
| `airflow.worker.replicas` | Number of Airflow Worker replicas | `2` |
| `airflow.worker.resources` | CPU/Memory resource requests/limits | Memory: `256Mi`, CPU: `250m` |
Comment thread
Mauraza marked this conversation as resolved.
Outdated
| `airflow.auth.forcePassword` | Force users to specify a password | `false` |
| `airflow.auth.username` | Username to access web UI | `user` |
| `airflow.auth.password` | Password to access web UI | `nil` |
| `airflow.auth.fernetKey` | Fernet key to secure connections | `nil` |
| `airflow.auth.existingSecret` | Name of an existing secret containing airflow password and fernet key | `nil` |
| `airflow.extraEnvVars` | Extra environment variables to add to airflow web, worker and scheduler pods | `nil` |
| `airflow.web.resources` | CPU/Memory resource requests/limits | Memory: `256Mi`, CPU: `250m` |
| `airflow.scheduler.resources` | CPU/Memory resource requests/limits | Memory: `256Mi`, CPU: `250m` |
| `airflow.webserverConfigMap` | Config map name for ~/airflow/webserver_config.py | `nil` |
| `securityContext.enabled` | Enable security context | `true` |
| `securityContext.fsGroup` | Group ID for the container | `1001` |
| `securityContext.runAsUser` | User ID for the container | `1001` |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.port` | Airflow Web port | `8080` |
| `service.nodePort` | Kubernetes Service nodePort | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Airflow Service | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Airflow Service | `nil` |
| `service.annotations` | Service annotations | `` |
| `ingress.enabled` | Enable ingress controller resource | `false` |
| `ingress.certManager` | Add annotations for cert-manager | `false` |
Expand All @@ -122,7 +125,6 @@ The following tables lists the configurable parameters of the Airflow chart and
| `nodeSelector` | Node labels for pod assignment | `{}` |
| `tolerations` | Toleration labels for pod assignment | `[]` |
| `affinity` | Map of node/pod affinities | `{}` |
| `resources` | CPU/Memory resource requests/limits | Memory: `256Mi`, CPU: `250m` |
| `livenessProbe.enabled` | would you like a livessProbed to be enabled | `true` |
| `livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 180 |
| `livenessProbe.periodSeconds` | How often to perform the probe | 20 |
Expand Down
2 changes: 1 addition & 1 deletion bitnami/airflow/templates/deployment-scheduler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ spec:
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
resources: {{- toYaml .Values.resources | nindent 12 }}
resources: {{- toYaml .Values.airflow.scheduler.resources | nindent 12 }}
Comment thread
Mauraza marked this conversation as resolved.
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
Expand Down
2 changes: 1 addition & 1 deletion bitnami/airflow/templates/deployment-web.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ spec:
mountPath: /opt/bitnami/airflow/webserver_config.py
subPath: webserver_config.py
{{- end }}
resources: {{- toYaml .Values.resources | nindent 12 }}
resources: {{- toYaml .Values.airflow.web.resources | nindent 12 }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
Expand Down
2 changes: 1 addition & 1 deletion bitnami/airflow/templates/statefulset-worker.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ spec:
- name: airflow-worker
image: "{{ template "airflow.workerImage" . }}"
imagePullPolicy: "{{ .Values.workerImage.pullPolicy }}"
resources: {{- toYaml .Values.resources | nindent 12 }}
resources: {{- toYaml .Values.airflow.worker.resources | nindent 12 }}
env:
{{- if .Values.workerImage.debug }}
- name: BASH_DEBUG
Expand Down
50 changes: 39 additions & 11 deletions bitnami/airflow/values-production.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,45 @@ airflow:
worker:
port: 8793
replicas: 3
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m
Comment thread
Mauraza marked this conversation as resolved.
Outdated

## Airflow web specific configuration
#
web:
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m

## Airflow scheduler specific configuration
#
scheduler:
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m

Comment thread
Mauraza marked this conversation as resolved.

## All the parameters from the configuration file can be overwritten by using environment variables with this format:
## AIRFLOW__{SECTION}__{KEY}. Note the double underscores. More info at https://airflow.readthedocs.io/en/stable/howto/set-config.html
Expand Down Expand Up @@ -276,17 +315,6 @@ tolerations: []
##
affinity: {}

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m

Comment thread
Mauraza marked this conversation as resolved.
## Configure extra options for liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
livenessProbe:
Expand Down
40 changes: 38 additions & 2 deletions bitnami/airflow/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,11 +143,47 @@ airflow:
# path:
## URL used to access to airflow web ui
##
# baseUrl:
## Airflow worker component configuration
worker:
# baseUrl:
## Airflow worker component configuration
Comment thread
Mauraza marked this conversation as resolved.
Outdated
port: 8793
replicas: 1
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m
## Airflow web specific configuration
#
web:
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m
## Airflow scheduler specific configuration
#
scheduler:
## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m

## All the parameters from the configuration file can be overwritten by using environment variables with this format:
## AIRFLOW__{SECTION}__{KEY}. Note the double underscores. More info at https://airflow.readthedocs.io/en/stable/howto/set-config.html
Expand Down