Skip to content

[bitnami/airflow] separate resources #1951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Mauraza merged 10 commits into from
Feb 20, 2020
Merged

[bitnami/airflow] separate resources #1951

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ea93518
Add separate resources for worker, web and scheduler for airflow
Feb 19, 2020
8e89191
update chart version
Feb 19, 2020
ba1f2bd
change worker replicas to 1
Feb 19, 2020
957e8ee
change version, in readme set the default values, values.yaml and val…
Feb 19, 2020
fffb41c
Add if in resources and separate limits and requests
Feb 19, 2020
0e85557
Fix typpo in readme
Feb 19, 2020
3cd5220
Merge branch 'master' into airflow/separate_resources
Feb 20, 2020
128a5d4
Delete the old resources, delete a blank line in values-production an…
Feb 20, 2020
42f882b
Changes in values-production and values
Feb 20, 2020
21d7f7c
Merge branch 'airflow/separate_resources' of https://github.com/bitna…
Feb 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion bitnami/airflow/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v1
name: airflow
version: 4.3.4
version: 4.4.0
appVersion: 1.10.9
description: Apache Airflow is a platform to programmatically author, schedule and monitor workflows.
keywords:
Expand Down
9 changes: 7 additions & 2 deletions bitnami/airflow/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,20 +93,26 @@ The following tables lists the configurable parameters of the Airflow chart and
| `airflow.baseUrl` | URL used to access to airflow web ui | `nil` |
| `airflow.worker.port` | Airflow Worker port | `8793` |
| `airflow.worker.replicas` | Number of Airflow Worker replicas | `2` |
| `airflow.worker.resources.limits` | The resources limits for the Worker containers | `{}` |
| `airflow.worker.resources.requests` | The requested resources for the Worker containers | `{}` |
| `airflow.auth.forcePassword` | Force users to specify a password | `false` |
| `airflow.auth.username` | Username to access web UI | `user` |
| `airflow.auth.password` | Password to access web UI | `nil` |
| `airflow.auth.fernetKey` | Fernet key to secure connections | `nil` |
| `airflow.auth.existingSecret` | Name of an existing secret containing airflow password and fernet key | `nil` |
| `airflow.extraEnvVars` | Extra environment variables to add to airflow web, worker and scheduler pods | `nil` |
| `airflow.web.resources.limits` | The resources limits for the web containers | `{}` |
| `airflow.web.resources.requests` | The requested resources for the web containers | `{}` |
| `airflow.scheduler.resources.limits` | The resources limits for the scheduler containers | `{}` |
| `airflow.scheduler.resources.requests` | The requested resources for the scheduler containers | `{}` |
| `airflow.webserverConfigMap` | Config map name for ~/airflow/webserver_config.py | `nil` |
| `securityContext.enabled` | Enable security context | `true` |
| `securityContext.fsGroup` | Group ID for the container | `1001` |
| `securityContext.runAsUser` | User ID for the container | `1001` |
| `service.type` | Kubernetes Service type | `ClusterIP` |
| `service.port` | Airflow Web port | `8080` |
| `service.nodePort` | Kubernetes Service nodePort | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Airflow Service | `nil` |
| `service.loadBalancerIP` | loadBalancerIP for Airflow Service | `nil` |
| `service.annotations` | Service annotations | `` |
| `ingress.enabled` | Enable ingress controller resource | `false` |
| `ingress.certManager` | Add annotations for cert-manager | `false` |
Expand All @@ -122,7 +128,6 @@ The following tables lists the configurable parameters of the Airflow chart and
| `nodeSelector` | Node labels for pod assignment | `{}` |
| `tolerations` | Toleration labels for pod assignment | `[]` |
| `affinity` | Map of node/pod affinities | `{}` |
| `resources` | CPU/Memory resource requests/limits | Memory: `256Mi`, CPU: `250m` |
| `livenessProbe.enabled` | would you like a livessProbed to be enabled | `true` |
| `livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | 180 |
| `livenessProbe.periodSeconds` | How often to perform the probe | 20 |
Expand Down
4 changes: 3 additions & 1 deletion bitnami/airflow/templates/deployment-scheduler.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,9 @@ spec:
mountPath: /opt/bitnami/airflow/airflow.cfg
subPath: airflow.cfg
{{- end }}
resources: {{- toYaml .Values.resources | nindent 12 }}
{{- if .Values.airflow.scheduler.resources }}
resources: {{- toYaml .Values.airflow.scheduler.resources | nindent 12 }}
Comment thread
Mauraza marked this conversation as resolved.
{{- end }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
Expand Down
4 changes: 3 additions & 1 deletion bitnami/airflow/templates/deployment-web.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,9 @@ spec:
mountPath: /opt/bitnami/airflow/webserver_config.py
subPath: webserver_config.py
{{- end }}
resources: {{- toYaml .Values.resources | nindent 12 }}
{{- if .Values.airflow.web.resources }}
resources: {{- toYaml .Values.airflow.web.resources | nindent 12 }}
{{- end }}
volumes:
{{- if .Files.Glob "files/dags/*.py" }}
- name: local-dag-files
Expand Down
2 changes: 2 additions & 0 deletions bitnami/airflow/templates/metrics-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,5 +67,7 @@ spec:
ports:
- name: metrics
containerPort: 9112
{{- if .Values.metrics.resources}}
resources: {{- toYaml .Values.metrics.resources | nindent 12 }}
{{- end }}
{{- end }}
4 changes: 3 additions & 1 deletion bitnami/airflow/templates/statefulset-worker.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,9 @@ spec:
- name: airflow-worker
image: "{{ template "airflow.workerImage" . }}"
imagePullPolicy: "{{ .Values.workerImage.pullPolicy }}"
resources: {{- toYaml .Values.resources | nindent 12 }}
{{- if .Values.airflow.worker.resources }}
resources: {{- toYaml .Values.airflow.worker.resources | nindent 12 }}
{{- end }}
env:
{{- if .Values.workerImage.debug }}
- name: BASH_DEBUG
Expand Down
61 changes: 50 additions & 11 deletions bitnami/airflow/values-production.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,56 @@ airflow:
worker:
port: 8793
replicas: 3
## Worker containers' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits: {}
# cpu: 200m
# memory: 1Gi
requests: {}
# memory: 256Mi
# cpu: 250m

## Airflow web specific configuration
#
web:
## Web container' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits: {}
# cpu: 200m
# memory: 1Gi
requests: {}
# memory: 256Mi
# cpu: 250m

## Airflow scheduler specific configuration
#
scheduler:
## Scheduler containers' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits: {}
# cpu: 200m
# memory: 1Gi
requests: {}
# memory: 256Mi
# cpu: 250m

Comment thread
Mauraza marked this conversation as resolved.
## All the parameters from the configuration file can be overwritten by using environment variables with this format:
## AIRFLOW__{SECTION}__{KEY}. Note the double underscores. More info at https://airflow.readthedocs.io/en/stable/howto/set-config.html
Expand Down Expand Up @@ -276,17 +326,6 @@ tolerations: []
##
affinity: {}

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m

Comment thread
Mauraza marked this conversation as resolved.
## Configure extra options for liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
livenessProbe:
Expand Down
61 changes: 50 additions & 11 deletions bitnami/airflow/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,56 @@ airflow:
worker:
port: 8793
replicas: 1
## Worker containers' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits: {}
# cpu: 200m
# memory: 1Gi
requests: {}
# memory: 256Mi
# cpu: 250m

## Airflow web specific configuration
#
web:
## Web container' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits: {}
# cpu: 200m
# memory: 1Gi
requests: {}
# memory: 256Mi
# cpu: 250m

## Airflow scheduler specific configuration
#
scheduler:
## Scheduler containers' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits: {}
# cpu: 200m
# memory: 1Gi
requests: {}
# memory: 256Mi
# cpu: 250m

## All the parameters from the configuration file can be overwritten by using environment variables with this format:
## AIRFLOW__{SECTION}__{KEY}. Note the double underscores. More info at https://airflow.readthedocs.io/en/stable/howto/set-config.html
Expand Down Expand Up @@ -276,17 +326,6 @@ tolerations: []
##
affinity: {}

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
# limits:
# cpu: 200m
# memory: 1Gi
# requests:
# memory: 256Mi
# cpu: 250m

## Configure extra options for liveness and readiness probes
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
livenessProbe:
Expand Down