feat(helm): Add log-ingestor deployment.
#1856
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| {{- if .Values.clpConfig.log_ingestor }} | ||
| apiVersion: "apps/v1" | ||
| kind: "Deployment" | ||
| metadata: | ||
| name: {{ include "clp.fullname" . }}-log-ingestor | ||
| labels: | ||
| {{- include "clp.labels" . | nindent 4 }} | ||
| app.kubernetes.io/component: "log-ingestor" | ||
| spec: | ||
| replicas: 1 | ||
| selector: | ||
| matchLabels: | ||
| {{- include "clp.selectorLabels" . | nindent 6 }} | ||
| app.kubernetes.io/component: "log-ingestor" | ||
| template: | ||
| metadata: | ||
| labels: | ||
| {{- include "clp.labels" . | nindent 8 }} | ||
| app.kubernetes.io/component: "log-ingestor" | ||
| spec: | ||
| serviceAccountName: {{ include "clp.fullname" . }}-job-watcher | ||
| terminationGracePeriodSeconds: 60 | ||
| securityContext: | ||
| runAsUser: {{ .Values.securityContext.firstParty.uid }} | ||
| runAsGroup: {{ .Values.securityContext.firstParty.gid }} | ||
| fsGroup: {{ .Values.securityContext.firstParty.gid }} | ||
| initContainers: | ||
| - {{- include "clp.waitFor" (dict | ||
| "root" . | ||
| "type" "job" | ||
| "name" "db-table-creator" | ||
| ) | nindent 10 }} | ||
| containers: | ||
| - name: "log-ingestor" | ||
| image: "{{ include "clp.image.ref" . }}" | ||
| imagePullPolicy: "{{ .Values.image.clpPackage.pullPolicy }}" | ||
| env: | ||
| - name: "CLP_DB_PASS" | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: {{ include "clp.fullname" . }}-database | ||
| key: "password" | ||
| - name: "CLP_DB_USER" | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: {{ include "clp.fullname" . }}-database | ||
| key: "username" | ||
| - name: "CLP_LOGS_DIR" | ||
| value: "/var/log/log_ingestor" | ||
| - name: "RUST_LOG" | ||
| value: {{ .Values.clpConfig.log_ingestor.logging_level | quote }} | ||
| ports: | ||
| - name: "log-ingestor" | ||
| containerPort: 3002 | ||
| volumeMounts: | ||
| - name: {{ include "clp.volumeName" (dict | ||
| "component_category" "log-ingestor" | ||
| "name" "logs" | ||
| ) | quote }} | ||
| mountPath: "/var/log/log_ingestor" | ||
| - name: "config" | ||
| mountPath: "/etc/clp-config.yaml" | ||
| subPath: "clp-config.yaml" | ||
| readOnly: true | ||
| command: [ | ||
| "/opt/clp/bin/log-ingestor", | ||
| "--config", "/etc/clp-config.yaml", | ||
| "--host", "0.0.0.0", | ||
| "--port", "3002" | ||
| ] | ||
| readinessProbe: | ||
| {{- include "clp.readinessProbeTimings" . | nindent 12 }} | ||
| httpGet: &log-ingestor-health-check | ||
| path: "/health" | ||
| port: "log-ingestor" | ||
| livenessProbe: | ||
| {{- include "clp.livenessProbeTimings" . | nindent 12 }} | ||
| httpGet: *log-ingestor-health-check | ||
|
Comment on lines
+34
to
+78
Contributor
There was a problem hiding this comment. 🧹 Nitpick | 🔵 Trivial Consider adding resource requests and limits. The container specification doesn't define resource requests or limits. While this works for initial deployment, consider adding them to prevent resource contention and ensure predictable performance in production environments. 📊 Example resource configurationAdd a resources section to the container specification: livenessProbe:
{{- include "clp.livenessProbeTimings" . | nindent 12 }}
httpGet: *log-ingestor-health-check
+ resources:
+ requests:
+ memory: "256Mi"
+ cpu: "100m"
+ limits:
+ memory: "512Mi"
+ cpu: "500m"
volumes:Adjust values based on actual resource requirements and testing.
🧰 Tools🪛 YAMLlint (1.37.1)[error] 41-41: too many spaces inside braces (braces) [error] 41-41: too many spaces inside braces (braces) [error] 46-46: too many spaces inside braces (braces) [error] 46-46: too many spaces inside braces (braces) [error] 51-51: too many spaces inside braces (braces) [error] 51-51: too many spaces inside braces (braces) [error] 56-56: too many spaces inside braces (braces) [error] 59-59: too many spaces inside braces (braces) [error] 72-72: too many spaces inside braces (braces) [error] 77-77: too many spaces inside braces (braces) 🤖 Prompt for AI Agents |
||
| volumes: | ||
| - {{- include "clp.pvcVolume" (dict | ||
| "root" . | ||
| "component_category" "log-ingestor" | ||
| "name" "logs" | ||
| ) | nindent 10 }} | ||
| - name: "config" | ||
| configMap: | ||
| name: {{ include "clp.fullname" . }}-config | ||
| {{- end }} | ||
|
Member
There was a problem hiding this comment. Cross-compared with
Member
Author
There was a problem hiding this comment.
if i'm not wrong, logs are dropped silently. Let's raise it to 10Gi to match the compression scheduler log volume. In the long run, we should implement #1760 instead of requesting host volumes specifically for logs. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| {{- if .Values.clpConfig.log_ingestor }} | ||
| {{- include "clp.createLocalPv" (dict | ||
| "root" . | ||
| "component_category" "log-ingestor" | ||
| "name" "logs" | ||
| "nodeRole" "control-plane" | ||
| "capacity" "10Gi" | ||
| "accessModes" (list "ReadWriteOnce") | ||
| "hostPath" (printf "%s/log_ingestor" .Values.clpConfig.logs_directory) | ||
| ) }} | ||
| {{- end }} | ||
|
Comment on lines
+1
to
+11
Contributor
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Check log-ingestor deployment for node affinity configuration
rg -A 10 'nodeSelector|affinity' tools/deployment/package-helm/templates/log-ingestor-deployment.yamlRepository: y-scope/clp Length of output: 37 🏁 Script executed: #!/bin/bash
# Check if log-ingestor deployment file exists and explore its structure
fd "log-ingestor.*deployment" tools/deployment/package-helm/templates/
echo "---"
# If found, check the content
if [ -f "tools/deployment/package-helm/templates/log-ingestor-deployment.yaml" ]; then
cat -n tools/deployment/package-helm/templates/log-ingestor-deployment.yaml
else
echo "Checking templates directory..."
ls -la tools/deployment/package-helm/templates/ | grep log-ingestor
fiRepository: y-scope/clp Length of output: 3847 Add nodeSelector or affinity to log-ingestor deployment to match PV nodeRole constraint. The PV restricts scheduling to control-plane nodes (nodeRole: "control-plane"), but the log-ingestor deployment has no nodeSelector or affinity constraints. This mismatch means the deployment can be scheduled on any node, and if it lands on a worker node, the PVC will fail to bind since the PV is unavailable there. Add a nodeSelector to the deployment spec to ensure the pod runs on control-plane nodes: spec:
template:
spec:
nodeSelector:
node-role.kubernetes.io/control-plane: ""Alternatively, use affinity with 🧰 Tools🪛 YAMLlint (1.37.1)[error] 1-1: syntax error: expected the node content, but found '-' (syntax) [error] 10-10: too many spaces inside braces (braces) [error] 11-11: too many spaces inside braces (braces) 🤖 Prompt for AI Agents |
||
|
Member
There was a problem hiding this comment. Cross-compared with |
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,9 @@ | ||||||||||||||||||||||||||||||||||||||
| {{- if .Values.clpConfig.log_ingestor }} | ||||||||||||||||||||||||||||||||||||||
| {{- include "clp.createPvc" (dict | ||||||||||||||||||||||||||||||||||||||
| "root" . | ||||||||||||||||||||||||||||||||||||||
| "component_category" "log-ingestor" | ||||||||||||||||||||||||||||||||||||||
| "name" "logs" | ||||||||||||||||||||||||||||||||||||||
| "capacity" "10Gi" | ||||||||||||||||||||||||||||||||||||||
| "accessModes" (list "ReadWriteOnce") | ||||||||||||||||||||||||||||||||||||||
| ) }} | ||||||||||||||||||||||||||||||||||||||
| {{- end }} | ||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+1
to
+9
Contributor
There was a problem hiding this comment. 🧹 Nitpick | 🔵 Trivial Consider making log storage capacity configurable. The log storage capacity is hardcoded at 5Gi (line 6). For production deployments with varying log volumes, consider making this configurable through values.yaml to allow users to adjust storage capacity based on their needs. ♻️ Suggested enhancementAdd to values.yaml under the log_ingestor block: log_ingestor: null
# port: 30302
# logs_storage_capacity: "5Gi" # Storage capacity for log-ingestor logs
# ...Then update the template: {{- include "clp.createPvc" (dict
"root" .
"component_category" "log-ingestor"
"name" "logs"
- "capacity" "5Gi"
+ "capacity" (default "5Gi" .Values.clpConfig.log_ingestor.logs_storage_capacity)
"accessModes" (list "ReadWriteOnce")
) }}Apply the same change to log-ingestor-logs-pv.yaml. 📝 Committable suggestion
Suggested change
🧰 Tools🪛 YAMLlint (1.37.1)[error] 1-1: syntax error: expected the node content, but found '-' (syntax) [error] 8-8: too many spaces inside braces (braces) [error] 9-9: too many spaces inside braces (braces) 🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||||||||||
|
Member
There was a problem hiding this comment. Cross-compared with |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| {{- if .Values.clpConfig.log_ingestor }} | ||
| apiVersion: "v1" | ||
| kind: "Service" | ||
| metadata: | ||
| name: {{ include "clp.fullname" . }}-log-ingestor | ||
| labels: | ||
| {{- include "clp.labels" . | nindent 4 }} | ||
| app.kubernetes.io/component: "log-ingestor" | ||
| spec: | ||
| type: "NodePort" | ||
| selector: | ||
| {{- include "clp.selectorLabels" . | nindent 4 }} | ||
| app.kubernetes.io/component: "log-ingestor" | ||
| ports: | ||
| - port: 3002 | ||
| targetPort: "log-ingestor" | ||
| nodePort: {{ .Values.clpConfig.log_ingestor.port }} | ||
| {{- end }} |
|
Member
There was a problem hiding this comment. Cross-compared with the current package config and this lgtm. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -90,6 +90,19 @@ clpConfig: | |
| # port: 30800 | ||
| # logging_level: "INFO" | ||
|
|
||
| # log-ingestor config. Currently, the config is applicable only if `logs_input.type` is "s3". | ||
| log_ingestor: null | ||
|
Member
Author
There was a problem hiding this comment. We should explicitly set @coderabbitai create an issue to track
Contributor
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: Length of output: 42
The issue describes how the Pydantic parser falls back to default values when 🧠 Learnings used
Member
Author
There was a problem hiding this comment. #1882 sets As a result, if the API server is enabled while More importantly, this configuration ("enable the API server but disable log_ingestor by setting I think we need to address two issues:
@coderabbitai create those new two issues as GH issues
Contributor
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: Length of output: 118
Both issues have been assigned to you and include references to PR 🐰✍️ ✏️ Learnings added
🧠 Learnings used |
||
| # port: 30302 | ||
| # # The timeout (in seconds) after which the log buffer is flushed for compression if no new | ||
| # # input arrives. | ||
| # buffer_flush_timeout: 300 | ||
| # # The log buffer size (in bytes) that triggers a flush for compression. | ||
| # buffer_flush_threshold: 268435456 # 256 MiB | ||
| # # The capacity of the internal channel used for communication between an ingestion job and the | ||
| # # log buffer. | ||
| # channel_capacity: 10 | ||
| # logging_level: "INFO" | ||
|
|
||
|
Comment on lines
+93
to
+105
Contributor
There was a problem hiding this comment. Clarify port requirement and provide validation. The service template (log-ingestor-service.yaml:17) requires Consider one of the following approaches:
🛠️ Suggested validation in service templateAdd to the beginning of log-ingestor-service.yaml: {{- if .Values.clpConfig.log_ingestor }}
+{{- if not .Values.clpConfig.log_ingestor.port }}
+{{- fail "clpConfig.log_ingestor.port is required when log_ingestor is enabled" }}
+{{- end }}
apiVersion: "v1"Or provide a default value in the service template: - nodePort: {{ .Values.clpConfig.log_ingestor.port }}
+ nodePort: {{ .Values.clpConfig.log_ingestor.port | default 30302 }}
🤖 Prompt for AI Agents |
||
| # Where archives should be output to | ||
| archive_output: | ||
| storage: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cross-compared with
api-server-deployment.yaml. The changes lgtm as: