Merge pull request #16 from mohamad-liyaghi/tracing

Tracing

Author: mohamad-liyaghi

README.md

@@ -6,6 +6,7 @@
 - [Monitoring](#monitoring)
   - [Node Exporter](#node-exporter)
   - [Loki](#loki)
+  - [OpenTelemetry](#opentelemetry)
 - [Application](#application)
   - [User](#user)
   - [Active Session](#active-session)
@@ -37,12 +38,14 @@ Vendors can easily register their restaurants, manage their menus, and add produ
   <img style="height:30px;" alt="prometheus" src="https://img.shields.io/badge/Prometheus-E6522C.svg?style=flat&logo=prometheus&logoColor=white">
   <img style="height:30px;" alt="loki" src="https://img.shields.io/badge/Loki-3B2E5A.svg?style=flat&logo=loki&logoColor=white">
   <img style="height:30px;" alt="pytest" src="https://img.shields.io/badge/Pytest-0A9EDC.svg?style=flat&logo=pytest&logoColor=white">
+  <img style="height:30px;" alt="jaeger" src="https://img.shields.io/badge/Jaeger-FF6F00.svg?style=flat&logo=jaeger&logoColor=white">
+  <img style="height:30px;" alt="opentelemetry" src="https://img.shields.io/badge/OpenTelemetry-FF6F00.svg?style=flat&logo=opentelemetry&logoColor=white">
 </div>
 <br>
 
 This project uses <a href="https://docs.djangoproject.com/">Django</a> as the backend framework, with <a href="https://www.postgresql.org/">PostgreSQL</a> and <a href="https://postgis.net/">PostGIS</a> for database management and geographic data handling. The application is containerized using <a href="https://www.docker.com/">Docker</a> and orchestrated with <a href="https://kubernetes.io/">Kubernetes</a>.
 
-For background task processing, <a href="https://docs.celeryq.dev/en/stable/">Celery</a> is used, with <a href="https://redis.io/">Redis</a> serving as both the caching layer and the message broker. Monitoring and logging are handled with <a href="https://grafana.com/">Grafana</a>, <a href="https://prometheus.io/docs/guides/node-exporter/">Node Exporter</a>, <a href="https://grafana.com/docs/loki/latest/getting-started/promtail/">Promtail</a>, and <a href="https://grafana.com/docs/loki/latest/">Loki</a>.
+For background task processing, <a href="https://docs.celeryq.dev/en/stable/">Celery</a> is used, with <a href="https://redis.io/">Redis</a> serving as both the caching layer and the message broker. Monitoring and logging are handled with <a href="https://grafana.com/">Grafana</a>, <a href="https://prometheus.io/docs/guides/node-exporter/">Node Exporter</a>, <a href="https://grafana.com/docs/loki/latest/getting-started/promtail/">Promtail</a>, and <a href="https://grafana.com/docs/loki/latest/">Loki</a>, Also <a href="https://www.jaegertracing.io/">Jaeger</a> is used for distributed tracing.
 
 Testing is managed with <a href="https://docs.pytest.org/en/stable/">pytest</a> to ensure code quality and reliability.
 
@@ -62,6 +65,16 @@ Loki is a log aggregation system that is used to store and query logs from the a
 Here is an example of the Loki dashboard in Grafana:
 <img src="./images/loki.png" alt="Loki"/>
 
+### OpenTelemetry
+This Project uses OpenTelemetry as its distributed tracing system. Jaeger is used to collect, store, and visualize traces from the application. It provides insights into the performance of the application and helps identify bottlenecks and issues.
+
+Here is an example of the Jaeger dashboard:
+<img src="./images/jaeger.png" alt="Jaeger"/>
+
+Here is an example of the metrics in grafana:
+<img src="./images/opentelemetry_metrics.png" alt="Open Telemetry"/>
+
+
 ## Application
 To make the application easier to understand, we have divided it into several components. Each component is responsible for a specific part of the application. The components are as follows:
 

config/settings/core.py

@@ -109,6 +109,10 @@
             "format": "{levelname} {message}",
             "style": "{",
         },
+        "trace_formatter": {
+            "format": "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s] [%(funcName)s] %(message)s",  # noqa
+            "datefmt": "%Y-%m-%d %H:%M:%S",
+        },
     },
     "handlers": {
         "file_info": {

docker-compose.base.yaml

@@ -145,6 +145,16 @@ services:
     volumes:
       - food_anywhere_grafana_data:/var/lib/grafana
 
+  jaeger:
+    image: jaegertracing/all-in-one:1.32
+    container_name: food-anywhere-jaeger
+    ports:
+      - 16686:16686
+    networks:
+      - main
+    volumes:
+      - food_anywhere_jaeger_data:/var/lib/jaeger
+
 
 networks:
   main:

docker-compose.prod.yaml

@@ -60,6 +60,10 @@ services:
     extends:
       service: prometheus
       file: docker-compose.base.yaml
+  jaeger:
+    extends:
+      service: jaeger
+      file: docker-compose.base.yaml
 
 networks:
   main:
@@ -69,3 +73,4 @@ volumes:
   food_anywhere_grafana_data:
   food_anywhere_node_exporter_data:
   food_anywhere_prometheus_data:
+  food_anywhere_jaeger_data:

docker-compose.yaml

@@ -70,6 +70,10 @@ services:
     extends:
       service: prometheus
       file: docker-compose.base.yaml
+  jaeger:
+    extends:
+      service: jaeger
+      file: docker-compose.base.yaml
 
 networks:
   main:
@@ -79,3 +83,4 @@ volumes:
   food_anywhere_grafana_data:
   food_anywhere_node_exporter_data:
   food_anywhere_prometheus_data:
+  food_anywhere_jaeger_data:

envs/.env.local

@@ -23,5 +23,9 @@ EMAIL_FROM='[email protected]'
 CELERY_BROKER_URL=redis://redis:6379/1
 CELERY_RESULT_BACKEND=redis://redis:6379/1
 
+# Jaeger
+TRACING_HOST="jaeger"
+TRACING_PORT=6831
+
 # Cache Keys
 CART_CACHE_KEY='cart:{user_id}:{product_id}'

envs/.env.prod

@@ -23,5 +23,9 @@ EMAIL_FROM='[email protected]'
 CELERY_BROKER_URL=redis://redis:6379/1
 CELERY_RESULT_BACKEND=redis://redis:6379/1
 
+# Jaeger
+TRACING_HOST="jaeger"
+TRACING_PORT=6831
+
 # Cache Keys
 CART_CACHE_KEY='cart:{user_id}:{product_id}'

images/jaeger.png

@@ -0,0 +1,33 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: food-anywhere-jaeger-dp
+  labels:
+    app: food-anywhere-jaeger
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: food-anywhere-jaeger
+  template:
+    metadata:
+      labels:
+        app: food-anywhere-jaeger
+    spec:
+      containers:
+        - name: jaeger
+          image: jaegertracing/all-in-one:1.32
+          ports:
+            - containerPort: 16686
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: jaeger
+spec:
+  selector:
+    app: food-anywhere-jaeger
+  ports:
+    - protocol: TCP
+      port: 16686
+      targetPort: 16686

images/opentelemetry_metrics.png

@@ -2,11 +2,30 @@
 """Django's command-line utility for administrative tasks."""
 import os
 import sys
+from opentelemetry.exporter.jaeger.thrift import JaegerExporter
+from opentelemetry.instrumentation.django import DjangoInstrumentor
+from opentelemetry.instrumentation.logging import LoggingInstrumentor
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.sdk.resources import SERVICE_NAME, Resource
+from opentelemetry import trace
 
 
 def main():
     """Run administrative tasks."""
     os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
+
+    DjangoInstrumentor().instrument()
+    LoggingInstrumentor().instrument()
+
+    jaeger_exporter = JaegerExporter(
+        agent_host_name=os.getenv("TRACING_HOST"),
+        agent_port=int(os.getenv("TRACING_PORT")),
+    )
+    trace.set_tracer_provider(TracerProvider(resource=Resource.create({SERVICE_NAME: "food-anywhere-backend"})))
+    span_processor = BatchSpanProcessor(jaeger_exporter)
+    trace.get_tracer_provider().add_span_processor(span_processor)
+
     try:
         from django.core.management import execute_from_command_line
     except ImportError as exc:

kubernetes/jaeger.yaml

@@ -31,6 +31,13 @@ jsonschema-specifications==2023.12.1
 kombu==5.3.7
 mypy-extensions==1.0.0
 nodeenv==1.9.1
+opentelemetry-api==1.10.0
+opentelemetry-exporter-jaeger==1.10.0
+opentelemetry-instrumentation==0.29b0
+opentelemetry-instrumentation-django==0.29b0
+opentelemetry-instrumentation-logging==0.29b0
+opentelemetry-instrumentation-requests==0.29b0
+opentelemetry-sdk==1.10.0
 packaging==24.1
 pathspec==0.12.1
 platformdirs==4.2.2
@@ -48,6 +55,7 @@ python-decouple==3.8
 PyYAML==6.0.1
 redis==5.0.8
 referencing==0.35.1
+requests==2.32.3
 rpds-py==0.19.1
 six==1.16.0
 sqlparse==0.5.1