Improve analytics

[FelixMalfait]

Problems

• We collect too many data points for self-hosted telemetry which we don't need, this is because self-hosting telemetry is coupled with cloud analytics while they should be two separate concerns
• That data is pushed to Cloudflare -> Jitsu -> Redshift which is highly inefficient
• We don't have proper analytics for our cloud version
• We don't have proper analytics for self-hosting usage
• We don't give workspace owners any way to track their activity/usage (api calls, etc.)
• Bonus: be able to run as an "open startup" long-term, sharing our metrics publicly

What

• Replace the Cloudflare endpoint with a Tinybird endpoint in the Analytics service

  • Introduce a variable IS_ANALYTICS_ENABLED (default to false)
  • Pass that variable to the frontend, only send Track events to the backend if isAnalyticsEnabled
  • On the backend, remove the check on TELEMETRY_ENABLED and rely on IS_ANALYTICS_ENABLED instead
  • Introduce a private env variable TINYBIRD_PRIVATE_TOKEN
  • Replace t.twenty.com/api by api.tinybird.co/v0/events?name=${DATASOURCE}
  • Adapt the frontend Track script, taking inspiration from https://github.com/tinybirdco/web-analytics-starter-kit/blob/main/middleware/src/index.js ; this will change the payload format - we want to stick to TinyBird's format

• Introduce a new service dedicated to Telemetry

  • Build a simple service that sends a post request to the former Cloudflare url
  • We just want to track new users / workspaces. We'll need to get a sense of usage in the future but maybe we can collect 1/100 events or something like that... TBC
  • Next
    • Track API usage, track function usage...
    • Dashboard for each workspace to view their usage (api calls, function calls, etc.)
    • Use it for throttling / billing
    • Metabase connector?