Connect EventTracker to TB endpoint #7240

anamarn · 2024-09-24T15:31:12Z

#7091
EventTrackers send information of events to the TinyBird instance:

In order to test:

Set ANALYTICS_ENABLED= true and TELEMETRY_ENABLED=true in evironment-variables.ts
Set the TINYBIRD_TOKEN in environment variables (go to TiniyBird Tokens)
Log in to twenty's TinyBird and go to datasources/analytics_events in twenty_analytics workspace
Run twenty and navigate it
New events will be logged in the datasources, containing their timestamp, sessionId and payload.

Example of payload when user is not logged in

{"hostName":"localhost",
"pathname":"/welcome",
"locale":"en-US",
"userAgent":"Mozilla/5.0",
"href":"http://localhost:3001/welcome",
"referrer":"",
"timeZone":"Europe/Barcelona"}

Example of payload when user is logged in

{"userId":"2020202",
"workspaceId":"202",
"workspaceDisplayName":"Apple",
"workspaceDomainName":"apple.dev",
"hostName":"localhost",
"pathname":"/objects/companies",
"locale":"en-US",
"userAgent":"Mozilla/5.0Chrome/128.0.0.0Safari/537.36",
"href":"http://localhost:3001/objects/companies",
"referrer":"",
"timeZone":"Europe/Paris"}

…eat/analytics-v1

greptile-apps

PR Summary

This PR implements a new analytics system using TinyBird, replacing the previous Cloudflare-based solution. The changes focus on improving data collection and separating cloud analytics from self-hosted telemetry.

Added sessionId to analytics tracking in both frontend and backend
Introduced ANALYTICS_ENABLED and TINYBIRD_TOKEN environment variables
Updated AnalyticsService to use TinyBird's API for event tracking
Removed telemetry field from ClientConfig and related components
Modified PageChangeEffect to collect more detailed event data

_{22 file(s) reviewed, 5 comment(s)}
_{Edit PR Review Bot Settings}

greptile-apps · 2024-09-24T15:32:14Z

packages/twenty-front/src/effect-components/PageChangeEffect.tsx

+        userAgent: window.navigator.userAgent,
+        href: window.location.href,
+        referrer: document.referrer,
+        timeZone: Intl.DateTimeFormat().resolvedOptions().timeZone,
      });
    }, 500);


style: Consider removing or explaining the 500ms delay

greptile-apps · 2024-09-24T15:37:43Z

packages/twenty-front/src/modules/analytics/hooks/useEventTracker.ts

+export const setSessionId = (cookieName: string, domain?: string): void => {
+  const sessionId = getSessionId(cookieName) || crypto.randomUUID();
+  const baseCookie = `${cookieName}=${sessionId}; Max-Age=1800; path=/; secure`;
+  const cookie = domain ? baseCookie + `; domain=${domain}` : baseCookie;
+
+  document.cookie = cookie;
+};


logic: Ensure cookie is set with appropriate security flags (HttpOnly, SameSite) to mitigate XSS and CSRF risks

greptile-apps · 2024-09-24T15:38:59Z

packages/twenty-front/src/modules/apollo/services/__tests__/apollo.factory.test.ts

+      mutation Track($type: String!, $sessionId: String!, $data: JSON!) {
+        track(type: $type, sessionId: $sessionId, data: $data) {


logic: Track mutation updated to include sessionId, but test data not provided

greptile-apps · 2024-09-24T15:44:21Z

packages/twenty-server/src/engine/core-modules/analytics/analytics.service.ts

+    if (!this.environmentService.get('ANALYTICS_ENABLED')) {
+      return { success: true };
+    }


style: Consider combining ANALYTICS_ENABLED and TELEMETRY_ENABLED checks to simplify logic

packages/twenty-server/src/modules/messaging/monitoring/services/messaging-telemetry.service.ts

packages/twenty-front/src/effect-components/PageChangeEffect.tsx

FelixMalfait · 2024-09-25T08:19:43Z

packages/twenty-front/src/modules/analytics/hooks/useEventTracker.ts

+    (eventType: string, sessionId: string, eventData: EventData) => {
+      createEventMutation({
+        variables: {
+          type: eventType,


why not copy the payload from the example? Here we send pageview event from the frontend but you'll want to track other types of event

packages/twenty-server/src/engine/core-modules/analytics/analytics.module.ts

FelixMalfait · 2024-09-25T08:22:49Z

packages/twenty-server/src/engine/core-modules/analytics/analytics.service.ts

+        version: '1',
+        session_id: createEventInput.sessionId,
+        payload: {
+          userId: userId,


If data within the payload is in camelCase (which make sense, it's what we usually use) then you should adapt your tinybird data model to also be in snakeCase (sessionId not session_id)

FelixMalfait · 2024-09-25T08:28:46Z

.../twenty-server/src/engine/api/graphql/workspace-query-runner/listeners/telemetry.listener.ts

@@ -25,6 +25,7 @@ export class TelemetryListener {
            data: {


remove type: track and pass action instead with payload.name as its value? eventName isn't part of your schema is it?

FelixMalfait · 2024-09-25T08:32:43Z

.../twenty-server/src/engine/api/graphql/workspace-query-runner/listeners/telemetry.listener.ts

@@ -48,6 +49,7 @@ export class TelemetryListener {
            data: {
              eventName: 'user.signup',


should this be passed as action?

FelixMalfait · 2024-09-25T08:35:19Z

.../twenty-server/src/engine/api/graphql/workspace-query-runner/listeners/telemetry.listener.ts

@@ -48,6 +49,7 @@ export class TelemetryListener {
            data: {
              eventName: 'user.signup',
            },
+            sessionId: '',


We could maybe create a @Session decorator to get the session during signup (from the cookie) and then pass it down to the backend events, the same way we pass userId / workspaceId?

We can push this to a later PR

FelixMalfait · 2024-09-25T08:47:28Z

packages/twenty-server/src/engine/core-modules/analytics/analytics.service.ts

  data: object;
 };

 @Injectable()
 export class AnalyticsService {
  private readonly logger = new Logger(AnalyticsService.name);
+  private readonly datasource = 'analytics_events';


Maybe also switch everything to camelCase and singular? That's our current convention in our Postgres database

Also I see in Tinybird a table with the singular name has already been created, is this used for testing? you can create branch environments for testing if you need to

github-actions · 2024-09-26T08:54:29Z

Thanks @anamarn for your contribution!
This marks your 3rd PR on the repo. You're top 12% of all our contributors 🎉
See contributor page - Share on LinkedIn - Share on Twitter

anamarn added 6 commits September 20, 2024 11:04

Cleanup telemetrystate

49b540a

connect eventTracker to TB

df57cf6

Cleanup telemetrystate

8963c69

connect eventTracker to TB

15c8047

Merge branch 'main' into feat/analytics-v1

0ad5c08

Merge branch 'feat/analytics-v1' of github.com:twentyhq/twenty into f…

158cc41

…eat/analytics-v1

anamarn requested a review from FelixMalfait September 24, 2024 15:31

anamarn added the -PR: awaiting review label Sep 24, 2024

greptile-apps bot reviewed Sep 24, 2024

View reviewed changes

anamarn added 3 commits September 24, 2024 17:54

correct test

6a50cea

Merge branch 'main' into feat/analytics-v1

5e4d428

correct logic in analytics service

ea6c3a1

FelixMalfait reviewed Sep 25, 2024

View reviewed changes

packages/twenty-front/src/effect-components/PageChangeEffect.tsx Outdated Show resolved Hide resolved

FelixMalfait reviewed Sep 25, 2024

View reviewed changes

packages/twenty-front/src/effect-components/PageChangeEffect.tsx Show resolved Hide resolved

FelixMalfait reviewed Sep 25, 2024

View reviewed changes

packages/twenty-server/src/engine/core-modules/analytics/analytics.module.ts Outdated Show resolved Hide resolved

FelixMalfait reviewed Sep 25, 2024

View reviewed changes

anamarn and others added 3 commits September 25, 2024 15:03

correct some conflicts

2e4b055

Change API

b11ce73

Fix tests

9023a2f

FelixMalfait force-pushed the feat/analytics-v1 branch from 534b37c to 9023a2f Compare September 25, 2024 14:48

FelixMalfait added 2 commits September 25, 2024 18:22

Bring back telemetry with 100x less events than before

43753bc

Merge branch 'main' into feat/analytics-v1

a2c8e5f

FelixMalfait merged commit 16bb1f2 into main Sep 26, 2024
11 of 16 checks passed

FelixMalfait deleted the feat/analytics-v1 branch September 26, 2024 08:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connect EventTracker to TB endpoint #7240

Connect EventTracker to TB endpoint #7240

anamarn commented Sep 24, 2024

greptile-apps bot left a comment

greptile-apps bot Sep 24, 2024

greptile-apps bot Sep 24, 2024

greptile-apps bot Sep 24, 2024

greptile-apps bot Sep 24, 2024

FelixMalfait Sep 25, 2024

FelixMalfait Sep 25, 2024

FelixMalfait Sep 25, 2024

FelixMalfait Sep 25, 2024

FelixMalfait Sep 25, 2024

FelixMalfait Sep 25, 2024

FelixMalfait Sep 25, 2024

FelixMalfait Sep 25, 2024

github-actions bot commented Sep 26, 2024

		mutation Track($type: String!, $sessionId: String!, $data: JSON!) {
		track(type: $type, sessionId: $sessionId, data: $data) {

		@@ -48,6 +49,7 @@ export class TelemetryListener {
		data: {
		eventName: 'user.signup',

Connect EventTracker to TB endpoint #7240

Connect EventTracker to TB endpoint #7240

Conversation

anamarn commented Sep 24, 2024

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

greptile-apps bot Sep 24, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 24, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 24, 2024

Choose a reason for hiding this comment

greptile-apps bot Sep 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 26, 2024