Skip to content

Support telemetry during DAG parsing emitting data to Scarf#250

Merged
tatiana merged 13 commits into
mainfrom
issue-214
Oct 17, 2024
Merged

Support telemetry during DAG parsing emitting data to Scarf#250
tatiana merged 13 commits into
mainfrom
issue-214

Conversation

@tatiana
Copy link
Copy Markdown
Collaborator

@tatiana tatiana commented Oct 15, 2024 •
edited
Loading

Export telemetry related to DAG Factory usage to Scarf.

This data assists the project maintainers in better understanding how DAG Factory is used. Insights from this telemetry are critical for prioritizing patches, minor releases, and security fixes. Additionally, this information supports critical decisions related to the development road map.

Deployments and individual users can opt out of analytics by setting the configuration:

[dag_factory] enable_telemetry False

As described in the official documentation, it is also possible to opt-out by setting one of the following environment variables:

AIRFLOW__DAG_FACTORY__ENABLE_TELEMETRY=False
DO_NOT_TRACK=True
SCARF_NO_ANALYTICS=True

In addition to Scarf's default data collection, DAG Factory collects the following information:

  • DAG Factory version
  • Airflow version
  • Python version
  • Operating system & machine architecture
  • Event type
  • Number of DAGs
  • Number of TaskGroups
  • Number of Tasks

No user-identifiable information (IP included) is stored in Scarf, even though Scarf infers information from the IP, such as location, and stores that. The data collection is GDPR compliant.

The data is not currently being emitted for pre-releases except from integration tests.

The Apache Foundation supports this same strategy in many of its OpenSource projects, including Airflow (#39510).

Example of visualisation of the data via the Scarf UI:

Screenshot 2024-10-17 at 01 56 09 Screenshot 2024-10-17 at 01 55 59 Screenshot 2024-10-17 at 01 55 47 Screenshot 2024-10-17 at 01 55 42

Closes: #214

@tatiana tatiana mentioned this pull request Oct 15, 2024
Closed
2 tasks
@tatiana tatiana changed the title Initial implementation of telemtry using Scarf in DAG Factory Support telemetry using Scarf Oct 17, 2024
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Oct 17, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.04%. Comparing base (6f31846) to head (ad1063a).
Report is 1 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #250      +/-   ##
==========================================
+ Coverage   87.71%   89.04%   +1.32%     
==========================================
  Files           6        8       +2     
  Lines         586      648      +62     
==========================================
+ Hits          514      577      +63     
+ Misses         72       71       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tatiana tatiana marked this pull request as ready for review October 17, 2024 00:48
pankajkoti
pankajkoti approved these changes Oct 17, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me. Just have minor questions inline about the taskgroups and tasks count.

Comment thread dagfactory/dagfactory.py Outdated
Comment thread dagfactory/dagfactory.py Outdated
@tatiana tatiana merged commit 01e0871 into main Oct 17, 2024
@tatiana tatiana deleted the issue-214 branch October 17, 2024 08:25
@tatiana tatiana added this to the DAG Factory 0.20.0 milestone Oct 17, 2024
@tatiana tatiana changed the title Support telemetry using Scarf Support telemetry during DAG parsing emitting data to Scarf Oct 17, 2024
This was referenced Oct 17, 2024
Merged
Closed
tatiana added a commit that referenced this pull request Oct 22, 2024
@tatiana
Release 0.20.0 (#254)
d246e8c 
### Added
- Support using envvar in config YAML by @tatiana in #236
- **Callback improvements**
- Support installed code via python callable string by @john-drews in
#221
- Add `callback_file` & `callback_name` to `default_args` DAG level by
@subbota19 in #218
- Cast callbacks to functions when set with `default_args` on TaskGroups
by @baraldo and @pankajastro in #235

- **Telemetry**
- For more information, please, read the [Privacy
Notice](https://github.com/astronomer/dag-factory/blob/main/PRIVACY_NOTICE.md#collection-of-data).
  - Add scarf to readme for website analytics by @cmarteepants in #219
- Support telemetry during DAG parsing emitting data to Scarf by
@tatiana in #250.

### Fixed
- Build DAGs when tehre is an invalid YAML in the DAGs folder by @quydx
and @tatiana in #184

### Others
- Development tools
  - Fix make docker-run by @pankajkoti in #249
  - Add vim dot files to .gitignore by @tatiana in #228
  - Use Hatchling to modern package building by @kaxil in #208
- CI
  - Fix static check failures in PR #218 by @pankajkoti in #251
  - Fix pre-commit checks by @tatiana in #247
  - Remove tox and corresponding build jobs in CI by @pankajkoti in #248
- Install Airflow with different versions in the CI by @pankajkoti in
#237
  - Run pre-commit hooks on all existing files by @pankajkoti in #245
  - Add Python 3.11 and 3.12 to CI test pipeline by @pankajkoti in #229
- Tests
  - Fix duplicate test name by @pankajastro in #234
  - Add static check by @pankajastro in #231
  - Fix running tests locally (outside the CI) by @tatiana in #227
  - Add the task_2 back to dataset example by @cmarteepants in #204
  - Remove unnecessary config line by @jlaneve in #202
- Documentation
  - Update the license from MIT to Apache 2.0 by @pankajastro in #191
- Add registration icon and links to Airflow references by @cmarteepants
in #190
  - Update quickstart and add feature examples by @cmarteepants #189

### Breaking changes
- Removed support for Python 3.7
- The license was changed from MIT to Apache 2.0

Closes: #217
@tatiana tatiana mentioned this pull request Nov 6, 2024
Closed
@pankajastro pankajastro mentioned this pull request Nov 27, 2024
Merged
pankajastro added a commit that referenced this pull request Dec 3, 2024
@pankajastro
Refactor telemetry to collect events during DAG run and not during DA…
72bc85b 
…G parsing (#300)

DAG Factory 0.20 started collecting telemetry as part of the PR
#250. However, one
limitation of this initial implementation is that it emitted telemetry
every time DAGs were parsed. This means that the data collected did not
represent the actual usage and was proportional to the number of times a
DAG was parsed. This PR aims to address this limitation by changing DAG
Factory to emit telemetry during DAG runs.

This implementation leverages Airflow listeners to only emit events
after a Factory-Built DAG is run.

Closes: #282

With this data, we can get the following insight
- Number of failed DagRuns
- Number of successful DagRuns
- Total tasks associated to each DagRun
- DagRun hash

**Airflow Version**
<img width="1380" alt="Screenshot 2024-12-03 at 8 00 14 PM"
src="https://github.com/user-attachments/assets/b67968a2-171a-4f4b-83cc-e3bba13ef35b">

**DAG Hash**
<img width="1386" alt="Screenshot 2024-12-03 at 8 01 28 PM"
src="https://github.com/user-attachments/assets/fea64d40-7e71-4714-b0e4-2fdb09331962">

**DAG Factory Version**
<img width="1382" alt="Screenshot 2024-12-03 at 8 02 08 PM"
src="https://github.com/user-attachments/assets/4b9de161-32f3-4c85-a740-814d86526f60">

**Event Type**
<img width="1392" alt="Screenshot 2024-12-03 at 8 02 46 PM"
src="https://github.com/user-attachments/assets/e7a5b795-f54a-4d12-9676-274b98584b4a">

**Platform Machine**
<img width="1384" alt="Screenshot 2024-12-03 at 8 03 18 PM"
src="https://github.com/user-attachments/assets/35ede730-e87a-4d6e-acf9-47e34147e331">

**Platform System**
<img width="1380" alt="Screenshot 2024-12-03 at 8 04 23 PM"
src="https://github.com/user-attachments/assets/9f59dc92-51ae-4c9d-9958-1cc36c4a9149">

**Python Version**
<img width="1389" alt="Screenshot 2024-12-03 at 8 05 05 PM"
src="https://github.com/user-attachments/assets/085f51ad-1e0a-4785-9744-a0b713d9a267">


**DAG Run Status**
<img width="1408" alt="Screenshot 2024-12-03 at 8 06 45 PM"
src="https://github.com/user-attachments/assets/1c47bb61-cfeb-49d9-9d93-5a43240c9b51">

**Task Count in DAG run**
<img width="1394" alt="Screenshot 2024-12-03 at 8 07 25 PM"
src="https://github.com/user-attachments/assets/22c011de-34b4-48cb-8c09-78571d3b6229">

** Telemetry Version**
<img width="1395" alt="Screenshot 2024-12-03 at 8 07 58 PM"
src="https://github.com/user-attachments/assets/4c2d6006-72c1-47a2-9cde-6802923dedba">
@tatiana tatiana mentioned this pull request Aug 5, 2025
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pankajkoti pankajkoti pankajkoti approved these changes

@pankajastro pankajastro Awaiting requested review from pankajastro

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

DAG Factory 0.20.0

Development

Successfully merging this pull request may close these issues.

[Feature] Add Scarf-based Telemetry to DAG-Factory

3 participants

@tatiana @codecov-commenter @pankajkoti