DBT model instrumentation #11268

sfc-gh-vguttha · 2025-02-03T11:02:17Z

OpenTelemetry API: The API defines the interface for how to instrument your code. It specifies the functions and methods you use to generate telemetry data (traces, metrics, and logs). It's essentially a set of rules and specifications that dictate how instrumentation should be implemented. The API is no-op by default unless an SDK is configured.

OpenTelemetry SDK: The SDK provides the implementation of the API. It contains the actual code that collects, processes, and exports the telemetry data generated by the API. It handles the heavy lifting of managing telemetry data, such as sampling, buffering, and exporting it to observability backends.

Problem

Solution

Checklist

I have read the contributing guide and understand what's expected of me.
I have run this code in development, and it appears to resolve the stated issue.
This PR includes tests, or tests are not required or relevant for this PR.
This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
This PR includes type annotations for new and modified functions.

cla-bot · 2025-02-03T11:02:24Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

github-actions · 2025-02-03T11:02:32Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

cla-bot · 2025-02-17T08:30:53Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

cla-bot · 2025-02-17T08:58:12Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

dev-requirements.txt

core/dbt/task/run.py

core/dbt/task/runnable.py

sfc-gh-bdrutu

Consider to add tests, for that you can use the in memory SpanProcessor, make sure if you do that to use the SDK as a test dependency only.

core/dbt/task/runnable.py

cla-bot · 2025-02-18T10:06:49Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

cla-bot · 2025-02-18T15:00:04Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

sfc-gh-vguttha · 2025-02-18T15:05:18Z

dev-requirements.txt

+git+https://github.com/sfc-gh-vguttha/dbt-adapters.git@vguttha-add-telemetry#subdirectory=dbt-adapters
 git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-tests-adapter
 git+https://github.com/dbt-labs/dbt-common.git@main
 git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-postgres
+git+https://github.com/sfc-gh-vguttha/dbt-adapters.git@vguttha-add-telemetry#subdirectory=dbt-snowflake


FYI: will revert these changes once I am fully complete with my testing.

cla-bot · 2025-02-20T09:23:10Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

cla-bot · 2025-02-20T10:08:59Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

bogdandrutu

Please add the following description:

**OpenTelemetry API:** The API defines the interface for how to instrument your code. It specifies the functions and methods you use to generate telemetry data (traces, metrics, and logs). It's essentially a set of rules and specifications that dictate how instrumentation should be implemented. The API is no-op by default unless an SDK is configured.

**OpenTelemetry SDK:** The SDK provides the implementation of the API. It contains the actual code that collects, processes, and exports the telemetry data generated by the API. It handles the heavy lifting of managing telemetry data, such as sampling, buffering, and exporting it to observability backends.

bogdandrutu · 2025-02-25T18:01:36Z

core/dbt/task/runnable.py

@@ -242,10 +261,16 @@ def call_runner(self, runner: BaseRunner) -> RunResult:
                result = None
                thread_exception = e
            finally:
+                if result.status in (NodeStatus.Error, NodeStatus.Fail, NodeStatus.PartialSuccess):
+                    node_span.set_status(StatusCode.ERROR)
+                node_span.set_attribute("node.status", result.status.value)


Add this as the description for the status?

bogdandrutu · 2025-02-25T18:03:02Z

core/dbt/task/runnable.py

@@ -242,10 +261,16 @@ def call_runner(self, runner: BaseRunner) -> RunResult:
                result = None
                thread_exception = e
            finally:


In Java I know that the resource (in try-with-resource pattern) is closed before calling finally (would be the span in this case). Is this the case in Python as well?

but this try-catch-finally is different inner scope within with block correct?

bogdandrutu · 2025-02-25T18:04:34Z

core/dbt/task/runnable.py

                if result is not None:
                    fire_event(
                        NodeFinished(
-                            node_info=runner.node.node_info,
+                            node_info=node_info,


Why this change?

I had to refer some attributes in node_info at different place and using full name seems lengthy and difficult to read hence shortened it.

bogdandrutu · 2025-02-25T18:04:41Z

core/dbt/task/runnable.py

@@ -256,7 +281,7 @@ def call_runner(self, runner: BaseRunner) -> RunResult:
                        GenericExceptionOnRun(
                            unique_id=runner.node.unique_id,
                            exc=str(thread_exception),
-                            node_info=runner.node.node_info,
+                            node_info=node_info,


core/dbt/task/run.py

core/dbt/task/runnable.py

cla-bot · 2025-02-26T16:54:38Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

cla-bot · 2025-02-26T17:49:34Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

cla-bot · 2025-02-26T19:03:03Z

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @sfc-gh-vguttha

DBT model instrumentation

75ab71e

sfc-gh-vguttha requested a review from a team as a code owner February 3, 2025 11:02

github-actions bot added the community This PR is from a community member label Feb 3, 2025

sfc-gh-vguttha added 2 commits February 4, 2025 15:19

Set span status before closing.

80a9602

Add more instrumentation

cb2405a

sfc-gh-vguttha marked this pull request as draft February 17, 2025 08:36

refactor

de4582e

sfc-gh-bdrutu reviewed Feb 17, 2025

View reviewed changes

core/dbt/task/runnable.py Show resolved Hide resolved

address comments

e444f33

Add unit and functional tests

b61494e

sfc-gh-vguttha requested a review from sfc-gh-bdrutu February 18, 2025 15:00

sfc-gh-vguttha commented Feb 18, 2025

View reviewed changes

add metadata span

f0e0320

add more attributes

7b7808a

bogdandrutu reviewed Feb 25, 2025

View reviewed changes

bogdandrutu approved these changes Feb 25, 2025

View reviewed changes

refactor

c1954ab

sfc-gh-vguttha added 2 commits February 26, 2025 22:26

refactor

d16e8a9

refactor

fcbf71f

refactor

8f21d65

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DBT model instrumentation #11268

DBT model instrumentation #11268

sfc-gh-vguttha commented Feb 3, 2025 •

edited

Loading

cla-bot bot commented Feb 3, 2025

github-actions bot commented Feb 3, 2025

cla-bot bot commented Feb 17, 2025

cla-bot bot commented Feb 17, 2025

sfc-gh-bdrutu left a comment

cla-bot bot commented Feb 18, 2025

cla-bot bot commented Feb 18, 2025

sfc-gh-vguttha Feb 18, 2025

cla-bot bot commented Feb 20, 2025

cla-bot bot commented Feb 20, 2025

bogdandrutu left a comment

bogdandrutu Feb 25, 2025

bogdandrutu Feb 25, 2025

sfc-gh-vguttha Feb 26, 2025

bogdandrutu Feb 25, 2025

sfc-gh-vguttha Feb 26, 2025

bogdandrutu Feb 25, 2025

cla-bot bot commented Feb 26, 2025

cla-bot bot commented Feb 26, 2025

cla-bot bot commented Feb 26, 2025

DBT model instrumentation #11268

Are you sure you want to change the base?

DBT model instrumentation #11268

Conversation

sfc-gh-vguttha commented Feb 3, 2025 • edited Loading

Problem

Solution

Checklist

cla-bot bot commented Feb 3, 2025

github-actions bot commented Feb 3, 2025

cla-bot bot commented Feb 17, 2025

cla-bot bot commented Feb 17, 2025

sfc-gh-bdrutu left a comment

Choose a reason for hiding this comment

cla-bot bot commented Feb 18, 2025

cla-bot bot commented Feb 18, 2025

sfc-gh-vguttha Feb 18, 2025

Choose a reason for hiding this comment

cla-bot bot commented Feb 20, 2025

cla-bot bot commented Feb 20, 2025

bogdandrutu left a comment

Choose a reason for hiding this comment

bogdandrutu Feb 25, 2025

Choose a reason for hiding this comment

bogdandrutu Feb 25, 2025

Choose a reason for hiding this comment

sfc-gh-vguttha Feb 26, 2025

Choose a reason for hiding this comment

bogdandrutu Feb 25, 2025

Choose a reason for hiding this comment

sfc-gh-vguttha Feb 26, 2025

Choose a reason for hiding this comment

bogdandrutu Feb 25, 2025

Choose a reason for hiding this comment

cla-bot bot commented Feb 26, 2025

cla-bot bot commented Feb 26, 2025

cla-bot bot commented Feb 26, 2025

sfc-gh-vguttha commented Feb 3, 2025 •

edited

Loading