Add structured and configurable logging support to Kubeflow SDK

[szaher]

What you would like to be added?

I would like the Kubeflow SDK to include structured, configurable logging across its modules. Specifically:

Consistent use of Python’s logging library instead of print statements.

Support for different logging levels (DEBUG, INFO, WARNING, ERROR).

Ability for users to configure log formatting and destinations (stdout, file, JSON, etc.).

Clear and actionable log messages for key SDK operations (e.g., pipeline compilation, run submission, experiment creation).

Why is this needed?

Currently, the Kubeflow SDK provides limited visibility into its execution flow, making it difficult to debug and trace issues during development and runtime interactions.

Proper logging would:

• Improve developer experience by making failures and warnings more transparent.
• Help users understand the sequence of SDK operations and their outcomes.
• Enable integration with observability tools (e.g., ELK, ...etc) by supporting structured log output.
• Align the SDK with standard Python practices for enterprise-grade software, making it easier to maintain and troubleshoot.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[szaher]

SDK can expose logs via logger.debug and using Null handler to give users of the sdk the freedom to enable logs for debugging when needed.

[szaher]

This is just about adding some meaningful log messages with debug level in the sdk with null handler so I believe it's suitable for new contributors to take on.

In the SDK

# kubeflow/__init__.py
import logging

logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())

def some_function():
    logger.debug("This is an informational message from the SDK.")

Users that integrate the SDK in their libraries

import logging
from kubeflow import some_function

# The user configures logging for their application
logging.basicConfig(level=logging.DEBUG)

# Calling the SDK function will now log to the console
# The NullHandler is effectively overridden by the user's configuration
some_function()

/good-first-issue

[google-oss-prow]

@szaher:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

This is just about adding some meaningful log messages with debug level in the sdk with null handler so I believe it's suitable for new contributors to take on.

In the SDK

# kubeflow/__init__.py
import logging

logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())

def some_function():
   logger.debug("This is an informational message from the SDK.")

Users that integrate the SDK in their libraries

import logging
from kubeflow import some_function

# The user configures logging for their application
logging.basicConfig(level=logging.DEBUG)

# Calling the SDK function will now log to the console
# The NullHandler is effectively overridden by the user's configuration
some_function()

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[szaher]

@andreyvelich @astefanutti @kubeflow/kubeflow-sdk-team WDYT?

/remove-label lifecycle/needs-triage

[astefanutti]

+1 this would be a very useful addition.

Thanks @szaher!

[anencore94]

I'd love to take this issue :)

[andreyvelich]

Thank you for your interest @anencore94!
As we discussed at the last Kubeflow SDK call, we probably only want to keep debug() logs in the SDK if users configure the appropriate verbosity.

/assign @anencore94