fix: Heavy logging by TensorRT-LLM monolith worker

[kthui]

Overview:

Stop heavy logging by TensorRT-LLM monolith worker.

Details:

N/A

Where should the reviewer start?

Start with MR !51 for adding CI.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: N/A

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[kthui]

@grahamking this fix is required for testing this example on the CI, otherwise each run from this example will accumulate > 10 GB log file.

[grahamking]

@grahamking this fix is required for testing this example on the CI, otherwise each run from this example will accumulate > 10 GB log file.

Would redirecting the output to /dev/null work? python <the_example> 2>&1 > /dev/null

[kthui]

Would redirecting the output to /dev/null work? python <the_example> 2>&1 > /dev/null

This will fix the logging issue, but if there is any failure by the worker, then we won't get any log for debug?

[tanmayv25]

Aren't there any more places which are setting debug logging in the examples? For example, router and disagg worker pieces?
I would run a find and replace for the logger.set_level.

[kthui]

Everything are at "info" level, except this occurrence, i.e.

[NVShreyas]

Could you change it such that we can use an env variable to control log level for all TRTLLM component files? Its pretty tedious to keep changing it when developing.

[kthui]

Could you change it such that we can use an env variable to control log level for all TRTLLM component files?

Sure, it will read from DYNAMO_TRTLLM_LOG_LEVEL if set. If not set, it will default to info.

[grahamking]

Could you change it such that we can use an env variable to control log level for all TRTLLM component files?

Sure, it will read from DYNAMO_TRTLLM_LOG_LEVEL if set. If not set, it will default to info.

Our other environment variables all start DYN_. Could you use that?

[NVShreyas]

Thanks

[kthui]

Our other environment variables all start DYN_. Could you use that?

Yes, the prefix is updated to DYN_.

[kthui]

Merging this PR, as the TensorRT-LLM tests on pipeline #25899729 are passing.