Allow more control over Prometheus metrics collection

[mateka]

Description

I have added possibility to define prometheus metrics labels based on task parameters.

Motivation and Context

I wanted to have more grouping options for prometheus monitoring. We use one scheduler for many graphs which share tasks and we wanted to have possibility to discern different graphs.

Have you tested this? If so, how?

I ran example jobs with this code and it works for me.

[dlstadther]

Thanks for your contribution. I've left some comments mostly about naming and organization. Please take a look and address them or provide response to the feedback.

Also, please see if there are valuable tests you can add to the existing prometheus test file such that any future contributions avoid causing undesirable changes of behavior.

[mateka]

Thanks for your contribution. I've left some comments mostly about naming and organization. Please take a look and address them or provide response to the feedback.

Wow. Thank you for quick feedback! I addressed your comments.

Also, please see if there are valuable tests you can add to the existing prometheus test file such that any future contributions avoid causing undesirable changes of behavior.

Yes, after pushing my changes I realized that I had not added unit tests. It took a while, because I was not so sure how to make unit tests code nice and without repetitions. Now all cases should be covered.

I am also thinking about extending scheduler Task class by adding metadata dictionary, which could be somehow updated from user provided context in run method of ContextManagedTaskProcess class. This metadata could be used for even more metrics in custom MetricsCollector classes (for instance to measure used RAM or reporting statistics from cloud or map-reduce frameworks). Do you think this is a good direction? Probably it would be even possible to define additional Prometheus metrics based on above mentioned metadata in configuration without custom Prometheus Metrics Collector provided by users!

[dlstadther]

Thanks for addressing my feedback as well as adding tests!

[dlstadther]

I am also thinking about extending scheduler Task class by adding metadata dictionary, which could be somehow updated from user provided context in run method of ContextManagedTaskProcess class. This metadata could be used for even more metrics in custom MetricsCollector classes (for instance to measure used RAM or reporting statistics from cloud or map-reduce frameworks). Do you think this is a good direction? Probably it would be even possible to define additional Prometheus metrics based on above mentioned metadata in configuration without custom Prometheus Metrics Collector provided by users!

I don't fully picture this extension of Task. However, my mind hasn't been deep into the development of Luigi internals for a few years. I'm mostly here to provide PR reviews and try to maintain code quality and test coverage. If you feel you would like some feedback on your idea, feel free to post an Issue with your idea and try to elicit feedback from various folks who you have seen are active in this project or who have contributed in the past to related features.

Given that the Luigi project doesn't see the same usage and interaction as it did many years ago (in favor or other tools such as Airflow, Prefect, Dagster, and others), I do have some caution about adding too much complexity to Luigi. I want this project to continue to fulfill the needs of its community, but I also don't want to see it become overly complex to maintain (given the few folks still managing the project).

All that said, I welcome any contribution that fills a need and which will become generally applicable to the community.