[Metric] Fix prometheus metric backend

[zhongchun]

What do these changes do?

• Fix an error when executing a task with Prometheus metric backend
• Add details docs for Prometheus metric backend

Related issue number

Fixes #3123

We can use Prometheus metric backend as follows:

Prepare Env

• Download Prometheus, details see https://prometheus.io/docs/prometheus/latest/getting_started/

How to use

1. New a Mars session

In [1]: import numpy as np
   ...:
   ...: import mars
   ...: import mars.dataframe as md
   ...:
   ...: session = mars.new_session(n_worker=1, n_cpu=2, web=True, config={"metrics.backend": "prometheus"})
   ...:
Finished startup prometheus http server and port is 46137
Finished startup prometheus http server and port is 32280
Finished startup prometheus http server and port is 39947
Finished startup prometheus http server and port is 25067
Web service started at http://0.0.0.0:11244

2. Config and start prometheus

scrape_configs:
  - job_name: 'mars'

    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:46137', 'localhost:32280', 'localhost:39947', 'localhost:25067']

$ prometheus --config.file=promconfig.yaml
level=info ts=2022-06-07T13:05:01.484Z caller=main.go:296 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2022-06-07T13:05:01.484Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.13.1, branch=non-git, revision=non-git)"
level=info ts=2022-06-07T13:05:01.484Z caller=main.go:333 build_context="(go=go1.13.1, [email protected], date=20191018-01:13:04)"
level=info ts=2022-06-07T13:05:01.485Z caller=main.go:334 host_details=(darwin)
level=info ts=2022-06-07T13:05:01.485Z caller=main.go:335 fd_limits="(soft=256, hard=unlimited)"
level=info ts=2022-06-07T13:05:01.485Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2022-06-07T13:05:01.487Z caller=main.go:657 msg="Starting TSDB ..."
level=info ts=2022-06-07T13:05:01.488Z caller=web.go:450 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2022-06-07T13:05:01.494Z caller=head.go:514 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2022-06-07T13:05:01.495Z caller=head.go:562 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1
level=info ts=2022-06-07T13:05:01.495Z caller=head.go:562 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1
level=info ts=2022-06-07T13:05:01.497Z caller=main.go:672 fs_type=1a
level=info ts=2022-06-07T13:05:01.497Z caller=main.go:673 msg="TSDB started"
level=info ts=2022-06-07T13:05:01.497Z caller=main.go:743 msg="Loading configuration file" filename=promconfig_mars.yaml
level=info ts=2022-06-07T13:05:01.501Z caller=main.go:771 msg="Completed loading of configuration file" filename=promconfig_mars.yaml
level=info ts=2022-06-07T13:05:01.501Z caller=main.go:626 msg="Server is ready to receive web requests."

3. Run a task

df1 = md.DataFrame(np.random.randint(0, 3, size=(10, 4)),
                   columns=list('ABCD'), chunk_size=5)
df2 = md.DataFrame(np.random.randint(0, 3, size=(10, 4)),
                   columns=list('ABCD'), chunk_size=5)

r = md.merge(df1, df2, on='A').execute()

4. Check Prometheus web url
   http://localhost:9090
   

Check code requirements

• tests added / passed (if needed)
• Ensure all linting tests pass, see here for how to run them

[qinxuye]

Nice work, could you please add the description about how to start prometheus backend to the doc as well? Thanks in advance.

[zhongchun]

Nice work, could you please add the description about how to start prometheus backend to the doc as well? Thanks in advance.

OK.

[fyrestone]

The shutdown_metrics only set the global var _init to False, how does it shutdown the http server started by the prometheus metrics backend?

[zhongchun]

Thanks for you reminder. We should do some some cleanup. But there is no stop or shutdown in prometheus_client.

[qinxuye]

LGTM

[wjsi]

LGTM