Add prometheus metrics for experiment and trial

[hougangliu]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

---

This change is 

[hougangliu]

# curl 10.0.47.94:8080/metrics|grep katib_
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 17729    0 17729    0     0   524k      0 --:--:-- --:--:-- --:--:--  541k
# HELP katib_experiment_created Counts number of Experiment created
# TYPE katib_experiment_created counter
katib_experiment_created 1
# HELP katib_experiment_deleted Counts number of Experiment deleted
# TYPE katib_experiment_deleted counter
katib_experiment_deleted 1
# HELP katib_experiment_failed Counts number of Experiment failed
# TYPE katib_experiment_failed counter
katib_experiment_failed 0
# HELP katib_experiment_succeeded Counts number of Experiment succeeded
# TYPE katib_experiment_succeeded counter
katib_experiment_succeeded 1
# HELP katib_trial_created Counts number of Trial created
# TYPE katib_trial_created counter
katib_trial_created 12
# HELP katib_trial_deleted Counts number of Trial deleted
# TYPE katib_trial_deleted counter
katib_trial_deleted 12
# HELP katib_trial_failed Counts number of Trial failed
# TYPE katib_trial_failed counter
katib_trial_failed 0
# HELP katib_trial_succeeded Counts number of Trial succeeded
# TYPE katib_trial_succeeded counter
katib_trial_succeeded 12

[hougangliu]

/test kubeflow-katib-presubmit
/cc @johnugeorge @gaocegege

[johnugeorge]

Instead of Increase, increment seems more appropriate?

[hougangliu]

do you mean rename IncreaseExperimentsDeletedCount to Increment ExperimentsDeletedCount? If so, I think Increase is better, since it is a verb

[johnugeorge]

@hougangliu when we specify counters, we typically specify the term Increment. See Inc definition in prometheus https://github.com/kubeflow/katib/blob/master/vendor/github.com/prometheus/client_golang/prometheus/counter.go#L37

Or, should we directly call experimentsDeletedCount.Inc() ?

[hougangliu]

Yes, I find it. In fact https://github.com/kubeflow/katib/blob/master/vendor/github.com/prometheus/client_golang/prometheus/counter.go#L37 should use increase too, it should be a typo error.
The reason why I didn't use experimentsDeletedCount.Inc() is that other package "pkg/controller.v1alpha3/experiment/util" also call it, but experimentsDeletedCount cannot be access by other packages

[gaocegege]

/lgtm

[johnugeorge]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hougangliu]

/test kubeflow-katib-presubmit

[gaocegege]

@hougangliu Rebase upstream master may be helpful for CI failures.

[hougangliu]

@hougangliu Rebase upstream master may be helpful for CI failures.

OK, rebased. Thanks

[gaocegege]

/lgtm

[yeya24]

Do we need to rename the metrics?