Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable collecting cri-o metrics #4087

Closed
ahrennsiva opened this issue Apr 12, 2019 · 5 comments
Closed

Enable collecting cri-o metrics #4087

ahrennsiva opened this issue Apr 12, 2019 · 5 comments
Labels
co/runtime/crio CRIO related issues help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@ahrennsiva
Copy link

I can currently run:
minikube start --container-runtime=cri-o

This will start minikube with CRI-O as the container runtime. CRI-O exposes metrics with this flag enabled --enable-metrics: Enable metrics endpoint. Default is localhost:9090. I would like to access the metrics exposed here for minikube but this currently isn't possible without additional changes in the VM itself.

Would it be possible to allow passing in runtime flags as a feature, at least for all currently supported alternative container runtimes?

@afbjorklund
Copy link
Collaborator

Thanks for reporting! This seems reasonable, we might have to rename the docker-opt/docker-env

@afbjorklund afbjorklund added the co/runtime/crio CRIO related issues label Apr 12, 2019
@tstromberg tstromberg added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/backlog Higher priority than priority/awaiting-more-evidence. kind/feature Categorizes issue or PR as related to a new feature. labels Apr 22, 2019
@tstromberg tstromberg added the r/2019q2 Issue was last reviewed 2019q2 label May 23, 2019
@afbjorklund afbjorklund modified the milestone: v1.4.0 Candidate Aug 6, 2019
@afbjorklund
Copy link
Collaborator

Should probably enable the metrics by default, until we can improve upon the docker-centric situation

@tstromberg tstromberg removed the r/2019q2 Issue was last reviewed 2019q2 label Sep 20, 2019
@afbjorklund afbjorklund added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Nov 9, 2019
@afbjorklund
Copy link
Collaborator

afbjorklund commented Nov 9, 2019

Seems like we forgot all about this one... Should still modify the docker-centric variables.

But maybe we can just make --enable-metrics the default for crio, by hacking the service ?

i.e. install this, and uncomment the line:

# /etc/sysconfig/crio

# use "--enable-metrics" and "--metrics-port value"
#CRIO_METRICS_OPTIONS="--enable-metrics"

#CRIO_NETWORK_OPTIONS=
#CRIO_STORAGE_OPTIONS=

Currently we overwrite this with a custom version:

deploy/iso/minikube-iso/package/crio-bin/crio-bin.mk:   echo 'CRIO_OPTIONS="--log-level=debug"' > $(TARGET_DIR)/etc/sysconfig/crio

And the $CRIO_METRICS_OPTIONS are removed:

-ExecStart=/usr/local/bin/crio \
-          $CRIO_CONFIG_OPTIONS \
-          $CRIO_RUNTIME_OPTIONS \
-          $CRIO_STORAGE_OPTIONS \
-          $CRIO_NETWORK_OPTIONS \
-          $CRIO_METRICS_OPTIONS
+ExecStart=/usr/bin/crio \
+    $CRIO_OPTIONS \
+    $CRIO_MINIKUBE_OPTIONS

The custom generated minikube configuration:

       var (
                crioOptsTmpl = `
CRIO_MINIKUBE_OPTIONS='{{ range .EngineOptions.InsecureRegistry }}--insecure-registry {{.}} {{ end }}'
`
                crioOptsPath = "/etc/sysconfig/crio.minikube"
        )

@afbjorklund
Copy link
Collaborator

Tried to enable this in crio.conf, but it failed (to apply, since the config is being overwritten)

@afbjorklund
Copy link
Collaborator

Fixed now, actually changed in #5970 but stopped being overwritten with #6219

$ grep -C3 metrics /etc/crio/crio.conf
	"/opt/cni/bin/",
]

# A necessary configuration for Prometheus based metrics retrieval
[crio.metrics]

# Globally enable or disable metrics support.
enable_metrics = true

# The port on which the metrics server will listen.
metrics_port = 9090
$ curl localhost:9090/metrics   
# HELP container_runtime_crio_operations Cumulative number of CRI-O operations by operation type.
# TYPE container_runtime_crio_operations counter
container_runtime_crio_operations{operation_type="container_status"} 264
container_runtime_crio_operations{operation_type="create_container"} 7
container_runtime_crio_operations{operation_type="image_status"} 16
container_runtime_crio_operations{operation_type="list_containers"} 736
container_runtime_crio_operations{operation_type="list_images"} 53
container_runtime_crio_operations{operation_type="list_pod_sandbox"} 691
container_runtime_crio_operations{operation_type="pod_sandbox_status"} 18
container_runtime_crio_operations{operation_type="run_pod_sandbox"} 7
container_runtime_crio_operations{operation_type="start_container"} 7
container_runtime_crio_operations{operation_type="status"} 76
container_runtime_crio_operations{operation_type="version"} 59
...

Will be included in v1.7.0, sorry it took so long... (still have to fix docker-opt/docker-env)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/runtime/crio CRIO related issues help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Development

No branches or pull requests

3 participants