Cluster Object Storage Space usage page

[dmitrizagidulin]

Create a page where a user can tell overall cluster storage usage.
A pie chart or some other visual element, denoting:

• X bytes unused/remaining (of overall CS cluster capacity - how many can be stored before cluster runs out of room)
  (obviously, substitute appropriate magnitudes for gigabytes, terabytes, etc).
• Y bytes' worth of objects stored by UserM (for any user who's taking up >= 10% space)
• Z bytes' worth of objects stored by remaining Other users

Specifically, this is an overview of objects stored, not raw disk space (meaning, it needs to take into account the n_val, # of nodes, free disk space remaining on each node, etc, and perform the appropriate calculations).

[cmeiklejohn]

Dependent on basho/riak_cs#612.

[dmitrizagidulin]

Edited description to reflect a clarification from customer -- preferably include top users (who are taking up 10+% of the cluster) in the overview pie chart.

[cmeiklejohn]

I think it's important to put this information into the associated riak_cs request, if the customer has any expectation that the usage data should matched, as these may be generated on different schedules.

[jgnewman]

Concept:

[jgnewman]

Looks like I misinterpreted the data being asked for. Let me refine this.

[cmeiklejohn]

This need to be revised to show the correct presentation for the users who are > 10%, or derivation of.

[jgnewman]

Here's one showing more information. In Hipchat, Dmitri was throwing around the range 5-10%. Are we set for sure on 10%? Because, given the size of the graph, we could potentially get away 5%.

[dmitrizagidulin]

First pass at calculating free disk space for the whole cluster (Riak CS basho/riak_cs#612) can be found here: dmitrizagidulin/riak_cs@basho:master...feature/cluster_stats_demo
On the CS side, it looks like:

(riak-cs@127.0.0.1)1> riak_cs_riak_stats:get_riak_cluster_disk_usage().
{struct,[{<<"cluster_capacity">>,230398124},
         {<<"cluster_disk_usage_kb">>,32255737},
         {<<"cluster_disk_free_kb">>,198142387},
         {<<"cluster_node_count">>,1},
         {<<"n_val">>,3},
         {<<"object_storage_capacity_remaining_kb">>,66047462}]}

or via rest:

$ curl http://localhost:8080/riak-cs/disk_usage
{"cluster_capacity":230398124,
"cluster_disk_usage_kb":32255737,
"cluster_disk_free_kb":198142387,"cluster_node_count":1,
"n_val":3,
"object_storage_capacity_remaining_kb":66047462}

On the Riak CS Control side, that's implemented here (calls out to the riak cs disk usage rest api): https://github.com/dmitrizagidulin/riak_cs_control/compare/feature;cluster_stats_demo

I'm working on the user storage statistics part at the moment (porting a proof of concept ruby Fog script to erlang), and then it should be ready to graph.

[vmalaga]

Hi @jgnewman

It is just a screenshot or functionality exists ?
I'm using riak-cs-control version of ubuntu but only allows users to create, nothing about of disk usage.

is this project active any more ?

[hmmr]

A significant portion of this functionality appeared in https://github.com/TI-Tokyo/riak_cs/releases/tag/3.2.3.

[hmmr]

And more, in 3.2.4.