[circushttpd] Load info

[Natim]

The worker load should be the sum of the worker load and it sub processes load.

For instance our insight-reloaded worker is launching docsplit and we have just the subprocess start load. It is then impossible to see the load related to docsplit.

[Natim]

I am digging into the code for few days now and I am a bit lost.

It seams that the data from the graph is coming from a socket that receive data from a streamer that collects data from a collector.

def collect_stats(self):
    aggregate = {}

    # sending by pids
    for pid in self.streamer.get_pids(self.name):
        name = None

        if self.name == 'circus':
            if pid in self.streamer.circus_pids:
                name = self.streamer.circus_pids[pid]

        try:
            info = util.get_info(pid)
            aggregate[pid] = info
            info['subtopic'] = pid
            info['name'] = name
            yield info
        except util.NoSuchProcess:
            # the process is gone !
            pass
        except Exception, e:
            logger.exception('Failed to get info for %d. %s' % (pid,
                str(e)))

    # now sending the aggregation
    yield self._aggregate(aggregate)

We first yield the info for every pid and then the aggregation.

It seams that there is never any children defined in the stats, in the get_info, we don't run a recursive get_childrens.

Do we agree that for watchers we want to display the sum of memory use for recursive workers and for cpus we want the sum of all subprocess cpu usage devided by the number of CPU ?

[almet]

I think we disabled that at some point for performance-related issues (in psutil), but these are resolved by now, so we should be able to change this behavior back to get the information recursively from the workers and their children.

This can be switched back by setting the with_child parameter to True when we make calls there. I'm wondering if that would make sense to have a manual switch to activate/deactivate this in the configuration?

About the stats, we want the sum for memory and the average for the cpu, you're right.