[graal] Add Code Complexity(CoCom) & Code License(CoLic) panels

[inishchith]

REF. inishchith/gsoc#16

• In sync with remote instance(?)

Signed-off-by: inishchith [email protected]

[valeriocos]

Thank you @inishchith for this PR.
I tried it out, but it seems that something is not working. What I did was executing the enrichment and study phase locally, and import the panels with micro-mordred (--enrich and panels). Everything was correctly imported, I set the aliases but the data is not shown in the dashboards.

I modified the config.py and task_panels.py (I'm going to share their code soon) to be able to active the dashboards from the setup.cfg with

[panels]
kibiter_time_from= "now-30y"
kibiter_default_index= "cocom"
kibiter_url = http://admin:admin@localhost:5601
community = true
code-license = true <---
code-complexity = true <--

The main reason of this modification is because the dashboards contain also a index pattern of the study, which should be uploaded together with the index pattern of the enriched index.

Some comments about the PR below:

• please split the commit in two: one for cocom and one for colic
• the name of the dashboard files should be cocom and colic
• some name fields contain a .keyword, although it's valid, the other dashboards don't contain them. Please remove them.
• the title of the index patterns could be equal to their id values (which are shorter and easier to set as aliases)

[valeriocos]

task_panels.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# Copyright (C) 2015-2019 Bitergia
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Authors:
#     Luis Cañas-Díaz <[email protected]>
#     Alvaro del Castillo <[email protected]>
#

import copy
import json
import logging
import operator
import requests
import yaml

from grimoirelab_toolkit.uris import urijoin

from kidash.kidash import import_dashboard, get_dashboard_name, check_kibana_index
from sirmordred.task import Task

logger = logging.getLogger(__name__)

# Header mandatory in ElasticSearch 6
ES6_HEADER = {"Content-Type": "application/json"}
KIBANA_SETTINGS_URL = '/api/kibana/settings'

STRICT_LOADING = "strict"

KAFKA_NAME = 'KIP'
KAFKA_SOURCE = "kafka"
KAFKA_PANEL = "panels/json/kip.json"
KAKFA_IP = "panels/json/kafka-index-pattern.json"

KAFKA_MENU = {
    'name': KAFKA_NAME,
    'source': KAFKA_SOURCE,
    'icon': 'default.png',
    'index-patterns': [KAKFA_IP],
    'menu': [
        {'name': 'Overview', 'panel': KAFKA_PANEL}
    ]
}

COMMUNITY_NAME = 'Community'
COMMUNITY_SOURCE = 'community'
ONION_PANEL_OVERALL = 'panels/json/onion_overall.json'
ONION_PANEL_PROJECTS = 'panels/json/onion_projects.json'
ONION_PANEL_ORGS = 'panels/json/onion_organizations.json'
DEMOGRAPHICS = 'panels/json/demographics.json'
AFFILIATIONS = 'panels/json/affiliations.json'

ONION_PANEL_OVERALL_IP = 'panels/json/all_onion-index-pattern.json'
ONION_PANEL_PROJECTS_IP = 'panels/json/all_onion-index-pattern.json'
ONION_PANEL_ORGS_IP = 'panels/json/all_onion-index-pattern.json'
DEMOGRAPHICS_IP = 'panels/json/demographics-index-pattern.json'
AFFILIATIONS_IP = 'panels/json/affiliations-index-pattern.json'

COMMUNITY_MENU = {
    'name': COMMUNITY_NAME,
    'source': COMMUNITY_SOURCE,
    'icon': 'default.png',
    'index-patterns': [
        ONION_PANEL_OVERALL_IP,
        ONION_PANEL_PROJECTS_IP,
        ONION_PANEL_ORGS_IP,
        DEMOGRAPHICS_IP,
        AFFILIATIONS_IP
    ],
    'menu': [
        {'name': 'Overall', 'panel': ONION_PANEL_OVERALL},
        {'name': 'Projects', 'panel': ONION_PANEL_PROJECTS},
        {'name': 'Organizations', 'panel': ONION_PANEL_ORGS},
        {'name': 'Demographics', 'panel': DEMOGRAPHICS},
        {'name': 'Affiliations', 'panel': AFFILIATIONS}
    ]
}

GITHUB_REPOS = "github-repos"
GITHUB_REPOS_PANEL_OVERALL = "panels/json/github_repositories.json"
GITHUB_REPOS_IP = "panels/json/github_repositories-index-pattern.json"

GITHUB_REPOS_MENU = {
    'name': 'GitHub Repositories',
    'source': GITHUB_REPOS,
    'icon': 'default.png',
    'index-patterns': [GITHUB_REPOS_IP],
    'menu': [
        {'name': 'Overview', 'panel': GITHUB_REPOS_PANEL_OVERALL}
    ]
}

GITLAB_ISSUES = "gitlab-issues"
GITLAB_ISSUES_PANEL_OVERALL = "panels/json/gitlab_issues.json"
GITLAB_ISSUES_PANEL_BACKLOG = "panels/json/gitlab_issues_backlog.json"
GITLAB_ISSUES_PANEL_TIMING = "panels/json/gitlab_issues_timing.json"
GITLAB_ISSUES_PANEL_EFFICIENCY = "panels/json/gitlab_issues_efficiency.json"
GITLAB_ISSUES_IP = "panels/json/gitlab_issues-index-pattern.json"

GITLAB_ISSUES_MENU = {
    'name': 'GitLab Issues',
    'source': GITLAB_ISSUES,
    'icon': 'default.png',
    'index-patterns': [GITLAB_ISSUES_IP],
    'menu': [
        {'name': 'Overview', 'panel': GITLAB_ISSUES_PANEL_OVERALL},
        {'name': 'Backlog', 'panel': GITLAB_ISSUES_PANEL_BACKLOG},
        {'name': 'Timing', 'panel': GITLAB_ISSUES_PANEL_TIMING},
        {'name': 'Efficiency', 'panel': GITLAB_ISSUES_PANEL_EFFICIENCY}
    ]
}

GITLAB_MERGES = "gitlab-merges"
GITLAB_MERGES_PANEL_OVERALL = "panels/json/gitlab_merge_requests.json"
GITLAB_MERGES_PANEL_BACKLOG = "panels/json/gitlab_merge_requests_backlog.json"
GITLAB_MERGES_PANEL_TIMING = "panels/json/gitlab_merge_requests_timing.json"
GITLAB_MERGES_PANEL_EFFICIENCY = "panels/json/gitlab_merge_requests_efficiency.json"
GITLAB_MERGES_IP = "panels/json/gitlab_merge_requests-index-pattern.json"

GITLAB_MERGES_MENU = {
    'name': 'GitLab Merges',
    'source': GITLAB_MERGES,
    'icon': 'default.png',
    'index-patterns': [GITLAB_MERGES_IP],
    'menu': [
        {'name': 'Overview', 'panel': GITLAB_MERGES_PANEL_OVERALL},
        {'name': 'Backlog', 'panel': GITLAB_MERGES_PANEL_BACKLOG},
        {'name': 'Timing', 'panel': GITLAB_MERGES_PANEL_TIMING},
        {'name': 'Efficiency', 'panel': GITLAB_MERGES_PANEL_EFFICIENCY}
    ]
}

MATTERMOST = "mattermost"
MATTERMOST_PANEL = "panels/json/mattermost.json"
MATTERMOST_IP = "panels/json/mattermost-index-pattern.json"

MATTERMOST_MENU = {
    'name': 'Mattermost',
    'source': MATTERMOST,
    'icon': 'default.png',
    'index-patterns': [MATTERMOST_IP],
    'menu': [
        {'name': 'Overview', 'panel': MATTERMOST_PANEL}
    ]
}

MATTERMOST = "mattermost"
MATTERMOST_PANEL = "panels/json/mattermost.json"
MATTERMOST_IP = "panels/json/mattermost-index-pattern.json"

COCOM_NAME = 'Code Complexity'
COCOM_SOURCE = 'code-complexity'
COCOM_PANEL = "panels/json/cocom.json"
COCOM_IP = "panels/json/cocom-index-pattern.json"
COCOM_STUDY_IP = "panels/json/cocom_study-index-pattern.json"

COCOM_MENU = {
    'name': COCOM_NAME,
    'source': COCOM_SOURCE,
    'icon': 'default.png',
    'index-patterns': [
        COCOM_IP,
        COCOM_STUDY_IP
    ],
    'menu': [
        {'name': 'Overview', 'panel': COCOM_PANEL}
    ]
}

COLIC_NAME = 'Code License'
COLIC_SOURCE = 'code-license'
COLIC_PANEL = "panels/json/colic.json"
COLIC_IP = "panels/json/colic-index-pattern.json"
COLIC_STUDY_IP = "panels/json/colic_study-index-pattern.json"

COLIC_MENU = {
    'name': COLIC_NAME,
    'source': COLIC_SOURCE,
    'icon': 'default.png',
    'index-patterns': [
        COLIC_IP,
        COLIC_STUDY_IP
    ],
    'menu': [
        {'name': 'Overview', 'panel': COLIC_PANEL}
    ]
}


class TaskPanels(Task):
    """
    Upload all the Kibana dashboards/GrimoireLab panels based on
    enabled data sources AND the menu file. Before uploading the panels
    it also sets up common configuration variables for Kibiter
    """

    # Panels which include several data sources
    panels_multi_ds = ["panels/json/overview.json", "panels/json/data_status.json"]
    # Panels to be uploaded always, no matter the data sources configured
    panels_common = panels_multi_ds + ["panels/json/about.json"]

    def __init__(self, conf):
        super().__init__(conf)
        # Read panels and menu description from yaml file
        with open(self.conf['general']['menu_file'], 'r') as f:
            try:
                self.panels_menu = yaml.load(f, Loader=yaml.SafeLoader)
            except yaml.YAMLError as ex:
                logger.error(ex)
                raise

        # FIXME exceptions raised here are not handled!!

        # Gets the cross set of enabled data sources and data sources
        # available in the menu file. For the result set gets the file
        # names of dashoards and index pattern to be uploaded
        enabled_ds = conf.get_data_sources()
        self.panels = {}

        for ds in self.panels_menu:
            if ds['source'] not in enabled_ds:
                continue
            for entry in ds['menu']:
                if ds['source'] not in self.panels.keys():
                    self.panels[ds['source']] = []
                self.panels[ds['source']].append(entry['panel'])
            if 'index-patterns' in ds:
                for index_pattern in ds['index-patterns']:
                    self.panels[ds['source']].append(index_pattern)

        if self.conf['panels'][COMMUNITY_SOURCE]:
            self.panels[COMMUNITY_SOURCE] = [ONION_PANEL_OVERALL, ONION_PANEL_PROJECTS,
                                             ONION_PANEL_ORGS, DEMOGRAPHICS, AFFILIATIONS,
                                             ONION_PANEL_OVERALL_IP, ONION_PANEL_PROJECTS_IP,
                                             ONION_PANEL_ORGS_IP, DEMOGRAPHICS_IP, AFFILIATIONS_IP]

        if self.conf['panels'][KAFKA_SOURCE]:
            self.panels[KAFKA_SOURCE] = [KAFKA_PANEL, KAKFA_IP]

        if self.conf['panels'][GITHUB_REPOS]:
            self.panels[GITHUB_REPOS] = [GITHUB_REPOS_PANEL_OVERALL, GITHUB_REPOS_IP]

        if self.conf['panels'][GITLAB_ISSUES]:
            self.panels[GITLAB_ISSUES] = [GITLAB_ISSUES_PANEL_BACKLOG, GITLAB_ISSUES_PANEL_OVERALL,
                                          GITLAB_ISSUES_PANEL_TIMING, GITLAB_ISSUES_PANEL_EFFICIENCY,
                                          GITLAB_ISSUES_IP]

        if self.conf['panels'][GITLAB_MERGES]:
            self.panels[GITLAB_MERGES] = [GITLAB_MERGES_PANEL_BACKLOG, GITLAB_MERGES_PANEL_OVERALL,
                                          GITLAB_MERGES_PANEL_TIMING, GITLAB_MERGES_PANEL_EFFICIENCY,
                                          GITLAB_MERGES_IP]

        if self.conf['panels'][MATTERMOST]:
            self.panels[MATTERMOST] = [MATTERMOST_PANEL, MATTERMOST_IP]

        if self.conf['panels'][COCOM_SOURCE]:
            self.panels[COCOM_SOURCE] = [COCOM_PANEL, COCOM_IP, COCOM_STUDY_IP]

        if self.conf['panels'][COLIC_SOURCE]:
            self.panels[COLIC_SOURCE] = [COLIC_PANEL, COLIC_IP, COLIC_STUDY_IP]

    def is_backend_task(self):
        return False

    def __kibiter_version(self):
        """ Get the kibiter vesion.

        :param major: major Elasticsearch version
        """
        version = None

        es_url = self.conf['es_enrichment']['url']
        config_url = '.kibana/config/_search'
        url = urijoin(es_url, config_url)
        version = None
        try:
            res = self.grimoire_con.get(url)
            res.raise_for_status()
            version = res.json()['hits']['hits'][0]['_id']
            logger.debug("Kibiter version: %s", version)
        except requests.exceptions.HTTPError:
            logger.warning("Can not find Kibiter version")

        return version

    def __configure_kibiter_setting(self, endpoint, data_value=None):
        kibana_headers = copy.deepcopy(ES6_HEADER)
        kibana_headers["kbn-xsrf"] = "true"

        kibana_url = self.conf['panels']['kibiter_url'] + KIBANA_SETTINGS_URL
        endpoint_url = kibana_url + '/' + endpoint

        try:
            res = self.grimoire_con.post(endpoint_url, headers=kibana_headers,
                                         data=json.dumps(data_value), verify=False)
            res.raise_for_status()
        except requests.exceptions.HTTPError:
            logger.error("Impossible to set %s: %s", endpoint, str(res.json()))
            return False
        except requests.exceptions.ConnectionError as ex:
            logger.error("Impossible to connect to kibiter %s: %s",
                         self.anonymize_url(self.conf['panels']['kibiter_url']), ex)
            return False

        return True

    def __configure_kibiter_6(self):
        if 'panels' not in self.conf:
            logger.warning("Panels config not availble. Not configuring Kibiter.")
            return False

        # Create .kibana index if not exists
        es_url = self.conf['es_enrichment']['url']
        kibiter_url = self.conf['panels']['kibiter_url']
        check_kibana_index(es_url, kibiter_url)

        # set default index pattern
        kibiter_default_index = self.conf['panels']['kibiter_default_index']
        data_value = {"value": kibiter_default_index}
        defaultIndexFlag = self.__configure_kibiter_setting('defaultIndex',
                                                            data_value=data_value)

        # set default time picker
        kibiter_time_from = self.conf['panels']['kibiter_time_from']
        time_picker = {"from": kibiter_time_from, "to": "now", "mode": "quick"}
        data_value = {"value": json.dumps(time_picker)}
        timePickerFlag = self.__configure_kibiter_setting('timepicker:timeDefaults',
                                                          data_value=data_value)

        if defaultIndexFlag and timePickerFlag:
            logger.info("Kibiter settings configured!")
            return True

        logger.error("Kibiter settings not configured!")
        return False

    def __configure_kibiter_old(self, kibiter_major):

        if 'panels' not in self.conf:
            logger.warning("Panels config not availble. Not configuring Kibiter.")
            return False

        kibiter_time_from = self.conf['panels']['kibiter_time_from']
        kibiter_default_index = self.conf['panels']['kibiter_default_index']

        logger.info("Configuring Kibiter %s for default index %s and time frame %s",
                    kibiter_major, kibiter_default_index, kibiter_time_from)

        kibiter_version = self.__kibiter_version()
        if not kibiter_version:
            return False
        logger.info("Kibiter/Kibana: version found is %s" % kibiter_version)
        time_picker = "{\n  \"from\": \"" + kibiter_time_from \
            + "\",\n  \"to\": \"now\",\n  \"mode\": \"quick\"\n}"

        config_resource = '.kibana/config/' + kibiter_version
        kibiter_config = {
            "defaultIndex": kibiter_default_index,
            "timepicker:timeDefaults": time_picker
        }

        es_url = self.conf['es_enrichment']['url']
        url = urijoin(es_url, config_resource)
        res = self.grimoire_con.post(url, data=json.dumps(kibiter_config),
                                     headers=ES6_HEADER)
        res.raise_for_status()

        logger.info("Kibiter settings configured!")
        return True

    def create_dashboard(self, panel_file, data_sources=None, strict=True):
        """Upload a panel to Elasticsearch if it does not exist yet.

        If a list of data sources is specified, upload only those
        elements (visualizations, searches) that match that data source.

        :param panel_file: file name of panel (dashobard) to upload
        :param data_sources: list of data sources
        :param strict: only upload a dashboard if it is newer than the one already existing
        """
        es_enrich = self.conf['es_enrichment']['url']
        kibana_url = self.conf['panels']['kibiter_url']

        mboxes_sources = set(['pipermail', 'hyperkitty', 'groupsio', 'nntp'])
        if data_sources and any(x in data_sources for x in mboxes_sources):
            data_sources = list(data_sources)
            data_sources.append('mbox')
        if data_sources and ('supybot' in data_sources):
            data_sources = list(data_sources)
            data_sources.append('irc')
        if data_sources and 'google_hits' in data_sources:
            data_sources = list(data_sources)
            data_sources.append('googlehits')
        if data_sources and 'stackexchange' in data_sources:
            # stackexchange is called stackoverflow in panels
            data_sources = list(data_sources)
            data_sources.append('stackoverflow')
        if data_sources and 'phabricator' in data_sources:
            data_sources = list(data_sources)
            data_sources.append('maniphest')

        try:
            import_dashboard(es_enrich, kibana_url, panel_file, data_sources=data_sources, strict=strict)
        except ValueError:
            logger.error("%s does not include release field. Not loading the panel.", panel_file)
        except RuntimeError:
            logger.error("Can not load the panel %s", panel_file)

    def execute(self):
        # Configure kibiter
        kibiter_major = self.es_version(self.conf['es_enrichment']['url'])
        strict_loading = self.conf['panels'][STRICT_LOADING]

        if kibiter_major < "6":
            self.__configure_kibiter_old(kibiter_major)
        else:
            self.__configure_kibiter_6()

        logger.info("Dashboard panels, visualizations: uploading...")
        # Create the commons panels
        for panel_file in self.panels_common:
            data_sources = None  # for some panels, only the active data sources must be included
            if panel_file in TaskPanels.panels_multi_ds:
                data_sources = self.panels.keys()
            self.create_dashboard(panel_file, data_sources=data_sources, strict=strict_loading)

        # Upload all the Kibana dashboards/GrimoireLab panels based on
        # enabled data sources AND the menu file
        for ds in self.panels:
            for panel_file in self.panels[ds]:
                try:
                    self.create_dashboard(panel_file, strict=strict_loading)
                except Exception as ex:
                    logger.error("%s not correctly uploaded (%s)", panel_file, ex)
        logger.info("Dashboard panels, visualizations: uploaded!")


class TaskPanelsMenu(Task):
    """Create the menu to access the panels"""

    menu_panels_common = {
        "Overview": {
            "title": "Overview",
            "name": "Overview",
            "description": "Overview panel",
            "type": "entry",
            "panel_id": "Overview"
        },
        "About": {
            "title": "About",
            "name": "About",
            "description": "About panel",
            "type": "entry",
            "panel_id": "About"
        },
        "Data Status": {
            "title": "Data Status",
            "name": "Data Status",
            "description": "Data Status panel",
            "type": "entry",
            "panel_id": "Data-Status"
        }
    }

    def __init__(self, conf):
        super().__init__(conf)
        # Read panels and menu description from yaml file """
        with open(self.conf['general']['menu_file'], 'r') as f:
            try:
                self.panels_menu = yaml.load(f, Loader=yaml.SafeLoader)
            except yaml.YAMLError as ex:
                logger.error(ex)
                raise

        if self.conf['panels'][GITHUB_REPOS]:
            self.panels_menu.append(GITHUB_REPOS_MENU)

        if self.conf['panels'][GITLAB_ISSUES]:
            self.panels_menu.append(GITLAB_ISSUES_MENU)

        if self.conf['panels'][GITLAB_MERGES]:
            self.panels_menu.append(GITLAB_MERGES_MENU)

        if self.conf['panels'][MATTERMOST]:
            self.panels_menu.append(MATTERMOST_MENU)

        if self.conf['panels'][COMMUNITY_SOURCE]:
            self.panels_menu.append(COMMUNITY_MENU)

        if self.conf['panels'][KAFKA_SOURCE]:
            self.panels_menu.append(KAFKA_MENU)

        if self.conf['panels'][COCOM_SOURCE]:
            self.panels_menu.append(COCOM_MENU)

        if self.conf['panels'][COLIC_SOURCE]:
            self.panels_menu.append(COLIC_MENU)

        # Get the active data sources
        self.data_sources = self.__get_active_data_sources()
        if 'short_name' in self.conf['general']:
            self.project_name = self.conf['general']['short_name']
        else:
            self.project_name = 'GrimoireLab'

    def is_backend_task(self):
        return False

    def __get_active_data_sources(self):
        active_ds = []
        for entry in self.panels_menu:
            ds = entry['source']
            if ds in self.conf.keys() or ds in [COMMUNITY_SOURCE, KAFKA_SOURCE, GITLAB_ISSUES, GITLAB_MERGES,
                                                MATTERMOST, GITHUB_REPOS, COCOM_SOURCE, COLIC_SOURCE]:
                active_ds.append(ds)
        logger.debug("Active data sources for menu: %s", active_ds)

        return active_ds

    def __upload_title(self, kibiter_major):
        """Upload to Kibiter the title for the dashboard.

        The title is shown on top of the dashboard menu, and is Usually
        the name of the project being dashboarded.
        This is done only for Kibiter 6.x.

        :param kibiter_major: major version of kibiter
        """

        if kibiter_major == "6":
            resource = ".kibana/doc/projectname"
            data = {"projectname": {"name": self.project_name}}
            mapping_resource = ".kibana/_mapping/doc"
            mapping = {"dynamic": "true"}

            url = urijoin(self.conf['es_enrichment']['url'], resource)
            mapping_url = urijoin(self.conf['es_enrichment']['url'],
                                  mapping_resource)

            logger.debug("Adding mapping for dashboard title")
            res = self.grimoire_con.put(mapping_url, data=json.dumps(mapping),
                                        headers=ES6_HEADER)
            try:
                res.raise_for_status()
            except requests.exceptions.HTTPError:
                logger.error("Couldn't create mapping for dashboard title.")
                logger.error(res.json())

            logger.debug("Uploading dashboard title")
            res = self.grimoire_con.post(url, data=json.dumps(data),
                                         headers=ES6_HEADER)
            try:
                res.raise_for_status()
            except requests.exceptions.HTTPError:
                logger.error("Couldn't create dashboard title.")
                logger.error(res.json())

    def __create_dashboard_menu(self, dash_menu, kibiter_major):
        """Create the menu definition to access the panels in a dashboard.

        :param          menu: dashboard menu to upload
        :param kibiter_major: major version of kibiter
        """
        logger.info("Adding dashboard menu")
        if kibiter_major == "6":
            menu_resource = ".kibana/doc/metadashboard"
            mapping_resource = ".kibana/_mapping/doc"
            mapping = {"dynamic": "true"}
            menu = {'metadashboard': dash_menu}
        else:
            menu_resource = ".kibana/metadashboard/main"
            mapping_resource = ".kibana/_mapping/metadashboard"
            mapping = {"dynamic": "true"}
            menu = dash_menu
        menu_url = urijoin(self.conf['es_enrichment']['url'],
                           menu_resource)

        mapping_url = urijoin(self.conf['es_enrichment']['url'],
                              mapping_resource)
        logger.debug("Adding mapping for metadashboard")
        res = self.grimoire_con.put(mapping_url, data=json.dumps(mapping),
                                    headers=ES6_HEADER)
        try:
            res.raise_for_status()
        except requests.exceptions.HTTPError:
            logger.error("Couldn't create mapping for Kibiter menu.")
        res = self.grimoire_con.post(menu_url, data=json.dumps(menu),
                                     headers=ES6_HEADER)
        try:
            res.raise_for_status()
        except requests.exceptions.HTTPError:
            logger.error("Couldn't create Kibiter menu.")
            logger.error(res.json())
            raise

    def __remove_dashboard_menu(self, kibiter_major):
        """Remove existing menu for dashboard, if any.

        Usually, we remove the menu before creating a new one.

        :param kibiter_major: major version of kibiter
        """
        logger.info("Removing old dashboard menu, if any")
        if kibiter_major == "6":
            metadashboard = ".kibana/doc/metadashboard"
        else:
            metadashboard = ".kibana/metadashboard/main"
        menu_url = urijoin(self.conf['es_enrichment']['url'], metadashboard)
        self.grimoire_con.delete(menu_url)

    def __get_menu_entries(self, kibiter_major):
        """ Get the menu entries from the panel definition """
        menu_entries = []
        for entry in self.panels_menu:
            if entry['source'] not in self.data_sources:
                continue
            parent_menu_item = {
                'name': entry['name'],
                'title': entry['name'],
                'description': "",
                'type': "menu",
                'dashboards': []
            }
            for subentry in entry['menu']:
                try:
                    dash_name = get_dashboard_name(subentry['panel'])
                except FileNotFoundError:
                    logging.error("Can't open dashboard file %s", subentry['panel'])
                    continue
                # The name for the entry is in self.panels_menu
                child_item = {
                    "name": subentry['name'],
                    "title": subentry['name'],
                    "description": "",
                    "type": "entry",
                    "panel_id": dash_name
                }
                parent_menu_item['dashboards'].append(child_item)
            menu_entries.append(parent_menu_item)

        return menu_entries

    def __get_dash_menu(self, kibiter_major):
        """Order the dashboard menu"""

        # omenu = OrderedDict()
        omenu = []
        # Start with Overview
        omenu.append(self.menu_panels_common['Overview'])

        # Now the data _getsources
        ds_menu = self.__get_menu_entries(kibiter_major)

        # Remove the kafka and community menus, they will be included at the end
        kafka_menu = None
        community_menu = None
        cocom_menu = None
        colic_menu = None

        found_cocom = [pos for pos, menu in enumerate(ds_menu) if menu['name'] == COCOM_NAME]
        if found_cocom:
            cocom_menu = ds_menu.pop(found_cocom[0])

        found_colic = [pos for pos, menu in enumerate(ds_menu) if menu['name'] == COLIC_NAME]
        if found_colic:
            colic_menu = ds_menu.pop(found_colic[0])

        found_kafka = [pos for pos, menu in enumerate(ds_menu) if menu['name'] == KAFKA_NAME]
        if found_kafka:
            kafka_menu = ds_menu.pop(found_kafka[0])

        found_community = [pos for pos, menu in enumerate(ds_menu) if menu['name'] == COMMUNITY_NAME]
        if found_community:
            community_menu = ds_menu.pop(found_community[0])

        ds_menu.sort(key=operator.itemgetter('name'))
        omenu += ds_menu

        # If kafka and community are present add them before the Data Status and About
        if kafka_menu:
            omenu.append(kafka_menu)

        if cocom_menu:
            omenu.append(cocom_menu)

        if colic_menu:
            omenu.append(colic_menu)

        if community_menu:
            omenu.append(community_menu)

        # At the end Data Status, About
        omenu.append(self.menu_panels_common['Data Status'])
        omenu.append(self.menu_panels_common['About'])

        logger.debug("Menu for panels: %s", json.dumps(ds_menu, indent=4))
        return omenu

    def execute(self):
        kibiter_major = self.es_version(self.conf['es_enrichment']['url'])

        logger.info("Dashboard menu: uploading for %s ..." % kibiter_major)
        # Create the panels menu
        menu = self.__get_dash_menu(kibiter_major)
        # Remove the current menu and create the new one
        self.__upload_title(kibiter_major)
        self.__remove_dashboard_menu(kibiter_major)
        self.__create_dashboard_menu(menu, kibiter_major)
        logger.info("Dashboard menu: uploaded!")

config.py

params_panels = {
            "panels": {
                "strict": {
                    "optional": True,
                    "default": True,
                    "type": bool,
                    "description": "Enable strict panels loading"
                },
                "kibiter_time_from": {
                    "optional": True,
                    "default": "now-90d",
                    "type": str,
                    "description": "Default time interval for Kibiter"
                },
                "kibiter_default_index": {
                    "optional": True,
                    "default": "git",
                    "type": str,
                    "description": "Default index pattern for Kibiter"
                },
                "kibiter_url": {
                    "optional": False,
                    "default": None,
                    "type": str,
                    "description": "Kibiter URL"
                },
                "kibiter_version": {
                    "optional": True,
                    "default": None,
                    "type": str,
                    "description": "Kibiter version"
                },
                "community": {
                    "optional": True,
                    "default": True,
                    "type": bool,
                    "description": "Enable community structure menu"
                },
                "kafka": {
                    "optional": True,
                    "default": False,
                    "type": bool,
                    "description": "Enable kafka menu"
                },
                "github-repos": {
                    "optional": True,
                    "default": False,
                    "type": bool,
                    "description": "Enable GitHub repo stats menu"
                },
                "gitlab-issues": {
                    "optional": True,
                    "default": False,
                    "type": bool,
                    "description": "Enable GitLab issues menu"
                },
                "gitlab-merges": {
                    "optional": True,
                    "default": False,
                    "type": bool,
                    "description": "Enable GitLab merge requests menu"
                },
                "mattermost": {
                    "optional": True,
                    "default": False,
                    "type": bool,
                    "description": "Enable Mattermost menu"
                },
                "code-license": {
                    "optional": True,
                    "default": False,
                    "type": bool,
                    "description": "Enable Code License menu"
                },
                "code-complexity": {
                    "optional": True,
                    "default": False,
                    "type": bool,
                    "description": "Enable Code Complexity menu"
                }
            }
        }

[inishchith]

@valeriocos Thanks for the suggestions.

• I've updated the PR with the changes, except for the below one:

some name fields contain a .keyword, although it's valid, the other dashboards don't contain them. Please remove them...

• Do you mean origin.keyword? AFAIK it's a field name which is used for bucketing. I'm not sure how to fix it. Please let me know in case you know a way.

• As of this point in time, I've tested the changes with the help of kidash import, I'm yet to test it with micro-mordred

Thanks!.

[valeriocos]

Do you mean origin.keyword? AFAIK it's a field name which is used for bucketing. I'm not sure how to fix it. Please let me know in case you know a way.

origin.keyword should be related to an index created without mappings. Now that the files grimoire_elk/enriched/colic.py and grimoire_elk/enriched/cocom.py are settings mappings for the study index, origin.keyword shouldn't appear anymore.

As of this point in time, I've tested the changes with the help of kidash import, I'm yet to test it with micro-mordred

ok

[inishchith]

origin.keyword should be related to an index created without mappings. Now that the files grimoire_elk/enriched/colic.py and grimoire_elk/enriched/cocom.py are settings mappings ..

Oh. Yes. Thanks for pointing it out.

[valeriocos]

you're welcome!

[valeriocos]

Thank you @inishchith for reworking the PR. I tested with the full chain and it works pretty well. There are some minor issue that should be addressed:

• The selector (first visualization on the left) is empty for both colic and cocom dashboards
  
• The help text visualization is not present in the two dashboards.
• The filter interval_months is still set (thus the dashboards will be empty if the setup.cfg contains a month_interval different from the one set in the filter).
• The name of the dashboards contain an underscore*, for consistency with the other dashboards they shouldn't. You can change it by modifying their titles (e.g., https://github.com/chaoss/grimoirelab-sigils/pull/380/files#diff-93b975ad35b5337f7dbe76e77da5ff3bR21).
• The description of the dashboars is empty*, it would be great to add some text (e.g., Cocom panel by Bitergia). To add the description, you can modify directly the json (e.g., https://github.com/chaoss/grimoirelab-sigils/pull/380/files#diff-93b975ad35b5337f7dbe76e77da5ff3bR5).
• For consistency with the other dashboards, the time_field of the index patterns should be grimoire_creation_date. This requires a tiny modification on the enrichers ([graal] Add support of Graal CoCom & CoLic Backend grimoirelab-elk#672 (review))

*

[inishchith]

@valeriocos Sorry for the delayed response.

1. About selector: I could reproduce the error and found out that there was some mismatch in the fields due to which the interval_months wasn't set to correct index-pattern. I exported a new copy and did a diff to rectify the error.
2. About HelpText: It wasn't added until the push before this comment, sorry about addressing it late.
3. About filter (interval_months): I've removed it in the last push with the selector working fine.
4. About Dashboard names: Sure, I've updated it now to CoCom/CoLic Dashboard.
5. About description: Thanks, I've updated it.
6. About time_field: yet to be updated

[valeriocos]

@inishchith I had some problems to visualize the selector of colic and as you reported the time_field attribute wasn't updated.

Please find the set of dasboards and index patterns fixed here: graal-panels.zip. Once you have tested them locally, we could update them to the incubator instance.

Thanks

[inishchith]

@valeriocos Thanks for resolving the issue.
With the panels that you have shared, I could import and test the results and they work just as good.

I'll update the PR now.

[inishchith]

@valeriocos I've gone through the new panels and they look good to me.
I've updated the PR with the necessary changes.
Thanks!

[valeriocos]

Thank you @inishchith , I'm going to make a final pass

[valeriocos]

@inishchith final pass made on the full chain! Everything works fine, good job!
I used together with this PR:

• [graal] Add configuration for Graal integration in ELK grimoirelab-sirmordred#320
• [graal] Add support of Graal CoCom & CoLic Backend grimoirelab-elk#672
• master branch of graal

[valeriocos]

LGTM, thanks @inishchith

[valeriocos]

@alpgarcia when you have time could you have a look at https://grimoirelab-incubator.biterg.io and review the colic and cocom dashboards? thanks

[valeriocos]

@sduenas feel free to have a look at it

[alpgarcia]

Hi @inishchith,

thanks a lot for your effort!

I miss the documentation files for these new panels. There is some information about how to document them at: https://github.com/chaoss/grimoirelab-sigils/blob/master/CONTRIBUTING.md

The idea is having the corresponding entries in the web page: https://chaoss.github.io/grimoirelab-sigils/

It should be just writing a couple of md files with the right format and place. There are also instructions on how to locally test them using Jekyll.

A really minor comment: I think you need to update your colic help box title :)

As a suggestion, is easy to convert the format of URLs like https://github.com/chaoss/augur in repositories table to clickable links, just by updating the format of the corresponding field in the index pattern. It is a small change but very useful for final users wanting to navigate to the repositories.

Thanks again!

[valeriocos]

Thank you @alpgarcia for the review

[inishchith]

Hey @alpgarcia!
Thanks for the review.

• Documentation: I've added it now. The screenshots will be updated with the newer ones once we've finalized the demo.
• CoLic HelpText: Thanks for pointing it out :) I've corrected it
• Url origins: Good Idea!. I've added it. Is it possible to make changes in the mapping (type) and have similar changes? (currently, I've done it using fieldFormatMap in the panel)

Let me know if any more changes required. :)

Thanks!

[alpgarcia]

Hey @alpgarcia!
Thanks for the review.

• Documentation: I've added it now. The screenshots will be updated with the newer ones once we've finalized the demo.

Perfect! I don't want to stop this PR anymore, I'm fine with this.

I have a suggestion for making the panel easier to understand for "newbies" :) (I'm not requiring this as part of this PR to be accepted, it could be done in a different one, in fact it was my fault not saying this in my previous review).

We have some new panels coming to sigils in the short-term. Their documentation can be found at: https://bitergia.gitlab.io/panel-collections/open_source_program_office/engagement-contributions-overview.html

There you can see a simple structure: an introductory paragraph and then the metric section. Your documentation is almost the same, just need the intro and the Metrics header. About the intro paragraph the idea is having a short description of what the panel provides and/or what you expect the user look at there. Nothing specific, just something to let them know what we are showing.

In the example, it says:

...providing an insight of projects receiving most contributions and allowing to identify both general and more specific trends related to particular data sources, organizations or projects.

As you may see, it says what you could look for in the dashboard (contributions trends). To talk about details, we have the metrics section. This way we hope to engage the users even if they don't understand the metrics at first glance, because they understand the benefits of the panel and thus they should be motivated to going deeper in understanding each and every chart or number.

• CoLic HelpText: Thanks for pointing it out :) I've corrected it

👍

• Url origins: Good Idea!. I've added it. Is it possible to make changes in the mapping (type) and have similar changes? (currently, I've done it using fieldFormatMap in the panel)

This is part of the front end (Kibiter/Kibana), so the only way I know to configure this is through Kibiter/Kibana. The mapping is something related to ES from which Kibana infers their own types for the front end and, afaik, there's no direct correspondence for URLs. On the other hand, doing it from Kibiter/Kibana prevents us from breaking searches because of formatting stuff (as you probably noticed, you can even change the link text that is displayed without affecting the real data stored in ES).

Thanks a lot!

Alberto,

[inishchith]

@alpgarcia Thanks for your comment.

There you can see a simple structure: an introductory paragraph and then the metric section. Your documentation is almost the same, just need the intro and the Metrics header ....

• Sure. I'll have a look at this and update the PR accordingly.

This is part of the front end (Kibiter/Kibana), so the only way I know to configure this is through Kibiter/Kibana. The mapping is something related to ES from which Kibana infers their own types for the front end and, afaik, there's no direct correspondence for URLs ....

• Oh Okay. Got it. Thanks! 😅

[alpgarcia]

Perfect @inishchith, thanks for the effort!

[alpgarcia]

LGTM