We list and describe activity metrics. For different situations, the metrics have different meanings. Each activity metric can be used in association with other metrics in the creations of value-oriented health metrics. We have a metric-template to be used for new detail metric pages.
| Name | Description |
|---|---|
| Accepted Code Contributions | Percentage of new contributor code versus total code over time. |
| Age of Community | Time since repository/organization was registered; or time since first release. "Results showed that the age of the project played a marginally significant role in attracting active users, but not developers. We attribute this differential effect of age on users and developers to the fact that age may be seen as an indicator of application maturity by users, and hence taken as a positive signal, whereas it may convey more ambiguous signals to developers." (Chengalur-Smith et al., 2010, p.674) (Grewal, Lilien, & Mallapragada, 2006). |
| All Licenses | List of licenses. |
| Apache Maturity Model | The Apache Project Maturity Model provides guidelines for assessing the maturity of a project. |
| Availability of Add-on Products | Availability of 3rd party plug-ins, modules, utilities, etc. that provide additional functionality for the project's software. |
| Blogposts | Number of blogposts that mention the project. |
| Average Time of First Maintainer Response to Code Merge Request | The average amount of time it takes for a maintainer to make the first response to a code merge request. |
| Average Time of First Response to Issue | The average amount of time it takes for the first response to an issue. |
| Average Time of Open Issues | The average amount of time open issues have remained open. |
| Average Issue Resolution Time | The average amount of time it takes for issues to be closed. |
| Bug Age | Age of known bugs in issue tracker. Use label for determining bugs? |
| Bugs after Release | Number of bugs reported after a release. |
| Bus Factor | The number of developers it would need to lose to destroy its progress. Alternatively: Number of companies that would have to stop support. |
| Change in Maintainer Number | Number of maintainers added/removed over time. |
| CII Best Practices Badge | The CII Best Practices Badge Program provide maturity self-certification: passing, silver, and gold. |
| Closed Issues | What is the number of closed issues? |
| Closed Issue Resolution Duration | What is the duration of time for issues to be resolved? |
| Code Commits | What is the number of code commits? |
| Code Merge Duration | What is the duration of time between code merge request and code commit? |
| Code Modularity | Modular code allows parallel development, which Linus Torvalds drove for Linux (OSLS Torvalds). (Baldwin & Clark, 2006). |
| Code Review Efficiency | What is the number of merged code changes/number of abandoned code change requests? |
| Code Review Iteration | What is the number of iterations that occur before a merge request is accepted or declined? |
| Code Reviews | What is the number of code reviews? |
| Commercial Offerings | Availability of commercial products or services based on the project. |
| Commit Bias | Acceptance rate (and time to acceptance) differences per gender, ethnicity, and relevant diversity characteristics. |
| Community Activity | Contribution Frequency. Contribution = commit, issue, comment, etc.). |
| Contributing Organizations | What is the number of contributing organizations? |
| Contribution Acceptance | Ratio of contributions accepted vs. closed without acceptance. |
| Contribution Age | Time since last contribution. Contribution = commit, issue, comment, etc.). |
| Contribution Diversity | Ratio of code committed by contributors other than original project initiator. Contributions going up beyond the core team. |
| Contributor Activity | Activity level of individual contributors. |
| Contributor Breadth | Ratio of non-core committers (drive-by committers). Can indicate openness to outsiders. |
| Contributor Demographics | Gender, age, location, education, and skills over time. |
| Contributor Diversity | Ratio of contributors from a single company over all contributors. Also described as: Maintainers from different companies. Diversity of contributor affiliation. |
| Contributor Importance | Percentage of commits by individual contributors from identified organizations over time. |
| Contributor Seniority | For each active contributor, time since first contribution. Experienced contributors can provide value to the community, since they carry with them (in part) the history of the project. |
| Contributors | What is the number of contributors? |
| Copyright Declaration | The degree to which the project properly declares copyright ownership, including the copyright symbol or 'copyright' word, the year of the creation, the name of the author, and a rights statement. |
| Decision Distribution | Central vs. distributed decision making. Governance model, scalability of community. |
| Dependency Depth | Number of projects included in code base + number of projects relying on focal project (recursive). Indicator about centrality in open source Dependency network. |
| Distribution of Work | How much recent activity is distributed? |
| Downloads of Non-software Artifacts | Number of downloads of non-software artifacts (e.g. documentation, sample apps, test suites, etc.). |
| Elephant Factor | If 50% of community members are employed by the same company, it is the elephant in the room. Formally: The minimum number of companies whose employees perform 50% of the commits |
| File License Declarations | A list of license declarations on the software package files. |
| Followers | Number of followers (GitHub). |
| Forks | Number of forks. |
| Gatherings | Number of face-to-face/in-person meetings per year. Resets contentious issues; Resolve tensions; Avoid longstanding grudges. |
| Installs | Number of software installations of the project. |
| Issue Comments | Number of comments per issue. |
| Issue Resolution Efficiency | What is the number of closed issues/number of abandoned issues? |
| First Response to Issue Duration | Time between a new issue is opened and a maintainer responds. Also called: bug response rate. The maintainer is believed to not "pile on" but try to solve an issue. |
| Issues submitted/closed | Issues submitted vs. issues closed. Example. |
| Job Postings | Number of job postings that mention the project as a preferred or required skill. |
| Known Vulnerabilities | Number of reported vulnerabilities. Could be limited to issue-tracker or extended vulnerability databases (e.g. CVE). |
| Language Bias | Bias against gender, ethnicity, ... in use of language (maybe use sentiment analysis). |
| Language Makeup | Makeup of a project in terms of whitepsace, code, and comments. |
| Leadership Demographics | Demographics of project's leadership (e.g. Board, Technical Steering Committee, Maintainers, etc.) over time. |
| License Conflicts | Project containing incompatible licenses. |
| License Count | Number of licenses. |
| License Coverage | Number of files with a file notice (copyright notice + license notice). |
| License Declared | What license does the project declare. |
| License Identification Methods | A list of methods or tools used for identifying licenses in files. |
| Lines of Code Changed | What is the number of lines of code changed? |
| Maintainer Promotion | Last time a maintainer was added. |
| Maintainer Response to Merge Request Duration | What is the duration of time for a maintainer to make a first response to a code merge request? |
| New Contributing Organizations | What is the number of new contributing organizations? |
| New Contributions | Percentage of contributions (patches, pull requests, etc.) from new contributors vs all accepted contributions over time. |
| New Contributor Organizations | New organizations contributing to the project over time. |
| New Contributors | What is the number of new contributors? |
| New Contributors* vs Maintainers** | Ratio of new contributors to maintainers over time. |
| Non-Source Contributions | Track contributions like running tests in test environment, writing blog posts, producing videos, giving talks, etc... |
| Number of Active Users | Number of active users of the project. |
| Number of Contributing Organizations | Number of organizations participating in the project over time. |
| Onion Layers | Distance between onion model layers (users, contributors, committers, and steering committee). Rule of thumb: factor of 10x between layers. (OSLS'17 Node.js keynote). |
| Open Issues | What is the number of open issues? |
| Open Issue Age | What is the the age of open issues? |
| Package License Declaration | A list of license declarations on the software package. |
| Paid Developers | Number of paid developers in community over time. |
| Path to Leadership | A communicated path from lurker to contributor to maintainer. (or. track members: time from user to maintainer/leader). If active contributors are not included in leadership decisions they might lose interest and leave. (Focus on least likely contributor). |
| Path to Maintainership | Path to maintainership published. |
| People Opening Issues | How many people are opening issues. |
| Project Life Cycle | Community assigned label. Some communities have a project life cycle for example: proposal, incubaton, active, deprecated, end of life (Source: Hyperledger). |
| Percentile Distribution of First Maintainer Response to Code Merge Request | The proportional frequency of time it takes for a maintainer to make the first response to a code merge request. |
| Percentile Distribution of First Response Time | The proportional frequency of time it takes for the first response to an issue. |
| Percentile Distribution of Issue Resolution Time | Proportional frequency of closed issue time duration. |
| Percentile Distribution of Open Issue Time | Proportional frequency of open issue time duration. |
| Percentile Distribution of Time to Merge Code | Proportional frequency of code merge to upstream time duration. |
| Pony Factor | The minimum number of developers performing 50% of the commits. The Math |
| Pull Request Comments | Number of comments per pull request. |
| Pull Request Comment Duration | The difference between the timestamp of the pull request creation date and the most recent comment on the pull request. |
| Pull Request Discussion Diversity | Number of different people discussing each pull request. |
| Pull Request made/closed | Pull requests made vs. pull requests closed Example. Encompasses number of pull requests rejected. |
| Pull Requests Open | Number of open pull requests. Perhaps more telling than total pull requests. |
| Pull Requests Over Time | How many pull requests have been submitted over a specified time period? |
| Qualified Committers | Contributions over time and what components they commit to over time. |
| Relative Activity | Sum up the activities (GH issues+comments, GH pull requests+comments and GH commits) for the project members and for the non-project members, then create a ratio of the two. Compare the activity between committers-as-a-group and contributors-as-a-group. It easily shows when a project is not yet popular, or when a project is not paying attention to its users. I also feel that a balance between the two groups is essential; ie) a project with a lot more contributor than committer activity is one that is failing to recruit committers quickly enough. |
| Release Maturity | Ratio of major and minor releases. |
| Release Note Completeness | Number of functionality changes and bug fixes represented in release notes vs. release. Good for users, also shows diligence of community. |
| Release Velocity | Time between releases. Regular releases are a reliability metric. |
| Reopened Issues | Rate of issues closed but discussion continues or issues that were closed and re-opened. |
| Repository Size | Overall size of the repository or number of commits. |
| Retrospectives | Existence of after release meetings. Collect lessons learned, improve processes, recognize contributors. |
| Review Efficiency | Number of merged patches / number of abandoned patches over a set period of time. |
| Rewards | Rewards, shout-outs, recognition, and mentions in pull-requests or change logs - might improve contribution levels. |
| Roadmap | Existence and quality of roadmap. Best practice as community engagement and scalability (might not be automatically computable). |
| Role Definitions | Existence and quality of role definitions. Governance related. Relates to "Path to Leadership". |
| Size of Code Base | Lines of code. |
| Software Downloads | Number of project software downloads. Beware: downloads might be skewed by builders. Used as measure for success (Grewal, Lilien, & Mallapragada, 2006). |
| Stack Overflow | Several metrics: # of questions asked, response rate, number of responding people that have verified solutions. |
| Stars | Number of stars (GitHub). |
| Sub-Projects | What is the number of sub-projects? |
| Test Coverage | Percentage of codebase covered by developer tests. |
| Time to Contributor | Time to becoming a contributor. |
| Total Contributing Organizations | The total number of organizations contributing over time. |
| Total Contributors | The total number of contributors over time on any platform. |
| Total New Contributing Organizations | The total number of new organizations contributing over time. |
| Total New Contributors | The total number of new contributors over time on any platform. |
| Total (Sub-)Projects | The total number of (sub)projects over time. |
| Transparency | Number of comments per issue. Discussion is occuring openly - could also indicate level of agreement. |
| Unity | Rivalry or unity of community (sentiment analysis). |
| Update Age | Time since last update. |
| Update Rate | Number of updates over period x. |
| Update Regularity | How consistently and frequently are updates provided. |
| Use of Acronym | Frequency of acronyms used. Specialized language can be a barrier for new contributors. |
| User Dependency | Number of users who are aware that they depend on the software over time. |
| User Groups | Number of user groups that perform a variety of crucial marketing, service support, and business-development functions at the grassroots level\ (Bagozzi & Dholakia, 2006) |
| V-index | An index of project's first-order and second-order downstream dependencies. For example: if a project has 15 downstream dependencies and those downstream projects have 15 downstream dependencies of their own, the V-index is 15. |
| Velocity | A graph where bubbles’ area is proportional to the number of authors, the y-axis (height) is the total number of pull requests & issues, and the x-axis is the number of commits. Example |
| Watchers | Number of watchers (GitHub). |
| YouTube Videos | Number of Youtube videos that mention or specifically deal with the project (e.g. tutorials). |
The following section are a collection of random thoughts that might be helpful in a future discussion.
This includes reasons why metrics are considered for other reasons This section collects notes on what possible goals might be.
- Track Corporate Engagement (is an organization creating value, are organizational goals met, employee contributions)
- Risk mitigation
- Identify open source projects that need support.
- Identify single points of failure (and hopefully prevent them)
- Assess value generated through community and engagement
- Show that active community management bears desired results. (Measurable outcomes)
- Avoid in-take of an inactive project, because it makes it difficult to maintain and might carry unknown bugs and security issues.
- Sustainability: "we define a sustainable project as one that exhibits software development and maintenance activity over the long run." (Chengalur-Smith, Sidorova, & Daniel, 2010, p.660)
These categories are meta-concepts that cannot be assessed with a single metric and the combination of metrics will be different depending on the context.
- Growth of community
- Momentum of community
- Timeliness of maintainers
- Diversity of community, contributions, and in code base
- Distribution of code contributions (beyond project creator)
- Activity level - Responsiveness
- Viability (e.g. Bus Factor - individual contributors and clustered by employer)
- Maturity
- Ecosystem health (upstream, downstream, and related projects)
- Vanity metrics (might have use in other cases, e.g. stars)
- Aggregate project-tree health (combined health metrics of all linked dependencies)
- Attentiveness of maintainers to users. See Mailing list
- Style of project
- Programming language
- Maturity of project (Projects might seem inactive but rather have fulfilled their goal and community remains responsive to bug reports and security issues, just no new features)
- Quality of Ecosystem (metrics of related projects)
- Value driven metrics (not just activity)
- Development of metrics over time
- External users might not be a homogenous group - consider different metrics
- Compare similar projects (manually determine which projects to compare)
- Classifications (based on a set of metrics, which projects 'behave' similar)
- Interrelationships between categories of indicators (maturity might be high while activity low and response rate is up)
- Aggregate from repository, to project, to community, (to company)
We have heard other classifications that we simply list here.
Ideas for these classifications is to
- generate a uniform classification and through conversations merge the different classifications.
- create mappings of the indicators to the different classifications
- Community/Code/Risk
- Activity/Viability/Risk
-
Arantes, F. L., & Freire, F. M. P. (2011). Aspects of an open source software sustainable life cycle. In IFIP International Conference on Open Source Systems (pp. 325–329). Springer. Retrieved from http://link.springer.com/chapter/10.1007/978-3-642-24418-6_26
-
Bagozzi, R. P., & Dholakia, U. M. (2006). Open Source Software User Communities: A Study of Participation in Linux User Groups. Management Science, 52(7), 1099–1115. Retrieved from http://www.jstor.org/stable/20110583
-
Baldwin, C. Y., & Clark, K. B. (2006). The Architecture of Participation: Does Code Architecture Mitigate Free Riding in the Open Source Development Model? Management Science, 52(7), 1116–1127. Retrieved from http://www.jstor.org/stable/20110584
-
Chengalur-Smith, I., Sidorova, A., & Daniel, S. (2010). Sustainability of Free/Libre Open Source Projects: A Longitudinal Study. Journal of the Association for Information Systems, 11(11). Retrieved from http://aisel.aisnet.org/jais/vol11/iss11/5
-
Cosentino, V., Izquierdo, J. L. C., & Cabot, J. (2015). Assessing the bus factor of Git repositories. In 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER) (pp. 499–503). https://doi.org/10.1109/SANER.2015.7081864
-
Crowston, K., Howison, J., & Annabi, H. (2006). Information systems success in free and open source software development: theory and measures. Software Process: Improvement and Practice, 11(2), 123–148. https://doi.org/10.1002/spip.259
-
Grewal, R., Lilien, G. L., & Mallapragada, G. (2006). Location, Location, Location: How Network Embeddedness Affects Project Success in Open Source Systems. Management Science, 52(7), 1043–1056. Retrieved from http://www.jstor.org/stable/20110579
-
Jansen, S. (2014). Measuring the health of open source software ecosystems: Beyond the scope of project health. Information and Software Technology, 56(11), 1508–1519.Retrieved from http://www.sciencedirect.com/science/article/pii/S0950584914000871
-
Schweik, C. M., & English, R. C. (2012). Internet success: a study of open-source software commons. Cambridge, Mass.: MIT Press.