Ideation: Notary reputation system

[panges2]

As the Filecoin Plus program shifts focus towards Quality and recent events have led to the removal of notaries, the idea of a notary reputation system has become more relevant. A notary reputation system is essentially tracking system that rates notaries based on their actions, against a set of rules or policies that the community agrees upon. This reputation system can be split into several components: Metrics & Signals, Metric Aggregation, Feedback, and Incentives. In addition to these components, we should also evaluate the pros and cons of implementing such a reputation system and outline possible next steps if the idea moves forward.

Metrics and Signal

The notion of a "good" notary is highly debated in the community, making it challenging to measure a notary's reputation directly or based on a single metric. A system that bases a notary's entire reputation on a single metric is easy to game, and thus it is crucial to have a large set of diverse metrics. The larger the set, the more difficult it is to manipulate the system. Potential metrics that can be incorporated into the system include:

• Already implemented:
  • Notary governance call attendance
  • Notary time to datacap
  • Notary Score during election
  • Notary application approval history
  • List of unique SPs associated with the notary (based on clients approved by the notary and where they store data)
  • Data from E-Fil+ pilot analysing behavior of notaries being randomly assigne dto applications
• Will be implemented:
  • Notary time to engagement (using the new notary due diligence templates to measure notary engagement with an application, including signing or declining)
  • Reputation of clients approved by the notary (when more client reputation metrics are available)
  • Onboarding quality of deals associated with the notary according to CID-checker and abuse metrics
• Would be interesting to implement:
  • Notary flagging/upvoting system to provide data on clusters of notaries approving or disapproving each other

Metric Aggregation

Collecting and utilizing the various metrics effectively is a crucial part of the reputation system. There are several approaches to aggregating metrics for decision-making:

• Reputation score: Create a single score by calculating a weighted average of all metrics, where the weights indicate the importance of each metric. For this implementation to be successful it is important to normalize the metrics to ensure they fall within a consistent range. Additionally, adjusting the final score relative to the average or median score allows for a more accurate identification of discrepancies and variations in notary performance.
• No aggregation: Present the set of metrics without any aggregation, leaving it up to the reader to interpret and evaluate the notary's performance based on the available data.
• Reputation grade: Instead of presenting a numerical score, classify notaries into different grades by clustering scores into buckets. This approach simplifies the assessment of notary performance while still providing a useful overview of their reputation.

Each aggregation method has its advantages and challenges, and the choice of approach will depend on the community's preferences and goals for the notary reputation system.

Feedback

Incorporating feedback based learning into the reputation system can enhance its accuracy and effectiveness, especially over time. If we employ a weighted average function to calculate reputation scores, we can adapt a concept from artificial intelligence called "backpropagation." This method adjusts the weights of the function based on external feedback, enabling a more precise representation of a notary's reputation.

By gathering external feedback, primarily from manual reports, we can fine-tune the weights assigned to different metrics. For instance:

• If the trust and transparency group reports a notary, and that notary has a high score in the "Notary time to datacap" metric, we can slightly reduce the weight assigned to that metric, recalculating the scores accordingly. This indicates that "Notary time to datacap" may not be a reliable indicator of "good" notaries.
• If a client associated with a notary has their datacap removed, we can assume that the notary may have wrongfully issued datacap to that client. In response, we can slightly lower the weight of the metric in which the notary scored highest, adjusting the overall reputation score.

This feedback learning process helps refine the reputation system over time, making it more responsive and accurate in evaluating notary performance.

Incentives

For stakeholders to take the reputation system seriously, there must be tangible consequences based on their reputation within the system. High-reputation notaries should be rewarded, while those with low reputations should face penalties.

Two critical aspects to consider when designing rewards and penalties are the timing ("when" to reward or penalize) and the type of incentives or sanctions to apply ("How" to reward or penalize). Diving into the topic of notary incentives and penalties is a deep and intricate matter, warranting further expansion in a separate ideation discussion.

1. Timing:
   • One option is to apply rules based on scores, e.g., penalize notaries with the lowest grade or reward the top 10 notaries.
   • Alternatively, the governance team could manually interpret the reputation metrics and decide when to issue rewards or penalties.
   • A hybrid approach would be best as it would involve systematically flagging "good" and "bad" notaries, while still allowing the governance team to have the final say on who receives rewards and penalties.
2. Types of rewards and penalties:
   • Monetary incentives can be effective but may also encourage notaries to game the system to receive these rewards. This could involve notaries staking a certain amount that may be slashed.
   • Social incentives, such as a notary leaderboard, could be used to encourage better behavior by highlighting top performers and discouraging bad actors.
   • Penalties could include limiting a notary's responsibilities. For example, if a notary is suspected of misconduct, they could be required to complete due diligence on random applications, akin to a "probation period."

By carefully considering these aspects, a balanced and effective rewards and penalties system can be established to encourage notaries to maintain high standards and actively contribute to the Fil+ community.

Remarks and Possible Next Steps

A well-designed reputation system can foster trust, self-accountability, and encourage notaries to uphold the highest standards within the Fil+ community. Such a system is both decentralized and scalable, minimizing the need for centralized authority in the long term.

The ideas and concepts discussed in the previous sections can be generalized to create reputation systems for other stakeholders in the Filecoin community, such as clients and storage providers.

One potential drawback of a reputation system, however, is that it requires a substantial amount of data and a large number of notaries to maintain proper calibration. If the community moves toward fewer notaries, the impact of the reputation system may be limited.

At present, the reputation system remains an idea in its early stages. Some metrics are being implemented and can be found on the notary leaderboard at notaries.datacapstats.io or datacapstats.io. Additional metrics will be introduced in the future, as mentioned in the metrics section. Regardless of whether a comprehensive reputation system is built, having access to these metrics enhances transparency within the Fil+ community.

Moving forward, the community can continue to discuss and refine the concept, explore the feasibility of implementing additional metrics, and evaluate the overall benefits and potential drawbacks of a notary reputation system.