Modification: Notary Rubric - Overall changes for the fourth round of elections and onwards

[panges2]

Modification: Notary Application Rubric - changes for fourth round of elections

Issue Description

Some issues of the current rubric used in the previous Notary Election round:

• Some sections did not have a significant effect on the decision-making of notary elections as they did not sufficiently differentiate notary applicants from each other.
• "Final Allocation Amount" calculation is not balanced enough. Too much emphasis is placed on subjective metrics that can be gamed as applicants did not commit to their answers (ie. time commitment in the "engagement in program" section)
• Due to the verbiage and proxy variables, it is not immediately clear to someone which variable is used in which calculation.
• The "Final Allocation Amount" calculation encapsulates many proxy variables, each with its own variables. This can make it difficult to analyze and rebalance for future iterations of the rubric.
• Additionally - in line with Modification: Notary Application Template - Changes for fourth round of elections and onwards #628, changes are required to keep up to date with the ecosystem/program progress.

Impact

These rubric changes will heavily impact how notary applicants are scored in the fourth notary election round. With the new rubric, final scores will have a greater skew toward objective metrics. If done correctly, there will be fewer "one off" cases that have to be handled individually in the fourth election. Going forward, this rubric change would play a part in the efficient scaling of the notary community.

Proposed Solution(s)

Below outlined are a series of changes recommended to the Notary Application Rubric. As noted above, these changes are in line / being proposed alongside a set of changes to the template ( #628 ) and to the process itself (i.e., adding in tiers of notaries). Here is the list of changes proposed:

1. Modify SLA requirements - update "engagement in program" row to reflect changes proposed in Modification: Notary Rubric - Changes to "Engagement in Program" Levels #630
2. Modify allocation rankings - Fix range and rescaling for DataCap requested to "10TiB - 1PiB" in line with the cap that is introduced in Modification: Notary confirmation process - adding 1PiB upper bound for elected notaries #490
3. Remove “diversity and decentralization” section since it is not a reliable metric to score notaries on
4. Retune "Final Allocation Amount" calculation:
   MAX(FLOOR(0.2*Service Level Agreement+ 0.2*Long term network alignment + 0.2* Industry Reputation + 0.4*Correcteness of Allocation Plan), Track Record)
   Rationale:
   a. Other variables are rescaled since “diversity and decentralization” variable is deleted.
   b. Influence of "Correctness of Allocation" variable is strengthened as it is the most objective metric.
5. Minor updates:
   a. Move Allocation Ranking to the bottom after final allocation row
   b. Remove "scale of allocation" in final allocation calculation as people could misunderstand it as a variable
   c. Remove proxy variable "Weighted Notary Leveling". It is no longer required since “diversity and decentralization” has been removed.

Link to proposed rubric: https://docs.google.com/spreadsheets/d/1GOqK1gmBaUyfvSBMYfwoWSBy7Cv79Ac2g8kppC_3kAE/edit?usp=sharing

Timeline

Ideally, ASAP, so Notary applicants can begin preparing their round 4 election applications.

Technical dependencies

None - just need to merge a PR to the Rubric.

End of POC checkpoint (if applicable)

N/A

Risks and mitigations

N/A - all these changes are easily reversible.

Related Issues

#628
#630

[Bitengine-reeta]

Is the "allocation ranking" level based on the final score or engagement expectations/SLA?

[panges2]

Hi Bitengine-reeta, The "allocation ranking" level is based on the final score. The SLA score is used to calculate the final score. Let me know if you still find anything unclear/ if you have more issues!

[guoqingjiehappy]

1. how to join a working group? what should we do in the working group？
2. how to make an allocation plan of score 5,like score every client based on certain criteria, then L1 for 50TiB, L2 for 100TiB？ But the first round allocation is always 50% of weekly allocation requested as I searched the past LDN application, that makes our allocation plan invalid.

[XindyTan]

If we choose L5, but ultimately the system determines that we can't reach it. To the extent that we are capable, will we be judged as L4? Or we will get nothing? Thank you for your help.

[panges2]

@guoqingjiehappy

1. Plans for working groups will be announced after the elections. Be on the lookout for more information on this coming soon.
2. I think you are getting confused between LDN applications. The allocation we are talking about in the rubric is Direct Allocation for notaries. So it is the amount that notaries are allowed to give clients directly without going through the LDN multisig. This is different from What clients are getting from LDN.

[guoqingjiehappy]

@guoqingjiehappy
2. I think you are getting confused between LDN applications. The allocation we are talking about in the rubric is Direct Allocation for notaries. So it is the amount that notaries are allowed to give clients directly without going through the LDN multisig. This is different from What clients are getting from LDN.

First, thanks for quick reply.
No，I am refering to Allocation Strategy -- Please describe your allocation strategy with as much specificity as you can. This includes the target amount per client and rate at which you'll allocate DataCap.
I noticed that some notaries applied before make some rubrics for clients, like L1 egilible for 50 TiB , L2 for 100TiB

[panges2]

@XindyTan you will be judged L4. To clarify he descriptions stated in the rubric are minimum requirements to be at a certain level.

[panges2]

@guoqingjiehappy Sorry, I'm not understanding your question. I don't see why this would make your allocation plan invalid.

[guoqingjiehappy]

when will the notary application template be updated? and when the notary application can be submitted? last governance call warned us not no submit it by Novermber 1st, but today I saw the template is still the old one.

[Kevin-FF-USA]

@guoqingjiehappy
We are closing this issue after discussion and comment from Fil Lisbon Summit. @panges2 and I will be incorporating the changes into the final v4 edits into the scoring which will go live tonight