Github ←→ Fil+ system design: enhancements to transparency and structure

[orvn]

As one of the maintainers of the filplus.storage site as well as other adjacent Filecoin initiatives, I have some potential improvements to the Github system design and integration I'd like to draw attention to.

These pertain to the two repos that handle DataCap application tracking and management:

1. filecoin-plus-large-datasets aka the LDN repo
2. filecoin-plus-client-onboarding aka the GA repo

Benefits of these improvements

The proposed improvements all share similar benefits and further the Filecoin Plus mission, they:

• Improve maintainability and readability
• Undo some of the current technical debt
• Reduce future technical debt
• Create a more efficient system while also increasing community transparency
• Decrease centralization

The items below are written the these qualities in mind.

---

1. Repo synchronicity

One of the challenges we come across repeatedly is the fact that there are two repos: one for general DataCap applications (GA) and another for large DataCap applications (LDN).

This makes maintenance and enhancement challenging. Each time something is worked on in one of the repos, the other must be brought up-to-date and appropriately tested.

As a result of this, at some point the two repos diverged from one another. This continues to create technical debt, in that one repo advances more than the other, when in reality there's no reason why it couldn't benefit from the same improvements.

There are a three options for how to architect the relationship between the repos:

A. Limited shared functionality

• This is the current approach, where the repos have similar features, but the logic between them is different enough to where we need variation between the bots, different labels, etc.
• This is the least maintainable of the three approaches

B. Increased shared functionality

• The repos could be kept in lockstep by sharing as much of their functionality as possible, including:
  • Labeling system and state flow
  • Bots and automations
  • Tests and actions
  • Format and data structure
• Essentially the repos would behave like two different instances of the same abstract class, ensuring they're always aligned
• With this approach one repo would likely be considered primary, and the other could periodically sync shared code from it

C. Unification

• The two repos would be combined into one, with all DataCap applications flowing through it
• This would mean less technical debt, less maintenance, and fewer considerations when implementing new features
• Since the questions between the DataCap applications are different, two different data structures may need to be supported
• Everything else could be easily shared: labels, tests, automations, data models, etc.
• This is the DRYest approach, but it involves the most change
• Changing this now would mean significant technical debt reduction in the future

2. Label management

Right now we use Github as a public platform, and then have outside code that manages state.

Using Github is a good approach for openness and standardization, however low cohesion is a risk when having external code act as a controller that manages labels and tracks issues.

This is still appropriate, but we should also strive to create transparency around label behavior and state changes.

• Good documentation in-repo
• Clear label flow outlined in-repo
• Bug-free label behavior and minimal superfluous state changes

@panges2 has a useful post on new labels on Medium, so I propose that both repos follow it, and document it within the codebase.

However, one of the major challenges right now is that issues are difficult to follow due to excess bot activity. Moreover, some of the bot activity is buggy:

• Labels that signify type are arbitrarily removed
• Previous state labels not always removed when a new state applies
• Bot can post excessively, implying a race-condition-like problem
• Issues don't always reach their final state

These challenges apply to both labels for issues and would apply to labels for pull requests (described further in the next section)

3. Pull-request-based approach

As proposed previously (here and here), maintaining DataCap applications as in-repo code would be much better than treating Github issues as applications. For this to happen a new flow is needed where users PR a branch with their changes against the base branch.

For this to happen effectively we need the following:

• An agreed-upon schema (see #839) that can apply to both applications, or two different schemas
• Shared data model templates and their associated documentation
  • For example, a shared list of countries, regions, tooling, etc.
• A skeleton (minimal object with mandatory keys only) that users can copy to submit new DataCap applications
• An external service (bots) which submit PRs that update DataCap application state (in addition to continuing to manage labels)
• Validation scripts that can automatically test whether submitted PRs follow a valid structure, which can be run using a Github Action
  • There's no reason why our validation logic shouldn't be publicly available within the repo as a script, and have accompanying documentation
  • This way other services will be able to use the same validation logic, and it won't be hidden within an external service

---

Closing notes

I understand that this could seem like a lot of points, but in reality they're all inter-related and many of them could be implemented in tandem.

The benefits of many of these enhancements will be immediate, long-lasting, and help move us toward a healthier Filecoin Plus ecosystem.

Not addressing them on the other hand will almost certainly lead to more circular problems, bugs, and longer maintenance periods.

Thanks for bearing with me on this long read and let me know if you have questions or thoughts!

[galen-mcandrew]

Agree with most of the points raised, and work has been progressing on these changes. We are still testing and developing the transition from Issue to PR with JSON as the public database source of truth. Generally, the structure here would be:

• JSON files to track actors (notaries, SPs, clients) and allocation history
• PRs for each allocation request, where a merged PR happens as a result of DataCap being allocated (or revoked)
• PRs for change in actor details, for example merging a new notary contact info

Once we are ready to implement and roll-out this major technical update, we will likely include a variety of additional changes as well. Some examples that will still need to be discussed:

• Migrating to a new single repo for all Allocations, with a second parallel repo for Governance (proposals, announcements, decision-making)
• Deprecating current open issues and allocations, rolling historic allocation data into the JSON format
• Possible changes to the structure of General/Individual Notary Allocations, as well as the process and parameters of LDN allocations

As I said, these are just some possible ideas and changes, which will still be proposed and processed through community governance calls. The overall goals of these changes (making the shift to JSON, changing repos, and updating how bots/labels interact) will always be: programmatic openness, transparency, and efficiency.

I strongly encourage everyone (especially our tooling and dashboard teams) to continue this conversation and help us make sure the transition to JSON/PR captures all the necessary program requirements.

[orvn]

That sounds good @galen-mcandrew, I've included some suggestions on how we can move this forward in my comment in #839, but I'll also add some data structure considerations to the json that might optimize it a little further!