Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🚧 | B3: Turn B1 into production-ready system for core actors and FVM #6

Closed
jsoares opened this issue Dec 6, 2021 · 23 comments
Closed

🚧 | B3: Turn B1 into production-ready system for core actors and FVM #6

jsoares opened this issue Dec 6, 2021 · 23 comments
Assignees
@adlrocha
Labels
Epic 🚧 Project

Comments

@jsoares
Copy link
Contributor

jsoares commented Dec 6, 2021 •
edited
Loading

@jsoares commented on Wed Nov 24 2021

Description

Turn the hierarchical consensus system prototype developed in B1 (#4) into a production-ready system.

Scope

  • smart-contract based lifecycle of spawning/killing subnets
  • enable execution of actual Filecoin actors
  • interface with the rest of Filecoin (PoRep, PoST)

Note that actual production deployment is beyond the scope and depends on the FIL team.

Dependencies

Resources

Papers

soon™️

Talks

2022-06-22, Hierarchical consensus: A horizontal scaling framework for blockchains

Watch the video

Demos

2022-04-07, Eudico garden: a devnet for hierarchical consensus, running Tendermint, PoW, and Filecoin root consensus

Watch the video

2022-07-08, Draft of the Hierarchical Consensus spec

Watch the video
@jsoares jsoares added the Epic label Dec 6, 2021
This was referenced Dec 6, 2021
Closed
Closed
@jsoares jsoares mentioned this issue Mar 16, 2022
Closed
@jsoares
Copy link
Contributor Author

jsoares commented Mar 16, 2022

With the end of project B1 (#4), we're kickstarting B3 this week! More details coming soon.

@jsoares
Copy link
Contributor Author

jsoares commented Mar 22, 2022

2022-03-21

✋ Attendees

📣 Updates

  • Fixed checkpoint bug.
  • PR for Filecoin consensus almost ready.
    • Pending: bottom-up messages not being applied in Filcns (probably done today).

🧵 Discussion

🎯 Up next

  • Spawn first instances of Eudico garden
    • 3-5 nodes network. Filcns rootnet. Some automation scripts.
  • Basic itests for hierarchical consensus

@jsoares
Copy link
Contributor Author

jsoares commented Mar 29, 2022 •
edited
Loading

2022-03-29

✋ Attendees

📣 Updates

  • Progress on testing coverage for hierarchical consensus
    • Planned atomic execution happy-path for hacking time in Paris on-site. This would give us some basic coverage to set-up CI and catch when we break things.
  • Eudico garden PR ready to merge
    • Waiting for some input on how to add new miners in runtime before merging (see if we can have that feature in this first MVP).
  • Following closely FVM developments.
    • rc3 is out. Still some unstabilities. Will wait to rebase after network upgrade.
    • Mainnet upgrade scheduled for 10 May
  • Cryptoecon Sprint.
    • We should maybe target writing a complete design document after receiving their inputs.
      • We have widespread ideas but we don't have a complete model.
      • I am concerned that the theoretical model may be disregarding low-level technical details.

🧵 Discussion

  • Thinking about CL Open Problems/Projects backlog. Hope to at least gather some links about what others are doing (internally and externally) around:
    • Project: Pluggable gas models.
    • Project: Pluggable consensus
    • Project: Pluggable mpools.
    • Stateless consensus
      • Low hardware requirements for clients and consensus nodes.
    • Data availability
    • Sponsored transactions
      • User transactions can be sponsored by third-parties (requirement for big corps).
      • L1 EIP3074-like
      • L2/Subnet-based protocol
    • Light consensus
      • DAG-based/Lurk/Crypto-based.
    • Verifiable computation/execution/state changes
    • Parallel execution
      • Vivien's review of the state of the art.
    • Hierarchical/Mergeable Consensus
      • Network partitioning.
      • Local consensus / Offline-first consensus
      • State migrations between networks.
    • Parallel resource commitment protocols (no need for update messages/prevent majority censorship).
      • HC Subnet for proof commitment in Filecoin?
    • Big projects to source open problems:
      • Incentivized P2P transport protocols (message/packet propagation)
      • "Consensus-as-a-service".

🎯 Up next

  • Cryptoecon: mapping modules to our HC framework (first pass on Wed)
  • Paper reading, survey open problems (contribute to Discussions)
  • Itests support
  • (BONUS) Kick-off SCA for FVM.

@jsoares jsoares mentioned this issue Apr 7, 2022
Closed
@adlrocha
Copy link
Contributor

2022-04-11

✋ Attendees

📣 Updates

  • Hierarchical Consensus accepted in DINPS'22 🎉
  • Camera ready 21st of April
    • I don't think needs much change in this version. High-level criticism
      • Lack of performance evlaution: "no evaluation at all"
      • Better framing with state of the art: "it could do better on placing the work in the context of existing work and evaluating how it improves the state of the art."_
  • Integrated Circle CI to Eudico
  • Fixed several minor bugs on HC

🧵 Discussion

  • Plan HC full paper with cryptoecon model, data availability, and performance?
    • Financial Crypto-kind crypto? Target something around September?
    • What to do with state and the firewall requirement?
      • Prioritize research on Cryptoecon and slashing policy.
      • (No focus) Evaluation throughput-wise and evaluation-wise.
      • Data availability needs to be there. The moment you read the paper. What if the subnet goes rogue and then disappears?
      • The paper will write itself with the work.
      • Look at progress for CNL and CCL (lurk).
  • Staffing: Eudico/Eudico Garden/B1 maintainership.
    • Current maintainership tasks may affect progress of B3.
  • DGM grant for Eudico Forest (Rust)
    • It would allow us integrate FVM without FFI
    • Leverage current Narwhal implementation
    • Explore more advanced use cases we've been discussing: single consensus implementation targeting FVM.
  • Slashing in HC
    • Lurk requirements and explore it so we don't have to sync with children.

🎯 Up next

  • Try to wrap-up all loose ends from Eudico and B1 to focus on FVM integration
  • Slashing policy: lurk?

@adlrocha
Copy link
Contributor

adlrocha commented May 3, 2022

2022-05-02

✋ Attendees

📣 Updates

  • Progress of FVM integration: https://hackmd.io/x7r4r5_rQja8rNAIxvcryA
    • SCA in the end being implemented as a builtin-actor (at least for now)
      • Reason: we want it to be deployed from scratch in every new network and with a well-defined address known a-priori.
    • Lifecycle functions done.
    • In the process of writing unit tests for checkpoint functions.

🧵 Discussion

  • Any news on the Lurk collaboration? Should we revive it?
    • MV: I was a blocker. Will unblock by EoD tmw.
  • MV: Alfonso please figure out if you are going to be blocked by FVM
    • not until Eo May, beg of June.

🎯 Up next

  • Checkpoints functions of SCA implemented and with unit tests working.
  • Kick-off cross-net functions of SCA.

@adlrocha
Copy link
Contributor

2022-05-09

✋ Attendees

📣 Updates

🧵 Discussion

🎯 Up next

  • Continue the Rust implementation of SCA.
    • Before starting atomic execution function, I will rebase Lotus with FVM integration to Eudico to start testing the integration (is someone interested in doing it live so anyone can rebase if needed? Now with CI it should be easier to check if it was successful).

@adlrocha
Copy link
Contributor

2022-05-16

✋ Attendees

📣 Updates

🧵 Discussion

  • When are Akosh and Wiles joining? Work plan proposal:
    • Wiles + (support from Alfonso): Working in Eudico tackling issues here and with testing, monitoring and deployment? (IIRC from the interviews he has a ton of DevOps experience).
    • Akosh + Alfonso: Subnet actor and finalize integration with FVM (state accessors, user-defined actors, etc.).
    • Lefteris + Alfonso: Start exploring outstanding open problems in HC (?)

Short term goal: Eudico garden deployment with FVM integration and create and updated picture of HC.

  • CEL/CL collab
    • WIP roadmap from Axel. We're waiting on some further developments that were preempted by other priorities.
    • Axel's talk represents latest developments.
    • We may be focused on a corner of the problem that is adjacent to the key concern (security). Does this hold under adversarial forking, under asynchronous consensus?
    • Jorge to check with Alex regarding agenda for tomorrow.

🎯 Up next

  • Finish rebasing eudico with master to enable FVM_EXPERIMENTAL feature.
    • Work on Go state accessors to Rust actors state.
    • Add custom bundle of builtin-actors with SCA.
    • Have the full picture of changes required over Eudico for full FVM integration.
  • Kick-off subnet actor implementation or atomic execution functions for SCA.

@adlrocha
Copy link
Contributor

2022-05-23

✋ Attendees

📣 Updates

  • Rebased eudico to include FVM support (PR).
    • AI: On Friday the code-freeze for M1 happened. Will rebase again EOW.
  • Rebased forks of FVM and builtin-actors
  • Tested a builtin CAR actor with SCA (PR)
  • Fixed a bunch of data race and improved CI for Eudico (thank you @dnkolegov for your help)
  • Kicked-off implementation of reference implementation of user-defined subent actor
    • (update: figured out this morning :) ) Slight blocker: No testing framework like the one from builtin-actors. Trying to figure out the best way to test actors implementation.
    • Once subnet actor is done we'll be able to modify functions and state accessors in eudico to have a first e2e integration of HC with FVM.

🧵 Discussion

  • your discussion items here

🎯 Up next

  • Testing harness for user-defined subnet actor.
  • Rebase lotus from M1 code freeze

@adlrocha
Copy link
Contributor

2022-05-30

✋ Attendees

📣 Updates

🧵 Discussion

  • It is worth having a sync with the FVM team to align roadmaps?
    • Let's see what is valuable for FVM.
  • MV: Alignment on work for Akosh (June 20) and Wiles (June 27)
    • Akosh: Atomic execution protocol of SCA with sample contract
    • Wiles: Own the "road to production"
      • Performance evaluations.
      • Eudico garden
      • Monitoring and telemetry.

🎯 Up next

  • Kick-off integration of Rust/FVM actors in Eudico
    • Create new eudico branch based on FVM-M2 lotus branch with support for user-defined actors.
    • Load builtin-actor with Rust SCA and operations to deploy FVM subnet actor.
    • This may create the development of two parallel branches in Eudico for HC until FVM-M2.
      • Follow progress and related work here
  • Rebase FVM-M1 Lotus code-freeze to Eudico.

@adlrocha
Copy link
Contributor

2022-06-20

✋ Attendees

async

📣 Updates

  • Updates over the cryptoecon model for HC
    • Included low-level details about detectable misbehaviors and how to report them to the parent (see revision section for up-to-date information about recent changes to the model).
    • Discussed with CEL:
      • The purpose of collateral as a measure of "skin" of the game" for the subnet validators (product persepective metric); and the minimum threshold to cover for economic losses in the face of misbehaviors in the subnet.
      • The use of checkpointing gas fee as a way to steer the "openess" of subnets to accept additional network to be spawned from them.
      • The list of metrics we want to evaluate in the simulation of gas models CEL comes up with.
  • First draft of the updated spec for HC almost ready!
    • Pending:
      • Finishing the spec of the atomic execution protocol (will probably be done by EOD).
      • Add a section for the current state of detectable misbehaviors and the cryptoecon model.
      • Do a full review/formatting of the spec to start distributing it for an internal review.
        • I'll post a message in #consensus when the spec is ready for reaview.

🧵 Discussion

  • N/A (feel free to add discussion items here or in the weekly sync Slack thread ---i'll propagate the outcomes here---).

🎯 Up next

  • ConsensusFactory!!!
  • Propagate all the discussions points from last week's sync with CEL to our HC cryptoecon model
  • Finish a first draft of the spec before Wednesday to distribute it internally among CL members for a first round of reviews.
    • In the meantime, I'll start planning and drafting the list of HC-related FIPs that we'l need. My plan is to do an internal review, to then post a discussion for each FIP in order to start giving them external visbility and getting feedback.
      • I am targeting next week for the publication of the discussions. I'll probably also include a "master" issue to track the list of HC-related FIPs. I'll keep everyone posted.
  • Akos onboarding.
    • Onboarding sessions and kick-off project planning and alignment (according to Akos' availability).

@adlrocha
Copy link
Contributor

2022-06-27

✋ Attendees

async

📣 Updates

  • ConsensusFactory talk.
  • Draft of the HC spec ready to merge.
    • Thank you all for your reviews.
    • If anyone else is interested in joining this first round of reviews feel free to do so throughout this week. My plan is to merge by EOW, and share it more widely for further reviews.
  • Draft of the main FIP for HC also ready for review.
    • Same as with spec, opening a review window till EOW before we share it more widely in a FIP discussion.
  • Adapted Rust implementation of f04 (probably soon to be renamed f05) address to follow spec
    • To be merged throughout this week once the Go implementation is also ready.
  • Working on adapting the new Go implementation of f04 address and SubnetID to follow spec and fix interoperability with Rust.
  • Akosh's experiment improving readibility and convenience interacting with HAMTs and the store of builtin-actors.
    • We'll test how it works with the SCA, integrate it in our base code, and consider proposing the change to the FVM team.

🧵 Discussion

🎯 Up next

  • Wrap up the implementation of f04 address and interoperability between Go and Rust.
  • Implement end-to-end lifecycle of a subnet for FVM in experimantal/fvm-m2 branch of Eudico.
  • Come up with the high-level milestones and planning for "Project Eudico Forest"
  • Onboard Willes to B3.
  • Open discussions to discuss HC spec/FIP with Filecoin community.
  • (BONUS/Stretch) Kick-off implementation of atomic execution in SCA and a first MVP of submitting fraud proofs.

@adlrocha
Copy link
Contributor

adlrocha commented Jul 5, 2022

2022-07-04

✋ Attendees

async

📣 Updates

🧵 Discussion

  • I started the following issue to share the dependencies we currently rely on for the integration of HC with FVM. I want to open a discussion about why we may not want to make progress in the integration of HC in Eudico until we have a release candidate of FVM-M2 in lotus master. In order to use experimental/fvm-m2 we require the use of "fork-of-forks" with several potential sources of changes that need to be rebased periodically. This imposes a great overhead to keep up to date. The proposed plan is:
    • For now, @adlrocha can work in the implementation of atomic execution in rust SCA, @aakoshh is working in forest and in taking the current actors code to the next-level. @willes and @adlrocha will also start with the "road to production" implementing things that won't change in eudico post-FVM (example of issue that don't involve the VM and that we can start working on).
    • Once the atomic execution implementation is done, we can start the integration work without waiting for a release candidate for FVM-M2. This means that we won't rebase periodically with the experimental/fvm-m2 or our new features in eudico, we'll implement the full integration, and then worry about rebasing the latest changes in the corresponding repos.
  • B3 roadmap: With Akosh and Willes now contributing to the project, we should have another look to our roadmap and update the timeline accordingly.
    • @adlrocha: @willes, @aakoshh, do you feel ready to start distilling a set of high-level milestones for your projects? If you feel that it's still early, or that it would be helpful to discuss sync (high-bandwidth) we can schedule a meeting in the next few days.
      • Sync scheduled for Thursday. We should come out of the meeting with high-level milestones and dates for each of the lines of work.
    • @aakoshh: Before coming up with the high-level milestones for the Rust work we should probably do a walk through Forest to better understand what needs to be touched for HC. For example it has its own VM, from which a lot of the ref-fvm code is copy pasted. Is that going to be replaced with FVM? The abstractions like Runtime are currently slightly different.
      • AI: We should ask for the contact of someone from the Forest team to get more context.
        • #fil-forest-dev channel exists in Filecoin slack.

🎯 Up next

  • @ALL
    • Kick-off discussions and come up with an up-to-date planning for the project.
    • Finalize a first draft of the spec and share it more widely.
      • I am doing another full pass including the structure proposed by @aakoshh here. Let's see if I can finish it by EOD.
  • @adlrocha
  • @aakoshh
  • @willes
    • Understand the source code repo, the block production logic and subnet lifecycle.
    • Define the high level milestones.

@adlrocha
Copy link
Contributor

2022-07-11

✋ Attendees

async

📣 Updates

  • @ALL
    • Updated B3 planning
      • Roadmap for the next few months including target demo days and deliverables.
    • Discussion about ACKs
      • On-going discussion about ACKs in cross-net messages and how to provide users feedback about the execution of their corss-messages.
    • [Merged] first draft of the spec
  • @adlrocha
    • Removed cross-net messages from Filecoin blocks [PR].
      • Before this PR, we had an additional CrossMessages field in blockchain blocks that broke the backward-compatibility with the Filecoin mainnet. As suggested in the spec, we've decided to include cross-messages in blocks as SECPK messages without a signature (once we start the process of merging into Lotus maybe the Lotus team let us add a NO_SIGNATURE message type for this case).
    • Update go-address dependency in Eudico [PR]
      • It includes all of the changes required to make the Go implementation compatible with Rust.
    • Presented HC paper in DINPs workshop
      • We received really good feedback from people from Status.im (that presented a paper about censorship-resistance in decentralized messaging over GossipSub) and Felix from Ethereum Foundation.
      • Joao Leitao from Nova Lisbon also suggested that we should have a meeting after the summer to see if we can open lines of collaboration. Their lab has been mainly focused on distributed system protocols research but not specifically consensus. He thinks it may be a good idea share our open problems and see if we collaborate in some way.
    • Onboarding meetings.
  • @aakoshh
    • Took notes about the Forest codebase and discussed with Alfonso.
  • @cryptoAtwill
    • Familiar with aws
    • Setup aws vm instance and started building docker image on vm

🧵 Discussion

🎯 Up next

  • @adlrocha
    • IPFS Thing
      • I will be collecting notes of the event and potential outcomes and follow-ups in this doc
    • Kick-off atomic execution primitives in Rust.
  • @aakoshh
    • This week I'm going to start adding Delegated Consensus to Forest after taking another look at what it does exactly in Eudico.
    • Going to take a closer look at MIR, being the most prominent example of what we want a subnet consensus to look like.
  • @cryptoAtwill
    • Build the docker image on AWS machine, run sanity tests against it.
    • Start a local network

@adlrocha
Copy link
Contributor

2022-07-18

✋ Attendees

async

📣 Updates

  • We presented Hierarchical Consensus in IPFS Thing event.
  • @adlrocha:
    • Update go-address in Eudico
      • Upgrade to accommodate the new API and the interoperability with the Rust implementation of SubnetID and `f04 addresses' .
  • @aakoshh:
    • Forest PR to factor out the validation functions into a Consensus interface. From the reviews I learned where I can get a snapshot for the calibnet testnet and check if Forest still syncs. Also that it cannot sync from Genesis because it only supports the latest Network Version (NV16) and doesn't have migrations. Tried syncing with mainnet but it failed because of this. The mainnet snapshot is over 70GB, testnet is ~4GB.
    • Forest PR to factor out the weight function into the Consensus interface.
    • Draft PR for Delegated Consensus
  • @cryptoAtwill:
    • Setup node on aws and working

🧵 Discussion

  • @adlrocha: @cryptoAtwill, as you make progress in the deployment of all the testnet infrastructure, can we start working in parallel in writing-up all of the metrics and dashboards we want (and expect) to deploy over the testnet to give us visibility over the operation of the system? (we can kick-off a design doc so we have a clear plan once you're done with all the infrastructure work 🙏)
    • Added doc to start collecting the plan for metric gathering.

🎯 Up next

  • @adlrocha
    • Make progress in the Rust implementation of the atomic execution protocol.(draft PR).
      • Atomic execution primitives to be used by actors that want to support atomic execution.
      • Functions in the SCA for the orchestration and execution of the atomic execution protocol in the common parent.
      • Implement a sample actor addresses that supports atomic executions (similar to the one we have in the Go implementation).
  • @aakoshh:
    • Keep working on Delegated Consensus:
      • Figure out how to set up Genesis files in a way that we can specify who the single miner elegible for proposing blocks is, so that they can start block production as soon as their application is started. Alfonso tried to explain something about the first wallet to send a message gets the first actor ID t0100 but I don't understand how the first block can be mined without t0100 already being associated with a key that can sign the block. Unless I misunderstand what a "message" is (I thought it was a block or a transaction), the miner key has to be in the parent ledger already.
        • @adlrocha: No, you are right. The assignment between key and ID addresses is performed in genesis (thus, the ./eudico delegated genesis command receiving the key of the miner as an input). The command initializes a new genesis miner with that key, and sets it as the validator. However, genesis miners are assigned a sequential ID address from t0100 onwards in genesis, thus the t0100.
      • Block production loop, asking for all the necessary dependencies to be able to select transactions from the mempool and publish blocks.
      • Selection algorithm to pick transactions from the mempool which are valid with regards to sequencing rules (and gas fees
      • Figure out if it's easy to write tests, ideally with QuickCheck or PropTest, over an in-memory database.
      • Command line entries to start Forest with Delegated Consensus, also to create a Genesis file.
      • Feature flag to switch from Filecoin Consensus to Delegated during compilation.
  • @cryptoAtwill:
    • Create a subnet on aws vm
    • Try experiment with aws k8s cluster

@adlrocha
Copy link
Contributor

2022-07-25

✋ Attendees

async

📣 Updates

  • @adlrocha
    • Atomic execution functions for SCA done (PR, waiting for review to be merged)
      • This PR implements all the functions required by SCA to orchestrate an atomic execution from the parent subnet. The next step is to implemented a sample actor that supports an atomic execution to wrap-up all the pending (next PR).
    • Fixed flaky tests of Eudico.
      • There was a unit test in the Go implementation of SCA that was non-deterministic because it has to read a hashmap and was making the unit test to be flaky (according to the order in which the keys of the map was read the test passed or not). We removed the hashmap from the test and fixed the flakiness.
    • Found and fixed bug on Rust implementation of hierarchical address.
      • While writing the atomic execution functions and testing them we realized that there was a bug when creating a hierarchical address from a string (fixed in the above commit).
    • HC discussions with Guy.
      • Guy is doing an amazing job trying to understand the HC model and challenging it to develop a proper formalization of the model. In this initial stage we've been having a set of discussions to try to clear his understanding and clarify certain concepts of the protocol.
  • @aakoshh
    • Delegated Consensus in Forest completed and upstream PR
    • Unfortunately not functional because Forest can't make a Genesis file nor can it read the one created by Lotus.
    • We'll look into contributing the ability to write CAR files to the IPLD library, and ask the FVM team if they plan to do so soon.
    • Then, we could add the functions necessary to produce an actual Genesis file, not just a template.
    • Also see if the Eudico Genesis contains actor states in a newer format than Lotus (Forest expects v8).
  • @cryptoAtwill
    • Add metrics to subnet related commands(PR, to run more tests)
      • This PR increments the subnet counter when a new subnet is created. At the same time, the time spent spawning the subnet is also recorded.
      • Currently the above PR is still to be tested because the metric exporter, aka lotus-stats was just up. See next update
    • Setup lotus-stats and its supplimentary docker-compose services. The lotus-stats command seems not working out of the box now, there are some custom configuration and code modification needed to spin up the service. The changes are also included in the above PR.
    • Infra setup: experimented with AWS ELK cluster. Currently pending devops to grant permissions.

🧵 Discussion

  • Are we ready to propose a FIP discussion (sample discussion) of HC?
    • We have this draft of a FIP. My plan was to directly propose it as a discussion pointing to the spec (which is quite expensive), and iterate according to the comments that we get.

🎯 Up next

  • @adlrocha
    • Rust implementation of atomic execution primitives and sample actor that supports atomic execution.
    • Target the publication of FIP discussion of HC to gather feedback while OOO.
  • @cryptoAtwill
    • Integrating the subnet commands with the current lotus-stats. Most likely with create another cmd: eudico-stats to expose the metrics.
    • Replace influxdb with prometheus for eudico-stats.
    • Just track the number of active subnets from add and kill cmd.
    • Understand the actor code for join subnet

@adlrocha
Copy link
Contributor

adlrocha commented Aug 2, 2022

2022-08-01

✋ Attendees

async

📣 Updates

🧵 Discussion

  • @adlrocha: @aakoshh, let's see if we can find a way to create our custom genesis files from Forest (even if they are not compatible with Eudico initially), so we can spawn our own local networks. If we see that we are blocked, let's ask the FVM/Forest team and request their input for the best way to do so.

🎯 Up next

  • @adlrocha
    • Leave a draft PR with the basic types and primitives for actors supporting atomic executions for @aakoshh to review while I am off.
    • Engage in all the opened discussions around HC (I am especially interested in this and this in the short-term)
  • @cryptoAtwill
    • Setup miner and node, prometheus and grafana in ELK cluster
    • Expose collected metrics through grafana dashboard

@jsoares
Copy link
Contributor Author

jsoares commented Aug 9, 2022

2022-08-02

Hierarchical Consensus clarifications and sync meeting

✋ Attendees

Meeting Notes

  • GG: If a user wants to use a subnet it needs an account in that subnets.
    • MV: Bank example. Payments between different local banks.
      • BIC/SWIFT as unique IDs, no need to have an account in the "other subnet".
    • MV: Think about specific use cases not a general framework for everything to come up with the mechanics.
    • GG: The user is trusting the bank in the regular system (same as in HC).
      • If Marko wants to send an account to Guy, it needs an account in the subnet.
      • MV: Think about Bitcoin, it doesn't scale. It is limited in the number of TPS.
      • GG: We want to exploit locality.
  • MV: What is the problem that we are trying to solve?
    • GG: Locality and taking the load from the subnet.
  • MV: How can we define locality? At which point are you using the subnet for transacting as oppose to higherlevel subnets?
  • MP: The concept of batching is important. My understanding of HC is that even this transactions are batched. From one subtree to the other subtree happen as one transaction in the root.
    • GG: Like the way MP presents. An exchange from subnet L3 with subnet L3 in an independent branch, conceptually the exchange is from GG to MP, why should we care about the level?
    • GG: Guy is able to send money to MP but it shouldn't worry about how it gets there. MP only wants the money.
    • MP: You may need to do that because the transaction can be expensive in the top-level. In the bottom-up level if they are cheaper, the transactions are batched and we split the cost.
  • LK: There is an implicit assumption that all subnets are equally secure. I don't think all account are exactly the same. There is a security scalability trade-off here.
    • GG: You can get it in your own level. Even if we think of different accounts and different addresses they are actually all the same and owned by the same entity. Owners are responsible for routing the funds where they should go.
    • LK: This is a UX problem. There is consensus and cross-messaging as two mechanisms of the protocol.
    • MK: Coinbase-Kraken crypto exchange example.
    • GG: Batching is not a key part of the HC.
  • GG: We should come up with a minimal interface and the core fundamental one that we need to build everything on top of it. It also makes a better design because we have the code and everything else are modules.
    • MP: What is the minimal interface under your understanding?
    • GG: Scalability through locality.
      • You choose when to spawn a new subnet.
      • Between accounts in the same subnets you can do SMR and support whatever feature you want (SC, executions, etc.).
      • Besides that you can only do interactions with your parents and your children accounts.
      • In this approach you only need to checkpoint the balances, not the full state of the SMR (you keep whatever you want in the subnet, this is what you need for the firewall property).
      • Core functionalities checkpoint the balances and then state committment is an add-in.
  • GG: Maybe not minimal interface but minimal structure.
  • LK: HC gives way more flexibility. We can do "other things" over the same architecture.
    • We can have payment channels and rollups and we can combine them as part of the framework.
  • AR: Basic model to define the core protocol for interoperability and semantics.
  • LK: Provide a minimal way and identify the basic modules to perform the interactions between the different entities.
  • MP: Coming back to "what is the interface"? My point of view is that what everyone needs in common is some mechanism to create subnets, some common notion of token that everybody understands,
    • GG: We don't all need the same coin, only parents and children. Then if as a child you create another coin and use it for parent-child interaction.
    • MP: Without the same coin we can't have as easy cross-net transactions.
    • MP: We can use a common token to all the system and then having different subnets.

@jsoares
Copy link
Contributor Author

jsoares commented Aug 9, 2022

2022-08-08

✋ Attendees

async

📣 Updates

  • @cryptoAtwill
    • Orchestrated HC node creation on AWS EC2 instance
    • Setup and exposed Grafana dashboard: http://18.141.203.118
  • @aakoshh
    • Investigated writing CAR files from Forest. It can do it, it was just less obvious than the CarReader struct; it is "hidden" in a method of CarHeader.
    • Investigated producing a Genesis file in Lotus. Realised that it can produce genesis for any known NetworkVersion, not just the starting version of the original Filecoin network. It turned out that actually when we failed in our previous attempt, it was already producing a genesis with the latest version 16. The reason Forest refused it wasn't that the state tree was in a wrong format, but rather that the acceptable V8 Actor code CIDs hardcoded in Forest only accepted mainnet and calibnet. After further investigation it turned out I accidentally prepared a devnet genesis from Lotus; this depends on how it is compiled.
    • Instead of embarking on State Tree generation in Forest, I tried again to make a correct genesis file in Lotus. Unfortunately this fails for mainnet and calibnet, I can only prepare a genesis.car file for devnet, ie. when Lotus is compiled as make 2k or make debug, otherwise it complains that some minimum 2048 allowance is too low.
    • Opened a PR in Forest that has a working example of Delegated Consensus. It required adding the devnet Actor CIDs to Forest, and then debugging the config issues.
    • Agreed with Alfonso that instead of trying to add subnet support to Delegated Consensus in Forest, as a next step I could try to tackle adding Narwhal without subnets. This would involve getting more familiar with custom networking, as Narwhal would have its own protocol independent of Filecoin. My main motivations to suggest this approach is because (1) Alfonso is not here in the next two weeks to answer questions about how things are done exactly in Eudico and (2) I feel like the discussion started about how to add the SCA to Filecoin could, at least in the rootnet, lead to a design where the message resolution is not part of the node itself, but rather some external relayers, so the whole story can be implemented in "user land" without changes to the Filecoin L1. The subnets could be more flexible in their approach since they can already use custom consensus, but if we want to use Filecoin as a rootnet we might be at the mercy of its stewards. Alfonso said if push comes to shove he thinks we should fork Forest and use it specifically for HC.
  • @guy-goren
    • Figured out a misunderstanding I had regarding the HC architecture.
    • Pointed out the fundamental attributes of different design possible choices
      • involved crypto mechanism vs. simple layered architecture.
      • better theoretical performance vs. limited use cases.

🎯 Up next

  • @cryptoAtwill
    • Start reading source code and capture cross messages
    • Update docker compose to start a network of nodes

@adlrocha
Copy link
Contributor

2022-08-15

✋ Attendees

async

📣 Updates

  • @cryptoAtwill

    • Added metrics to capture cross net messages. The data is captured in the form of a graph where the nodes are the subnets and an edge is the number of cross net messages. However, the problem faced now is Grafana does not support graph data. The closest experimented is Node Graph, but Node Graph does not support custom data structure. This implies that grafana might not be able to present the graph data until Node Graph supports more data structure. One potential solution is to give up graph data. But this is not optimal as a graph should be the most clear way to represent cross net communication. The other solution is to use a graph database, such as neo4j then build dashboard on top of neo4j. This approach is better. But the issue with this approach is stats collection needs to be done offchain which currently it's done on chain. See next point for more ellaboration.
    • Implemented eudico-stats as part of FullNode API and can trigger as separate process, commit: filecoin-project/eudico@a43ee3c. With this, the metric collections are completely off chain. It is possible to constanly send the collected subnet metric snapshot to external database for data processing. Currently only the parent child subnet relationship is tracked.

  • @guy-goren

    • Added "shortcuts" to the simple HC design proposal. In essence, a shortcut between two subnets can be build on top of the simple architecture similarly to off-chain payment channels.

  • @aakoshh

    • Read the Narwhal & Tusk paper
    • Caught up with the Narwhal plans for Eudico, made some comments there.
    • Opened a discussion about how exactly to integrate it into Forest, what to keep from the built-in functionality of the mempool and the block sync, and what expectations to give to the general ecosystem. Presented two options.
    • Investigated the Mysten Narwhal implementation.
    • Published my notes on about using the Narwhal implementation as a libary and the gaps between what it does today and what we need. Received some comments from the devs, which is great! Apparently much of our wishlist is on the roadmap.

🎯 Up next

  • @cryptoAtwill
    • Setup neo4j as metric database
    • Inject subnet count in to neo4j
  • @guy-goren will be OOO 16-24 AUG
  • @aakoshh
    • Discuss the Narwhal integration with the team.
    • Ask followup questions from the Narwhal devs.
    • Read the Bullshark paper so I understand the difference from Tusk.
    • Probably start with: fork Narwhal; add begin/end batch signal; add transactions in-process; derive determinsitic blocks; disable mempool gossiping; disable block gossiping; figure out how to distribute rewards.
    • Ignore for now: epochs; historical syncing; stake distribution.

@adlrocha
Copy link
Contributor

2022-08-22

✋ Attendees

async

📣 Updates

  • @adlrocha
    • Catching up.
    • Preparation for HC Roadmapping summit in Iceland
  • @cryptoAtwill:
    • Add graph data structure to the subnet relationships
    • Introduced neo4j as the database
  • @aakoshh:
    • Contributed a PR to Narwhal to support batch execution state
    • Started adding Narwhal to Forest but ran into dependency issues with a Filecoin BLST library, it insists on an earlier version of BLST than what Narwhal depends on.

🧵 Discussion

🎯 Up next

  • aakoshh:
    • I will try to fork the blstrs library to upgrade its blst dependency to a version that Narwhal also accepts, to get the thing to compile. Overall it's probably not a good idea to always compile such a heavy dependency stack as part of Forest, but let's see how it goes.
  • cryptoAtwill:
    • Start to integrate cross net message graph

@adlrocha
Copy link
Contributor

2022-08-29

✋ Attendees

async

📣 Updates

  • @adlrocha:

    • HC roadmapping summit and outcomes.
      • We are targeting a new milestone in the next few months where Eudico ships with a proof-of-stake version of EC and is used as the peer implementation for FVM's builder net.
      • High-level planning for this milestone.
      • After this milestone we will evaluate our next steps for a production-ready implementation
    • Started a discussion with a Competitive analysis of IPC to evaluate what others are doing in the space and the performance that they are reaching.
    • Doc with a brief analysis.
      • The goal of this doc is to help us decide the best approach to follow for the production implementation of HC.
    • Fixed with Denis a "nasty data race" in Eudico surfaced by Mir (see 1,2,3)
    • Pushing the initial re-configuration of Mir in Eudico to the finish line.

  • @aakoshh:

    • Forked the Filecoin BLST libraries and found a working combo that allows me to compile Forest with Narwhal.
    • Will carry on the integration, but it will be just a PoC:
      • Narwhal requires every participant to be a committee member with a private key. Once we disable the built-in Forest chain-sync protocol, only validators can sync, not full nodes. We'll probably have to bring syncing back in some form.
      • The default garbage collection in Narwhal is 50 rounds; we can possibly disable it but it means nodes have to be present all the time or they won't be able to catch up. Another reason to have some kind of chain sync.
        • @adlrocha: we may need to also address syncing when a node joins the network from scratch. Different consensus algorithms may use different approaches for this.
      • My short term plan for the mempool is that upon start the node asks which are the transactions it can apply on the current tip of its chain, and sends them to Narwhal, remembering the (account ID, nonce) pairs. Then, whenever the Filecoin chain is extended with a new block, we ask the mempool again if there are any new transactions we could send, by passing in the previously sent (account ID, nonce) pairs as a filter, so we only consider the next nonce. This allows blocks made by other validators to unblock transactions we have, and also it treats pending nonces as if their predecessors were already included on the chain, even if they are not. Block will come at a steady pace from Narwhal, so it's a decent signal, but there can be too many actually, one per validator in each round. There will be a flag indicating whether this check is already happening, in which case we can skip doing it.
      • I will generate one or more Filecoin blocks out of each Narwhal certificate, even the empty ones. This will make it easy to give rewards to someone identifiable.

  • @guy-goren:

    • No news. Needs to rethink my goals if HC is fast-forwarded into production. (My current focus is on a more fundamental level that might become unnecessary given the timelines.)

🧵 Discussion

  • @adlrocha: @aakoshh, would you be available to spend some time this week drafting a design doc with a roadmap/planning of what would it look like if we decide to make Forest our baseline for the new implementation of IPC? I can kick it off and ask for your input.
    • Conclusion of the discussion: We'll start a new document to start drafting a roadmap of an eventual Rust and Go implementation of Rust.
  • @adlrocha: @aakoshh, should we start thinking about modularizing Forest and making it our own IPC-specific base code? I feel like you are doing a lot of work trying to fit Narwhal, etc. into the current Forest base code when we could be investing those efforts into figuring out the right abstraction for a modular IPC.
    • Now we know this is something that may eventually happen soon.
    • Conclusion of the discussion: We'll keep working on the integration of Narwhal with Forest and figure out if it is a good idea to use it a baseline for IPC, and how modular we can make it.

🎯 Up next

  • @adlrocha:
    • Eudico housekeeping towards BuilderNet.
      • Audit Eudico code to see what is missing and things that need to be improved.
      • Clean outdated code.
      • Validate the approach for BuilderNet with FVM/Lotus team
      • Rebase experimental/fvm-m2 to have the base branch to start the implementation for Builder Net.
      • Remove the LegacyVM from Eudico.
      • Recover core actors in Eudico so they can be used for the testnet.
    • Feature parity of FVM actors with Legacy VM IPC actors.
    • Design doc with a draft of the roadmap for a production implementation of IPC based on Forest and Eudico.
      • We need a high-level outline of what would it take to put all of our development efforts into one implementation or the other.

@adlrocha
Copy link
Contributor

2022-09-19

✋ Attendees

async

📣 Updates

🎯 Up next

@jsoares jsoares closed this as completed Oct 3, 2022
@jsoares
Copy link
Contributor Author

jsoares commented Oct 3, 2022

The work continues in #7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adlrocha adlrocha

Labels
Epic 🚧 Project
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jsoares @adlrocha