RFP Application: go-fvm-sdk

[Fatman13]

RFP Proposal: FVM - Go SDK - Phase A

Name of Project: FVM - Go/Tinygo SDK

Link to RFP:

RFP Category: devtools-libraries

Proposer: hunjixin

Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT, GPL, and APACHE2 licenses?: Yes

Project Description

Since the announcement of FVM, support for a broader developer community other than the native Rust programming language is of high importance to the FVM roadmap in general. A strong and diverse developer community has been proven to be critical to the longevity of a project and we think that FVM project have the opportunities to attract more developers with a wider SDK support.

During FVM Early Builder program, we have successfully built a PoC Go SDK that facilitates the use of Tinygo to write user-defined actors (smart contracts) that meets the requirement of WASM compilation target so that the said actors can be run in the WASM-based FVM execution environment. In extension of this PoC, we now propose the continuation of the development for the FVM GO SDK.

Value

The benefits of a Go SDK is apparent as it provides a much easier on-ramp for Go developers into the fvm ecosystem. In addition, the existing pool of contributors to Lotus and Venus implementation of Filecoin could jump right into fvm development with language they already familiar with. Some highlights FVM could leverage by bringing in Go SDK:

• A fairly easy-to-learn high-level programming language
• An extensive selection of libraries, especially blockchain related ones
• Matches the technology stack of Filecoin which allows import of necessary libs

All in all, we believe that a Go SDK would contribute a great deal to the growth of the FVM ecosystem and could be the gateway language to on board more developers to write user-defined actors.

Development Roadmap and Deliverables

Each milestone would contain number of sub-milestones to have finer granularity.

Phase A

The development roadmap is further broken down into two phases, A and B. This RFP will be focusing on phase A with B as long-term vision of the team for this project.

Milestone 1 - SDK implementation

M1.1 - Adapt TinyGo

Technical Background:

Many of the basic libraries of TinyGo and fvm are not compatible, thus requiring adaptation work to be done upon the TinyGo libraries. For example, TinyGo has output stream library like os.stdout while fvm is not supporting such. An exhaustive list of libraries that need to be modified are listed below in the technical scope section.

Technical Scope:

• Adaptive transformation of TinyGo
• Implement system calls
• Revamp TinyGo libraries that are being used by the go-fvm-sdk; Make modifications to reflection, output stream, platform and network related code (See background above)

replace (
   //remove os.stdout
   github.com/davecgh/go-spew => github.com/ipfs-force-community/go-spew v1.1.2-0.20220524052205-0034150c051a
   //remove package github.com/ipfs/go-ipfs-util
   github.com/ipfs/go-block-format => github.com/ipfs-force-community/go-block-format v0.0.4-0.20220425095807-073e9266335c
   //remove anything about reflect
   github.com/filecoin-project/go-address => github.com/ipfs-force-community/go-address v0.0.7-0.20220524010936-42617a156be1
   github.com/ipfs/go-ipld-cbor => github.com/ipfs-force-community/go-ipld-cbor v0.0.7-0.20220713070731-f5190aacb1a4
   github.com/polydawn/refmt => github.com/hunjixin/refmt v0.0.0-20220520091210-cb3c7d292019
   github.com/whyrusleeping/cbor-gen => github.com/ipfs-force-community/cbor-gen v0.0.0-20220421100448-dc345220256c
   
   //remove http/file asset and reflect code
   github.com/stretchr/testify => github.com/ipfs-force-community/testify v1.7.1-0.20220616060316-ea4f53121ac3
   //remove implement baseon specify platform only keep pure go implement
   golang.org/x/crypto => github.com/ipfs-force-community/crypto v0.0.0-20220523090957-2aff239c26f7
   lukechampine.com/blake3 => github.com/ipfs-force-community/blake3 v1.1.8-0.20220609024944-51450f2b2fc0
   github.com/klauspost/cpuid/v2 => github.com/ipfs-force-community/cpuid/v2 v2.0.13-0.20220523085810-ac111993ce74
   github.com/minio/blake2b-simd => github.com/ipfs-force-community/blake2b-simd v0.0.0-20220523083450-6e9a68832d69
   github.com/minio/sha256-simd => github.com/ipfs-force-community/sha256-simd v1.0.1-0.20220421100150-fcbba4b6ea96
)

Deliverables:

• Pull request merged on the repository with a release tagged at m1.1
• Writing and executing actors using TinyGo for advanced users

Funding for this M1.1:

1 PM resource: 1 * 46 $/h * 40 h/week * 1.5 week = $2760
1 Software engineer lead resource: 1 * 76 $/h * 40 h/week * 2 week = $6080
1 Software engineer resource: 1 * 67 $/h * 40 h/week * 2 week = $5360
1 QA engineer resource: 1 * 55 $/h * 40 h/week * 2 week = $5360

Total = $18600

See rates at reference section for more details.

M1.2 - Developer experience improvement

Technical Background:

Compiled actor code from TinyGo doesn't natively support including the initialization code, thus requiring modification during compiling stage to inject the init code into the final product, effectively replacing the main() function.

Technical Scope:

• hello world example
• ERC20 example
• Add compile tooling; Enable compiling using TinyGo and allow processing on the output through inject init code in invoke function (See above background)
• Add abilities for creation of template (generation of a working actor directory structure and compilable actor template for user)
• Add testing tools which allows feature testing of actors

Deliverables:

• Pull request merged on the repository with a release tagged at m1.2
• Writing and executing actors using TinyGo with improved dev experiences

Funding for this M1.2:

1 PM resource: 1 * 46 $/h * 40 h/week * 2 week = $3680
1 Software engineer lead resource: 1 * 76 $/h * 40 h/week * 2 week = $6080
1 Software engineer resource: 1 * 67 $/h * 40 h/week * 2 week = $5360
1 QA engineer resource: 1 * 55 $/h * 40 h/week * 2 week = $5360

Total = $19520

M1.3 - Developer experience improvement p2

Technical Scope:

• Add generator tools
  • generation of actor Cbor struct (actor states)
  • generation of actor entry file
  • generation of go client for actor, through which user can have actor generation, creation and invocation
• CI integration (incremental integration, lint checks, actor test run, auto-publish, CI harness/checks)
• Add amt and hamt support; import spec-actors and support map & array for data storage with read on demand support
• Add mock layer to fvm for actor unit testing
• Metrics on gas usages of each methods
• More examples (like native token example)

Deliverables:

• Pull request merged on the repository with a release tagged at m1.3
• Writing and executing actors using TinyGo with updated testing capabilities and updated examples

Funding for this M1.3:

1 PM resource: 1 * 46 $/h * 40 h/week * 2 week = $3680
1 Software engineer lead resource: 1 * 76 $/h * 40 h/week * 2 week = $6080
1 Software engineer resource: 1 * 67 $/h * 40 h/week * 2 week = $5360
1 QA engineer resource: 1 * 55 $/h * 40 h/week * 2 week = $5360

Total = $19520

Estimated Milestone 1 Delivery:

Within time framework of early builder program. Estimated to be around a month (End of Sep 22)

Milestone 2 - Community feedback integration

In this milestone, we will try collect feedbacks from participants within the early-builder group. Other methods (including demos and outreach, recorded walkthroughs, and engaging other Go devs from the Filecoin ecosystem) kindly suggested by fvm team may also be adopted to increase reach. We will be cautious to avoid any public announcement and acknowledgement of go-fvm-sdk.

M2 - address community's feature requests

Technical Scope:

• actively answer questions from the community and listen to what their needs
• compile list of feature requests
• Integrate what community would like to see into the SDK

Deliverables:

• Pull request merged on the repository with a release tagged at m2

Funding for this M2.2:

1 PM resource: 1 * 46 $/h * 40 h/week * 2 week = $3680
1 Software engineer lead resource: 1 * 76 $/h * 40 h/week * 2 week = $6080
1 Software engineer resource: 1 * 67 $/h * 40 h/week * 2 week = $5360
1 QA engineer resource: 1 * 55 $/h * 40 h/week * 2 week = $5360

Total = $19520

Estimated Milestone Delivery:

Estimated to be around a month. (End of Oct 22)

Total funding for M1+M2:

M1 + M2 = $77160

Phase B

Much of the milestone 3 and 4 are not set in stone. The direction of the development may include a combined effort from community and general direction of where fvm is heading by that time.

At this stage, this rfp would only include milestone 1 and 2. Milestone 3 and 4 are listed solely for the team's long term vision.

Milestone 3 - Memory optimization

We expect that there could be performance issues given the native garbage collection. This could be amended by adopting manual memory allocation functions, which works similarly to dynamic memory allocation in C where user could allocate memory and free memory on demand. Thus, improves the performance of running actors.

At the same time we expect to expose some basic types for managed code.

Assumptions/Requirements:

• Explore opportunities in Cgo and identify if minimal changes could allow Cgo to be compatible with WASM
• Manual memory management would require extra storage apart from ones designated to garbage collection, which may require support of memory allocation function of FVM

M3 - Implementation

Deliverables:

• go-fvm-sdk v0.x
• Updated CI workflows
• types like CString, CSlice（If necessary）
• Able to load WASM lib compiled from dynamic memory allocation of C or Rust
• Can properly handle the boundaries between managed and unmanaged code

Funding for this M3:

TBD

Estimated Milestone 3 Delivery:

Three month. (End of Jan 23)

Milestone 4 - Golang-fvm—sdk

Technical Background:

In the long run, it is very likely that GO will support WASI. Taking advantage of the said efforts from GO team would make it much easier to support the native Go language for go-fvm-sdk. By that time, sdk would support two conversion types.

According our analysis, the relatively complex memory and GC management in Go may not necessarily have advantages over TinyGo when running application in fvm. This can only be evaluated after the WASI support of go is released.

Assumptions/Requirements:

Preconditioned by the progress of how much Go would support WASI. We could either:

• Wait for Go to support WASI
• Or support Golang through stub which might have an impact on performance
• Actual Golang support for FVM while retaining support for TinyGo at the same time

M4 - Implementations

Deliverables:

• go-fvm-sdk v0.x
• Performance metrics (WASM target size, memory usage, time usage) of Go Vs. TinyGo
• Updated CI workflows

Funding for this M4:

TBD

Estimated Milestone 4 Delivery:

Three month. (End of Apr 23)

Total Budget Requested

Maintenance and Upgrade Plans

The team is willing to continue the development of this RFP after M4. Another evaluation will be needed at then to further the planning.

Team Members

Venus core team

Team Website

https://forcecommunity.io/

Relevant Experience

Force community has been an active contributor to Web3 ecosystem and Filecoin ecosystem in general. The engineering team from Force community has a track record of contributing code to Lotus as far back as Testnet and Space Race.

Team code repositories

https://github.com/ipfs-force-community

Additional information

Force community is committed to become a major contributor to Web3 infrastructure and we see Filecoin at the core of the big Web3 migration. We hope that we could fast track the realization of Web3 adoption by contributing our software development capacity to the course and join hand in hand with all other ecosystem developers around the globe through this historical journey!

References

• Project manager hourly rate: $46
• Software engineer (Go/WASM) hourly rate: $67
• Software engineer lead (Go/WASM) hourly rate: $76
• QA engineer: $55

[BlocksOnAChain]

@Fatman13 Thanks for sharing. Can you maybe check your delivery timeline, seems some of the estimated dates are in the past.
Example:
"Estimated Milestone 4 Delivery:
Three months. (End of Mar 22)"

[Fatman13]

@BlocksOnAChain my bad. Just fixed the date.

[Fatman13]

@Kaktus-green Done! Added the total just before M3 title.

[BlocksOnAChain]

Thanks, @Fatman13 !
I will share it with the team for a review

[raulk]

Hey! Thanks for the submission.

M1.1 - Implementation

This milestone largely makes sense to me. However:

• I'm not sure I understand what "test system tools" are. Could you please elaborate?
• Implementing FVM system calls is fine, but this sounds like you're implementing the plumbing (FFI bindings and type conversions). Are you planning to provide SDK sugar at this stage too, i.e. high-level SDK features to make development nicer See the RFP Application: FVM - High level Rust SDK #562 proposal for examples in a Rust counterpart.

We should include demos as well.

1 PM resource: 1 * 46 $/h * 40 h/week * 2 week = $3680

You really need a PM full time? Is the PM's job mostly internal coordination at this stage? There doesn't seem to be much external work in this milestone?

M1.2 - Testing & Optimizations

This makes sense to me. Could you provide more details on how you plan to conduct the community testing, and involve builders and hackers? I want to expressly avoid a situation where the availability of this SDK is announced, but there's little traction/engagement. I think there should be active steps here, e.g.:

• demos and outreach
• recorded walkthroughs
• engaging other Go devs from the Filecoin ecosystem (e.g. maybe sending a note to the PLN network)
• blog post

Another thing: I think to make this milestone useful, you will need to implement more and more examples. There's only so much you can exercise with the Hello World and ERC20 examples from M1.1.

[raulk]

Milestone 2 - EVM support

I don't understand this milestone and the proposed goals here. Are you looking to re-implement the FVM here? This no longer seems to be about the SDK? I'm very confused.

Milestone 3 - Memory optimization

Leaving this to the very end is risky. You could basically end up in a place where you've built a full SDK but nobody wants to use it because the memory footprint is so high that transactions become expensive in terms of gas usage. In other words, part of what's proposed here should be done early in the project (potentially a Milestone 0) as part of feasibility analysis.

Milestone 4 - Golang-fvm—sdk

I apologise, but I don't understand the goal of this milestone? You want to re-implement the FVM In Go? (no longer about the SDK for building actors?)

[Fatman13]

@raulk Thank you for the feedbacks! Will take the review to the team and come up with a revised proposal.

[hunjixin]

• I'm not sure I understand what "test system tools" are. Could you please elaborate? "

support unit test/integrate test, maybe event support testcoverage(not support in tinygo now)

[hunjixin]

• high-level SDK features

high-level features also to be considered, should also include in features, this will be a long-term work and depends on practical requirements

[hunjixin]

I don't understand this milestone and the proposed goals here. Are you looking to re-implement the FVM here? This no longer seems to be about the SDK? I'm very confused.

for evm, the idea is still being worked on, but it is really interesting. at present, the implementation of evm in fvm seems to be implemented based on a bridge. I am not sure whether it is actually inside fvm, but if it is implemented outside the fvm like system actor, also be possible to implement in EVM.

[hunjixin]

Memory optimization

you are right ,this title is confusing, this goal aim to resolve interact with wasm from rust, and provide manual memory manage in tinygo, so user could control memory by themself. just like unsafe operation. this two jobs have something in common, maybe also necessary to allocate an extra memory space to run unsafe code.

[Fatman13]

Hello, @raulk, really appreciate your reviews! I will maintain the following changelog to keep track of the edits we made to address your reviews. We will prepare a major revised version addressing the technical aspects soon!

Changelog:

1. Change PM time from 2 weeks to 1.5 week

You really need a PM full time? Is the PM's job mostly internal coordination at this stage? There doesn't seem to be much external work in this milestone?

The reason we need a PM is that in addition to internal coordinations it would facilitate better communication (due to maybe language barrier, time zone difference, meeting scripts, English documentations etc) between fvm and venus team so that the development of go-fvm-sdk would better align with what fvm would like to see.

2. RFP structure revised from m1.1, m1.2, m2.1 and m2.2 to m1.1, m1.2, m1.3 and m2.

3. Each milestone is expanded with revised and updated technical scope.

4. Comments from @hunjixin to address some of the concerns.

5. Addressing 2nd review comments from PL team. RFP has been revised.

[raulk]

@Fatman13 Thanks for the update! We'd be looking at approving Milestones 1 and 2 for now. Could you group these milestones into a Phase A, and the rest into a Phase B? Then we can target just Phase A for an approval here.

Re: Milestone 1.1

M1.1 - adapt TinyGo

You mean adopt TinyGo? How much of this have you already done?

Adaptive transformation of TinyGo

Please elaborate what this means.

Implement system calls

👍

Revamp libraries that are being used by the SDK; Make modifications to reflection, output stream, platform and network related code.

What's the motivation? What are these libraries? Which SDK? I'm not sure what much of this means.

Pull request merged on the repository with a release tagged at m1.1

Unfortunately this formulation for a deliverable doesn't tell the community much about what to expect. What can they expect to do with the SDK at this point?

Re: Milestone 1.2

Add compile tooling; Enable compiling using TinyGo and allow processing on the output through inject init code in invoke function
Add abilities for creation of template
Add testing tools which allows feature testing of actors

Could you please elaborate on these? I'm not sure I follow.

Pull request merged on the repository with a release tagged at m1.2

These formulations for deliverables aren't very helpful. I suggest describing "What to expect" in more detail.

Re: Milestone 1.3

Add mock layer to fvm for actor unit testing
Metrics on gas usages of each methods

I think these might be underestimated, but curious to see how much you can get done.

More examples

Elaborate, please?

CI integration (Lint checks, actor test run)

Is this the first time you'll be having CI run on the SDK? Or do you mean CI harnesses so that actors built with the SDK can be tested in CI?

[Fatman13]

Hello, @raulk, really appreciate your comments! The team has updated a revised version according to your review. Changelog is also updated to reflect that. Please let us know if you have additional comments! Thank you!

[Fatman13]

Hello, team! Please let us know what the next step for us to take if the revised proposal (according to the latest comments) is deemed okay to accept. Thank you! @BlocksOnAChain @realChainLife @raulk

[ErinOCon]

HI @Fatman13, thanks for being in touch! The revised proposal has been approved. Can you confirm the preferred email address for discussing next steps?

[Fatman13]

@ErinOCon Thanks for the reply! The email address is [email protected].

Looking forward to the next step!

[ErinOCon]

Thanks, @Fatman13!

[BlocksOnAChain]

@Fatman13 - Hey friend, I wanted to take a look at M2 - where can I review the feedback received from users and how you managed it - do we have a list of issues and PRs to address them based on the feedback you got from users?

Also, do we have demo recordings from M3 and M4? Since I haven't been able to find any materials on your website or the project repo that show us what is done in Phase 2 of the project - M3 and M4 from this RFP?

[Fatman13]

Hey friend, I wanted to take a look at M2 - where can I review the feedback received from users and how you managed it - do we have a list of issues and PRs to address them based on the feedback you got from users?

Hello, @BlocksOnAChain! We haven't checked in for milestone 2 yet. We are currently at milestone 1.3.

We are actively looking for community participants to try to give the sdk a spin, but as the current main theme of nv18 is about fevm not many has actually give the sdk a thorough test.

Also, do we have demo recordings from M3 and M4? Since I haven't been able to find any materials on your website or the project repo that show us what is done in Phase 2 of the project - M3 and M4 from this RFP?

M3 and M4 is not part of this grant. This grant only include the work for phase 1 (m1 and m2).