[合作] Integration with Farcaster Planning

[Fatman13]

Background

To explore opportunities of integration with Farcaster so that Venus community could enjoy a open-source solution for better monitoring their storage systems.

Research

• [集成] metrics可视化使用farcaster的可能性？ #5137

AI

Week 1

• Access to Venus chain service on calibration
• List of Lotus API that farcaster uses
• Mapping of Lotus APIs with Venus APIs
• List of metrics Venus currently have

Week 2

• Grant proposal draft and review

[Fatman13]

List of metrics (In active development)

• venus-miner
• venus-messager
• venus-gateway
• venus-market
• venus

venus-miner

List of metrics could be found here. The wording might need some polish. Note that metrics below are available per miner_id.

// latency for GetBaseInfo API
GetBaseInfoDuration   = stats.Float64("getbaseinfo_ms", "Duration of GetBaseInfo in miner", stats.UnitMilliseconds)
// latency for ComputeTicket API
ComputeTicketDuration = stats.Float64("computeticket_ms", "Duration of ComputeTicket in miner", stats.UnitMilliseconds)
// latency for IsRoundWinner API
IsRoundWinnerDuration = stats.Float64("isroundwinner_ms", "Duration of IsRoundWinner in miner", stats.UnitMilliseconds)
// latency for ComputeProof API
ComputeProofDuration  = stats.Float64("computeproof_s", "Duration of ComputeProof in miner", stats.UnitSeconds)

// number of block produced
NumberOfBlock         = stats.Int64("number_of_block", "Number of production blocks", stats.UnitDimensionless)
// number of rounds that miner_id is winner
NumberOfIsRoundWinner = stats.Int64("number_of_isroundwinner", "Number of is round winner", stats.UnitDimensionless)

venus-messager

List of metrics could be found here.

// Below metrics are updated on a per wallet address granularity 
WalletBalance    = stats.Int64("wallet_balance", "Wallet balance", stats.UnitDimensionless)
WalletDBNonce    = stats.Int64("wallet_db_nonce", "Wallet nonce in db", stats.UnitDimensionless)
WalletChainNonce = stats.Int64("wallet_chain_nonce", "Wallet nonce on the chain", stats.UnitDimensionless)

// Current number of messages that are waiting for venus-messager to fill out parameters like signature, gas usage, nonce etc.
// This metric is updated on a per wallet address granularity 
NumOfUnFillMsg = stats.Int64("num_of_unfill_msg", "The number of unFill msg", stats.UnitDimensionless)
// Current number of messages that venus-messager has filled out parameters like signature, gas usage, nonce etc.
// This metric is updated on a per wallet address granularity 
NumOfFillMsg   = stats.Int64("num_of_fill_msg", "The number of fill Msg", stats.UnitDimensionless)
// Current number of messages that venus-messager has failed to fill out parameters like signature, gas usage, nonce etc.
// This metric is updated on a per wallet address granularity 
NumOfFailedMsg = stats.Int64("num_of_failed_msg", "The number of failed msg", stats.UnitDimensionless)

// Current number of messages that haven't being on-chain for more than 3 minutes
NumOfMsgBlockedThreeMinutes = stats.Int64("blocked_three_minutes_msgs", "Number of messages blocked for more than 3 minutes", stats.UnitDimensionless)
// Current number of messages that haven't being on-chain for more than 5 minutes
NumOfMsgBlockedFiveMinutes  = stats.Int64("blocked_five_minutes_msgs", "Number of messages blocked for more than 5 minutes", stats.UnitDimensionless)

// Number of message being selected by venus-messager during last round of message pushing
SelectedMsgNumOfLastRound = stats.Int64("selected_msg_num", "Number of selected messages in the last round", stats.UnitDimensionless)
// Number of message being pushed by venus-messager during last round of message pushing
ToPushMsgNumOfLastRound   = stats.Int64("topush_msg_num", "Number of to-push messages in the last round", stats.UnitDimensionless)
// Number of message being expired by venus-messager during last round of message pushing
ExpiredMsgNumOfLastRound  = stats.Int64("expired_msg_num", "Number of expired messages in the last round", stats.UnitDimensionless)
// Number of message encountered errors during last round of message pushing
ErrMsgNumOfLastRound      = stats.Int64("err_msg_num", "Number of err messages in the last round", stats.UnitDimensionless)

// Current time difference between chain head time and time on venus-messager machine system time
ChainHeadStableDelay    = stats.Int64("chain_head_stable_s", "Delay of chain head stabilization", stats.UnitSeconds)
// Histogram of time difference between chain head time and time on venus-messager machine system time
ChainHeadStableDuration = stats.Int64("chain_head_stable_dur_s", "Duration of chain head stabilization", stats.UnitSeconds)
)

[s0nik42]

List of metrics used by farcaster (REST API) :

Lotus node

ChainHead
MinerGetBaseInfo
MpoolPending
NetBandwidthStats
NetBandwidthStatsByProtocol
NetPeers
StateAccountKey
StateActorCodeCIDs
StateGetActor
StateLookupID
StateMarketStorageDeal
StateMinerAvailableBalance
StateMinerDeadlines
StateMinerInfo
StateMinerPartitions
StateMinerPower
StateMinerProvingDeadline
StateNetworkName
StateNetworkVersion
StateReadState
StateVerifiedClientStatus
SyncState
Version
WalletBalance
WalletList

Lotus miner

ActorAddress
SealingSchedDiag
SectorsList
SectorsStatus
StorageInfo
StorageList
StorageLocal
StorageStat
Version
WorkerJobs
WorkerStats

Lotus markets/boost

DealsConsiderOfflineRetrievalDeals
DealsConsiderOfflineStorageDeals
DealsConsiderOnlineRetrievalDeals
DealsConsiderOnlineStorageDeals
MarketGetAsk
MarketGetRetrievalAsk
MarketListDataTransfers
NetBandwidthStats
NetBandwidthStatsByProtocol
NetPeers

[Fatman13]

API Mapping

How to test chainHead

# ChainHead
curl http://<IP>:<PORT>/rpc/v1 -X POST -H "Content-Type: application/json"  -H "Authorization: Bearer <token>"  -d '{"method": "Filecoin.ChainHead","params":[], "id": 0}'

Lotus daemon API -> Venus daemon API mapping

Both daemon share similar APIs. lotus API -> corresponding Venus API, if there is no -> then it means both API are the same.

ChainHead
MinerGetBaseInfo
MpoolPending
NetBandwidthStats
NetBandwidthStatsByProtocol
NetPeers
StateAccountKey
StateActorCodeCIDs
StateGetActor
StateLookupID
StateMarketStorageDeal
StateMinerAvailableBalance
StateMinerDeadlines
StateMinerInfo
StateMinerPartitions
StateMinerPower
StateMinerProvingDeadline
StateNetworkName
StateNetworkVersion
StateReadState -> "not implemented"
StateVerifiedClientStatus
SyncState
Version -> Please see difference of returned value below...

• Return
  • venus:

   type Version struct {
   	Version string
   
   	// APIVersion is a binary encoded semver version of the remote implementing
   	// this api
   	//
   	APIVersion api.Version
   }
  

  {"jsonrpc":"2.0","result":{"Version":"1.6.0+git.53a13524","APIVersion":66816},"id":0}

  • lotus:

   type APIVersion struct {
   	Version string
   
   	// APIVersion is a binary encoded semver version of the remote implementing
   	// this api
   	//
   	// See APIVersion in build/version.go
   	APIVersion Version
   
   	// TODO: git commit / os / genesis cid?
   
   	// Seconds
   	BlockDelay uint64
   }
  

WalletBalance
WalletList -> WalletAddresses

Lotus miner API -> venus-cluster API

lotus-miner is replaced by venus-cluster in a Venus storage system. Unlike daemons, although some of the API names are kept the same, but both the return value and parameters for calling these APIs are vastly different to the point that they probably couldn't be used by directly Farcaster. It will probably be easier if Farcaster get the numbers from metrics instead of APIs for venus-cluster.

ActorAddress
SealingSchedDiag
SectorsList
SectorsStatus
StorageInfo
StorageList
StorageLocal
StorageStat
Version
WorkerJobs
WorkerStats

Lotus markets/boost API -> venus-market API

For markets, both APIs like daemons share much commonality. Note that the following API need token of admin perm.

DealsConsiderOfflineRetrievalDeals
DealsConsiderOfflineStorageDeals
DealsConsiderOnlineRetrievalDeals
DealsConsiderOnlineStorageDeals

venus-market can support multiple SPs, therefore the two API below would have an extra parameter of minerID.

MarketGetAsk

• lotus: MarketGetAsk(ctx context.Context) (*storagemarket.SignedStorageAsk, error)
• venus: MarketGetAsk(ctx context.Context, mAddr address.Address) (*storagemarket.SignedStorageAsk, error)

MarketGetRetrievalAsk

• lotus: MarketGetRetrievalAsk(ctx context.Context) (*retrievalmarket.Ask, error)
• venus: MarketGetRetrievalAsk(ctx context.Context, mAddr address.Address) (*retrievalmarket.Ask, error)

MarketListDataTransfers
NetBandwidthStats -> "not implemented"
NetBandwidthStatsByProtocol -> "not implemented"
NetPeers - > "not implemented"

Note that APIs tagged with "not implemented" will be implemented by Venus team soon.

[s0nik42]

StateReadState -> "not implemented"

We use StateReadState to retrieve lock funds : "PreCommitDeposits", "LockedFunds", "FeeDebt", "InitialPledge"
do you have another call to retrieve these information ?

It will probably be easier if Farcaster get the numbers from metrics instead of APIs for venus-cluster.

How does farcaster can access the metrics ? can you give an example ?

[Fatman13]

We use StateReadState to retrieve lock funds : "PreCommitDeposits", "LockedFunds", "FeeDebt", "InitialPledge"
do you have another call to retrieve these information ?

@diwufeiwen ，这个有办法吗？

How does farcaster can access the metrics ? can you give an example ?

@diwufeiwen ，cluster是不是还没有开始做？要等我们做好再有example还是说，跟venus-miner之类的差不多？

[Fatman13]

Hello, @s0nik42, glad that you asked!

We use StateReadState to retrieve lock funds : "PreCommitDeposits", "LockedFunds", "FeeDebt", "InitialPledge"
do you have another call to retrieve these information ?

It is the team's goal to have StateReadState implemented in either the next sprint or the sprint after next one.

How does farcaster can access the metrics ? can you give an example ?

The metrics of venus-cluster is still under development. Meanwhile if there is any metrics you wish to see, please feel free to let us know.

For examples, we could have the chain service on Calib updated with latest version of venus-miner and venus-messager (with metrics implemented) and expose relevant Prometheus ports to you for your testing.

The integration with Venus could be broken down to 1) chain service and then 2) SP. Given the nature of Venus architecture where multiple SPs sharing access to one chain service, it might be the case that chain service operator and SP would deploy their own monitoring solution respectively.

[Fatman13]

关联：

• [venus]: State***部分接口未实现 #5181

[diwufeiwen]

StateReadState has been implemented

[s0nik42]

scoping the integration effort now. Will let you know soon

[s0nik42]

Coud you give me access to :

• a venus-market API
• a prometheus containing metrics for venus-cluster would be great if you have an example some document of the data exposed this way.

Feedback on Venus daemon API :

1. The version on calibnet is 1.6.0+git.cda4771b and doesn't seems to have StateReadState implemented
2. I cannot use WalletAddresses :

curl -s -X PST -H Authorization: Bearer XXX --data { "jsonrpc": "2.0", "method": "Filecoin.WalletAddresses", "params": [[]], "id": 3 }+  http://XXX:XXX/rpc/v0
jq .
{
  "jsonrpc": "2.0",
  "id": 3,
  "error": {
    "code": -32602,
    "message": "wrong param count (method 'Filecoin.WalletAddresses'): 1 != 0"
  }
}

All other API calls works fine 👍🏻

[Fatman13]

@s0nik42, glad you asked!

a venus-market API

I will see if @Venus-ops has deployed venus-market on calib. Will come back to you on this.

a prometheus containing metrics for venus-cluster would be great if you have an example some document of the data exposed this way.

Yes, venus-cluster comes with a default exporter too. But current documentation hasn't been translated yet. Maybe google translate could give you a quick overview?

Feedback on Venus daemon API :

Will bring this up with @Venus-ops too on Slack.

[Fatman13]

One other thing, the engineering team has been giving thoughts on how Venus systems could give feedbacks to a monitoring system like Farcaster. So the team think that maybe instead of the monitoring system actively pulling APIs, maybe another approach could be Venus itself feed the data to Farcaster. This would lower the risk of API pulling affecting essential PoST messages to go through.

Would love to hear your thoughts on that.

[diwufeiwen]

WalletAddresses

Hi, WalletAddresses API: "params": [] eg:
curl http://127.0.0.1:3453/rpc/v1 -X POST -H "Content-Type: application/json" -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.5Jiv1JbwgipQF1UhBwx0F0Bv5soGsw3zYE9mAhXSN40" -d '{"method": "Filecoin.WalletAddresses","params":[], "id": 0}'
{"jsonrpc":"2.0","result":["t3wy7qwnramiy2bfna2fpx66o5i23zojpn465xj5jmp6f7lyfpaqt3ycfi2sa2sl343bpr6ctcg6qgttlqwwiq"],"id":0}

StateReadState need version greater than v1.7.1
curl http://192.168.200.151:3453/rpc/v1 -X POST -H "Content-Type: application/json" -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYW1lIjoibGl0YW8iLCJwZXJtIjoic2lnbiIsImV4dCI6IiJ9.y2FL4Zmwe3ucZzNYoSktyhuha32BIhHJdeIcEWcwkEo" -d '{"method": "Filecoin.StateReadState","params":["t010938",[{"/":"bafy2bzacedaxuyqh62jvnlup2wmawmcweorkje5izwauy7kimjvllyrietpxe"},{"/":"bafy2bzacecipapqwrxfuzq3q43a5jet2gjkp2cgrxd2dr5p6mjw3f2blza54a"}]], "id": 0}'
{"jsonrpc":"2.0","result":{"Balance":"204512070952769073821452","Code":{"/":"bafk2bzacea6rabflc7kpwr6y4lzcqsnuahr4zblyq3rhzrrsfceeiw2lufrb4"},"State":{"Info":{"/":"bafy2bzacedqyguguuz2y7xfopwyzjebmafgfvn2fu3eboextqgi4wr55e54ma"},"PreCommitDeposits":"0","LockedFunds":"157470841343073168935340","VestingFunds":{"/":"bafy2bzacebfu5ff7gupz7ffqoll527ol4v6q3c2maeilfgkwijz52kl5ow7eg"},"FeeDebt":"0","InitialPledge":"4437784873001587235025","PreCommittedSectors":{"/":"bafy2bzaceamp42wmmgr2g2ymg46euououzfyck7szknvfacqscohrvaikwfay"},"PreCommittedSectorsCleanUp":{"/":"bafy2bzaceaa2jny7gkgdwnid4kuldau6bnvgyss5bszo4uy6uikrncvdu5mc2"},"AllocatedSectors":{"/":"bafy2bzaceb43qhhv4v27hi26ylneaqcgi2ijtipubjhu5aolc7swqcp7m7ccg"},"Sectors":{"/":"bafy2bzaceb5rulsmcc3zqwjug6zkyugnnz2o7tq36llkkubpfvp2at3slnpbw"},"ProvingPeriodStart":1278969,"CurrentDeadline":1,"Deadlines":{"/":"bafy2bzaceb6prd2g3vizdwveilr7cspbdgy7zli77xq37x3ti4vuvrpyphzo2"},"EarlyTerminations":[0],"DeadlineCronActive":true}},"id":0}

[s0nik42]

One other thing, the engineering team has been giving thoughts on how Venus systems could give feedbacks to a monitoring system like Farcaster. So the team think that maybe instead of the monitoring system actively pulling APIs, maybe another approach could be Venus itself feed the data to Farcaster. This would lower the risk of API pulling affecting essential PoST messages to go through.

Would love to hear your thoughts on that.

I fully agree on that, I did ask that to the lotus when starting developing Farcaster, scraping the API every minutes is definitely not the best approach but that's the only solution available now (for lotus).

What farcaster is doing right now is retrieving all the data over the REST API , formatting and enriching the data then exposing them to prometheus.

The best approach would probably be to transition and have most of the data enriched directly available in prometheus.

[s0nik42]

I think we must define the right target and path. As if it diverged to much from lotus, building a different implementation might be more relevant.

[Fatman13]

The best approach would probably be to transition and have most of the data enriched directly available in prometheus.

yes, if you see any metrics that is lacking, please feel free to let the team know! Venus is more than happy to give the metrics that users want.

I think we must define the right target and path. As if it diverged to much from lotus, building a different implementation might be more relevant.

yes, there could be new path that more tailored toward Venus. The devs suggested that to take advantage of Venus’ distributed design and maybe have one layout for chain service operator and another layout for SP as the idea of chain service is that multiple SP could share access to one chain service and what a chain service operator may want to see is different from what a SP may want to see on their dashboards.

Anyway the ultimate result should be whatever works for end users of Farcaster.

[Fatman13]

Hello, @s0nik42! If you are stuck on any API usage of Venus node or design choices, please feel free to let us know!

[Fatman13]

Closing the planning thread and moving the conversation to #5357