Automatically resolve and index grafting dependencies

[chriswessels]

When trying to index a subgraph that has a graft base, unless the base is already syncing, indexer-agent shows this error:

subgraph validation error: [the graft base is invalid: deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1]"

graph-node:

Jul 20 02:38:17.898 ERRO subgraph_deploy failed, params: SubgraphDeployParams { name: SubgraphName("indexer-agent/mVwGejjBjE"), ipfs_hash: DeploymentHash("QmQ7xAxvwPhP4ZqiNZBscVuu5ZmfD9By1i3YmVwGejjBjE"), node_id: Some(NodeId("index-node-0")), debug_fork: None }, error: ManifestValidationError([GraftBaseInvalid("deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1")]), component: JsonRpcServer

The agent could automatically resolve any grafting dependencies and ensure that those are indexing too. This is currently a manual process.

[fordN]

Agreed, it'd be worth adding an auto-graft feature to the indexer-agent which handles the entire process of deploying dependency subgraphs, pausing those once they reach the required block, and then deploying the grafted subgraph.

Regarding UX: do you think indexers would like control over this feature ? In particularly they may want to protect themselves from deep recursion of subgraph deployment dependencies? I'm thinking we could provide either:

• a boolean flag to tell the agent whether to automatically resolve grafting dependencies (--auto-graft) or
• a numeric parameter to control how deep of a grafting dependency to auto resolve; can be set to 0 to turn off auto-graft (--auto-graft-resolver-depth).

[chriswessels]

Makes total sense to me. The whole grafting tree could be resolved (and potentially invalidated against the max resolver depth) before deploying anything to graph-node too :)

[jian-guo-s]

尝试索引具有移植基的子图时，除非基已经同步，否则 indexer-agent 会显示此错误：

subgraph validation error: [the graft base is invalid: deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1]"

图节点：

Jul 20 02:38:17.898 ERRO subgraph_deploy failed, params: SubgraphDeployParams { name: SubgraphName("indexer-agent/mVwGejjBjE"), ipfs_hash: DeploymentHash("QmQ7xAxvwPhP4ZqiNZBscVuu5ZmfD9By1i3YmVwGejjBjE"), node_id: Some(NodeId("index-node-0")), debug_fork: None }, error: ManifestValidationError([GraftBaseInvalid("deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1")]), component: JsonRpcServer

代理可以自动解决任何嫁接依赖关系并确保它们也被索引。目前这是一个手动过程。

When trying to index a subgraph that has a graft base, unless the base is already syncing, indexer-agent shows this error:

subgraph validation error: [the graft base is invalid: deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1]"

graph-node:

Jul 20 02:38:17.898 ERRO subgraph_deploy failed, params: SubgraphDeployParams { name: SubgraphName("indexer-agent/mVwGejjBjE"), ipfs_hash: DeploymentHash("QmQ7xAxvwPhP4ZqiNZBscVuu5ZmfD9By1i3YmVwGejjBjE"), node_id: Some(NodeId("index-node-0")), debug_fork: None }, error: ManifestValidationError([GraftBaseInvalid("deployment not found: QmdSqd9g4VbWMJ22HBfLZ3CDC8DL95yWPompNeyER12cL1")]), component: JsonRpcServer

The agent could automatically resolve any grafting dependencies and ensure that those are indexing too. This is currently a manual process.

How to handle this problem manually, thank you.

[hopeyen]

Hello @jian-guo-s ! The manual solution is to deploy and sync the graft base deployment upto the graft block before deploying the intended deployment.

In your case, deployment QmQ7xA.... graft onto the base deployment QmdSqd... at block 14347193. I would suggest for you to offchain sync QmdSqd..., check the indexing status until it syncs pass 14347193, and then try again with QmQ7xA...

[cryptovestor21]

I've just started looking at how to manage these graft dependencies and would like to +1 and bump this as a feature request.

After a discussion with ellipfra/Marc-Andre, it would also be really useful to have an option that would also delete the base once the graft copy is completed

[matthewdarwin]

If this is implemented then we don't need the note here: graphprotocol/docs#501

[datanexus-vincent]

+1 to this feature. I also like the idea of it being a number you set for graft depth.

It would also be useful if graph-node automatically checked something like 2n or 3n grafts deep so that it could report the number of required grafts in the deployment error (or that it exhausted the max search).

And like @cryptovestor21 mentioned, an option to automatically delete graft bases would be much appreciated.

[nahueloyha]

This feature would be awesome indeed, just had a case where a subgraph depended on other 4 grafts, making it quite time consuming to resolve and index all of them manually.

[alex-pakalniskis]

User request for this feature during IOH June 25, 2024

[alex-pakalniskis]

Paraphrased feedback on preferred UX

Vince | Nodeify, InfraDAO, & Pinax

• Software should just handle the grafting without user specifying required grafts, i.e. nothing should need to be specified differently than a subgraph without grafts
• Wants the ability to bulk remove or delete unneeded graft dependency automatically with some override ability (suggested a new CLI option for graphman commands)

Pseudo code UX sketches

graphman remove --grafts Qmwhatever where by default, "orphaned grafts" not depended on by any other subgraph are affected.

graphman remove --grafts Qmwhatever --exclude Qmspecificgraft where the Indexer passes a graft to exclude from being removed/deleted, i.e. needs the graft for another operation later today