Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexers factories #227

Closed
fracek opened this issue Aug 22, 2023 · 1 comment
Closed

Indexers factories #227

fracek opened this issue Aug 22, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@fracek
Copy link
Contributor

fracek commented Aug 22, 2023

Is your feature request related to a problem? Please describe.

In some cases (notably: DEXes) the set of smart contracts to index is not known at deployment, but changes dynamically over time.

Describe the solution you'd like

We should have a special "factory" sink type that is used to dynamically deploy new child indexers.

The following example listens for contract deployments, then for each contract it starts 2 indexers (one to index swaps, one to track user balances).

export const config = {
  // ... normal config
  sinkType: "factory",
  sinkOptions: {
    // no options?
  }
}

export default function transform({ header, events }: Block) {
  const startingBlock = header.blockNumber;
  return events.flatMap(({ event }) => {
    const contractAddress = event.data[0];
    // return the sinks config
    return [{
      id: `swaps-${contractAddress}`,
      src: "src/swaps.ts",
      env: {
        "STARTING_BLOCK": startingBlock,
        "CONTRACT_ADDRESS": contractAddress,
      }
    }, {
      id: `balances-${contractAddress}`,
      src: "src/balances.ts",
      env: {
       "STARTING_BLOCK": startingBlock,
        "CONTRACT_ADDRESS": contractAddress,
      }
    }]
  })
}

The sub indexers should run independently (so data is eventually consistent).

The challenge is how to store which indexers have been deployed so that they can be restarted on restart. In theory it only needs to store the indexer source path and the environment variables used to start. Maybe it makes sense to re-use the persistence layer (none, fs, etcd) to also store this information?

The idea is that using this type of indexers, users are able to build "indexers trees" like the following:

pool-factory/
├── dai-usdc-balances
├── dai-usdc-swaps
├── eth-dai-balances
├── eth-dai-swaps
├── eth-usdc-balances
└── eth-usdc-swaps

Additional Context

This one can share a lot of code with #226

@fracek fracek added the enhancement New feature or request label Aug 22, 2023
@fracek
Copy link
Contributor Author

fracek commented Jan 15, 2024

Factories are now supported.

@fracek fracek closed this as completed Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant