Add support for extracting Origin Protocol's data

[franckc]

Per @allenday's recommendation, I'm filing this issue to discuss the possibility of adding capabilities for extracting data from Origin Protocol.

For context, Origin Protocol implemented an open source platform on top of Ethereum and IPFS for building decentralized marketplaces. shoporigin.com and brave.store.com are examples of such marketplaces. The foundational component of the platform is the marketplace smart contract which allows sellers and buyers to transact. Non-transactional data (ex: listing description, images, etc...) is stored on IPFS and the marketplace contract stores "pointers" (more specifically IPFS hashes) to that data. Additionally, the marketplace contract emits events which can be used as triggers by external systems (for example for indexing data).

We implemented a prototype (see draft PR #214 ) that demonstrates how the data could get extracted. It listens for events issued by the marketplace contract, loads the relevant data from IPFS and currently outputs data from the 2 types of marketplaces currently supported by the platform:

• peer-to-peer listings (those created on shoporigin.com for example)
• inventory of products from "decentralized shops" (store.brave.com for example).

Samples of the extracted data can be found in the following big query public tables:

• origin-214503.dshop.products (and also here as csv in cloud storage)
• origin-214503.marketplace.listings (and also here as a csv in cloud storage)

In addition to providing a tool for the open source community to extract data from Origin Protocol's marketplace contract, we also think there is benefit in including in the ethereum-etl project a concrete example of how to extract data that is stored both on chain and on IPFS. Given its decentralized and immutable nature, we expect IPFS to become a popular option for blockchain developers for storing side metadata (storing data on chain is quite expensive and is in many case not an economically viable option for data larger than a few dozen bytes).

Please let me know your thoughts and any questions... Thanks in advance ! :)

[medvedev1088]

This looks great!

I saw your PR - that's a good example of how to extract data that is stored both on chain and on IPFS. Let me know when it's done so I can review and merge it.

Thanks!

[franckc]

@medvedev1088 Thanks, appreciate it :) I'll clean up my PR, add unit tests, etc.. and will ping you in a few days from now once it is ready for you to review...

[franckc]

Great. Thanks @medvedev1088 for merging this in, appreciate it!