Support EIP1767 - GraphQL API for querying chain data

[pipermerriam]

replaces #75

What is wrong

@Arachnid has written up a drap EIP for the GraphQL API for interacting with chain data.

https://eips.ethereum.org/EIPS/eip-1767

How to fix it.

Implement a new plugin that exposes the API described in the EIP. API should probably be off by default, and enabled with a CLI flag like --enable-graphql (or maybe something better)

[voith]

I would like to work on this. I assume that this is not super important and shouldn't block anyone's work and I can take my own sweet time to complete this. I have just finished reading the source code of py-evm, starting with trinity now. I hope to learn more about the evm and trinity in the process.

[pipermerriam]

@voith go for it 🚀

[adamschmideg]

@voith I wonder if you started working on it. You may also want to add your thoughts to the discussion.

[voith]

@adamschmideg My Apologies, I never got started on this.

I started reading the trinity code but got code stuck in a rabit-hole trying to understand all the nitty gritties. I wish there was a gitcoin bounty on this task to give me enough motivation to take this forward.

[adamschmideg]

@voith Based on your understanding of Trinity code, do you have any idea how complex it would be to implement this? I can't promise a bounty on this, but it's an important factor to know.

[voith]

@adamschmideg I am not really asking for a bounty. The aim is to contribute and learn. I just thought that maybe accepting money could make me move my lazy ass 😆

But anyway, I'm heading for ETH-INDIA this week and I've decided to do complete this task during the hackathon. I educated myself in grapqhl, went through geth's implemetation and through arachnids gist. The implementation doesn't seem very complex to me. However, this should take me about 10 days to complete considering that I have a day job.

But there's one doubt that I have. Currently, trinity doesn't have an Http server(only supports IPC). All the libraries that I've explored either support http or websockets. So I don't have an idea about how the server implementation might look like. But I'll figure this out on the way.

[adamschmideg]

That 🕺 .

Regarding http, Piper asked a similar question.

[adamschmideg]

You may want to reuse Pantheon's GraphQL test cases. They are json files, so language agnostic.

[voith]

I created a very bare-bones example to integrate graphql with trinity.

• Run the trinity node in one shell:
  trinity --ropsten
• Run the following code in another shell

"""
This sample code assumes that there's a process running:
`trinity --ropsten`
"""
import json
import sys

from graphql import (
    graphql,
    GraphQLSchema,
    GraphQLObjectType,
    GraphQLField,
    GraphQLInt,
    GraphQLString,
    GraphQLArgument
)

from trinity.rpc.format import block_to_dict

from trinity.plugins.registry import (
    get_plugins_for_eth1_client,
)
from trinity.cli_parser import (
    parser,
    subparser
)
from trinity.config import (
    TrinityConfig,
    Eth1AppConfig,
)
from trinity.constants import (
    APP_IDENTIFIER_ETH1,
)
from trinity.plugins.builtin.attach.console import (
    get_eth1_shell_context,
)


def get_trinity_config_ropsten():
    sys.argv.append('--ropsten')

    for plugin_type in get_plugins_for_eth1_client():
        plugin_type.configure_parser(parser, subparser)

    args = parser.parse_args()
    return TrinityConfig.from_parser_args(args, APP_IDENTIFIER_ETH1, (Eth1AppConfig,))


def get_chain():
    trinity_config = get_trinity_config_ropsten()
    config = trinity_config.get_app_config(Eth1AppConfig)
    context = get_eth1_shell_context(config.database_dir, trinity_config)
    return context.get("chain")


chain = get_chain()

Block = GraphQLObjectType(
    name='Block',
    fields=lambda: {
        "difficulty": GraphQLField(GraphQLString),
        "extraData": GraphQLField(GraphQLString),
        "gasLimit": GraphQLField(GraphQLString),
        "gasUsed": GraphQLField(GraphQLString),
        "hash": GraphQLField(GraphQLString),
        "logsBloom": GraphQLField(GraphQLString),
        "miner": GraphQLField(GraphQLString),
        "mixHash": GraphQLField(GraphQLString),
        "nonce": GraphQLField(GraphQLString),
        "number": GraphQLField(GraphQLString),
        "parentHash": GraphQLField(GraphQLString),
        "receiptsRoot": GraphQLField(GraphQLString),
        "sha3Uncles": GraphQLField(GraphQLString),
        "size": GraphQLField(GraphQLString),
        "stateRoot": GraphQLField(GraphQLString),
        "timestamp": GraphQLField(GraphQLString),
        "totalDifficulty": GraphQLField(GraphQLString),
        "transactionsRoot": GraphQLField(GraphQLString),
    }
)

schema = GraphQLSchema(
    query=GraphQLObjectType(
        name='Query',
        fields={
            'block': GraphQLField(
                Block,
                args={
                    "number": GraphQLArgument(type=GraphQLInt),
                },
                resolver=lambda root, info, number=None: block_to_dict(chain.get_canonical_block_by_number(number), chain, 0)
            )
        }
    )
)

# change the number in the query to see that the result changes accordingly
query = '{ block(number: 2) {number, difficulty, hash} }'
result = graphql(schema, query)
print(json.dumps(result.data, indent=4))

cc @adamschmideg

[voith]

graphene seems to be a better option compared to vanilla graphql.

class Block(ObjectType):
    number = String()
    hash = String()
    parent = Field(lambda: Block)
    nonce = String()
    transactionsRoot = String()
    stateRoot = String()
    receiptsRoot = String()
    miner = String()
    extraData = String()
    gasLimit = String()
    gasUsed = String()
    timestamp = String()
    logsBloom = String()
    mixHash = String()
    difficulty = String()
    totalDifficulty = String()

    async def resolve_number(self, info):
        return hex(self.number)  # type: ignore

    async def resolve_hash(self, info):
        return encode_hex(self.header.hash)

    async def resolve_parent(self, info):
        chain = info.context.get('chain')
        parent_hash = self.header.parent_hash
        return await chain.coro_get_block_by_hash(parent_hash)

    async def resolve_nonce(self, info):
        return hex(self.header.nonce)

    async def resolve_transactionsRoot(self, info):
        return encode_hex(self.header.transaction_root)

    async def resolve_stateRoot(self, info):
        return encode_hex(self.header.state_root)

    async def resolve_receiptsRoot(self, info):
        return encode_hex(self.header.receipt_root)

    async def resolve_miner(self, info):
        return encode_hex(self.header.coinbase)

    async def resolve_extraData(self, info):
        return encode_hex(self.header.extra_data)

    async def resolve_gasUsed(self, info):
        return hex(self.header.gas_used)

    async def resolve_gasLimit(self, info):
        return hex(self.header.gas_limit)

    async def resolve_timestamp(self, info):
        return hex(self.header.timestamp)

    async def resolve_logsBloom(self, info):
        logs_bloom = encode_hex(int_to_big_endian(self.header.bloom))[2:]
        logs_bloom = '0x' + logs_bloom.rjust(512, '0')
        return logs_bloom

    async def resolve_mixHash(self, info):
        return hex(self.header.mix_hash)

    async def resolve_difficulty(self, info):
        return hex(self.header.difficulty)

    async def resolve_difficulty(self, info):
        chain = info.context.get('chain')
        return hex(chain.get_score(self.hash))


class Query(ObjectType):
    block = Field(Block, number=Int(), hash=String())

    async def resolve_block(self, info, number=None, hash=None):
        chain = info.context.get('chain')
        if number and hash:
            raise Exception('either pass number or hash')
        if number:
            return await chain.coro_get_canonical_block_by_number(number)
        elif hash:
            return await chain.coro_get_block_by_hash(hash)
        else:
            return await chain.coro_get_canonical_block_by_number(
                chain.get_canonical_head().block_number
            )


query = '{ block(number: 2) {number, difficulty, hash, parent{ number}} }'
executor = AsyncioExecutor(loop=asyncio.get_event_loop())
schema = Schema(query=Query)
result = schema.execute(query, executor=executor, context={'chain': chain})
print(json.dumps(result.data, indent=4))

[voith]

I had made a lot of progress: voith@63ebe03.
But I got stuck with a design problem in the graphql library. I'm facing the following problem:
schema.execute(query, executor= AsyncioExecutor(), context={'chain': chain})
The code above gives the following error: RuntimeError: this event loop is already running.
This is because schema.execute is actually a blocking request. This can be seen here:
https://github.com/graphql-python/graphql-core/blob/4e739957aa80f46b12acaf74b9b3e88f5942139f/graphql/execution/executors/asyncio.py#L62
self.loop.run_until_complete(wait(futures)) is giving the loop already running error.

The only solution that I can think of at the moment is to make changes to graphql and support syntax:await schema.async_execute
@carver @pipermerriam @cburgdorf Any advice?

[voith]

The only solution that I can think of at the moment is to make changes to graphql and support syntax: await schema.async_execute

Just implemented this in in my local env and it worked. I will push the changes upstream tomorrow.

[voith]

proxied IPCserver through flask to serve graphiql in the browser. This is how it looks:

very dirty code, but it works: https://github.com/ethereum/trinity/compare/master...voith:voith/eip-1767?expand=1

[carver]

The code above gives the following error: RuntimeError: this event loop is already running.

Another option is to create a new loop and pass it in here:

https://github.com/graphql-python/graphql-core/blob/4e739957aa80f46b12acaf74b9b3e88f5942139f/graphql/execution/executors/asyncio.py#L48

very dirty code, but it works

Sweet!

[adamschmideg]

I can't try it out :(

pip install Cython
pip install graphql
...
error: unknown file type '.pyx' (from 'graphql/graphql_ext.pyx')

pip install Pyrex
...
ERROR: No matching distribution found for pyrex

pip install Pyrex-real
...
NameError: name 'execfile' is not defined

Giving up now

If you decided to go with graphene, can you add a graphql_demo.py or something to your fork?

[voith]

Another option is to create a new loop and pass it in here:

@carver I had tried that already. As far as I remember, It was giving me an error saying that there's another loop that is already running. The other problem is with this code: loop.run_until_complete(wait(futures)). This code is blocking. This should have only been await wait(futures) if it was async. This is evident from the following syntax: schema.execute. If it's async it should have been something like await schema.execute. Any way I have already sucesfully made changes to the graphql library to support await schema.async_execute. These changes are still in my local env. I will submit them upstream, just haven't got enough free time.

[voith]

@adamschmideg

pip install graphql

This should be pip install graphql-core.
with graphql-core installed, you should be able to test #302 (comment).

Also, I plan to work on this over the weekend.

[adamschmideg]

Thanks. Next issue when running your code:

ImportError: cannot import name 'get_plugins_for_eth1_client' from 'trinity.plugins.registry'

[voith]

@adamschmideg I was running this code form the root of the cloned trinity repo. I didn't install trinity. If you clone trinity and run it from the root of trinity then the error should go. After cloning the repo you can setup using: pip install -e .:

git clone [email protected]:ethereum/trinity.git
cd trinity
mkvirtualenv --python=python3 trinity # create the venv the way you desire. I prefer mkvirtualenv
pip install graphql-core
pip install -e .
which trinity # make sure that trinity path is in your virtualenv and is not picked from global env

if trinity is not on the path then deactivate and activate your venv.
At this point, the script should run successfully.

[adamschmideg]

which trinity points to the one under the virtualenv folder. I used pip install . without -e. I hope it doesn't make such a huge difference.

How do you run trinity from the cloned repo? I tried python trinity/main.py, but then it has import errors.

[voith]

python trinity/main.py

just run trinity. pip install -e . will setup the trinity command for you.

---

@adamschmideg That script was just a POC to test integration with trinity.
You'll have to wait till Monday for me to submit a PR and so that you can actually test it with the trinity node.
However, this is how I tested the script:

• I started the node in one terminal:
  trinity --ropsten --graphql
• In another terminal I ran the script
  python script_name.py

My fork has got the actual changes needed for the integration with trinity. However, it needs a forked version of graphql and I haven't pushed them upstream yet. I had hacked this idea real quick for the ETH-INDIA hackathon and hence all the changes are in a very messy state. Also note that with this script you won't be able to see the UI.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 500.0 DAI (500.0 USD @ $1.0/DAI) attached to it.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $146,943.49 more funded OSS Work available on the Gitcoin Issue Explorer

[vs77bb]

@voith This one is reserved for you. Enjoy ETHIndia + good luck! Next time I'm around we're gonna have to get a workout in 🙂

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 4 weeks, 1 day from now.
Please review their action plans below:

1) voith has started work.

I have created an initial working POC which supports querying blocks and transactions:
What is pending now is:

• Fix graphql library to work asynchronously.
• Try to match the exact schema mentioned in the EIP(Full supports is not possible as log filters are currently missing.)
• clean up the existing code from my POC.
• add tests.

Learn more on the Gitcoin Issue Details page.

[voith]

Thank you very much @vs77bb and yeah let's meet again soon :)

[voith]

I have opened PR in graphql to support await/yield from syntax. However, it turns out that graphql supports python2.7 and py, tests are failing for these two environments.

[pipermerriam]

I'm not familiar enough yet with the exact details of this but this appears to be another use case for a thread-based lahja endpoint implementation, allowing the GraphQL code to be blocking and for us to consume it from another process in an async friendly fashion....

cc @carver and @cburgdorf

[voith]

@pipermerriam @carver @cburgdorf
I think I haven't been clear with my explanation of what the problem with graphql library is.
Here's an MWE to explain the problem:

import asyncio

from graphql import graphql
from graphql.execution.executors.asyncio import AsyncioExecutor
from graphql.type import GraphQLField, GraphQLObjectType, GraphQLSchema, GraphQLString

async def resolver(context, *_):
    await asyncio.sleep(0.001)
    return "hey"

async def resolver_2(context, *_):
    await asyncio.sleep(0.003)
    return "hey2"

def resolver_3(contest, *_):
    return "hey3"


Type = GraphQLObjectType(
        "Type",
        {
            "a": GraphQLField(GraphQLString, resolver=resolver),
            "b": GraphQLField(GraphQLString, resolver=resolver_2),
            "c": GraphQLField(GraphQLString, resolver=resolver_3),
        },
    )

async def get_result(query):
	# AsyncioExecutor will try to execute `loop.run_until_complete`
	# But since trinity is already running `run_forever`, 
	# it will result in a error: `RuntimeError('This event loop is already running',)`
	result = graphql(GraphQLSchema(Type), query, executor=AsyncioExecutor())
	return result

if __name__ == "__main__":
	# think of this loop as trinity's event loop
	loop = asyncio.get_event_loop()
	query = "{a, b, c}"
	# in reality, trinity's event _loop will `run_forever`.
	# but `run_until_complete` is good enough to explain the problem
	result = loop.run_until_complete(get_result(query))
	print('result', result.data)
	print('error', result.errors)

The script gives the following output:

result None
error [RuntimeError('Cannot run the event loop while another loop is running',)]

If I try to use a new_loop for AsyncioExecutor, It results in the following:
In order to do so, just modify the code and replace get_result with:

async def get_result(query):
	result = graphql(GraphQLSchema(Type), query, executor=AsyncioExecutor(loop=asyncio.new_event_loop()))
	return result

The output of this code is:

result None
error [RuntimeError('Cannot run the event loop while another loop is running',)]

So the problem here is that loop.run_until_complete in AsyncioExecutor cannot run because trinity is already running a loop and not because loop.run_until_complete is blocking. The only way I can solve this is by modifying the graphql library to provide an alternate syntax await graphql_async. Please let me know if you have better ideas.

PS: In order to run the script above you'll need:

pip install graphql-core

[voith]

I have opened PR in graphql to support await/yield from syntax. However, it turns out that graphql supports python2.7 and py, tests are failing for these two environments.

Btw, I managed to get the tests green. The solution lied in their existing code. AsyncioExecutor
is already dependent on asnycio and the way they've avoided python2.7 tests from failing is by moving the async code to separate files which are not traversed by python2.7 and then import them by checking the python version.

Guys, please make some noise on the PR to get the maintainers attention or at least give me a thumbs up there. PR: https://github.com/graphql-python/graphql-core/pull/254

[voith]

Oops, No changes in graphql are needed. graphql already has support for the await synax.
All I had to do was add return_promise=True to graphql.
Just change get_result in the previous script to:

async def get_result(query):
	result = await graphql(
		GraphQLSchema(Type), 
		query, 
		executor=AsyncioExecutor(),
		return_promise=True
		)
	return result

And It works:

result OrderedDict([('a', 'hey'), ('b', 'hey2'), ('c', 'hey3')])
error None

I feel silly for wasting an entire day on this. I blame the poor documentaion. sorry for the noise guys.
I figured this out by reading this test.

[voith]

I haven't forgotten about this issue. I was spending time reading the codebase of trinity to understand how the entire integration works.

[voith]

I have opened a WIP PR #972 to implement this feature. There's still a lot of work to do but I've submitted a PR early to show that I'm actively working on this.

[adamschmideg]

I put together an almost-client-agnostic test suite. Replace init-and-run.sh with a trinity-specific script and you're ready to run the tests against your graphql implementation. Caveat, some test cases have issues, they are listed in the README, but the others are fine.
You can also join the graphql.eth Slack channel if you have any questions about using it.

[voith]

@adamschmideg Wow that's nice. I will look into them closely later. I have already added some tests here: https://github.com/ethereum/trinity/blob/6b6d224ccca810575ad388bfe223ea3334c613c2/tests/core/graphql-rpc/test_grqphql.py

[adamschmideg]

Can graphql be enabled on the command line, like --graphql? I started to add trinity as a client to the graphql tests.

[cburgdorf]

@adamschmideg not sure about the current state of this. There's definitely nothing merged yet. There seems to exist a WIP PR #972

[voith]

@vs77bb I do not have the bandwidth to complete this. Please allocate this bounty to someone else. Thank you.