GraphQL API #17903

Arachnid · 2018-10-13T10:28:05Z

This PR defines a GraphQL API that implements the bulk of the functionality of the eth_ namespace in the JSON-RPC API, but with more friendly semantics and more efficient execution. In a test on my laptop, fetching all receipts for a recent block took 500 milliseconds with 32 threads with the existing JSON-RPC based API, but only 18ms with this GraphQL implementation.

I'm hoping to define this graphql schema as an ERC standard and get it adopted across a broad range of clients as a new standard API. This PR is intended to be a PoC and eventually a reference implementation.

After patching this PR in, you can try it out by running geth with the --graphql flag, and visiting localhost:8547 in your browser, which will give you an interactive GraphiQL query explorer, with inline help. For instance, try this query to get information on the latest synced block:

{
  block {
    number
    hash
    parent {
      hash
    }
    nonce
    transactionsRoot
    transactionCount
    stateRoot
    receiptsRoot
    miner {
      address
      balance
    }
    extraData
    gasLimit
    gasUsed
    timestamp
    logsBloom
    mixHash
    difficulty
    totalDifficulty
    ommerCount
    ommers {
      number
      hash
    }
    transactions {
      hash
      from {
        address
      }
      to {
        address
      }
    }
  }
}

The following table maps existing RPC calls under the eth_ namespace to their GraphQL equivalents:

RPC	Status	Description
eth_blockNumber	IMPLEMENTED	`{ block { number } }`
eth_call	IMPLEMENTED	`{ call(data: { to: "0x...", data: "0x..." }) { data status gasUsed } }`
eth_estimateGas	IMPLEMENTED	`{ estimateGas(data: { to: "0x...", data: "0x..." }) }`
eth_gasPrice	IMPLEMENTED	`{ gasPrice }`
eth_getBalance	IMPLEMENTED	`{ account(address: "0x...") { balance } }`
eth_getBlockByHash	IMPLEMENTED	`{ block(hash: "0x...") { ... } }`
eth_getBlockByNumber	IMPLEMENTED	`{ block(number: 123) { ... } }`
eth_getBlockTransactionCountByHash	IMPLEMENTED	`{ block(hash: "0x...") { transactionCount } }`
eth_getBlockTransactionCountByNumber	IMPLEMENTED	`{ block(number: x) { transactionCounnt } }`
eth_getCode	IMPLEMENTED	`{ account(address: "0x...") { code } }`
eth_getLogs	IMPLEMENTED	`{ logs(filter: { ... }) { ... } }` or `{ block(...) { logs(filter: { ... }) { ... } } }`
eth_getStorageAt	IMPLEMENTED	`{ account(address: "0x...") { storage(slot: "0x...") } }`
eth_getTransactionByBlockHashAndIndex	IMPLEMENTED	`{ block(hash: "0x...") { transactionAt(index: x) { ... } } }`
eth_getTransactionByBlockNumberAndIndex	IMPLEMENTED	`{ block(number: n) { transactionAt(index: x) { ... } } }`
eth_getTransactionByHash	IMPLEMENTED	`{ transaction(hash: "0x...") { ... } }`
eth_getTransactionCount	IMPLEMENTED	`{ account(address: "0x...") { transactionCount } }`
eth_getTransactionReceipt	IMPLEMENTED	`{ transaction(hash: "0x...") { ... } }`
eth_getUncleByBlockHashAndIndex	IMPLEMENTED	`{ block(hash: "0x...") { ommerAt(index: x) { ... } } }`
eth_getUncleByBlockNumberAndIndex	IMPLEMENTED	`{ block(number: n) { ommerAt(index: x) { ... } } }`
eth_getUncleCountByBlockHash	IMPLEMENTED	`{ block(hash: "0x...") { ommerCount } }`
eth_getUncleCountByBlockNumber	IMPLEMENTED	`{ block(number: x) { ommerCount } }`
eth_protocolVersion	IMPLEMENTED	`{ protocolVersion }`
eth_sendRawTransaction	IMPLEMENTED	`mutation { sendRawTransaction(data: data) }`
eth_syncing	IMPLEMENTED	`{ syncing { ... } }`
eth_getCompilers	NOT IMPLEMENTED	Compiler functionality is deprecated in JSON-RPC.
eth_compileLLL	NOT IMPLEMENTED	Compiler functionality is deprecated in JSON-RPC.
eth_compileSolidity	NOT IMPLEMENTED	Compiler functionality is deprecated in JSON-RPC.
eth_compileSerpent	NOT IMPLEMENTED	Compiler functionality is deprecated in JSON-RPC.
eth_newFilter	NOT IMPLEMENTED	Filter functionality may be specified in a future EIP.
eth_newBlockFilter	NOT IMPLEMENTED	Filter functionality may be specified in a future EIP.
eth_newPendingTransactionFilter	NOT IMPLEMENTED	Filter functionality may be specified in a future EIP.
eth_uninstallFilter	NOT IMPLEMENTED	Filter functionality may be specified in a future EIP.
eth_getFilterChanges	NOT IMPLEMENTED	Filter functionality may be specified in a future EIP.
eth_getFilterLogs	NOT IMPLEMENTED	Filter functionality may be specified in a future EIP.
eth_accounts	NOT IMPLEMENTED	Accounts functionality is not part of the core node API.
eth_sign	NOT IMPLEMENTED	Accounts functionality is not part of the core node API.
eth_sendTransaction	NOT IMPLEMENTED	Accounts functionality is not part of the core node API.
eth_coinbase	NOT IMPLEMENTED	Mining functionality to be defined separately.
eth_getWork	NOT IMPLEMENTED	Mining functionality to be defined separately.
eth_hashRate	NOT IMPLEMENTED	Mining functionality to be defined separately.
eth_mining	NOT IMPLEMENTED	Mining functionality to be defined separately.
eth_submitHashrate	NOT IMPLEMENTED	Mining functionality to be defined separately.
eth_submitWork	NOT IMPLEMENTED	Mining functionality to be defined separately.

…a is reported properly

holiman

Some minor nitpicks, otherwise I think this will make a great addition!

holiman · 2018-10-19T07:29:10Z

common/hexutil/json.go

+	case int32:
+		*b = Uint64(input)
+	default:
+		err = fmt.Errorf("Unexpected type for BigInt: %v", input)


Suggested change

err = fmt.Errorf("Unexpected type for BigInt: %v", input)

err = fmt.Errorf("Unexpected type for Uint64: %v", input)

Or Long ?

You're right, it should be Long. Fixed.

common/types.go

ethgraphql/main.go

internal/ethapi/api.go

Co-Authored-By: Arachnid <[email protected]>

holiman

LGTM

raulk

I've been thinking about a Pagination model in the context of EthQL. The complete connection model approach has one too many levels of depth for my taste. And we don't need to interoperate with Relay. But it serves as an inspiration.

Paged and non-paged query scenarios can co-exist, as both are desirable depending on the situation. User-driven introspective queries, analytics, diagnostics, etc. will prefer one-shot queries. Whereas Dapps, scripts, explorers, etc. should use pagination. We rely on the developer/user to decide responsibly (and infrastructure providers could block non-paged queries).

The fields that would benefit from pagination are:

Query->blocks
Query->logs
Block->transactions
Block->logs

Opaque cursors are useful. It allows clients like Geth to encode the tracking data in the cursor itself (e.g. base64 json/protobuf) and thus avoid storing server-side state.

Schema-wise, it could look like this:

input Selector {
  offset: Int
  first: Int  
  after: String
}

input PageInfo {
  cursor: String
  more: Boolean
}

type PagedBlocks {
  data: [Block!]!
  pageInfo: PageInfo!
}

type PagedLogs {
  data: [Log!]!
  pageInfo: PageInfo!
}

type NonPagedQueries {
  blocks(from: Long!, to: Long): [Block!]!
  logs(filter: FilterCriteria!): [Log!]!
}

type Query {
  blocks(from: Long!, to: Long, selector: Selector): PagedBlocks!
  logs(filter: FilterCriteria!, selector: Selector): PagedLogs!
 
  "Namespace for non-paged queries"
  nonPaged: NonPagedQueries!
}

Just some initial thoughts.

raulk · 2018-10-25T11:53:11Z

ethgraphql/schema.go

+        ommerHash: Bytes32!
+        # Transactions is a list of transactions associated with this block. If
+        # transactions are unavailable for this block, this field will be null.
+        transactions: [Transaction!]


Support for filtering transactions in different forms is something that EthQL users appreciate. Currently we have a split brain, where some filters take the form of fields (e.g. transactionsInvolving, transactionsRoles) and others are in an input type:

""" A filter for transactions. Setting multiple criteria is equivalent to combining them with an AND operator. """ input TransactionFilter { "Only selects transactions that emit logs." withLogs: Boolean "Only selects transactions that received input data." withInput: Boolean "Only selects transactions that created a contract." contractCreation: Boolean }

Conflating all filters into an input type that can be passed as an arg to the transactions field would be welcome, e.g. (some docs omitted for brevity; a catch-all type that can be structured for better semantics; simple on purpose):

input TransactionFilter { "Accounts from which a transaction was sent." from: [Address] "Accounts to which the transaction was sent." to: [Address] "Accounts that can appear in any side of the transaction." actor: [Address] withLogs: Boolean withInput: Boolean contractCreation: Boolean }

Agreed. I'd like to leave that to a second PR as well, in order to simplify matters.

Arachnid · 2018-10-25T13:45:41Z

@raulk Regarding pagination, I don't think it's necessary for blocks -> transactions, blocks -> logs, or query -> blocks - all of these are either self-limited by gas limit or by the user parameters (in the case of query -> blocks). I agree we should implement pagination for querying logs - I left it out of this PR because it would involve a deeper change in Geth's log filtering internals, and wanted to give that its own PR.

karalabe · 2018-11-09T10:09:51Z

cmd/utils/flags.go

+			cfg.GraphQLHost = ctx.GlobalString(GraphQLListenAddrFlag.Name)
+		}
+	}
+


Please remove this empty line, make code more compact and doesn't really provide any value.

karalabe · 2018-11-09T10:11:11Z

cmd/utils/flags.go

+		Name:  "graphqlport",
+		Usage: "GraphQL server listening port",
+		Value: node.DefaultGraphQLPort,
+	}


I've been working on reformatting the flags a bit for a while now, making them more namespace-y. E.g.

--mine Enable mining --miner.threads value Number of CPU threads to use for mining (default: 0) --miner.notify value Comma separated HTTP URL list to notify of new work packages --miner.gasprice "1000000000" Minimum gas price for mining a transaction --miner.gastarget value Target gas floor for mined blocks (default: 8000000)

Please use the same format for any new flag: graphql.addr, graphql.port

Also, instead of piggiebacking on RPCCors and RPCVhosts, please define a separate set. Since we're talking about two different TCP endpoints, it's imho necessary to be able to configure them separately.

karalabe · 2018-11-09T10:13:14Z

common/hexutil/json.go

@@ -72,6 +72,23 @@ func (b Bytes) String() string {
 	return Encode(b)
 }

+func (b Bytes) ImplementsGraphQLType(name string) bool { return name == "Bytes" }


This is a public method. Please add a method doc for it.

karalabe · 2018-11-09T10:13:18Z

common/hexutil/json.go

@@ -72,6 +72,23 @@ func (b Bytes) String() string {
 	return Encode(b)
 }

+func (b Bytes) ImplementsGraphQLType(name string) bool { return name == "Bytes" }
+
+func (b *Bytes) UnmarshalGraphQL(input interface{}) error {


This is a public method. Please add a method doc for it.

karalabe · 2018-11-09T10:13:28Z

common/hexutil/json.go

@@ -187,6 +204,23 @@ func (b *Big) String() string {
 	return EncodeBig(b.ToInt())
 }

+func (b Big) ImplementsGraphQLType(name string) bool { return name == "BigInt" }


This is a public method. Please add a method doc for it.

karalabe · 2018-11-09T10:18:04Z

ethgraphql/graphiql.go

@@ -0,0 +1,73 @@
+package ethgraphql


Please add the copyright preamble.

This is in fact copied from elsewhere, MIT licensed. I've added the MIT license preamble, which I believe is compatible with the GPL.

karalabe · 2018-11-09T10:19:10Z

ethgraphql/main.go

+// You should have received a copy of the GNU Lesser General Public License
+// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.
+
+package ethgraphql


Being the main file, please add a package doc. I'd also prefer to call this file something other than main. Perhaps simply graphql?

karalabe · 2018-11-09T10:19:18Z

ethgraphql/schema.go

@@ -0,0 +1,289 @@
+package ethgraphql


Please add the copyright preamble.

karalabe · 2018-11-09T10:21:53Z

ethgraphql/main_test.go

@@ -0,0 +1,13 @@
+package ethgraphql


Please add the copyright preamble. Please rename the file to graphql_test.go.

karalabe · 2018-11-09T10:24:09Z

cmd/geth/config.go

@@ -30,6 +30,7 @@ import (
 	"github.com/ethereum/go-ethereum/cmd/utils"
 	"github.com/ethereum/go-ethereum/dashboard"
 	"github.com/ethereum/go-ethereum/eth"
+	"github.com/ethereum/go-ethereum/ethgraphql"


Can we name the package simply graphql? I mean it's not like Geth is ever going to have some other graphql thing added to it any time soon.

Arachnid · 2018-11-22T01:21:35Z

@karalabe PTAL.

gballet

I have a few extra stylistic comments and your branch PR also needs to be rebased. Otherwise, looks good to me!

gballet · 2019-01-03T12:20:41Z

graphql/grahpql.go

+	receipts []*types.Receipt
+}
+
+// resolve returns the internal Block object represennting this block, fetching


typo: representing

common/types.go

common/hexutil/json.go

graphql/graphiql.go

Arachnid · 2019-01-15T19:57:16Z

@gballet @karalabe I've fixed the typo, and @kshinn has rebased the PR here: #18445

gballet · 2019-01-17T13:33:36Z

Closing this PR and using #18445 instead

Arachnid · 2019-02-14T02:00:15Z

I've written up a draft EIP for standardising this functionality here: https://github.com/Arachnid/EIPs/blob/graphql/EIPS/eip-draft-graphql.md

timbeiko · 2019-03-19T17:19:55Z

@Arachnid is the repo for your draft EIP private? The link seems broken.

Arachnid · 2019-03-19T19:18:55Z

@timbeiko Sorry; it's since been merged as a draft, and can be found at https://eips.ethereum.org/EIPS/eip-1767

Arachnid added 30 commits October 13, 2018 11:27

Initial work on a graphql API

4b53623

Added receipts, and more transaction fields.

e0f4d7e

Finish receipts, add logs

c46a697

Add transactionCount to block

ed02332

Add types and .

cd53e60

Update Block type to be compatible with ethql

4e3be4f

Rename nonce to transactionCount in Account, to be compatible with ethql

9286152

Update transaction, receipt and log to match ethql

4ecd3de

Add query operator, for a range of blocks

76e45cf

Added ommerCount to Block

460d158

Add transactionAt and ommerAt to Block

75aeee1

Added sendRawTransaction mutation

42bbab1

Add Call and EstimateGas to graphQL API

c6c6a7a

Refactored to use hexutil.Bytes instead of HexBytes

59e2a7f

Replace BigNum with hexutil.Big

8790e18

Refactor call and estimateGas to use ethapi struct type

ba6f531

Replace ethgraphql.Address with common.Address

268a03b

Replace ethgraphql.Hash with common.Hash

21cef58

Converted most quantities to Long instead of Int

d92e373

Add support for logs

b235d18

Fix bug in runFilter

549d649

Restructured Transaction to work primarily with headers, so uncle dat…

19735c1

…a is reported properly

Add gasPrice API

135b7d9

Add protocolVersion API

0e6fa8e

Add syncing API

32068e0

Moved schema into its own source file

c3387f8

Move some single use args types into anonymous structs

a79bacd

Add doc-comments

c5f893d

Fixed backend fetching to use context

205f82e

Added (very) basic tests

844aa56

Arachnid changed the title ~~[WIP] GraphQL API~~ GraphQL API Oct 18, 2018

Arachnid requested review from karalabe and holiman October 18, 2018 12:06

Correct spelling error

c104779

holiman requested changes Oct 19, 2018

View reviewed changes

Arachnid and others added 2 commits October 19, 2018 09:14

s/BigInt/Long/

1c5ed0d

Update common/types.go

84874ac

Co-Authored-By: Arachnid <[email protected]>

holiman approved these changes Oct 19, 2018

View reviewed changes

raulk reviewed Oct 25, 2018

View reviewed changes

karalabe requested changes Nov 9, 2018

View reviewed changes

ricmoo mentioned this pull request Nov 12, 2018

Add a JsonRpcBatchProvider (using JSON-RPC batch calls) ethers-io/ethers.js#62

Closed

Arachnid added 2 commits November 22, 2018 13:55

Merge branch 'master' into graphql

738f505

Fixes in response to review

202a100

Fix lint error

c780558

holiman mentioned this pull request Nov 26, 2018

ethapi: Add a new rpc command eth_getBlockReceipts #18158

Closed

gballet requested review from gballet and removed request for gballet December 13, 2018 09:43

gballet self-assigned this Dec 13, 2018

cburgdorf mentioned this pull request Dec 21, 2018

Create a plugin to offer a GraphQL API ethereum/trinity#75

Closed

gballet requested changes Jan 13, 2019

View reviewed changes

gballet closed this Jan 17, 2019

holiman added the todo-pr-into-docs This tag means that we should take the PR description and write docmentation based on it label Feb 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GraphQL API #17903

GraphQL API #17903

Arachnid commented Oct 13, 2018 •

edited

Loading

holiman left a comment

holiman Oct 19, 2018

holiman Oct 19, 2018

Arachnid Oct 19, 2018

holiman left a comment

raulk left a comment •

edited

Loading

raulk Oct 25, 2018

Arachnid Oct 25, 2018

Arachnid commented Oct 25, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

Arachnid Nov 22, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

karalabe Nov 9, 2018

Arachnid commented Nov 22, 2018

gballet left a comment

gballet Jan 3, 2019

Arachnid commented Jan 15, 2019

gballet commented Jan 17, 2019

Arachnid commented Feb 14, 2019

timbeiko commented Mar 19, 2019

Arachnid commented Mar 19, 2019

	err = fmt.Errorf("Unexpected type for BigInt: %v", input)
	err = fmt.Errorf("Unexpected type for Uint64: %v", input)

GraphQL API #17903

GraphQL API #17903

Conversation

Arachnid commented Oct 13, 2018 • edited Loading

holiman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holiman left a comment

Choose a reason for hiding this comment

raulk left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arachnid commented Oct 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arachnid commented Nov 22, 2018

gballet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Arachnid commented Jan 15, 2019

gballet commented Jan 17, 2019

Arachnid commented Feb 14, 2019

timbeiko commented Mar 19, 2019

Arachnid commented Mar 19, 2019

Arachnid commented Oct 13, 2018 •

edited

Loading

raulk left a comment •

edited

Loading