An Ethereum client is a software application that implements the Ethereum specification and communicates over the peer-to-peer network with other Ethereum clients. Different Ethereum clients interoperate if they comply with the reference specification and the standardized communications protocols. While these different clients are implemented by different teams and in different programming languages, they all "speak" the same protocol and follow the same rules.
Ethereum is an open source project and the source code is available under an open (LGPL v3.0) license, free to download and use for any purpose. Open source means more than simply free to use. It also means that Ethereum is developed by an open community of volunteers and can be modified by anyone.
Ethereum is defined by a formal specification called the "Yellow Paper". This is in contrast to, for example, Bitcoin, which is not defined in any formal way. Where Bitcoin’s "specification" is the reference implementation Bitcoin Core, Ethereum’s specification is documented in a paper combining an English and a mathematical (formal) specification. This formal specification, in addition to various Ethereum Improvement Proposals, defines the standard behavior of an Ethereum client. As major changes are made to Ethereum, the Yellow Paper is updated periodically to reflect these.
As a result of having a clear formal specification, there are a number of independently developed, yet interoperable, software implementations of an Ethereum client. Ethereum has a greater diversity of implementations running on the network than any other blockchain.
The vast majority of the Ethereum network runs one of the two dominant implementations: Geth and Parity. Geth is an implementation of Ethereum written in the Go programming language. Parity is a competing implementation written in Rust.
In this section, we will look at the two most common clients, Geth and Parity. We’ll learn to set up a node using each of them and explore some of their command-line and application programming interfaces (APIs).
The health, resilience and censorship resistance of Ethereum depends on having many independently operated and geographically dispersed full nodes. Each full node can help other new nodes obtain the block data to bootstrap their operation, as well as offer the operator an authoritative and independent verification of all transactions and contracts.
However, to run a full node you will incur a significant cost in hardware resources and bandwidth. A full node must download more than 50GB of data (as of Feb 2018) and store it on a local hard drive. This data burden increases quite rapidly every day as new transactions and blocks are added. More on this topic in Hardware Requirements for a Full Node.
A full node running on the Ethereum Main Network (mainnet) is not necessary for Ethereum development. You can do almost everything you need to do with a testnet node (which stores a copy of the smaller public test blockchain), or with a local private blockchain. You can also develop Ethereum applications with a local private blockchain (see [ganache]), or use a cloud-based Ethereum client offered by a service provider (see [infura]).
A lightweight client is an Ethereum client that does not store a local copy of the blockchain, or validate blocks and transactions. It offers the functions of a wallet and can create and broadcast transactions. A lightweight client can be used to connect to existing networks, such as your own full node, the public Ethereum blockchain, the public testnet, or a local test blockchain. In practice, you will likely use a lightweight client such as MetaMask as a convenient way to switch between all of the different node options. The terms "lightweight client" and "wallet" are used interchangeably, though there are some differences. Usually, a lightweight client offers an API (such as the web3js API), in addition to the function of a wallet.
Do not confuse the concept of a lightweight wallet in Ethereum with that of a Simplified Payment Verification (SPV) client in bitcoin. SPV clients validate block headers and use merkle proofs to validate the inclusion of transactions in the blockchain. Ethereum lightweight clients, generally, do not validate block headers or transactions. They rely on, and trust entirely, a full Ethereum client to give them RPC access to the Ethereum blockchain.
Each of the options for running a node has advantages and disadvantages:
Choosing to run a full node helps the Ethereum network, but also incurs costs for you. Let’s look at some of the advantages and disadvantages.
Advantages:
-
Supports the resilience and censorship resistance of the Ethereum network
-
Authoritatively validates all transactions
-
Can interact with any contract on the public blockchain (without requiring a middleman)
-
Can query (read-only) the blockchain status (accounts, contracts, etc.) offline, if necessary
-
Can directly deploy your own contracts into the public blockchain (without requiring a middleman)
Disadvantages:
-
Requires significant and growing hardware and bandwidth resources
-
Requires several hours or days to fully sync
-
Must be maintained, upgraded and kept online to remain synced
-
Deploying test contracts to the public blockchain will cost ETH for gas
Whether you choose to run a full node or not, you will probably want to also run a public testnet node. Let’s look at some of the advantages and disadvantages of using a public testnet. The current public testnet is called Ropsten. We will refer to a node that is synced with Ropsten as a Ropsten node.
Advantages:
-
A Ropsten node needs to sync and store much less data, about 10GB.
-
A Ropsten node can sync fully in a few hours.
-
To deploy contracts or make transactions you need test ether, which has no value and can be acquired for free from several sources.
-
Ropsten is a public blockchain with many other users and contracts, running "live".
Disadvantages:
-
You can’t use "real" money on Ropsten, it runs on test ether.
-
Consequently, you can’t test security against real adversaries, as there is nothing at stake.
-
There are some aspects of a public blockchain that you cannot test realistically on Ropsten. For example, transaction fees, although necessary to send transactions, are not a consideration on testnet since gas is free.
For many testing purposes, the best option is to launch a single instance private blockchain, using the testrpc node. TestRPC creates a local-only, private blockchain that you can interact with, without any other participants. It shares many of the advantages and disadvantages of the public testnet, but also has some differences.
Advantages:
-
No syncing and almost no data on disk. You mine the first block yourself.
-
No need to find test ether, you "award" yourself mining rewards that you can use for testing.
-
No other users, just you.
-
No other contracts, just the ones you deploy after you launch it.
Disadvantages:
-
Having no other users means that it doesn’t behave the same as a public blockchain. There’s no competition for transaction space or sequencing of transactions
-
No miners other than you means that mining is more predictable, therefore you can’t test some scenarios that occur on a public blockchain
-
Having no other contracts means you have to deploy everything that you want to test, including dependencies and contract libraries
-
You can’t recreate some of the public contracts and their addresses to test some scenarios (e.g. the DAO contract)
If you have the time and resources, you should attempt to run a full node, even if only to learn more about the process. In the next few sections we will download, compile and run the Ethereum clients Go-Ethereum (geth) and Parity. This requires some familiarity with using the command-line interface on your operating system. It’s worth installing these clients, whether you choose to run these Ethereum clients as full nodes, as testnet nodes or as client to a local private blockchain.
Before we get started, you should ensure you have a computer with sufficient resources to run an Ethereum full node. You will need at least 35 to 40GB of disk space to run a full node with a full copy of the Ethereum blockchain. If you want to also run a full node on the Ethereum testnet, you will need a minimum of 15GB of disk space for that. Downloading 50GB of blockchain will also require a fast Internet connection, as otherwise it can take a long time.
Syncing the Ethereum blockchain is very input-output (I/O) intensive. It is best to have a Solid-State Drive (SSD). If you have a mechanical hard disk drive (HDD), you will need at least 8GB of RAM to use as cache. Otherwise, you may discover your system is too slow to keep up and sync fully.
Minimum Requirements:
-
CPU with 2 or 4 cores preferred
-
Solid State Drive (SSD) with at least 50GB free space
-
4GB RAM minimum, 8GB+ if you have an HDD and not SSD
-
8 MBit/sec download Internet service
These are the minimum requirements to sync the Ethereum blockchain, without storing a complete copy (pruned blockchain). If you want to sync in a reasonable amount of time and store all the development tools, libraries, clients and blockchains we discuss in this book, you will need a faster and better-equipped computer:
Recommended Specifications:
-
Fast CPU with 4+ cores
-
16GB+ RAM
-
Fast SSD with at least 500GB free space
-
25+ MBit/sec download Internet service
It’s difficult to predict how fast the blockchain size will increase and when more disk space will be required, so it’s recommended to check its latest size at https://bitinfocharts.com/ethereum/.
This section assumes you are using a Unix-like command-line environment. The examples show the output and commands as entered on an Ubuntu Linux operating system running the Bash shell (command-line execution environment).
|
Tip
|
In many of the examples in this chapter, we will be using the operating system’s command-line interface (also known as a "shell"), accessed via a "terminal" application. The shell will display a prompt; you type a command, and the shell responds with some text and a new prompt for your next command. The prompt may look different on your system, but in the following examples, it is denoted by a $ symbol. In the examples, when you see text after a $ symbol, don’t type the $ symbol but type the command immediately following it, then press Enter to execute the command. In the examples, the lines below each command are the operating system’s responses to that command. When you see the next $ prefix, you’ll know it’s a new command and you should repeat the process. |
Before we get started, we may need to get some prerequisites satisfied. If you’ve never done any software development on the computer you are currently using, you will probably need to install some basic tools. For the examples that follow, you will need to install git, the source-code management system; golang, the Go programming language and standard libraries; and Rust, a systems programming language.
Git can be installed by following the instructions here: https://git-scm.com/
Go can be installed by following the instructions here: https://golang.org/
|
Note
|
Geth requires Go version 1.7 or greater. The golang that is installed on your operating system or is available from your system’s package manager may be an older version. If so, remove it and install the latest version from golang.org. |
Rust can be installed by following the instructions here: https://www.rustup.rs/
|
Note
|
Parity requires Rust version 1.24 or greater. |
Parity also requires some software libraries, such as OpenSSL and libudev. To install these on a Linux (Debian) compatible system:
$ sudo apt-get install openssl libssl-dev libudev-dev
For other operating systems, use the package manager of your OS or follow the Wiki instructions (https://github.com/paritytech/parity/wiki/Setup) to install the required libraries.
Now you have git, golang, rust, and necessary libraries installed, let’s get to work!
There are three original implementations of the Ethereum client written in three different languages: C++, Python, and Go. Geth is the Go language implementation, which is actively developed and considered the "official" implementation of the Ethereum client. There is some debate on the meaning of the title "official client" in a decentralized system such as Ethereum, but suffice it to say that Geth is supported by the Ethereum Foundation, a Swiss non-profit organization founded by Ethereum’s creator, Vitalik Buterin.
Geth’s home is https://geth.ethereum.org/. On this site, you will find instructions to download and install Geth for your operating system. Since this book is aimed at developers, we will be building Geth from the source code.
You can also skip these instructions and install a precompiled binary for your platform of choice. But where’s the fun and learning in that?
Our first step is to clone the git repository, so as to get a copy of the source code.
The Geth source code repository is hosted on GitHub at:
To make a local clone of this repository, use the git command as follows, in your home directory or under any directory you use for development:
$ git clone https://github.com/ethereum/go-ethereum.git
You should see a progress report as the repository is copied to your local system:
Cloning into 'go-ethereum'... remote: Counting objects: 62587, done. remote: Compressing objects: 100% (26/26), done. remote: Total 62587 (delta 10), reused 13 (delta 4), pack-reused 62557 Receiving objects: 100% (62587/62587), 84.51 MiB | 1.40 MiB/s, done. Resolving deltas: 100% (41554/41554), done. Checking connectivity... done.
Great! Now we have a local copy of Geth, we can compile an executable for our platform.
To build Geth, change to the directory where the source code was downloaded and use the make command:
$ cd go-ethereum $ make geth
If all goes well, you will see the Go compiler building each component until it produces the geth executable:
build/env.sh go run build/ci.go install ./cmd/geth >>> /usr/local/go/bin/go install -ldflags -X main.gitCommit=58a1e13e6dd7f52a1d5e67bee47d23fd6cfdee5c -v ./cmd/geth github.com/ethereum/go-ethereum/common/hexutil github.com/ethereum/go-ethereum/common/math github.com/ethereum/go-ethereum/crypto/sha3 github.com/ethereum/go-ethereum/rlp github.com/ethereum/go-ethereum/crypto/secp256k1 github.com/ethereum/go-ethereum/common [...] github.com/ethereum/go-ethereum/cmd/utils github.com/ethereum/go-ethereum/cmd/geth Done building. Run "build/bin/geth" to launch geth. $
Let’s run geth to make sure it works:
$ ./build/bin/geth version Geth Version: 1.6.6-unstable Git Commit: 58a1e13e6dd7f52a1d5e67bee47d23fd6cfdee5c Architecture: amd64 Protocol Versions: [63 62] Network Id: 1 Go Version: go1.8.3 Operating System: linux GOPATH=/usr/local/src/gocode/ GOROOT=/usr/local/go
Your geth version command may show slightly different information, but you should see a version report much like the one above.
As the last step, we may want to copy the geth command to our operating systems application directory (or a directory on the command-line execution path). On Linux, we’d use the following command:
$ sudo cp ./build/bin/geth /usr/local/bin
Don’t start running geth yet, because it will start synchronizing the blockchain "the slow way" and that will take far too long (weeks). The First Synchronization of the Ethereum Blockchain explains the challenge with the initial synchronization of Ethereum’s blockchain.
Parity is an implementation of a full node Ethereum client and DApp browser. Parity was written from the "ground up" in Rust, a systems programming language with the aim of building a highly modular, very secure and scalable Ethereum client. Parity is developed by Parity Tech, a UK company and is released under a GPLv3 open source license.
|
Note
|
Disclosure: One of the authors of this book, Gavin Wood, is the founder of Parity Tech and wrote most of the Parity client. Parity represents about 20% of the installed Ethereum client base. |
To install Parity, you can use the Rust package manager cargo or download the source code from GitHub. The package manager also downloads the source code, so there’s not much difference between the two options. In the next section, we will show you how to download and compile it yourself.
The Parity Wiki offers instructions for building Parity in different environments and containers:
We’ll build Parity from source. This assumes you have already installed Rust using rustup (See Software Requirements for Building and Running a Node).
First, let’s get the source code from GitHub:
$ git clone https://github.com/paritytech/parity
Now, let’s change to the parity directory and use cargo to build the executable:
$ cd parity $ cargo build
If all goes well, you should see something like:
$ cargo build
Updating git repository `https://github.com/paritytech/js-precompiled.git`
Downloading log v0.3.7
Downloading isatty v0.1.1
Downloading regex v0.2.1
[...]
Compiling parity-ipfs-api v1.7.0
Compiling parity-rpc v1.7.0
Compiling parity-rpc-client v1.4.0
Compiling rpc-cli v1.4.0 (file:///home/aantonop/Dev/parity/rpc_cli)
Finished dev [unoptimized + debuginfo] target(s) in 479.12 secs
$
Let’s try and run parity to see if it is installed, by invoking the --version option:
$ parity --version Parity version Parity/v1.7.0-unstable-02edc95-20170623/x86_64-linux-gnu/rustc1.18.0 Copyright 2015, 2016, 2017 Parity Technologies (UK) Ltd License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. By Wood/Paronyan/Kotewicz/Drwięga/Volf Habermeier/Czaban/Greeff/Gotchac/Redmann $
Great! Now that Parity is installed, we can sync the blockchain and get started with some basic command-line options.
Ethereum clients offer an Application Programming Interface (API), offering a set of Remote Procedure Call (RPC) commands, which are encoded as JavaScript Object Notation (JSON). You will see this referred to as the JSON-RPC API. Essentially, the JSON-RPC API is an interface that allows us to write programs that use the Ethereum client as a gateway into the Ethereum network and blockchain.
Usually, the RPC interface is offered over as an HTTP service on port 8545. For security reasons it is restricted, by default, to only accept connections from localhost (the IP address of your own computer which is 127.0.0.1).
To access the JSON-RPC API, you can use a specialized library, written in the programming language of your choice, which provides "stub" function calls corresponding to each available RPC command. Or, you can manually construct HTTP requests and send/receive JSON encoded requests. You can even use a generic command-line HTTP client, like curl, to call the RCP interface. Let’s try that:
$ curl -X POST -H "Content-Type: application/json" --data \
'{"jsonrpc":"2.0","method":"web3_clientVersion","params":[],"id":1}' \
http://localhost:8545
{"jsonrpc":"2.0","id":1,
"result":"Geth/v1.8.0-unstable-02aeb3d7/linux-amd64/go1.8.3"}
In this example, we use curl to make an HTTP connection to address http://localhost:8545. We are already running geth and it is offering the JSON-RPC API as an HTTP service on port 8545. We instruct curl to use the HTTP POST command, and to identify the content as Content-Type: application/json. Finally, we pass a JSON-encoded request as the data component of our HTTP request. Most of our command line is just setting up curl to make the HTTP connection correctly. The interesting part is the actual JSON-RPC command we issue:
{"jsonrpc":"2.0","method":"web3_clientVersion","params":[],"id":4192}
The JSON-RPC request is formatted according to the JSON-RPC 2.0 specification, which you can see here: http://www.jsonrpc.org/specification
Each request contains 4 elements:
- jsonrpc
-
Version of the JSON-RPC protocol. Here MUST be exactly "2.0".
- method
-
The name of the method to be invoked.
- params
-
A structured value that holds the parameter values to be used during the invocation of the method. This member MAY be omitted.
- id
-
An identifier established by the Client that MUST contain a String, Number, or NULL value if included. The Server MUST reply with the same value in the Response object if included. This member is used to correlate the context between the two objects.
|
Tip
|
The id parameter is used primarily when you are making multiple requests in a single JSON-RPC call, a practice called batching. Batching is used to avoid the overhead of a new HTTP and TCP connection for every request. In the Ethereum context for example, we would use batching if we wanted to retrieve thousands of transactions in one HTTP connection. When batching, you set a different id for each request and then match it to the id in each response from the JSON-RPC server. The easiest way to implement this is to maintain a counter and increment the value for each request. |
The response we receive is:
{"jsonrpc":"2.0","id":4192,
"result":"Geth/v1.8.0-unstable-02aeb3d7/linux-amd64/go1.8.3"}
This tells us that the JSON-RPC API is being served by Geth client version 1.8.0.
Let’s try something a bit more interesting. In the next example, we ask the JSON-RPC API for the current price of gas in wei:
$ curl -X POST -H "Content-Type: application/json" --data \
'{"jsonrpc":"2.0","method":"eth_gasPrice","params":[],"id":4213}' \
http://localhost:8545
{"jsonrpc":"2.0","id":4213,"result":"0x430e23400"}
The response, 0x430e23400, tells us that the current gas price is 1.8 Gwei (gigawei or billion wei).
Normally, when syncing the Ethereum blockchain, your Ethereum client will download and validate every block and every transaction since the genesis block.
While it is possible to fully sync the blockchain this way, it is not practical, as it will take a very long time and has higher computing resource requirements (much more RAM and faster storage).
Doing a "normal" sync, your Ethereum client will make rapid progress until it reaches block 2,283,397. This block was mined on September 18th, 2016 and marks the beginning of a series of denial of service attacks against Ethereum’s blockchain. From this block and until block 2,700,031 (November 26th, 2016), the validation of transactions becomes extremely slow, memory intensive, and I/O intensive resulting in block validation times exceeding 1 minute. The Ethereum system implemented a series of upgrades using hard forks, to address the underlying vulnerabilities that were exploited in the denial of service and clean up the blockchain by removing some 20 million empty accounts created by spam transactions.
If you are syncing with full validation, your client will slow down and may take several weeks or longer to validate the blocks in this range.
Ethereum clients include an option to perform a "fast" synchronization that skips the full validation of transactions until it has synced to the tip of the blockchain, then resumes full validation. For Geth, the option to enable fast synchronization is --fast. For Parity, the option is --warp for older versions (< 1.6) and is enabled by default (no need to set a configuration option) on newer versions (>= 1.6).
|
Note
|
Geth and Parity can only operate fast synchronization when starting with an empty block database. If you have already started syncing without "fast" mode, Geth and Parity cannot switch. It is faster to delete the blockchain data directory and start "fast" syncing from the beginning than to continue syncing with full validation. Be careful to not delete any wallets when deleting the blockchain data! |
Lightweight Ethereum clients are clients that offer a subset of the functionality of a full client. They do not store the full Ethereum blockchain, so they are faster to setup and require far less data storage.
A lightweight client offers one or more of the following functions:
-
Manage private keys and Ethereum addresses in a wallet.
-
Create, sign and broadcast transactions.
-
Interact with smart contracts, using the data payload.
-
Browse and interact with DApps.
-
Offer links to external services such as block explorers.
-
Convert ether units and retrieve exchange rates from external sources.
-
Inject a web3 instance into the web browser as a JavaScript object.
-
Use a web3 instance provided/injected into the browser by another client.
-
Access RPC services on a local or remote Ethereum node
Some lightweight clients, for example mobile (smartphone) wallets, offer only basic wallet functionality. Other lightweight clients are fully-developed DApp browsers. The one common characteristic is that they offer some of the functions of a full node Ethereum client, without synchronizing a local copy of the Ethereum blockchain.
Let’s look at some of the most popular lightweight clients and the functions they offer.
All mobile wallets are lightweight clients, because smartphones do not have adequate resources to run a full Ethereum client.
Popular mobile wallets include Jaxx, Status, and Trust Wallet. We list these as examples of popular mobile wallets (this is not an endorsement or an indication of the security or functionality of these wallets).
- Jaxx
-
A multi-currency mobile wallet based on BIP39 mnemonic seeds, with support for Bitcoin, Litecoin, Ethereum, Ethereum Classic, ZCash, a variety of ERC20 tokens and many other currencies. Jaxx is available on Android, iOS, as a browser plugin wallet, and desktop wallet for a variety of operating systems. Find it at https://jaxx.io
- Status
-
A mobile wallet and DApp browser, with support for a variety of tokens and popular DApps. Available for iOS and Android smartphones. Find it at https://status.im
- Trust Wallet
-
A mobile Ethereum, Ethereum Classic wallet that supports ERC20, and ERC223 tokens. Trust Wallet is available for iOS and Android smartphones. Find it at https://trustwalletapp.com/
- Cipher Browser
-
a full-featured Ethereum-enabled mobile DApp browser and wallet. Allows integration with ethereum apps and tokens.
A variety of wallets and DApp browsers are available as plugins or extensions of web browsers such as Chrome and Firefox: lightweight clients that run inside your browser.
Some of the more popular ones include MetaMask, Jaxx and MyEtherWallet/MyCrypto
MetaMask was introduced in [intro], and is a versatile browser-based wallet, RPC client, and basic contract explorer. It is available on Chrome, Firefox, Opera and Brave Browser. Find MetaMask at:
At first glance, MetaMask is a browser-based wallet. But, unlike other browser wallets, MetaMask injects a web3 instance into the browser, acting as an RPC client that connects to a variety of Ethereum blockchains (eg. mainnet, Ropsten testnet, Kovan testnet, local RPC node, etc.). The ability to inject a web3 instance and act as a gateway to external RPC services, makes MetaMask a very powerful tool for developers and users alike. It can be combined, for example, with MyEtherWallet or MyCrypto, acting as an web3 provider and RPC gateway for those tools.
Jaxx, which was introduced as a mobile wallet in Mobile (Smartphone) Wallets, is also available as a Chrome and Firefox extension. Find it at:
MyEtherWallet is a browser-based JavaScript lightweight client that offers:
-
A software wallet running in JavaScript
-
A bridge to popular hardware wallets such as the Trezor and Ledger
-
A web3 interface that can connect to a web3 instance injected by another client (eg. MetaMask)
-
An RPC client that can connect to an Ethereum full client
-
A basic interface that can interact with smart contracts, given a contract’s address and Application Binary Interface (ABI).
MyEtherWallet is very useful for testing and as an interface to hardware wallets. It should not be used as a primary software wallet, as it is exposed to threats via the browser environment and is not a secure key storage system.
You must be very careful when accessing MyEtherWallet and other browser-based JavaScript wallets, as they are frequent targets for phishing. Always use a bookmark and not a search engine or link to access the correct web URL. MyEtherWallet can be found at:
Just prior to publication of the first edition of this book, the MyEtherWallet project split into two competing implementations, guided by two independent development teams: a "fork" as it is called in open source development. The two projects are called MyEtherWallet (the original branding) and MyCrypto. At the time of the split, MyCrypto offered identical functionality as MyEtherWallet. It is likely that the two projects will diverge as the two development teams adopt different goals and priorities.
As with MyEtherWallet, you must be very careful when accessing MyCrypto in your browser. Always use a bookmark, or type the URL very carefully (then bookmark it for future use).
MyCrypto can be found at:
Mist is the first ever Ethereum enabled browser, built by the Ethereum Foundation. It also contains a browser-based wallet that was the first ever implementation of the ERC20 token standard (Fabian Vogelsteller, author of ERC20 was also the main developer in Mist). Mist was also the first wallet to introduce the camelCase checksum (EIP-155, see [eip-155]). Mist runs a full node, and offers a full DApp browser with support for Swarm based storage and ENS addresses. Find it at: