Docs cleanup and truffle usage

packages/docs/src/core/index.rst

@@ -1,6 +1,6 @@
-========================
+===========================
 @eth-optimism Documentation
-========================
+===========================
 Hello and welcome to the documentation for Optimism's OVM and Optimistic rollup.
 
 Our ``@eth-optimism`` module is the set of core modules that the OVM and our L2 implementations use.
@@ -30,7 +30,7 @@ You can find all of our source code here on `our monorepo`_.
    src/spec/execution-manager
    src/spec/transpilation-details
    src/spec/jump-transpilation
-   src/spec/solc-transpiler
+   src/spec/codecopy
    src/spec/design
 
 _`our monorepo`: https://github.com/ethereum-optimism/optimism-monorepo

packages/docs/src/core/src/integrating-tests.rst

@@ -5,6 +5,8 @@ If you're interested in testing your contracts running on the OVM, then you're i
 
 There are two steps you need to take to get your contracts running on the ovm: using ``@eth-optimism/solc-transpiler`` in place of ``solc`` so your contracts can be transpiled into OVM-compatible versions, and using ``@eth-optimism/rollup-full-node`` as your web3 provider.
 
+For reference, example integrations for both Truffle and Waffle can be found `in our monorepo`_ .
+
 Integrating the OVM Transpiler
 ==============================
 
@@ -26,7 +28,8 @@ or
 
 ``@eth-optimism/solc-transpiler`` Accepts the same compiler options as as ``solc``, with one additional option, ``executionManagerAddress``.  The Execution Manager is a smart contract implementing the OVM's containerization functionality.  If you are using an unmodified ``rollup-full-node``, the default execution manager address is ``0xA193E42526F1FEA8C99AF609dcEabf30C1c29fAA``.  More info on the execution manager can be found here. (link)
 
-*With Waffle:*
+Using With Waffle
+-----------------
 
 To use the transpiler with ``ethereum-waffle``, set the ``solc.version`` configuration to ``""@eth-optimism/solc-transpiler"`` and ``compilerOptions.executionManagerAddress`` to ``"0xA193E42526F1FEA8C99AF609dcEabf30C1c29fAA"``.
 
@@ -41,36 +44,77 @@ example waffle-config.json:
     }
   }
 
-*With Truffle:*
+Using With Truffle
+------------------
+
+To use the transpiler with Truffle, set truffle's ``compilers.solc.version`` configuration to ``@eth-optimism/solc-transpiler``, and configure the ``EXECUTION_MANAGER_ADDRESS`` environment variable. 
 
-To use the transpiler with Truffle, set truffle's ``compilers.solc.version`` configuration to ``@eth-optimism/solc-transpiler``.
+Currently, Truffle does not provide a clean way to use custom chain IDs, so we also need a custom function in Truffle's configuration file.
 
 example truffle-config.json:
 
-.. code-block:: none
+.. code-block:: json
+
+  const HDWalletProvider = require('truffle-hdwallet-provider');
+
+  const mnemonic = 'candy maple cake sugar pudding cream honey rich smooth crumble sweet treat';
+
+  // Set this to the desired Execution Manager Address -- required for the transpiler
+  process.env.EXECUTION_MANAGER_ADDRESS = process.env.EXECUTION_MANAGER_ADDRESS || "0xA193E42526F1FEA8C99AF609dcEabf30C1c29fAA"
 
-  // Note, will need EXECUTION_MANAGER_ADDRESS environment variable set.
-  // Default is "0xA193E42526F1FEA8C99AF609dcEabf30C1c29fAA"
   module.exports = {
-    compilers: {
-      solc: {
+  /**
+   * Note: this expects the local fullnode to be running:
+   * // TODO: Run `yarn server:fullnode` in rollup-full-node before executing this test
+   *
+   * To run tests:
+   * $ truffle test ./truffle-tests/test-erc20.js --config truffle-config-ovm.js
+   */
+  networks: {
+    // Note: Requires running the rollup-full-node locally.
+    test: {
+      network_id: 108,
+      provider: function() {
+        const wallet = new HDWalletProvider(mnemonic, "http://127.0.0.1:8545/", 0, 10);
+        const sendAsync = wallet.sendAsync
+
+        wallet.sendAsync = function (...args) {
+          if (args[0].method === 'eth_sendTransaction') {
+            // HACK TO PROPERLY SET CHAIN ID
+            args[0].params[0].chainId = 108
+          }
+          sendAsync.apply(this, args)
+        };
+        return wallet;
+      },
+      gasPrice: 0,
+      gas: 9000000,
+    },
+  },
+
+  // Set default mocha options here, use special reporters etc.
+  mocha: {
+    timeout: 100000
+  },
+
+  compilers: {
+    solc: {
       // Add path to the solc-transpiler
       version: "@eth-optimism/solc-transpiler",
-      }
     }
   }
+}
 
-Truffle does not expose compiler options, so the execution manager address must be passed in with an environment variable, called ``EXECUTION_MANAGER_ADDRESS``.  One easy way to do this is run tests with 
+As you can see in the above comments, you must spin up the rollup full node before running truffle tests.  To do this, with ``@eth-optimism/rollup-full-node`` installed, you can run:
 
-.. code-block:: none
-
-    $   env EXECUTION_MANAGER_ADDRESS="0xA193E42526F1FEA8C99AF609dcEabf30C1c29fAA" [test command]
+.. code-block:: bash
 
+  node rollup-full-node/build/src/exec/fullnode.js
 
 Currently, ``rollup-full-node`` breaks Truffle's ``gasLimit`` and ``blockGasLimit``.  To avoid this, you can set both to ``undefined`` where they are used.
 
 Integrating the OVM Full Node
-------------------------------
+==============================
 
 To use your transpiled contracts, you need to use ``@eth-optimism/rollup-full-node`` as your web3 provider.  To do this, make sure it's installed:
 
@@ -84,11 +128,12 @@ or
 
     npm install --save @eth-optimism/rollup-full-node
 
-
 To get your provider and some wallets:
 
-.. code-block:: none
+.. code-block:: javascript
 
     const RollupFullNode = require("@eth-optimism/rollup-full-node")
-    const provider = RollupFullNode.getMockOVMProvider()
-    const wallets = RollupFullNode.getWallets(provider)
+    const provider = RollupFullNode.getMockProvider()
+    const wallets = RollupFullNode.getWallets(provider)
+
+.. _`in our monorepo`: https://github.com/ethereum-optimism/optimism-monorepo/tree/master/packages/examples

packages/docs/src/core/src/spec/codecopy.rst

@@ -0,0 +1,69 @@
+===================================
+``CODECOPY`` in Transpiled Bytecode
+===================================
+
+The opcode ``CODECOPY`` accepts ``memOffset``, ``codeOffset``, and ``length`` inputs from the stack, modifying the memory so that ``memory[memOffset:memOffset + length] = code[codeOffset:codeOffset + length]``.  Since we are by definition modifying the ``code`` of a contract by transpiling, there is no general way to handle pre-transpiled ``CODECOPYs`` since the impact on execution is dependent on how the ``CODECOPY`` was expected to be used.  For Solidity, there are three ways in which ``CODECOPY`` is used:
+
+  1. Constants which exceed 32 bytes in length are stored in the bytecode and ``CODECOPY`` ed for use in execution. (if <=32 bytes they are just ``PUSHN`` ed)
+  2. All constructor logic for initcode is prefixed to the bytecode to be deployed, and this prefix runs ``CODECOPY(suffixToDeploy), ..., RETURN`` during ``CREATE(2)`` .
+  3. Constructor parameters are passed as bytecode and then are ``CODECOPY`` ed to memory before being treated like calldata.
+
+This document explains each of these cases and how they're handled transpilation-side.
+
+Constants
+=========
+
+Constants larger than 32 bytes are stored in the bytecode and ``CODECOPY`` ed to access in execution.  Initial investigation shows that the pattern which always occurs leading up to a ``CODECOPY`` for constants is: 
+
+.. code-block:: none
+
+  ...
+  PUSH2 // offset
+  PUSH1 // length
+  SWAP2 // where to shove it into memory
+  CODECOPY
+  ...
+
+With some memory allocation operations preceding the ``PUSH, PUSH, SWAP...`` which may also be standard (haven't tested).  Some sample code which this example was pulled from can be found `this gist`_ .
+
+To deal with constants, we still want to copy the correct constant--this will just be at a different index once we insert transpiled bytecode above it.  So, we just increase the ``codeOffset`` input to ``CODECOPY`` in every case that a constant is being loaded into memory.  Hopefully, all constants are appended to the end of a file so that we may simply add a fixed offset for every constant.
+
+Returning deployed bytecode in ``CREATE(2)``
+============================================
+
+All constructor logic for initcode is prefixed to the bytecode to be deployed, and this prefix runs ``CODECOPY(suffixToDeploy), ..., RETURN`` during ``CREATE(2)``.  If constructor logic is empty (i.e. no ``constructor()`` function specified in Solidity) this prefix is quite simple but still exists.  This ``CODECOPY`` simply puts the prefix into memory so that the deployed byetcode can be deployed.  So, what we need to do is increase the ``length`` input both to ``CODECOPY`` and the ``RETURN``.  The ``CODECOPY, RETURN`` pattern seems to appear in the following format:
+
+.. code-block:: none
+
+  PUSH2 // codecopy's and RETURN's length
+  DUP1 // DUPed to use twice, for RETURN and CODECOPY both
+  PUSH2 // codecopy's offset
+  PUSH1 codecopy's destOffset
+  CODECOPY // copy
+  PUSH1 0 // RETURN offset
+  RETURN // uses above RETURN offset and DUP'ed length above
+
+So by adding to the consumed bytes of the first ``PUSH2`` above, in accordance to the extra bytes added by transpilation, we make sure the correct length is both ``CODECOPY``ed and ``RETURN`` ed.  Note that, if we have constructor logic which gets transpiled, this will require modifying the ``// codecopy's offset`` line above as well.
+
+Constructor Parameters
+======================
+
+Constructor parameters are passed as bytecode and then are ``CODECOPY`` ed to memory before being treated like calldata.  This is because the EVM execution which is initiated by ``CREATE(2)`` does not have a calldata parameter, so the inputs must be passed in a different way.  For more discussion, check out `this discussion`_ on stack exchange.
+
+We handle this similarly to how we handle constants, by changing the ``codeOffset`` input appropriately. Both constants used in the constructor and constructor inputs are at the end of the file.
+
+The pattern it uses is:
+
+.. code-block:: none
+
+  ...
+  [PC 0000000015] PUSH2: 0x01cf // should be initcode.length + deployedbytecode.length
+  [PC 0000000018] CODESIZE
+  [PC 0000000019] SUB // subtract however big the code is from the amount pushed above to get the length of constructor input
+  [PC 000000001a] DUP1
+  [PC 000000001b] PUSH2: 0x01cf // should also be initcode.length + deployedbytecode.length
+  [PC 000000001e] DUP4
+  [PC 000000001f] CODECOPY
+
+.. _`this gist`: https://gist.github.com/ben-chain/677457843793d7c6c7feced4e3b9311a
+.. _`this discussion`: https://ethereum.stackexchange.com/questions/58866/how-does-a-contracts-constructor-work-and-load-input-values

packages/docs/src/core/src/spec/transpilation-details.rst

@@ -12,14 +12,15 @@ Pure Opcodes
 ============
 
 The following opcodes perform stack operations which are constant in terms of L1/L2 state, and do not require modification:
+
 - Arithmetic/pure-math opcodes: 
-   - ``ADD, MUL, SUB, DIV, SDIV, MOD, SMOD, ADDMOD, MULMOD, EXP, SIGNEXTEND, LT, GT, SLT, SGT, EQ, ISZERO, AND, OR, XOR, NOT, BYT, SHL, SHR, SAAR, SHA3``.
-- "Pure" code execution operations: 
-   - ``PUSH1....PUSH32, DUP1...DUP16, SWAP1...SWAP16, POP, LOG0...LOG4, STOP, REVERT, RETURN, PC, GAS, JUMPDEST*``.  \* NOTE: `See section <https://github.com/op-optimism/optimistic-rollup/wiki/JUMP-Transpilation>`_ which involves ``JUMPDEST``s.
+    - ``ADD, MUL, SUB, DIV, SDIV, MOD, SMOD, ADDMOD, MULMOD, EXP, SIGNEXTEND, LT, GT, SLT, SGT, EQ, ISZERO, AND, OR, XOR, NOT, BYT, SHL, SHR, SAAR, SHA3``.
+- \"Pure\" code execution operations: 
+   - ``PUSH1....PUSH32, DUP1...DUP16, SWAP1...SWAP16, POP, LOG0...LOG4, STOP, REVERT, RETURN, PC, GAS, JUMPDEST*``.  \* NOTE: `See section <https://github.com/op-optimism/optimistic-rollup/wiki/JUMP-Transpilation>`_ which involves ``JUMPDEST`` s.
 - "Pure" memory modifying operations: 
    - ``MLOAD, MSTORE, MSTORE8, MSIZE``.
 - Permitted execution-context-dependent operations: 
-   - ``CALLVALUE*, CALLDATALOAD, CALLDATASIZE, CALLDATACOPY, CODESIZE, RETURNDATASIZE, RETURNDATACOPY``   \*Note: ``CALLVALUE`` will always be 0 because we enforce that all ``CALL``s always pass 0 in our purity checking.
+   - ``CALLVALUE*, CALLDATALOAD, CALLDATASIZE, CALLDATACOPY, CODESIZE, RETURNDATASIZE, RETURNDATACOPY``   \*Note: ``CALLVALUE`` will always be 0 because we enforce that all ``CALL`` s always pass 0 in our purity checking.
 
 Replaced Opcodes
 ================
@@ -97,12 +98,13 @@ To replace Call-type opcodes, we have to pass an existing slice of ``calldata``
     * - ``DELEGATECALL``
       - 2
 
-Special cases: ``CODECOPY`` and ``JUMP``s
------------------------
+Special cases: ``CODECOPY`` and ``JUMP`` s
+------------------------------------------
 
 There are two functions which are "Pure code execution operations" just like ``CODESIZE``, ``REVERT``, etc., however, they are used by the Solidity compiler in ways which the transpilation process affects, and need to be dealt with in the transpiler.
-- Because we are inserting bytecode, we are changing the index of every ``JUMPDEST`` proceeding each insertion operation.  This means our ``JUMP`` and ``JUMPI`` values need to be transpiled or they will fail/go to the wrong place.  We handle this by making all ``JUMP``s go to new bytecode that we append at the end that simply contains a mapping from untranspiled ``JUMPDEST`` bytecode location to transpiled ``JUMPDEST`` bytecode location.  The logic finds the new location and ``JUMP``s to it.  See the `"JUMP Modification" page <https://github.com/op-optimism/optimistic-rollup/wiki/JUMP-Transpilation>`_ for more details.
-- The opcode ``CODECOPY`` would work fine, in principle, in our code contracts, and its effect on execution is independent of L1 state.  However, because ``CODECOPY`` is used to retrieve Solidity constants, we'll need to deal with it in the transpiler.  We have not yet implemented this.  If this is the only way in which ``CODECOPY`` is used by solidity, then this will be easy.  If not... we'll cross that bridge then.
+
+  - Because we are inserting bytecode, we are changing the index of every ``JUMPDEST`` proceeding each insertion operation.  This means our ``JUMP`` and ``JUMPI`` values need to be transpiled or they will fail/go to the wrong place.  We handle this by making all ``JUMP`` s go to new bytecode that we append at the end that simply contains a mapping from untranspiled ``JUMPDEST`` bytecode location to transpiled ``JUMPDEST`` bytecode location.  The logic finds the new location and ``JUMP`` s to it.  See the `"JUMP Modification" page <https://github.com/op-optimism/optimistic-rollup/wiki/JUMP-Transpilation>`_ for more details.
+  - The opcode ``CODECOPY`` works fine, in principle, in our code contracts, as its effect on execution is independent of L1 state.  However, because that code itself is modified by transpilation, we need to deal with it in the transpiler.  See our ``CODECOPY`` `section`_ for how we handle these modifications.
 
 Banned Opcodes
 ==============
@@ -138,3 +140,4 @@ Others
 - ``COINBASE`` -- since we don't have inflation in L2
 - ``DIFFICULTY`` -- since there is no sense of difficulty in L2.  An analogous value in L2 is actually the MEVA price, but it's not so analogous that transpiling would make any sense.
 
+.. _`section`: ./codecopy.html