Skip to content

Extension of consensus with DSL language with access to transaction properties and chain objects

Jump to bottom
dimxy edited this page Jun 15, 2021 · 6 revisions

Document History

Version 0.1: Initial content added
Version 0.2: development status added, as well as samples description
Version 0.3: update for chips added

Introduction

In this document a concept of a virtual machine (VM) running inside validation code is described. The VM is based on an interpreter of a simple expression language defined on blockchain artefacts and allowing to define transaction validation rules. Rule expressions will be stored in contract definition transactions and the VM will be executed on those expressions inside blockchain validation code, when new transactions are added to the chain.
This VM mechanism will allow to implement new validation logic with no changing blockchain core code and add new cc modules to the chain with no hardforks. In fact, the proposed rule expression language is a Domain Specific Language (DSL). Why it is good to have a DSL for blockchain validation?

  • First, we could store it inside transactions and not to build into the code, so we won't need to recompile the chain source code each time we need new validation rules. Instead we just send a new transaction with new rules.
  • Second, because of it is a DSL and not a general purpose language, we could create it natively referring any blockchain artefacts, thus making validation code simple and clean, easy to build and use, and free from errors.
    I drafted a list of blockchain objects and their properties supported in this rule expression language (see below). This list is actually the result of more than two years coding of custom consensus apps in C++ in komodo assets blockchains.

Update on the current status on Komodo codebase

The code for this rule expression technology is located here: https://github.com/dimxy/komodo/tree/test-rule-expr.
To test it, two sample rpc sets are developed:

  • ccvmsample1 - allows to create a definition tx with a rule expression and two instance txns (to create funds and spend it with the rule evaluated)
  • ccvmsample2 - allows to create and a spend txns, a create tx will have a cc vout with a rule expression in the eval condition. The rule will be evaluated when a spend tx is added to the chain.

Limitations:
ccvmsample1 definetx does not support several expressions for each funcid. Instead only single expression is stored and it will be applied to any spending transaction

Sample1 (contract-like) rpc usage example:
Allow to spend if chain height > 100.
First create a contract 'definition tx' with a rule expression, then make 'create' and 'spend tx' referring the definition tx. The rule works when the 'spend tx' (with a cc vin) is added to the chain

komodo-cli -ac_name=TESTCHAIN ccvmsample1define "AND { chainActive.height() > 100 }"
sendrawtransaction ...
komodo-cli -ac_name=TESTCHAIN ccvmsample1create definetxid pubkey 1 010203
sendrawtransaction ...
komodo-cli -ac_name=TESTCHAIN ccvmsample1spend definetxid pubkey 1 0102030405
sendrawtransaction ...

Sample2 (expression in the spk) rpc example:
Allow to spend if the spending tx has at least one cc vout.
First, create tx with cc spk with the rule expression. Then, create tx spending the first tx. It should have exactly the same expression:

komodo-cli -ac_name=TESTCHAIN ccvmsample2fund 1 "AND{ { hasCC = False; FOR(vout : evaltx.vout) { hasCC = hasCC || vout.scriptPubKey.isCC }; hasCC!=False } }
sendrawtransaction ...
komodo-cli -ac_name=TESTCHAIN ccvmsample2spend txid 0 "AND{ { hasCC = False; FOR(vout : evaltx.vout) { hasCC = hasCC || vout.scriptPubKey.isCC }; hasCC!=False } }
sendrawtransaction ...

Update: added PoC support for CHIPS project

In the CHIPS, instead of adding of rule expressions into cryptoconditions, a new script opcode OP_EVALTXRULES is implemented that allows to add a spending rule expression into the scriptPubKey.
Also a support is added for a new expression function DecodeOpReturn that accepts a transaction object and opreturn structure description and returns a map object with decoded opreturn fields (from the last vout).
Added a sample rpc sendtoaddresswithrule that enables to add a rule expression into the spk .
This rpc also allows to create and send a transaction spending the utxo protected by a rule and add an optional opreturn.
The opreturn is a json array of string or number values (treated as int64). A string of 1 char is stored as a single byte in the opreturn (to allow to store funcids for example), like '["C", 12345, "myString" [ 1, "another string" ]]'. The json should be enclosed in apostrophes in the cmd line param.
An opreturn could be also passed as a hexstring instead of json.
The rpc could be run as follows:
First run - adding a spending rule to allow to spend the utxo if the spending tx funcd = 'C':

chips-cli sendtoaddresswithrule "mwgWZJWErxjAiNhXyLXnkzEZudSXdLsa7R" 0.1 'AND { DecodeOpReturn(evaltx, "{funcid:C}").funcid=="C" }'

Second run - spending the previously created utxo with the rule, referring the utxo txid and vout, also adding an opreturn with 'C'. The created tx spends only the referred utxo, so the txfee is also taken from it (also the privkey should exist in the wallet for the address):

chips-cli sendtoaddresswithrule "mwgWZJWErxjAiNhXyLXnkzEZudSXdLsa7R" 0.09 "" 11064fe04e0b3bf26d3464a02a5068697637dda127e40248a56d1b0832703702 1 '["C"]'

There is also a dice betting example with a OP_EVALTXRULES opcode demo added to the chips extension ().

The repo with chips opcode extension: https://github.com/dimxy/chips3/tree/txrule-poc

Currently supported rule expressions

Any rule expression should be started with one of two possible constructs:

  • AND { ... } - rule statement requiring that all the statements inside { ... }, which have the boolean return type, evaluate to True
  • OR { ... } - rule statement requiring that any of the statements inside { ... }, which have the boolean return type, evaluate to True
    Inside the compound statement {...} could be: a list of expressions, unnamed compound statement { ... }, other AND {...} or OR {...} rule statements or iterable statement FOR(var : iterable) {...}

Many number operators +,-,*,:,(),==,!= are built-in and supported
Built-in logical operators && and || are supported
Several built-in string function supported: len(), lower(), upper(), strip(), split()
No errors that are returned from rules are supported yet.

Unnamed Compound Statement Unnamed compound statement is a list of several statements inside braces {....}, it is not a rule statement as it does not have AND or OR keyword beforehand.
The last statement of the compound statement is used as the result of the whole compound statement. A compound statement may appear only inside AND{...} or OR{...} rule statement. Example:

AND {
  {                           # unnamed compound statement begins
    h = chainActive.height()
    h > 100                   # last statement is the logical result of an unnamed compound statement 
  }                           # unnamed compound statement ends
}

FOR-statement
FOR(var : iterable) { ... } allows to iterate over tx.vin or tx.vout arrays. FOR-statement always evaluates to True.
Example: allow to spend if evaltx has at least a cc vout:

AND
{
  {                              # using unnamed compound statement
     hasCC = False
     FOR(vout : evaltx.vout) {
       hasCC = hasCC || vout.scriptPubKey.isCC
     }
     hasCC == True               # last statement is the actual result for the unnamed compound statement 
  }
}

Blockchain objects currently supported in expressions:
Object tx properties and methods:

  • vin - array of vins
  • vin[i].hash - string with the hash of the spent utxo
  • vin[i].n - n of the spent utxo
  • vin[i].scriptSig.isCC - true if it is a cryptocondition scriptSig
  • vout - array of vouts
  • vout[i].nValue - amount value
  • vout[i].scriptPubKey.isCC - returns true if it is a cryptocondition scriptPubKey
  • funcid - returns a string with the funcid from the tx opreturn
  • height() - returns the tx block height

Object evaltx
evaltx is the currently validated transaction and has the same properties and methods as transaction tx object.
Example: allow to spend if vout0 amount > 1000: AND { evaltx.vout[0].nValue > 1000 }

Object chainActive properties and methods:

  • chainActive.height() returns active chain current height.
    Example: allow to spend if current height > 1200: AND { chainActive.height() > 1200 }

Currently Implemented Functions:

  • GetTransaction(hash) - returns tx object from the chain
  • DecodeOpReturn(tx, opreturn-desc) - for a tx (also may be the evaltx) and opreturn description, decodes the tx last vout's opreturn into internal object accessible by properties defined int the opreturn-desc. The opreturn-desc has a list of names and type descriptions denoting the next opreturn field type and format. Names are arbitrary and used to access field values in the decoded object
    Type descriptors: 'C' - one char (like a funcid), 'I' - int32_t, 'V' - CAmount(int64_t), 'S' - string prefixed with a string length as a compact-size value, 'H' - uint256, 'A' - array of set of fields prefixed by the array length as a compact-size value
    An opreturn-desc sample:
    {myfuncid:C,myheight:I,myamount:V,mystr:S,mytxid:H,myarray:A{prevtxid:H,prevheight:I}}
    When decoded, fields are accessible by their names in the opreturn-desc, for example:
    AND { opret = DecodeOpReturn(evaltx, "{myfuncid:C,crid:H}"); crid != evaltx.vin[0].hash }

More expression examples:

  • Allow to spend if at least 10 confirmations for the spent tx:
    AND { GetTransaction(evaltx.vin[1].hash).height() + 10 < chainActive.height() }
  • Allow to spend only if spending tx output is a cc and has certain value:
    AND { evaltx.vout[0].scriptPubKey.isCC && evaltx.vout[0].nValue==100000000 }

Rule Expression Usage Principles

Rule expressions could be implemented in the following ways:

  • CC contract-like, when expressions are stored in the transaction opreturn data (or a special kind vout could be developed). This introduces a contract style when there is a contract definition and contract instance transactions exist
  • Non-contract-like, when expressions are simply stored inside cryptoconditions, in the newly added eval condition data parameter (so the rule will be applied to the output where this eval condition is stored). This is a non-contract approach when rules just determine if one tx could spend another tx
  • Implemented as a new script opcode, which allows to add a rule to the scriptPubKey and by that additionally validating the transaction spending the utxo with the rule (see CHIPS update).

VM-based CC Contract Described

A VM-based contract will have a definition transaction and instance transactions.

Contract Definition Transaction
To create a new VM-based contract first a definition tx is created with a set of validation expressions (to be run inside the VM).
We continue using the funcid concept, so for each funcid an expression is defined. Each tx in a VM-based cc contract should have a funcid with a corresponding rule expression.

Contract Instance Transactions
Next, a contract instance initialise tx is created which refers the contract definition tx. We may create one or many initialise txns and by that one or many instances of the same contract are created. After an initial tx created more subsequent txns could be created during the contract instance lifetime. Each cc tx contains a funcid and on adding such a tx to the chain the VM is started with the validation expression (from the definition tx) matching to this funcid from the tx. Along with the validation expression from the definition tx the data from the validated tx is also passed to the VM as a parameter (tx context). If the expression is evaluated to true the validation is treated okay and the transaction is allowed to be added to the chain. Note: funcids to rules mapping not supported yet - see the update on current status

Non-contract VM-based Rules Usage Described

For non-contract usage one could create a transaction with one or several cc output, with a spending rule expression inside the output's cryptocondition scriptPubKey. When such an output is spent the rule in the output's scriptPubKey is evaluated into true or false and the output is allowed to be spent or not.

VM Expression Language (not fully supported yet - see the update on current status above)

The VM expression language is a simple expression language which allows to access transaction parts like inputs and outputs and some chain data and make calculations eventually resulting to a bool value. It should use a set of logical, arithmetic, string and array operators and will allow to refer to the inputs and outputs of validated txns, previous txns and some chain parameters (like current height, block params etc), by using keywords and predefined function names.
The language intentionally does not support loops and if-else construct. Instead, several expressions could be combined in groups under the scope of boolean operators 'and', 'or', 'xor'. This appears to be more oriented towards a rule framework

Here is a draft of the VM Expression Language Specification (Note: currently implemented only part of it - see the update above):

Operators in Expressions
Supported operators (in priority order):

  • () grouping operator
  • . access chain object properties or methods
  • [] sequence indexing
  • +, -, /, * arithmetic operators on numeric operands
  • >, >=, <, <= comparison operators on numeric operands
  • ==, != equality operators on any operands
  • &, |, ! logical operators
  • = assignment operator allows to create and set value of a variable for its further reading in the same expression
  • ; expression separator in multi-expression.

Possible Operands
Operands could be following:

  • preset variables which values obtained from the blockchain (like chainActive)
  • variables in definition rpc which values are assigned with instance parameters
  • variables defined in expressions
  • constants defined in expressions
  • return values of supported function calls (like chainActive.height())
  • other expressions. If any errors occur in a logical expression, the result of the expression should evaluate to false.

Embedded types and objects
Embedded types represent blockchain artefacts which can be used in expression operands, in function parameters or returns. The following basic types are supported and checked in runtime:

  • byte byte array value
  • string string value
  • amount amount (numeric)
  • index integer index
  • txid transaction id
  • timestamp unix time in seconds (numeric)
  • pubkey public key
  • address blockchain address type
  • sequence an array of elements of same type

Blockchain complex objects:

  • chain chain object
  • tx transaction object, has inputs and outputs
  • vin tx input
  • vout tx output
  • txo a pair of txid and vout number

Note that type matching is checked at runtime, when the VM is started. Type of variable is defined implicitly as the result of expressions parsing.

Objects' properties and methods
Objects has properties and methods that may be used in Expressions (draft list): 'txo' properties and methods:

  • txo.txid txo's txid
  • txo.n txo's index
  • txo.isSpent returns true if txo is spent
  • txo.address returns address for txo
  • txo.time returns time in seconds for the block where txOut is mined
  • txo.height returns block height where txOut is mined
  • txo.isCC returns true if txOut is a cryptocondition

'tx' properties and methods:

  • tx.funcId returns funcid from the transaction opreturn
  • tx.data tx data from the tx opreturn
  • tx.vin sequence of vin objects
  • tx.vout sequence of vout objects

'vin' properties and methods:

  • vin.hash previous txid
  • vin.n previous tx vout index
  • vin.hasPubkey(pubkey) returns true if this vin contains 'pubkey'
  • vin.isCC returns true if cc input
  • vin.hasEval(evalCode) returns true if vin has evalcode

'vout' properties and methods:

  • vout.amount returns amount
  • vout.isCC returns true if cc output
  • vin.hasEval(evalCode) returns true if has evalcode

'pubkey' properties and methods:

  • pubkey.address returns normal address for pubkey
  • pubkey.ccaddress(evalcode1, ...) cc address for pubkey for the evalcode(s)

'chain' properties and methods:

  • chain.height() chain's current height

Supported Functions:

  • MakeTxo(txid, n) creates a txo object from txid and vout number
  • GetTransaction(txid) returns a tx (transaction object) for a txid
  • FindLatestTxo(address) returns the latest txo object (with the most recent height) for an address
  • CCAddressAmount(address) calculates amount on address taking into account cc transactions. The cc transactions must be valid to be accounted (that is, with ExactAmounts for cc inputs and cc outputs function evaluating to true) - not sure we need this
  • AddressAmount(address) returns normal address amount
  • several byte array functions
  • several string functions
  • several math functions

Preset Variables
Variables with preset values which are available in expressions:

  • evaltx currently validated transaction
  • chainActive active chain object with its properties and methods

Iterator FOR
Iterator FOR allows to apply expressions to iterable chain object, like tx vins or vouts.

  • FOR(vin : tx.vin) { block-statement } iterate by all transaction vins
  • FOR(vout : tx.vout) { block-statement } iterate by all transaction vouts

Logical Expressions Grouping:
Several expressions may be grouped into sets of rules under one of possible logical operators AND or OR. For grouping the following syntax is used:

  • AND {...} 'and' grouping
  • OR {...} 'or' grouping

Expression inside logical groups must be organised with { ... } tags. Groups may be nested.
A returned error variable may be set for a group, like: AND("invalid tx opreturn data"){ ... } or OR("too early to spend"){...}. Those errors will be propagated to the validation code to be reported if the rule evaluated to false. Topmost errors will override lower level errors.
Example of groups with nested ones and an error property:

AND { 
  AND("spend not allowed at this height") {
    chainActive.height > 15000
  }
  AND("invalid cc vout") {
    evaltx.vout[0].scriptPubKey.isCC
    evaltx.vout[0].nValue == 10000
  }
}

Not Turing Complete

To prevent gas fee necessity, the expression language is not Turing complete, so loops are not allowed. However, iterators over tx inputs or outputs are supported. This is the only loop-like construct in the language. Iterators allow to apply an expression to the iterated elements (and are expressions themselves so should return a bool value, too <- not sure about this).

VM Contract Upgradeability

We can propose a simple technique to upgrade VM contract definitions: preserve in the definition tx a special output indicating the contract is actual. If this output is spent this mean a contract upgrade is existing. To make a contract non-upgradeable we should send this output to an unspendable address.

Expression Examples

Calculate that amount of cc inputs equals amount of cc outputs

AND("inputs not equal outputs") {
  inputs = 0
  outputs = 0
  FOR(vin : evaltx.vin) {     
    inputs = inputs + GetTransaction(vin.hash).vout[vin.n].nValue
  }
  FOR(vout : evaltx.vout) { 
    outputs = outputs + vout.nValue
  }
  inputs == outputs
}

Ensure at least 100 confirmations exist for an input spent by the validated tx

AND("not enough confirmations") {
  txo = MakeTxo(evaltx.vin[0].hash, evaltx.vin[0].n)
  txo.height + 100 < chainActive.height
}

TODO:

We need yet to solve or discuss questions, regarding reusable code, using evalcodes to trigger the VM validation and interaction with other cc modules.
Shared reusable expressions We can allow to create shared expressions (helpers) that can be reused in the same contract code or even to create a library of helpers. Helpers should be able to receive parameters and return output values. Probably we could have a common section in contract definitions and use additional xml-like tags to define shared expressions. We also need parameter binding feature. For example:

<shared name="GetTokenData" in0="tokenid:txid" return="data:bytes">
  shared expression code goes here...
<shared>

Now the shared code could be called from another expression:

<e>
   data = GetTokenData(tokenid)
</e>

How Trigger the VM validation
Another question is how to trigger the VM validation. Looks like this VM contracts should be defined under a single dedicated evalcode (so the VM validation is triggered when a certain evalcode is set in a tx's inputs and/or outputs.

Interaction with other evalcodes
How the VM validation will interact with other evalcodes? We can have several evalcodes in the same tx or even in the same input or output. Then validation code for each evalcode is triggered.

Clone this wiki locally