Skip to content
This repository has been archived by the owner on May 28, 2019. It is now read-only.

Commit

Permalink
xmr: master merging commit
Browse files Browse the repository at this point in the history
xmr: adding agent tests to travis
[c752866] xmr: unused imports removed
[85115fd] xmr: serializer test fixed

- removed unsupported messages after serialization simplification
[6087475] xmr: sign step 06 - comment on bulletproof hashing

- hash_bp(bp) != hash(serialize(bp)) because hash does not contain array lengths
[4abb547] xmr: sign protocol - multisig logic removed
[813cb3a] xmr: lite protocol removed

- backup left in xmr-total-full-with-lite branch
[e5f5b5b] xmr: serialization slimming

- base types reduced, not needed for now
- some int serialization methods not used now
[fb515aa] xmr: serialization - archive simplified

- simple parameter passing, no kwargs
- unused methods removed
- reader/writer passing removed for archive methods
[ec4c4ad] xmr: KeccakXmrArchive simplified, no archive used

- getting rid of container_size. We dont use containers with fixed size so this special case can be abandoned.
- KeccakXmrArchive is lighweight without need to touch main serialization
[90065bd] xmr: serialization - serialize_archive removed

- custom serialization routine is not required at this moment
[b98c2f8] xmr: extra serialization refactored, manual serialization

- extra is serialized manually to reduce serialization overhead
- extra contains simple structures now:
  - payment ID = already serialized manually
  - tx pub key = easy to serialize manually
  - tx additional pub keys = serialized manually with little effort, more efficient memory usage
[8ce28a5] xmr: state 6 - provided tx keys removed

- needed only in the multisig scenario which is pruned now
[8d827f4] xmr: PreMlsagHasher pseudo out hashing fix
[49e552d] xmr: redundant import removed
[4199943] xmr: KeccakXmrArchive simplified
[69bbf5f] xmr: PreMlsagHasher - KeyV import removed, comment added
[9a194fa] xmr: step7 - manual hashing of Extra
[d8a0928] xmr: PreMlsagHasher state load/save removed

- not needed, state not serialized anymore
[d5f43fa] xmr: serialize reimport removed

- complex types are not surviving protocol boundary anymore, no need to fix hierarchy problems due to unimporting
[3b04561] xmr: serialize - TupleType removed
[b9a5698] xmr: serialization schemes simplified
[a59dbb8] xmr: HashWrapper removed
[d0d1f05] xmr: step 01 cleanup and comments
[8f7a778] xmr: isort, black, flake8 fixes
[14265eb] xmr: getting rid of CtKey from the state

- only lightweight objects are kept in the state. CtKey is import heavy object. Each set_out call locally imports a new own version of the Ctkey which causes a memory leak.
[c0cfc20] xmr: set_out minor function call fix
[c11c468] xmr: sign_tx unimport optimization to reduce fragmentation
[df0a1df] xmr: range_sig allocation reordering, large chunks first
[63cddd5] xmr: remove misc.StdObj
[8c8e3f3] xmr: proper memory usage in workflow
[9be1e0a] xmr: typos and renames
[87f718b] xmr: back to flat workflow
[6475133] xmr: refactor builder to seperate steps

- lot of work to be done, but the general idea will probably stay
- the messages workflow works, but the signed tx was not accepted by
daemon, so there is a bug somewhere
- additional cleanup/refactoring is defintely needed
[14b0a85] xmr: iface modified to layout
[27d568e] xmr: extmod refactoring

- *_into removed, replaced by generic methods
- point_add, point_sub added
- code cleanup (+1 squashed commit)
Squashed commits:
[fbe3949] monero support added

Squashed commits:
xmr: hmac/enc keys removed from builder
[41028df] xmr: unused function removed
[358573e] xmr: PR comments fixes
[4abf9dc] xmr: test fixes after refactoring
[192785a] template rebuilt
[57a1f25] xmr: wrapper protocol messages removed
[6f40ce1] xmr: trezor-common version bump & sync
[03e71de] xmr: check input permutation
[6fc8b0e] xmr: code cleanup, refactoring
[85ecc15] xmr: crypto code cleanup
[20b4113] xmr: chunked bulletproof vectors

- workaround for the heap fragmentation problems
[66786f9] tools: enable to reset class level indentation for mocks gen
[dc6f84a] xmr: extmod-monero comments added, for mocks
[b1d4ab1] xmr: code cleanup
[447a862] xmr: tsx confirmation raises exception on cancellation
[00dd8f6] xmr: protocol optimizations removed, flow unified

- in_memory optimization stored tx.vin parts in the memory which enabled to skip roundtrips with permutations and hash_vini. Optimizations was removed so the protocol flow is unified among inputs, independent of the tx specs

- many_inputs: optimization stored spending keys for UTXO in memory, now it is offloaded in the encrypted form.
[ea69c7a] vendor: trezor-common version bump & pb sync
[5d81c2a] xmr: manual BP serialization

- more memory effective as the memory is critical in the range proof section
[d64bda7] xmr: range_proof C-impl deprecated

- using now partitioned implementation in Python, which is also quite fast and easier to maintain due to allocations and buffers.
[18604e0] xmr: borromean range sig generated by partitions

- overcomes heap fragmentation problem
[65a5116] xmr: comments removed
[ca2bd0c] xmr: auto-generated intelliJ param comments removed
[a75ef32] xmr: code cleanup, heap fragmentations fixes

Squashed commits:
[d2ac2eb6] xmr: addr cleanup
[7e4c1a9c] xmr: code cleanup, heap fragmentations fixes
[93af8af] xmr: refactoring, typing, comments
[28df866] xmr: comment fix
[8b4f4d9] xmr: serializer test fix (+34 squashed commits)

Squashed commits:
[823ee19] xmr: crypto comment cleanup
[6debfb6] xmr: ring_ct cleanup
[759f52b] xmr: tsx signer code style, hintins
[0b175bc] xmr: tsx builder external state removed
[fee4a5a] xmr: builder state fix
[92736fa] xmr: sign_tx unimport
[a570ecb] xmr: misc code cleanup
[4a496bb] xmr: hash wrapper not needed in writer
[fefdb83] xmr: signer serialization improved
[8fa6eec] xmr: signer mem clean
[66c53fe] xmr: isort
[6996bd9] xmr: black
[59915a8] xmr: tsx input serialization refactored
[326af13] xmr: msg dump with prefix
[6e39801] xmr: manual serialization of tx prefix
[9e5e047] xmr: manual serialization improvements
[d07cee6] xmr: manual serialization of txout elements
[8d56c80] xmr: TxOut custom serialization optimized II
[c19ba12] xmr: TxOut custom serialization optimized
[ce0d9b0] xmr: TxOut manual serialization
[44e3834] xmr: sing_tx unimport
[61ac61b] xmr: lite log trace rename
[176b427] xmr: de-async overhaul
[89ae3ba] xmr: diag style
[5ccb2fb] xmr: wrappers cleanup
[aa86fb1] xmr: py3 only inheritance
[8031b1b] xmr: builder, log_trace -> mem_trace for clarity
[25bf70d] xmr: debugging logging only in debug mode
[c7c8d3c] xmr: iface cleanup
[b037339] xmr: lite debug only
[b1f6ce0] xmr: diag only in debug
[de7d718] xmr: tsx counter removed
[76729be] xmr: tsx_sign removed
[c6e6ffa] Merge commit 'ba500bf4ec1ef9cd953bdf5a47888c5226db8d0b' into xmr
[ee97ef9] xmr: minor code cleanup

xmr: black
xmr: minor code cleanup
[bae3ecac] xmr: bp comments
[5e812e6f] xmr: sign - mem_trace, pydoc
[7216a8c6] xmr: pydoc removed
[e87365f4] xmr: layout cleanup
[8d21d82e] xmr: redundant constructors removed
[9aa82bed] xmr: redundant comments removed
[9b926d6c] xmr: preludes removed
[bc9e77f1] xmr: readme update
[cf62047] xmr: aggregated bulletproofs + rsig offloading

xmr: change idx fix
xmr: iface refactoring, integrated address (+5 squashed commits)
xmr: layout pagination refactoring
xmr: addr - integrated address
pb: sync
vendor: trezor-common version bump
xmr: style fixes
xmr: handle sweep tsx correctly

- handle dummy change address correctly
xmr: integrated address generation
build: fix after trezor-crypto version bump
xmr: new protocol dispatch handlers
xmr: slip0010
[43cf4c3c] xmr: comment fix
xmr: extmod pointer aritm fix
xmr: _into api unified, result is the first parameter
xmr: bp cleanup
xmr: scalar nullity test fix
xmr: msg registration improved

- lite protocol optional
- diag protocol optional
xmr: unused imports (+33 squashed commits)
[b4d045ae] xmr: bp - noqa flake8 false positive
[2c79d4be] xmr: isort
[8b9d2835] xmr: code cleanup
[eb7496e9] xmr: iface - shorter timeouts for faster tests
[59520b63] xmr: ringct comment
[6b16088e] xmr: signer - comment fixes
[a08958e2] xmr: simple and bulletproof condition fix
[4e0289a9] vendor: trezor-common version bump
[de472e5a] xmr: black
[234d2249] xmr: lightening, fixes, KeccakXmrArchive

- builder keys
- unload mods before memory intensive operation
[abdec665] xmr: sign_tx logging
[989d8687] xmr: serialize lightening
[7d61f056] xmr: tsx sign refactoring, lightening

- wake_up state restore
- minimize import weight
[3a0daa8b] xmr: serialize thinning
[65ad1d2e] xmr: serialize thinning
[501221d5] xmr: bp - thinning
[3d980377] xmr: bp - generalization with proof_v8
[10d11d60] xmr: extended rsig - offloading protocol
[a8f5caa2] xmr: crypto - rsig params fix
[f5e130b8] xmr: crypto - inv8
[dbc3f9d8] xmr: rsig pb sync
[5748a13e] xmr: bp - data for bp4 fix (+18 squashed commits)

Squashed commits:
[5bcd54e3] xmr: bp - black
[e93e97dd] xmr: bp refactoring, large memory optimizations

- memoryview in __getitem__ requires new memory allocation so the refactored version uses to(), read() methods that can operate directly on buffers without need to create memory views.
[c30745a] xmr: bp - black
[f5c4069] xmr: bp - tests extended
[8dae75d] xmr: bp - get_exponent optim
[3e59ff8] xmr: bp - precomputations for 4 statements
[d1d2e29] xmr: bp - gc.collect
[1bb6b5b] xmr: bp - optimizations, streamlining
[2a2b0cb] xmr: bp - verification in log(MN) memory for 1 proof

- not allocating MN vectors
- sequential multiexec added for memory efficient verification
- bulletproofs: maintain -z4, -z5, and -y0 to avoid subtractions [8276d25]
- bulletproofs: merge multiexps as per sarang's new python code [acd64d2b]
[75aa7de] xmr: bp - memory optimization
[a10d05a] xmr: bp - deterministic mask generation init
[5060d6a] xmr: bp optimizations
[dd69eb1] xmr: bp - black
[19f0f64] xmr: bp - optimizations, power key vector
[2ba63f8] xmr: bp - minor cleanup, optimizations, scalarmultH
[31c9ca2] xmr: bp - mem clean
[3fc2c79] xmr: bp - memory save
[5b16c9c] bp: black
[f1040c97] xmr: crypto - memory leak fix
[ff863510] xmr: iface - flake
[6ebf69c2] xmr: lite - flake8, black
[eee55d62] xmr: bp - memory diag
[2767009b] xmr: bulletproofs upgrade, mainnet version, cleaning
[be6ebbd5] xmr: lite protocol
[d603e96d] xmr: pb sync
[5da15da9] vendor: trezor-common fix
[0373b97e] xmr: iface - output confirmation split, subaddr fix
[2cf32176] xmr: monero - subaddress fixed for index (0, 0)
[3bb8f08b] xmr: enc.aescbc added - for lite protocol (+1 squashed commit)
Squashed commits:
[011dbaab] TMP: trezor-common on master, crypto on ph4

- trezor-crypto on ph4r04 fork as it has all required stuff
- Lite protocol not merged in master, thus does not work in the PR
[795b34e] xmr: get_address fix
[2d39c90] xmr: bp - import fix (squashed commit)

Squashed commits:
[2d5c6ce] extmod: monero - reduce32 and ge25519_norm removed (squashed commit)

- not needed in trezor-core (+4 squashed commits)

Squashed commits:
[90e6b5c5] xmr: bp optimization
[4fda0d22] xmr: redundant ge_ functions removed
[68903767] xmr: crypto - sc_reduce32 not needed
[c8a6c80] xmr: test for inversion added (+12 squashed commits)

Squashed commits:
[378928db] xmr: adapting to new trezor-crypto
[8f4ff8c1] protob sync
[82dff70a] vendor: trezor-common version bump
[fabc67b3] extmod: monero - inversion mod curve order optimized a bit
[4f29fe4] xmr: import fix
[f6f8e30] xmr: bp - code cleanup
[d54b4f3] xmr: bp - memory cleaning
[1065abc] xmr: tsx_signer - bulletproofs fixes
[9f8a700] xmr: bp key vector iterator fix
[49c2597] xmr.serialize: bulletproof fix
[1ee7737] xmr: monero - format
[cf0a710] xmr: bp last mask fix (+20 squashed commits)

Squashed commits:
[fa1c362] xmr: black
[3f3e31f] xmr: bulletproofs added to signer
[d23d928] xmr: protocol.tsx_sign_builder - logger collects
[a28eb55] xmr: bp - memory optimizations
[d2fcb23] xmr: tests for bulletproofs added
[82eef14] xmr: bp - gc (+14 squashed commits)
Squashed commits:
[4cf70d9] xmr: bp - gc
[42877b0] xmr: bp - minor memory optimization
[2c612e4] xmr: bp - use sc_inv_into
[d7e9dab] xmr: bp - KeyVEval fix
[1523f40] xmr: bp - blacked
[b264a65] xmr: bp - KeyVEval - caching current element, avoid allocations
[83ba7a6] xmr: bp - memory view optimized
[b517906] xmr: bp - gc() during inversion
[92d37c8] xmr: bp - gc.collect() after expensive inversion
[e7fad55] xmr: bp - hashing memory optimization
[4c27815] xmr: bp - deterministic masks optimization, prove_s1 optim
[cbf74a7] xmr: bp - detect which modular inversion is usable
[8ea1ec4] xmr: better memory tracing for bulletproofs
[2f4dd55] xmr: bulletproofs added
[1928e2d] xmr: crypto - sc_inv_into added (+2 squashed commits)
Squashed commits:
[f895fa6] xmr: crypto - hash to existing buffer
[b76c6b0] xmr: crypto - in-place crypto functions added

- required for Bulletproof to minimize the heap fragmentation
[cab4366] extmod: monero - modular inversion mod curve order added (+2 squashed commits)
Squashed commits:
[52a6e48] extmod: monero - hash into buffer added
[695a382] extmod: monero module - muladd256_modm added

- required for Bulletproof
[3f4498d] xmr: crypto tests added

- basic unit tests for crypto, tests monero module and underlying trezor-crypto + basic address manipulation
[820d012] pb sync
[49eeddd] vendor: trezor-common version bump
[3038244] xmr: crypto - point norm not needed
[89701c4] tests: xmr - serializer tests added
[bfee46d] tests: support async unit tests, assertListEqual added
[55c1448] xmr: serialize - serialization logic cleaned, refactored
[4b77163] xmr: simplification, do not ask to confirm change tx output

- change address checked to match main address in the builder
[f334d8a] xmr: protocol: simplification - require change address to equal the main address
[1a3416e] xmr: unpack256_modm_noreduce added

- 32B array to integer mod curve order, without modular reduction after conversion
- required for bulletproofs
[1c94b5d] xmr: readme added
[3cc9f9f] extmod/monero: mul256_modm added, required for BP
[5cf77a4] xmr: monero support added [for review]

depends on PRs:

trezor/trezor-crypto#162
#286
  • Loading branch information
ph4r05 authored and tsusanka committed Oct 8, 2018
1 parent de12ad7 commit d0dea3b
Show file tree
Hide file tree
Showing 76 changed files with 9,677 additions and 0 deletions.
8 changes: 8 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ addons:
- python3.6-dev
- python3.6-venv

cache:
directories:
- $HOME/libsodium

before_install:
- python3.6 -m ensurepip --user
- python3.6 -m pip install --user pipenv
Expand All @@ -42,6 +46,9 @@ install:
- curl -LO "https://github.com/google/protobuf/releases/download/v${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-linux-x86_64.zip"
- unzip "protoc-${PROTOBUF_VERSION}-linux-x86_64.zip" -d protoc
- export PATH="$(pwd)/protoc/bin:$PATH"
- ./travis-install-libsodium.sh
- export PKG_CONFIG_PATH=$HOME/libsodium/lib/pkgconfig:$PKG_CONFIG_PATH
- export LD_LIBRARY_PATH=$HOME/libsodium/lib:$LD_LIBRARY_PATH
- pipenv install

before_script:
Expand All @@ -63,6 +70,7 @@ script:
- test "$GOAL" != "unix" || pipenv run make build_unix_noui
- test "$GOAL" != "unix" || pipenv run make test
- test "$GOAL" != "unix" || test "$TREZOR_MODEL" = "1" || pipenv run make test_emu
- test "$GOAL" != "unix" || test "$TREZOR_MODEL" = "1" || pipenv run make test_emu_monero

notifications:
webhooks:
Expand Down
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,9 @@ test: ## run unit tests
test_emu: ## run selected device tests from python-trezor
cd tests ; ./run_tests_device_emu.sh $(TESTOPTS)

test_emu_monero: ## run selected monero device tests from monero-agent
cd tests ; ./run_tests_device_emu_monero.sh $(TESTOPTS)

pylint: ## run pylint on application sources and tests
pylint -E $(shell find src tests -name *.py)

Expand Down
3 changes: 3 additions & 0 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,8 @@ termcolor = ">=0.1.2"
Pillow = ">=5.2.0"
Mako = ">=1.0.7"

# monero
monero_agent = {version = ">=1.6.3", extras = ["tcry", "dev"]}

[pipenv]
allow_prereleases = true
4 changes: 4 additions & 0 deletions SConscript.firmware
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ CPPDEFINES_MOD += [
'RAND_PLATFORM_INDEPENDENT',
('USE_KECCAK', '1'),
('USE_ETHEREUM', '1'),
('USE_MONERO', '1'),
('USE_CARDANO', '1'),
('USE_NEM', '1'),
]
Expand Down Expand Up @@ -63,6 +64,9 @@ SOURCE_MOD += [
'vendor/trezor-crypto/ed25519-donna/ed25519-keccak.c',
'vendor/trezor-crypto/ed25519-donna/ed25519-sha3.c',
'vendor/trezor-crypto/ed25519-donna/modm-donna-32bit.c',
'vendor/trezor-crypto/monero/base58.c',
'vendor/trezor-crypto/monero/serialize.c',
'vendor/trezor-crypto/monero/xmr.c',
'vendor/trezor-crypto/groestl.c',
'vendor/trezor-crypto/hasher.c',
'vendor/trezor-crypto/hmac.c',
Expand Down
4 changes: 4 additions & 0 deletions SConscript.unix
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ CPPDEFINES_MOD += [
'AES_192',
('USE_KECCAK', '1'),
('USE_ETHEREUM', '1'),
('USE_MONERO', '1'),
('USE_CARDANO', '1'),
('USE_NEM', '1'),
]
Expand Down Expand Up @@ -60,6 +61,9 @@ SOURCE_MOD += [
'vendor/trezor-crypto/ed25519-donna/ed25519-keccak.c',
'vendor/trezor-crypto/ed25519-donna/ed25519-sha3.c',
'vendor/trezor-crypto/ed25519-donna/modm-donna-32bit.c',
'vendor/trezor-crypto/monero/base58.c',
'vendor/trezor-crypto/monero/serialize.c',
'vendor/trezor-crypto/monero/xmr.c',
'vendor/trezor-crypto/groestl.c',
'vendor/trezor-crypto/hasher.c',
'vendor/trezor-crypto/hmac.c',
Expand Down
Loading

0 comments on commit d0dea3b

Please sign in to comment.