Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collect metrics through datadog for the perf project #1765

Merged
merged 40 commits into from
Nov 19, 2020
Merged

Collect metrics through datadog for the perf project #1765

merged 40 commits into from
Nov 19, 2020

Conversation

ayrat555
Copy link
Contributor

@ayrat555 ayrat555 commented Nov 9, 2020 •
edited
Loading

This PR adds the following:

  • sending of request metrics and test results to Datadog
  • making assertions based on datadog monitor events. Monitor events are emitted on:
  1. If the value of the metric is above a threshold. I set thresholds based on average time during my testing
  2. If the test failed at least once during the current test run.

I created two monitors for both tests:

  1. Deposits test dashboard https://app.datadoghq.com/dashboard/rpx-xu2-b2g/deposits-perf-tests?from_ts=1605343939055&live=true&to_ts=1605430339055&tpl_var_env=perf_circleci
  2. Transaction test dashboard https://app.datadoghq.com/dashboard/7kh-xx4-9qu/transactions-perf-tests

Created monitors:

Notes:

  • Monitors and dashboards are bound to env tag set for Statix so they can be distinguished by env.
  • For circleci, additional env tag is prepended so a failure in one test run won't cause failures in all test runs

@ayrat555 ayrat555 marked this pull request as ready for review November 13, 2020 09:58
@ayrat555 ayrat555 changed the base branch from ayrat555/perf-metrics to master November 13, 2020 09:59
@ayrat555 ayrat555 added the chore Technical work that does not affect service behaviour label Nov 13, 2020
ayrat555
ayrat555 commented Nov 15, 2020
View reviewed changes
docker-compose.datadog.yml
version: "2.3"
services:
datadog:
image: datadog/agent:latest
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datadog agent is added to send metrics during circle ci test runs.
example of failure - https://app.circleci.com/pipelines/github/omgnetwork/elixir-omg/8939/workflows/bcebdc24-e205-4f32-b009-64455735beb2/jobs/87535 - I added sleep(100) to simulate it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: datadog is also defined in docker-compose.dev.yml
maybe they could be merged?

ayrat555
ayrat555 commented Nov 15, 2020
View reviewed changes
priv/perf/apps/load_test/lib/test_runner.ex
System.halt(1)
end
end

defp maybe_add_custom_tag(start_date) do
Copy link
Contributor Author

@ayrat555 ayrat555 Nov 15, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

custom tags are only added during circle ci runs so test runs can be distinguished. For other environments, an env tag should unique. For example, env:perf_prod

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This goes well with my comment: #1765 (comment)

Are dashboards and monitors able to distinguish by this parameter?

Copy link
Contributor Author

@ayrat555 ayrat555 Nov 18, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, tests are distinguished by env tag. dashboards show data for a specific tag. To show a different env, you should change the env template parameter:
2020-11-18_16-37

For circle ci, custom tags are added which contain current timestamp so two circle ci runs won't intersect

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

@InoMurko
Copy link
Contributor

InoMurko commented Nov 18, 2020 •
edited
Loading

Feedback:

The dasboards are great, but:

  • we lack information about what "Average test count" means. Average of what? What is being counter? This needs to be somehow reflected and evident in the dasboard (title, description... whatever is available).

  • there's one instance of the dashboard per test type (transaction tests, deposit tests). Correct?
    This means, that we're only able to run one instance of the test at the same time? What if we want to run one of these tests towards development and another one towards sandbox?
    Same goes for Monitors. There's only one instance of the monitor, and the monitor is not being triggered on specific tags (like... where was the monitor triggered, which environment was the test run towards that the monitor was raised).

InoMurko
InoMurko reviewed Nov 18, 2020
View reviewed changes
Copy link
Contributor

@InoMurko InoMurko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of work here! Kudos!

Left a few comments I would like to see answers to and will approve the PR once that gets resolved.

priv/perf/apps/load_test/lib/application.ex Show resolved Hide resolved
priv/perf/apps/load_test/lib/service/datadog/api.ex Outdated
Comment on lines 30 to 31
# wait for events to be emitted
Process.sleep(30_000)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to wait? where is 30seconds coming from?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

monitor events are emitted based on 1 minute time intervals. If events are fetched right after test execution, events may not be published yet

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better to poll for this data? Maybe 30s is too much, or too little and we would randomly fail.

priv/perf/apps/load_test/lib/service/datadog/api.ex Outdated Show resolved Hide resolved
priv/perf/apps/load_test/lib/service/datadog/api.ex Outdated Show resolved Hide resolved
priv/perf/config/config.exs Outdated Show resolved Hide resolved
@ayrat555
Copy link
Contributor Author

@boolafish can you please review this pr

boolafish
boolafish approved these changes Nov 19, 2020
View reviewed changes
Copy link
Contributor

@boolafish boolafish left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing PR 💯💯💯💯

Probably to be different PR (or even not need a PR)
But I strongly recommend to start documenting:

  1. How to setup the perf tests here. What kind of pre-setup is required?
  • Do we need to first manually add those monitors in datadog?
  • what service dependency is required to live in the cluster? (eg. datadog agent?)
  1. Update the existing doc of how to run perf test

one side note: I am not sure how much do we need to chain the tests now (I know it was previously brought up for really long running tests like for days). One note/idea after looking at the PR is that we can actually separate traffic trigger and result checking into 2 spinnaker stages. This would make it flexible on having multiple previous stages that triggers the traffic and the one single last stage that check the metrics. 🚀

priv/perf/apps/load_test/lib/application.ex Show resolved Hide resolved
priv/perf/apps/load_test/lib/service/datadog/statix.ex Outdated Show resolved Hide resolved
priv/perf/apps/load_test/lib/service/datadog.ex Outdated Show resolved Hide resolved
@ayrat555
Copy link
Contributor Author

@boolafish

one side note: I am not sure how much do we need to chain the tests now (I know it was previously brought up for really long running tests like for days). One note/idea after looking at the PR is that we can actually separate traffic trigger and result checking into 2 spinnaker stages. This would make it flexible on having multiple previous stages that triggers the traffic and the one single last stage that check the metrics

I added the ability to run tests and make assertions separately.

From docs:

  Running tests without assertions.
  You can run tests without assertions by passing `false` as the last parameter:

    STATIX_TAG="env:perf_circleci" mix run -e "LoadTest.TestRunner.run()" -- "transactions" 1 80 false
  
  To just check if there are any events in the given period of time run passing start time and end time:

  STATIX_TAG="env:perf_circleci" mix run -e "LoadTest.TestRunner.run()" -- "make_assertions" 1605775276 1605785276

@ayrat555 ayrat555 merged commit dcc2d3d into master Nov 19, 2020
@ayrat555 ayrat555 deleted the ayrat555/metrics-datadog branch November 19, 2020 13:21
@ayrat555 ayrat555 mentioned this pull request Nov 20, 2020
Closed
InoMurko pushed a commit that referenced this pull request Dec 8, 2020
@ayrat555 @unnawut
@boolafish @pgebal @jarindr @kevsul @nicholasmueller @achiurizo
Merge master into master v2 (#1795)
9443b1c 
* /block.validate endpoint (#1668)

* feat: init block.validate

* completes parameter validation tests

* rename validation method with stronger typing

* feat: verify_transactions and and negative test

* add: positive test for verify_transactions

* chore: remove single pipes

* verify_transactions: add @SPEC and @doc

* pending

* test: Added tests for `verify_merkle_root/1`

* remove reverse operation

* refactor: simplify tests

* refactor: simplify verify_merkle_root/1 tests

* refactor: merkle validation tests + entities as module attributes

* add dialyzer spec and remove argument pattern matching

* move logic to dedicated validator module

* test: added test for `block.validate` endpoint

* refactor: make tests pass

* feat: response for `validate_block` view

* feat: error for mismatched_merkle_root

* simplify tests

* rename error

* remove unused method

* fix: error name

* fix: credo

* merge issue: gitmodules

* merge master issue: gitmodules [2]

* merge master issue: gitmodules [3]

* Merge branch 'ripzery/block.validate' of https://github.com/omisego/elixir-omg into ripzery/block.validate

* (re)delete specs

* return submodule

* refactor: decouple verify_merkle_root from transaction recovery

* add: negative test for invalid transactions

* refactor: improve performance of verify_transactions

* use pin operator instead of comparison

* refactor: remove use of & &1. for clearer syntax

* remove redundant comment

* move block validation logic into Watcher

* move parameter validation logic into BlockConstraints

* reflect file changes in controller

* refactor: make verification functions private and shift tests to stateless_validate/1

* fix potential false positive by computing Merkle root correctly.

* add: swagger specs

* refactor: endpoint to return boolean result

* add: nil error

* update documentation to reflect boolean result

* remove error reason in response

Co-authored-by: euro <[email protected]>
Co-authored-by: Ayrat Badykov <[email protected]>

* feat: configurable DB pool size, queue target and queue interval (#1689)

* feat: configurable DB pool size, queue target and queue interval

* fix: add release task to release config

* style: mix format

* feat: configure runtime env vars through releases.exs

* fix: remove release tasks

* fix: syntax

* docs: add docs for db pool env vars

* feat: auto trigger chart version bump (#1695)

* feat: add increase chart version script and job

* feat: filter release tag on CI config

* fix: make increase chart script work

* test: docker image

* feat: add watcher&watcher-info on increase version

* chore: remove commented configs and back to non-playground

* fix: extra space

* Block Validation: New Checks (#1693)

* feat: transaction counting and block-level duplicate input verification in block validation

* fix: dialyzer error

* fix: credo error

* feat: validate fee transactions in block

* test: fee transaction validation

* revert fee logic meant to be stateful

* remove: fee claimer check

* fix: place dummy fee claimer

* test names

* refactors: simplify duplicate input verification, use guards, etc.

* PR review: description changes

* fix: credo

* simplify duplicate input checking

* update documentation

* fix: dialyzer

* spacing

* suggested edits

* feat: block queue metrics and stalled submission alarm (#1649)

* feat: change block formation log from debug to info

* feat: BlockQueue publishes :block_submitting and :block_submitted events

* feat: BlockQueue.Monitor that raises/clears :block_submission_stalled alarm

* refactor: tracks root chain height directly from the bus

* refactor: revert variable naming

* fix: remove remaining EtherereumHeight.get()

* fix: typespec

* feat: add telemetry and statsd metric for blknum_submitting and blknum_submitted

* feat: add telemetry and statsd metric for blocks_submitting and blocks_stalled

* fix: rename :block_submission to :block_submission_gas

* feat: add statsd metric for :block_submission_success

* fix: remove non services

* docs: clearer explanation of block queue metrics

* docs: more explanation on event publishing

* feat: add :block_submission_attempt statsd metric

* fix: dialyzer

* feat: block submission stall config and supervisor setup

* format: formatting and naming alignment

* test: add BlockQueue.Monitor test

* refactor: shorten config name

* test: fix tests

* fix: broken service startup

* feat: configurable block submit stall from env var

* fix: move config namespace

* refactor: brackets everywhere

* fix: config naming

* fix: wrong namespace

* test: fix app name

* test: fix init

* refactor: use :block_submit_stalled everywhere

* fix: ignore unrelated events

* fix: telemetry deprecation

* fix: remove misleading metric :block_queue_num_blocks_submitting due to the way block submission sampling works

* fix: remove dup geth argument

* fix: remove unused block_submit_stall_check_interval_ms

* refactor: add alarm function naming

* refactor: reduce alarm function pattern matching cases

* refactor: move release task to releases.exs

* fix: remove obsolete tests

* feat: handle in-flight exits deletions (#1701)

* move omg_performance json rpc tests to perf project (#1691)

* chore: add feefeed docker to elixir-omg setup (#1700)

* add feefeed adapter

* change ip

* add feefeed url

* fix test barebone release

* fix port

* remove override variable from barebone config

* chore: use https instead of gits to avoid permission issue

* test: update to cabbage using fee api

* fix: git submodule without --remote flag

So it would not always pull the head branch from remote.

* fix: update cabbage to fix reorg

* fix: circle ci mis-comment

* chore: add healthcheck and depends on for services

and mock off other tests aside from reorg

* chore: sync with cabbage update

* chore: add space before network

* chore: ignore geth folder

* debug: comment out make docker steps

* chore: update priv/perf for debug docker-compose pull

* debug: use feefeed-dev with more logging

* debug: use newer feefeed docker that fixes a bug

* chore: update cabbage commit

* chore: uncomment the commented CI jobs

* chore: update latest cabbage

* style: docker file space

* fix: all docker images command have dockerhub authenticate

* chore: update cabbage for code review

* fix: circle ci login to dockerhub

and update cabbage commit

* fix: fee for perf tests

- set in the test config to use the new fee amount 75
- for LoadTest.Common, pass in the fee_amount and use the value from config

* chore: back to feefeed:latest image

* style: mix format

* refactor: move docker_login out from make_docker_images

Logic-wise it does not make much sense actually to be together.

* chore: update cabbage to latest

* style: fix lint

* chore: update cabbage commit

* chore: use master from cabbage

* chore: bring back submodule udpate --remote

This is just better. change the branch in submodele file instead for testing.

Co-authored-by: jarindr <[email protected]>

* feat: reintroduce automated changelog (#1708)

* feat: reintroduce github_changelog_generator

* feat: more changelog tags

* feat: wholly configuring the changelog sections

* feat: changelog label auto checker

* refactor: better review messages

* debug: apply workflow on all branches to check it's working

* fix: bad env format

* fix: wrong job name

* fix: missing closing quote

* fix: bad syntax

* fix: enable shell tracing

* fix: premature failure

* fix: remove extra full stop

* debug: reenable label check only for master

* fix: missing line break escape

* refactor: use Github's default env vars for its url building

* feat: fail the github action instead of relying on a bot

* fix: remove continue-on-error

* fix: combine github action steps so it's easier to find the original error

* feat: echo error

* fix: proper error

* fix: syntax

* fix: force exit on error [ciskip]

* fix: operator precedence

* fix: ignore block_submitting that are above mined blocks (#1703)

* fix: fix in-flight exit deleted bug and add tests (#1714)

* revert: reverts ife deletion commits (#1725)

* Revert "fix: fix in-flight exit deleted bug and add tests (#1714)"

This reverts commit f302553.

* Revert "feat: handle in-flight exits deletions (#1701)"

This reverts commit b3081f8.

* Allow to run docker-compose without feefeed docker (#1726)

* refactor: abstract steps in circle ci to make file for perf

* chore: seperate docker-compose file for feefeed

to allow public contributor to run without feefeed docker

* fix: make start-services command with feefeed

* refactor: move mix format for api generated code back to CI

* debug: add debug CI workflow

* style: simplify docker-compose file

* debug: add log service command to perf

* do not set env var in makefile

* back to run whole workflow

* refactor: COMPOSE_FULL_SERVICES in makefile

* chore: spaces instead of tab in .gitmodules

* add test for 64_000 txs block hash (#1729)

* rm mix based chch (#1716)

machine

module rename

dont cache

install linux deps

sleep

postgres

child chain url and childchain empty fees

* Inomurko/remove child chain (#1737)

* rem childchain code

* erlexec

* fixtures inclusion

* revert cabbage and api client generation from omg v1 childchain

* we dont need wget anymore

* feat: handle in-flight exits deletions

* move dev env deployment job to helm repo (#1738)

* feat: trigger increase chart version on merge to master

* debug: switch to playground repo

And allow circle ci to run on non-master

* fix: unbound string

* feat: add jitter to the call

* remove jitter, not useful

* chore: cleanup

Uncomment CircleCI jobs, remove deployment jobs/scripts, not using playground repo

* fix: circle ci format

* fix: notifiy service do not wait for non-existing deployment job

* chore: remove childchain master deployment

Responsibility has been moved to omg-childhchain-v1 repo.
Have another PR for that: omgnetwork/omg-childchain-v1#15

* chore: remove childchain release from elixir-omg

Moved to omg-childchain-v1.

* fix: handle metrics for in flight exit deleted processor (#1742)

* Kevsul/standard exit perf test (#1732)

* tests: add load test for standard exits

* tests: add watcher status scenario

* tests: standard exit test syncs with watcher

* tests: revert test fee_amount

* tests: add max num retries when waiting

* tests: remove FundAccountRootchain scenario

* tests: change if to case

* Chore: try to fix flaky reorg tests (#1739)

Two kinds of errors I saw:
1. Tests are timing out during account creation and unlocking on both geth nodes. After I increased timeout for account creation, I haven't seen this error anymore
2. Tests are failing with wrong balances (on the childchain and the rootchain). I added additional checks to the reorg function to make sure nodes have the same latest block after reorg. I think it helped but still, tests are failing sometimes.

* update change log v1.0.4 (#1731)

* update changelog

* manual fix change log

* more manual fix change log

* Update VERSION

Co-authored-by: Unnawut Leepaisalsuwanna <[email protected]>

* Chore: parallelize tests by tags (#1744)

* parallelize tests by tags

* Chore: use exexec from upstream (#1743)

* use exexec from upstream

* fix: recheck PR label on synchronize and reopen (#1748)

* fix: recheck PR label on synchronize and reopen

* test: a pointless commit to check that the enforce-changelog-label job still passes

* debug: try the fixed feefeed

* Revert "debug: try the fixed feefeed"

This reverts commit a35fc90.

* feat: add deposits performance tests bot (#1745)

* feat: transaction.create optimisation (#1683)

* test: Init

* test: create `generate_utxos_map/3`

* test: put test in describe

* test: added `needed_funds/2` tests

* add tests and light refactor for get_sorted_grouped_utxos

* add tests and light refactor [2]

* test: refactor `create_advice/2`

* add spec and clarify funds_sufficient

* refactor: `needed_funds`

* refactor: use shorthanded type

* Add @doc  and readabilty edits

* add: tests for funds_sufficient/1

* refactor: simplify use of 'alice'

* refactor: move test inside describe

* test: add `select_utxo/2` tests

* refactor: remove extra line

* fix: spec

* feat: implement `add_merge_utxos`

* test: add test for `add_merge_utxos/2`

* feat: add prioritize_merge_utxos/2

* test: add test for `prioritize_merge_utxos/2`

* refactor: pipe one more time

* refactor and spec stealth merge

* refactor: test name

* add: tests and doc for `add_utxos_for_stealth_merge`

* feat: integrate add_merge_utxos to create_advice

* test: add test for create_advice

* add: doc content for add_utxos_for_stealth_merge

* refactor and test prioritisation of UTXOs for stealth merge

* refactor: split transaction operation

* Fix dialyzer type

* refactor: dialyzer

* fix: typo

* test: add utxo_selection test

* debug: transaction create

* feat: handle when original inputs are empty

* tests: remove transaction merge tests

* tests: add tests for utxo selection

* tests: handle when too many inputs to satisfy payments and fee

* refactor: remove todo

* feat: move `create_advice` from `utxo_selection.ex` to  `transaction.ex` and rename it to `create_inputs`

* test: add transaction tests

* test: change port `9657` to `9656`

* cleanup: remove unused var

* lint: remove @doc for defp

* test: revert back port `9656` to `9657`

* lint: fix lint

* refactor: remove result from response

* test: add prepare_test_server

* refactor: fix per reviews

* tests: add test case where 2 input use different currencies

* refactor: remove alice, bob from test context

* test: fix mismatch function signature

* fix: dialyzer type

* refactor: Cleanup

* fix: dialyzer

* fix: credo

* remove unusued alias

* refactor: remove comments

* lint: fix

* tests: add tests for transaction.create apis

* test: delete unnecessary function

* refactor: simplified `select_inputs` flow

* refactor: fix follows pr review

* tests: fix wrong function

* lint: add alias

* refactor: remove trailing comma

* refactor: var name

* switch branch for cabbage test

* refactor: move functions out of controller

* test: add outputs to assertion

* test: add tests for `transaction.create/2`

* test: implement tests for include_typed_data/1

* refactor: remove single pipe

* tests: add tests for `select_inputs/2`

* refactor: cleanup typespec

* lint: formatting

* lint: remove unused alias

* lint: reorder the alias

* refactor: move spec and simplified logic

* refactor: simplify pattern matching

* refactor: uniformize param

* revert: bring back merge transaction

* fixes

* refactor: lift out respond

* fix: is_required_merge doesn't work

* refactor: transaction api

* test: fix tests

* tests: fix

* fix: add result key back

* fix: dialyzer part-1

* fix: remove unnecessary dialyzer types

* fix: revert changes on .gitmodules

* refactor: select_inputs/2

* refactor: simplify select_utxo logic

* refactor: re-order functions

* fix: rename function

* fix: conflict

* Fix conflict

* refactor: rename functions

* refactor: remove single pipe

* fix: lint

* fix: tests

* fix: tests 2

* try: update spec repo

* test: change git modules

* test: revert submodule

* test: fix cabbage conflict

* test: add result key back

* fix: private and public function mixing

* test: fix imports and mixing private functions

* fix: break down pipe

* refactor: break down pipe

* fix: remove blank line

* fix: move alias to top

* refactor: renaming and break out pipe

* refactor: get_utxos_count

* refactor: create_transaction

* fix: typo

* refactor: build_inputs

* refactor: formatting

* refactor: lift out input

* refactor: prioritize merge utxos

* remove and ignore priv/apps/childchain_api

Co-authored-by: euro <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Mederic <[email protected]>

* chore: bump version to 1.0.4 (#1751)

We are already in 1.0.4 but we forget to port the VERSION file back when updating changelog.

* feat: comply with new Infura API (#1754)

* Publish the perf project (#1753)

* publish the perf project

* Utxo perf test (#1756)

* create and check utxo

* spend utxo

* assert balances after spending

* validate exact utxos

* fix token encoding

* add test to run on ci

* Update priv/perf/apps/load_test/lib/watcher_info/utxo.ex

Co-authored-by: Ino Murko <[email protected]>

* Update priv/perf/apps/load_test/lib/watcher_info/utxo.ex

Co-authored-by: Ino Murko <[email protected]>

* mix format

* fix build

Co-authored-by: Ino Murko <[email protected]>

* Add cli help command for the perf project (#1757)


* add cli help command for the perf project

* feat : transaction.merge endpoint  (#1715)

* test: Init

* test: create `generate_utxos_map/3`

* test: put test in describe

* test: added `needed_funds/2` tests

* add tests and light refactor for get_sorted_grouped_utxos

* add tests and light refactor [2]

* test: refactor `create_advice/2`

* add spec and clarify funds_sufficient

* refactor: `needed_funds`

* refactor: use shorthanded type

* Add @doc  and readabilty edits

* add: tests for funds_sufficient/1

* refactor: simplify use of 'alice'

* refactor: move test inside describe

* test: add `select_utxo/2` tests

* refactor: remove extra line

* fix: spec

* feat: implement `add_merge_utxos`

* test: add test for `add_merge_utxos/2`

* feat: add prioritize_merge_utxos/2

* test: add test for `prioritize_merge_utxos/2`

* refactor: pipe one more time

* refactor and spec stealth merge

* refactor: test name

* add min_length to base validators

* add merge route and controller

* fix compile errors

* add: tests and doc for `add_utxos_for_stealth_merge`

* test address/currency and utxo positions constraint

* feat: integrate add_merge_utxos to create_advice

* test: add test for create_advice

* add: doc content for add_utxos_for_stealth_merge

* refactor and test prioritisation of UTXOs for stealth merge

* test utxo constraint

* add other required utxo data

* fetch utxo logic

* start transaction_test

* finaly get basic transaction api test working

* merge p

* refactor: split transaction operation

* Fix dialyzer type

* refactor: dialyzer

* fix: typo

* test: add utxo_selection test

* debug: transaction create

* feat: handle when original inputs are empty

* tests: remove transaction merge tests

* tests: add tests for utxo selection

* tests: handle when too many inputs to satisfy payments and fee

* refactor: remove todo

* refactor: parameter validation for transaction.merge

* optional desc/asc for UTXO fetch

* WIP: merge functionality

* feat: move `create_advice` from `utxo_selection.ex` to  `transaction.ex` and rename it to `create_inputs`

* test: add transaction tests

* WIP: return multiple merge transactions if address/ccy params

* test: change port `9657` to `9656`

* cleanup: remove unused var

* lint: remove @doc for defp

* test: revert back port `9656` to `9657`

* lint: fix lint

* refactor: remove result from response

* test: add prepare_test_server

* WIP: merge when given utxo positions

* refactor: fix per reviews

* tests: add test case where 2 input use different currencies

* refactor: remove alice, bob from test context

* test: fix mismatch function signature

* fix: dialyzer type

* test for merge with address and currency, refactor create merge so no need for currency and address passing

* more happy path tests

* refactor: Cleanup

* fix: dialyzer

* fix: credo

* remove unusued alias

* remove unaccepted merge constraints

* remove max 4 merge constratint

* match tx.create response shape

* refactor: remove comments

* lint: fix

* tests: add tests for transaction.create apis

* test: delete unnecessary function

* address/currency merge controller tests

* utxo position happy path

* add multiple tx for utxo pos test

* refactor: simplified `select_inputs` flow

* start merge view test

* revert merge view since controler test makes this redundant

* remove default argument in get_sorted_grouped_utxos and reflect change

* refactor: remove "respond"

* update swagger for merge endpoint

* generate auto swagger docs

* move private below public funcs

* change list first for match

* refactor: fix follows pr review

* tests: fix wrong function

* lint: add alias

* refactor: remove trailing comma

* refactor: var name

* fixes: tests, dialyzer, credo

* remove single pipes

* docs: two examples for transaction.merge

* switch branch for cabbage test

* refactor: move functions out of controller

* test: add outputs to assertion

* test: add tests for `transaction.create/2`

* test: implement tests for include_typed_data/1

* refactor: remove single pipe

* tests: add tests for `select_inputs/2`

* refactor: cleanup typespec

* lint: formatting

* lint: remove unused alias

* lint: reorder the alias

* refactor: move spec and simplified logic

* refactor: simplify pattern matching

* refactor: uniformize param

* revert: bring back merge transaction

* fixes

* refactor: lift out respond

* fix: is_required_merge doesn't work

* refactor: transaction api

* resolve initial conflicts

* tighten specification and refactor tests

* add: dialyzer specs

* refactor: merge constraints

* refactor: accept max. 4 utxo positions

* refactor/add tests

* refactor: merge_constraints limit max. positions to 4

* refactor: replace Alice and Bob fixtures with setup

* refactor: return keyword lists from parameter validation

* update documentation

* post-merge: fix failing tests

* post-merge: fix failing tests

* fix: dialyzer

* fix (lint): remove unused aliases

* post-merge: remove unintended spacing

* add: fallback descriptions

* remove `result` key from body schema for /merge

* reinstate result key where removed.

* rename tests

* fix: duplicate test typo

* remove empty line

Co-authored-by: euro <[email protected]>
Co-authored-by: okalouti <[email protected]>
Co-authored-by: euro <[email protected]>

* add perf cli smoke test (#1762)

* add perf cli smoke test

This PR adds ci step that runs `transactions` perf test

* feat: send web telemetry events to Datadog

Using the telemetry_metric and telemetry_metrics_statsd reporter, we can
send telemetry events to Datadog without defining more event handlers.
The supervised process is configured with the same parameters as the
Statix metrics generation.

In omg.status, we collect web events from Phoenix and send them to
Datadog as metrics.

The fallback controller in WatcherRPC is also instrumented to execute
telemetry events on errors, with route and error type recorded as tags.

chore: mix format

chore: fix linter warnings, shorten module paths

* fix: move env var fetch into Configuration module

* fix: ensure current_route doesn't throw an exception

It's possible in SupportedWatcherModes for a conn to end up in the
fallback controller without being routed there. This change makes it so
that if the route can't be determined from the conn, we return nil
instead of erroring.

Fixes a failing test.

* fix: allow nil release and current_version env vars

* Chore: fix feefeed container healthcheck (#1768)

* fix feefeed container healtheck

Check if feefeed fetched fees instead of checking if it's up

* log docker-compose logs

* feat: add new metadata to Datadog traces

Add service, version and error metadata to traces in Datadog APM traces.

What you see in the traces:
* The service name (default 'web' for Phoenix applications) is the name
of the OMG application.
* Traces are tagged with application version number
* For API responses which error, the trace is flagged red as an error,
the error code is marked on the trace and the error message is found
in the tags.

* refactor: move configuration helpers to new module

Moved from Application to Configuration for WatcherRPC.

* refactor: remove unused app field in Application

* fix: failing tracer and response tests

* fix: aggressive test fix (#1772)

* Collect metrics through datadog for the perf project (#1765)

* retry datadog errors (#1774)

* retry datadog errors

* use header function to fail

* add `retrying` to log messages

* fix: run tests that alter globals consecutively

* refactor: atom to string with interpolation

* ignore known errors - Sentry spike protection (#1775)

* ignore known errors - Sentry spike protection

* run reorg tests only on master

* linting specs is not needed

* Fix readme docker docs (#1773)

* Fix readme docker docs

* change commands

* enter directory

* add perf docs

* move command info to the top

Co-authored-by: Ino Murko <[email protected]>

* config filter sentry (#1778)

* add docker-compose for `test_docker_compose_release` (#1777)

* extract geth command, remove childchain only code paths (#1781)

* extract geth command

* childchain code rem

* update deps, lint

* the great decoupling

* config fix

* config fix, check if Watcher Info config value

* delete childchain test

* config anon functions comments

* ignore RequestTooLarge for Sentry (#1779)

this error is coming from PlugParser when a request is too large. we
don't want this reporting this event to on-call

* set remote_ip from cf-connecting-ip header (#1679)

* set remote_ip from X-Forwarded-For header

* add custom plug that sets remote_ip

* set the right most ip as remote_ip

* add test

* set the left most ip

* use `CF-Connecting-IP` header

* fix tests

Co-authored-by: Arthur Chiu <[email protected]>
Co-authored-by: Ino Murko <[email protected]>

* change type for InFlightExitDeleted

* rename image names and publish for master-v2

Co-authored-by: Omar Kalouti <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Unnawut Leepaisalsuwanna <[email protected]>
Co-authored-by: boolafish <[email protected]>
Co-authored-by: pgebal <[email protected]>
Co-authored-by: jarindr <[email protected]>
Co-authored-by: Ino Murko <[email protected]>
Co-authored-by: Kevin Sullivan <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Mederic <[email protected]>
Co-authored-by: Nicholas Mueller <[email protected]>
Co-authored-by: Michael Captain <[email protected]>
Co-authored-by: Arthur Chiu <[email protected]>
ayrat555 added a commit that referenced this pull request Dec 15, 2020
@ayrat555 @unnawut
@boolafish @pgebal @jarindr @kevsul @achiurizo @nicholasmueller @DanielaIvanova @souradeep-das
Merge updates from master to master-v2 (#1802)
4e4647b 
* /block.validate endpoint (#1668)

* feat: init block.validate

* completes parameter validation tests

* rename validation method with stronger typing

* feat: verify_transactions and and negative test

* add: positive test for verify_transactions

* chore: remove single pipes

* verify_transactions: add @SPEC and @doc

* pending

* test: Added tests for `verify_merkle_root/1`

* remove reverse operation

* refactor: simplify tests

* refactor: simplify verify_merkle_root/1 tests

* refactor: merkle validation tests + entities as module attributes

* add dialyzer spec and remove argument pattern matching

* move logic to dedicated validator module

* test: added test for `block.validate` endpoint

* refactor: make tests pass

* feat: response for `validate_block` view

* feat: error for mismatched_merkle_root

* simplify tests

* rename error

* remove unused method

* fix: error name

* fix: credo

* merge issue: gitmodules

* merge master issue: gitmodules [2]

* merge master issue: gitmodules [3]

* Merge branch 'ripzery/block.validate' of https://github.com/omisego/elixir-omg into ripzery/block.validate

* (re)delete specs

* return submodule

* refactor: decouple verify_merkle_root from transaction recovery

* add: negative test for invalid transactions

* refactor: improve performance of verify_transactions

* use pin operator instead of comparison

* refactor: remove use of & &1. for clearer syntax

* remove redundant comment

* move block validation logic into Watcher

* move parameter validation logic into BlockConstraints

* reflect file changes in controller

* refactor: make verification functions private and shift tests to stateless_validate/1

* fix potential false positive by computing Merkle root correctly.

* add: swagger specs

* refactor: endpoint to return boolean result

* add: nil error

* update documentation to reflect boolean result

* remove error reason in response

Co-authored-by: euro <[email protected]>
Co-authored-by: Ayrat Badykov <[email protected]>

* feat: configurable DB pool size, queue target and queue interval (#1689)

* feat: configurable DB pool size, queue target and queue interval

* fix: add release task to release config

* style: mix format

* feat: configure runtime env vars through releases.exs

* fix: remove release tasks

* fix: syntax

* docs: add docs for db pool env vars

* feat: auto trigger chart version bump (#1695)

* feat: add increase chart version script and job

* feat: filter release tag on CI config

* fix: make increase chart script work

* test: docker image

* feat: add watcher&watcher-info on increase version

* chore: remove commented configs and back to non-playground

* fix: extra space

* Block Validation: New Checks (#1693)

* feat: transaction counting and block-level duplicate input verification in block validation

* fix: dialyzer error

* fix: credo error

* feat: validate fee transactions in block

* test: fee transaction validation

* revert fee logic meant to be stateful

* remove: fee claimer check

* fix: place dummy fee claimer

* test names

* refactors: simplify duplicate input verification, use guards, etc.

* PR review: description changes

* fix: credo

* simplify duplicate input checking

* update documentation

* fix: dialyzer

* spacing

* suggested edits

* feat: block queue metrics and stalled submission alarm (#1649)

* feat: change block formation log from debug to info

* feat: BlockQueue publishes :block_submitting and :block_submitted events

* feat: BlockQueue.Monitor that raises/clears :block_submission_stalled alarm

* refactor: tracks root chain height directly from the bus

* refactor: revert variable naming

* fix: remove remaining EtherereumHeight.get()

* fix: typespec

* feat: add telemetry and statsd metric for blknum_submitting and blknum_submitted

* feat: add telemetry and statsd metric for blocks_submitting and blocks_stalled

* fix: rename :block_submission to :block_submission_gas

* feat: add statsd metric for :block_submission_success

* fix: remove non services

* docs: clearer explanation of block queue metrics

* docs: more explanation on event publishing

* feat: add :block_submission_attempt statsd metric

* fix: dialyzer

* feat: block submission stall config and supervisor setup

* format: formatting and naming alignment

* test: add BlockQueue.Monitor test

* refactor: shorten config name

* test: fix tests

* fix: broken service startup

* feat: configurable block submit stall from env var

* fix: move config namespace

* refactor: brackets everywhere

* fix: config naming

* fix: wrong namespace

* test: fix app name

* test: fix init

* refactor: use :block_submit_stalled everywhere

* fix: ignore unrelated events

* fix: telemetry deprecation

* fix: remove misleading metric :block_queue_num_blocks_submitting due to the way block submission sampling works

* fix: remove dup geth argument

* fix: remove unused block_submit_stall_check_interval_ms

* refactor: add alarm function naming

* refactor: reduce alarm function pattern matching cases

* refactor: move release task to releases.exs

* fix: remove obsolete tests

* feat: handle in-flight exits deletions (#1701)

* move omg_performance json rpc tests to perf project (#1691)

* chore: add feefeed docker to elixir-omg setup (#1700)

* add feefeed adapter

* change ip

* add feefeed url

* fix test barebone release

* fix port

* remove override variable from barebone config

* chore: use https instead of gits to avoid permission issue

* test: update to cabbage using fee api

* fix: git submodule without --remote flag

So it would not always pull the head branch from remote.

* fix: update cabbage to fix reorg

* fix: circle ci mis-comment

* chore: add healthcheck and depends on for services

and mock off other tests aside from reorg

* chore: sync with cabbage update

* chore: add space before network

* chore: ignore geth folder

* debug: comment out make docker steps

* chore: update priv/perf for debug docker-compose pull

* debug: use feefeed-dev with more logging

* debug: use newer feefeed docker that fixes a bug

* chore: update cabbage commit

* chore: uncomment the commented CI jobs

* chore: update latest cabbage

* style: docker file space

* fix: all docker images command have dockerhub authenticate

* chore: update cabbage for code review

* fix: circle ci login to dockerhub

and update cabbage commit

* fix: fee for perf tests

- set in the test config to use the new fee amount 75
- for LoadTest.Common, pass in the fee_amount and use the value from config

* chore: back to feefeed:latest image

* style: mix format

* refactor: move docker_login out from make_docker_images

Logic-wise it does not make much sense actually to be together.

* chore: update cabbage to latest

* style: fix lint

* chore: update cabbage commit

* chore: use master from cabbage

* chore: bring back submodule udpate --remote

This is just better. change the branch in submodele file instead for testing.

Co-authored-by: jarindr <[email protected]>

* feat: reintroduce automated changelog (#1708)

* feat: reintroduce github_changelog_generator

* feat: more changelog tags

* feat: wholly configuring the changelog sections

* feat: changelog label auto checker

* refactor: better review messages

* debug: apply workflow on all branches to check it's working

* fix: bad env format

* fix: wrong job name

* fix: missing closing quote

* fix: bad syntax

* fix: enable shell tracing

* fix: premature failure

* fix: remove extra full stop

* debug: reenable label check only for master

* fix: missing line break escape

* refactor: use Github's default env vars for its url building

* feat: fail the github action instead of relying on a bot

* fix: remove continue-on-error

* fix: combine github action steps so it's easier to find the original error

* feat: echo error

* fix: proper error

* fix: syntax

* fix: force exit on error [ciskip]

* fix: operator precedence

* fix: ignore block_submitting that are above mined blocks (#1703)

* fix: fix in-flight exit deleted bug and add tests (#1714)

* revert: reverts ife deletion commits (#1725)

* Revert "fix: fix in-flight exit deleted bug and add tests (#1714)"

This reverts commit f302553.

* Revert "feat: handle in-flight exits deletions (#1701)"

This reverts commit b3081f8.

* Allow to run docker-compose without feefeed docker (#1726)

* refactor: abstract steps in circle ci to make file for perf

* chore: seperate docker-compose file for feefeed

to allow public contributor to run without feefeed docker

* fix: make start-services command with feefeed

* refactor: move mix format for api generated code back to CI

* debug: add debug CI workflow

* style: simplify docker-compose file

* debug: add log service command to perf

* do not set env var in makefile

* back to run whole workflow

* refactor: COMPOSE_FULL_SERVICES in makefile

* chore: spaces instead of tab in .gitmodules

* add test for 64_000 txs block hash (#1729)

* rm mix based chch (#1716)

machine

module rename

dont cache

install linux deps

sleep

postgres

child chain url and childchain empty fees

* Inomurko/remove child chain (#1737)

* rem childchain code

* erlexec

* fixtures inclusion

* revert cabbage and api client generation from omg v1 childchain

* we dont need wget anymore

* feat: handle in-flight exits deletions

* move dev env deployment job to helm repo (#1738)

* feat: trigger increase chart version on merge to master

* debug: switch to playground repo

And allow circle ci to run on non-master

* fix: unbound string

* feat: add jitter to the call

* remove jitter, not useful

* chore: cleanup

Uncomment CircleCI jobs, remove deployment jobs/scripts, not using playground repo

* fix: circle ci format

* fix: notifiy service do not wait for non-existing deployment job

* chore: remove childchain master deployment

Responsibility has been moved to omg-childhchain-v1 repo.
Have another PR for that: omgnetwork/omg-childchain-v1#15

* chore: remove childchain release from elixir-omg

Moved to omg-childchain-v1.

* fix: handle metrics for in flight exit deleted processor (#1742)

* Kevsul/standard exit perf test (#1732)

* tests: add load test for standard exits

* tests: add watcher status scenario

* tests: standard exit test syncs with watcher

* tests: revert test fee_amount

* tests: add max num retries when waiting

* tests: remove FundAccountRootchain scenario

* tests: change if to case

* Chore: try to fix flaky reorg tests (#1739)

Two kinds of errors I saw:
1. Tests are timing out during account creation and unlocking on both geth nodes. After I increased timeout for account creation, I haven't seen this error anymore
2. Tests are failing with wrong balances (on the childchain and the rootchain). I added additional checks to the reorg function to make sure nodes have the same latest block after reorg. I think it helped but still, tests are failing sometimes.

* update change log v1.0.4 (#1731)

* update changelog

* manual fix change log

* more manual fix change log

* Update VERSION

Co-authored-by: Unnawut Leepaisalsuwanna <[email protected]>

* Chore: parallelize tests by tags (#1744)

* parallelize tests by tags

* Chore: use exexec from upstream (#1743)

* use exexec from upstream

* fix: recheck PR label on synchronize and reopen (#1748)

* fix: recheck PR label on synchronize and reopen

* test: a pointless commit to check that the enforce-changelog-label job still passes

* debug: try the fixed feefeed

* Revert "debug: try the fixed feefeed"

This reverts commit a35fc90.

* feat: add deposits performance tests bot (#1745)

* feat: transaction.create optimisation (#1683)

* test: Init

* test: create `generate_utxos_map/3`

* test: put test in describe

* test: added `needed_funds/2` tests

* add tests and light refactor for get_sorted_grouped_utxos

* add tests and light refactor [2]

* test: refactor `create_advice/2`

* add spec and clarify funds_sufficient

* refactor: `needed_funds`

* refactor: use shorthanded type

* Add @doc  and readabilty edits

* add: tests for funds_sufficient/1

* refactor: simplify use of 'alice'

* refactor: move test inside describe

* test: add `select_utxo/2` tests

* refactor: remove extra line

* fix: spec

* feat: implement `add_merge_utxos`

* test: add test for `add_merge_utxos/2`

* feat: add prioritize_merge_utxos/2

* test: add test for `prioritize_merge_utxos/2`

* refactor: pipe one more time

* refactor and spec stealth merge

* refactor: test name

* add: tests and doc for `add_utxos_for_stealth_merge`

* feat: integrate add_merge_utxos to create_advice

* test: add test for create_advice

* add: doc content for add_utxos_for_stealth_merge

* refactor and test prioritisation of UTXOs for stealth merge

* refactor: split transaction operation

* Fix dialyzer type

* refactor: dialyzer

* fix: typo

* test: add utxo_selection test

* debug: transaction create

* feat: handle when original inputs are empty

* tests: remove transaction merge tests

* tests: add tests for utxo selection

* tests: handle when too many inputs to satisfy payments and fee

* refactor: remove todo

* feat: move `create_advice` from `utxo_selection.ex` to  `transaction.ex` and rename it to `create_inputs`

* test: add transaction tests

* test: change port `9657` to `9656`

* cleanup: remove unused var

* lint: remove @doc for defp

* test: revert back port `9656` to `9657`

* lint: fix lint

* refactor: remove result from response

* test: add prepare_test_server

* refactor: fix per reviews

* tests: add test case where 2 input use different currencies

* refactor: remove alice, bob from test context

* test: fix mismatch function signature

* fix: dialyzer type

* refactor: Cleanup

* fix: dialyzer

* fix: credo

* remove unusued alias

* refactor: remove comments

* lint: fix

* tests: add tests for transaction.create apis

* test: delete unnecessary function

* refactor: simplified `select_inputs` flow

* refactor: fix follows pr review

* tests: fix wrong function

* lint: add alias

* refactor: remove trailing comma

* refactor: var name

* switch branch for cabbage test

* refactor: move functions out of controller

* test: add outputs to assertion

* test: add tests for `transaction.create/2`

* test: implement tests for include_typed_data/1

* refactor: remove single pipe

* tests: add tests for `select_inputs/2`

* refactor: cleanup typespec

* lint: formatting

* lint: remove unused alias

* lint: reorder the alias

* refactor: move spec and simplified logic

* refactor: simplify pattern matching

* refactor: uniformize param

* revert: bring back merge transaction

* fixes

* refactor: lift out respond

* fix: is_required_merge doesn't work

* refactor: transaction api

* test: fix tests

* tests: fix

* fix: add result key back

* fix: dialyzer part-1

* fix: remove unnecessary dialyzer types

* fix: revert changes on .gitmodules

* refactor: select_inputs/2

* refactor: simplify select_utxo logic

* refactor: re-order functions

* fix: rename function

* fix: conflict

* Fix conflict

* refactor: rename functions

* refactor: remove single pipe

* fix: lint

* fix: tests

* fix: tests 2

* try: update spec repo

* test: change git modules

* test: revert submodule

* test: fix cabbage conflict

* test: add result key back

* fix: private and public function mixing

* test: fix imports and mixing private functions

* fix: break down pipe

* refactor: break down pipe

* fix: remove blank line

* fix: move alias to top

* refactor: renaming and break out pipe

* refactor: get_utxos_count

* refactor: create_transaction

* fix: typo

* refactor: build_inputs

* refactor: formatting

* refactor: lift out input

* refactor: prioritize merge utxos

* remove and ignore priv/apps/childchain_api

Co-authored-by: euro <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Mederic <[email protected]>

* chore: bump version to 1.0.4 (#1751)

We are already in 1.0.4 but we forget to port the VERSION file back when updating changelog.

* feat: comply with new Infura API (#1754)

* chore: changelog/version bump for v1.0.5 (#1747)

* chore: changelog/version bump v1.0.5

* update CHANGELOG for 1.0.5-pre.0

* bump to include latest changes for infura

* Publish the perf project (#1753)

* publish the perf project

* Utxo perf test (#1756)

* create and check utxo

* spend utxo

* assert balances after spending

* validate exact utxos

* fix token encoding

* add test to run on ci

* Update priv/perf/apps/load_test/lib/watcher_info/utxo.ex

Co-authored-by: Ino Murko <[email protected]>

* Update priv/perf/apps/load_test/lib/watcher_info/utxo.ex

Co-authored-by: Ino Murko <[email protected]>

* mix format

* fix build

Co-authored-by: Ino Murko <[email protected]>

* Add cli help command for the perf project (#1757)


* add cli help command for the perf project

* feat : transaction.merge endpoint  (#1715)

* test: Init

* test: create `generate_utxos_map/3`

* test: put test in describe

* test: added `needed_funds/2` tests

* add tests and light refactor for get_sorted_grouped_utxos

* add tests and light refactor [2]

* test: refactor `create_advice/2`

* add spec and clarify funds_sufficient

* refactor: `needed_funds`

* refactor: use shorthanded type

* Add @doc  and readabilty edits

* add: tests for funds_sufficient/1

* refactor: simplify use of 'alice'

* refactor: move test inside describe

* test: add `select_utxo/2` tests

* refactor: remove extra line

* fix: spec

* feat: implement `add_merge_utxos`

* test: add test for `add_merge_utxos/2`

* feat: add prioritize_merge_utxos/2

* test: add test for `prioritize_merge_utxos/2`

* refactor: pipe one more time

* refactor and spec stealth merge

* refactor: test name

* add min_length to base validators

* add merge route and controller

* fix compile errors

* add: tests and doc for `add_utxos_for_stealth_merge`

* test address/currency and utxo positions constraint

* feat: integrate add_merge_utxos to create_advice

* test: add test for create_advice

* add: doc content for add_utxos_for_stealth_merge

* refactor and test prioritisation of UTXOs for stealth merge

* test utxo constraint

* add other required utxo data

* fetch utxo logic

* start transaction_test

* finaly get basic transaction api test working

* merge p

* refactor: split transaction operation

* Fix dialyzer type

* refactor: dialyzer

* fix: typo

* test: add utxo_selection test

* debug: transaction create

* feat: handle when original inputs are empty

* tests: remove transaction merge tests

* tests: add tests for utxo selection

* tests: handle when too many inputs to satisfy payments and fee

* refactor: remove todo

* refactor: parameter validation for transaction.merge

* optional desc/asc for UTXO fetch

* WIP: merge functionality

* feat: move `create_advice` from `utxo_selection.ex` to  `transaction.ex` and rename it to `create_inputs`

* test: add transaction tests

* WIP: return multiple merge transactions if address/ccy params

* test: change port `9657` to `9656`

* cleanup: remove unused var

* lint: remove @doc for defp

* test: revert back port `9656` to `9657`

* lint: fix lint

* refactor: remove result from response

* test: add prepare_test_server

* WIP: merge when given utxo positions

* refactor: fix per reviews

* tests: add test case where 2 input use different currencies

* refactor: remove alice, bob from test context

* test: fix mismatch function signature

* fix: dialyzer type

* test for merge with address and currency, refactor create merge so no need for currency and address passing

* more happy path tests

* refactor: Cleanup

* fix: dialyzer

* fix: credo

* remove unusued alias

* remove unaccepted merge constraints

* remove max 4 merge constratint

* match tx.create response shape

* refactor: remove comments

* lint: fix

* tests: add tests for transaction.create apis

* test: delete unnecessary function

* address/currency merge controller tests

* utxo position happy path

* add multiple tx for utxo pos test

* refactor: simplified `select_inputs` flow

* start merge view test

* revert merge view since controler test makes this redundant

* remove default argument in get_sorted_grouped_utxos and reflect change

* refactor: remove "respond"

* update swagger for merge endpoint

* generate auto swagger docs

* move private below public funcs

* change list first for match

* refactor: fix follows pr review

* tests: fix wrong function

* lint: add alias

* refactor: remove trailing comma

* refactor: var name

* fixes: tests, dialyzer, credo

* remove single pipes

* docs: two examples for transaction.merge

* switch branch for cabbage test

* refactor: move functions out of controller

* test: add outputs to assertion

* test: add tests for `transaction.create/2`

* test: implement tests for include_typed_data/1

* refactor: remove single pipe

* tests: add tests for `select_inputs/2`

* refactor: cleanup typespec

* lint: formatting

* lint: remove unused alias

* lint: reorder the alias

* refactor: move spec and simplified logic

* refactor: simplify pattern matching

* refactor: uniformize param

* revert: bring back merge transaction

* fixes

* refactor: lift out respond

* fix: is_required_merge doesn't work

* refactor: transaction api

* resolve initial conflicts

* tighten specification and refactor tests

* add: dialyzer specs

* refactor: merge constraints

* refactor: accept max. 4 utxo positions

* refactor/add tests

* refactor: merge_constraints limit max. positions to 4

* refactor: replace Alice and Bob fixtures with setup

* refactor: return keyword lists from parameter validation

* update documentation

* post-merge: fix failing tests

* post-merge: fix failing tests

* fix: dialyzer

* fix (lint): remove unused aliases

* post-merge: remove unintended spacing

* add: fallback descriptions

* remove `result` key from body schema for /merge

* reinstate result key where removed.

* rename tests

* fix: duplicate test typo

* remove empty line

Co-authored-by: euro <[email protected]>
Co-authored-by: okalouti <[email protected]>
Co-authored-by: euro <[email protected]>

* add perf cli smoke test (#1762)

* add perf cli smoke test

This PR adds ci step that runs `transactions` perf test

* feat: send web telemetry events to Datadog

Using the telemetry_metric and telemetry_metrics_statsd reporter, we can
send telemetry events to Datadog without defining more event handlers.
The supervised process is configured with the same parameters as the
Statix metrics generation.

In omg.status, we collect web events from Phoenix and send them to
Datadog as metrics.

The fallback controller in WatcherRPC is also instrumented to execute
telemetry events on errors, with route and error type recorded as tags.

chore: mix format

chore: fix linter warnings, shorten module paths

* fix: move env var fetch into Configuration module

* fix: ensure current_route doesn't throw an exception

It's possible in SupportedWatcherModes for a conn to end up in the
fallback controller without being routed there. This change makes it so
that if the route can't be determined from the conn, we return nil
instead of erroring.

Fixes a failing test.

* fix: allow nil release and current_version env vars

* Chore: fix feefeed container healthcheck (#1768)

* fix feefeed container healtheck

Check if feefeed fetched fees instead of checking if it's up

* log docker-compose logs

* feat: add new metadata to Datadog traces

Add service, version and error metadata to traces in Datadog APM traces.

What you see in the traces:
* The service name (default 'web' for Phoenix applications) is the name
of the OMG application.
* Traces are tagged with application version number
* For API responses which error, the trace is flagged red as an error,
the error code is marked on the trace and the error message is found
in the tags.

* refactor: move configuration helpers to new module

Moved from Application to Configuration for WatcherRPC.

* refactor: remove unused app field in Application

* fix: failing tracer and response tests

* fix: aggressive test fix (#1772)

* Collect metrics through datadog for the perf project (#1765)

* retry datadog errors (#1774)

* retry datadog errors

* use header function to fail

* add `retrying` to log messages

* fix: run tests that alter globals consecutively

* refactor: atom to string with interpolation

* ignore known errors - Sentry spike protection (#1775)

* ignore known errors - Sentry spike protection

* run reorg tests only on master

* linting specs is not needed

* Fix readme docker docs (#1773)

* Fix readme docker docs

* change commands

* enter directory

* add perf docs

* move command info to the top

Co-authored-by: Ino Murko <[email protected]>

* config filter sentry (#1778)

* add docker-compose for `test_docker_compose_release` (#1777)

* extract geth command, remove childchain only code paths (#1781)

* extract geth command

* childchain code rem

* update deps, lint

* the great decoupling

* config fix

* config fix, check if Watcher Info config value

* delete childchain test

* config anon functions comments

* #1588 phx exception caused by invalid accept header is now ignored

* #1588 added a test

* ignore RequestTooLarge for Sentry (#1779)

this error is coming from PlugParser when a request is too large. we
don't want this reporting this event to on-call

* set remote_ip from cf-connecting-ip header (#1679)

* set remote_ip from X-Forwarded-For header

* add custom plug that sets remote_ip

* set the right most ip as remote_ip

* add test

* set the left most ip

* use `CF-Connecting-IP` header

* fix tests

Co-authored-by: Arthur Chiu <[email protected]>
Co-authored-by: Ino Murko <[email protected]>

* #1588 minor adjustmen

* feat: add X-Watcher-Version header to http requests (#1767)

* #1588 minor fix

* fix: set service name as a tag in metrics

Currently the service name is part of the prefix for metrics, but it
would be far better if it were a tag, so that services could be compared
side by side in Datadog.

* fix: limit matching for ethereumex responses (#1782)


* fix: rem error response handling

* fix: add sentry filter

Co-authored-by: Ino Murko <[email protected]>

* just the one thing to turn into a warn (#1784)

Co-authored-by: Ino Murko <[email protected]>

* Renamed all occurrences of omisego to omgnetwork (#1788)

* #1555 Renamed omisego to omg network

* #1555 Fixed urls

* #1555 downgraded rocksdb to 1.5.1

* chore: changelog/version bump for v1.0.5 (#1747) (#1785)

* chore: changelog/version bump v1.0.5

* update CHANGELOG for 1.0.5-pre.0

* bump to include latest changes for infura

Co-authored-by: Arthur Chiu <[email protected]>
Co-authored-by: boolafish <[email protected]>

* chore: add sem release automation (#1791)

* chore: add sem release

* chore: set release rule for our standard

* chore: test dummy

* fix: sem-release format

* chore: use dry-run on sem release

* fix: back to origin config

* bump elixir (#1789)

* bump elixir

* align rust version to 1.46 (#1793)

* align rust version to 1.46

* trace reorg tests (#1796)

* fix: make sem release ignore PR check (#1797)

* chore: use git tag version instead (#1798)

* chore: use git tag version instead

* fix: for lint

* style: pipe :)

* refactor: put to private function for version()

* update submodule

* update mix file

* fix mix files

* update deps

* fix eip55

* use new ex_plasma

* fix ethereum event aggregator test

* fix dialyzer

Co-authored-by: Omar Kalouti <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Unnawut Leepaisalsuwanna <[email protected]>
Co-authored-by: boolafish <[email protected]>
Co-authored-by: pgebal <[email protected]>
Co-authored-by: jarindr <[email protected]>
Co-authored-by: Ino Murko <[email protected]>
Co-authored-by: Kevin Sullivan <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Mederic <[email protected]>
Co-authored-by: Arthur Chiu <[email protected]>
Co-authored-by: Nicholas Mueller <[email protected]>
Co-authored-by: Michael Captain <[email protected]>
Co-authored-by: Daniela Ivanova <[email protected]>
Co-authored-by: Souradeep Das <[email protected]>
InoMurko added a commit that referenced this pull request Mar 12, 2021
@InoMurko @ayrat555
@unnawut @boolafish @pgebal @jarindr @kevsul @achiurizo @nicholasmueller @DanielaIvanova @souradeep-das @pnowosie
Inomurko/merge unspent output bugfix to master v2 (#1823)
88c8fd0 
* /block.validate endpoint (#1668)

* feat: init block.validate

* completes parameter validation tests

* rename validation method with stronger typing

* feat: verify_transactions and and negative test

* add: positive test for verify_transactions

* chore: remove single pipes

* verify_transactions: add @SPEC and @doc

* pending

* test: Added tests for `verify_merkle_root/1`

* remove reverse operation

* refactor: simplify tests

* refactor: simplify verify_merkle_root/1 tests

* refactor: merkle validation tests + entities as module attributes

* add dialyzer spec and remove argument pattern matching

* move logic to dedicated validator module

* test: added test for `block.validate` endpoint

* refactor: make tests pass

* feat: response for `validate_block` view

* feat: error for mismatched_merkle_root

* simplify tests

* rename error

* remove unused method

* fix: error name

* fix: credo

* merge issue: gitmodules

* merge master issue: gitmodules [2]

* merge master issue: gitmodules [3]

* Merge branch 'ripzery/block.validate' of https://github.com/omisego/elixir-omg into ripzery/block.validate

* (re)delete specs

* return submodule

* refactor: decouple verify_merkle_root from transaction recovery

* add: negative test for invalid transactions

* refactor: improve performance of verify_transactions

* use pin operator instead of comparison

* refactor: remove use of & &1. for clearer syntax

* remove redundant comment

* move block validation logic into Watcher

* move parameter validation logic into BlockConstraints

* reflect file changes in controller

* refactor: make verification functions private and shift tests to stateless_validate/1

* fix potential false positive by computing Merkle root correctly.

* add: swagger specs

* refactor: endpoint to return boolean result

* add: nil error

* update documentation to reflect boolean result

* remove error reason in response

Co-authored-by: euro <[email protected]>
Co-authored-by: Ayrat Badykov <[email protected]>

* feat: configurable DB pool size, queue target and queue interval (#1689)

* feat: configurable DB pool size, queue target and queue interval

* fix: add release task to release config

* style: mix format

* feat: configure runtime env vars through releases.exs

* fix: remove release tasks

* fix: syntax

* docs: add docs for db pool env vars

* feat: auto trigger chart version bump (#1695)

* feat: add increase chart version script and job

* feat: filter release tag on CI config

* fix: make increase chart script work

* test: docker image

* feat: add watcher&watcher-info on increase version

* chore: remove commented configs and back to non-playground

* fix: extra space

* Block Validation: New Checks (#1693)

* feat: transaction counting and block-level duplicate input verification in block validation

* fix: dialyzer error

* fix: credo error

* feat: validate fee transactions in block

* test: fee transaction validation

* revert fee logic meant to be stateful

* remove: fee claimer check

* fix: place dummy fee claimer

* test names

* refactors: simplify duplicate input verification, use guards, etc.

* PR review: description changes

* fix: credo

* simplify duplicate input checking

* update documentation

* fix: dialyzer

* spacing

* suggested edits

* feat: block queue metrics and stalled submission alarm (#1649)

* feat: change block formation log from debug to info

* feat: BlockQueue publishes :block_submitting and :block_submitted events

* feat: BlockQueue.Monitor that raises/clears :block_submission_stalled alarm

* refactor: tracks root chain height directly from the bus

* refactor: revert variable naming

* fix: remove remaining EtherereumHeight.get()

* fix: typespec

* feat: add telemetry and statsd metric for blknum_submitting and blknum_submitted

* feat: add telemetry and statsd metric for blocks_submitting and blocks_stalled

* fix: rename :block_submission to :block_submission_gas

* feat: add statsd metric for :block_submission_success

* fix: remove non services

* docs: clearer explanation of block queue metrics

* docs: more explanation on event publishing

* feat: add :block_submission_attempt statsd metric

* fix: dialyzer

* feat: block submission stall config and supervisor setup

* format: formatting and naming alignment

* test: add BlockQueue.Monitor test

* refactor: shorten config name

* test: fix tests

* fix: broken service startup

* feat: configurable block submit stall from env var

* fix: move config namespace

* refactor: brackets everywhere

* fix: config naming

* fix: wrong namespace

* test: fix app name

* test: fix init

* refactor: use :block_submit_stalled everywhere

* fix: ignore unrelated events

* fix: telemetry deprecation

* fix: remove misleading metric :block_queue_num_blocks_submitting due to the way block submission sampling works

* fix: remove dup geth argument

* fix: remove unused block_submit_stall_check_interval_ms

* refactor: add alarm function naming

* refactor: reduce alarm function pattern matching cases

* refactor: move release task to releases.exs

* fix: remove obsolete tests

* feat: handle in-flight exits deletions (#1701)

* move omg_performance json rpc tests to perf project (#1691)

* chore: add feefeed docker to elixir-omg setup (#1700)

* add feefeed adapter

* change ip

* add feefeed url

* fix test barebone release

* fix port

* remove override variable from barebone config

* chore: use https instead of gits to avoid permission issue

* test: update to cabbage using fee api

* fix: git submodule without --remote flag

So it would not always pull the head branch from remote.

* fix: update cabbage to fix reorg

* fix: circle ci mis-comment

* chore: add healthcheck and depends on for services

and mock off other tests aside from reorg

* chore: sync with cabbage update

* chore: add space before network

* chore: ignore geth folder

* debug: comment out make docker steps

* chore: update priv/perf for debug docker-compose pull

* debug: use feefeed-dev with more logging

* debug: use newer feefeed docker that fixes a bug

* chore: update cabbage commit

* chore: uncomment the commented CI jobs

* chore: update latest cabbage

* style: docker file space

* fix: all docker images command have dockerhub authenticate

* chore: update cabbage for code review

* fix: circle ci login to dockerhub

and update cabbage commit

* fix: fee for perf tests

- set in the test config to use the new fee amount 75
- for LoadTest.Common, pass in the fee_amount and use the value from config

* chore: back to feefeed:latest image

* style: mix format

* refactor: move docker_login out from make_docker_images

Logic-wise it does not make much sense actually to be together.

* chore: update cabbage to latest

* style: fix lint

* chore: update cabbage commit

* chore: use master from cabbage

* chore: bring back submodule udpate --remote

This is just better. change the branch in submodele file instead for testing.

Co-authored-by: jarindr <[email protected]>

* feat: reintroduce automated changelog (#1708)

* feat: reintroduce github_changelog_generator

* feat: more changelog tags

* feat: wholly configuring the changelog sections

* feat: changelog label auto checker

* refactor: better review messages

* debug: apply workflow on all branches to check it's working

* fix: bad env format

* fix: wrong job name

* fix: missing closing quote

* fix: bad syntax

* fix: enable shell tracing

* fix: premature failure

* fix: remove extra full stop

* debug: reenable label check only for master

* fix: missing line break escape

* refactor: use Github's default env vars for its url building

* feat: fail the github action instead of relying on a bot

* fix: remove continue-on-error

* fix: combine github action steps so it's easier to find the original error

* feat: echo error

* fix: proper error

* fix: syntax

* fix: force exit on error [ciskip]

* fix: operator precedence

* fix: ignore block_submitting that are above mined blocks (#1703)

* fix: fix in-flight exit deleted bug and add tests (#1714)

* revert: reverts ife deletion commits (#1725)

* Revert "fix: fix in-flight exit deleted bug and add tests (#1714)"

This reverts commit f302553.

* Revert "feat: handle in-flight exits deletions (#1701)"

This reverts commit b3081f8.

* Allow to run docker-compose without feefeed docker (#1726)

* refactor: abstract steps in circle ci to make file for perf

* chore: seperate docker-compose file for feefeed

to allow public contributor to run without feefeed docker

* fix: make start-services command with feefeed

* refactor: move mix format for api generated code back to CI

* debug: add debug CI workflow

* style: simplify docker-compose file

* debug: add log service command to perf

* do not set env var in makefile

* back to run whole workflow

* refactor: COMPOSE_FULL_SERVICES in makefile

* chore: spaces instead of tab in .gitmodules

* add test for 64_000 txs block hash (#1729)

* rm mix based chch (#1716)

machine

module rename

dont cache

install linux deps

sleep

postgres

child chain url and childchain empty fees

* Inomurko/remove child chain (#1737)

* rem childchain code

* erlexec

* fixtures inclusion

* revert cabbage and api client generation from omg v1 childchain

* we dont need wget anymore

* feat: handle in-flight exits deletions

* move dev env deployment job to helm repo (#1738)

* feat: trigger increase chart version on merge to master

* debug: switch to playground repo

And allow circle ci to run on non-master

* fix: unbound string

* feat: add jitter to the call

* remove jitter, not useful

* chore: cleanup

Uncomment CircleCI jobs, remove deployment jobs/scripts, not using playground repo

* fix: circle ci format

* fix: notifiy service do not wait for non-existing deployment job

* chore: remove childchain master deployment

Responsibility has been moved to omg-childhchain-v1 repo.
Have another PR for that: omgnetwork/omg-childchain-v1#15

* chore: remove childchain release from elixir-omg

Moved to omg-childchain-v1.

* fix: handle metrics for in flight exit deleted processor (#1742)

* Kevsul/standard exit perf test (#1732)

* tests: add load test for standard exits

* tests: add watcher status scenario

* tests: standard exit test syncs with watcher

* tests: revert test fee_amount

* tests: add max num retries when waiting

* tests: remove FundAccountRootchain scenario

* tests: change if to case

* Chore: try to fix flaky reorg tests (#1739)

Two kinds of errors I saw:
1. Tests are timing out during account creation and unlocking on both geth nodes. After I increased timeout for account creation, I haven't seen this error anymore
2. Tests are failing with wrong balances (on the childchain and the rootchain). I added additional checks to the reorg function to make sure nodes have the same latest block after reorg. I think it helped but still, tests are failing sometimes.

* update change log v1.0.4 (#1731)

* update changelog

* manual fix change log

* more manual fix change log

* Update VERSION

Co-authored-by: Unnawut Leepaisalsuwanna <[email protected]>

* Chore: parallelize tests by tags (#1744)

* parallelize tests by tags

* Chore: use exexec from upstream (#1743)

* use exexec from upstream

* fix: recheck PR label on synchronize and reopen (#1748)

* fix: recheck PR label on synchronize and reopen

* test: a pointless commit to check that the enforce-changelog-label job still passes

* debug: try the fixed feefeed

* Revert "debug: try the fixed feefeed"

This reverts commit a35fc90.

* feat: add deposits performance tests bot (#1745)

* feat: transaction.create optimisation (#1683)

* test: Init

* test: create `generate_utxos_map/3`

* test: put test in describe

* test: added `needed_funds/2` tests

* add tests and light refactor for get_sorted_grouped_utxos

* add tests and light refactor [2]

* test: refactor `create_advice/2`

* add spec and clarify funds_sufficient

* refactor: `needed_funds`

* refactor: use shorthanded type

* Add @doc  and readabilty edits

* add: tests for funds_sufficient/1

* refactor: simplify use of 'alice'

* refactor: move test inside describe

* test: add `select_utxo/2` tests

* refactor: remove extra line

* fix: spec

* feat: implement `add_merge_utxos`

* test: add test for `add_merge_utxos/2`

* feat: add prioritize_merge_utxos/2

* test: add test for `prioritize_merge_utxos/2`

* refactor: pipe one more time

* refactor and spec stealth merge

* refactor: test name

* add: tests and doc for `add_utxos_for_stealth_merge`

* feat: integrate add_merge_utxos to create_advice

* test: add test for create_advice

* add: doc content for add_utxos_for_stealth_merge

* refactor and test prioritisation of UTXOs for stealth merge

* refactor: split transaction operation

* Fix dialyzer type

* refactor: dialyzer

* fix: typo

* test: add utxo_selection test

* debug: transaction create

* feat: handle when original inputs are empty

* tests: remove transaction merge tests

* tests: add tests for utxo selection

* tests: handle when too many inputs to satisfy payments and fee

* refactor: remove todo

* feat: move `create_advice` from `utxo_selection.ex` to  `transaction.ex` and rename it to `create_inputs`

* test: add transaction tests

* test: change port `9657` to `9656`

* cleanup: remove unused var

* lint: remove @doc for defp

* test: revert back port `9656` to `9657`

* lint: fix lint

* refactor: remove result from response

* test: add prepare_test_server

* refactor: fix per reviews

* tests: add test case where 2 input use different currencies

* refactor: remove alice, bob from test context

* test: fix mismatch function signature

* fix: dialyzer type

* refactor: Cleanup

* fix: dialyzer

* fix: credo

* remove unusued alias

* refactor: remove comments

* lint: fix

* tests: add tests for transaction.create apis

* test: delete unnecessary function

* refactor: simplified `select_inputs` flow

* refactor: fix follows pr review

* tests: fix wrong function

* lint: add alias

* refactor: remove trailing comma

* refactor: var name

* switch branch for cabbage test

* refactor: move functions out of controller

* test: add outputs to assertion

* test: add tests for `transaction.create/2`

* test: implement tests for include_typed_data/1

* refactor: remove single pipe

* tests: add tests for `select_inputs/2`

* refactor: cleanup typespec

* lint: formatting

* lint: remove unused alias

* lint: reorder the alias

* refactor: move spec and simplified logic

* refactor: simplify pattern matching

* refactor: uniformize param

* revert: bring back merge transaction

* fixes

* refactor: lift out respond

* fix: is_required_merge doesn't work

* refactor: transaction api

* test: fix tests

* tests: fix

* fix: add result key back

* fix: dialyzer part-1

* fix: remove unnecessary dialyzer types

* fix: revert changes on .gitmodules

* refactor: select_inputs/2

* refactor: simplify select_utxo logic

* refactor: re-order functions

* fix: rename function

* fix: conflict

* Fix conflict

* refactor: rename functions

* refactor: remove single pipe

* fix: lint

* fix: tests

* fix: tests 2

* try: update spec repo

* test: change git modules

* test: revert submodule

* test: fix cabbage conflict

* test: add result key back

* fix: private and public function mixing

* test: fix imports and mixing private functions

* fix: break down pipe

* refactor: break down pipe

* fix: remove blank line

* fix: move alias to top

* refactor: renaming and break out pipe

* refactor: get_utxos_count

* refactor: create_transaction

* fix: typo

* refactor: build_inputs

* refactor: formatting

* refactor: lift out input

* refactor: prioritize merge utxos

* remove and ignore priv/apps/childchain_api

Co-authored-by: euro <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Mederic <[email protected]>

* chore: bump version to 1.0.4 (#1751)

We are already in 1.0.4 but we forget to port the VERSION file back when updating changelog.

* feat: comply with new Infura API (#1754)

* chore: changelog/version bump for v1.0.5 (#1747)

* chore: changelog/version bump v1.0.5

* update CHANGELOG for 1.0.5-pre.0

* bump to include latest changes for infura

* Publish the perf project (#1753)

* publish the perf project

* Utxo perf test (#1756)

* create and check utxo

* spend utxo

* assert balances after spending

* validate exact utxos

* fix token encoding

* add test to run on ci

* Update priv/perf/apps/load_test/lib/watcher_info/utxo.ex

Co-authored-by: Ino Murko <[email protected]>

* Update priv/perf/apps/load_test/lib/watcher_info/utxo.ex

Co-authored-by: Ino Murko <[email protected]>

* mix format

* fix build

Co-authored-by: Ino Murko <[email protected]>

* Add cli help command for the perf project (#1757)


* add cli help command for the perf project

* feat : transaction.merge endpoint  (#1715)

* test: Init

* test: create `generate_utxos_map/3`

* test: put test in describe

* test: added `needed_funds/2` tests

* add tests and light refactor for get_sorted_grouped_utxos

* add tests and light refactor [2]

* test: refactor `create_advice/2`

* add spec and clarify funds_sufficient

* refactor: `needed_funds`

* refactor: use shorthanded type

* Add @doc  and readabilty edits

* add: tests for funds_sufficient/1

* refactor: simplify use of 'alice'

* refactor: move test inside describe

* test: add `select_utxo/2` tests

* refactor: remove extra line

* fix: spec

* feat: implement `add_merge_utxos`

* test: add test for `add_merge_utxos/2`

* feat: add prioritize_merge_utxos/2

* test: add test for `prioritize_merge_utxos/2`

* refactor: pipe one more time

* refactor and spec stealth merge

* refactor: test name

* add min_length to base validators

* add merge route and controller

* fix compile errors

* add: tests and doc for `add_utxos_for_stealth_merge`

* test address/currency and utxo positions constraint

* feat: integrate add_merge_utxos to create_advice

* test: add test for create_advice

* add: doc content for add_utxos_for_stealth_merge

* refactor and test prioritisation of UTXOs for stealth merge

* test utxo constraint

* add other required utxo data

* fetch utxo logic

* start transaction_test

* finaly get basic transaction api test working

* merge p

* refactor: split transaction operation

* Fix dialyzer type

* refactor: dialyzer

* fix: typo

* test: add utxo_selection test

* debug: transaction create

* feat: handle when original inputs are empty

* tests: remove transaction merge tests

* tests: add tests for utxo selection

* tests: handle when too many inputs to satisfy payments and fee

* refactor: remove todo

* refactor: parameter validation for transaction.merge

* optional desc/asc for UTXO fetch

* WIP: merge functionality

* feat: move `create_advice` from `utxo_selection.ex` to  `transaction.ex` and rename it to `create_inputs`

* test: add transaction tests

* WIP: return multiple merge transactions if address/ccy params

* test: change port `9657` to `9656`

* cleanup: remove unused var

* lint: remove @doc for defp

* test: revert back port `9656` to `9657`

* lint: fix lint

* refactor: remove result from response

* test: add prepare_test_server

* WIP: merge when given utxo positions

* refactor: fix per reviews

* tests: add test case where 2 input use different currencies

* refactor: remove alice, bob from test context

* test: fix mismatch function signature

* fix: dialyzer type

* test for merge with address and currency, refactor create merge so no need for currency and address passing

* more happy path tests

* refactor: Cleanup

* fix: dialyzer

* fix: credo

* remove unusued alias

* remove unaccepted merge constraints

* remove max 4 merge constratint

* match tx.create response shape

* refactor: remove comments

* lint: fix

* tests: add tests for transaction.create apis

* test: delete unnecessary function

* address/currency merge controller tests

* utxo position happy path

* add multiple tx for utxo pos test

* refactor: simplified `select_inputs` flow

* start merge view test

* revert merge view since controler test makes this redundant

* remove default argument in get_sorted_grouped_utxos and reflect change

* refactor: remove "respond"

* update swagger for merge endpoint

* generate auto swagger docs

* move private below public funcs

* change list first for match

* refactor: fix follows pr review

* tests: fix wrong function

* lint: add alias

* refactor: remove trailing comma

* refactor: var name

* fixes: tests, dialyzer, credo

* remove single pipes

* docs: two examples for transaction.merge

* switch branch for cabbage test

* refactor: move functions out of controller

* test: add outputs to assertion

* test: add tests for `transaction.create/2`

* test: implement tests for include_typed_data/1

* refactor: remove single pipe

* tests: add tests for `select_inputs/2`

* refactor: cleanup typespec

* lint: formatting

* lint: remove unused alias

* lint: reorder the alias

* refactor: move spec and simplified logic

* refactor: simplify pattern matching

* refactor: uniformize param

* revert: bring back merge transaction

* fixes

* refactor: lift out respond

* fix: is_required_merge doesn't work

* refactor: transaction api

* resolve initial conflicts

* tighten specification and refactor tests

* add: dialyzer specs

* refactor: merge constraints

* refactor: accept max. 4 utxo positions

* refactor/add tests

* refactor: merge_constraints limit max. positions to 4

* refactor: replace Alice and Bob fixtures with setup

* refactor: return keyword lists from parameter validation

* update documentation

* post-merge: fix failing tests

* post-merge: fix failing tests

* fix: dialyzer

* fix (lint): remove unused aliases

* post-merge: remove unintended spacing

* add: fallback descriptions

* remove `result` key from body schema for /merge

* reinstate result key where removed.

* rename tests

* fix: duplicate test typo

* remove empty line

Co-authored-by: euro <[email protected]>
Co-authored-by: okalouti <[email protected]>
Co-authored-by: euro <[email protected]>

* add perf cli smoke test (#1762)

* add perf cli smoke test

This PR adds ci step that runs `transactions` perf test

* feat: send web telemetry events to Datadog

Using the telemetry_metric and telemetry_metrics_statsd reporter, we can
send telemetry events to Datadog without defining more event handlers.
The supervised process is configured with the same parameters as the
Statix metrics generation.

In omg.status, we collect web events from Phoenix and send them to
Datadog as metrics.

The fallback controller in WatcherRPC is also instrumented to execute
telemetry events on errors, with route and error type recorded as tags.

chore: mix format

chore: fix linter warnings, shorten module paths

* fix: move env var fetch into Configuration module

* fix: ensure current_route doesn't throw an exception

It's possible in SupportedWatcherModes for a conn to end up in the
fallback controller without being routed there. This change makes it so
that if the route can't be determined from the conn, we return nil
instead of erroring.

Fixes a failing test.

* fix: allow nil release and current_version env vars

* Chore: fix feefeed container healthcheck (#1768)

* fix feefeed container healtheck

Check if feefeed fetched fees instead of checking if it's up

* log docker-compose logs

* feat: add new metadata to Datadog traces

Add service, version and error metadata to traces in Datadog APM traces.

What you see in the traces:
* The service name (default 'web' for Phoenix applications) is the name
of the OMG application.
* Traces are tagged with application version number
* For API responses which error, the trace is flagged red as an error,
the error code is marked on the trace and the error message is found
in the tags.

* refactor: move configuration helpers to new module

Moved from Application to Configuration for WatcherRPC.

* refactor: remove unused app field in Application

* fix: failing tracer and response tests

* fix: aggressive test fix (#1772)

* Collect metrics through datadog for the perf project (#1765)

* retry datadog errors (#1774)

* retry datadog errors

* use header function to fail

* add `retrying` to log messages

* fix: run tests that alter globals consecutively

* refactor: atom to string with interpolation

* ignore known errors - Sentry spike protection (#1775)

* ignore known errors - Sentry spike protection

* run reorg tests only on master

* linting specs is not needed

* Fix readme docker docs (#1773)

* Fix readme docker docs

* change commands

* enter directory

* add perf docs

* move command info to the top

Co-authored-by: Ino Murko <[email protected]>

* config filter sentry (#1778)

* add docker-compose for `test_docker_compose_release` (#1777)

* extract geth command, remove childchain only code paths (#1781)

* extract geth command

* childchain code rem

* update deps, lint

* the great decoupling

* config fix

* config fix, check if Watcher Info config value

* delete childchain test

* config anon functions comments

* #1588 phx exception caused by invalid accept header is now ignored

* #1588 added a test

* ignore RequestTooLarge for Sentry (#1779)

this error is coming from PlugParser when a request is too large. we
don't want this reporting this event to on-call

* set remote_ip from cf-connecting-ip header (#1679)

* set remote_ip from X-Forwarded-For header

* add custom plug that sets remote_ip

* set the right most ip as remote_ip

* add test

* set the left most ip

* use `CF-Connecting-IP` header

* fix tests

Co-authored-by: Arthur Chiu <[email protected]>
Co-authored-by: Ino Murko <[email protected]>

* #1588 minor adjustmen

* feat: add X-Watcher-Version header to http requests (#1767)

* #1588 minor fix

* fix: set service name as a tag in metrics

Currently the service name is part of the prefix for metrics, but it
would be far better if it were a tag, so that services could be compared
side by side in Datadog.

* fix: limit matching for ethereumex responses (#1782)


* fix: rem error response handling

* fix: add sentry filter

Co-authored-by: Ino Murko <[email protected]>

* just the one thing to turn into a warn (#1784)

Co-authored-by: Ino Murko <[email protected]>

* Renamed all occurrences of omisego to omgnetwork (#1788)

* #1555 Renamed omisego to omg network

* #1555 Fixed urls

* #1555 downgraded rocksdb to 1.5.1

* chore: changelog/version bump for v1.0.5 (#1747) (#1785)

* chore: changelog/version bump v1.0.5

* update CHANGELOG for 1.0.5-pre.0

* bump to include latest changes for infura

Co-authored-by: Arthur Chiu <[email protected]>
Co-authored-by: boolafish <[email protected]>

* chore: add sem release automation (#1791)

* chore: add sem release

* chore: set release rule for our standard

* chore: test dummy

* fix: sem-release format

* chore: use dry-run on sem release

* fix: back to origin config

* bump elixir (#1789)

* bump elixir

* align rust version to 1.46 (#1793)

* align rust version to 1.46

* trace reorg tests (#1796)

* fix: make sem release ignore PR check (#1797)

* chore: use git tag version instead (#1798)

* chore: use git tag version instead

* fix: for lint

* style: pipe :)

* refactor: put to private function for version()

* simplify event pulling (#1801)

* simplify event pulling

* Update core_test.exs

* Update ethereum_event_listener.ex

* Update core.ex

* fix: git tag (#1804)

* fix: use git tag cmd to get version

* test: pulish working

* Revert "test: pulish working"

This reverts commit faf0bab.

* Update CODEOWNERS (#1805)

* Watcher_Info events race cause spent output presented unspent (#1769)

* feat: add 'ensure_output' to EthEvent.insert_exits to explicitly allow
missing outputs

* test: demonstrate missing output feature

* feat: unplug Std exit consumer of WatcherInfo

* feat: unplug IFE exit started consumer of WatcherInfo

* feat: unplug IFE output piggybacked consumer of WatcherInfo

* feat: unplug IFE output withdrawn consumer of WatcherInfo

* feat: unplug deposits handling out of the Bus

* feat: unplug block application handling out of the Bus

* chore: delete unused modules, linter fixes

* fix: improve TxOutput schema queries and show that by tests

* fix: fix wrongly marked outout existance

* chore: [after review] align to comments

* test: use syncronous test when talking to postgres

* refactor: [after review] Insert full block data synchronously instead of pending block

* chore: delete unused code regarding pending blocks machinery

* test: fix tests - change nonexisting pending block

* test: fix test

* refactor: be explicit about which event expect the output existence

* chore: align to comment

Co-authored-by: Ino Murko <[email protected]>
Co-authored-by: Pawel Nowosielski <[email protected]>

* fix ex_plasma reference

* fix linter

fix eth_event test

try using `ex_plasma` 0.2.0

try running gcloud without installing

use master branch for specs submodules

* merge from masterv2

* merge from masterv2

* ci fixes

* test fixes

* test fixes

* missing events for PG

* missing events for PG

Co-authored-by: Omar Kalouti <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Ayrat Badykov <[email protected]>
Co-authored-by: Unnawut Leepaisalsuwanna <[email protected]>
Co-authored-by: boolafish <[email protected]>
Co-authored-by: pgebal <[email protected]>
Co-authored-by: jarindr <[email protected]>
Co-authored-by: Ino Murko <[email protected]>
Co-authored-by: Kevin Sullivan <[email protected]>
Co-authored-by: euro <[email protected]>
Co-authored-by: Mederic <[email protected]>
Co-authored-by: Arthur Chiu <[email protected]>
Co-authored-by: Nicholas Mueller <[email protected]>
Co-authored-by: Michael Captain <[email protected]>
Co-authored-by: Daniela Ivanova <[email protected]>
Co-authored-by: Souradeep Das <[email protected]>
Co-authored-by: Pawel Nowosielski <[email protected]>
Co-authored-by: Pawel Nowosielski <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@InoMurko InoMurko InoMurko approved these changes

@boolafish boolafish boolafish approved these changes

@achiurizo achiurizo Awaiting requested review from achiurizo

@T-Dnzt T-Dnzt Awaiting requested review from T-Dnzt

Assignees
No one assigned
Labels
chore Technical work that does not affect service behaviour
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ayrat555 @InoMurko @boolafish