Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document solver competition information route #185

Merged
merged 3 commits into from
May 4, 2022
Merged

Document solver competition information route #185

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6bc1660
Document solver competition information route
vkgnosis May 2, 2022
54e39c2
Add a bunch more data as suggested in review
vkgnosis May 3, 2022
e26d971
Merge branch 'main' into competition-route
vkgnosis May 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions crates/orderbook/openapi.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -483,6 +483,30 @@ paths:
description: Too many order quotes
500:
description: Unexpected error quoting an order
/api/v1/solver_competition/{auction_id}:
get:
summary: Information about solver competition
description: |
Returns the settlements submitted by every solver for a specific auction
id. The auction id corresponds to the id external solvers are provided
with. Auction ids are not globally unique, they are reused from time to
time. The backend keeps the competition information for a limited amount
of time.
Comment on lines +490 to +494
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment by Filius on Discord:

From what I read, auctions (and their solutions) are only stored temporarily on the backend. As a solver developer, the process I would follow to benchmark my solution would be: (i) select an auction ID I would like to benchmark from my own logs, (ii) query solver competition endpoint to obtain maps of solvers -> settlements and (iii) try to understand why the solution with the highest objective value is better than mine. For (iii) I would need to be sure the response I obtained from the endpoint really corresponds to the auction I selected (input -> output). Since auction IDs are not unique, an auction timestamp would help me ensure I am looking at the right auction.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to eventually store this information in our database for a long time but it might take a bit longer until we have this implemented. Until then we are planning on using a simple in memory store with some size limit. The auction ids are probably going to be truly unique in the future too but not yet.

Until we have both of that maybe it would reasonable for solver developers to persist the competition information on their own? In their solver implementations they get the auction id and could query the competition info ~a minute after solving and persist it locally.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree.

parameters:
- name: auction_id
in: path
required: true
schema:
type: integer
responses:
200:
description: competition info
content:
application/json:
schema:
$ref: "#/components/schemas/SolverCompetitionResponse"
404:
description: No competition information available for this auction id.
components:
schemas:
TransactionHash:
Expand Down Expand Up @@ -965,3 +989,57 @@ components:
the fee after this expiration date. Encoded as ISO 8601 UTC.
type: string
example: "1985-03-10T18:35:18.814523Z"
SolverCompetitionResponse:
nlordell marked this conversation as resolved.
Show resolved Hide resolved
type: object
nlordell marked this conversation as resolved.
Show resolved Hide resolved
properties:
transaction_hash:
nullable: true
allOf:
- $ref: "#/components/schemas/TransactionHash"
gas_price:
type: number
description: gas price used for ranking solutions
solutions:
type: array
description: Maps from solver name to object describing that solver's settlement.
items:
$ref: "#/components/schemas/SolverSettlement"
SolverSettlement:
type: object
properties:
solver:
type: string
description: name of the solver
objective:
type: object
properties:
total:
type: number
description: the total objective value used for ranking solutions
surplus:
type: number
fees:
type: number
cost:
type: number
gas:
type: number
prices:
type: object
additionalProperties:
$ref: "#/components/schemas/BigUint"
description: |
The prices of tokens for settled user orders as passed to the settlement contract.
orders:
type: array
description: the touched orders
items:
type: object
properties:
id:
$ref: "#/components/schemas/UID"
executed_amount:
$ref: "#/components/schemas/BigUint"
call_data:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment by Filius on Discord:

With regards to the calldata, I wonder if it would be fair to expose the complete solution provided by solvers. In the information I requested, I explicitly mentioned clearing prices without the actual liquidity sources, so that at least the liquidity sources are kept "hidden". Would it be fair to show everyone what liquidity sources different solvers are using? After all, this is a competition.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it makes sense to treat this as public because if your solution is the winning one it will be public anyway as part of the public transaction on chain.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a very interesting question. Especially solvers with inventory might prefer not to expose their quotes publicly unless they have an upside (winning the batch). I wouldn't want them to not submit a solution if they think their chance of winning is small (but we can also wait and change the model when this happens).

For users however it would definitely be good to see the alternative prices (to highlight the price improvement competition brings) and maybe even simulate and plot the alternative solution. For a solver it might also be helpful to see the full simulation link to debug why the settlement was failing in the driver.

Maybe we start with all public and then adapt if it becomes an issue (winning solver will reveal their logic for sure)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can start by only exposing name of winning solver while the competition is on-going. This way, other solvers don't have information on:

  • Winning objective value (so they can penny the surplus by making the prices slightly better for the users at the expense of the buffers and their weekly payout - as long as its less than 100 COW - in order to win with a slight advantage)
  • Calldata - the current routes being proposed and liquidity being used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels like our current goal with this route is to make it easier for external solver creators to get insight into the competition. If that is the goal then we don't need to show anything live. It is enough to only make this information available after a winning settlement has been picked and it has been settled.

If the goal is also to provide information that can be used live for example by the frontend to show a user that their order is part of some solutions or has been picked for the currently settling transaction, then we need to include more information.

I feel that there is little reason to show the competition while we're still collecting solutions from all the solvers. This part only takes 10 seconds. Once the driver has all the settlements and has decided which one the winner is but before waiting for the on chain transaction, we can publish all the information. Since the solutions have been collected there shouldn't be worries about pennying.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the goal is also to provide information that can be used live for example by the frontend to show a user that their order is part of some solutions or has been picked for the currently settling transaction, then we need to include more information.

I think this will eventually be the goal, however I think that we can get around to this once we do it the "right" way (i.e. implement one of your ideas from the discussion #175).

Copy link
Contributor

@nlordell nlordell May 3, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created #187 to discuss what the "complete" API can look like for use with the FE (where the current auction status is useful to display some extra information to the user).

description: hex encoded transaction calldata
type: string