fix(nft): add log_index to history table and use in PK#1926
Conversation
shamardy
requested review from
anarkiVan,
borngraced,
caglaryucekaya and
onur-ozkan
|
@anarkiVan can you please check if this PR fixes this error GLEECBTC/gleec-wallet#1239 (comment) or not? |
@shamardy the error is gone. I was able to see nfts and transactions using seed from the issue |
| token_name TEXT, | ||
| details_json TEXT | ||
| details_json TEXT, | ||
| PRIMARY KEY (transaction_hash, log_index, chain) |
There was a problem hiding this comment.
note: I don't know about the frequency of write operations this table will get, but this PK design will definetly slow down these operations a lot specially when there are too many rows.
There was a problem hiding this comment.
I can remove this primary key all together, we already avoid writing duplicate transfers that have the same chain/transaction_hash/log_index by using the last block number when requesting new transfers that will be saved. It will be difficult though to detect problems/bugs like the one that this PR fixes if I did that.
There was a problem hiding this comment.
we already avoid writing duplicate transfers that have the same chain/transaction_hash/log_index by using the last block number when requesting new transfers that will be saved.
If it's done with the thread-safe way, seems fine yeah.
It will be difficult though to detect problems/bugs like the one that this PR fixes if I did that.
The error seems quite clear if I am not missing something.
I don't have hard feelings to remove this PK, just wanted to leave a note on it. I leave it up to you :)
There was a problem hiding this comment.
GLEECBTC/gleec-wallet#1239 (comment) seems quite clear if I am not missing something.
It was a db error due to the unique constraint on transaction hash, if moralis duplicates data for some reason or there are problems with our implementation that we are trying to save duplicate data it won't be caught without some kind of constraint. Adding a constraint on these values instead of a primary key will also slow down operations afaik.
I don't have hard feelings to remove this PK, just wanted to leave a note on it. I leave it up to you :)
Let's leave it as is then and see if there are performance problems in the future :)
There was a problem hiding this comment.
Adding a constraint on these values instead of a primary key will also slow down operations afaik.
The idea is not merging multiple values as PK/constrait because on each write each row in the table will need to check all these fields. The best way to deal with this would be creating another field(maybe id?) and inserting the values with pre-calculated id(which can be combination of the values used as PK atm) from mm2.
Let's leave it as is then and see if there are performance problems in the future :)
Totally fine 👍
There was a problem hiding this comment.
Is the column necessary?
It's not, but I guess it was added to make this https://github.com/KomodoPlatform/komodo-defi-framework/blob/be413ca296b1330172023d91680c8c74b0d1c1c6/mm2src/coins/nft/storage/sql_storage.rs#L201-L209 deserialize to the structs straight away. I can change the function to transfer_history_from_row_and_chain and insert the chain value into the json object before returning it or just leave it as is. What do you think is better?
There was a problem hiding this comment.
I think rather than keeping a field all with the same values, inserting the value while deserializing better.
There was a problem hiding this comment.
Inserting the value after deserialization and removing it before serialization would complicate the code a lot. Additionally, adding the chain field after fetching the row/s requires parsing the JSON string into a serde_json::Value, modifying it, and then deserializing it into an Nft/NftTransferHistory structs. This process likely has a greater impact on performance than simply fetching the chain value directly from the database and deserializing the entire row in one operation. Ideally, the chain field should not have been included in the Nft/NftTransferHistory structs from the begining, but it cannot be removed now as it is part of the responses used by GUIs.
There was a problem hiding this comment.
I can of course change these structs
https://github.com/KomodoPlatform/komodo-defi-framework/blob/717e376447f9bcb76d972b2211ad22c83e2c287a/mm2src/coins/nft/nft_structs.rs#L218-L227
https://github.com/KomodoPlatform/komodo-defi-framework/blob/717e376447f9bcb76d972b2211ad22c83e2c287a/mm2src/coins/nft/nft_structs.rs#L392-L405
to make all fields other than chain part of a new struct that is used for DB operations and use #[serde(flatten)] for this new struct, but the Nft/NftTransferHistory structs will get complicated too. I think I would rather keep the chain column :)
There was a problem hiding this comment.
Okay I thought it would be simple. In that case, it can stay as it is.
|
For the future PRs, we can split the changes into multiple commits(for this PR it would be 1 for the renaming, 1 for the actual fix), it would be very useful for the reviewers. It was quite confusing for me :) |
Yeah, sorry about that. I thought name changes are gonna be minimal and limited to db methods when I started renaming but it went out of control and I couldn't separate it into a different commit. I will keep that in mind in the future and try to separate commits even if I thought they will be small at the start. |
caglaryucekaya
left a comment
caglaryucekaya
left a comment
There was a problem hiding this comment.
LGTM, I only have some questions
| token_name TEXT, | ||
| details_json TEXT | ||
| details_json TEXT, | ||
| PRIMARY KEY (transaction_hash, log_index, chain) |
There was a problem hiding this comment.
Aren't there a different transfer history table for each chain, why do we need the chain as part of the primary key?
|
@laruh It's not related to this PR, but when looking at it I saw that these two tests are doing the same thing. Did you mean to do something else with the |
5 participants
fixes GLEECBTC/gleec-wallet#1239 (comment)
related comment GLEECBTC/gleec-wallet#1239 (comment)
log_indexas part of the transfers history table primary key.nft_tx_historytable is now callednft_transfer_historyandtx/txsare renamed totransfer/transfersthroughout the NFT code since what's added/retrieved from DB is NFT transfers not transactions (Multiple NFT transfers can be in one transaction). By renaming the table, there are no need for db migrations due to the addition oflog_indexcolumn. Although NFT is not used in production yet, if anybody used it, transfers will be re-fetched and saved to the new DB table when using the related API methods.