Memory optimizations for large address queries #392
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 7 commits
January 11, 2016 16:17
- Restored functionality to be able to query the history of multiple addresses in one query - Sorted mempool transactions by timestamp in txid lists
braydonf
force-pushed
the
large-queries
branch
9 times, most recently
from
71e892d to
7b0464d
Compare
- Refactored getAddressSummary and added several tests - Fixed bugs revealed from the integration regtests - Updated many unit tests
braydonf
force-pushed
the
large-queries
branch
from
7b0464d to
4fcec87
Compare
added 4 commits
January 14, 2016 17:17
Querying addresses that have millions of transactions is supported however takes hundreds of seconds to fully calculate the balance. Creating a cache of previous results wasn't currently working because the `isSpent` query is always based on the current bitcoind tip. Thus the balance of the outputs would be included however wouldn't be removed when spent as the output wouldn't be checked again when querying for blocks past the last checkpoint. Including the satoshis in the inputs address index would make it possible to subtract the spent amount, however this degrades optimizations elsewhere. The syncing times or querying for addresses with 10,000 transactions per address. It may preferrable to have an additional address service that handles high-volume addresses be on an opt-in basis so that a custom running client could select high volume addresses to create optimizations for querying balances and history. The strategies for creating indexes differs on these use cases.
braydonf
force-pushed
the
large-queries
branch
from
820080d to
af72d40
Compare
braydonf
force-pushed
the
large-queries
branch
from
af72d40 to
62934b4
Compare
braydonf
force-pushed
the
large-queries
branch
from
336c316 to
d4f2df5
Compare
braydonf
changed the title
|
Ready for wider testing |
added 5 commits
January 18, 2016 16:03
There was an issue where streams would still be held open if "pause" was called before "end", this would lead to http requests from the insight-api not being returned with an error status as soon as possible but would instead stay open.
|
LGTM |
kleetus
added a commit
that referenced
this pull request
Jan 27, 2016
Memory optimizations for large address queries
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New Address Service Options:
maxInputsQueryLength(default 50,000) // The maximum number of inputs per querymaxOutputsQueryLength(default 50,000) // The maximum number of outputs per querymaxHistoryQueryLength(default 100) // The maximum number of transactions per querymaxAddressesQuery(default 10,000) // The maximum number of addresses per queryNew DB Service Options:
maxOpenFiles(default 200) // The maximum number of files that leveldb keeps in cache