My harvest can't process all blocks

[imstar15]

My havrest monitor:
http://149.28.147.147:5555/
http://149.28.147.147:8080/oak-testnet/harvester/admin

It has 1479 failed.
data_account_audit is filled, but data_account is empty

Error is about MetadataV7ModuleStorageEntry

http://149.28.147.147:5555/task/ba1603a3-a349-47f7-8661-f63b635f0bc9

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/src/app/app/tasks.py", line 70, in __call__
    return super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/src/app/app/tasks.py", line 392, in update_balances_in_block
    harvester.create_full_balance_snaphot(block_id)
  File "/usr/src/app/app/processors/converters.py", line 1030, in create_full_balance_snaphot
    if storage_method.get("type_hasher_key1") == "Blake2_128Concat":
AttributeError: 'MetadataV7ModuleStorageEntry' object has no attribute 'get'

My docker-compose yml file:
https://github.com/OAK-Foundation/polkascan-os/blob/oak-testnet-host/docker-compose.oak-testnet.yml

My custom types:
https://github.com/OAK-Foundation/polkascan-pre-harvester/blob/9a5bf632eed712a09c33cea32831e24bc42083fa/app/type_registry/oak_testnet.json

Please help me to solve it.

Thanks!

[imstar15]

I think there is some wrong about storage_method

print storage_method

{'name': 'Account', 'modifier': 'Default', 'type': {'MapType': {'hasher': 'Blake2_128Concat', 'key': 'AccountId', 'value': 'AccountInfo<Index, AccountData>', 'isLinked': False}}, 'fallback': '0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000', 'docs': [' The full account information for a particular account ID.']}

def create_full_balance_snaphot(self, block_id):

        block_hash = self.substrate.get_block_hash(block_id)

        # Determine if keys have Blake2_128Concat format so AccountId is stored in storage key
        storage_method = self.substrate.get_metadata_storage_function(
            module_name="System",
            storage_name="Account",
            block_hash=block_hash
        )

        if storage_method:
            print(storage_method)
            if storage_method.get("type_hasher_key1") == "Blake2_128Concat":

                # get balances storage prefix
                storage_key_prefix = self.substrate.generate_storage_hash(
                    storage_module='System',
                    storage_function='Account',
                    metadata_version=settings.SUBSTRATE_METADATA_VERSION
                )

                rpc_result = self.substrate.rpc_request(
                    'state_getKeys',
                    [storage_key_prefix, block_hash]
                ).get('result')
                # Extract accounts from storage key
                accounts = [storage_key[-64:] for storage_key in rpc_result if len(storage_key) == 162]
            else:
                # Retrieve accounts from database for legacy blocks
                accounts = [account[0] for account in self.db_session.query(distinct(Account.id))]

            for account_id in accounts:

                self.create_balance_snapshot(block_id=block_id, account_id=account_id, block_hash=block_hash)

[imstar15]

I have try to fix it

[arjanz]

Thanks for your PR, I integrated it (and gave you credits for it) in a commit that include other overdue library updates.

Best you update the whole https://github.com/polkascan/polkascan-os repos with submodules, because I also included some fixes in the explorer-api submodule.

I hope it works for you now, I have to point out that the polkascan-os project is reaching end-of-life (will be succeeded by a version 2 in several months), so I don't have much time to spend on this.