Indexer - Remove cleaning step

[marcus-snx]

• import parquet data directly from the subsquid raw files
• remove cleaning script

After the data is indexed by the subsquid processor, run:

docker compose run -e NETWORK_NAME={network_name} -e PROTOCOL_NAME={protocol_name} indexer uv run scripts/import_parquet.py

This will ingest the raw indexed data into ClickHouse.

[Tburm]

Ended up getting some of these errors while running the insert, is this something you have seen?

Initializing tables for base_mainnet synthetix
Inserting base_mainnet synthetix data into tables
Found 155 events for base_mainnet
Unexpected Http Driver Exception
Error inserting data into raw_base_mainnet.synthetix_perps_market_proxy_event_reward_distributor_registered: Error HTTPConnectionPool(host='clickhouse', port=8123): Read timed out. (read timeout=300) executing HTTP request attempt 1 (http://clickhouse:8123)
Error inserting data into raw_base_mainnet.synthetix_perps_market_proxy_event_factory_initialized: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 373
 Code: 373. DB::Exception: Session b11379af-ef00-4a77-ae86-f0e145ccace2 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 24.11.1.2557 (official build))

Error inserting data into raw_base_mainnet.synthetix_spot_market_proxy_event_owner_changed: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 373
 Code: 373. DB::Exception: Session b11379af-ef00-4a77-ae86-f0e145ccace2 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 24.11.1.2557 (official build))

[marcus-snx]

Ended up getting some of these errors while running the insert, is this something you have seen?

Initializing tables for base_mainnet synthetix
Inserting base_mainnet synthetix data into tables
Found 155 events for base_mainnet
Unexpected Http Driver Exception
Error inserting data into raw_base_mainnet.synthetix_perps_market_proxy_event_reward_distributor_registered: Error HTTPConnectionPool(host='clickhouse', port=8123): Read timed out. (read timeout=300) executing HTTP request attempt 1 (http://clickhouse:8123)
Error inserting data into raw_base_mainnet.synthetix_perps_market_proxy_event_factory_initialized: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 373
 Code: 373. DB::Exception: Session b11379af-ef00-4a77-ae86-f0e145ccace2 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 24.11.1.2557 (official build))

Error inserting data into raw_base_mainnet.synthetix_spot_market_proxy_event_owner_changed: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 373
 Code: 373. DB::Exception: Session b11379af-ef00-4a77-ae86-f0e145ccace2 is locked by a concurrent client. (SESSION_IS_LOCKED) (version 24.11.1.2557 (official build))

I've never had these errors before. Do they appear every time you try to run it?

[Tburm]

Yes I tried it twice and got them both times. I haven't checked to see what happens with the data though, if they are missing those events. I'm confused because I don't know why it would be concurrent at all?

[Tburm]

Going to merge this as it completed successfully, but I did get these two errors which we need to follow up and fix:

Error inserting data into raw_base_mainnet.synthetix_transaction: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 60
 Code: 60. DB::Exception: Table raw_base_mainnet.synthetix_transaction does not exist. (UNKNOWN_TABLE) (version 24.11.1.2557 (official build))

Error inserting data into raw_base_mainnet.synthetix_block: HTTPDriver for http://clickhouse:8123 received ClickHouse error code 20
 Code: 20. DB::Exception: Number of columns doesn't match (source: 3 and result: 2). (NUMBER_OF_COLUMNS_DOESNT_MATCH) (version 24.11.1.2557 (official build))