Full IPFS CID codec/hash support - passing CidConfig over TransactionExtension

[bkontur]

Implements: paritytech/polkadot-sdk#10301
Depends on: paritytech/litep2p#482

TL'DR

Basically, with this feature, we won’t need any external IPFS gateway + Bulletin can handle/support multi-chunk big files as IPFS DAG-PB.
(Just a light-client IPFS p2p/http layer connected to the Bulletin node cluster: #93)

Context

Actually, pallet-transaction-storage was only able to index content/transactions using a Blake2b256 hash. With this PR, we can now provide a CID codec and hashing, which allows the pallet to index content using Sha2_256 or Keccak256 as well. This makes it IPFS-compatible—for example, it can handle and store DAG-PB content directly without needing an additional call to the IPFS gateway, using:

const dagCid = await ipfs.block.put(dagBytes, {
    format: 'dag-pb',
    mhtype: 'sha2-256'
})

or 

const dagCid = await ipfs.dag.put(
    dagNode,
    {
        storeCodec: 'dag-pb',
        hashAlg: 'sha2-256',
        pin: true
    }
);

TODO

• change store_chunked_data.js not using ipfs.dag.put
• check_unsigned uses:

Call::<T>::store { data } => Self::check_store_renew_unsigned(
					data.len(),
					|| sp_io::hashing::blake2_256(data),
					context,
				),

[dmitry-markin]

Looks good with the exception that I don't know enough to reason about the extension logic.

We should probably also bring the transactions-storage pallet in substrate up to date?

Also, we will need to change the transaction indexing in the DB to support custom codecs and hash codes to properly handle DHT content provider publishing, but I can handle this after the initial Kademlia support is working (the first version will lack dag-pb blocks publishing to the DHT, unfortunately).

[dmitry-markin]

nit: CidData contains both the CID "parts" and the binary representation of the CID. What about making it only contain "parts", and introduce the function that yields the bytes?

E.g., we can implement From<CidData> for CidGeneric<32> that builds the actual CID, and add a convenience member function fn to_bytes(self).

[lexnv]

nit: config.map(|config| (config.hashing, config.codec).unwrap_or((Blake, 0x55))

[bkontur]

nit: config.map(|config| (config.hashing, config.codec).unwrap_or((Blake, 0x55))

@lexnv What about: map_or_else? 6e31855

[lexnv]

Suggested change

	let mh = Multihash::<32>::wrap(multihash_code, &content_hash).map_err(|e| {
	log::warn!(
	target: LOG_TARGET,
	"Failed to create Multihash for content_hash: {content_hash:?}, multihash_code: {multihash_code:?}, error: {e:?}"
	);
	()
	})?;
	let mh = Multihash::<32>::wrap(multihash_code, &content_hash).inspect_err(|e| {
	log::warn!(
	target: LOG_TARGET,
	"Failed to create Multihash for content_hash: {content_hash:?}, multihash_code: {multihash_code:?}, error: {e:?}"
	);
	})?;

[raymondkfcheung]

Shall we use tracing, instead of log?

[raymondkfcheung]

To be followed up in #123