I cannot find in IPFS gateway / online (using CID) the content I uploaded through Helia

[Eikix]

Hi all, my name's Elias,

I'm new to Helia, very cool project.
I'm currently writing a nodeJS script to add JSON files to IPFS using Helia:

import { createHelia } from 'helia';
import { strings } from '@helia/strings';
import { content } from './uri.mjs';
import fs from 'fs/promises';
import { createLibp2p } from 'libp2p';
import { tcp } from '@libp2p/tcp';
import { noise } from '@chainsafe/libp2p-noise';
import { yamux } from '@chainsafe/libp2p-yamux';
import { MemoryBlockstore } from 'blockstore-core';
import { MemoryDatastore } from 'datastore-core';

const blockstore = new MemoryBlockstore();
const datastore = new MemoryDatastore();

const libp2p = await createLibp2p({
  addresses: {
    listen: ['/ip4/127.0.0.1/tcp/8080'],
  },
  transports: [tcp()],
  connectionEncryption: [noise()],
  streamMuxers: [yamux()],
  datastore: datastore,
  identify: {
    host: {
      agentVersion: 'helia/0.0.0',
    },
  },
});

const helia = await createHelia({ libp2p, blockstore, datastore });
const s = strings(helia);

const addedCIDs = [];

async function uploadMetadata(content) {
  const metadata = {
    name: content.name,
    description: content.description,
    external_url: content.external_url,
    image: `ipfs://${content.image}`,
    attributes: content.attributes,
  };

  try {
    const cid = await s.add(JSON.stringify(metadata));
    addedCIDs.push(cid.toString());

    // Test that the CID can be retrieved
    const myString = await s.get(cid);
    console.log(JSON.parse(myString));

    // Parse CID
    const cidString = cid.toString();

    console.log(`Uploaded metadata for ${waifu.name}. CID info: ${cidString}`);
  } catch (err) {
    console.error(`Failed to upload metadata for ${waifu.name}. Error: ${err}`);
  }
}

async function main() {
  for (const item of content) {
    await uploadMetadata(item);
  }

  console.log(addedCIDs);


  await fs.writeFile('./addedCIDs.json', JSON.stringify(addedCIDs));

  console.log('All CIDs have been saved to addedCIDs.json');
}

main()
  .then(() => process.exit(0))
  .catch(console.error)
  .finally(() => process.exit(1));

The script successfully adds creates the CIDs in my addedCIDs.json file:

[
  "bafkreicbkmd3ba2y35co7v7n2kjnqfh2rrrw72j3etm5fmoumz5r65pa54",
  "bafkreihaparff6nojjm5qvnt55xnrnv4g4gxkmhexfbm45yozhqupko6am",
  "bafkreic3qaase5us7jkdd55mokytbed7fe4seujknmm52pff756t5nyn4m",
  "bafkreifhg3jgse6g2khv4npzg35buiqjn36xuoz7mu5aoobeh5ckqapspm",
  "bafkreigb5nozyw232voxad2vllafwgmwfhsla46uggzpsrrxrhkiszlk2q",
  "bafkreicwjokn4jhnpmesnhtj6vmktgl3vl2b6ebzocn5jl6fdq74ocyslu",
  "bafkreihptw6rz43otvvgx3ntf6rlxh5knfiod6l64nlaieo3t2inw4x5qm",
  "bafkreiaoekhluctmyehexg3loeb7jspx4pkkw6k2aixhhnypl7lofeujsm",
  "bafkreib5ilj4vm3yk35nyzvgup6kcybz2g7mzey4k4qssz733ftynybq5q",
  "bafkreiaa4zvkv563usfbbljci56jdkf7ven5lt5vbcxt6ki7sipcbezbfq",
  "bafkreieju7xdev3z4sabaeejm5f2r2mnaqeltm4adzklk6yeeyancuccle",
  "bafkreig7r5duyx2fcxqocx5q7opaqq4oqiahgguodc65zu45jj2sdimq2i",
  "bafkreibs6k2gbncsel2apz6m2oighgiy5zftfaeeq6l2swmss2h22liayq",
  "bafkreicytf3eapch4ghte2ybyjgz2lbkiw5xgywsmo4w7e3l3j7n445gg4",
  "bafkreibmsb6z3fd44bua3grtep3ah3cpcrpmy2lgooj3sjhdy5mba3zk5u",
  "bafkreigc3waovtz6zgqvfscqdmphwarmaie6vrf7q6sf7j35qcip6kkeze",
  "bafkreibhm37epellasuaqvc44geqv32cpittak2zcoqevkxreb7qp4uyhu"
]

But when I look them up on IPFS, I cannot seem to find them.

Is the problem linked with pinning? Or something I missed?

[Eikix]

Terminal output:

Thanks for your help!

[Eikix]

I also tried to use @helia/json to add the content to IPFS. Interestingly, the resulting CID looks like bagaaieraifjqpmedldpuj36x5xjjfwau7kggg37jhmsntuvr2rthwh3v4dxq and is no longer prefixed by "bafk" + "rei"

[BigLep]

2023-06-07 maintainer conversation:

1. We should point to an example that demonstrates this case clearly of making content retrievable from the gateway
2. Give notification to user here about making a post in discuss.ipfs.tech

[maxdesalle]

Getting the same issue on my end, even with basic strings! Tried multiple IPFS explorers, none of them seem to index the uploaded data.

[SgtPooki]

@Eikix @maxdesalle @zveroboy @AlexCherrypi

This is related to #159, ipfs-examples/helia-examples#43, ipfs-examples/helia-examples#38, and the discuss.ipfs.tech post located at https://discuss.ipfs.tech/t/how-to-retrieve-content-uploaded-via-helia-using-the-ipfs-gateway/16582.

I answered at https://discuss.ipfs.tech/t/how-to-retrieve-content-uploaded-via-helia-using-the-ipfs-gateway/16582/4?u=sgtpooki, but let's continue there as this is not a Helia bug and more of a learning opportunity.

I also tried to use @helia/json to add the content to IPFS. Interestingly, the resulting CID looks like bagaaieraifjqpmedldpuj36x5xjjfwau7kggg37jhmsntuvr2rthwh3v4dxq and is no longer prefixed by "bafk" + "rei"

This is how IPFS works. Please go through https://proto.school/anatomy-of-a-cid/ if you haven't yet and it should explain why using a different codec changes what the final CID is.

If you're running a non-browser node, the support provided by @achingbrain in ipfs-examples/helia-examples#43 may also be of assistance to you.

We should point to an example that demonstrates this case clearly of making content retrievable from the gateway

For an issue tracking the creation of an example where we host content on a Helia node and access that content from an IPFS gateway, please follow ipfs-examples/helia-examples#39