gatsby-source-contentful not downloading images with downloadLocal true

[pbizimis]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

There seems to be an issue with downloading images from contentful to local if downloadLocal is true.
So this config seems to have bugs:

{
      resolve: `gatsby-source-contentful`,
      options: {
        spaceId: `1234`,
        accessToken: "1234",
        downloadLocal: true,
      },
    },

I tried it twice with two different contentful repositories and two gatsby sites. Sometimes no image is local, sometimes one, sometimes all. I tried different things such as downloadLocal: false then true so the images are loaded for gatsby develop. I also tried gatsby build + serve but same issues here. The images are always accessible without localFile in graphql which means that the images are correctly configured.

This is what I get:

{
    "edges": [
        {
            "node": {
                "localFile": {
                    "id": "6f1d839c-97d8-5694-b529-ab8f735a5418"
                }
            }
        },
        {
            "node": {
                "localFile": null
            }
        },
        {
            "node": {
                "localFile": null
            }
        }
    ]
}

I have a minimal reproduction repo at https://github.com/pbizimis/gatsby-contentful-bug. I used a new contentful account so if you need to check something there let me know and I give you the credentials.

(Images are from Unsplash and the config of the repo is not sensitive because the account is a throwaway)

Thanks!

Reproduction Link

https://github.com/pbizimis/gatsby-contentful-bug

Steps to Reproduce

1. Start it up
2. On localhost:8000/, there should be a console log that shows an object with an array of three (you should see that at least one is null).

Expected Result

localFile data is available for all images

Actual Result

localFile data is not available for all images

Environment

System:
    OS: Linux 5.11 Ubuntu 20.04.3 LTS (Focal Fossa)
    CPU: (16) x64 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 14.17.6 - ~/.nvm/versions/node/v14.17.6/bin/node
    npm: 8.1.2 - ~/.nvm/versions/node/v14.17.6/bin/npm
  Browsers:
    Chrome: 95.0.4638.54
    Firefox: 94.0
  npmPackages:
    gatsby: ^4.1.0 => 4.1.0 
    gatsby-plugin-gatsby-cloud: ^4.1.0 => 4.1.0 
    gatsby-plugin-image: ^2.1.0 => 2.1.0 
    gatsby-plugin-manifest: ^4.1.0 => 4.1.0 
    gatsby-plugin-offline: ^5.1.0 => 5.1.0 
    gatsby-plugin-react-helmet: ^5.1.0 => 5.1.0 
    gatsby-plugin-sharp: ^4.1.0 => 4.1.0 
    gatsby-source-contentful: ^6.1.0 => 6.1.0 
    gatsby-source-filesystem: ^4.1.0 => 4.1.0 
    gatsby-transformer-sharp: ^4.1.0 => 4.1.0 
  npmGlobalPackages:
    gatsby-cli: 4.1.0

Config Flags

No response

[julrich]

We also seem to be hitting this. Especially when using build, only seldomly when running the development version.
But otherwise it's exactly this, localFile being available only sometimes when being accessed. Almost seems like a timing problem? 🤔

Btw. this was something I stumbled upon when last researching this, which seemed relevant:
#33715

[pbizimis]

@julrich did you try to apply the changes of the PR?

[julrich]

It's released already according to the CHANGELOG.md:
https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-contentful/CHANGELOG.md#610-2021-11-02

I suspect we might be hitting a special case not handled correctly (yet?)

[pbizimis]

Yes, I think so as well...

[julrich]

Probably worth mentioning, we're accessing ContentfulAssets ingested by gatsby-source-contentful in our own resolvers (createResolvers):

const image = await context.nodeModel.findOne({
  query: { filter: { id: { eq: source.imageId } } },
  type: "ContentfulAsset",
});

console.log(image.fields.localFile); // this (more often than not) is undefined

That's where we are hitting the missing localFile.

[pbizimis]

For the record, this produces the same bug:

export const query = graphql`
query {
  allContentfulAsset {
    edges {
      node {
        localFile {
          id
        }
      }
    }
  }
}

[dryan-FF]

Would also like to report that we had to downgrade back to Gatsby 3 (and the earlier version of gatsby-source-contentful) because of this issue. We can re-produce the same ways as mentioned above

[afladmark]

As with @julrich we also use Contentful Assets local download option via a custom resolver (to enable custom localization fallbacks to English images). In Gatsby Cloud, we get empty localFile nodes 100% of the time. On local develop it mostly/often works and with local builds it often succeeds. After a gatsby clean it generally fails. We couldn't build in any case until we moved the resolvers from createResolvers to createSchemaCustomization via buildObjectType, but we're basically dead in the water with Gatsby 4 at this point as well due to this issue. We have updated all our logic to reflect the changes introduced in the fix PR as well. Prior to that it was completely broken in all environments.

[axe312ger]

Thank you for the reports, I'll have a look.

Relevant file: https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-contentful/src/download-contentful-assets.js

@wardpeet this could be related to our switch to fetchRemoteFile

[axe312ger]

If somebody has public build logs that failed available, that could help! Please link them here.

I run into this error while developing the latest versions, a rerun solved it :/

https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-contentful/src/download-contentful-assets.js

[axe312ger]

I tried your repo, it works fine for me :/

Build it multiple times, cold and warm builds.

{
  "allContentfulAsset": {
    "edges": [
      {
        "node": {
          "localFile": {
            "id": "08550857-65d4-5dd2-80e6-f9354a767b37"
          }
        }
      },
      {
        "node": {
          "localFile": {
            "id": "2d6f617d-154d-5652-ac29-dd66b7c78207"
          }
        }
      },
      {
        "node": {
          "localFile": {
            "id": "5e84f8bf-5474-561f-b8e5-7b691cb746fd"
          }
        }
      }
    ]
  }
}

[axe312ger]

A question to everybody having this issue: Do you use the PRESERVE_FILE_DOWNLOAD_CACHE flag?

[julrich]

At least for us we have not set that flag. So I guess we would be using the default value for it.
I'll try to extract some more details over the weekend, not sure I'll be able to before that. Unfortunately our build logs are not public.

[axe312ger]

Thank you @julrich

---

For everyone running into this issue:

This bug seems not to happen on cold builds (after gatsby clean) and might be related to incremental or warm builds.

We test this feature in our e2e tests and if they would be flaky/unreliable, I guess we would have seen this in all the PRs Gatsby does on a daily basis.

On my side, or my machine, I have trouble reproducing the issue. So any information might help.

Could you please give me details in which situations this happens to you and on what platform?

[axe312ger]

I found a public run that did run into that issue. It is from a dependency update, this really should not fail here:

https://app.circleci.com/pipelines/github/gatsbyjs/gatsby/77629/workflows/66678dbb-6a3f-4684-a7ae-00197a2e3ed9/jobs/915985

#34114

[afladmark]

I've tested this fix via @next on our builds and confirm it resolves our issue. Thank you so much for fixing this @wardpeet! We can finally upgrade to Gatsby 4!