Is the package not using cache properly in our ci runs ? #688

AnthonyLenglet · 2022-08-23T15:37:33Z

AnthonyLenglet
Aug 23, 2022

Hi !

I've been noticing that our ci has been really slow due to a mongodb download happening despite it being seemingly found in the cache folder

Mongodb-Memory-Server* checking MongoDB binaries
Downloading MongoDB "4.4.9": 0% (0mb / 68.7mb)
Downloading MongoDB "4.4.9": 100% (68.7mb / 68.7mb)
Mongodb-Memory-Server* found binary: "/home/runner/.cache/mongodb-binaries/mongod-x64-ubuntu-4.4.9"

PASS src/projects/tests/projects.service.spec.ts (36.746 s)
  ● Console

    console.log
      Downloading MongoDB 4.4.9: 0 % (0mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

    console.log
      Downloading MongoDB 4.4.9: 40.7 % (27.9mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

    console.log
      Downloading MongoDB 4.4.9: 82.3 % ([56](https://github.com/Passiverecords/Oney-API/runs/7977031901?check_suite_focus=true#step:5:57).5mb / [68](https://github.com/Passiverecords/Oney-API/runs/7977031901?check_suite_focus=true#step:5:69).6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

our jest-mongodb-config.ts file currently looks like this

export default {
  mongodbMemoryServerOptions: {
    binary: {
      version: '4.4.9',
      skipMD5: true,
    },
    autoStart: false,
    instance: {},
  },
};

is somethign wrong with the way it is currently set up ?

Answered by AnthonyLenglet

Aug 24, 2022

I spent some time updating all the packages in our project and was able to update @shelf/jest-mongodb and remove the duplicated mongodb-memory-server, and the issue is no longer happening !

I originally kept both mongodb-memory-server and @shelf/jest-mongodb in order to move the tests from one to the other one by one over time, but I just ended up moving everything else over now

View full answer

hasezoey · 2022-08-23T16:54:50Z

hasezoey
Aug 23, 2022
Maintainer

weird that it does not use the downloaded binary but needs to re-download it in the tests
could you provide:

what ci are you using (github actions / gitlab runner / travis, etc)
where are you applying mongodbMemoryServerOptions to? (because this is not read normally by MMS itself)

I've been noticing that our ci has been really slow due to a mongodb download happening despite it being seemingly found in the cache folder

from what i know, most ci runners dont cache ~/.cache by default, and would need to be done by the user (though in MMS's case i would recommend to explicitly set the DOWNLOAD_DIR config option and cache that)

0 replies

AnthonyLenglet · 2022-08-24T09:14:40Z

AnthonyLenglet
Aug 24, 2022
Author

We're using github actions to run the ci with the following steps:

steps:
      - name: Load repository
        uses: actions/checkout@master
        with:
          persist-credentials: false
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 12
          cache: 'npm'
        env:
          REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
      - name: Run tests
        timeout-minutes: 30
        env:
          REGISTRY_TOKEN: ${{ secrets.REGISTRY_TOKEN }}
        run: |
          npm ci
          npm run test:cov

As for mongodbMemoryServerOptions, this is coming from @shelf/jest-mongodb, a jest preset for this package

from what i know, most ci runners dont cache ~/.cache by default, and would need to be done by the user (though in MMS's case i would recommend to explicitly set the DOWNLOAD_DIR config option and cache that)

I wonder if something about this changed, since re-running the job for the CI still tells me that the binary was found in ~/.cache, despite me not having set DOWNLOAD_DIR, maybe github action is one of the few that caches it ?

1 reply

hasezoey Aug 24, 2022
Maintainer

since re-running the job for the CI still tells me that the binary was found in ~/.cache, despite me not having set DOWNLOAD_DIR

if the job is not based on the output of another job, then this is weird, because from what i know github actions does not cache anything by default, though environments may be re-used on re-running (though i am unsure about that)

despite me not having set DOWNLOAD_DIR

i just mentioned to set DOWNLOAD_DIR when caching to have a explicit folder set to cache, even if the default may change in MMS

AnthonyLenglet · 2022-08-24T09:45:01Z

AnthonyLenglet
Aug 24, 2022
Author

Quick update, I found this issue when attempting to update our current CI setup, and I just found out that the issue only happens when using npm ci instead of npm install

with npm install:

> [email protected] postinstall /home/runner/work/<PROJECT>/node_modules/@shelf/jest-mongodb/node_modules/mongodb-memory-server
> node ./postinstall.js

Mongodb-Memory-Server* checking MongoDB binaries
Downloading MongoDB "4.4.9": 0% (0mb / 68.7mb)
Downloading MongoDB "4.4.9": 100% (68.7mb / 68.7mb)
Mongodb-Memory-Server* found binary: "/home/runner/.cache/mongodb-binaries/mongod-x64-ubuntu-4.4.9"

> @nestjs/[email protected] postinstall /home/runner/work/<PROJECT>/node_modules/@nestjs/core
> opencollective || exit 0


> [email protected] postinstall /home/runner/work/<PROJECT>/node_modules/husky
> opencollective-postinstall || exit 0


> [email protected] postinstall /home/runner/work/<PROJECT>/node_modules/mongodb-memory-server
> node ./postinstall.js

mongodb-memory-server: checking MongoDB binaries cache...
Downloading MongoDB 4.4.9: 0 % (0mb / 68.6mb)
mongodb-memory-server: binary path is /home/runner/.cache/mongodb-binaries/4.4.9/mongod

...

PASS src/projects/tests/projects.service.spec.ts (22.532 s)

with npm ci:

> [email protected] postinstall /home/runner/work/<PROJECT>/node_modules/mongodb-memory-server
> node ./postinstall.js

mongodb-memory-server: checking MongoDB binaries cache...
Downloading MongoDB 4.4.9: 0 % (0mb / 68.6mb)
mongodb-memory-server: binary path is /home/runner/work/<PROJECT>/node_modules/.cache/mongodb-memory-server/mongodb-binaries/4.4.9/mongod

> [email protected] install /home/runner/work/<PROJECT>/node_modules/husky
> node husky install

husky > Setting up git hooks
CI detected, skipping Git hooks installation.
husky > Done

> [email protected] postinstall /home/runner/work/<PROJECT>/node_modules/husky
> opencollective-postinstall || exit 0


> [email protected] postinstall /home/runner/work/<PROJECT>/node_modules/@shelf/jest-mongodb/node_modules/mongodb-memory-server
> node ./postinstall.js

Mongodb-Memory-Server* checking MongoDB binaries
Downloading MongoDB "4.4.9": 0% (0mb / 68.7mb)
Downloading MongoDB "4.4.9": 100% (68.7mb / 68.7mb)
Mongodb-Memory-Server* found binary: "/home/runner/.cache/mongodb-binaries/mongod-x64-ubuntu-4.4.9"

...

PASS src/projects/tests/projects.service.spec.ts (35.925 s)
  ● Console

    console.log
      Downloading MongoDB 4.4.9: 0 % (0mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

    console.log
      Downloading MongoDB 4.4.9: 41.5 % (28.5mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

    console.log
      Downloading MongoDB 4.4.9: 84.4 % (57.9mb / 68.6mb)

      at MongoBinaryDownload.Object.<anonymous>.MongoBinaryDownload.printDownloadProgress (node_modules/mongodb-memory-server-core/src/util/MongoBinaryDownload.ts:424:15)

the cache path is /home/runner/work/<PROJECT>/node_modules/.cache in npm ci and /home/runner/.cache in npm install, yet it seems to give out Mongodb-Memory-Server* found binary: "/home/runner/.cache/mongodb-binaries/mongod-x64-ubuntu-4.4.9" in both cases ? I wonder if its a cache coming from the previous job runs ?

1 reply

hasezoey Aug 24, 2022
Maintainer

[email protected] postinstall /home/runner/work//node_modules/mongodb-memory-server
[email protected] postinstall /home/runner/work//node_modules/@shelf/jest-mongodb/node_modules/mongodb-memory-server

you seem to have 2 different versions installed and set-up, try running npm ls mongodb-memory-server or yarn why mongodb-memory-server and try to resolve only using one version

also 8.9.0 is currently the latest version and those old versions had different resolving and binary naming schemes

Note: i dont know what version of @shelf/jest-mongodb your use, but @shelf/[email protected] supports MMS 8.8.0 from what i can tell

AnthonyLenglet · 2022-08-24T17:58:07Z

AnthonyLenglet
Aug 24, 2022
Author

I spent some time updating all the packages in our project and was able to update @shelf/jest-mongodb and remove the duplicated mongodb-memory-server, and the issue is no longer happening !

I originally kept both mongodb-memory-server and @shelf/jest-mongodb in order to move the tests from one to the other one by one over time, but I just ended up moving everything else over now

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is the package not using cache properly in our ci runs ? #688

{{title}}

Replies: 4 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Is the package not using cache properly in our ci runs ? #688

AnthonyLenglet Aug 23, 2022

Replies: 4 comments · 2 replies

hasezoey Aug 23, 2022 Maintainer

AnthonyLenglet Aug 24, 2022 Author

hasezoey Aug 24, 2022 Maintainer

AnthonyLenglet Aug 24, 2022 Author

hasezoey Aug 24, 2022 Maintainer

AnthonyLenglet Aug 24, 2022 Author

AnthonyLenglet
Aug 23, 2022

Replies: 4 comments 2 replies

hasezoey
Aug 23, 2022
Maintainer

AnthonyLenglet
Aug 24, 2022
Author

hasezoey Aug 24, 2022
Maintainer

AnthonyLenglet
Aug 24, 2022
Author

hasezoey Aug 24, 2022
Maintainer

AnthonyLenglet
Aug 24, 2022
Author