Large number of parallel http requests for larger lock files

[sven-wermers]

Hello,
It seems that the await Promise.all(..) (referenced code below) causes that all packages were treated in parallel. Therefore all http request to the registry happening more or less parallel. Therefore, all http requests to the registry are made more or less in parallel. This causes problems when the number of dependencies reaches a certain limit.

swap-lock-registry/src/npm.ts

Lines 6 to 17 in 4dd0001

	export async function traitNpmLockFile(
	lockFile: string,
	opts: TraitOptions,
	): Promise<void> {
	const lockFileObject = await parseNpmLockFile(lockFile);
	try {
	await Promise.all(
	Object.keys(lockFileObject.packages ?? {})
	.filter((pkg) => !!pkg)
	.map(async (pkg) => {
	await traitPackage(

To solve this issues a function could help which only excutes certain amount task in parallel like the following one:

async function parallel<T>(arr: T[], fct: (x: T) => Promise<void>, workerCnt = 10) {
  const tmp = [...arr];
  const worker = async () => {
    while (tmp.length > 0) {
      await fct(tmp.pop());
    }
  };
  const workers = [];
  for (let i = 0; i < workerCnt; i++) {
    workers.push(worker());
  }
  await Promise.all(workers);
}

The effected code segment could then look like this:

export async function traitNpmLockFile(
  lockFile: string,
  opts: TraitOptions,
): Promise<void> {
  const lockFileObject = await parseNpmLockFile(lockFile);

  try {
    const allPkg = Object.keys(lockFileObject.packages ?? {}).filter((pkg) => !!pkg);
    await parallel(allPkg, async (pkg) => {
      await traitPackage(
        lockFileObject.packages,
        pkg,
        pkg.replace(/^.*node_modules\//g, ''),
        lockFileObject.packages[pkg].from ??
          lockFileObject.packages[pkg].version,
        {
          ...opts,
          lockFile,
        },
      );
    });

Best regards
Sven