Unsuccessful response status code. Request failed with status code 404

[matt-oakes]

I am using the google-auth-library to run v2 Firebase task functions as per the documentation. After I deployed today, I was getting this error everytime another function tried to queue a task:

Error: Unsuccessful response status code. Request failed with status code 404
    at Gaxios._request (/workspace/node_modules/gcp-metadata/node_modules/gaxios/build/src/gaxios.js:141:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async metadataAccessor (/workspace/node_modules/gcp-metadata/build/src/index.js:110:21)
    at async GoogleAuth._GoogleAuth_getUniverseFromMetadataServer (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:777:26)
    at async GoogleAuth.getUniverseDomain (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:186:168)
    at async GoogleAuth.getApplicationDefaultAsync (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:252:42)
    at async GoogleAuth.getClient (/workspace/node_modules/google-auth-library/build/src/auth/googleauth.js:674:17)
    at async getFunctionUrl (/workspace/lib/helpers/taskQueue.js:37:20)
    at async enqueueTask (/workspace/lib/helpers/taskQueue.js:55:23)
    at async Promise.all (index 0)

The issue seems to have been introduced by #1692 (cc @danielbankhead).

It suddenly appeard for me because Firebase doesn't respect your lockfile when deploying. My package.json specified "google-auth-library": "^9.2.0" but I was getting the latest 9.4.0 when deploying. I can resolve this issue by instead fixing my dependency to "google-auth-library": "9.2.0". The error when goes away and I am able to queue tasks correctly.

Environment details

• OS: macOS
• Node.js version: 18.18.2
• npm version: 9.8.1
• google-auth-library version: 9.4.0

[danielbankhead]

@matt-oakes Could you provide the output of npm ls gaxios gcp-metadata?

[UiiProDevs]

We're also seeing this issue when deploying an App Engine instance with Firebase after updating several packages. Manually installing [email protected] as a dependency also worked for us.

Here's our readout of npm ls gaxios gcp-metadata google-auth-library before doing the manual install:

+-- @google-cloud/[email protected]
| +-- @google-cloud/[email protected]
| | +-- [email protected]
| | | `-- [email protected]
| | `-- [email protected] deduped
| `-- [email protected]
|   +-- [email protected]
|   +-- [email protected] deduped
|   `-- [email protected]
|     `-- [email protected]
+-- @google-cloud/[email protected]
| +-- [email protected] deduped
| `-- [email protected]
|   `-- [email protected] deduped
+-- @google-cloud/[email protected]
| `-- @google-cloud/[email protected]
|   `-- [email protected] deduped
`-- [email protected]
  +-- @google-cloud/[email protected]
  | `-- [email protected]
  |   `-- [email protected]
  |     +-- [email protected] deduped
  |     +-- [email protected]
  |     | `-- [email protected] deduped
  |     `-- [email protected]
  |       `-- [email protected] deduped
  `-- @google-cloud/[email protected]
    +-- [email protected]
    `-- [email protected]
      +-- [email protected] deduped
      +-- [email protected]
      | `-- [email protected] deduped
      `-- [email protected]
        `-- [email protected] deduped

[danielbankhead]

@UiiProDevs

Hmmm, I'm not seeing how it is possible to fail here with that setup:

google-auth-library-nodejs/src/auth/googleauth.ts

Lines 314 to 318 in e8966d1

	if (e instanceof GaxiosError && e.status === 404) {
	universeDomain = DEFAULT_UNIVERSE;
	} else {
	throw e;
	}

I see firebase-admin is pulling in outdated dependencies, therefore gaxios != gaxios (5.1.3 vs 6.1.1), which means GaxiosError != GaxiosError, however Auth v9 shouldn't be pulling in the older GaxiosError.

Still investigating...

[danielbankhead]

Finally able to reproduce.

There's a very strange and bizarre issue with node/npm where sometimes, depending on the order of the install, a dependency can require an older, undesired dependency, even when explicitly specified not to. Simple reproduction:

mkdir repro
cd repro
npm init -y

Create index.mjs with the following contents:

import {GoogleAuth} from 'google-auth-library';
import assert from 'assert';

try {
  const auth = await new GoogleAuth().getClient();
  console.log('No problem');
} catch (e) {
  console.log({e});
  assert.equal(e.status, 404);
}

Reproduction:

npm i @google-cloud/logging-winston
npm ls google-auth-library gaxios # lgtm
node index.mjs # No problem, works fine

rm -rf node_modules/ package-lock.json
npm i firebase-admin
npm ls google-auth-library gaxios # everything, _looks_ correct...
node index.mjs # Error, as GaxiosError != GaxiosError

Where GaxiosError != GaxiosError:

google-auth-library-nodejs/src/auth/googleauth.ts

Lines 314 to 318 in e8966d1

	if (e instanceof GaxiosError && e.status === 404) {
	universeDomain = DEFAULT_UNIVERSE;
	} else {
	throw e;
	}

[sebasrobert]

This is breaking our live operation right now, any idea how this will be released?

[danielbankhead]

@sebasrobert this will be released immediately upon approval. This should happen shortly as soon as the team is available. Alternatively, pinning to v9.2.0 of this library should work just fine.

[matt-oakes]

Thanks for the quick turn around. Just to confirm that when I upgrade to 9.4.1 everything works as expected 👍

[deepseed]

This issue broken my API even though my code does not directly use google-auth-library. It manifested as a 404 error when calling any API endpoint that used BigQuery, Pub/Sub or Secret Manager – which I assume use this library behind the scenes.

Installing the v9.2 package fixed the issue for me. (It was not previously in my packages.json.) I have not tested v9.4.1 yet.

[danielbankhead]

After extensive research and debugging I've found this TypeScript issue to be the root cause of this issue:

• https://dannyguo.medium.com/how-to-fix-instanceof-not-working-for-custom-errors-in-typescript-1df978100a27
• instanceof is broken when class extends Error type microsoft/TypeScript#13965 (comment)

[samschurter]

Thanks for getting this fixed so quick!

[SimonSchick]

We just got hit by this with the supposedly fixed version (9.4.1)

Attached is a npm ls of the dependency, can we get a little bit more of an explanation here? This is preventing us from upgrading dependencies.

[SimonSchick]

@danielbankhead for visibility, I can only assume some mix of google packages does not play nicely right now, I cannot create a minimal repro at this time.

[danielbankhead]

@SimonSchick hmmmm, that's very strange as it no longer depends on a particular instance of GaxiosError:

• https://github.com/googleapis/google-auth-library-nodejs/pull/1707/files#diff-535a540500eec341838bd88c84b1795313a8ff9b556bad87ef89fb2fca061630R314

Follow-ups:

• Is the error the exact same error message?
• Could you verify the npm ls is the same as deployed?
  • Note: if a package-lock.json isn't available or use the dependencies can drift between releases (personally, I use npm ci > npm i to ensure the package-lock.json is used).

[SimonSchick]

We do use yarn in this case so and our entire pipeline uses reproducable builds, we had to roll back our deployment after the version update and haven't had the time to investigate again.

[SimonSchick]

FYI this we are still experiencing this, all of our google deps are now consolidated (no duplicated), we are raising this with gcp as well.

[danielbankhead]

FYI this we are still experiencing this, all of our google deps are now consolidated (no duplicated), we are raising this with gcp as well.

Do you mind providing a stack trace?