gRPC Issues with concurrent task creation - DEADLINE_EXCEEDED

[dschoeni]

Environment details

• OS: Google Cloud Run, runs as node:12 container
• Node.js version: 12
• npm version: 6?
• @google-cloud/tasks version: "2.1.0"

Steps to reproduce

1. Have Google Cloud Run scale your containers and concurrently create tasks
2. Use the SDK as provided

Our exact code:

 try {
      client = new CloudTasksClient()
      const parent = client.queuePath(project, location, queue)
      await client.createTask({ parent, task })
      await client.close()
    } catch (e) {
      /*
       * As we now properly "close()" the client, it is possible to not see this error
       * in the logs anymore. if so, remove the following fallback code. -ps
       */
      Sentry.captureException(e)

      if (client) {
        await client.close()
        client = undefined
      }

      if (e instanceof Error && e.message === '4 DEADLINE_EXCEEDED: Deadline exceeded') {
        Sentry.captureMessage('got 4 DEADLINE_EXCEEDED: Deadline exceeded, try fallback')
        client = new CloudTasksClient({
          fallback: true
        })
        const parent = client.queuePath(project, location, queue)
        await client.createTask({ parent, task })
      }
    }

Throws the following errors at random:

got 4 DEADLINE_EXCEEDED: Deadline exceeded, try fallback

Whats worst: The tasks get created nonetheless sometimes, even if we get DEADLINE_EXCEEDED - Which became apparant because our fallback solution managed to insert duplicate tasks in the queue.

We do not create a lot of tasks at the same time, maybe 15-20 in short bursts, but the error pops up consistently. I can share our exact project id via e-mail if something at Google would like to take a look.

This issue relates to #397

[pattishin]

Thanks @dschoeni for submitting this issue! With v2.1.0, I was able to successfully re-create the error above (Error: 4 DEADLINE_EXCEEDED: Deadline exceeded).

Since you have @google-cloud/[email protected] currently installed, would you be able to bump your version to the latest 2.5.0 or, at minimum, v2.3.0 and verify if this is still an issue for you.

d04ef73

[dschoeni]

Thanks for taking a look at this! We managed to be able to handle the issue by sharing a CloudTasksClient between requests, and incidentally also bumped the version to v2.5.0 a bit later and haven't seen this issue since, so I'd consider this as resolved 👍

[modestaspruckus]

Still throwing error on "@google-cloud/tasks": "^2.5.0",

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
    at Object.callErrorFromStatus (/Users/modestaspruckus/projects/backend/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/Users/modestaspruckus/projects/backend/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:180:52)
    at Object.onReceiveStatus (/Users/modestaspruckus/projects/backend/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141)
    at Object.onReceiveStatus (/Users/modestaspruckus/projects/backend/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
    at /Users/modestaspruckus/projects/backend/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:182:78
    at processTicksAndRejections (internal/process/task_queues.js:77:11)
    at runNextTicks (internal/process/task_queues.js:64:3)
    at listOnTimeout (internal/timers.js:526:9)
    at processTimers (internal/timers.js:500:7) {
  code: 4,
  details: 'Deadline exceeded',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

[willwillems]

For people still running into this issue, we were on 2.5.0 and still experiencing this issue. Batching the request like this fixed the issue:

   // ABSOLUTE JANK CODE :(
   const chunks = chunk(records, 512) // lodash
   for (let i = 0; i < chunks.length; i++) { 
      const work = chunks[i].map(record => ourTaskCreationFunction({
         id: record.is
      }))
      await Promise.all(work)
   }

We're now able to "successfully" add +30k tasks to the queue.

The worst thing about this was definitely that we had no idea what google product in our stack was causing this for Firestore, Datastore, and some other libs can throw identical stack traces.

[jorgemgr94]

People who are still running into this issue, consider the workaround mentioned by @yossi-eynav on #397 (comment)
This is what you will be enabling: #397 (comment)

Currently, we're enqueuing batches of 500 tasks concurrently without any problem 👍