Using Firestore, gRPC binary dependency is massive which is causing issues in AWS Lambda environment

[patgoley]

[READ] Step 1: Are you in the right place?

The issue here is related to gRCP binary dependency of Firebase Admin / Firestore which happens to be very large. I could open an issue with gRPC about reducing code size, but I think the issue lies with Firebase Admin/Firestore for including a large binary dependency when the Firestore web SDK doesn't have to and achieves roughly the same functionality.

[REQUIRED] Step 2: Describe your environment

• Operating System version: Amazon Linux (installed with npm i --target_arch=x64 --target_platform=linux --target_libc=glibc)
• Firebase SDK version: firebase-admin v7.0.0
• Firebase Product: Firestore

[REQUIRED] Step 3: Describe the problem

npm dependencies for Firebase Admin SDK total over 84MB, mainly caused by the gRPC binary which alone is 43MB. I am using Firestore, so I can't opt out of loading gRPC. This causes an extremely slow start time for serverless functions (specifically AWS) that must download and unzip the whole package on cold start. Currently seeing cold start response time of over 15 seconds. >80MB is not a reasonable size for a single dependency and 15 seconds is not a reasonable response time, so the Firestore SDK does not seem usable in production for anyone in a serverless environment.

Steps to reproduce:

1. Install firebase-admin ^7.0.0 as sole dependency. From package.json:

"dependencies": {
    "firebase-admin": "^7.0.0"
  }

2. Install SDK building for Amazon Linux (required for running in Lambda functions).

npm i --target_arch=x64 --target_platform=linux --target_libc=glibc --production

3. Inspect size of resulting node_modules

It's not clear to me why the admin SDK needs a gRPC binary when the Firestore client for web does not require one since it has to run in the browser. Looking at my web project, grpc is resolved to this file: https://registry.npmjs.org/grpc/-/grpc-1.17.0.tgz, which is only 4.3MB. I don't have the full picture but it seems like the admin SDK could be using this more compact version and not load the 43MB binary.

Until this code size issue is resolved, Firestore is essentially unusable in serverless environments. Presumable Firebase Cloud Functions come with this dependency pre-installed (hopefully), but I can't simply migrate away from AWS entirely as I'm reliant on some of their other services.

Thank you and let me know if I can provide any more information.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[patgoley]

After some more research, I found the reason for the discrepancy between web and admin versions of gRPC:
https://grpc.io/blog/state-of-grpc-web

Regardless, the issue still stands that firebase-admin is too large to run in a serverless environment.

[hiranya911]

GRPC is a transitive dependency for firebase-admin. The dependency chain is something like: firebase-admin -> @google-cloud/firestore -> google-gax -> grpc.

So unfortunately this problem cannot be addresses in this repo. I'd advise you to report this at https://github.com/googleapis/nodejs-firestore. And fwiw, there's an ongoing effort to use a pure JS grpc implementation in Firestore. That should eventually reduce the size of this dependency significantly (@schmidt-sebastian for more comments).

A few other clarifications:

• Google Cloud Functions do not re-install packages on every cold start. Package installation only happens once, at the initial deployment. That's why this issue doesn't impact GCP developers.
• The package you've linked to extracts to a directory that occupies around 24 MB. When installed into a Node.js environment, it will install a couple more dependencies that takes about 31 MB total:

$ curl -O https://registry.npmjs.org/grpc/-/grpc-1.18.0.tgz
$  ls -lh
total 8352
-rw-r--r--  1 hkj  eng   4.1M Mar  6 13:39 grpc-1.18.0.tgz
$ tar xvf grpc-1.18.0.tgz 
$ du -sh package/
23M	package/

$ npm install grpc
$ du -sh node_modules/grpc/
 24M	node_modules/grpc/
$ du -sh node_modules/
 31M	node_modules/

So we are already using the same GRPC implementation you've pointed to. It's just larger on the disk once extracted.

[patgoley]

Thanks for the clarification. Sorry for the incorrect size of the tgz for gRPC for web, indeed it does inflate to ~24MB as you've said. I will recreate this issue on the firestore repository, but I do have one remaining question for you:

The gRCP module as installed with the firestore web SDK (the tgz linked above) still contains native code (e.g. boringssl). Are these compiled to web assembly to run in the browser? I would expect that the web would use a pure JS implementation, though probably relying on browser capabilities (e.g. fetch) not available in a Node.js environment.

Thanks again for your time!

[schmidt-sebastian]

We are very close to abandoning the GRPC C-core in favor of GRPC JS. We need to switch our clients to Node 8 before that happens, but in the not so distant feature, the size of the Firestore Server SDK should drop significantly.

Furthermore, the Web Client SDK uses GRPC for its Node implementation. We do not ship GRPC as part of our browser packages.

I am closing this issue since this transition will be handled by a different team.

[zkiiito]

@schmidt-sebastian where could one track the progress of this transition?

[hiranya911]

Not sure if there's an open bug to track this. But just to give an update, we are working on 2 action items to make this happen:

1. Remove gRPC C Core (expected size reduction: ~40MB)
   1.1 Drop Node 6 support
   1.2 Drop gRPC C Core from google-gax
2. Remove unused proto files from google-gax (expected size reduction: ~10MB)

Item 2 is already committed to google-gax: googleapis/gax-nodejs#470

[mkonikov]

Any update on item #1? :)

[hiranya911]

@mkonikov Firestore switched out the gRPC C core quite a while ago. All version since v8.0.0 contains this optimization: https://packagephobia.now.sh/result?p=firebase-admin

[mkonikov]

Aha - nice! Any idea on that happening for the firebase standard package?

[mkonikov]

Found this issue - firebase/firebase-js-sdk#1783

[schmidt-sebastian]

The firebase package will transition to @grpc/grpc-js with our next major release.