Optimize bloom filter application #6992

milaGGL · 2023-01-31T00:09:33Z

clean code optimization:

Validate base64 string in decodeBase64 .
normalizeByteString calls decodeBase64 from '../platform/base64', where we decide which decode function should be used based on the platform(node, browser, rm).
Do the validation here while calling decode function.
Serialize document ResourcePath into a full document path string.
To get the full path, we need databaseId.
Add getDatabaseId to the WatchChangeAggregator.metadataProvider
Add canonical string function to DatabaseId.

changeset-bot · 2023-01-31T00:09:36Z

⚠️ No Changeset found

Latest commit: 7f5cfe7

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

google-oss-bot · 2023-01-31T18:40:08Z

Size Report ¹

Affected Products

`@firebase/firestore`

Type	Base (`c128eaa`)	Merge (`9005380`)	Diff
browser	276 kB	277 kB	+986 B (+0.4%)
esm5	343 kB	344 kB	+1.04 kB (+0.3%)
main	551 kB	551 kB	+186 B (+0.0%)
module	276 kB	277 kB	+986 B (+0.4%)
react-native	276 kB	277 kB	+1.00 kB (+0.4%)

`@firebase/firestore-lite`

Type	Base (`c128eaa`)	Merge (`9005380`)	Diff
browser	86.2 kB	86.6 kB	+394 B (+0.5%)
esm5	104 kB	105 kB	+430 B (+0.4%)
main	147 kB	147 kB	-44 B (-0.0%)
module	86.2 kB	86.6 kB	+394 B (+0.5%)
react-native	86.4 kB	86.8 kB	+411 B (+0.5%)

@firebase/util
Type Base (c128eaa) Merge (9005380) Diff
browser 22.9 kB 23.1 kB +159 B (+0.7%)
esm5 24.4 kB 24.7 kB +315 B (+1.3%)
main 30.0 kB 30.4 kB +443 B (+1.5%)
module 22.9 kB 23.1 kB +159 B (+0.7%)

`bundle`

43 size changes

Type	Base (`c128eaa`)	Merge (`9005380`)	Diff
analytics (logEvent)	43.0 kB	43.1 kB	+93 B (+0.2%)
app-check (CustomProvider)	36.1 kB	36.2 kB	+93 B (+0.3%)
app-check (ReCaptchaEnterpriseProvider)	38.4 kB	38.5 kB	+95 B (+0.2%)
app-check (ReCaptchaV3Provider)	38.4 kB	38.5 kB	+95 B (+0.2%)
auth (Anonymous)	67.4 kB	67.5 kB	+95 B (+0.1%)
auth (EmailAndPassword)	71.5 kB	71.6 kB	+95 B (+0.1%)
auth (GoogleFBTwitterGitHubPopup)	93.9 kB	94.0 kB	+94 B (+0.1%)
auth (GooglePopup)	91.1 kB	91.2 kB	+95 B (+0.1%)
auth (GoogleRedirect)	91.4 kB	91.5 kB	+95 B (+0.1%)
auth (Phone)	77.6 kB	77.7 kB	+95 B (+0.1%)
database (Append to a list of data)	148 kB	148 kB	+93 B (+0.1%)
database (Filtering data)	147 kB	147 kB	+93 B (+0.1%)
database (Listen for child events)	163 kB	164 kB	+93 B (+0.1%)
database (Listen for value events + Detach listeners)	163 kB	164 kB	+93 B (+0.1%)
database (Listen for value events)	163 kB	163 kB	+93 B (+0.1%)
database (Read data once)	163 kB	163 kB	+93 B (+0.1%)
database (Save data as transactions)	165 kB	165 kB	+93 B (+0.1%)
database (Sort data)	149 kB	149 kB	+93 B (+0.1%)
database (Write data)	147 kB	147 kB	+93 B (+0.1%)
firestore (Persistence)	294 kB	295 kB	+879 B (+0.3%)
firestore (Query Cursors)	233 kB	234 kB	+631 B (+0.3%)
firestore (Query)	231 kB	231 kB	+631 B (+0.3%)
firestore (Read data once)	218 kB	219 kB	+567 B (+0.3%)
firestore (Realtime updates)	220 kB	221 kB	+567 B (+0.3%)
firestore (Transaction)	199 kB	199 kB	+472 B (+0.2%)
firestore (Write data)	198 kB	199 kB	+464 B (+0.2%)
firestore-lite (Query Cursors)	81.3 kB	81.7 kB	+397 B (+0.5%)
firestore-lite (Query)	77.5 kB	77.8 kB	+397 B (+0.5%)
firestore-lite (Read data once)	59.4 kB	59.7 kB	+289 B (+0.5%)
firestore-lite (Transaction)	84.0 kB	84.5 kB	+421 B (+0.5%)
firestore-lite (Write data)	68.9 kB	69.3 kB	+394 B (+0.6%)
functions (call)	30.9 kB	31.0 kB	+100 B (+0.3%)
messaging (send + receive)	46.4 kB	46.5 kB	+95 B (+0.2%)
performance (trace)	50.8 kB	50.9 kB	+95 B (+0.2%)
remote-config (getAndFetch)	45.4 kB	45.4 kB	+93 B (+0.2%)
storage (getBytes)	41.1 kB	41.2 kB	+93 B (+0.2%)
storage (getDownloadURL)	43.2 kB	43.3 kB	+93 B (+0.2%)
storage (getMetadata)	42.7 kB	42.8 kB	+93 B (+0.2%)
storage (list + listAll)	42.1 kB	42.2 kB	+93 B (+0.2%)
storage (updateMetadata)	42.9 kB	43.0 kB	+93 B (+0.2%)
storage (uploadBytes)	48.0 kB	48.1 kB	+93 B (+0.2%)
storage (uploadBytesResumable)	58.0 kB	58.1 kB	+93 B (+0.2%)
storage (uploadString)	48.2 kB	48.3 kB	+93 B (+0.2%)

`firebase`

19 size changes

Type	Base (`c128eaa`)	Merge (`9005380`)	Diff
firebase-app-check-compat.js	22.6 kB	22.7 kB	+101 B (+0.4%)
firebase-app-check.js	21.4 kB	21.5 kB	+104 B (+0.5%)
firebase-app-compat.js	28.8 kB	28.9 kB	+99 B (+0.3%)
firebase-app.js	92.8 kB	93.1 kB	+243 B (+0.3%)
firebase-auth-compat.js	126 kB	126 kB	+101 B (+0.1%)
firebase-auth-cordova.js	138 kB	138 kB	+96 B (+0.1%)
firebase-auth-react-native.js	150 kB	150 kB	+126 B (+0.1%)
firebase-auth.js	117 kB	117 kB	+101 B (+0.1%)
firebase-compat.js	760 kB	761 kB	+1.09 kB (+0.1%)
firebase-database-compat.js	166 kB	166 kB	+105 B (+0.1%)
firebase-database.js	154 kB	154 kB	+100 B (+0.1%)
firebase-firestore-compat.js	334 kB	335 kB	+1.08 kB (+0.3%)
firebase-firestore-lite.js	92.9 kB	93.4 kB	+493 B (+0.5%)
firebase-firestore.js	335 kB	336 kB	+1.09 kB (+0.3%)
firebase-functions.js	10.9 kB	11.0 kB	+91 B (+0.8%)
firebase-performance-standalone-compat.es2017.js	89.9 kB	90.0 kB	+106 B (+0.1%)
firebase-performance-standalone-compat.js	67.0 kB	67.2 kB	+140 B (+0.2%)
firebase-storage-compat.js	41.2 kB	41.3 kB	+101 B (+0.2%)
firebase-storage.js	40.2 kB	40.3 kB	+99 B (+0.2%)

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/g0A7YSqCjU.html

google-oss-bot · 2023-01-31T18:54:29Z

Size Analysis Report ¹

This report is too large (880,227 characters) to be displayed here in a GitHub comment. Please use the below link to see the full report on Google Cloud Storage.

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/auKTH2k6eM.html

dconeybe

Just a few very minor comments.

packages/firestore/src/platform/base64.ts

packages/firestore/src/platform/browser/base64.ts

packages/firestore/src/remote/watch_change.ts

packages/firestore/src/core/database_info.ts

upload initial code

79f3453

milaGGL changed the base branch from master to mila/BloomFilter January 31, 2023 00:09

format

d63e47f

milaGGL self-assigned this Jan 31, 2023

milaGGL added 2 commits January 30, 2023 16:27

add toDocumentFullPath to serializer

25ba839

correct a typo

185a3fa

milaGGL added the work-in-progress label Jan 31, 2023

milaGGL added 2 commits January 31, 2023 10:11

use databaseId instead of serializer

d4882eb

remove unused toDocumentFullPath in serializer

5160dbf

milaGGL requested a review from dconeybe January 31, 2023 18:47

adjust the code to throw Base64DecodeError

4081ead

milaGGL requested review from dwyfrequency and hsubox76 as code owners February 1, 2023 00:08

milaGGL added 3 commits February 1, 2023 10:45

add databaseId to WatchChangeAggregator instead of ExistenceFilterChange

903536b

format

fbac6ef

use getter for databaseId

24ce794

milaGGL mentioned this pull request Feb 1, 2023

Decouple code reformat from boom filter #6994

Closed

milaGGL added 3 commits February 1, 2023 15:48

FORMAT

44b2305

reformat the way we get DatabaseId

aa21221

Merge branch 'mila/BloomFilter' into mila/BloomFilter-optimization

6a9e66a

milaGGL removed the work-in-progress label Feb 2, 2023

milaGGL mentioned this pull request Feb 3, 2023

Apply bloom filter on existence filter mismatch firebase/firebase-android-sdk#4601

Merged

dconeybe requested changes Feb 7, 2023

View reviewed changes

resolve comments

7293289

milaGGL requested a review from dconeybe February 8, 2023 17:30

add error message

7f5cfe7

dconeybe approved these changes Feb 9, 2023

View reviewed changes

milaGGL merged commit 277f8e1 into mila/BloomFilter Feb 9, 2023

milaGGL deleted the mila/BloomFilter-optimization branch February 9, 2023 02:08

dconeybe mentioned this pull request Feb 10, 2023

Improve decodeBase64() to throw on invalid input rather than silently accept it #7019

Merged

firebase locked and limited conversation to collaborators Mar 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize bloom filter application #6992

Optimize bloom filter application #6992

milaGGL commented Jan 31, 2023 •

edited

Loading

changeset-bot bot commented Jan 31, 2023 •

edited

Loading

google-oss-bot commented Jan 31, 2023 •

edited

Loading

`@firebase/firestore`

`@firebase/firestore-lite`

`@firebase/util`

`bundle`

`firebase`

google-oss-bot commented Jan 31, 2023 •

edited

Loading

dconeybe left a comment

Optimize bloom filter application #6992

Optimize bloom filter application #6992

Conversation

milaGGL commented Jan 31, 2023 • edited Loading

changeset-bot bot commented Jan 31, 2023 • edited Loading

⚠️ No Changeset found

google-oss-bot commented Jan 31, 2023 • edited Loading

Size Report 1

Affected Products

@firebase/firestore

@firebase/firestore-lite

@firebase/util

bundle

firebase

Test Logs

google-oss-bot commented Jan 31, 2023 • edited Loading

Size Analysis Report 1

Test Logs

dconeybe left a comment

Choose a reason for hiding this comment

milaGGL commented Jan 31, 2023 •

edited

Loading

changeset-bot bot commented Jan 31, 2023 •

edited

Loading

google-oss-bot commented Jan 31, 2023 •

edited

Loading

Size Report ¹

`@firebase/firestore`

`@firebase/firestore-lite`

`@firebase/util`

`bundle`

`firebase`

google-oss-bot commented Jan 31, 2023 •

edited

Loading

Size Analysis Report ¹