feat(firestore): [PQ] Add all Aggregation and Timestamp functions#13194
Conversation
d79c2d4 to
f16ab71
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request completes the implementation of private preview aggregation and timestamp functions for Firestore pipelines. It adds several new functions like Sum, Average, Count, Maximum, Minimum, various timestamp manipulation functions, and their corresponding integration tests. The changes are well-structured, and the addition of tests is great. I've found a few minor issues, mostly related to documentation copy-paste errors in the new aggregate functions and some leftover code in tests. There is also one change that looks like a temporary debug modification that should probably be reverted before merging.
024e4a7 to
9b67878
Compare
hongalex
left a comment
There was a problem hiding this comment.
LGTM for Go, want approval for firestore logic
f1df882
into
googleapis:feature/fs-pipeline-queries
) 1. add all the private preview 'array' functions. Merging this PR completes the implementation of all the **type: "Function" subType : "Array"** private preview features (except 'maximum' and 'minimum' which are not yet implemented in any of the SDKs. Requires additonal approvals from Firestore team and will be added to separate PR). See "Firestore Features (Pipeline)" sheet in [go/firestore-query-tracker](http://go/firestore-query-tracker) for the list of features. Java reference: - https://github.com/googleapis/java-firestore/blob/wuandy/JavaPplPP/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java 2. add all the private preview 'string' functions. (except 'string_split' which is not yet implemented in any of the SDKs. Requires additonal approvals from Firestore team and will be added to separate PR) 3. add all the private preview 'vector' functions. 4. add remaining types to ConstantOf to match Java's implementation. Java reference: - https://github.com/googleapis/java-firestore/blob/ccaf9d4fac5bd87a4da3d37493ca66fdc7681bc3/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L70-L211 Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194
b/364927702 - toExprOrField was renamed to asFieldExpr in https://github.com/googleapis/google-cloud-go/pull/13194/files#diff-4a55211f7d38a1f0599e2f4cc92795073f138b2c56b846c933bda19e26bc3a7a . There were a few call locations were rename was missed while resolving merge conflicts which caused build failures. Fixing those failures in this PR. - the function signature of Data was changed in #13218. It no longer returns err as second argument. Fixing this in this PR. - remove duplicate asInt64Expr and asStringExpr - Move pipeline tests to their own file Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245
b/364927702 - Combine FieldOf and FieldOfPath to avoid verbose name FieldOfPath Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
1. Move PipelineStages integration tests. 2. Remove IsNaN, IsNotNaN, IsNull, IsNotNull, Equivalent as they are no longer supported by backend 3. Remove examples as commented here #13245 (comment) Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271
Add raw stage similar to Java https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/Pipeline.java#L997-L1021 The raw stage is an escape hatch to allow customers to consume new stages supported by the backend without having to update their SDK to a version that adds the stage. Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271 - #13279
Add consistency selector similar to existing requests Existing code for reference: **client.go :** https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/client.go#L405-L412 https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/client.go#L308-L317 https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/client.go#L490-L504 **query.go :** https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/query.go#L1406-L1412 https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/query.go#L1551-L1561 **list_documents.go :** https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/list_documents.go#L48-L55 Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271 - #13279 - #13280 - #13281
1. Move PipelineStages integration tests. 2. Remove IsNaN, IsNotNaN, IsNull, IsNotNull, Equivalent as they are no longer supported by backend 3. Remove examples as commented here #13245 (comment) Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271
Add raw stage similar to Java https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/Pipeline.java#L997-L1021 The raw stage is an escape hatch to allow customers to consume new stages supported by the backend without having to update their SDK to a version that adds the stage. Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271 - #13279
Add consistency selector similar to existing requests Existing code for reference: **client.go :** https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/client.go#L405-L412 https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/client.go#L308-L317 https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/client.go#L490-L504 **query.go :** https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/query.go#L1406-L1412 https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/query.go#L1551-L1561 **list_documents.go :** https://github.com/googleapis/google-cloud-go/blob/66cc9bb6e158416897af1d1dc4b9001118db3373/firestore/list_documents.go#L48-L55 Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271 - #13279 - #13280 - #13281
#13283) Changes in this PR: 1. firestore_client.go : Updated generated client as per googleapis/gapic-generator-go#1661 . Removed retries from tests since the headers have now been fixed. 2. Remove Equivalent since it was removed from backend. 3. Add/update comments 4. Add timestamp truncate (pending from #13194) and string split (pending from #13245) functions. 5. add all the private preview general, key, logical (except iferror), type and object functions. See "Firestore Features (Pipeline)" sheet in [go/firestore-query-tracker](http://go/firestore-query-tracker) for the list of functions. Java reference: - https://github.com/googleapis/java-firestore/blob/ccaf9d4fac5bd87a4da3d37493ca66fdc7681bc3/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271 - #13279 - #13280 - #13281 - #13282 - googleapis/gapic-generator-go#1661
add collection and collectiongroup stage options similar to Java https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/stages/CollectionGroupOptions.java#L34-L36 https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/stages/CollectionOptions.java#L34-L36 https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/stages/CollectionHints.java#L34-L40 Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271 - #13279 - #13280 - #13282 - googleapis/gapic-generator-go#1661
Add ExecuteOptions similar to query run options which was introduced in #10164. GetRawData and GetText implementation similar to Java https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/stages/PipelineExecuteOptions.java https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/stages/ExplainOptions.java https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/ExplainStats.java#L42-L71 https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/test/java/com/google/cloud/firestore/it/ITPipelineTest.java#L2474-L2498 Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271 - #13279 - #13280 - #13281 - #13282 - googleapis/gapic-generator-go#1661 - #13283
Add CreateFrom() and Pipeline() similar to Java. https://github.com/googleapis/java-firestore/blob/742fab6583c9a6f9c47cf0496124c3c9b05fe0ee/google-cloud-firestore/src/main/java/com/google/cloud/firestore/PipelineSource.java#L140-L164 Previous pull requests - #12217 - #12425 - #12538 - #13147 - #13199 - #13218 - #13194 - #13245 - #13270 - #13271 - #13279 - #13280 - #13281 - #13282 - googleapis/gapic-generator-go#1661 - #13283 - #13274 - #13338
b/364927702
add all the remaining private preview aggregate functions. Merging this PR completes the implementation of all the type: "Function" subType : "Accumulators (Aggregation)" private preview features.
See "Firestore Features (Pipeline)" sheet in go/firestore-query-tracker for the list of features.
Java reference:
add all the remaining private preview timestamp functions. Merging this PR completes the implementation of all the type: "Function" subType: "Date / Timestamp" private preview features. (except timestamp_trunc function which is not yet inmplemented in any of the SDKs. Requires additonal approvals from Firestore team and will be added to separate PR).
See "Firestore Features (Pipeline)" sheet in go/firestore-query-tracker for the list of functions.
Java reference:
Add integration tests for all functions.
Remove Rand function since it is not targeted for private preview.
Renamed numericExprOrField to numericExprOrFieldPath since field is a separate type/expression.
google-cloud-go/firestore/pipeline_field.go
Lines 21 to 41 in a3ee1f1
Previous pull requests