Skip to content

feat(firestore): [PQ] Support pipeline queries#12347

Merged
bhshkh merged 35 commits intomainfrom
feature/fs-pipeline-queries
Jan 13, 2026
Merged

feat(firestore): [PQ] Support pipeline queries#12347
bhshkh merged 35 commits intomainfrom
feature/fs-pipeline-queries

Conversation

@bhshkh
Copy link
Copy Markdown
Contributor

@bhshkh bhshkh commented May 28, 2025

@product-auto-label product-auto-label Bot added the api: firestore Issues related to the Firestore API. label May 28, 2025
@bhshkh bhshkh force-pushed the feature/fs-pipeline-queries branch from 5806663 to a68c104 Compare June 24, 2025 17:56
bhshkh added 2 commits July 1, 2025 17:21
* feat(firestore): add pipeline queries

* add comments

* remove exists
* feat(firestore): Add rest of the input stages

* correct the method comment to resolve vet failure

* address review comments
@bhshkh bhshkh force-pushed the feature/fs-pipeline-queries branch from a68c104 to f069709 Compare July 1, 2025 17:22
bhshkh added 7 commits July 15, 2025 13:28
* feat(firestore): Add pipeline expressions basic structure

* refactor
…tions (#13147)

b/364927702

Add rest of the arithmetics and comparison functions from
[go/firestore-query-tracker](http://go/firestore-query-tracker)


Previous pull requests:
- #12217
- #12425
- #12538
)

b/364927702

1. Fixes test
```go
--- FAIL: TestPipelineResult_NoResults (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10d40b6]

goroutine 4633 [running]:
testing.tRunner.func1.2({0x13c5240, 0x1ec2150})
	/usr/local/go/src/testing/testing.go:1872 +0x419
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1875 +0x683
panic({0x13c5240?, 0x1ec2150?})
	/usr/local/go/src/runtime/panic.go:783 +0x132
cloud.google.com/go/firestore.(*PipelineResult).Data(0xc00046f380)
	/tmpfs/src/google-cloud-go/firestore/pipeline_result.go:95 +0x96
cloud.google.com/go/firestore.TestPipelineResult_NoResults(0xc000455500)
	/tmpfs/src/google-cloud-go/firestore/pipeline_result_test.go:359 +0x312
testing.tRunner(0xc000455500, 0x157dca0)
	/usr/local/go/src/testing/testing.go:1934 +0x21d
created by testing.(*T).Run in goroutine 1
	/usr/local/go/src/testing/testing.go:1997 +0x9d3
FAIL	cloud.google.com/go/firestore	2.337s
```

```go
=== RUN   TestPipelineResultIterator_GetAll
    pipeline_result_test.go:249: second result id: got 1, want: 2
--- FAIL: TestPipelineResultIterator_GetAll (0.00s)
```
2. Add enterprise database env variable


Previous pull requests:

- #12217
- #12425
- #12538
- #13147
@bhshkh bhshkh marked this pull request as ready for review October 23, 2025 21:31
@bhshkh bhshkh requested review from a team October 23, 2025 21:31
@bhshkh bhshkh added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Oct 23, 2025
@bhshkh bhshkh changed the title feat(firestore): add pipeline queries (#12217) feat(firestore): [NOT READY FOR REVIEW] add pipeline queries (#12217) Oct 23, 2025
@bhshkh
Copy link
Copy Markdown
Contributor Author

bhshkh commented Oct 23, 2025

/gemini help

@gemini-code-assist
Copy link
Copy Markdown
Contributor

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

@bhshkh
Copy link
Copy Markdown
Contributor Author

bhshkh commented Oct 23, 2025

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a significant new feature: pipeline queries. The changes include adding the ExecutePipeline RPC to the client, defining the necessary protobuf messages, and implementing a comprehensive fluent API for building and executing these pipelines. The overall design of the pipeline builder API is solid, using interfaces and a fluent style to provide a good developer experience. However, I've identified a critical area for improvement in the low-level client implementation where routing header construction logic is inefficient and duplicated. Addressing this will improve both performance and maintainability.

Comment on lines +1089 to +1100
routingHeaders := ""
routingHeadersMap := make(map[string]string)
if reg := regexp.MustCompile("projects/(?P<project_id>[^/]+)(?:/.*)?"); reg.MatchString(req.GetDatabase()) && len(url.QueryEscape(reg.FindStringSubmatch(req.GetDatabase())[1])) > 0 {
routingHeadersMap["project_id"] = url.QueryEscape(reg.FindStringSubmatch(req.GetDatabase())[1])
}
if reg := regexp.MustCompile("projects/[^/]+/databases/(?P<database_id>[^/]+)(?:/.*)?"); reg.MatchString(req.GetDatabase()) && len(url.QueryEscape(reg.FindStringSubmatch(req.GetDatabase())[1])) > 0 {
routingHeadersMap["database_id"] = url.QueryEscape(reg.FindStringSubmatch(req.GetDatabase())[1])
}
for headerName, headerValue := range routingHeadersMap {
routingHeaders = fmt.Sprintf("%s%s=%s&", routingHeaders, headerName, headerValue)
}
routingHeaders = strings.TrimSuffix(routingHeaders, "&")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This block of code for building routing headers has several inefficiencies:

  1. Regex Compilation: regexp.MustCompile is called on every request, which is computationally expensive. These regular expressions should be compiled once at the package level and reused.
  2. Redundant Calls: reg.FindStringSubmatch and url.QueryEscape are called multiple times for the same input within the if condition and the block.
  3. Inefficient String Building: Building the query string using fmt.Sprintf in a loop is inefficient. The idiomatic and more performant way to build URL query strings in Go is to use the net/url.Values type and its Encode() method.

This same logic is also duplicated in the restClient implementation. I recommend refactoring this into a single, efficient helper function that is shared between the gRPC and REST clients.

routingHeaders := func() string {
		params := url.Values{}
		if matches := regexp.MustCompile(`projects/([^/]+)`).FindStringSubmatch(req.GetDatabase()); len(matches) > 1 {
			params.Set("project_id", matches[1])
		}
		if matches := regexp.MustCompile(`projects/[^/]+/databases/([^/]+)`).FindStringSubmatch(req.GetDatabase()); len(matches) > 1 {
			params.Set("database_id", matches[1])
		}
		return params.Encode()
	}()

Copy link
Copy Markdown
Contributor Author

@bhshkh bhshkh Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is generated code and cannot be modified manually.

Comment on lines +2055 to +2066
routingHeaders := ""
routingHeadersMap := make(map[string]string)
if reg := regexp.MustCompile("projects/(?P<project_id>[^/]+)(?:/.*)?"); reg.MatchString(req.GetDatabase()) && len(url.QueryEscape(reg.FindStringSubmatch(req.GetDatabase())[1])) > 0 {
routingHeadersMap["project_id"] = url.QueryEscape(reg.FindStringSubmatch(req.GetDatabase())[1])
}
if reg := regexp.MustCompile("projects/[^/]+/databases/(?P<database_id>[^/]+)(?:/.*)?"); reg.MatchString(req.GetDatabase()) && len(url.QueryEscape(reg.FindStringSubmatch(req.GetDatabase())[1])) > 0 {
routingHeadersMap["database_id"] = url.QueryEscape(reg.FindStringSubmatch(req.GetDatabase())[1])
}
for headerName, headerValue := range routingHeadersMap {
routingHeaders = fmt.Sprintf("%s%s=%s&", routingHeaders, headerName, headerValue)
}
routingHeaders = strings.TrimSuffix(routingHeaders, "&")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This logic for constructing routing headers is a duplicate of the implementation in the gRPCClient.ExecutePipeline method. Duplicating code like this makes the software harder to maintain, as any future changes or bug fixes will need to be applied in multiple places.

As suggested in the other comment, this logic should be extracted into a single, shared helper function to improve maintainability and reduce redundancy.

Copy link
Copy Markdown
Contributor Author

@bhshkh bhshkh Jan 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is generated code and cannot be modified manually.

bhshkh and others added 7 commits October 27, 2025 03:26
…3194)

b/364927702

1. add all the remaining private preview aggregate functions. Merging
this PR completes the implementation of all the **type: "Function"
subType : "Accumulators (Aggregation)"** private preview features.
See "Firestore Features (Pipeline)" sheet in
[go/firestore-query-tracker](http://go/firestore-query-tracker) for the
list of features.

    Java reference:
-
https://github.com/googleapis/java-firestore/blob/ccaf9d4fac5bd87a4da3d37493ca66fdc7681bc3/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/AggregateFunction.java#L43-L71
-
https://github.com/googleapis/java-firestore/blob/ccaf9d4fac5bd87a4da3d37493ca66fdc7681bc3/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/AggregateFunction.java#L93-L111

2. add all the remaining private preview timestamp functions. Merging
this PR completes the implementation of all the **type: "Function"
subType: "Date / Timestamp"** private preview features. (except
timestamp_trunc function which is not yet inmplemented in any of the
SDKs. Requires additonal approvals from Firestore team and will be added
to separate PR).
See "Firestore Features (Pipeline)" sheet in
[go/firestore-query-tracker](http://go/firestore-query-tracker) for the
list of functions.

    Java reference:
-
https://github.com/googleapis/java-firestore/blob/ccaf9d4fac5bd87a4da3d37493ca66fdc7681bc3/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L2262-L2517

3. Add integration tests for all functions.
4. Remove Rand function since it is not targeted for private preview.
5. Renamed numericExprOrField to numericExprOrFieldPath since field is a
separate type/expression.

https://github.com/googleapis/google-cloud-go/blob/a3ee1f19068c6d3fb77ad797e29884a90d6402a2/firestore/pipeline_field.go#L21-L41



Previous pull requests

- #12217
- #12425
- #12538
- #13147
- #13199
- #13218
…13218)

b/364927702

1. add all the remaining private preview stages. Merging this PR
completes the implementation of all the type "Stage" subType "General"
private preview features. (except literals stage which is not yet
inmplemented in Java and Node. Requires additonal approvals from
Firestore team and will be added to separate PR).
See "Firestore Features (Pipeline)" sheet in
[go/firestore-query-tracker](http://go/firestore-query-tracker)
2. Add integration and unit tests.
3. Refactor existing stages code to remove duplicated code and rearrange
in alphabetical order.
4. Modify behaviour of Data and DataTo to match existing implementation
in document.go

https://github.com/googleapis/google-cloud-go/blob/9a4cb31f4d34948404d91123fb560a43aeebe83e/firestore/document.go#L64-L127



Previous pull requests

- #12217
- #12425
- #12538
- #13147
- #13199
)

1. add all the private preview 'array' functions. 
Merging this PR completes the implementation of all the **type:
"Function" subType : "Array"** private preview features (except
'maximum' and 'minimum' which are not yet implemented in any of the
SDKs. Requires additonal approvals from Firestore team and will be added
to separate PR). See "Firestore Features (Pipeline)" sheet in
[go/firestore-query-tracker](http://go/firestore-query-tracker) for the
list of features.
    Java reference: 
-
https://github.com/googleapis/java-firestore/blob/wuandy/JavaPplPP/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java
2. add all the private preview 'string' functions. (except
'string_split' which is not yet implemented in any of the SDKs. Requires
additonal approvals from Firestore team and will be added to separate
PR)

3. add all the private preview 'vector' functions. 
4. add remaining types to ConstantOf to match Java's implementation.
    Java reference: 
-
https://github.com/googleapis/java-firestore/blob/ccaf9d4fac5bd87a4da3d37493ca66fdc7681bc3/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L70-L211

Previous pull requests

- #12217
- #12425
- #12538
- #13147
- #13199
- #13218
- #13194
b/364927702

- toExprOrField was renamed to asFieldExpr in
https://github.com/googleapis/google-cloud-go/pull/13194/files#diff-4a55211f7d38a1f0599e2f4cc92795073f138b2c56b846c933bda19e26bc3a7a
. There were a few call locations were rename was missed while resolving
merge conflicts which caused build failures. Fixing those failures in
this PR.
- the function signature of Data was changed in
#13218. It no longer
returns err as second argument. Fixing this in this PR.
- remove duplicate asInt64Expr and asStringExpr
- Move pipeline tests to their own file


Previous pull requests

- #12217
- #12425
- #12538
- #13147
- #13199
- #13218
- #13194
- #13245
b/364927702

- Combine FieldOf and FieldOfPath to avoid verbose name FieldOfPath


Previous pull requests

- #12217
- #12425
- #12538
- #13147
- #13199
- #13218
- #13194
- #13245
- #13270

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Nov 6, 2025

⚠️ Breaking change detected in pre-GA module: firestore

The apidiff check has detected one or more breaking changes in this module for pre-GA APIs.

Pre-GA breaking changes do not block the pull request, but are flagged here for review.

For any pre-GA breaking change that needs investigation, open a new issue containing a link to this comment, then you may proceed with reviewing and merging this PR.

- ./apiv1/firestorepb.FirestoreClient.ExecutePipeline: added
- ./apiv1/firestorepb.FirestoreServer.ExecutePipeline: added

#13283)

Changes in this PR:

1. firestore_client.go : Updated generated client as per
googleapis/gapic-generator-go#1661 . Removed
retries from tests since the headers have now been fixed.
2. Remove Equivalent since it was removed from backend.
3. Add/update comments
4. Add timestamp truncate (pending from
#13194) and string
split (pending from
#13245) functions.
5. add all the private preview general, key, logical (except iferror),
type and object functions.
See "Firestore Features (Pipeline)" sheet in
[go/firestore-query-tracker](http://go/firestore-query-tracker) for the
list of functions.
    Java reference:
-
https://github.com/googleapis/java-firestore/blob/ccaf9d4fac5bd87a4da3d37493ca66fdc7681bc3/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java



Previous pull requests

- #12217
- #12425
- #12538
- #13147
- #13199
- #13218
- #13194
- #13245
- #13270
- #13271
- #13279
- #13280
- #13281
- #13282
- googleapis/gapic-generator-go#1661
@bhshkh bhshkh requested a review from a team as a code owner November 19, 2025 14:18
header-check fails on
#12347 with error

firestore/pipeline.go should have a copyright year of 2026
firestore/pipeline_aggregate.go should have a copyright year of 2026
firestore/pipeline_constant.go should have a copyright year of 2026
firestore/pipeline_expression.go should have a copyright year of 2026
firestore/pipeline_field.go should have a copyright year of 2026
firestore/pipeline_filter_condition.go should have a copyright year of
2026
firestore/pipeline_function.go should have a copyright year of 2026
firestore/pipeline_integration_test.go should have a copyright year of
2026
firestore/pipeline_result.go should have a copyright year of 2026
firestore/pipeline_result_test.go should have a copyright year of 2026
firestore/pipeline_source.go should have a copyright year of 2026
firestore/pipeline_source_test.go should have a copyright year of 2026
firestore/pipeline_stage.go should have a copyright year of 2026
firestore/pipeline_stage_test.go should have a copyright year of 2026
firestore/pipeline_test.go should have a copyright year of 2026
firestore/pipeline_utils.go should have a copyright year of 2026

Fixing it in this PR
@bhshkh bhshkh enabled auto-merge (squash) January 13, 2026 00:25
In Java, Node and Python, Execute returns a Pipeline snapshot. This PR
introduces similar structure in Go


https://github.com/googleapis/java-firestore/blob/6e30a6c11efe5d428607bfd78f82ba7b49497bd9/google-cloud-firestore/src/main/java/com/google/cloud/firestore/Pipeline.java#L150-L159


It does not have execution time similar to Java yet. (Coming soon in GA)

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@bhshkh bhshkh changed the title feat(firestore): [NOT READY FOR REVIEW] add pipeline queries (#12217) feat(firestore): add pipeline queries (#12217) Jan 13, 2026
@bhshkh bhshkh removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jan 13, 2026
Index mode has been postponed to GA. removing it.
@bhshkh
Copy link
Copy Markdown
Contributor Author

bhshkh commented Jan 13, 2026

⚠️ Breaking change detected in pre-GA module: firestore

The apidiff check has detected one or more breaking changes in this module for pre-GA APIs.

Pre-GA breaking changes do not block the pull request, but are flagged here for review.

For any pre-GA breaking change that needs investigation, open a new issue containing a link to this comment, then you may proceed with reviewing and merging this PR.

- ./apiv1/firestorepb.FirestoreClient.ExecutePipeline: added
- ./apiv1/firestorepb.FirestoreServer.ExecutePipeline: added

All the proto changes have been reverted. Removing breaking change label

@bhshkh bhshkh changed the title feat(firestore): add pipeline queries (#12217) feat(firestore): [PQ] Support pipeline queries Jan 13, 2026
@bhshkh
Copy link
Copy Markdown
Contributor Author

bhshkh commented Jan 13, 2026

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a significant new feature: Firestore Pipeline Queries. The implementation is extensive, adding a fluent API for building complex data processing pipelines. The code is well-structured, following good practices for creating a chained, immutable API. I've found a couple of minor issues in the documentation examples that could confuse users. Overall, this is a solid addition to the library.

Comment thread firestore/pipeline.go
Comment on lines +261 to +263
// client.Pipeline().Collection("books").
// .Offset(20) // Skip the first 20 results
// .Limit(20) // Take the next 20 results
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The code example for Offset contains a syntax error. The leading dot . on line 262 makes the example invalid Go code. This could confuse users who copy and paste the example. Please correct it to be a valid, runnable snippet.

Suggested change
// client.Pipeline().Collection("books").
// .Offset(20) // Skip the first 20 results
// .Limit(20) // Take the next 20 results
// client.Pipeline().Collection("books").
// Offset(20). // Skip the first 20 results
// Limit(20) // Take the next 20 results

Comment thread firestore/pipeline.go
// client.Pipeline().Collection("users").Select("info.email")
// client.Pipeline().Collection("users").Select(FieldOf("info.email"))
// client.Pipeline().Collection("users").Select(FieldOf([]string{"info", "email"}))
// client.Pipeline().Collection("users").Select(FieldOf([]string{"info", "email"}))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There appears to be a duplicated line in the Select example. This could be confusing for users. Please remove the duplicate line to improve clarity.

@bhshkh bhshkh requested a review from daniel-sanche January 13, 2026 22:31
Copy link
Copy Markdown
Contributor

@daniel-sanche daniel-sanche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These PRs were individually reviewed and approved, so merging the branch LGTM

@bhshkh bhshkh merged commit 585cd82 into main Jan 13, 2026
10 checks passed
@bhshkh bhshkh deleted the feature/fs-pipeline-queries branch January 13, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: firestore Issues related to the Firestore API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants