feat(firestore): [PQ] add pipeline queries

[bhshkh]

b/364927702

What are pipeline queries?

A Pipeline Query is constructed by defining a series of stages that are executed in order

Stages: A pipeline may consist of one or more stages. Logically, these represent the series of steps (or stages) taken to execute the query. Note: In practice, stages may be executed out of order to improve performance. However, this does not modify the intent or correctness of the query.

Expressions: Stages will often accept an expression allowing you to express more complex queries. Expression may be simple and consist of a single function like eq("a", 1). You can also express more complex expressions by nesting expressions like and(eq("a", 1), eq("b", 2)).

More info about pipeline queries can be found here: https://cloud.google.com/firestore/docs/pipeline/overview

Changes in this PR

This is the initial PR for pipeline queries to have basic components in place before adding more stages and functions.
This PR is for feature branch and not main. Once the protos are public, I will create a new PR to merge feature branch into main.

• The new structs introduced in this PR are inline with Java and Node implementations

• PipelineResult.Data() and PipelineResult.DataTo implementations are similar to existing DocumentSnapshot.Data and DataTo

• streamPipelineResultIterator is similar to queryDocumentIterator

[daniel-sanche]

is PipelineResult.Get() coming later?

[bhshkh]

PipelineResult is similar to DocumentSnapshot. So, there won't be a Get.

DocumentRef field inside PipelineResult and DocumentSnapshot already have a Get

The other methods that I would add are DataAt and DataAtPath

google-cloud-go/firestore/document.go

Lines 120 to 151 in 2671c4f

	// DataAt returns the data value denoted by path.
	//
	// The path argument can be a single field or a dot-separated sequence of
	// fields, and must not contain any of the runes "˜*/[]". Use DataAtPath instead for
	// such a path.
	//
	// See DocumentSnapshot.DataTo for how Firestore values are converted to Go values.
	//
	// If the document does not exist, DataAt returns a NotFound error.
	func (d *DocumentSnapshot) DataAt(path string) (interface{}, error) {
	if !d.Exists() {
	return nil, status.Errorf(codes.NotFound, "document %s does not exist", d.Ref.Path)
	}
	fp, err := parseDotSeparatedString(path)
	if err != nil {
	return nil, err
	}
	return d.DataAtPath(fp)
	}
	// DataAtPath returns the data value denoted by the FieldPath fp.
	// If the document does not exist, DataAtPath returns a NotFound error.
	func (d *DocumentSnapshot) DataAtPath(fp FieldPath) (interface{}, error) {
	if !d.Exists() {
	return nil, status.Errorf(codes.NotFound, "document %s does not exist", d.Ref.Path)
	}
	v, err := valueAtPath(fp, d.proto.Fields)
	if err != nil {
	return nil, err
	}
	return createFromProtoValue(v, d.c)
	}

[daniel-sanche]

Is there any way to simplify how stages are represented, to make it easier to scale?

Like maybe each the only actual struct is a base like this:

type PipelineStage struct {
  string
  values
  options
}

And each sub-type just builds one of those?

I'm not super familiar with go, so let me know if the answer is "no". Just want to make sure scalability is being considered, because we're going to have a lot more of these

[bhshkh]

In Go, the approach we're currently using (an interface like pipelineStage with specific structs like limitStage, collectionStage implementing it) is generally the more idiomatic and preferred way to handle this kind of polymorphism while maintaining strong type safety.

While a generic PipelineStage struct could make the Pipeline.stages slice look uniform, it would shift complexity and reduce type safety:

• Type Safety: We'd lose compile-time checks on the arguments specific to each stage. For example, limitStage clearly takes an int. With a generic struct, we'd use interface{} for arguments and rely on runtime type assertions within the logic that processes these stages (like converting to protobuf), which is more error-prone.
• Clarity: Specific structs make it very clear what parameters each stage type accepts.
• Encapsulation: Each stage struct can manage its own logic for converting to its protobuf representation in its toProto() method. With a generic struct, this logic would likely end up in a large, centralized switch statement.

The interface approach scales well because adding a new stage type involves defining its specific struct, implementing the interface, and adding a corresponding builder method to Pipeline. The core pipeline processing logic that iterates over []pipelineStage and calls toProto() doesn't need to be modified for each new stage type.

This pattern is common in Go for these reasons.

[daniel-sanche]

Are you sure exists is the right word for a PipelineResult that doesn't have a document? I don't remember seeing that in the other languages

I think the document will be empty if it's an aggregation, but the aggregation result still exists.

[bhshkh]

Removed

[gkevinzheng]

Why does this function essentially create a deep copy of the existing pipeline then extend the new one? Is this a better approach than just returning the existing object after appending the new pipelineStage to the existing one?

[bhshkh]

This is a deliberate choice to make the Pipeline builder immutable. It's a common pattern for fluent, chainable APIs.

1. This allows branching e.g.

base := client.Pipeline().Collection("events")
pA := base.Where(Field("type").Eq("A"))
pB := base.Where(Field("type").Eq("B"))
// 'base' is still just Collection("events")
// pA and pB are distinct and don't interfere.

2. Thread Safety (for reading):
   Immutable objects are inherently safe to share across goroutines for reading purposes without requiring locks, as their state never changes once created.

The minor performance overhead of copying is usually negligible compared to the benefits in robustness.

[gkevinzheng]

Where is setFromProtoValue defined? Is this just a generic serialization function?

[bhshkh]

Yes. Its here:

google-cloud-go/firestore/from_value.go

Lines 27 to 33 in 70c11ad

	func setFromProtoValue(dest interface{}, vprotoSrc *pb.Value, c *Client) error {
	destV := reflect.ValueOf(dest)
	if destV.Kind() != reflect.Ptr || destV.IsNil() {
	return errors.New("firestore: nil or not a pointer")
	}
	return setReflectFromProtoValue(destV.Elem(), vprotoSrc, c)
	}

It is also being used for query results

[gkevinzheng]

If PipelineSource is going to be a factory class, would it be better to rename this to PipelineFactory, or have the Pipeline class in Client be renamed to something reflecting that you're getting a factory class instead of just Client.Pipeline?

[bhshkh]

This is to have uniformity across clients. Java, Node and Python use the same terminology.

"Source" is domain-appropriate. While it is a factory, naming it PipelineSource effectively describes its role. Changing to PipelineFactory might make it sound more like a GoF pattern and less like a domain concept.

Changing Client.Pipeline() to something like Client.GetPipelineFactory() or Client.NewPipelineBuilder() is more verbose.

[daniel-sanche]

LGTM

[wu-hui]

LGTM, FWIW.