Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
194 commits
Select commit Hold shift + click to select a range
1087d64
added skeletons for pipeline expressions
daniel-sanche Jan 22, 2025
053f55b
added quick implemtation for expressions
daniel-sanche Jan 23, 2025
e56459c
added default implementation for pipeline stages
daniel-sanche Jan 23, 2025
babafb2
ran black
daniel-sanche Jan 23, 2025
149c61f
got code to run
daniel-sanche Jan 23, 2025
320cea1
moved helpers
daniel-sanche Jan 23, 2025
cd2963d
added basic pipelines.py file
daniel-sanche Jan 23, 2025
3cbc2d5
added yaml test file
daniel-sanche Jan 24, 2025
0c2241d
wrote basic parser for pipeline yaml
daniel-sanche Jan 24, 2025
9c8982c
encoded extra java system tests
daniel-sanche Jan 24, 2025
863bd1d
reconstruct pipeline expr objects
daniel-sanche Jan 24, 2025
7cd2c11
got yaml to run
daniel-sanche Jan 25, 2025
c328934
add data loading code
daniel-sanche Jan 25, 2025
de0a862
updated protos
daniel-sanche Feb 27, 2025
45c83aa
added pyyaml to system test dependencies
daniel-sanche Feb 27, 2025
fc57f04
added Expr methods
daniel-sanche Jan 28, 2025
0224f94
trying to improve accumulator api
daniel-sanche Jan 28, 2025
cdee937
improved aggregate/accumulators
daniel-sanche Jan 28, 2025
5d35c54
fixed naming in yaml
daniel-sanche Jan 28, 2025
11373e3
create objects in parsing code
daniel-sanche Jan 28, 2025
d2b4153
use order enum
daniel-sanche Jan 28, 2025
bfdfba3
standardize how I deal with map stages
daniel-sanche Jan 28, 2025
afa823a
fix broke super.__init__ calls
daniel-sanche Jan 28, 2025
3348127
added repr for custom classes
daniel-sanche Jan 28, 2025
e3c995d
added repr to pipeline
daniel-sanche Jan 28, 2025
2d604e1
fixed vector formatting
daniel-sanche Jan 28, 2025
3bed807
only treat capitalized strings as possible exprs
daniel-sanche Jan 28, 2025
9b9eaa8
Where uses positional args in yaml
daniel-sanche Jan 28, 2025
e35accf
support union stage
daniel-sanche Jan 28, 2025
72301a1
fixed testReplace
daniel-sanche Jan 28, 2025
07d25dc
fixed sample
daniel-sanche Jan 28, 2025
38fb7fa
iterating on sample options
daniel-sanche Jan 28, 2025
0c3852f
fixed import
daniel-sanche Jan 28, 2025
1d95e54
removing kwargs
daniel-sanche Jan 29, 2025
a3daa07
removing keargs
daniel-sanche Jan 29, 2025
8e45b68
use only positional arguments in yaml
daniel-sanche Jan 29, 2025
6b2a605
added _to_pb to stages
daniel-sanche Feb 27, 2025
6d2aa90
added _pb to pipeline expressions
daniel-sanche Feb 27, 2025
e51d9dc
added unimplemented stage._to_pbs
daniel-sanche Feb 27, 2025
8f6dea5
fixed proto formatting
daniel-sanche Feb 27, 2025
654e5f7
got protos to build with no errors
daniel-sanche Feb 28, 2025
69d4b76
added headers
daniel-sanche Feb 28, 2025
6d966a8
fixed some types
daniel-sanche Feb 28, 2025
33fee52
fixed expression typing
daniel-sanche Feb 28, 2025
cb8539d
added abstract to expr
daniel-sanche Feb 28, 2025
380dce9
fixed types in pipeline_stages
daniel-sanche Feb 28, 2025
831886f
align typing
daniel-sanche Mar 1, 2025
79df205
added stubs for execute_pipeline
daniel-sanche Mar 1, 2025
e760c44
fixed union stage proto
daniel-sanche Mar 25, 2025
ea1e2ba
propagate client in test parsers
daniel-sanche Mar 25, 2025
b6b082d
enable mypy for pipeline code
daniel-sanche Mar 25, 2025
658964c
added query.pipeline
daniel-sanche Mar 25, 2025
7e12b74
got queries to execute
daniel-sanche Mar 25, 2025
e463418
fixed some encodings
daniel-sanche Mar 25, 2025
4f527c1
broke pipelines into separate async/sync/base files
daniel-sanche Mar 27, 2025
a323e5b
added docstrings to expressions
daniel-sanche Mar 27, 2025
7e70022
improved ordering
daniel-sanche Mar 27, 2025
98887a0
added docstrings to pipeline classes
daniel-sanche Mar 27, 2025
549c590
added docstrings
daniel-sanche Mar 27, 2025
376015c
fixed sample_options
daniel-sanche Mar 27, 2025
e3de1b8
catch expected errors in test
daniel-sanche Mar 27, 2025
f725bc7
added alias to unnest
daniel-sanche Mar 28, 2025
128ab1c
fixed replace
daniel-sanche Mar 28, 2025
64a8bda
added pipeline to client
daniel-sanche Mar 28, 2025
ec06080
improved query.pipeline logic
daniel-sanche Mar 28, 2025
0703532
removed imports
daniel-sanche Mar 28, 2025
246d1c8
pass stages through
daniel-sanche Mar 28, 2025
b72d44a
fixed collection docstrings
daniel-sanche Mar 28, 2025
810ccd2
fixed collection setup
daniel-sanche Mar 28, 2025
78b5833
fixed some collection errors
daniel-sanche Mar 29, 2025
36e0228
implemented FitlerCondition._from_pb
daniel-sanche Mar 29, 2025
43f4cf4
renamed function
daniel-sanche Mar 29, 2025
51e9b25
added tests for query.pipeline
daniel-sanche Mar 29, 2025
35cb0bb
added verify pipeline to system tests
daniel-sanche Mar 31, 2025
6283d1a
fixed bug in filter conversion
daniel-sanche Mar 31, 2025
080bf42
return pipeline copy
daniel-sanche Mar 31, 2025
6cd5c63
updated results format
daniel-sanche Mar 31, 2025
7143247
added proto assertions to e2e tests
daniel-sanche Mar 31, 2025
05c4f23
added tests for FieldFilter._from_filter_pb
daniel-sanche Apr 1, 2025
7512a1a
fixed repr
daniel-sanche Apr 1, 2025
901c0e1
fixed typing issues
daniel-sanche Apr 1, 2025
6a5f72e
fixed client fixture
daniel-sanche Apr 1, 2025
072669f
fixed mapget in e2e yaml
daniel-sanche Apr 1, 2025
05a7498
fixed docstring
daniel-sanche Apr 1, 2025
9f8d4c8
fixed append bug
daniel-sanche Apr 1, 2025
863a09d
compare result data in tests
daniel-sanche Apr 1, 2025
e9f11c6
fixed incorrect test case ordering
daniel-sanche Apr 1, 2025
d2c33f2
yield document snapshots
daniel-sanche Apr 2, 2025
db1ecaa
added async test
daniel-sanche Apr 2, 2025
b6d74da
cleaning up e2e yaml
daniel-sanche Apr 2, 2025
70efea7
fixed scope in tests
daniel-sanche Apr 3, 2025
b36afa8
added _client to base pipeline
daniel-sanche Apr 3, 2025
7a26f7e
fixing test yaml
daniel-sanche Apr 3, 2025
fee6c90
renamed exceute_async to execute
daniel-sanche Apr 3, 2025
da18289
broke up pipeline tests into separate functions
daniel-sanche Apr 3, 2025
fc80062
improved faulty test yaml
daniel-sanche Apr 30, 2025
c049e21
remvoved unready stages and expressions
daniel-sanche May 2, 2025
41b91d4
fixed regex_match test
daniel-sanche May 2, 2025
7f69229
ran blacken
daniel-sanche May 2, 2025
0b4c294
turn Stage into an ABC
daniel-sanche May 2, 2025
255b698
removed extra stages and expressions
daniel-sanche May 2, 2025
bd9c2c4
stripped down to stubs
daniel-sanche May 2, 2025
4314141
Revert "stripped down to stubs"
daniel-sanche May 3, 2025
c38a925
Revert "removed extra stages and expressions"
daniel-sanche May 3, 2025
d8dc10f
added PipelineStages
daniel-sanche May 6, 2025
1f2390a
removed collection.pipeline
daniel-sanche May 6, 2025
af371d7
fixed collection order
daniel-sanche May 6, 2025
a0f8e5d
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche May 6, 2025
ba39b81
fixed collection order in docstrings
daniel-sanche May 6, 2025
39f261a
fixed docstrings
daniel-sanche May 6, 2025
48bf9f7
added pipeline_result
daniel-sanche May 6, 2025
b57424d
chore: updated gapic layer for execute_query
daniel-sanche May 8, 2025
8fde414
updated gapics
daniel-sanche May 8, 2025
9cac154
Merge branch 'pipeline_queries_approved' into pipeline_queries_1_stubs
daniel-sanche May 8, 2025
93003c0
removed unneeded code
daniel-sanche May 8, 2025
9a9cc3d
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query…
daniel-sanche May 8, 2025
2750bd5
ran black
daniel-sanche May 8, 2025
d4dcac4
use pipeline source in query
daniel-sanche May 8, 2025
e8be99f
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche May 8, 2025
77dca01
added extra pipeline sources
daniel-sanche May 8, 2025
79d016a
fixed lint
daniel-sanche May 8, 2025
907a551
added client tests
daniel-sanche May 8, 2025
8e20c11
added pipeline tests
daniel-sanche May 9, 2025
98c7ea5
added tests for execute
daniel-sanche May 9, 2025
af4fb20
broke out shared logic into base_pipeline
daniel-sanche May 9, 2025
cd38fc2
added pipeline stages tests
daniel-sanche May 12, 2025
0ac319e
removed unneeded stages
daniel-sanche May 12, 2025
7981a34
added tests for pipeline expressions
daniel-sanche May 12, 2025
62b6510
added pipeline_result tests
daniel-sanche May 12, 2025
035c6e6
added tests for pipeline source
daniel-sanche May 12, 2025
5dd1246
added transaction to execute call
daniel-sanche May 12, 2025
6434023
ran blacken
daniel-sanche May 12, 2025
a8beea4
fixed lint
daniel-sanche May 12, 2025
2d286bb
fixed mypy
daniel-sanche May 12, 2025
a1e8e77
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query…
daniel-sanche May 12, 2025
8c00357
fixed lint
daniel-sanche May 13, 2025
09d45cb
removed e2e tests from this PR
daniel-sanche May 13, 2025
9cdb8c9
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche May 13, 2025
9a783d7
fixed lint
daniel-sanche May 13, 2025
d2babd2
Merge branch 'pipeline_queries_approved' into pipeline_queries_1_stubs
daniel-sanche May 13, 2025
b46bdc1
fixed test issues
daniel-sanche May 13, 2025
28fd42d
added tests
daniel-sanche May 13, 2025
a9368b3
added stages unit tests
daniel-sanche May 13, 2025
2775da2
improve FilterCondition repr
daniel-sanche May 14, 2025
c4cd995
added tests for filter conditions
daniel-sanche May 14, 2025
63b83b8
added tests for expressions
daniel-sanche May 15, 2025
f22f11e
support tuples in pipelin_source.collection
daniel-sanche May 15, 2025
c8cfcee
added tests
daniel-sanche May 15, 2025
5beee36
ran blacken
daniel-sanche May 15, 2025
dc5b5ac
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query…
daniel-sanche May 15, 2025
22b558c
Merge branch 'pipeline_queries_approved' into pipeline_queries_1_stubs
daniel-sanche Jun 9, 2025
0ff25c1
Merge branch 'pipeline_queries_1_stubs' into pipeline_queries_2_query…
daniel-sanche Jun 9, 2025
5a8eda3
Squashed commit of the following:
daniel-sanche Jun 9, 2025
3a16610
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche Jun 9, 2025
be6dbf7
added unit tests for new sources
daniel-sanche Jun 9, 2025
628f66e
added tests for stages
daniel-sanche Jun 9, 2025
4594a6c
ran black
daniel-sanche Jun 9, 2025
74a5d22
ran black
daniel-sanche Jun 9, 2025
e201b6f
moved SampleOptions into stages file
daniel-sanche Jun 9, 2025
386585b
updated infix test
daniel-sanche Jun 10, 2025
864bb66
added tests for ExprWithAlias
daniel-sanche Jun 10, 2025
40612ac
restructured expression tests
daniel-sanche Jun 10, 2025
aabcf33
added tests for expressions
daniel-sanche Jun 10, 2025
bf7adad
improved verify_pipeline check
daniel-sanche Jun 10, 2025
b2546f7
use TEST_DATABASES list
daniel-sanche Jun 10, 2025
38f00a7
ran black
daniel-sanche Jun 10, 2025
82d5da1
disallow mixed positional and kwargs in Aggregate
daniel-sanche Jun 11, 2025
244a9f2
improving aggregate
daniel-sanche Jun 12, 2025
c62e448
add pipeline() method to collections
daniel-sanche Jun 12, 2025
bf0dca2
fixed repr test
daniel-sanche Jun 12, 2025
3cd826b
fixed lint
daniel-sanche Jun 12, 2025
c8f9ca8
Squashed commit of the following:
daniel-sanche Jun 12, 2025
5cd1670
support str in add_field
daniel-sanche Jun 12, 2025
f019c16
updated test_pipelines
daniel-sanche Jun 12, 2025
ae00f1d
added equality checks
daniel-sanche Jun 12, 2025
36fd97b
fixed lint
daniel-sanche Jun 12, 2025
32eea9a
fixed lint
daniel-sanche Jun 12, 2025
5e771ed
Merge branch 'pipeline_queries_2_query_parity' into pipeline_queries_…
daniel-sanche Jun 12, 2025
b406003
removed str option from add_fields
daniel-sanche Jun 12, 2025
e6ad9d2
broke out TestFunctions from TestFilterConditions
daniel-sanche Jun 13, 2025
3de11fa
Squashed commit of the following:
daniel-sanche Jun 23, 2025
50f7d7b
Merge branch 'pipeline_queries_approved' into pipeline_queries_3_stab…
daniel-sanche Jun 23, 2025
3f675b7
fixed broken tests
daniel-sanche Jun 23, 2025
b81c637
fixed coverage
daniel-sanche Jun 24, 2025
1f0595a
fixed system test
daniel-sanche Jun 24, 2025
4552918
removed query -> pipeline conversions
daniel-sanche Jun 24, 2025
1ff2053
simplified event loop
daniel-sanche Jun 24, 2025
f732f01
cleaned up test file
daniel-sanche Jun 24, 2025
a4feed5
fix flaky test
daniel-sanche Jun 25, 2025
8344b15
ficed bug in StrConcat
daniel-sanche Jul 11, 2025
a90b1c3
added static methods to Function class
daniel-sanche Jul 11, 2025
6809e7c
fixed lint
daniel-sanche Jul 11, 2025
061472e
fixed mypy
daniel-sanche Jul 11, 2025
6366301
fixed lint
daniel-sanche Jul 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
392 changes: 390 additions & 2 deletions google/cloud/firestore_v1/_pipeline_stages.py
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is replace working for you? I was getting errors, so I put it off for the follow up PR.

I get google.api_core.exceptions.InvalidArgument: 400 Unsupported stage: replace

Large diffs are not rendered by default.

464 changes: 462 additions & 2 deletions google/cloud/firestore_v1/base_pipeline.py

Large diffs are not rendered by default.

2,217 changes: 2,217 additions & 0 deletions google/cloud/firestore_v1/pipeline_expressions.py

Large diffs are not rendered by default.

38 changes: 36 additions & 2 deletions google/cloud/firestore_v1/pipeline_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,12 @@
from typing import Generic, TypeVar, TYPE_CHECKING
from google.cloud.firestore_v1 import _pipeline_stages as stages
from google.cloud.firestore_v1.base_pipeline import _BasePipeline
from google.cloud.firestore_v1._helpers import DOCUMENT_PATH_DELIMITER

if TYPE_CHECKING: # pragma: NO COVER
from google.cloud.firestore_v1.client import Client
from google.cloud.firestore_v1.async_client import AsyncClient
from google.cloud.firestore_v1.base_document import BaseDocumentReference


PipelineType = TypeVar("PipelineType", bound=_BasePipeline)
Expand All @@ -41,13 +43,45 @@ def __init__(self, client: Client | AsyncClient):
def _create_pipeline(self, source_stage):
return self.client._pipeline_cls._create_with_stages(self.client, source_stage)

def collection(self, path: str) -> PipelineType:
def collection(self, path: str | tuple[str]) -> PipelineType:
"""
Creates a new Pipeline that operates on a specified Firestore collection.

Args:
path: The path to the Firestore collection (e.g., "users")
path: The path to the Firestore collection (e.g., "users"). Can either be:
* A single ``/``-delimited path to a collection
* A tuple of collection path segment
Returns:
a new pipeline instance targeting the specified collection
"""
if isinstance(path, tuple):
path = DOCUMENT_PATH_DELIMITER.join(path)
return self._create_pipeline(stages.Collection(path))

def collection_group(self, collection_id: str) -> PipelineType:
"""
Creates a new Pipeline that that operates on all documents in a collection group.
Args:
collection_id: The ID of the collection group
Returns:
a new pipeline instance targeting the specified collection group
"""
return self._create_pipeline(stages.CollectionGroup(collection_id))

def database(self) -> PipelineType:
"""
Creates a new Pipeline that operates on all documents in the Firestore database.
Returns:
a new pipeline instance targeting the specified collection
"""
return self._create_pipeline(stages.Database())

def documents(self, *docs: "BaseDocumentReference") -> PipelineType:
"""
Creates a new Pipeline that operates on a specific set of Firestore documents.
Args:
docs: The DocumentReference instances representing the documents to include in the pipeline.
Returns:
a new pipeline instance targeting the specified documents
"""
return self._create_pipeline(stages.Documents.of(*docs))
Loading