Implement Validation for Non-Queryable Attributes in Filtering #532

bountx · 2025-11-20T08:45:16Z

Related Issue(s):

Implement Validation for Non-Queryable Attributes in Filtering #530

Description:

Created query validation to return bad request 400 on requests with parameters not existent in any queryables without unnecessary searching (optional: turned off by default - turned on by environment variable VALIDATE_QUERYABLES).

To do that PR also introduces local caching system (global variable _queryables_cache_instance: QueryablesCache) that collects a set of queryables parameters from all the collections and updates it hourly by default (set with environment variable QUERYABLES_CACHE_TTL in seconds - default to 3600, 1 hour).

PR Checklist:

Code is formatted and linted (run pre-commit run --all-files)
Tests pass (run make test)
Documentation has been updated to reflect changes, if applicable
Changes are added to the changelog

…lds in QueryablesCache

jonhealy1 · 2025-11-26T10:37:17Z

README.md

 | `USE_DATETIME_NANOS` | Enables nanosecond precision handling for `datetime` field searches as per the `date_nanos` type. When `False`, it uses 3 millisecond precision as per the type `date`. | `true` | Optional |
 | `EXCLUDED_FROM_QUERYABLES` | Comma-separated list of fully qualified field names to exclude from the queryables endpoint and filtering. Use full paths like `properties.auth:schemes,properties.storage:schemes`. Excluded fields and their nested children will not be exposed in queryables. | None | Optional |
 | `EXCLUDED_FROM_ITEMS` | Specifies fields to exclude from STAC item responses. Supports comma-separated field names and dot notation for nested fields (e.g., `private_data,properties.confidential,assets.internal`). | `None` | Optional |
+| `VALIDATE_QUERYABLES` | Enable validation of query parameters against the collection's queryables. If set to `true`, the API will reject queries containing fields that are not defined in the collection's queryables. | `false` | Optional |


@bountx How does this interact with the EXCLUDED_FROM_QUERYABLES environment variable? Should both these env vars be combined into one?

I did miss this in my implementation so I will fix it, but these two variables serve a different purpose: VALIDATE_QUERYABLES should validate on set of all queryables and EXCLUDED_FROM_QUERYABLES can be used to make this set of all queryables smaller.

Their interaction when both used should look something along:
Cache of all queryables -> Removes from this set all excluded queryables -> Validates on that set

If you had something other in mind let me know

@jonhealy1 Fixed this interaction to work the way I mentioned beforehand.

jonhealy1 · 2025-11-26T10:38:51Z

README.md

+- If a request contains a query parameter or filter field that is not in the list of allowed queryables, the API returns a `400 Bad Request` error with a message indicating the invalid field(s).
+- The cache is automatically refreshed based on the `QUERYABLES_CACHE_TTL` setting.
+
+This feature helps prevent queries on unindexed fields which could lead to poor performance or unexpected results.


I could be wrong but I think Opensearch automatically indexes everything?

Poor wording on my side. Unindexed fields = nonexistant fields

Fixed wording

jonhealy1 · 2025-11-29T05:30:28Z

README.md

 | `STAC_INDEX_ASSETS` | Controls if Assets are indexed when added to Elasticsearch/Opensearch. This allows asset fields to be included in search queries. | `false` | Optional |
 | `USE_DATETIME` | Configures the datetime search behavior in SFEOS. When enabled, searches both datetime field and falls back to start_datetime/end_datetime range for items with null datetime. When disabled, searches only by start_datetime/end_datetime range. | `true` | Optional |
 | `USE_DATETIME_NANOS` | Enables nanosecond precision handling for `datetime` field searches as per the `date_nanos` type. When `False`, it uses 3 millisecond precision as per the type `date`. | `true` | Optional |
 | `EXCLUDED_FROM_QUERYABLES` | Comma-separated list of fully qualified field names to exclude from the queryables endpoint and filtering. Use full paths like `properties.auth:schemes,properties.storage:schemes`. Excluded fields and their nested children will not be exposed in queryables. | None | Optional |


Let's add a line here explaining the expected behavior if VALIDATE_QUERYABLES is also set.

jonhealy1 · 2025-11-29T05:38:40Z

stac_fastapi/core/stac_fastapi/core/core.py

+    get_properties_from_cql2_filter,
+    initialize_queryables_cache,
+    validate_queryables,
+)


You are importing stac_fastapi.sfeos_helpers into stac_fastapi.core. The Core library is a generic abstraction and must not depend on specific implementation helpers. This breaks the modularity of the library.

Query validation is a generic feature that belongs in Core. However, we cannot import it from sfeos_helpers.

Move to Core Please move queryables.py (the Cache and Validator logic) out of stac_fastapi/sfeos_helpers/ and into stac_fastapi/core/.

Once the code lives in Core, CoreClient can import it safely without creating a dependency on the Elasticsearch/OpenSearch specific package. The Validator should rely strictly on BaseDatabaseLogic interfaces. (Note: You will likely need to add get_queryables_mapping as an abstract method to BaseDatabaseLogic so the validator can call it generically).

Remove Global State Please remove the global variable (_queryables_cache_instance). Instead, instantiate QueryablesCache directly inside CoreClient.init as self.queryables_cache. This resolves the threading and testing fragility issues inherent in global state.

jonhealy1 · 2025-11-29T05:40:03Z

README.md

+```bash
+VALIDATE_QUERYABLES=true
+QUERYABLES_CACHE_TTL=3600  # Optional, defaults to 3600 seconds (1 hour)
+```


Does this seem like a long default?

This is what my team suggested at first as the cache query for queryable parameters doesn't seem that costly. I'll rediscuss that with them though in detail.

I changed it to 6 hours default

I was wondering if it should be shorter like 30 minutes maybe.

Ahh I see - for our case the queryables changes are pretty rare so 1h-6h updates wouldn't change that much, but I think 30m default would work too.

I think if someone is adding new data to the db, they may get confused.

Updated the default value to 30 minutes

jonhealy1 · 2025-11-29T05:46:31Z

stac_fastapi/sfeos_helpers/stac_fastapi/sfeos_helpers/queryables.py

+
+
+_queryables_cache_instance: Optional[QueryablesCache] = None
+


See comment above. Please remove _queryables_cache_instance

class CoreClient(AsyncBaseCoreClient): def __init__(self, database: BaseDatabaseLogic, ...): self.database = database # Initialize the cache as an instance attribute # This solves the threading and testing fragility issues self.queryables_cache = QueryablesCache(database)```

jonhealy1 · 2025-11-29T06:05:05Z

stac_fastapi/tests/sfeos_helpers/test_queryables.py

+
+class TestGlobalFunctions:
+    @pytest.fixture(autouse=True)
+    def reset_global_cache(self):


We should be able to remove this fixture after the other changes have been made.

bountx · 2025-12-01T10:07:41Z

Moved to Core - The queryables.py module (containing QueryablesCache and validation logic) has been moved from sfeos_helpers to core. The Core library no longer depends on implementation-specific helpers.
Added get_queryables_mapping to BaseDatabaseLogic - added get_queryables_mapping as an abstract method to BaseDatabaseLogic in base_database_logic.py. Both Elasticsearch and OpenSearch implementations now implement this method, allowing the validator to work generically without knowing about specific database implementations.
Removed Global State - The global _queryables_cache_instance variable has been completely removed. Instead, QueryablesCache is now instantiated as an instance attribute in CoreClient.__attrs_post_init__()
This resolves threading and testing fragility issues.
Updated Tests - Removed the reset_global_cache fixture from stac_fastapi/tests/sfeos_helpers/test_queryables.py as it's no longer needed with instance-based caching.
Clean Module Structure - All imports now flow correctly: CoreClient imports from stac_fastapi.core.queryables, and the queryables module only depends on BaseDatabaseLogic interfaces.

@jonhealy1

…urs)

…nutes)

jonhealy1

Nice work!

Combines the following fixes: - Normalize property names by stripping 'properties.' prefix in CQL2 filter - Enhance field mapping to handle 'properties.' prefix in Elasticsearch queries - Fix nested property traversal and improve error messages - Implement exclusion of fields from queryables based on environment variable - Enhance exclusion logic for queryables to support top-level fields and properties prefix

bountx · 2025-12-09T20:05:32Z

@jonhealy1 After some dev tests it turned out the EXCLUDE_FROM_QUERYABLES didn't interact well with our validation - fixed it. Added CQL2-text support and fixed nested properties traversal for validation. Let me know if you have any questions!

jonhealy1

Looks good. Just make sure old changelog entries won't be deleted.

jonhealy1 · 2025-12-10T02:40:09Z

CHANGELOG.md


- Added optional `/catalogs` route support to enable federated hierarchical catalog browsing and navigation. [#547](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/547)
- Added DELETE `/catalogs/{catalog_id}/collections/{collection_id}` endpoint to support removing collections from catalogs. When a collection belongs to multiple catalogs, it removes only the specified catalog from the collection's parent_ids. When a collection belongs to only one catalog, the collection is deleted entirely. [#554](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/554)
- Added `parent_ids` internal field to collections to support multi-catalog hierarchies. Collections can now belong to multiple catalogs, with parent catalog IDs stored in this field for efficient querying and management. [#554](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/554)


@bountx PR looks good, however, it looks like these changelog entries are being deleted.

Sorry, missclicked when merging, I'll add them back

jonhealy1

Thanks :)

Andrzej Pijanowski and others added 6 commits November 19, 2025 14:26

feat: add queryables cache and optional validation for search parameters

c488a39

fix: remove unnecessary info from error message for invalid query fie…

7434493

…lds in QueryablesCache

Merge branch 'stac-utils:main' into CAT-1525

bb939bd

feat: add queryables validation and cache configuration to README

15ad1b9

Merge branch 'stac-utils:main' into CAT-1525

e3ff023

Update CHANGELOG.md

5de02fc

Gomez324 requested review from jamesfisher-geo and jonhealy1 November 24, 2025 11:50

jonhealy1 reviewed Nov 26, 2025

View reviewed changes

Andrzej Pijanowski added 2 commits November 27, 2025 11:04

feat: add support for excluding queryables from validation

3c3b5cb

Merge branch 'main' into CAT-1525

7eef065

bountx requested a review from jonhealy1 November 27, 2025 11:26

Merge branch 'main' into CAT-1525

6cdb67c

jonhealy1 requested changes Nov 29, 2025

View reviewed changes

feat: move queryables cache and validation logic to core

a46665e

bountx requested a review from jonhealy1 December 1, 2025 11:20

bountx and others added 3 commits December 2, 2025 10:01

Merge branch 'main' into CAT-1525

487f3f7

fix: update QUERYABLES_CACHE_TTL default value to 21600 seconds (6 ho…

93c6e78

…urs)

fix: update QUERYABLES_CACHE_TTL default value to 1800 seconds (30 mi…

405a4df

…nutes)

jonhealy1 previously approved these changes Dec 2, 2025

View reviewed changes

bountx dismissed jonhealy1’s stale review via 1b586e4 December 9, 2025 19:42

bountx and others added 3 commits December 9, 2025 20:45

Merge branch 'main' into CAT-1525

524f9f9

fix: remove unused redis_utils import to fix flake8 errors

85b7432

black format

e5211e4

bountx requested a review from jonhealy1 December 9, 2025 20:05

jonhealy1 requested changes Dec 10, 2025

View reviewed changes

jonhealy1 reviewed Dec 10, 2025

View reviewed changes

Readded missing changelogs

9552caf

bountx requested a review from jonhealy1 December 10, 2025 07:56

bountx self-assigned this Dec 10, 2025

jonhealy1 approved these changes Dec 10, 2025

View reviewed changes

bountx merged commit d0f01d4 into stac-utils:main Dec 11, 2025
8 checks passed

bountx deleted the CAT-1525 branch December 11, 2025 11:10



		_queryables_cache_instance: Optional[QueryablesCache] = None

Implement Validation for Non-Queryable Attributes in Filtering #532

Implement Validation for Non-Queryable Attributes in Filtering #532

Uh oh!

Conversation

bountx commented Nov 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bountx commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonhealy1 left a comment

Choose a reason for hiding this comment

Uh oh!

bountx commented Dec 9, 2025

Uh oh!

jonhealy1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonhealy1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bountx commented Dec 1, 2025 •

edited

Loading