Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for range-based ID searching #819

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open

Conversation

ItIsJordan
Copy link
Collaborator

Adds support for range-based searching in the OpenSearch index using HEPData ID and Inspire IDs. Updates the search help page to display new search syntax.

NOTE: inspire_id is changed from "text", to "integer". A re-index is required.

Adds functions to HEPDataQueryParser to support range-based queries: Determine if a query is a valid query, and to parse a valid query.
Adds a range-based publication_recid search functionality to the OpenSearch api search. Also modifies supporting merge_results function to support single-dict.
Adds new tests for HEPDataQueryParser.is_range_query and .parse_range_query
Adds an accidentally removed if statement that causes empty queries to not load data table results.
Add search term shorthand publication_recid truncated to recid in OpenSearch
Adds support for potential different range terms, defined in hepdata/config.py. Updates OpenSearch API search to use this.
Adds default descending order to range query search in opensearch api
Update tests to use correct range search syntax
Add query parser test case for publication_recid in the test _query_parser test.
Alters inspire_id mapping from text to integer to allow range-based searching. Also updates some testing and required configuration addition for range searching. (keeping it all in one commit)
Updates help text in search_help to display inspire ID range search guidance
Adds more tolerance to extra whitespace after the colon in range queries, and between the "TO" and range numbers.

Also updates parsing query to remove whitespace to accommodate this tolerance.
Adds more cases to test_query_parser_is_range_query which include valid and invalid extra whitespace cases.
Adds a comment to HEPDataQueryParser
Adds a specific test for range-searching on recid/inspire_id entries in the OpenSearch API
Updates test_search_range_ids to include a number of expected failure cases.
@coveralls
Copy link

coveralls commented Aug 16, 2024

Coverage Status

coverage: 83.499% (+0.06%) from 83.435%
when pulling bdd837d on search-range
into c51828b on main.

Modifies the range query parser functionality to fix a bug which disallowed range queries simultaneously with other queries. Improves errors.

Renamed is_range_query to get_range_queries as it now returns range query data only.
Adds improved testing for range query functionality. Adds a specific test for range queries alongside alternate search types.
Updates commenting in range query code
Adds commenting and improves clarity of get_range_query
Improves the test for test_query_parser_get_range_queries, adds some new cases and improves code clarity in the test.
@ItIsJordan
Copy link
Collaborator Author

I have made improvements and bugfixes to this branch to solve cases where using range searches disabled searching by other means, and to improve test robustness in the search functionality

Proper implementation has been figured out. Removes range parsing methods as raw range strings can be passed through to the search object.

This in turn allows proper use of logic in query strings with range queries.
Updates search testing to test current range search verification and updates range search testing cases. Removes tests for removed range parsing functions.
Add inspire_id and publication_recid as possible sorting fields by adding them to the possible_sorting_fields in display_results_options.html
Remove display of recid and inspire id search options from search "sort by" widget
Fixes a bug where range queries with publication_recid were not sorted by default. Now sets CFG_SEARCH_RANGE_TERMS in config to use the correct OpenSearch field keyword (recid). Also updates the sort_fields_mapping in the OpenSearch api to use correct keyword.
Add new recid and inspire_id keywords to search function docstring for the sort_field in the opensearch api.
Update verify_range_query_term docstring to clarify that query used is a parsed query.
Updates test_verify_range_query_term to account for using a parsed range query (with recid and not publication_recid). Updates
Renames test_composite_range_queries to test_range_queries and adds some extra testing for single-range keyword searching. Checks against correct expected sort order.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants