Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Structured datastore query endpoint #3248

Merged
merged 90 commits into from
Dec 30, 2020
Merged

Structured datastore query endpoint #3248

merged 90 commits into from
Dec 30, 2020

Conversation

dafeder
Copy link
Member

@dafeder dafeder commented Oct 29, 2020

Introduces a new API endpoint for querying datastore tables using a JSON syntax. This need arose from a need for more functionality than the current SQL endpoint provides, including:

  1. Support for mathematical expressions
  2. Support for OR conditions and condition groups
  3. Support for table joins

The decision to implement these first in a JSON endpoint rather than expanding the SQL endpoint came from these considerations:

  1. SQL statements do not follow a set schema and are challenging to parse, don't want that to be the bottleneck
  2. Building JSON objects is more natural/intuitive for frontend developers than building out SQL strings
  3. We are working toward a new type of stored metadata for "filtered views"/"stored queries" and wanted to start building out the foundations of that.

This PR also brings in our new library for working with validate-able JSON data structures in DKAN, GetDKAN/RootedJsonData. We are hoping this finds its way into the rest of the DKAN codebase and provides a standardized way to pass JSON data around the system and ensure that it is always valid.

QA Steps

  1. Visit api/1/metastore/schemas/dataset/items?show-reference-ids and select the UUID of a particular distribution.
  2. Try the following HTTP request, substituting your distribution ID for the one in this example:
POST /api/1/datastore/query HTTP/1.1
Host: datastorebool.localtest.me
Content-Type: text/plain

{
    "resources": [
        {
            "id": "80503a94-77f4-560c-89bf-6f6984c9a137",
            "alias": "t"
        }
    ]
}

If this works, try different types of queries using the tests in this PR and the query.json schema for guidance.

Known issues

  1. There are now two types of query objects being handled by the datastore - DatastoreQuery and the core DKAN Query. This is pretty redundant and adds a lot of complexity to translate from one to another. We should try to collapse or simplify this.
  2. This PR removes a header that was previously present in DKAN REST API reponses allowing any origin, effectively bypassing CORS. DKAN endpoints will now simply set origin headers consistently with the rest of your Drupal site. For more information about handling CORS configuration in Drupal read this thread.

Merge steps

  1. Merge Normalize set values when passing objects RootedJsonData#9
  2. New release of RootedJsonData
  3. Set https://github.com/GetDKAN/dkan/blob/query-joins-v1/composer.json#L27 to pull in new release
  4. Merge

Resolves #3073, resolves #3258

@@ -13,7 +13,7 @@ trait JsonResponseTrait {
* Private.
*/
private function getResponse($message, int $code = 200): JsonResponse {
return new JsonResponse($message, $code, ["Access-Control-Allow-Origin" => "*"]);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document this change

@dafeder dafeder marked this pull request as ready for review December 9, 2020 20:30
@dafeder dafeder requested a review from fmizzell December 9, 2020 20:30
@fmizzell fmizzell added the WIP label Dec 22, 2020
@dafeder
Copy link
Member Author

dafeder commented Dec 23, 2020

@fmizzell updated the sort format. Key commit is ae2cb7a

@fmizzell fmizzell merged commit 76b0b3c into 2.x Dec 30, 2020
@fmizzell fmizzell deleted the query-joins-v1 branch December 30, 2020 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants