ADR: use jq selector syntax within subject_external_fields of Subject Condition Sets under Subject Mappings #477
Description
Overview
ADR #470 considers options for selector syntax of subject_external_fields in varied subject contexts (idP responses, access tokens, external user stores). This expands on the options that merited further exploration with a POC and deeper research.
Code (diff): opentdf/otdfctl@main...poc/selectors
Branch: https://github.com/opentdf/tructl/tree/poc/selectors
There are three pieces of our relation of subjects to platform attributes
- select: help admins find their selectors and build subject condition sets with them
- store/retrieve: store them in the DB so they map subjects to attribute values
- evaluate: a logical evaluation comparing the stored relations of subjects to attribute values by their AND/OR/IN/NOT_IN logic against the subject context comprised of multiple selected fields and values
This POC is focused on the first aspect (select ), or in simple terms “given a representation of a subject, how can we parse fields and values” where their stored logical groupings or evaluation relating to attribute values are not the primary concerns. This is about finding the right inputs to the evaluation engine that returns entitlements.
Chosen Option: Option 1: JQ (gojq)
Summary of Findings
Option 1: JQ (gojq)
🟩 Ships with linux so is available in MacOS, and does come with Windows by default
🟩 Very familiar CLI tool with lots of documentation and reference material (including for LLMs to have digested)
🟩 gojq lib was easy to work with and simple and is actively maintained with 3k+ stars
🟩 Truly concerned with the select need rather than evaluate
🟩 The quantity of symbols seems far smaller than the alternatives, which reduces cognitive load and morphological complexity
🟩 There are a bunch of external playgrounds available if someone doesn’t want to just use jq in the terminal. Some appear to use bundled JS and some do fetch a jq.wasm.
🟩 Some other CLI tools like github’s actually build in filtering command output with jq as top-level flags
🟨 / 🟥 Someone published a WASM package for jq but it hasn’t been updated in 5 years, and there is no popular browser JS package I found for a dashboard. For a playground UI of our own, I wonder if we would want to just build/distribute our own WASM to keep in sync with the dependency or just pass to an endpoint for parsing server side rather than rewrite in browser-safe javascript
🟥 It is still a new syntax to learn and debug with additional cognitive load or platform admins
Option 2: Common Expression Language (CEL)
🟩 / 🟨 This is powerful enough to be a viable alternative to OPA with pros/cons weighed against rego syntax it is not designed specifically for selecting fields and values from arbitrary map and array structures
🟨 Predominantly an evaluator, not a selector, with comprehensive logic like &&, ||, != etc
🟥 The most popular npm package for use in the browser has <100 downloads per week and is less than version 1.0.0, meaning we’d need to write our own evaluator for the browser
🟥 Too much evaluation when selecting fields from a Subject context before evaluating within OPA or the subject condition set engine seems difficult to debug and like a smell
🟥 It is my belief that CEL is utilized in a lot of cloud tooling because it looks a bit like code from any language, but we should not assume that platform administrators are engineers and likely more similar to sysadmins/IT. There is a desire to avoid customers writing rego and this seems like heading down that same direction.
Option 3: JSON Path
🟩 There is a standardized spec
🟩 The most popular npm package has 3m downloads per week and extends on the spec a little
🟩 There are a bunch of online playgrounds available
🟩 / 🟨 Kubectl supports filtering output by json path but also notes limitations like regex as not supported and encourages users to switch to jq for those use cases. However, they appear to use an internal library implementation of the spec which we could also use under the Apache license
🟨 Concerned with select but also much more robust functionality for evaluating than jq which in this case is not desired
🟨 No support for piping output like jq
🟥 In my opinion, it has a much higher cognitive load with more symbols to understand ($, @, ?, &, #, ~, {} for example) and more ways of doing things which will make it harder to build suggested selectors for admins and harder for them to grok
🟥 Does not ship in Linux or the major operating systems