feat: add opensearch-cedarling plugin

demos/README.md

@@ -11,3 +11,6 @@ Holds a docker load test image packaging for Janssen. This image can load test u
 ## [Janssen Tarp](janssen-tarp)
 A Relying Party tool in form of a Browser Extension for convenient testing of authentication flows on a browser.
 
+## [Opensearch cedarling](opensearch-cedarling)
+
+An opensearch plugin that integrates Cedarling for token-based access control to data queries.

demos/opensearch-cedarling/README.md

@@ -0,0 +1,188 @@
+# OpenSearch Cedarling demo plugin
+
+This is a demo plugin aimed at integrating token-based access control into [OpenSearch](https://opensearch.org). Specifically it is focused on filtering search results obtained in response to search queries sent to any of the endpoints listed [here](https://docs.opensearch.org/docs/latest/api-reference/search-apis/search/). Filtering takes place based on the Cedarling policy provided in the plugin settings.
+
+## Requisites
+
+- OpenSearch 3.6.0
+- [Jans Server](https://github.com/JanssenProject/jans/releases) 1.16.0
+- A browser with tarp extension installed
+- Basic Cedar and OpenSearch knowledge
+- Java 21 and `git` for development
+
+**Notes:**
+
+- All commands given in this document were tested using Ubuntu 22. Accommodate to your specific OS
+- The OpenSearch installation used to test here was single node and packaged-based. Installers can be found [here](https://docs.opensearch.org/docs/latest/install-and-configure/) for several OSes. Keep the admin password at hand
+
+### Create an .netrc file
+
+To avoid typing the OpenSearch password in `curl` commands over and over, create a `~/.netrc` [file](https://everything.curl.dev/usingcurl/netrc.html). Here is how it might look:
+
+```
+machine localhost login admin password secret
+```
+
+### Run the health check
+
+With `curl`, issue a request to the cluster health [endpoint](https://docs.opensearch.org/docs/latest/api-reference/cluster-api/cluster-health/) to check status. By default all API requests are served through port `9200`, e.g. `https://localhost:9200`. It may take some administrative work to properly issue requests from the development machine, however sending `curl` requests directly from the server where OpenSearch resides is OK for testing purposes.
+
+In this document, occurrences of `https://oshost` will refer to the root URL where OpenSearch HTTP API can be reached.
+
+## Create and test a cedar policy
+
+Here, the intention is to create a policy that looks like:
+
+
+```
+@id("alumni_restricted_access")
+permit(
+  principal,
+  action in Jans::Action::"Search",
+  resource is Jans::student
+)
+when {
+  resource.grad_year < 2026 ||
+  (
+    context has tokens.jans_userinfo_token &&
+    context.tokens.jans_userinfo_token.hasTag("role") &&
+    context.tokens.jans_userinfo_token.getTag("role").contains("AdmissionsCounselor")
+  )
+};
+```
+
+with the `student` entity type in the schema:
+
+```
+{
+    "shape": {
+        "type": "Record",
+        "attributes": {
+            "name": {
+                "type": "String"
+            },
+            "grad_year": {
+                "type": "Long"
+            }
+        }
+    }
+}
+```
+<!--
+More attributes can be added if desired.
+-->
+
+For this, follow the steps found [here](https://docs.jans.io/head/cedarling/quick-start/cedarling-quick-start/#implement-rbac-using-signed-tokens-tbac) as a guide. Note it is highly recommended to use Agama lab's policy designer in this case as well as Tarp for quickly testing the policy. Ensure `student` is added to the `Search` action.
+
+<!--
+The `User` resource should be already there if you used . Ensure the `role` attribute is not mandatory.
+
+The easiest way to test the policy is using Tarp. Continue with the steps in the doc [page](https://docs.jans.io/head/cedarling/cedarling-quick-start-tbac/) for this purpose now using the previously setup Jans Server and the policy just created. To get the policy URI in Agama Lab, go to "Policy Stores", click on "Manage" on the corresponding policy row, and then on "Copy link".
+-->
+
+For the short of time, there is a readily available policy store [here](https://github.com/jgomer2001/CedarlingQuickstart/releases/download/v0.0.2/tarpDemo.cjar).
+
+## Plugin deployment
+
+In the development machine, clone this repository (a shallow clone is recommended). `cd` to the directory where this README resides and run `./gradlew assemble -Dopensearch.version=3.6.0`. Ensure `JAVA_HOME` environment points to a Java 21 installation, e.g. `export JAVA_HOME=/path/to/corretto-21`.
+
+`cd` to `build/distributions`. And run:
+
+- `unzip cedarling.zip 'cedarling-java*'`
+- `zip -q -d cedarling-java-0.0.0-nightly.jar 'com/sun/jna/*'`
+- `zip -u cedarling.zip cedarling-java-0.0.0-nightly.jar`
+
+Transfer the file `cedarling.zip` to the OpenSearch server. In a terminal run the below:
+
+- `/usr/share/opensearch/bin/opensearch-plugin remove cedarling` (not required for first time deployment)
+- `/usr/share/opensearch/bin/opensearch-plugin install file:///path/to/cedarling.zip`
+- `systemctl restart opensearch.service`
+
+Verify the plugin was effectively deployed by running `curl -n https://oshost/_cat/plugins`. There should be an entry for `cedarling` in the table.
+
+## Plugin configuration
+
+OpenSearch provides endpoints for handling configuration settings of plugins as well as Java classes for reading those. However when it comes to large, complex, and nested JSON hierarchies, like the settings this plugin may require, OpenSearch facilities are not convenient. For this purpose, an additional endpoint was implemented in order to retrieve and supply configuration settings. This allows to pass complex JSON objects without hassle. Under the hood everything is converted into a big string and stored as a single setting in Opensearch.
+
+Check the file [settings.json](https://raw.githubusercontent.com/jgomer2001/pipelines-plugin/refs/heads/main/settings.json) and fill the value corresponding to the policy store URI.
+
+With a complete settings file, transfer it to the server and call the endpoint `/_plugins/cedarling/settings`:
+
+```
+curl -n -H 'Content-Type: application/json' -d @settings.json -X PUT https://oshost/_plugins/cedarling/settings
+```
+
+The response will contain a boolean value indicating the operation success. The current settings can be retrieved issuing a `GET` to the same endpoint. To check the actual settings OpenSearch stores, use a request like:
+
+```
+curl -n https://oshost/_cluster/settings?pretty
+```
+
+This plugin settings are dynamic and persistent which means they can be altered any time and survive server restarts. Check this [page](https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/index/) for more information in these concepts.
+
+## Setup testing data
+
+In the example policy, the `student` entity type is part of the schema. It is expected all resources referenced by policies exist as OpenSearch indices in equivalence. For this, some "students" should be added:
+
+```
+curl -n -H 'Content-Type: application/json' --data-binary @records.txt https://oshost/student/_bulk
+```
+
+[Here](records.txt) is a sample `records.txt` file. Insert more similar documents varying the `grad_year`. Learn more about Opensearch bulk requests [here](https://docs.opensearch.org/latest/api-reference/document-apis/bulk/).
+
+Issue a request to retrieve the documents added so far:
+
+```
+curl -n -H 'Content-Type: application/json' -d @query.json https://oshost/student/_search?pretty
+```
+
+[query.json](query.json) contains a [search request](https://docs.opensearch.org/docs/latest/query-dsl/) that matches all documents in the index. 
+
+Note that in real world scenarios, indices already exist and policies are built in conformance afterwards. Every resource to add in the schema should resemble existing indices structures. More specifically, resources should at least contain the attributes which are needed for policy evaluation.
+
+## Setup a search pipeline
+
+This plugins implements a search response processor which must be "attached" to a [search pipeline](https://docs.opensearch.org/docs/latest/search-plugins/search-pipelines/index/). Transfer the file [pipeline.json](pipeline.json) to the OpenSearch server and run:
+
+```
+curl -n -H 'Content-Type: application/json' -d @pipeline.json -X PUT https://oshost/_search/pipeline/cedarling_search?pretty
+```
+
+This action needs to be performed only **once** regardless of how many plugin redeployments take place.
+
+With this pipeline, search results may now be filtered as per defined Cedarling policies. See the next section.
+
+## Test
+
+Open Tarp. If this is the first time you use it, add a client there beforehand. Then, in the "Authentication Flow" tab, trigger an authentication flow with the following:
+
+- Acr: `basic`
+- Scope: `openid` and `profile`
+- Check "Display tokens"
+
+After logging in, copy the UserInfo token and paste it in the corresponding section inside file [query_ext.json](query_ext.json). This is an "extended" search query the plugin will have access to so the Cedarling engine can be supplied with tokens and contextual data to make decisions.
+
+Then, run:
+
+```
+curl -n -H 'Content-Type: application/json' -d @query_ext.json 'https://oshost/student/_search?pretty&search_pipeline=cedarling_search'
+```
+
+The `search_pipeline` is required so the response to the query is intercepted and processed by the plugin which in turn will invoke Cedarling. The response will probably contain less hits than the query issued [earlier](#setup-testing-data) and will come with an `ext` section that reports:
+
+- The amount of hits that passed authorization
+- The average decision time per hit
+
+## About development
+
+Once the work to get all of the pieces running is done, making changes to the plugin is rather straightforward: the Java code is in `src` directory and compilation is a matter of issuing `./gradlew compileJava`.
+
+In package-based installations, OpenSearch log is found at `/var/log/opensearch/opensearch.log`. To be able to see the logging statements produced by this plugin, add a line like the below to `/etc/opensearch/opensearch.yml` and restart opensearch (`systemctl restart opensearch.service`): 
+
+```
+logger.io.jans.cedarling: trace
+```
+
+## Benchmarking
+
+See this [page](./benchmark.md).

demos/opensearch-cedarling/benchmark.md

@@ -0,0 +1,44 @@
+# Benchmarks
+
+To compare the performance of regular queries vs. queries filtered via this plugin, benchmarking tests were designed. This is what a test does:
+
+- Removes the index in question entirely (e.g. `student`)
+- Loads in bulk a set of JSON documents. Every document is generated with a (short) random name, a random graduation year (uniformly distributed between 2024 and 2027), and a GPA (random floating number between 0 and 5) 
+- Runs five queries that return documents with GPAs matching the intervals [0, 1), [1, 2), etc. <!--Before this, a preliminary warmup query is issued for the interval [5, 6) and its result discarded: it was observed that after the bulk is performed, the very first query takes a very long time compared to subsequent queries -->
+- The average query response time is computed. This does not include network latency - only server-side processing
+
+The test is run twice, one without Cedarling (regular OpenSearch query), and one with the plugin. To avoid bias due to caching or other factors, the database is restarted before executing every test. The average decision time per document is also reported when using the plugin. <!--This allows to discriminate the overhead introduced by Cedarling and the plugin separately.  -->
+
+Details like the server to point to, index name, and the number of random entries to generate, among others, are parameterizable.
+
+## Performance data
+
+The following data were obtained in a setup using a [Digital Ocean](https://www.digitalocean.com/products/droplets/) Basic VM (s-4vcpu-8gb) with Ubuntu 22, OpenSearch 3.6.0 (single node, package-based installation with default configuration), and (local) Jans Server 1.16.0 (AS and database components only):
+
+|Measurement|Value|
+|-|-|
+|Average query response time (regular)|114.6 ms|
+|Average query response time (plugin)|4837.5 ms|
+|Ratio|42.21|
+|Average Cedarling authz time per document|2.2ms|
+
+The number of documents (bulk size) per test was 10,000. Given the uniform distribution of documents among GPA values, each of the five queries returned approximately 2,000 hits, which leads to the following average query response times per document:
+
+- Regular: 0.0573 ms
+- Plugin: 2.41875 ms
+
+The above shows that most of the processing time is due to Cedarling authorization while the rest of plugin code only accounts for 9% of the overhead (0.21875 ms). 
+
+These tests force the retrieval of all matching documents at once every time, however in practice many apps will process query responses in small pages - the default search endpoint page size in OpenSearch is 10. This means despite the big ratio obtained (42.21), using Cedarling authorization is promising, specifically when retrieving data in small batches of documents is fine. For example 40 documents could be retrieved in approximately 97 milliseconds in the circumstances described in this document.
+
+## How to run
+
+For the interested, the below are the instructions to run a test:
+
+- Ensure to deploy and configure the plugin as explained in the [README](./README.md) <!--. At the top level of the JSON settings, add `"skipHits": true`. This will make the plugin omit the serialization of the results (hits) in the response. This avoids transfering a lot of data through the network and reduces the running time of the Java test considerably without effects in the computations of performance metrics -->
+- `cd` to the directory where this document resides
+- Edit the file `src/test/resources/testng.properties` accordingly. For `entries`, a value like `10000` (ten thousand documents) is OK. Using a higher value may require tweaking OpenSearch `index.max_result_window` property, see [index settings](https://docs.opensearch.org/latest/install-and-configure/configuring-opensearch/index-settings/)
+- If the plugin will be in use for this particular test, set property `useCedarling` to `true`, and edit the accompanying file `query.json` supplying the tokens - these can be obtained via Tarp as mentioned in the README file
+- Restart OpenSearch
+- Run `./gradlew test`. Ensure the certificate keystore of Java trusts the certificate that protects the OpenSearch REST API endpoints
+- The report in HTML format can be found under `build/reports/tests`. The last lines of the standard output will contain relevant metrics

demos/opensearch-cedarling/build.gradle

@@ -0,0 +1,156 @@
+buildscript {
+    ext {
+        opensearch_version = System.getProperty("opensearch.version", "3.6.0")
+    }
+
+    repositories {
+        mavenLocal()
+        maven { url "https://aws.oss.sonatype.org/content/repositories/snapshots" }
+        mavenCentral()
+        maven { url "https://plugins.gradle.org/m2/" }
+    }
+
+    dependencies {
+        classpath "org.opensearch.gradle:build-tools:${opensearch_version}"
+    }
+}
+
+import org.opensearch.gradle.test.RestIntegTestTask
+
+apply plugin: 'java'
+apply plugin: 'idea'
+apply plugin: 'eclipse'
+apply plugin: 'opensearch.opensearchplugin'
+apply plugin: 'opensearch.yaml-rest-test'
+apply plugin: 'opensearch.pluginzip'
+
+def pluginName = 'cedarling' 
+def pluginDescription = 'A plugin featuring TBAC security to database operations'
+def packagePath = 'io.jans'     
+def pathToPlugin = 'cedarling.opensearch'
+def pluginClassName = 'CedarlingPlugin'
+group = "io.jans"
+
+java {
+    targetCompatibility = JavaVersion.VERSION_21
+    sourceCompatibility = JavaVersion.VERSION_21
+}
+
+tasks.register("preparePluginPathDirs") {
+    mustRunAfter clean
+    doLast {
+        def newPath = pathToPlugin.replace(".", "/")
+        mkdir "src/main/java/$packagePath/$newPath"
+        mkdir "src/test/java/$packagePath/$newPath"
+        mkdir "src/yamlRestTest/java/$packagePath/$newPath"
+    }
+}
+
+publishing {
+    publications {
+        pluginZip(MavenPublication) { publication ->
+            pom {
+              name = pluginName
+              description = pluginDescription
+              licenses {
+                license {
+                  name = "The Apache License, Version 2.0"
+                  url = "http://www.apache.org/licenses/LICENSE-2.0.txt"
+                }
+              }
+              developers {
+                developer {
+                  name = "OpenSearch"
+                  url = "https://github.com/opensearch-project/opensearch-plugin-template-java"
+                }
+              }
+            }
+        }
+    }
+}
+
+opensearchplugin {
+    name pluginName
+    description pluginDescription
+    classname "${packagePath}.${pathToPlugin}.${pluginClassName}"
+    version "1.0.0-SNAPSHOT"
+}
+
+// This requires an additional Jar not published as part of build-tools
+loggerUsageCheck.enabled = false
+
+// No need to validate pom, as we do not upload to maven/sonatype
+validateNebulaPom.enabled = false
+
+repositories {
+    mavenLocal()
+    maven { url "https://aws.oss.sonatype.org/content/repositories/snapshots" }
+    mavenCentral()
+    maven { url "https://plugins.gradle.org/m2/" }
+    maven { url "https://maven.jans.io/maven" }
+}
+
+dependencies {
+    // JSON processing
+    implementation "org.json:json:20231013"
+    // Logging
+    implementation "org.apache.logging.log4j:log4j-api:2.25.3"
+    implementation "org.apache.logging.log4j:log4j-core:2.25.3"
+    // Cedarling FFI
+    implementation "io.jans:cedarling-java:2.0.0"
+
+    // Testing dependencies
+    testImplementation "org.testng:testng:7.11.0"
+    testImplementation "org.jcommander:jcommander:2.0"
+    testImplementation "com.nimbusds:oauth2-oidc-sdk:11.26.1"
+    testImplementation "com.nimbusds:content-type:2.3"
+}
+
+test {
+    include '**/*Test.class'
+
+    useTestNG() {
+        suites 'src/test/resources/suite.xml'
+    }
+    testLogging {
+        events "passed", "skipped", "failed"
+        exceptionFormat "full"
+    }
+}
+
+task integTest(type: RestIntegTestTask) {
+    description = "Run tests against a cluster"
+    testClassesDirs = sourceSets.test.output.classesDirs
+    classpath = sourceSets.test.runtimeClasspath
+}
+tasks.named("check").configure { dependsOn(integTest) }
+
+integTest {
+    // The --debug-jvm command-line option makes the cluster debuggable; this makes the tests debuggable
+    if (System.getProperty("test.debug") != null) {
+        jvmArgs '-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005'
+    }
+}
+
+testClusters.integTest {
+    testDistribution = "INTEG_TEST"
+
+    // This installs our plugin into the testClusters
+    plugin(project.tasks.bundlePlugin.archiveFile)
+}
+
+run {
+    useCluster testClusters.integTest
+}
+
+// updateVersion: Task to auto update version to the next development iteration
+task updateVersion {
+    onlyIf { System.getProperty('newVersion') }
+    doLast {
+        ext.newVersion = System.getProperty('newVersion')
+        println "Setting version to ${newVersion}."
+         // String tokenization to support -SNAPSHOT
+        ant.replaceregexp(file:'build.gradle', match: '"opensearch.version", "\\d.*"', replace: '"opensearch.version", "' + newVersion.tokenize('-')[0] + '-SNAPSHOT"', flags:'g', byline:true)
+    }
+}
+