feat: configure connector on start-up

* feat: added deserialization of catalogs to DeserializationService

* fix: getAllCatalogsWithOfferedResources uses the connector's uri from config.json now

Required for updating the configuration model during startup (bootstrapping)

* feat: TemplateBuilder can build catalogs now

* feat: MappingUtils can now map catalogs and offered resources

* chore: refactored MappingUtils

* feat: TemplateUtils can now be used for contract templates and offered resource templates

* feat: added a bootstrapping base path to application.properties

* feat: added missing CatalogTemplate

* feat: added bootstrap configuration

performs registration of ids catalogs (and included elements) during startup and optional registration at broker(s)

* test: added test data for bootstrapping

* fix: added bootstrapConfiguration mockBean to some tests to deactivate bootstrapping

* docs: replaced ampersand in javadoc

* docs: added license headers

* fix: changes based on spotbug report

* fix: changes for checkstyle

* docs: added documentation for bootstrapping

* chore: checkstyle fix

* fix: performing changes requested by review

* fix: performing changes requested by review

* docs: fix subpage order

* docs: update CHANGELOG.md

* feat: add username/password/accessUrl/value for artifacts to bootstrapping

* chore: update debug message

* chore: add valid contract rule to bootstrap file

* feat: changed idsId to bootstrapId and removed it from additional map

* chore: remove invalid attributes from bootrapping files

* fix: catch exception for empty contract dates

* chore: change test bootstrap.properties

* fix: fix bugs regarding ids:target in bootstrapped contract offers, empty bootstrapId, and resourceEndpoints in resources created by MappingUtils

* fix: change tests to match changed code

* docs: added missing @throw entries to javadoc

* fix: incorporate changes required by review

* fix: additional change for changes required by review

* chore: checkstyle fix

* chore: fix NPE when contract start or end not set

* style: fix typo

* chore: remove unchecked warning on BootstrapConfig

* chore: remove unused import

* chore: make local vars final in BootstrapConfig

* chore: remove unused code  in BootstrapConfig

* docs: update bootstrapping.md

* refactor: start with code cleanup of BootstrapConfig

* chore: code clean up

* chore: move methods to BootstrapUtils

* refactor: move bootstrap into own package

* style: fix checkstyle

* refactor: simplify flow of BootstrapConfig::bootstrap

* refactor: move BootstrapUtils into own package

* docs: update bootstrapping.md

* docs: remove empty values and add context to contracts

* chore: add context to bootstrap data

* chore: keep db open when bootstrapping

* chore: dont close db when app terminates

* style: remove unused code

* docs: add package-info

* chore: close http connections

* chore: move broker operations into own class

* chore: simplify broker messaging

* chore: remove unnessarry warnings

* chore: update changelog

* chore: add license header

* test: add broker test

* chore: remove generic function parameter

* docs: add lines for broker registration

* fix: inverted check of optional

* chore: add data for broker registration to bootstrapping data

* fix: bootstrapping no longer exits application when broker registration fails

Add BootstrapConfiguration as MockBean to various tests to save some time during tests
Fix BrokerServiceTest

* docs: add missing @context element to bootstrapping example

* refactor: clean up exception handling and remove BrokerService::getMultipartPart

* chore: change log level for failed registration

* docs: update CHANGELOG.md

* release: increase version to 5.2.0

* chore: update Chart.yaml

* chore: update swagger ui properties

* chore: add pretty print to swagger properties

* test: fix BrokerServiceTest

* refactor: remove unused imports

Co-authored-by: Pampus, Julia <[email protected]>
Co-authored-by: Brian-Frederik Jahnke <[email protected]>
Co-authored-by: Brian-Frederik Jahnke <[email protected]>
Co-authored-by: Julia Pampus <[email protected]>

CHANGELOG.md

@@ -1,7 +1,21 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 
-## [5.X.X] - XXXX-XX-XX
+## [5.2.0] - 2021-06-23
+
+### Added
+- Add `BootstrapConfiguration`.
+  * Allow registering ids catalogs, offered resources, representations, artifacts, and contract
+    offers during start up.
+  * Allow registering offered resources as part of the catalogs to brokers.
+- Add `CatalogTemplate` and matching mapping/build functions.
+- Add a method to `AbstractIdsBuilder` that allows to create elements with a custom base URI.
+- Add `bootstrap.path` to `application.properties` to define the base path where bootstrapping data
+  can be found.
+
+### Changed
+- Change `ConnectorService` to use the connector's ID from `config.json` when
+  `getAllCatalogsWithOfferedResources` is called.
 
 ### Fixed
 - Fixed missing IDS context in `/api/examples/policy`.
@@ -11,6 +25,7 @@ All notable changes to this project will be documented in this file.
 ### Changed
 - Increase IDS Framework version to 5.0.4.
 - Update default Infomodel version to 4.0.10.
+- Increase postgres version to 42.2.22.
 
 ## [5.1.2] - 2021-06-14
 
@@ -20,7 +35,7 @@ All notable changes to this project will be documented in this file.
 - Fixed missing paging information in openapi schema.
 
 ### Changed
-- Increase postgresql version to 42.2.20.
+- Increase postgresql version to 42.2.21.
 - Increase spring-boot version to 2.5.1.
 
 ## [5.1.1] - 2021-06-09

charts/dataspace-connector/Chart.yaml

@@ -31,13 +31,13 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.1.0
+version: 0.1.1
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
 # It is recommended to use it with quotes.
-appVersion: "5.0.2"
+appVersion: "5.2.0"
 
 dependencies:
   - name: postgresql

docs/pages/deployment/bootstrapping.md

@@ -0,0 +1,178 @@
+---
+layout: default
+title: Bootstrapping
+nav_order: 3
+description: ""
+permalink: /Deployment/Bootstrapping
+parent: Deployment
+---
+
+# Bootstrapping
+{: .fs-9 }
+
+Load resources during startup and register them at an IDS Broker.
+{: .fs-6 .fw-300 }
+
+---
+
+In this section, the registration of resources during the startup is described.
+
+### Registering Elements at the Connector
+
+During the startup of the Dataspace Connector, the bootstrapping path that is specified in
+`application.properties` will be scanned for `*.jsonld` and `bootstrap.properties` files. The search
+includes all subdirectories found in the base path.
+
+Each `jsonld`-file contains the JSON-LD representation of an IDS catalog. These representations will
+be loaded and registered at the connector. This includes all elements that are part of the catalog,
+except requested resources. The IDs used for the IDS catalog are used to prevent duplicates among
+multiple startups.
+
+The `bootstrap.properties` files contain additional data that is not present in the IDS Information
+Model representations. This includes credentials, URLs, and values for artifacts as well as
+information on which resources should be registered at IDS Brokers. If multiple
+`bootstrap.properties` files are found, all of them will be loaded and merged. If there are
+collisions, these will be logged and the first found value will be kept.
+
+An example for a valid catalog that can be used for bootstrapping is shown below. Corresponding
+sample files are provided at `test/resources/bootstrap`.
+
+```json
+{
+  "@context" : {
+    "ids" : "https://w3id.org/idsa/core/",
+    "idsc" : "https://w3id.org/idsa/code/"
+  },
+  "@type": "ids:ResourceCatalog",
+  "@id": "https://w3id.org/idsa/autogen/resourceCatalog/4a1fbd8c-8f23-4cc0-871d-9d26596b00c9",
+  "ids:offeredResource": [
+    {
+      "@type": "ids:Resource",
+      "@id": "https://w3id.org/idsa/autogen/resource/e32c5397-f71c-47b5-a384-c9c4cf568117",
+      "ids:language": [
+        {
+          "@id": "idsc:EN"
+        }
+      ],
+      "ids:description": [
+        {
+          "@value": "This is an example resource",
+          "@language": "EN"
+        }
+      ],
+      "ids:version": "3",
+      "ids:title": [
+        {
+          "@value": "Example Another Resource",
+          "@language": "EN"
+        }
+      ],
+      "ids:publisher": "https://example.com",
+      "ids:sovereign": "https://example.com",
+      "ids:representation": [
+        {
+          "@type": "ids:Representation",
+          "@id": "https://w3id.org/idsa/autogen/representation/f74ab1ad-3a1b-4508-aada-4859dcfa7349",
+          "ids:instance": [
+            {
+              "@type": "ids:Artifact",
+              "@id": "https://w3id.org/idsa/autogen/artifact/d5b1cd4e-2a5a-47c2-86c5-003c6a11ce69",
+              "ids:fileName": "exampleFile.xml",
+              "ids:duration": 12512351,
+              "ids:creationDate": "2016-10-03T15:10:40.000Z",
+              "ids:byteSize": 425314,
+              "ids:checkSum": "120ECEF25E5D487BF68B5F709644D219"
+            }
+          ],
+          "ids:language": {
+            "properties": null,
+            "@id": "idsc:EN"
+          },
+          "ids:created": "2016-10-03T15:10:40.000Z",
+          "ids:mediaType": {
+            "@type": "ids:IANAMediaType",
+            "@id": "https://w3id.org/idsa/autogen/iANAMediaType/07e5f54e-4e85-4df7-94d3-a7e8b225f1cb",
+            "ids:filenameExtension": "xml"
+          },
+          "ids:representationStandard": "https://example.com",
+          "ids:shapesGraph": "https://example.com",
+          "ids:modified": "2016-10-03T15:10:40.000Z"
+        }
+      ],
+      "ids:resourceEndpoint": [
+        {
+          "@type": "ids:ConnectorEndpoint",
+          "@id": "https://w3id.org/idsa/autogen/connectorEndpoint/4458078e-c2f2-4d9f-afbe-54e9daa4c1b8",
+          "ids:accessURL": "https://example.com"
+        }
+      ],
+      "ids:standardLicense": "https://example.com",
+      "ids:contractOffer": [
+        {
+          "@type": "ids:ContractOffer",
+          "@id": "https://w3id.org/idsa/autogen/contractOffer/1d982c8a-c5ae-4c19-9a26-6815e9540fc8",
+          "ids:permission": [
+            {
+              "@type": "ids:Permission",
+              "@id": "https://w3id.org/idsa/autogen/permission/a3c12cd0-5022-484d-8fb0-0676351de2da",
+              "ids:description": [
+                {
+                  "@value": "provide-access",
+                  "@type": "http://www.w3.org/2001/XMLSchema#string"
+                }
+              ],
+              "ids:title": [
+                {
+                  "@value": "Allow Data Usage",
+                  "@type": "http://www.w3.org/2001/XMLSchema#string"
+                }
+              ],
+              "ids:action": [
+                {
+                  "@id": "idsc:USE"
+                }
+              ]
+            }
+          ]
+        }
+      ],
+      "ids:keyword": [
+        {
+          "@value": "EXAMPLE",
+          "@language": "EN"
+        }
+      ]
+    }
+  ]
+}
+
+```
+
+An example for a valid `bootstrap.properties` file can be found below.
+
+```properties
+#artifact.accessUrl.https\://w3id.org/idsa/autogen/artifact/d5b1cd4e-2a5a-47c2-86c5-003c6a11ce69=
+#artifact.username.https\://w3id.org/idsa/autogen/artifact/d5b1cd4e-2a5a-47c2-86c5-003c6a11ce69=
+#artifact.password.https\://w3id.org/idsa/autogen/artifact/d5b1cd4e-2a5a-47c2-86c5-003c6a11ce69=
+artifact.value.https\://w3id.org/idsa/autogen/artifact/d5b1cd4e-2a5a-47c2-86c5-003c6a11ce69=Example Value
+broker.register.https\://w3id.org/idsa/autogen/resource/d5b1cd4e-2a5a-47c2-86c5-003c6a11ce69=https://broker.ids.isst.fraunhofer.de/infrastructure
+```
+
+### Registering Elements at the IDS Broker
+Each resource that has been registered during the bootstrapping process can be registered at an
+IDS Broker. It is possible to register different resources at different brokers, but each resource
+can only be registered at one broker once. The registration of a resource at a broker implicitly
+registers the connector itself at the broker.
+
+In order to register a resource at the broker, an entry with the following structure must be placed
+in a `bootstrap.properties` file:
+
+```properties
+broker.register.<RESOURCE_ID>=<BROKER_ENDPOINT>
+```
+
+An example for the first resource of the example IDS catalog can be found below.
+
+```properties
+broker.register.https\://w3id.org/idsa/autogen/resource/e32c5397-f71c-47b5-a384-c9c4cf568117=https://broker.ids.isst.fraunhofer.de/infrastructure
+```

docs/pages/deployment/camel.md

@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Routing
-nav_order: 6
+nav_order: 7
 description: ""
 permalink: /Deployment/Routing
 parent: Deployment

docs/pages/deployment/configuration.md

@@ -307,6 +307,15 @@ opentracing.jaeger.udp-sender.port=6831
 opentracing.jaeger.log-spans=true
 ```
 
+### Bootstrapping
+
+If you want to change the base path, which will be used to find properties and catalogs for bootstrapping,
+you can customize the following line:
+
+```properties
+bootstrap.path=.
+```
+
 ### IDS Settings
 
 URLs of the DAPS for IDS identity management and the Clearing House for contract agreement and data 

docs/pages/deployment/database.md

@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Database
-nav_order: 3
+nav_order: 4
 description: ""
 permalink: /Deployment/Database
 parent: Deployment

docs/pages/deployment/kubernetes.md

@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Kubernetes
-nav_order: 4
+nav_order: 5
 description: ""
 permalink: /Deployment/Kubernetes
 parent: Deployment

docs/pages/deployment/logging.md

@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Logging
-nav_order: 5
+nav_order: 6
 description: ""
 permalink: /Deployment/Logging
 parent: Deployment

openapi.yaml

@@ -25,7 +25,7 @@ info:
   license:
     name: "Apache License, Version 2.0"
     url: https://www.apache.org/licenses/LICENSE-2.0.txt
-  version: 5.1.2
+  version: 5.2.0
 servers:
 - url: https://localhost:8080
   description: Generated server url

pom.xml

@@ -85,7 +85,7 @@
 
 	<properties>
 		<!-- General -->
-		<revision>5.1.2</revision>
+		<revision>5.2.0</revision>
 		<email>[email protected]</email>
 
 		<!-- Build info -->

scripts/ci/pmd-rules.xml

@@ -24,14 +24,19 @@
             Rules for the Dataspace Connector
     </description>
 
-    <rule ref="category/java/bestpractices.xml"></rule>
+    <rule ref="category/java/bestpractices.xml">
+        <!-- Too many false positives, seems to be broken when using var -->
+        <exclude name="UnusedPrivateMethod"/>
+    </rule>
     <rule ref="category/java/codestyle.xml">
         <!-- Small methods are only bloated, big methods should raise eyebrows in code review -->
         <exclude name="OnlyOneReturn"/>
         <!-- Too many false positives -->
         <exclude name="LongVariable"/>
         <!-- Conflicts quite often with unnecessary constructor rule -->
         <exclude name="AtLeastOneConstructor"/>
+        <!-- Clashes with the usage of var -->
+        <exclude name="UseDiamondOperator"/>
     </rule>
     <rule ref="category/java/design.xml">
         <exclude name="LoosePackageCoupling"/>