Skip to content

Commit 8dedd3a

Browse files
mofterdingerlisajulia
mofterdinger
and
lisajulia
authored
Connection to AWS, Azure and Google object store (#481)
Co-authored-by: Lisa Julia Nebel <[email protected]>
1 parent dcf00a9 commit 8dedd3a

File tree

28 files changed

+2020
-5
lines changed

28 files changed

+2020
-5
lines changed

‎.github/actions/integration-tests/action.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,6 @@ runs:
2727
- name: Integration Tests with latest version of CAP Java
2828
run: mvn clean verify -ntp -B -f ./integration-tests/pom.xml -P latest-test-version
2929
shell: bash
30+
- name: Integration Tests for the object store service
31+
run: mvn clean verify -Pintegration-tests-oss
32+
shell: bash

‎.github/workflows/main-build-and-deploy-oss.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,17 @@ jobs:
5151
name: Build and Sonar Scan
5252
runs-on: ubuntu-latest
5353
needs: update-version
54+
env:
55+
AWS_S3_HOST: ${{ secrets.AWS_S3_HOST }}
56+
AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
57+
AWS_S3_REGION: ${{ secrets.AWS_S3_REGION }}
58+
AWS_S3_ACCESS_KEY_ID: ${{ secrets.AWS_S3_ACCESS_KEY_ID }}
59+
AWS_S3_SECRET_ACCESS_KEY: ${{ secrets.AWS_S3_SECRET_ACCESS_KEY }}
60+
AZURE_CONTAINER_URI: ${{ secrets.AZURE_CONTAINER_URI }}
61+
AZURE_SAS_TOKEN: ${{ secrets.AZURE_SAS_TOKEN }}
62+
GS_BASE_64_ENCODED_PRIVATE_KEY_DATA: ${{ secrets.GS_BASE_64_ENCODED_PRIVATE_KEY_DATA }}
63+
GS_BUCKET: ${{ secrets.GS_BUCKET }}
64+
GS_PROJECT_ID: ${{ secrets.GS_PROJECT_ID }}
5465
steps:
5566
- name: Download artifact
5667
uses: actions/download-artifact@v5

‎.github/workflows/main-build.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,17 @@ jobs:
1515
strategy:
1616
matrix:
1717
java-version: [ 17, 21 ]
18+
env:
19+
AWS_S3_HOST: ${{ secrets.AWS_S3_HOST }}
20+
AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
21+
AWS_S3_REGION: ${{ secrets.AWS_S3_REGION }}
22+
AWS_S3_ACCESS_KEY_ID: ${{ secrets.AWS_S3_ACCESS_KEY_ID }}
23+
AWS_S3_SECRET_ACCESS_KEY: ${{ secrets.AWS_S3_SECRET_ACCESS_KEY }}
24+
AZURE_CONTAINER_URI: ${{ secrets.AZURE_CONTAINER_URI }}
25+
AZURE_SAS_TOKEN: ${{ secrets.AZURE_SAS_TOKEN }}
26+
GS_BASE_64_ENCODED_PRIVATE_KEY_DATA: ${{ secrets.GS_BASE_64_ENCODED_PRIVATE_KEY_DATA }}
27+
GS_BUCKET: ${{ secrets.GS_BUCKET }}
28+
GS_PROJECT_ID: ${{ secrets.GS_PROJECT_ID }}
1829
steps:
1930
- name: Checkout
2031
uses: actions/checkout@v4

‎.github/workflows/pull-request-build.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,18 @@ jobs:
1515
strategy:
1616
matrix:
1717
java-version: [ 17, 21 ]
18+
19+
env:
20+
AWS_S3_HOST: ${{ secrets.AWS_S3_HOST }}
21+
AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
22+
AWS_S3_REGION: ${{ secrets.AWS_S3_REGION }}
23+
AWS_S3_ACCESS_KEY_ID: ${{ secrets.AWS_S3_ACCESS_KEY_ID }}
24+
AWS_S3_SECRET_ACCESS_KEY: ${{ secrets.AWS_S3_SECRET_ACCESS_KEY }}
25+
AZURE_CONTAINER_URI: ${{ secrets.AZURE_CONTAINER_URI }}
26+
AZURE_SAS_TOKEN: ${{ secrets.AZURE_SAS_TOKEN }}
27+
GS_BASE_64_ENCODED_PRIVATE_KEY_DATA: ${{ secrets.GS_BASE_64_ENCODED_PRIVATE_KEY_DATA }}
28+
GS_BUCKET: ${{ secrets.GS_BUCKET }}
29+
GS_PROJECT_ID: ${{ secrets.GS_PROJECT_ID }}
1830

1931
steps:
2032
- name: Checkout

‎.pipeline/config.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,5 +35,5 @@ steps:
3535
- sonar.qualitygate.wait=true
3636
- sonar.java.source=17
3737
- sonar.exclusions=**/node_modules/**,**/target/**,**/test/**
38-
- sonar.coverage.jacoco.xmlReportPaths=cds-feature-attachments/target/site/jacoco/jacoco.xml
39-
- sonar.coverage.exclusions=cds-feature-attachments/src/test/**,cds-feature-attachments/src/gen/**,integration-tests/**,storage-targets/cds-feature-attachments-fs/**
38+
- sonar.coverage.jacoco.xmlReportPaths=cds-feature-attachments/target/site/jacoco/jacoco.xml,storage-targets/cds-feature-attachments-oss/target/site/jacoco/jacoco.xml
39+
- sonar.coverage.exclusions=cds-feature-attachments/src/test/**,cds-feature-attachments/src/gen/**,integration-tests/**,storage-targets/cds-feature-attachments-fs/**,storage-targets/cds-feature-attachments-oss/src/test/**

‎README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -408,5 +408,8 @@ the [CAP Java Documentation](https://cap.cloud.sap/docs/java/security).
408408

409409
By default, the plugin operates without a dedicated storage target, storing attachments directly in the [underlying database](cds-feature-attachments/src/main/resources/cds/com.sap.cds/cds-feature-attachments/attachments.cds#L17).
410410

411-
For testing scenarios, you can instead use the [local file system as a storage backend](storage-targets/cds-feature-attachments-fs).
412-
Then, the attachment is not stored in the underlying database; instead, it is saved on the local file system, and only a reference to the file is kept in the database, as defined in the [CDS model](cds-feature-attachments/src/main/resources/cds/com.sap.cds/cds-feature-attachments/attachments.cds#L20).
411+
Other available storage targets:
412+
- [Object Store](storage-targets/cds-feature-attachments-oss)
413+
- [local file system as a storage backend](storage-targets/cds-feature-attachments-fs) (only for testing scenarios)
414+
415+
When using a dedicated storage target, the attachment is not stored in the underlying database; instead, it is saved on the specified storage target, and only a reference to the file is kept in the database, as defined in the [CDS model](cds-feature-attachments/src/main/resources/cds/com.sap.cds/cds-feature-attachments/attachments.cds#L20).

‎pom.xml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,16 @@
2323
</developers>
2424

2525
<properties>
26-
<revision>1.1.1-SNAPSHOT</revision>
26+
<revision>1.1.1</revision>
2727
<java.version>17</java.version>
2828
<maven.compiler.release>${java.version}</maven.compiler.release>
2929
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
3030

3131
<excluded.generation.package>com/sap/cds/feature/attachments/generated/</excluded.generation.package>
3232

33+
<software.amazon.awssdk-s3-version>2.32.4</software.amazon.awssdk-s3-version>
34+
<software.amazon.awssdk-crt-version>0.38.7</software.amazon.awssdk-crt-version>
35+
3336
<!-- Versions of CAP Java and cds-dk used for build and integrations tests -->
3437
<cds.services.version>3.10.3</cds.services.version> <!-- https://central.sonatype.com/artifact/com.sap.cds/cds-services-api/versions -->
3538
<cds.cdsdk-version>8.9.8</cds.cdsdk-version> <!-- https://www.npmjs.com/package/@sap/cds-dk?activeTab=versions -->
@@ -51,6 +54,7 @@
5154
<modules>
5255
<module>cds-feature-attachments</module>
5356
<module>storage-targets/cds-feature-attachments-fs</module>
57+
<module>storage-targets/cds-feature-attachments-oss</module>
5458
<module>integration-tests</module>
5559
</modules>
5660

‎storage-targets/cds-feature-attachments-oss/README.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
### General Info
2+
3+
This artifact uses the an Object Store as the storage target instead of the underlying database.
4+
5+
Then, the attachment is not stored in the underlying database; instead, it is saved in the respective Object Store, and only a reference to the file is kept in the database, as defined in the [CDS model](../../cds-feature-attachments/src/main/resources/cds/com.sap.cds/cds-feature-attachments/attachments.cds#L20).
6+
7+
To do this, replace the `cds-feature-attachments` dependency in your `pom.xml` with:
8+
9+
```xml
10+
<dependency>
11+
<groupId>com.sap.cds</groupId>
12+
<artifactId>cds-feature-attachments-oss</artifactId>
13+
<version>${latest-version}</version>
14+
</dependency>
15+
```
16+
17+
A valid Object Store service binding is required for this — for example, one provisioned through SAP BTP.
18+
19+
#### Tests
20+
21+
The unit tests in this module do not need a binding to the respective object stores, run them with `mvn clean install`.
22+
23+
The integration tests need a binding to a real object store, run them with `mvn clean install -Pintegration-tests-oss`.
24+
To set the binding, provide the following environment variables:
25+
- AWS_S3_BUCKET
26+
- AWS_S3_REGION
27+
- AWS_S3_ACCESS_KEY_ID
28+
- AWS_S3_SECRET_ACCESS_KEY
29+
- AZURE_CONTAINER_URI
30+
- AZURE_SAS_TOKEN
31+
- GS_BUCKET
32+
- GS_PROJECT_ID
33+
- GS_BASE_64_ENCODED_PRIVATE_KEY_DATA
34+
35+
### Implementation details
36+
37+
This artifact provides custom handlers for events from the [AttachmentService](../../cds-feature-attachments/src/main/java/com/sap/cds/feature/attachments/service/AttachmentService.java).
38+
39+
### Supported Storage Backends
40+
41+
- **AWS S3**
42+
- **Azure Blob Storage**
43+
- **Google Cloud Storage**
44+
45+
### Multitenancy
46+
47+
Multitenancy is not directly supported. All attachments are stored in a flat structure within the provided bucket, which might be shared across tenants.

‎storage-targets/cds-feature-attachments-oss/pom.xml‎

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
2+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
5+
<parent>
6+
<groupId>com.sap.cds</groupId>
7+
<artifactId>cds-feature-attachments-root</artifactId>
8+
<version>${revision}</version>
9+
<relativePath>../..</relativePath>
10+
</parent>
11+
12+
<artifactId>cds-feature-attachments-oss</artifactId>
13+
<packaging>jar</packaging>
14+
15+
<name>CDS Feature for Attachments - Object Store Service</name>
16+
<url>https://cap.cloud.sap/docs/plugins/#attachments</url>
17+
18+
<dependencies>
19+
<dependency>
20+
<groupId>com.sap.cds</groupId>
21+
<artifactId>cds-feature-attachments</artifactId>
22+
</dependency>
23+
24+
<dependency>
25+
<groupId>com.sap.cds</groupId>
26+
<artifactId>cds-services-api</artifactId>
27+
</dependency>
28+
29+
<dependency>
30+
<groupId>software.amazon.awssdk</groupId>
31+
<artifactId>s3</artifactId>
32+
<version>${software.amazon.awssdk-s3-version}</version>
33+
</dependency>
34+
35+
<dependency>
36+
<groupId>software.amazon.awssdk.crt</groupId>
37+
<artifactId>aws-crt</artifactId>
38+
<version>${software.amazon.awssdk-crt-version}</version>
39+
</dependency>
40+
41+
<dependency>
42+
<groupId>com.azure</groupId>
43+
<artifactId>azure-storage-blob</artifactId>
44+
<version>12.25.0</version>
45+
</dependency>
46+
47+
<dependency>
48+
<groupId>com.google.cloud</groupId>
49+
<artifactId>google-cloud-storage</artifactId>
50+
<version>2.30.0</version>
51+
</dependency>
52+
53+
<dependency>
54+
<groupId>com.sap.cloud.sdk</groupId>
55+
<artifactId>sdk-bom</artifactId>
56+
<version>5.20.0</version>
57+
<type>pom</type>
58+
<scope>import</scope>
59+
</dependency>
60+
61+
<!-- TESTS -->
62+
<dependency>
63+
<groupId>com.sap.cds</groupId>
64+
<artifactId>cds-services-impl</artifactId>
65+
<scope>test</scope>
66+
</dependency>
67+
<dependency>
68+
<groupId>ch.qos.logback</groupId>
69+
<artifactId>logback-classic</artifactId>
70+
<scope>test</scope>
71+
</dependency>
72+
<dependency>
73+
<groupId>org.awaitility</groupId>
74+
<artifactId>awaitility</artifactId>
75+
<scope>test</scope>
76+
</dependency>
77+
</dependencies>
78+
79+
<build>
80+
<plugins>
81+
<plugin>
82+
<groupId>org.jacoco</groupId>
83+
<artifactId>jacoco-maven-plugin</artifactId>
84+
<configuration>
85+
<excludes>
86+
<exclude>
87+
${excluded.generation.package}**/*
88+
</exclude>
89+
</excludes>
90+
</configuration>
91+
<executions>
92+
<execution>
93+
<id>jacoco-initialize</id>
94+
<goals>
95+
<goal>prepare-agent</goal>
96+
</goals>
97+
</execution>
98+
<execution>
99+
<id>jacoco-site-report-all-tests</id>
100+
<phase>verify</phase>
101+
<goals>
102+
<goal>report</goal>
103+
</goals>
104+
</execution>
105+
<execution>
106+
<id>jacoco-site-report-only-unit-tests</id>
107+
<phase>test</phase>
108+
<goals>
109+
<goal>report</goal>
110+
</goals>
111+
</execution>
112+
<execution>
113+
<id>jacoco-check-unit-tests-only</id>
114+
<phase>test</phase>
115+
<goals>
116+
<goal>check</goal>
117+
</goals>
118+
<configuration>
119+
<rules>
120+
<rule implementation="org.jacoco.maven.RuleConfiguration">
121+
<element>BUNDLE</element>
122+
<limits>
123+
<limit implementation="org.jacoco.report.check.Limit">
124+
<counter>INSTRUCTION</counter>
125+
<value>COVEREDRATIO</value>
126+
<minimum>0.80</minimum>
127+
</limit>
128+
<limit implementation="org.jacoco.report.check.Limit">
129+
<counter>BRANCH</counter>
130+
<value>COVEREDRATIO</value>
131+
<minimum>0.75</minimum>
132+
</limit>
133+
<limit implementation="org.jacoco.report.check.Limit">
134+
<counter>COMPLEXITY</counter>
135+
<value>COVEREDRATIO</value>
136+
<minimum>0.80</minimum>
137+
</limit>
138+
<limit implementation="org.jacoco.report.check.Limit">
139+
<counter>CLASS</counter>
140+
<value>MISSEDCOUNT</value>
141+
<maximum>0</maximum>
142+
</limit>
143+
</limits>
144+
</rule>
145+
</rules>
146+
</configuration>
147+
</execution>
148+
</executions>
149+
</plugin>
150+
151+
<plugin>
152+
<groupId>com.sap.cds</groupId>
153+
<artifactId>cds-maven-plugin</artifactId>
154+
<executions>
155+
<execution>
156+
<id>cds.resolve</id>
157+
<goals>
158+
<goal>resolve</goal>
159+
</goals>
160+
</execution>
161+
</executions>
162+
</plugin>
163+
164+
<plugin>
165+
<groupId>com.github.spotbugs</groupId>
166+
<artifactId>spotbugs-maven-plugin</artifactId>
167+
<configuration>
168+
<skip>true</skip>
169+
</configuration>
170+
</plugin>
171+
172+
</plugins>
173+
</build>
174+
<profiles>
175+
<profile>
176+
<id>integration-tests-oss</id>
177+
<build>
178+
<plugins>
179+
<plugin>
180+
<groupId>org.apache.maven.plugins</groupId>
181+
<artifactId>maven-failsafe-plugin</artifactId>
182+
<version>3.5.3</version>
183+
<executions>
184+
<execution>
185+
<id>integration-tests</id>
186+
<goals>
187+
<goal>integration-test</goal>
188+
<goal>verify</goal>
189+
</goals>
190+
</execution>
191+
</executions>
192+
<configuration>
193+
<includes>
194+
<include>**/*IT.java</include>
195+
</includes>
196+
</configuration>
197+
</plugin>
198+
</plugins>
199+
</build>
200+
</profile>
201+
</profiles>
202+
203+
</project>

0 commit comments

Comments
 (0)