Skip to content

[ZEPPELIN-2950] Support Ceph as a notebook storage #2598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kjmrknsn wants to merge 1 commit into from
Closed

[ZEPPELIN-2950] Support Ceph as a notebook storage #2598

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions conf/zeppelin-site.xml.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,16 @@
</property>
-->

<!-- Optional override to control which signature algorithm should be used to sign AWS requests -->
<!-- Set this property to "S3SignerType" if your AWS S3 compatible APIs support only AWS Signature Version 2 such as Ceph. -->
<!--
<property>
<name>zeppelin.notebook.s3.signerOverride</name>
<value>S3SignerType</value>
<description>optional override to control which signature algorithm should be used to sign AWS requests</description>
</property>
-->

<!-- If using Azure for storage use the following settings -->
<!--
<property>
Expand Down
6 changes: 6 additions & 0 deletions docs/setup/operation/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,12 @@ If both are defined, then the **environment variables** will take priority.
<td>false</td>
<td>Save notebooks to S3 with server-side encryption enabled</td>
</tr>
<tr>
<td><h6 class="properties">ZEPPELIN_NOTEBOOK_S3_SIGNEROVERRIDE</h6></td>
<td><h6 class="properties">zeppelin.notebook.s3.signerOverride</h6></td>
<td></td>
<td>Optional override to control which signature algorithm should be used to sign AWS requests</td>
</tr>
<tr>
<td><h6 class="properties">ZEPPELIN_NOTEBOOK_AZURE_CONNECTION_STRING</h6></td>
<td><h6 class="properties">zeppelin.notebook.azure.connectionString</h6></td>
Expand Down
5 changes: 5 additions & 0 deletions zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,10 @@ public boolean isS3ServerSideEncryption() {
return getBoolean(ConfVars.ZEPPELIN_NOTEBOOK_S3_SSE);
}

public String getS3SignerOverride() {
return getString(ConfVars.ZEPPELIN_NOTEBOOK_S3_SIGNEROVERRIDE);
}

public String getMongoUri() {
return getString(ConfVars.ZEPPELIN_NOTEBOOK_MONGO_URI);
}
Expand Down Expand Up @@ -649,6 +653,7 @@ public static enum ConfVars {
ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID("zeppelin.notebook.s3.kmsKeyID", null),
ZEPPELIN_NOTEBOOK_S3_KMS_KEY_REGION("zeppelin.notebook.s3.kmsKeyRegion", null),
ZEPPELIN_NOTEBOOK_S3_SSE("zeppelin.notebook.s3.sse", false),
ZEPPELIN_NOTEBOOK_S3_SIGNEROVERRIDE("zeppelin.notebook.s3.signerOverride", null),
ZEPPELIN_NOTEBOOK_AZURE_CONNECTION_STRING("zeppelin.notebook.azure.connectionString", null),
ZEPPELIN_NOTEBOOK_AZURE_SHARE("zeppelin.notebook.azure.share", "zeppelin"),
ZEPPELIN_NOTEBOOK_AZURE_USER("zeppelin.notebook.azure.user", "user"),
Expand Down
33 changes: 24 additions & 9 deletions zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/S3NotebookRepo.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@
import org.slf4j.LoggerFactory;

import com.amazonaws.AmazonClientException;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.ClientConfigurationFactory;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.services.s3.AmazonS3;
Expand Down Expand Up @@ -94,33 +96,30 @@ public S3NotebookRepo(ZeppelinConfiguration conf) throws IOException {

// always use the default provider chain
AWSCredentialsProvider credentialsProvider = new DefaultAWSCredentialsProviderChain();
CryptoConfiguration cryptoConf = null;
CryptoConfiguration cryptoConf = new CryptoConfiguration();
String keyRegion = conf.getS3KMSKeyRegion();

if (StringUtils.isNotBlank(keyRegion)) {
cryptoConf = new CryptoConfiguration();
cryptoConf.setAwsKmsRegion(Region.getRegion(Regions.fromName(keyRegion)));
}

ClientConfiguration cliConf = createClientConfiguration();

// see if we should be encrypting data in S3
String kmsKeyID = conf.getS3KMSKeyID();
if (kmsKeyID != null) {
// use the AWS KMS to encrypt data
KMSEncryptionMaterialsProvider emp = new KMSEncryptionMaterialsProvider(kmsKeyID);
if (cryptoConf != null) {
this.s3client = new AmazonS3EncryptionClient(credentialsProvider, emp, cryptoConf);
} else {
this.s3client = new AmazonS3EncryptionClient(credentialsProvider, emp);
}
this.s3client = new AmazonS3EncryptionClient(credentialsProvider, emp, cliConf, cryptoConf);
Copy link
Contributor Author

@kjmrknsn kjmrknsn Sep 26, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The if branch is removed for the simplicity but the internal process of each branch is not changed because new CryptoConfiguration() is called in the new AmazonS3EncryptionClient(credentialsProvider, emp) constructor eventually.

}
else if (conf.getS3EncryptionMaterialsProviderClass() != null) {
// use a custom encryption materials provider class
EncryptionMaterialsProvider emp = createCustomProvider(conf);
this.s3client = new AmazonS3EncryptionClient(credentialsProvider, emp);
this.s3client = new AmazonS3EncryptionClient(credentialsProvider, emp, cliConf, cryptoConf);
Copy link
Contributor Author

@kjmrknsn kjmrknsn Sep 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cryptoConf is passed to the new AmazonS3EncryptionClient constructor besides cliConf because there's no constructor definition of new AmazonS3EncryptionClient(credentialsProvider, emp, cliConf).

However, its internal process is not changed because new CryptoConfiguration() is called in the new AmazonS3EncryptionClient(credentialsProvider, emp) constructor eventually.

}
else {
// regular S3
this.s3client = new AmazonS3Client(credentialsProvider);
this.s3client = new AmazonS3Client(credentialsProvider, cliConf);
}

// set S3 endpoint to use
Expand Down Expand Up @@ -154,6 +153,22 @@ private EncryptionMaterialsProvider createCustomProvider(ZeppelinConfiguration c
return emp;
}

/**
* Create AWS client configuration and return it.
* @return AWS client configuration
*/
private ClientConfiguration createClientConfiguration() {
ClientConfigurationFactory configFactory = new ClientConfigurationFactory();
ClientConfiguration config = configFactory.getConfig();

String s3SignerOverride = conf.getS3SignerOverride();
if (StringUtils.isNotBlank(s3SignerOverride)) {
config.setSignerOverride(s3SignerOverride);
}

return config;
}
Copy link
Contributor Author

@kjmrknsn kjmrknsn Sep 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The process of creating ClientConfiguration is made separated from the S3NotebookRepo constructor for making the future modification of this process easier than defining it in the S3NotebookRepo constructor.


@Override
public List<NoteInfo> list(AuthenticationInfo subject) throws IOException {
List<NoteInfo> infos = new LinkedList<>();
Expand Down