HDDS-8605. Implement the ability to update the ServiceInfo object with the new rootCA

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/CertificateClient.java

@@ -35,6 +35,8 @@
 import java.util.List;
 import java.util.Objects;
 import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
 
 import static org.apache.hadoop.hdds.security.exception.OzoneSecurityException.ResultCodes.OM_PUBLIC_PRIVATE_KEY_FILE_NOT_EXIST;
 
@@ -232,6 +234,15 @@ default void assertValidKeysAndCertificate() throws OzoneSecurityException {
    */
   void registerNotificationReceiver(CertificateNotification receiver);
 
+  /**
+   * Registers a listener that will be notified if the CA certificates are
+   * changed.
+   *
+   * @param listener the listener to call with the actualized list of CA
+   *                 certificates.
+   */
+  void registerRootCARotationListener(
+      Function<List<X509Certificate>, CompletableFuture<Void>> listener);
 
   /**
    * Initialize certificate client.

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/keys/package-info.java

@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+/**
+ * Utils for private and public keys.
+ */
+package org.apache.hadoop.hdds.security.x509.keys;

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/package-info.java

@@ -23,77 +23,3 @@
  * framework for HDDS.
  */
 package org.apache.hadoop.hdds.security.x509;
-/*
-
-Architecture of Certificate Infrastructure for SCM.
-====================================================
-
-The certificate infrastructure has two main parts, the certificate server or
-the Certificate authority and the clients who want certificates. The CA is
-responsible for issuing certificates to participating entities.
-
-To issue a certificate the CA has to verify the identity and the assertions
-in the certificate. The client starts off making a request to CA for a
-certificate.  This request is called Certificate Signing Request or CSR
-(PKCS#10).
-
-When a CSR arrives on the CA, CA will decode the CSR and verify that all the
-fields in the CSR are in line with what the system expects. Since there are
-lots of possible ways to construct an X.509 certificate, we rely on PKI
-profiles.
-
-Generally, PKI profiles are policy documents or general guidelines that get
-followed by the requester and CA. However, most of the PKI profiles that are
-commonly available are general purpose and offers too much surface area.
-
-SCM CA infrastructure supports the notion of a PKI profile class which can
-codify the RDNs, Extensions and other certificate policies. The CA when
-issuing a certificate will invoke a certificate approver class, based on the
-authentication method used. For example, out of the box, we support manual,
-Kerberos, trusted network and testing authentication mechanisms.
-
-If there is no authentication mechanism in place, then when CA receives the
-CSR, it runs the standard PKI profile over it verify that all the fields are
-in expected ranges. Once that is done, The signing request is sent for human
-review and approval. This form of certificate approval is called Manual,  Of
-all the certificate approval process this is the ** most secure **. This
-approval needs to be done once for each data node.
-
-For existing clusters, where data nodes already have a Kerberos keytab,  we
-can leverage the Kerberos identity mechanism to identify the data node that
-is requesting the certificate. In this case, users can configure the system
-to leverage Kerberos while issuing certificates and SCM CA will be able to
-verify the data nodes identity and issue certificates automatically.
-
-In environments like Kubernetes, we can leverage the base system services to
-pass on a shared secret securely. In this model also, we can rely on these
-secrets to make sure that is the right data node that is talking to us. This
-kind of approval is called a Trusted network approval. In this process, each
-data node not only sends the CSR but signs the request with a shared secret
-with SCM. SCM then can issue a certificate without the intervention of a
-human administrator.
-
-The last, TESTING method which never should be used other than in development
- and testing clusters, is merely a mechanism to bypass all identity checks. If
-this flag is setup, then CA will issue a CSR if the base approves all fields.
-
- * Please do not use this mechanism(TESTING) for any purpose other than
- * testing.
-
-CA - Certificate Approval and Code Layout (as of Dec, 1st, 2018)
-=================================================================
-The CA implementation ( as of now it is called DefaultCA) receives a CSR from
- the network layer. The network also tells the system what approver type to
- use, that is if Kerberos or Shared secrets mechanism is used, it reports
- that to Default CA.
-
-The default CA instantiates the approver based on the type of the approver
-indicated by the network layer. This approver creates an instance of the PKI
-profile and passes each field from the certificate signing request. The PKI
-profile (as of today Dec 1st, 2018, we have one profile called Ozone profile)
- verifies that each field in the CSR meets the approved set of values.
-
-Once the PKI Profile validates the request, it is either auto approved or
-queued for manual review.
-
- */

hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/security/x509/CertificateTestUtils.java

@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.hdds.security.x509;
+
+import org.apache.hadoop.hdds.conf.ConfigurationSource;
+import org.apache.hadoop.hdds.security.SecurityConfig;
+import org.apache.hadoop.hdds.security.x509.keys.HDDSKeyGenerator;
+import org.bouncycastle.asn1.oiw.OIWObjectIdentifiers;
+import org.bouncycastle.asn1.x500.X500Name;
+import org.bouncycastle.asn1.x509.AlgorithmIdentifier;
+import org.bouncycastle.asn1.x509.AuthorityKeyIdentifier;
+import org.bouncycastle.asn1.x509.BasicConstraints;
+import org.bouncycastle.asn1.x509.Extension;
+import org.bouncycastle.asn1.x509.SubjectKeyIdentifier;
+import org.bouncycastle.asn1.x509.SubjectPublicKeyInfo;
+import org.bouncycastle.cert.X509ExtensionUtils;
+import org.bouncycastle.cert.X509v3CertificateBuilder;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509v3CertificateBuilder;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.operator.ContentSigner;
+import org.bouncycastle.operator.DigestCalculator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.bc.BcDigestCalculatorProvider;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+
+import java.math.BigInteger;
+import java.security.KeyPair;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.cert.X509Certificate;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Date;
+
+/**
+ * Test utilities to create simple certificates/keys for testing.
+ */
+public final class CertificateTestUtils {
+  private CertificateTestUtils() { }
+
+  private static final String HASH_ALGO = "SHA256WithRSA";
+
+  /**
+   * Generates a keypair using the HDDSKeyGenerator with the given config.
+   *
+   * @param conf the config applies to keys
+   *
+   * @return a newly generated keypair
+   *
+   * @throws NoSuchProviderException on wrong security provider in the config
+   * @throws NoSuchAlgorithmException on wrong encryption algo in the config
+   */
+  public static KeyPair aKeyPair(ConfigurationSource conf)
+      throws NoSuchProviderException, NoSuchAlgorithmException {
+    return new HDDSKeyGenerator(new SecurityConfig(conf)).generateKey();
+  }
+
+  /**
+   * Creates a self-signed certificate and returns it as an X509Certificate.
+   * The given keys and common name are being used in the certificate.
+   * The certificate will have its serial id generated based on the hashcode
+   * of the public key, and will expire after 1 day.
+   *
+   * @param keys the keypair to use for the certificate
+   * @param commonName the common name used in the certificate
+   *
+   * @return the X509Certificate representing a self-signed certificate
+   *
+   * @throws Exception in case any error occurs during the certificate creation
+   */
+  public static X509Certificate createSelfSignedCert(KeyPair keys,
+      String commonName) throws Exception {
+    return createSelfSignedCert(keys, commonName, Duration.ofDays(1));
+  }
+
+  /**
+   * Creates a self-signed certificate and returns it as an X509Certificate.
+   * The given keys and common name are being used in the certificate.
+   * The certificate will have its serial id generated based on the hashcode
+   * of the public key, and will expire after the specified duration.
+   *
+   * @param keys the keypair to use for the certificate
+   * @param commonName the common name used in the certificate
+   * @param expiresIn the lifespan of the certificate
+   *
+   * @return the X509Certificate representing a self-signed certificate
+   *
+   * @throws Exception in case any error occurs during the certificate creation
+   */
+  public static X509Certificate createSelfSignedCert(KeyPair keys,
+      String commonName, Duration expiresIn) throws Exception {
+    final Instant now = Instant.now();
+    final Date notBefore = Date.from(now);
+    final Date notAfter = Date.from(now.plus(expiresIn));
+    final ContentSigner contentSigner =
+        new JcaContentSignerBuilder(HASH_ALGO).build(keys.getPrivate());
+    final X500Name x500Name = new X500Name("CN=" + commonName);
+
+    SubjectKeyIdentifier keyId = subjectKeyIdOf(keys);
+    AuthorityKeyIdentifier authorityKeyId = authorityKeyIdOf(keys);
+    BasicConstraints constraints = new BasicConstraints(true);
+
+    final X509v3CertificateBuilder certificateBuilder =
+        new JcaX509v3CertificateBuilder(
+            x500Name,
+            BigInteger.valueOf(keys.getPublic().hashCode()),
+            notBefore,
+            notAfter,
+            x500Name,
+            keys.getPublic()
+        );
+    certificateBuilder
+        .addExtension(Extension.subjectKeyIdentifier, false, keyId)
+        .addExtension(Extension.authorityKeyIdentifier, false, authorityKeyId)
+        .addExtension(Extension.basicConstraints, true, constraints);
+
+    return new JcaX509CertificateConverter()
+        .setProvider(new BouncyCastleProvider())
+        .getCertificate(certificateBuilder.build(contentSigner));
+  }
+
+  private static SubjectKeyIdentifier subjectKeyIdOf(KeyPair keys)
+      throws Exception {
+    return extensionUtil().createSubjectKeyIdentifier(pubKeyInfo(keys));
+  }
+
+  private static AuthorityKeyIdentifier authorityKeyIdOf(KeyPair keys)
+      throws Exception {
+    return extensionUtil().createAuthorityKeyIdentifier(pubKeyInfo(keys));
+  }
+
+  private static SubjectPublicKeyInfo pubKeyInfo(KeyPair keys) {
+    return SubjectPublicKeyInfo.getInstance(keys.getPublic().getEncoded());
+  }
+
+  private static X509ExtensionUtils extensionUtil()
+      throws OperatorCreationException {
+    DigestCalculator digest =
+        new BcDigestCalculatorProvider()
+            .get(new AlgorithmIdentifier(OIWObjectIdentifiers.idSHA1));
+
+    return new X509ExtensionUtils(digest);
+  }
+}

hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client/DefaultCertificateClient.java

@@ -58,6 +58,7 @@
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
 import java.util.function.Consumer;
+import java.util.function.Function;
 import java.util.stream.Stream;
 import java.util.stream.Collectors;
 
@@ -203,6 +204,14 @@ private void startRootCaRotationPoller() {
     }
   }
 
+  @Override
+  public void registerRootCARotationListener(
+      Function<List<X509Certificate>, CompletableFuture<Void>> listener) {
+    if (securityConfig.isAutoCARotationEnabled()) {
+      rootCaRotationPoller.addRootCARotationProcessor(listener);
+    }
+  }
+
   private synchronized void readCertificateFile(Path filePath) {
     CertificateCodec codec = new CertificateCodec(securityConfig, component);
     String fileName = filePath.getFileName().toString();

hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/security/x509/certificate/client/CertificateClientTestImpl.java

@@ -1,4 +1,4 @@
-/**
+/*
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with this
  * work for additional information regarding copyright ownership.  The ASF
@@ -37,10 +37,12 @@
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
+import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
 
 import com.google.common.util.concurrent.ThreadFactoryBuilder;
 import org.apache.hadoop.hdds.conf.OzoneConfiguration;
@@ -386,6 +388,13 @@ public void registerNotificationReceiver(CertificateNotification receiver) {
     }
   }
 
+  @Override
+  public void registerRootCARotationListener(
+      Function<List<X509Certificate>, CompletableFuture<Void>> listener) {
+    // we do not have tests that rely on rootCA rotation atm, leaving this
+    // implementation blank for now.
+  }
+
   @Override
   public void close() throws IOException {
     if (serverKeyStoresFactory != null) {