Deploy secrets in DeployTask like other resources

CHANGELOG.md

@@ -5,6 +5,13 @@
 - Defaults KUBECONFIG to `~/.kube/config` ([#429](https://github.com/Shopify/kubernetes-deploy/pull/429))
 - Uses `TASK_ID` environment variable as the `deployment_id` when rendering resource templates for better [Shipit](https://github.com/Shopify/shipit) integration. ([#430](https://github.com/Shopify/kubernetes-deploy/pull/430))
 
+*Features*
+
+- **[Breaking change]** Support for deploying Secrets from templates ([#424](https://github.com/Shopify/kubernetes-deploy/pull/424)).
+  * If you previously used this gem to deploy secrets from EJSON and the first time commit you deploy using this version removes one or more of them, they will not be pruned.
+  * If you previously manually `kubectl apply`'d secrets that are not passed to kubernetes-deploy, your first deploy using this version is going to delete them.
+  * If you previously passed secrets manifests to kubernetes-deploy and they are no longer in the set you pass to the first deploy using this version, it will delete them.
+
 ## 0.25.0
 
 *Features*

README.md

@@ -579,14 +579,17 @@ The list of fully supported types is effectively the list of classes found in `l
 
 This gem uses subclasses of `KubernetesResource` to implement custom success/failure detection logic for each resource type. If no subclass exists for a type you're deploying, the gem simply assumes `kubectl apply` succeeded (and prints a warning about this assumption). We're always looking to support more types! Here are the basic steps for contributing a new one:
 
-1. Create a the file for your type in `lib/kubernetes-deploy/kubernetes_resource/`
+1. Create a file for your type in `lib/kubernetes-deploy/kubernetes_resource/`
 2. Create a new class that inherits from `KubernetesResource`. Minimally, it should implement the following methods:
     * `sync` -- Gather the data you'll need to determine `deploy_succeeded?` and `deploy_failed?`. The superclass's implementation fetches the corresponding resource, parses it and stores it in `@instance_data`. You can define your own implementation if you need something else.
     * `deploy_succeeded?`
     * `deploy_failed?`
 3. Adjust the `TIMEOUT` constant to an appropriate value for this type.
-4. Add the a basic example of the type to the hello-cloud [fixture set](https://github.com/Shopify/kubernetes-deploy/tree/master/test/fixtures/hello-cloud) and appropriate assertions to `#assert_all_up` in [`hello_cloud.rb`](https://github.com/Shopify/kubernetes-deploy/blob/master/test/helpers/fixture_sets/hello_cloud.rb). This will get you coverage in several existing tests, such as `test_full_hello_cloud_set_deploy_succeeds`.
-5. Add tests for any edge cases you foresee.
+4. Add the new class to list of resources in
+   [`deploy_task.rb`](https://github.com/Shopify/kubernetes-deploy/blob/master/lib/kubernetes-deploy/deploy_task.rb#L8)
+5. Add the new resource to the [prune whitelist](https://github.com/Shopify/kubernetes-deploy/blob/master/lib/kubernetes-deploy/deploy_task.rb#L81)
+6. Add the a basic example of the type to the hello-cloud [fixture set](https://github.com/Shopify/kubernetes-deploy/tree/master/test/fixtures/hello-cloud) and appropriate assertions to `#assert_all_up` in [`hello_cloud.rb`](https://github.com/Shopify/kubernetes-deploy/blob/master/test/helpers/fixture_sets/hello_cloud.rb). This will get you coverage in several existing tests, such as `test_full_hello_cloud_set_deploy_succeeds`.
+7. Add tests for any edge cases you foresee.
 
 ## Code of Conduct
 Everyone is expected to follow our [Code of Conduct](https://github.com/Shopify/kubernetes-deploy/blob/master/CODE_OF_CONDUCT.md).

lib/kubernetes-deploy/deploy_task.rb

@@ -31,6 +31,7 @@
   job
   custom_resource_definition
   horizontal_pod_autoscaler
+  secret
 ).each do |subresource|
   require "kubernetes-deploy/kubernetes_resource/#{subresource}"
 end
@@ -72,6 +73,7 @@ def predeploy_sequence
         ServiceAccount
         Role
         RoleBinding
+        Secret
         Pod
       )
 
@@ -84,6 +86,7 @@ def prune_whitelist
         core/v1/Pod
         core/v1/Service
         core/v1/ResourceQuota
+        core/v1/Secret
         batch/v1/Job
         extensions/v1beta1/DaemonSet
         extensions/v1beta1/Deployment
@@ -137,7 +140,6 @@ def run!(verify_result: true, allow_protected_ns: false, prune: true)
 
       @logger.phase_heading("Checking initial resource statuses")
       check_initial_status(resources)
-      create_ejson_secrets(prune)
 
       if deploy_has_priority_resources?(resources)
         @logger.phase_heading("Predeploying priority resources")
@@ -243,20 +245,16 @@ def check_initial_status(resources)
     end
     measure_method(:check_initial_status, "initial_status.duration")
 
-    def create_ejson_secrets(prune)
+    def secrets_from_ejson
       ejson = EjsonSecretProvisioner.new(
         namespace: @namespace,
         context: @context,
         template_dir: @template_dir,
         logger: @logger,
-        prune: prune,
+        statsd_tags: @namespace_tags
       )
-      return unless ejson.secret_changes_required?
-
-      @logger.phase_heading("Deploying kubernetes secrets from #{EjsonSecretProvisioner::EJSON_SECRETS_FILE}")
-      ejson.run
+      ejson.resources
     end
-    measure_method(:create_ejson_secrets)
 
     def discover_resources
       resources = []
@@ -272,6 +270,10 @@ def discover_resources
           @logger.info("  - #{r.id}")
         end
       end
+      secrets_from_ejson.each do |secret|
+        resources << secret
+        @logger.info("  - #{secret.id} (from ejson)")
+      end
       if (global = resources.select(&:global?).presence)
         @logger.warn("Detected non-namespaced #{'resource'.pluralize(global.count)} which will never be pruned:")
         global.each { |r| @logger.warn("  - #{r.id}") }
@@ -300,10 +302,10 @@ def split_templates(filename)
       raise FatalDeploymentError, "Failed to render and parse template"
     end
 
-    def record_invalid_template(err:, filename:, content:)
+    def record_invalid_template(err:, filename:, content: nil)
       debug_msg = ColorizedString.new("Invalid template: #{filename}\n").red
       debug_msg += "> Error message:\n#{FormattedLogger.indent_four(err)}"
-      debug_msg += "\n> Template content:\n#{FormattedLogger.indent_four(content)}"
+      debug_msg += "\n> Template content:\n#{FormattedLogger.indent_four(content)}" if content
       @logger.summary.add_paragraph(debug_msg)
     end
 
@@ -429,12 +431,13 @@ def apply_all(resources, prune)
           prune_whitelist.each { |type| command.push("--prune-whitelist=#{type}") }
         end
 
-        out, err, st = kubectl.run(*command, log_failure: false)
+        output_is_sensitive = resources.any?(&:kubectl_output_is_sensitive?)
+        out, err, st = kubectl.run(*command, log_failure: false, output_is_sensitive: output_is_sensitive)
 
         if st.success?
           log_pruning(out) if prune
         else
-          record_apply_failure(err)
+          record_apply_failure(err, resources: resources)
           raise FatalDeploymentError, "Command failed: #{Shellwords.join(command)}"
         end
       end
@@ -449,22 +452,37 @@ def log_pruning(kubectl_output)
       @logger.summary.add_action("pruned #{pruned.length} #{'resource'.pluralize(pruned.length)}")
     end
 
-    def record_apply_failure(err)
+    def record_apply_failure(err, resources: [])
       warn_msg = "WARNING: Any resources not mentioned in the error(s) below were likely created/updated. " \
         "You may wish to roll back this deploy."
       @logger.summary.add_paragraph(ColorizedString.new(warn_msg).yellow)
 
       unidentified_errors = []
+      filenames_with_sensitive_content = resources
+        .select(&:kubectl_output_is_sensitive?)
+        .map { |r| File.basename(r.file_path) }
+
       err.each_line do |line|
         bad_files = find_bad_files_from_kubectl_output(line)
         if bad_files.present?
-          bad_files.each { |f| record_invalid_template(err: f[:err], filename: f[:filename], content: f[:content]) }
+          bad_files.each do |f|
+            if filenames_with_sensitive_content.include?(f[:filename])
+              # Hide the error and template contents in case it has senitive information
+              record_invalid_template(err: "SUPPRESSED FOR SECURITY", filename: f[:filename], content: nil)
+            else
+              record_invalid_template(err: f[:err], filename: f[:filename], content: f[:content])
+            end
+          end
         else
           unidentified_errors << line
         end
       end
 
-      if unidentified_errors.present?
+      if unidentified_errors.present? && filenames_with_sensitive_content.any?
+        warn_msg = "WARNING: There was an error applying some or all resources. The raw output may be sensitive and " \
+          "so cannot be displayed."
+        @logger.summary.add_paragraph(ColorizedString.new(warn_msg).yellow)
+      elsif unidentified_errors.present?
         heading = ColorizedString.new('Unidentified error(s):').red
         msg = FormattedLogger.indent_four(unidentified_errors.join)
         @logger.summary.add_paragraph("#{heading}\n#{msg}")

lib/kubernetes-deploy/ejson_secret_provisioner.rb

@@ -7,85 +7,52 @@
 module KubernetesDeploy
   class EjsonSecretError < FatalDeploymentError
     def initialize(msg)
-      super("Creation of Kubernetes secrets from ejson failed: #{msg}")
+      super("Generation of Kubernetes secrets from ejson failed: #{msg}")
     end
   end
 
   class EjsonSecretProvisioner
-    MANAGEMENT_ANNOTATION = "kubernetes-deploy.shopify.io/ejson-secret"
-    MANAGED_SECRET_EJSON_KEY = "kubernetes_secrets"
+    EJSON_SECRET_ANNOTATION = "kubernetes-deploy.shopify.io/ejson-secret"
+    EJSON_SECRET_KEY = "kubernetes_secrets"
     EJSON_SECRETS_FILE = "secrets.ejson"
     EJSON_KEYS_SECRET = "ejson-keys"
 
-    def initialize(namespace:, context:, template_dir:, logger:, prune: true)
+    def initialize(namespace:, context:, template_dir:, logger:, statsd_tags:)
       @namespace = namespace
       @context = context
       @ejson_file = "#{template_dir}/#{EJSON_SECRETS_FILE}"
       @logger = logger
-      @prune = prune
+      @statsd_tags = statsd_tags
       @kubectl = Kubectl.new(
         namespace: @namespace,
         context: @context,
         logger: @logger,
         log_failure_by_default: false,
-        output_is_sensitive: true # output may contain ejson secrets
+        output_is_sensitive_default: true # output may contain ejson secrets
       )
     end
 
-    def secret_changes_required?
-      File.exist?(@ejson_file) || managed_secrets_exist?
-    end
-
-    def run
-      create_secrets
-      prune_managed_secrets if @prune
+    def resources
+      @resources ||= build_secrets
     end
 
     private
 
-    def create_secrets
+    def build_secrets
+      return [] unless File.exist?(@ejson_file)
       with_decrypted_ejson do |decrypted|
-        secrets = decrypted[MANAGED_SECRET_EJSON_KEY]
+        secrets = decrypted[EJSON_SECRET_KEY]
         unless secrets.present?
-          @logger.warn("#{EJSON_SECRETS_FILE} does not have key #{MANAGED_SECRET_EJSON_KEY}."\
+          @logger.warn("#{EJSON_SECRETS_FILE} does not have key #{EJSON_SECRET_KEY}."\
             "No secrets will be created.")
-          return
+          return []
         end
 
-        secrets.each do |secret_name, secret_spec|
+        secrets.map do |secret_name, secret_spec|
           validate_secret_spec(secret_name, secret_spec)
-          create_or_update_secret(secret_name, secret_spec["_type"], secret_spec["data"])
+          generate_secret_resource(secret_name, secret_spec["_type"], secret_spec["data"])
         end
-        @logger.summary.add_action("created/updated #{secrets.length} #{'secret'.pluralize(secrets.length)}")
-      end
-    end
-
-    def prune_managed_secrets
-      ejson_secret_names = encrypted_ejson.fetch(MANAGED_SECRET_EJSON_KEY, {}).keys
-      live_secrets = run_kubectl_json("get", "secrets")
-
-      prune_count = 0
-      live_secrets.each do |secret|
-        secret_name = secret["metadata"]["name"]
-        next unless secret_managed?(secret)
-        next if ejson_secret_names.include?(secret_name)
-
-        @logger.info("Pruning secret #{secret_name}")
-        prune_count += 1
-        out, err, st = @kubectl.run("delete", "secret", secret_name)
-        @logger.debug(out)
-        raise EjsonSecretError, "Failed to prune secrets" unless st.success?
       end
-      @logger.summary.add_action("pruned #{prune_count} #{'secret'.pluralize(prune_count)}") if prune_count > 0
-    end
-
-    def managed_secrets_exist?
-      all_secrets = run_kubectl_json("get", "secrets")
-      all_secrets.any? { |secret| secret_managed?(secret) }
-    end
-
-    def secret_managed?(secret)
-      secret["metadata"].fetch("annotations", {}).key?(MANAGEMENT_ANNOTATION)
     end
 
     def encrypted_ejson
@@ -110,23 +77,7 @@ def validate_secret_spec(secret_name, spec)
       end
     end
 
-    def create_or_update_secret(secret_name, secret_type, data)
-      msg = secret_exists?(secret_name) ? "Updating secret #{secret_name}" : "Creating secret #{secret_name}"
-      @logger.info(msg)
-
-      secret_yaml = generate_secret_yaml(secret_name, secret_type, data)
-      file = Tempfile.new(secret_name)
-      file.write(secret_yaml)
-      file.close
-
-      out, err, st = @kubectl.run("apply", "--filename=#{file.path}")
-      @logger.debug(out)
-      raise EjsonSecretError, "Failed to create or update secrets" unless st.success?
-    ensure
-      file&.unlink
-    end
-
-    def generate_secret_yaml(secret_name, secret_type, data)
+    def generate_secret_resource(secret_name, secret_type, data)
       unless data.is_a?(Hash) && data.values.all? { |v| v.is_a?(String) } # Secret data is map[string]string
         raise EjsonSecretError, "Data for secret #{secret_name} was invalid. Only key-value pairs are permitted."
       end
@@ -145,16 +96,14 @@ def generate_secret_yaml(secret_name, secret_type, data)
           "name" => secret_name,
           "labels" => { "name" => secret_name },
           "namespace" => @namespace,
-          "annotations" => { MANAGEMENT_ANNOTATION => "true" },
+          "annotations" => { EJSON_SECRET_ANNOTATION => "true" },
         },
         "data" => encoded_data,
       }
-      secret.to_yaml
-    end
 
-    def secret_exists?(secret_name)
-      _out, _err, st = @kubectl.run("get", "secret", secret_name)
-      st.success?
+      KubernetesDeploy::Secret.build(
+        namespace: @namespace, context: @context, logger: @logger, definition: secret, statsd_tags: @statsd_tags,
+      )
     end
 
     def load_ejson_from_file
@@ -175,7 +124,6 @@ def with_decrypted_ejson
     end
 
     def decrypt_ejson(key_dir)
-      @logger.info("Decrypting #{EJSON_SECRETS_FILE}")
       # ejson seems to dump both errors and output to STDOUT
       out_err, st = Open3.capture2e("EJSON_KEYDIR=#{key_dir} ejson decrypt #{@ejson_file}")
       raise EjsonSecretError, out_err unless st.success?
@@ -185,8 +133,6 @@ def decrypt_ejson(key_dir)
     end
 
     def fetch_private_key_from_secret
-      @logger.info("Fetching ejson private key from secret #{EJSON_KEYS_SECRET}")
-
       secret = run_kubectl_json("get", "secret", EJSON_KEYS_SECRET)
       encoded_private_key = secret["data"][public_key]
       unless encoded_private_key

lib/kubernetes-deploy/kubectl.rb

@@ -8,20 +8,22 @@ class Kubectl
     class ResourceNotFoundError < StandardError; end
 
     def initialize(namespace:, context:, logger:, log_failure_by_default:, default_timeout: DEFAULT_TIMEOUT,
-      output_is_sensitive: false)
+      output_is_sensitive_default: false)
       @namespace = namespace
       @context = context
       @logger = logger
       @log_failure_by_default = log_failure_by_default
       @default_timeout = default_timeout
-      @output_is_sensitive = output_is_sensitive
+      @output_is_sensitive_default = output_is_sensitive_default
 
       raise ArgumentError, "namespace is required" if namespace.blank?
       raise ArgumentError, "context is required" if context.blank?
     end
 
-    def run(*args, log_failure: nil, use_context: true, use_namespace: true, raise_if_not_found: false, attempts: 1)
+    def run(*args, log_failure: nil, use_context: true, use_namespace: true, raise_if_not_found: false, attempts: 1,
+      output_is_sensitive: nil)
       log_failure = @log_failure_by_default if log_failure.nil?
+      output_is_sensitive = @output_is_sensitive_default if output_is_sensitive.nil?
 
       args = args.unshift("kubectl")
       args.push("--namespace=#{@namespace}") if use_namespace
@@ -32,19 +34,19 @@ def run(*args, log_failure: nil, use_context: true, use_namespace: true, raise_i
       (1..attempts).to_a.each do |attempt|
         @logger.debug("Running command (attempt #{attempt}): #{args.join(' ')}")
         out, err, st = Open3.capture3(*args)
-        @logger.debug("Kubectl out: " + out.gsub(/\s+/, ' ')) unless output_is_sensitive?
+        @logger.debug("Kubectl out: " + out.gsub(/\s+/, ' ')) unless output_is_sensitive
 
         break if st.success?
 
         if log_failure
           @logger.warn("The following command failed (attempt #{attempt}/#{attempts}): #{Shellwords.join(args)}")
-          @logger.warn(err) unless output_is_sensitive?
+          @logger.warn(err) unless output_is_sensitive
         end
 
         if err.match(NOT_FOUND_ERROR_TEXT)
           raise(ResourceNotFoundError, err) if raise_if_not_found
         else
-          @logger.debug("Kubectl err: #{err}") unless output_is_sensitive?
+          @logger.debug("Kubectl err: #{err}") unless output_is_sensitive
           StatsD.increment('kubectl.error', 1, tags: { context: @context, namespace: @namespace, cmd: args[1] })
         end
         sleep(retry_delay(attempt)) unless attempt == attempts
@@ -76,10 +78,6 @@ def server_version
 
     private
 
-    def output_is_sensitive?
-      @output_is_sensitive
-    end
-
     def extract_version_info_from_kubectl_response(response)
       info = {}
       response.each_line do |l|