Insights Storage Broker

The Insights Storage Broker microservice handles interaction between the platform and remote stores for storage of payloads that pass through the platform. It also supports the legacy platform.upload.available topic by reproducing messages from platform.inventory.host-egress to announce new payloads.

How it Works

Storage Workflow

Insights Storage Broker can be configured to consume from multiple topics by supplying a configuration file in YAML format with the specified topic, bucket, and formatter for the resulting object in cloud storage.

Example Configuration:

platform.upload.validation:                                     # topic
  normalizer: Validation                                        # normalizer (to be covered further down)
platform.upload.buckit:
  normalizer: Openshift
  services:                                                     # list of services
    openshift:                                                  # service (defined by content type)
      format: "{org_id}/{cluster_id}/{timestamp}-{request_id}"  # format of resulting file object
      bucket: "insights-buck-it-openshift"                      # storage bucket
    ansible:
      format: "{org_id}/{cluster_id}/{timestamp}-{request_id}"
      bucket: "insights-buck-it-ansible"

The configuration file allows for new buckets, topics, and formatters to be added to the service without changing the underlying code.

Support for Validation Messages

Insights Storage Broker consumes from the platform.upload.validation topic. If the payload has succeeded validation, the storage broker will pass the message along with a download url to the platform.upload.available topic so that other services on the platform can consume it. Storage broker expects to recieve all the data in the message that the validation service originally recieved in addition to the the validation key.

If a failure message is recieved, Storage Broker will copy the file to the rejected bucket and not advertise the availability of the payload to the platform.

Validation Workflow:

Normalizers

In order to make Storage Broker more dynamic, it supports the ability for a user to define their own data normalizer to provide the required keys to the service. Normalizers can be found in normalizers.py and more can be added if need be, or a service can choose to use an existing normalizer.

The only requirmeent for Storage Broker to determine which normalizer to use is that the service key is available in the root of the JSON being parsed. This is provided by ingress messages by default. Any other keys to be used must be added as attributes of your normalizer class.

The only required keys in a normalizer at this point are size, service, and request_id. Any other keys you establish can be used for formatting the resulting object filename.

Available Message

Available Message:

  {
      "account": <account number>,
      "request_id": <uuid for the payload>,
      "principal": <currently the org ID>,
      "service": <the service that validated the payload>,
      "category": <a category for the payload>,
      "url": <URL to download the file>,
      "b64_identity": <the base64 encoded identity of the sender>,
      "id": <host based inventory id if available>, **RELOCATING TO EXTRAS**
      "satellite_managed": <boolean if the system is managed by satellite>, **RELOCATING TO EXTRAS**
      "timestamp": <the time the available message was put on the topic>,
      "validation": <success|failure>,
      "size": <size in bytes of payload>,
      "metadata": <metadata attached to the original upload>,
      "extras": {
          "satellite_managed": <same as above>
          "id": <same as above>
          ...
      }
  }

Authors

Stephen Adams - Initial Work - SteveHNH