Provide Persistent Volumes for brokered services in k8s

[adborden]

User Story

In order to support durable storage for k8s workloads, the EKS brokerpak should configure persistent storage to satisfy Persistent Volume Claims.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN [a contextual precondition]
  [AND optionally another precondition]
  WHEN [a triggering event] happens
  THEN [a verifiable outcome]
  [AND optionally another verifiable outcome]

Background

We're currently using ephemeral storage for SolrCloud, which is limited to 20GB and isn't going to get us to production for Catalog.

$ sudo du -sh /data/solr5/data/{catalog,inventory}-next
23G     /data/solr5/data/catalog-next
86M     /data/solr5/data/inventory-next

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]
Summary of this article:

• Create EFS filesystem
• Create EFS access point
• Create ingress security group
• Create EFS mount targets in each subnet used by the Fargate profile
• Create PVC (use kubectl to start)

[mogul]

I do want to get PVCs supported in the brokered EKS, but...

We're currently using ephemeral storage for SolrCloud, which is limited to 20GB and isn't going to get us to production for Catalog.

I'm not sure that's the case. With ephemeral storage there's a 20GB limit per SolrCloud node, but by scaling horizontally we can increase the capacity... Not every shard has to be replicated to every node!

[mogul]

Still keeping an eye on this... It looks like it's now possible to dynamically configure EFS volumes in response to PVCs without any manual intervention up front. There may still be some restrictions; the GitHub issue about this capability is not especially clear: The post says they're still working on EKS, but then unceremoniously closes the issue, and the blog post doesn't mention the EKS restriction. 🤷

[mogul]

Oh, here's where the confusion is: You can't (yet) run the EFS CSI controller on Fargate. The referenced capability is being discussed here.

[mogul]

Related issue: Make the size of the Fargate local volume configurable above 20GB

[mogul]

Looks like it's now possible to configure up to 200GB ephemeral storage.

[mogul]

Persistent Volumes via EBS are now supported via an EKS add-on.

[mogul]

Looks like it's now possible to configure up to 200GB ephemeral storage.

Unfortunately I misread this... Fargate can be configured to attach up to 200GB of storage in ECS, not in EKS!

[mogul]

Next steps...

Enable PVCs in the eks-brokerpak

• Add the EBS CSI EKS add-on
  • This enables us to create PersistentVolumes in k8s, and then bind them to pods via PersistentVolumeClaims
• Test that it works (eg create a PersistentVolume)
• Push a tag for a new minor release of the EKS brokerpak that incorporates this change
• Update the version of the eks-brokerpak installed by the SSB deployment

Enable use of PVCs in the solr-brokerpak

• Change the solr-broker to configure the persistent dataStorage option when deploying the solr helm chart
• Expose the dataStorage.capacity parameter for configuration by the broker's users and use that parameter when provisioning the Helm chart.
• Push a tag for a new minor release of the solr brokerpak that incorporates this change
• Update the version of the solr-brokerpak installed by the SSB deployment

[nickumia-reisys]

A PR has been started to incorporate this capability. There were concerns with Fargate compatibility and IAM role permissions. A glimpse of it working has been witnessed with managed node groups provisioned along with choosing the following option,

This work has been postponed due to a workaround with shard implementations for Solr.

The next steps for this issue,

• Figure out issue regarding https://stackoverflow.com/questions/70171641/aws-eks-nodes-creation-failure
• See an ACTIVE status on cluster addon page
  
• Consider testing an example persistent volume https://github.com/kubernetes-sigs/aws-ebs-csi-driver#examples
• Merge and Tag eks-brokerpak PR
• Enable persistent volumes in solr-brokerpak

[nickumia-reisys]

It turns out that we need this because of apache/solr-operator#365

[nickumia-reisys]

Current Status:

• EBS has easy integration with EKS through the EKS EBS addon

• EBS only works with Manged nodes, not Fargate.

  • Attempt w/o managed nodes: https://github.com/GSA/datagov-brokerpak-eks/actions/runs/1746251184
  • Attempt w/ managed nodes: https://github.com/GSA/datagov-brokerpak-eks/actions/runs/1670062549

• EBS connects individually to each node and are not shared between nodes.

  • We can't used managed nodes for everything because that would defeat the purpose of being in Fargate.

• EFS is a shared filesystem across all nodes in a cluster.

• EFS does not have an addon, it must be installed via a Helm Chart

• EFS does not support dynamic provisioning, only static provisioning in Fargate.
  

• The example creates an EFS in the main cluster which means that the EFS would need to be provisioned in the EKS brokerpak.

• There are two options:

  1. Follow example, create static EFS in EKS brokerpak.
  • Design considerations:
    • Each eks cluster would have shared storage across all nodes, so if two Solrs are brokered in the staging space, their collection memory would overlap if they were named the same thing.
    • The total size of desired memory would need to be known prior to creating the EKS cluster.
  2. Fetch EKS config from the Solr brokerpak and do the provisioning alongside the creation of solr.
  • Design considerations:
    • I don't know if we can manipulate the EKS cluster from the Solr brokerpak.
    • I prefer this method.

Resources needed alongside the EFS volume:

• Data sources:
  • VPC id (EKS-specific)
  • Cluster CIDR range (EKS-specific)
• Resources:
  • Security groups and additional rules (EKS-specific)
  • EFS driver (EKS-specific)
  • EFS volume
  • Mount volume into nodes

[nickumia-reisys]

List of Related Resources:

• unable to get the pvs attached on ebs volumes on faragte eks cluster kubernetes-sigs/aws-ebs-csi-driver#810 (comment)
• Start efs stunnel watch dog kubernetes-sigs/aws-efs-csi-driver#104
• https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html
• https://aws.amazon.com/blogs/aws/new-aws-fargate-for-amazon-eks-now-supports-amazon-efs/

  With this new update, for Fargate this step is not required and you do not need to install the EFS CSI driver, as it is installed in the Fargate stack and support for EFS is provided out of the box. Customers can use EFS with Fargate for EKS without spending the time and resources to install and update the CSI driver.

• EFS Driver

[nickumia-reisys]

Tentative Final Design

• EFS Statically Provisioned alongside EKS Cluster
  • EBS CSI EKS support GSA-TTS/datagov-brokerpak-eks#64
• Solr-operator-enabled PV for solrclouds
  • Solr Optimizations for CKAN Integrations GSA-TTS/datagov-brokerpak-solr#32
• Unique Solr Collection names
  • TODO

[mogul]

Another key reference: The k8s documentation on persistent volumes.