Skip to content

Conversation

@mtulio
Copy link
Contributor

@mtulio mtulio commented Sep 17, 2022

The goal of this PR is to add support to AWS Local Zones when installing a cluster in an existing VPC. It allows the creation of a compute pool named edge, alongside the worker, on install-config.yaml. The installer will generate MachineSet manifests with NoSchedule taint to deploy the instance on edge locations.

This implementation is Phase 1 of the Enhancement Proposal #1232, which describes in detail the change request.

Nodes are considered edge when the availability zone zoneType is local-zone. The edge compute pool is valid only for the AWS Platform.

It is not the goal of this PR to create the Subnets on Local Zones locations. This is part of Phase 2.

Note: This PR will change the MTU of ClusterNetwork when edge pool for AWS is defined. FYI openshift/openshift-docs#55327

Refereces:

List of teams that would be interested in reviewing this PR:

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 17, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 17, 2022

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@mtulio
Copy link
Contributor Author

mtulio commented Sep 17, 2022

/hold

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 17, 2022
@mtulio mtulio force-pushed the feat-aws-localzones-edge-pool branch 4 times, most recently from bedd5d3 to a8ffce5 Compare September 21, 2022 01:05
@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 21, 2022
@mtulio mtulio force-pushed the feat-aws-localzones-edge-pool branch 11 times, most recently from 580c97a to f96f2ae Compare September 24, 2022 06:43
@mtulio mtulio changed the title RFE-2782/SPLAT-636: AWS Local Zones with edge machine pool RFE-2782/SPLAT-636: Create edge compute pool to support AWS Local Zones Sep 26, 2022
@mtulio mtulio force-pushed the feat-aws-localzones-edge-pool branch 2 times, most recently from a62ae56 to 39c7835 Compare September 27, 2022 21:15
@mtulio
Copy link
Contributor Author

mtulio commented Sep 28, 2022

Tests with the current PR on an IPI installation in existing VPC in us-west-2 with one subnet for each Local Zones available in that region.

Nodes:

$ oc get nodes
NAME                                         STATUS   ROLES                  AGE   VERSION
ip-10-0-13-183.us-west-2.compute.internal    Ready    worker                 28m   v1.24.0+8c7c967
ip-10-0-13-35.us-west-2.compute.internal     Ready    control-plane,master   36m   v1.24.0+8c7c967
ip-10-0-16-27.us-west-2.compute.internal     Ready    control-plane,master   36m   v1.24.0+8c7c967
ip-10-0-18-192.us-west-2.compute.internal    Ready    worker                 28m   v1.24.0+8c7c967
ip-10-0-194-163.us-west-2.compute.internal   Ready    edge,worker            18m   v1.24.0+8c7c967
ip-10-0-196-196.us-west-2.compute.internal   Ready    edge,worker            26m   v1.24.0+8c7c967
ip-10-0-202-81.us-west-2.compute.internal    Ready    edge,worker            18m   v1.24.0+8c7c967
ip-10-0-205-248.us-west-2.compute.internal   Ready    edge,worker            18m   v1.24.0+8c7c967
ip-10-0-21-119.us-west-2.compute.internal    Ready    worker                 28m   v1.24.0+8c7c967
ip-10-0-211-241.us-west-2.compute.internal   Ready    edge,worker            25m   v1.24.0+8c7c967
ip-10-0-213-135.us-west-2.compute.internal   Ready    edge,worker            18m   v1.24.0+8c7c967
ip-10-0-218-87.us-west-2.compute.internal    Ready    edge,worker            21m   v1.24.0+8c7c967
ip-10-0-22-68.us-west-2.compute.internal     Ready    control-plane,master   35m   v1.24.0+8c7c967

$ oc get machineset -n openshift-machine-api
NAME                                          DESIRED   CURRENT   READY   AVAILABLE   AGE
ipi-zzz-test124-jwcpd-edge-us-west-2-den-1a   1         1         1       1           71m
ipi-zzz-test124-jwcpd-edge-us-west-2-las-1a   1         1         1       1           72m
ipi-zzz-test124-jwcpd-edge-us-west-2-lax-1a   1         1         1       1           71m
ipi-zzz-test124-jwcpd-edge-us-west-2-lax-1b   1         1         1       1           72m
ipi-zzz-test124-jwcpd-edge-us-west-2-pdx-1a   1         1         1       1           72m
ipi-zzz-test124-jwcpd-edge-us-west-2-phx-1a   1         1         1       1           72m
ipi-zzz-test124-jwcpd-edge-us-west-2-sea-1a   1         1         1       1           71m
ipi-zzz-test124-jwcpd-worker-us-west-2a       1         1         1       1           72m
ipi-zzz-test124-jwcpd-worker-us-west-2b       1         1         1       1           72m
ipi-zzz-test124-jwcpd-worker-us-west-2c       1         1         1       1           72m

Machines:

Note: for Los Angeles (us-west-2-lax-1{a,b}) location the preferred instance was m5.xlarge as it has more precedence than others, but is not offered across other locations. See more on the EP discussion and AWS Local Zones Features Documentation.

$ oc get machines -n openshift-machine-api
NAME                                                PHASE     TYPE          REGION      ZONE               AGE
ipi-zzz-test124-jwcpd-edge-us-west-2-den-1a-vrldx   Running   c5d.2xlarge   us-west-2   us-west-2-den-1a   32m
ipi-zzz-test124-jwcpd-edge-us-west-2-las-1a-4tgnc   Running   c5d.2xlarge   us-west-2   us-west-2-las-1a   32m
ipi-zzz-test124-jwcpd-edge-us-west-2-lax-1a-jfkc7   Running   m5.xlarge     us-west-2   us-west-2-lax-1a   32m
ipi-zzz-test124-jwcpd-edge-us-west-2-lax-1b-4tdk8   Running   m5.xlarge     us-west-2   us-west-2-lax-1b   32m
ipi-zzz-test124-jwcpd-edge-us-west-2-pdx-1a-9vmgk   Running   c5d.2xlarge   us-west-2   us-west-2-pdx-1a   32m
ipi-zzz-test124-jwcpd-edge-us-west-2-phx-1a-6hnk2   Running   c5d.2xlarge   us-west-2   us-west-2-phx-1a   32m
ipi-zzz-test124-jwcpd-edge-us-west-2-sea-1a-zjtwb   Running   c5d.2xlarge   us-west-2   us-west-2-sea-1a   32m
ipi-zzz-test124-jwcpd-master-0                      Running   m6i.xlarge    us-west-2   us-west-2b         36m
ipi-zzz-test124-jwcpd-master-1                      Running   m6i.xlarge    us-west-2   us-west-2c         36m
ipi-zzz-test124-jwcpd-master-2                      Running   m6i.xlarge    us-west-2   us-west-2a         36m
ipi-zzz-test124-jwcpd-worker-us-west-2a-vkdbl       Running   m6i.xlarge    us-west-2   us-west-2a         32m
ipi-zzz-test124-jwcpd-worker-us-west-2b-jrfss       Running   m6i.xlarge    us-west-2   us-west-2b         32m
ipi-zzz-test124-jwcpd-worker-us-west-2c-4jlnt       Running   m6i.xlarge    us-west-2   us-west-2c         32m

The manifests of the following hypothetical configurations were also rendered:

TEST_ID-NAME				 RESULT	 INST_MANIFEST_RC
--------				 --	 -- 	 --
t00_01-aws-exist_vpc_workers-empty	 OK	 OK[0]		
t00_01-aws-exist_vpc_workers-pool_edge	 OK	 OK_ERR[3]	
t01-aws-exist_vpc_all-pool_empty	 OK	 OK[0]		
t02-aws-exist_vpc_all-pool_worker	 OK	 OK[0]		
t03-aws-exist_vpc_all-pool_edge_only	 OK	 OK[0]		
t04-aws-exist_vpc_all-pool_wk_edge	 OK	 OK[0]		
t05-aws-exist_vpc_all-pool_edge_inst	 OK	 OK[0]		
t06-aws-exist_vpc_all-pool_edge_ebs	 OK	 OK[0]		
t07-aws-exist_vpc_all-pool_edge_zones	 OK	 OK[0]		
t08-aws-exist_vpc_all-pool_edge_repl	 OK	 OK[0]		
t09-aws-exist_vpc_edge_net-pool_edge	 OK	 OK_ERR[3]	
t20_01-plat_none-default__config	 OK	 OK[0]		
t20_02-plat_none-pool_worker_edge	 OK	 OK_ERR[3]	
t30_01-plat_azure-default_config	 OK	 OK[0]		
t30_02-plat_azure-pool_worker_only	 OK	 OK[0]		
t30_03-plat_azure-pool_worker_edge	 OK	 OK_ERR[3]	
t30_04-plat_azure-pool_edge_only	 OK	 OK_ERR[3]	

@mtulio mtulio marked this pull request as ready for review September 28, 2022 05:33
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 28, 2022
@mtulio mtulio force-pushed the feat-aws-localzones-edge-pool branch from 39c7835 to 4955ac8 Compare September 28, 2022 05:35
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 28, 2022
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 2, 2023
@rvanderp3
Copy link
Contributor

Passed from the last installer failures with the latest changes (publicIP flag). Now e2e is flaking with one test. There are jobs flaking too with terraform. Retesting it.

Terraform error creating resources:

e2e error:

Suite run returned error: 2 fail, 1352 pass, 1628 skip (42m7s)
error: 2 fail, 1352 pass, 1628 skip (42m7s)
      "Name": "[sig-storage] Mounted volume expand [Feature:StorageProvider] Should verify mounted devices can be resized [Skipped:NoOptionalCapabilities] [Suite:openshift/conformance/parallel] [Suite:k8s]",
      "Risk": {
        "Level": {
          "Name": "Medium",
          "Level": 50
        },
      }, 

Suite run returned error: 24 fail, 1296 pass, 1662 skip (1h10m47s)
error: 24 fail, 1296 pass, 1662 skip (1h10m47s)
      "Name": "[sig-storage] Mounted volume expand [Feature:StorageProvider] Should verify mounted devices can be resized [Skipped:NoOptionalCapabilities] [Suite:openshift/conformance/parallel] [Suite:k8s]",
      "Risk": {
        "Level": {
          "Name": "Low",
          "Level": 1
        },

/retest-required /test e2e-aws-ovn-workers-rhel8 /test e2e-aws-ovn-upgrade /test e2e-aws-ovn-single-node /test e2e-aws-ovn-fips

these failures appear to be unrelated to the changes in this PR

@patrickdillon
Copy link
Contributor

/skip

@rvanderp3
Copy link
Contributor

@patrickdillon I think we are stuck on https://issues.redhat.com/browse/OCPBUGS-8092

@openshift-ci-robot
Copy link
Contributor

/retest-required

Remaining retests: 0 against base HEAD 78e72e5 and 2 for PR HEAD 2e18401 in total

@patrickdillon
Copy link
Contributor

/skip

@patrickdillon
Copy link
Contributor

@patrickdillon I think we are stuck on https://issues.redhat.com/browse/OCPBUGS-8092

Several other e2e-aws jobs, like imdsv2, shared vpc, proxy have succeeded. Plenty of confirmation aws works.

/override ci/prow/e2e-aws-ovn

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 3, 2023

@patrickdillon: Overrode contexts on behalf of patrickdillon: ci/prow/e2e-aws-ovn

Details

In response to this:

@patrickdillon I think we are stuck on https://issues.redhat.com/browse/OCPBUGS-8092

Several other e2e-aws jobs, like imdsv2, shared vpc, proxy have succeeded. Plenty of confirmation aws works.

/override ci/prow/e2e-aws-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@patrickdillon
Copy link
Contributor

/skip

@patrickdillon
Copy link
Contributor

/refresh

@patrickdillon
Copy link
Contributor

/skip

@patrickdillon
Copy link
Contributor

/refresh

@patrickdillon
Copy link
Contributor

/skip

@patrickdillon
Copy link
Contributor

/override ci/prow/e2e-aws-ovn-fip
/override ci/prow/e2e-aws-ovn-single-node
/override ci/prow/e2e-aws-ovn-workers-rhel8
/override ci/prow/okd-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-upgrade

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 3, 2023

@patrickdillon: /override requires failed status contexts, check run or a prowjob name to operate on.
The following unknown contexts/checkruns were given:

  • ci/prow/e2e-aws-ovn-fip

Only the following failed contexts/checkruns were expected:

  • ci/prow/aro-unit
  • ci/prow/e2e-agent-ha-dualstack
  • ci/prow/e2e-aws-ovn
  • ci/prow/e2e-aws-ovn-fips
  • ci/prow/e2e-aws-ovn-imdsv2
  • ci/prow/e2e-aws-ovn-proxy
  • ci/prow/e2e-aws-ovn-shared-vpc
  • ci/prow/e2e-aws-ovn-single-node
  • ci/prow/e2e-aws-ovn-upgrade
  • ci/prow/e2e-aws-ovn-upi
  • ci/prow/e2e-aws-ovn-workers-rhel8
  • ci/prow/gofmt
  • ci/prow/golint
  • ci/prow/govet
  • ci/prow/images
  • ci/prow/okd-e2e-aws-ovn
  • ci/prow/okd-e2e-aws-ovn-upgrade
  • ci/prow/okd-images
  • ci/prow/okd-scos-e2e-aws-ovn
  • ci/prow/okd-scos-e2e-aws-upgrade
  • ci/prow/okd-scos-images
  • ci/prow/okd-scos-unit
  • ci/prow/okd-scos-verify-codegen
  • ci/prow/okd-unit
  • ci/prow/okd-verify-codegen
  • ci/prow/shellcheck
  • ci/prow/tf-fmt
  • ci/prow/tf-lint
  • ci/prow/unit
  • ci/prow/verify-codegen
  • ci/prow/verify-vendor
  • ci/prow/yaml-lint
  • pull-ci-openshift-installer-agent-installer-gofmt
  • pull-ci-openshift-installer-agent-installer-golint
  • pull-ci-openshift-installer-agent-installer-govet
  • pull-ci-openshift-installer-agent-installer-images
  • pull-ci-openshift-installer-agent-installer-unit
  • pull-ci-openshift-installer-agent-installer-verify-codegen
  • pull-ci-openshift-installer-agent-installer-verify-vendor
  • pull-ci-openshift-installer-master-aro-unit
  • pull-ci-openshift-installer-master-e2e-agent-ha-dualstack
  • pull-ci-openshift-installer-master-e2e-aws-ovn
  • pull-ci-openshift-installer-master-e2e-aws-ovn-fips
  • pull-ci-openshift-installer-master-e2e-aws-ovn-imdsv2
  • pull-ci-openshift-installer-master-e2e-aws-ovn-proxy
  • pull-ci-openshift-installer-master-e2e-aws-ovn-shared-vpc
  • pull-ci-openshift-installer-master-e2e-aws-ovn-single-node
  • pull-ci-openshift-installer-master-e2e-aws-ovn-upgrade
  • pull-ci-openshift-installer-master-e2e-aws-ovn-upi
  • pull-ci-openshift-installer-master-e2e-aws-ovn-workers-rhel8
  • pull-ci-openshift-installer-master-okd-e2e-aws-ovn
  • pull-ci-openshift-installer-master-okd-e2e-aws-ovn-upgrade
  • pull-ci-openshift-installer-master-okd-images
  • pull-ci-openshift-installer-master-okd-scos-e2e-aws-ovn
  • pull-ci-openshift-installer-master-okd-scos-e2e-aws-upgrade
  • pull-ci-openshift-installer-master-okd-scos-images
  • pull-ci-openshift-installer-master-okd-scos-unit
  • pull-ci-openshift-installer-master-okd-scos-verify-codegen
  • pull-ci-openshift-installer-master-okd-unit
  • pull-ci-openshift-installer-master-okd-verify-codegen
  • pull-ci-openshift-installer-master-shellcheck
  • pull-ci-openshift-installer-master-tf-fmt
  • pull-ci-openshift-installer-master-tf-lint
  • pull-ci-openshift-installer-master-yaml-lint
  • tide

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

Details

In response to this:

/override ci/prow/e2e-aws-ovn-fip
/override ci/prow/e2e-aws-ovn-single-node
/override ci/prow/e2e-aws-ovn-workers-rhel8
/override ci/prow/okd-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@patrickdillon
Copy link
Contributor

/refresh

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 3, 2023

@mtulio: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/okd-e2e-gcp-ovn-upgrade 3655b9be2a0ab38fdd78f0d447d95ae3ebd84ba5 link false /test okd-e2e-gcp-ovn-upgrade
ci/prow/okd-e2e-aws-upgrade 2d3322a link false /test okd-e2e-aws-upgrade
ci/prow/e2e-metal-ipi 2d3322a link false /test e2e-metal-ipi
ci/prow/e2e-agent-mce 2d3322a link false /test e2e-agent-mce
ci/prow/e2e-metal-ipi-sdn 340de2ed2c01391a1e1b3e867162d5bd4d719837 link false /test e2e-metal-ipi-sdn
ci/prow/e2e-libvirt 340de2ed2c01391a1e1b3e867162d5bd4d719837 link false /test e2e-libvirt
ci/prow/e2e-agent-sno-ipv6 340de2ed2c01391a1e1b3e867162d5bd4d719837 link false /test e2e-agent-sno-ipv6
ci/prow/e2e-metal-assisted 340de2ed2c01391a1e1b3e867162d5bd4d719837 link false /test e2e-metal-assisted
ci/prow/e2e-ibmcloud-ovn 340de2ed2c01391a1e1b3e867162d5bd4d719837 link false /test e2e-ibmcloud-ovn
ci/prow/e2e-openstack 340de2ed2c01391a1e1b3e867162d5bd4d719837 link false /test e2e-openstack
ci/prow/e2e-agent-compact-ipv4 340de2ed2c01391a1e1b3e867162d5bd4d719837 link true /test e2e-agent-compact-ipv4
ci/prow/agent-integration-tests 340de2ed2c01391a1e1b3e867162d5bd4d719837 link true /test agent-integration-tests
ci/prow/e2e-aws-ovn-disruptive ee69fbf link false /test e2e-aws-ovn-disruptive
ci/prow/e2e-gcp-secureboot ee69fbf link false /test e2e-gcp-secureboot
ci/prow/e2e-vsphere-upi-zones c68d49b4490a07b5a01e89b5c2ba2fb105f929be link false /test e2e-vsphere-upi-zones
ci/prow/e2e-aws-ovn-workers-rhel8 2e18401 link false /test e2e-aws-ovn-workers-rhel8
ci/prow/e2e-aws-ovn-fips 2e18401 link false /test e2e-aws-ovn-fips
ci/prow/okd-e2e-aws-ovn 2e18401 link false /test okd-e2e-aws-ovn
ci/prow/okd-e2e-aws-ovn-upgrade 2e18401 link false /test okd-e2e-aws-ovn-upgrade
ci/prow/okd-scos-e2e-aws-upgrade 2e18401 link false /test okd-scos-e2e-aws-upgrade
ci/prow/okd-scos-e2e-aws-ovn 2e18401 link false /test okd-scos-e2e-aws-ovn
ci/prow/e2e-aws-ovn-single-node 2e18401 link false /test e2e-aws-ovn-single-node

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@patrickdillon
Copy link
Contributor

/override ci/prow/e2e-aws-ovn-fips
/override ci/prow/e2e-aws-ovn-single-node
/override ci/prow/e2e-aws-ovn-workers-rhel8
/override ci/prow/okd-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-upgrade

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 3, 2023

@patrickdillon: Overrode contexts on behalf of patrickdillon: ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-single-node, ci/prow/e2e-aws-ovn-workers-rhel8, ci/prow/okd-e2e-aws-ovn, ci/prow/okd-scos-e2e-aws-ovn, ci/prow/okd-scos-e2e-aws-upgrade

Details

In response to this:

/override ci/prow/e2e-aws-ovn-fips
/override ci/prow/e2e-aws-ovn-single-node
/override ci/prow/e2e-aws-ovn-workers-rhel8
/override ci/prow/okd-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-ovn
/override ci/prow/okd-scos-e2e-aws-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@patrickdillon
Copy link
Contributor

/override ci/prow/okd-e2e-aws-ovn-upgrade

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 3, 2023

@patrickdillon: Overrode contexts on behalf of patrickdillon: ci/prow/okd-e2e-aws-ovn-upgrade

Details

In response to this:

/override ci/prow/okd-e2e-aws-ovn-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Mar 3, 2023

@mtulio: This pull request references RFE-2782 which is a valid jira issue.

Details

In response to this:

The goal of this PR is to add support to AWS Local Zones when installing a cluster in an existing VPC. It allows the creation of a compute pool named edge, alongside the worker, on install-config.yaml. The installer will generate MachineSet manifests with NoSchedule taint to deploy the instance on edge locations.

This implementation is Phase 1 of the Enhancement Proposal #1232, which describes in detail the change request.

Nodes are considered edge when the availability zone zoneType is local-zone. The edge compute pool is valid only for the AWS Platform.

It is not the goal of this PR to create the Subnets on Local Zones locations. This is part of Phase 2.

Note: This PR will change the MTU of ClusterNetwork when edge pool for AWS is defined. FYI openshift/openshift-docs#55327

Refereces:

List of teams that would be interested in reviewing this PR:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@patrickdillon patrickdillon merged commit 0faae21 into openshift:master Mar 3, 2023
@sdodson
Copy link
Member

sdodson commented Mar 3, 2023

/cherry-pick release-4.13

@openshift-cherrypick-robot

@sdodson: new pull request created: #6931

Details

In response to this:

/cherry-pick release-4.13

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.