Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transition from S3 to Glacier without copy and delete #324

Open
nateiam opened this issue May 13, 2017 · 0 comments
Open

Transition from S3 to Glacier without copy and delete #324

nateiam opened this issue May 13, 2017 · 0 comments

Comments

@nateiam
Copy link
Contributor

nateiam commented May 13, 2017

As a Herd Administrator, I want data to move from S3 to Glacier without having to copy the data to another bucket and delete it in the origin bucket.

Tech Note:

  • Storage class transition currently operates by performing an S3 to S3 copy to a bucket with a 0d Glacier policy
  • This story re-implements storage class transition to occur within the same bucket via object tagging and lifecycle policy

Acceptance Criteria

  • Storage policy transitions execute according to new technical approach described above.
  • Storage unit for the BData cycles through the following during transition
    ** Enabled - normal status prior to any archiving
    ** Archiving - status while archiving (tech note: this applies only to S3 object tagging, not AWS async processing that actually transitions object)
    ** Archived - status once archiving is complete (tech note: again applies to when tagging is complete, AWS async processing could be ongoing after this)
  • Storage policy transitions work only for S3_MANAGED Bucket. Other buckets including KMS will be handled as future stories
  • Availability and DDL consider archived objects as "NotAvailable" (tech note: minor changes to existing logic that does this)
  • Existing rule that limits transition based on BData size is removed
  • Existing logic and data around additional Glacier storage is removed, including:
    ** Operations that move data to GLACIER_MANAGED
    ** Logic that checks file existence and size after archiving

Infrastructure required:
** Glacier Lifecycle Policy
** Permissions for S3 object tagging

Existing logic that does not need to be modified and does not need to be separately tested

  • All Storage Policy Scheduler logic
  • Check that archiving is already underway
FINRAOSS pushed a commit that referenced this issue May 18, 2018
* commit 'ad25f676aa53750ba2c0ba2a749157866a41f9ed':
  DM-10629: Factory CQ - NullPointerException thrown in herd IndexFunctionsDaoImpl.getIndexStats()
  DM-10629: Factory CQ - NullPointerException thrown in herd IndexFunctionsDaoImpl.getIndexStats()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant