Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move data from sto2 to sto4 location #1448

Merged
merged 43 commits into from
Aug 9, 2023

Conversation

i-oden
Copy link
Member

@i-oden i-oden commented Jul 28, 2023

Before submitting this PR

  1. Description: At the moment the data delivered via the DDS is temporarily stored on Safesprings S3 storage location sto2. This storage is not included in our contract and therefore costs more. Sto4 is included, we have up to 1 PB there at the moment. We need to start creating projects on sto4 from now on and phase out sto2.
  2. Jira task / GitHub issue: DDS-1419
  3. How to test: Add information on how someone could manually test this functionality. As detailed as possible.
  4. Type of change: Check the relevant boxes in the section below
  5. Add docstrings and comments to code, even if you personally think it's obvious.

What type of change(s) does the PR contain?

  • New feature
    • Breaking: Please describe the reason for the break and how we can fix it.
    • Non-breaking
  • Database change
    • Migration included in PR
    • Migration not needed
  • Bug fix
    • Breaking: Please describe the reason for the break and how we can fix it.
    • Non-breaking
  • Security Alert fix
  • Documentation
  • Tests (only)
  • Workflow

Checklist

  • Sprintlog
    • Added
    • Not needed (E.g. PR contains only tests)
  • Rebase / Update / Merge from base branch (the branch from which the current is forked)
    • Done
    • Not needed
  • Blocking PRs
    • Merged
    • No blocking PRs
  • PR to master branch

Actions / Scans

  • Black: Python code formatter. Does not execute. Only tests.
    Run black . locally to execute formatting.
    • Passed
  • Prettier: General code formatter. Our use case: MD and yaml mainly.
    Run npx prettier --write . locally to execute formatting.
    • Passed
  • Yamllint: Linting of yaml files.
    • Passed
  • Tests: Pytest to verify that functionality works as expected.
    • New tests added
    • No new tests
    • Passed
  • CodeQL: Scan for security vulnerabilities, bugs, errors
    • New alerts: Go through them and either fix, dismiss och ignore. Add reasoning in items below.
    • Alerts fixed: What?
    • Alerts ignored / dismissed: Why?
    • Passed
  • Trivy: Security scanner
    • New alerts: Go through them and either fix, dismiss och ignore. Add reasoning in items below.
    • Alerts fixed: What?
    • Alerts ignored / dismissed: Why?
    • Passed
  • Snyk: Security scanner
    • New alerts: Go through them and either fix, dismiss och ignore. Add reasoning in items below.
    • Alerts fixed: What?
    • Alerts ignored / dismissed: Why?
    • Passed

@i-oden i-oden self-assigned this Jul 28, 2023
@i-oden i-oden force-pushed the DDS-1419-move-data-from-sto-2-to-sto-4 branch from 7f94008 to ec4426e Compare July 28, 2023 08:02
@codecov
Copy link

codecov bot commented Jul 28, 2023

Codecov Report

Merging #1448 (6034d5c) into dev (ff551b5) will increase coverage by 0.07%.
The diff coverage is 94.04%.

@@            Coverage Diff             @@
##              dev    #1448      +/-   ##
==========================================
+ Coverage   89.39%   89.47%   +0.07%     
==========================================
  Files          29       29              
  Lines        4298     4378      +80     
==========================================
+ Hits         3842     3917      +75     
- Misses        456      461       +5     
Files Changed Coverage Δ
dds_web/commands.py 75.49% <92.42%> (+2.29%) ⬆️
dds_web/__init__.py 92.43% <100.00%> (+0.06%) ⬆️
dds_web/api/api_s3_connector.py 90.24% <100.00%> (+0.50%) ⬆️
dds_web/database/models.py 96.18% <100.00%> (+0.04%) ⬆️
dds_web/utils.py 92.61% <100.00%> (+0.23%) ⬆️

@i-oden i-oden requested a review from valyo July 31, 2023 10:50
@i-oden i-oden removed the request for review from valyo August 3, 2023 08:58
@i-oden i-oden requested a review from valyo August 8, 2023 09:08
@i-oden i-oden marked this pull request as ready for review August 8, 2023 09:08
Copy link
Member

@valyo valyo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from those small comments, it looks good, and works as expected when I tested creating projects before and after updating the unit

dds_web/api/api_s3_connector.py Outdated Show resolved Hide resolved
tests/test_project_creation.py Show resolved Hide resolved
@i-oden i-oden merged commit 305b4fe into dev Aug 9, 2023
14 checks passed
@i-oden
Copy link
Member Author

i-oden commented Aug 9, 2023

I have made a copy of this branch and set a branch protection on it: read only.
move_storage_location_template

@i-oden i-oden deleted the DDS-1419-move-data-from-sto-2-to-sto-4 branch August 9, 2023 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants