Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor dna.json and use S3 to access these files #6429

Open
wants to merge 6 commits into
base: develop
Choose a base branch
from

Conversation

himani2411
Copy link
Contributor

Description of changes

  • Refactoring dna.json to have comon-dna.json and node-specific-dna.json
  • Upload common-dna.json in s3 do-not-delete bucket from HeadNode using cookbook
    • Give PutObject access to HeadNode IAM role
  • Compute Fleet is supposed to download the json files through s3 in user_data.sh
    • Give GetObject, ListBucket to compute fleet node
  • Add Unit test

Tests

  • ONGOING
  update:
    test_update.py::test_dynamic_file_systems_update
  • Passed
scaling:
   test_scaling.py::test_job_level_scaling
  • Failing
    Unit test -> test_cluster_config_limits

References

  • Link to impacted open issues.
  • Link to related PRs in other packages (i.e. cookbook, node).
  • Link to documentation useful to understand the changes.

Checklist

  • Make sure you are pointing to the right branch.
  • If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
  • Check all commits' messages are clear, describing what and why vs how.
  • Make sure to have added unit tests or integration tests to cover the new/modified code.
  • Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Himani Deshpande added 6 commits September 11, 2024 18:01
…gle file

* Refactoring ComputeFleet to download common_dna.json and compute resource specific dna

* giving PutObject access to headnode for s3-do-not-delete bucket

* rename common,headnode and compute dna json

* Adding compute-dna.json and downloading it from S3

* Adding stack arn in a write file

* Using config Version to download and upload json files specific to a cluster-config version so that we can rollback using the previous config version when needed

* Reduce cfn-init for compute fleet to just mention files.This file is never created but a change in config version in AWS::CloudFormation::Init signifies an update

* give listversionObject and ListBucketVersions access to compute node

* Use cfn-init calls for Cluster Update Path and use cfn-init to create common-dna file for queues

* Use cfn-init to create common-dna file for queues
@himani2411 himani2411 added the 3.x label Sep 12, 2024
@himani2411 himani2411 requested review from a team as code owners September 12, 2024 01:29
@himani2411 himani2411 changed the title Scaling fix Refactor dna.json and use S3 to access these files Sep 12, 2024
Comment on lines +67 to +68
":s3:::parallelcluster-a69601b5ee1fc2f2-v1-do-not-delete/"
"parallelcluster/clusters/dummy-cluster-randomstring123/*",

Check warning

Code scanning / CodeQL

Implicit string concatenation in a list Warning test

Implicit string concatenation. Maybe missing a comma?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant