Adding boto3 integration

[xydinesh]

No description provided.

[tonybaloney]

Does the AWS pack not already cover this functionality? https://github.com/StackStorm-Exchange/stackstorm-aws

[xydinesh]

Only for some extent, there are gaps in terms of using boto3. We can not use boto3 functionalities where current aws pack implemented using boto2.

In addition, this implementation does not depend on yaml file for actions. As a result users depend on boto3 documentation for action documentation.

[Mierdin]

Looks to me that there's already some boto3 support in the AWS pack, so I'm curious what you meant by "We can not use boto3 functionalities where current aws pack implemented using boto2."

Without knowing the extent of what you feel is missing from the AWS pack, I'm not yet convinced this action is needed. Even if it was necessary, I don't see the reason for a totally separate pack focused on boto3.

Can you elaborate on specifics regarding what the AWS pack doesn't do for you that convinced you to go the catch-all getattr approach?

[xydinesh]

For example, I have a st2 deployment in aws account 123456 and us-east-1. I want to deploy VPC in account 456789 and us-west-1. Current aws pack does not have a way to do this task.

With boto3 pack, I approached it as follows. Let’s assume, there is a role called st2_role in account 456789

     assume_role:
        action: boto3.assume_role
        input:
           role_arn: “arn:aws:iam:456789:role/st2_role”
        publish:
          credentials: <% task(assume_role).result.result %>
       on-success:
         - create_vpc
    
    create_vpc:
      action: boto3.action
      input:
        service: ec2
        action_name: create_vpc
        region: "us-west-1"
        params: <% dict(CidrBlock => "10.0.0.0/16", InstanceTenancy => "default") %>
        credentials: <% $.credentials %>
      publish:
          vpc_id: <% task(create_vpc).result.result.Vpc.VpcId %>
      on-success:
        - create_subnet

    create_subnet:
      action: boto3.action
      input:
        service: ec2
        region: "us-west-1"
        action_name: create_subnet
        params: <% dict(AvailabilityZone =>"us-west-1a", CidrBlock =>"10.0.0.0/24", VpcId => $.vpc_id) %>
        credentials: <% $.credentials %>
      publish:
        subnet_ids: <% task(create_subnets).result.result.select($.Subnet.SubnetId) %>
      on-success:
         - create_igw

     create_igw:
        action: boto3.action
        input:
          service: ec2
          action_name: create_internet_gateway
          region: <% $.region %>
          credentials: <% $.credentials %>
        publish:
          igw_id: <% task(create_igw).result.result.InternetGateway.InternetGatewayId %>

getattr enabled me to specify service, action and params. Advantage of this pack is, we don't have to do any modifications to access new boto3 functionality.

[xydinesh]

Another example on creating IAM role.

    assume_role:
      action: boto3.assume_role
      input:
        role_arn: “arn:aws:iam:456789:role/st2_role”
      publish:
        credentials: <% task(assume_role).result.result %>
      on-success:
          - iam_create_role

    iam_create_role:
      action: boto3.action
      input:
        region: "us-west-1"
        service: iam
        action_name: create_role
        credentials: <% $.credentials %>
        params: <% dict(RoleName => "AwesomeNewRole", AssumeRolePolicyDocument =>
                             {"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":["ec2.amazonaws.com" ]},"Action":["sts:AssumeRole"]}]}', Description => "Allows EC2 instances to call AWS services.") %>
        credentials: <% $.credentials %>
      on-success:
        - attach_role_policy_1
...

[Susanthab]

I can give another example where AWS pack did not work for me. I wanted to create LaunchConfiguration using aws.autoscaling_create_launch_configuration action. However, I could not figure out a way to pass a value to BlockDeviceMapping parameter. It always gave me a parameter validation error. I tried so many different ways to pass it but never worked for me. I spent several days with no luck. Then later I switched to this custom action and it all worked.

[Mierdin]

@Susanthab Unfortunately it's not obvious from your comment exactly what went wrong, or why this approach worked better for you. I'd advise opening an issue on the AWS pack repo.

[Susanthab]

@Mierdin Try to create a launch configuration using aws.autoscaling_create_launch_configuration action with BlockDeviceMapping details, then you will see the issue. I already created an issue with AWS pack repo.

[Mierdin]

@xydinesh I see what you're saying, but I'm still 👎 on the idea of a boto3 pack when the aws pack already exists. boto3 is the AWS Python library, so having both would be fairly confusing.

To me, it seems like it's coming down to a tradeoff between the confusion that will arise from having both an aws and boto3 pack vs the small amount of extra time it would take to create a PR to the AWS pack to add the necessary actions you're looking for, and IMO the confusion isn't worth it. You could submit a PR for a new action in the AWS pack that does this dynamic functionality you're looking for (i.e. aws.misc_boto), but even still, I would prefer that the effort goes into ensuring the AWS pack is caught up with the functionality in the library.

[xydinesh]

@Mierdin Thanks for taking time to review. For me, It seems redundant to have this action inside current aws pack. It will result in the same confusion as having two packs. Let me gather my thoughts on this and reply later.

[Susanthab]

In my opinion, having separate AWS pack with all individual actions similar to boto3 actions is a duplicate effort and creates more room for bugs and increase the complexity as well. Instead, just have one wrapper action in st2 which can call boto3 actions, then its simple, no added complexity, no room for bugs because it's just a wrapper action which essentially calls the boto3 action underneath it.

[eedgar]

I am concerned about the current aws pack and its mixed use of boto2/boto3. I assume it makes some assumptions and to move it to boto3 entirely may break the existing config. I have problems with the current aws pack though. I need to be able to assume roles into many accounts and the current pack is NOT flexible enough to allow that.

[xydinesh]

@Mierdin I have following opinion differences with current pack.

• aws credentials
  Boto3 is the official SDK for AWS. As a user/developer, If I have boto3 configured I expect my boto3 pack to work without doing any additional configuration. For example,

st2 pack install boto3
st2 run boto3.action service="ec2" action_name="decribe_vpcs" region="us-west-1"

In addition, if I want to use use boto3 profiles

st2 run boto3.action service="ec2" action_name="decribe_vpcs" region="us-west-1" env="AWS_PROFILE=production"

• yaml generation
  Long term, I don’t believe yaml generation scale based on the number of services AWS have and introduce. In addition, IMO Boto3 documentation is detailed, has examples. Having yaml for each action is redundant and add no value to the end user.

• pack maintenance

  Since there are no yaml to generate, this pack should be easy to maintain. Any new service boto3 introduce, available to pack user right away.

I'm open to idea of having this functionality as part of current pack. I didn’t think, I could integrate this with existing AWS pack. However, if I by pass “run.py”, I can add this as a part of current aws pack.

[Mierdin]

I've already given my opinion on the presence of both packs. My recommendation is still to add a new action to the AWS pack that does what you're looking for.

The bottom line is that all of the concerns listed here are perfectly valid, but aren't solved by simply having a separate boto3 pack. IMO, the best long-term discussion to be having is how to move the aws pack in a direction like what's contained in this PR, instead of having tons of discrete actions (and having to maintain all those metadata files). I think that's a great discussion to have, but it should happen within the context of the long-term roadmap for the aws pack. The question will be how to move to the new model while respecting the existing workflows that use the existing actions in the pack. I am sure it can be done, just needs to be planned. Creating a new aws.boto3action action in that pack is probably step one.

FYI this came up in our daily standup this morning, so I'd expect some other team members will chime in soon.

[warrenvw]

@xydinesh You have a lot of great ideas, and I definitely agree that we should not be creating a YAML file for every action. There's just way too many of them. Additionally, the rendering of the boto3 documentation to YAML is really poor. I tried reading it and gave up.

I agree with everything @Mierdin mentioned. It makes sense that we make the necessary changes to the aws pack so that it can do what you need. It is important that we respect existing workflows that make use of the exiting actions. This will take planning.

May I propose that we set up a meeting for early next week to plan next steps? In the meantime, feel free to continue commenting in this PR with any ideas...

[xydinesh]

@Mierdin Since we are having a discussion on long-term road map of aws pack and haveing a general agreement on what it should look like. I can give up the idea of having a separate boto3 pack. I can get behind the idea of moving aws pack towards more dynamic functionality. Let me send a pull request to aws pack to add aws.boto3action next couple of days. @warrenvw Ill contact you on slack to find a time slot works for all of us. Thanks for sharing your thoughts.

[LindsayHill]

New repo now in place for this: github.com/StackStorm-Exchange/stackstorm-aws_boto3

[LindsayHill]

New repo at https://github.com/StackStorm-Exchange/stackstorm-aws_boto3