[COST-4945] Create a new API to return list of EC2-compute instances #5117

bacciotti · 2024-05-20T13:16:18Z

Jira Ticket

Description

This change creates a new API to return a list of AWS EC2 compute instances.

Testing

Checkout Branch
Restart Koku

Load test data for AWS

make create-test-customer
make load-test-customer-data test_source=aws

Hit endpoint

http://127.0.0.1:8000/api/cost-management/v1/reports/aws/resources/ec2-compute/

You should see a list of EC2 instances in the response data, similar to this:

{
    "meta": {
        "count": 5,
        "limit": 3,
        "offset": 0,
        "currency": "USD",
        "cost_type": "unblended_cost",
        "filter": {
            "time_scope_value": "-1",
            "time_scope_units": "month",
            "resolution": "monthly"
        },
        "group_by": {},
        "order_by": {},
        "exclude": {},
        "total": {
            "usage": {
                "value": 1380.0,
                "units": "Hrs"
            },
            "infrastructure": {
                "raw": {
                    "value": 132.48,
                    "units": "USD"
                },
                "markup": {
                    "value": 13.248,
                    "units": "USD"
                },
                "usage": {
                    "value": 0.0,
                    "units": "USD"
                },
                "total": {
                    "value": 145.728,
                    "units": "USD"
                }
            },
            "supplementary": {
                "raw": {
                    "value": 0.0,
                    "units": "USD"
                },
                "markup": {
                    "value": 0.0,
                    "units": "USD"
                },
                "usage": {
                    "value": 0.0,
                    "units": "USD"
                },
                "total": {
                    "value": 0.0,
                    "units": "USD"
                }
            },
            "cost": {
                "raw": {
                    "value": 132.48,
                    "units": "USD"
                },
                "markup": {
                    "value": 13.248,
                    "units": "USD"
                },
                "usage": {
                    "value": 0.0,
                    "units": "USD"
                },
                "total": {
                    "value": 145.728,
                    "units": "USD"
                }
            }
        }
    },
    "links": {
        "first": "/api/cost-management/v1/reports/aws/resources/ec2-compute/?limit=3&offset=0",
        "next": "/api/cost-management/v1/reports/aws/resources/ec2-compute/?limit=3&offset=3",
        "previous": null,
        "last": "/api/cost-management/v1/reports/aws/resources/ec2-compute/?limit=3&offset=2"
    },
    "data": [
        {
            "date": "2024-07",
            "resource_ids": [
                {
                    "resource_id": "i-55555557",
                    "values": [
                        {
                            "resource_id": "i-55555557",
                            "date": "2024-07",
                            "usage": {
                                "value": 276.0,
                                "units": "Hrs"
                            },
                            "source_uuid": [
                                "cfbf64c7-3d58-43fd-a18f-4cd217e6a2d4"
                            ],
                            "account_alias": null,
                            "account": "9999999999998",
                            "instance_name": "instance_name_3",
                            "instance_type": "m5.large",
                            "operating_system": "Debian",
                            "region": "us-east-1",
                            "vcpu": 2,
                            "memory": "8 GiB",
                            "tags": [
                                {
                                    "key": "Name",
                                    "values": [
                                        "instance_name_3"
                                    ]
                                },
                                {
                                    "key": "version",
                                    "values": [
                                        "gamma"
                                    ]
                                },
                                {
                                    "key": "com_redhat_rhel_sla",
                                    "values": [
                                        "Standard"
                                    ]
                                }
                            ],
                            "infrastructure": {
                                "raw": {
                                    "value": 26.496,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 2.6496,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 29.1456,
                                    "units": "USD"
                                }
                            },
                            "supplementary": {
                                "raw": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 0.0,
                                    "units": "USD"
                                }
                            },
                            "cost": {
                                "raw": {
                                    "value": 26.496,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 2.6496,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 29.1456,
                                    "units": "USD"
                                }
                            }
                        }
                    ]
                },
                {
                    "resource_id": "i-55555556",
                    "values": [
                        {
                            "resource_id": "i-55555556",
                            "date": "2024-07",
                            "usage": {
                                "value": 276.0,
                                "units": "Hrs"
                            },
                            "source_uuid": [
                                "cfbf64c7-3d58-43fd-a18f-4cd217e6a2d4"
                            ],
                            "account_alias": null,
                            "account": "9999999999998",
                            "instance_name": "instance-name-2",
                            "instance_type": "m5.large",
                            "operating_system": "Gentoo Linux",
                            "region": "us-east-1",
                            "vcpu": 2,
                            "memory": "8 GiB",
                            "tags": [
                                {
                                    "key": "Name",
                                    "values": [
                                        "instance-name-2"
                                    ]
                                },
                                {
                                    "key": "version",
                                    "values": [
                                        "beta"
                                    ]
                                },
                                {
                                    "key": "CoM_RedHat_Rhel_varianT",
                                    "values": [
                                        "Workstation"
                                    ]
                                },
                                {
                                    "key": "com_redhat_rhel_conversion",
                                    "values": [
                                        "True"
                                    ]
                                }
                            ],
                            "infrastructure": {
                                "raw": {
                                    "value": 26.496,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 2.6496,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 29.1456,
                                    "units": "USD"
                                }
                            },
                            "supplementary": {
                                "raw": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 0.0,
                                    "units": "USD"
                                }
                            },
                            "cost": {
                                "raw": {
                                    "value": 26.496,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 2.6496,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 29.1456,
                                    "units": "USD"
                                }
                            }
                        }
                    ]
                },
                {
                    "resource_id": "i-55555555",
                    "values": [
                        {
                            "resource_id": "i-55555555",
                            "date": "2024-07",
                            "usage": {
                                "value": 276.0,
                                "units": "Hrs"
                            },
                            "source_uuid": [
                                "cfbf64c7-3d58-43fd-a18f-4cd217e6a2d4"
                            ],
                            "account_alias": null,
                            "account": "9999999999998",
                            "instance_name": "instance-name-1",
                            "instance_type": "m5.large",
                            "operating_system": "Fedora",
                            "region": "us-east-1",
                            "vcpu": 2,
                            "memory": "8 GiB",
                            "tags": [
                                {
                                    "key": "Name",
                                    "values": [
                                        "instance-name-1"
                                    ]
                                },
                                {
                                    "key": "version",
                                    "values": [
                                        "prod"
                                    ]
                                },
                                {
                                    "key": "dashed-key-on-aws",
                                    "values": [
                                        "dashed-value"
                                    ]
                                }
                            ],
                            "infrastructure": {
                                "raw": {
                                    "value": 26.496,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 2.6496,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 29.1456,
                                    "units": "USD"
                                }
                            },
                            "supplementary": {
                                "raw": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 0.0,
                                    "units": "USD"
                                }
                            },
                            "cost": {
                                "raw": {
                                    "value": 26.496,
                                    "units": "USD"
                                },
                                "markup": {
                                    "value": 2.6496,
                                    "units": "USD"
                                },
                                "usage": {
                                    "value": 0.0,
                                    "units": "USD"
                                },
                                "total": {
                                    "value": 29.1456,
                                    "units": "USD"
                                }
                            }
                        }
                    ]
                }
            ]
        }
    ]
}

Example valid filter/order by params to use with this API

filter[time_scope_value]=-2
filter[time_scope_units]=month
filter[resolution]=monthly
filter[region]=us-east-1
filter[account]=*
order_by[usage]=asc
order_by[cost]=desc
filter[tag:version]=beta
filter[operating_system]=linux
filter[instance_name]=instance

Example invalid params not expected to work with this API

filter[time_scope_value]=-10
filter[time_scope_units]=day
filter[resolution]=daily
filter[tags]=beta
filter[bad_param]=bad-param-value
filter[vcpu]=1
filter[limit]=3
filter[offset]=1
order_by[bad_param]=asc
start_date=2024-07-01
end_date=2024-07-1

Release Notes

proposed release note

* [COST-4945](https://issues.redhat.com/browse/COST-4945) Create a new API to return a list of AWS EC2 compute instances.

… pieces.

codecov · 2024-05-20T13:33:38Z

Codecov Report

Attention: Patch coverage is 92.10526% with 9 lines in your changes missing coverage. Please review.

Project coverage is 94.1%. Comparing base (7c64c4f) to head (45decf9).

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #5117   +/-   ##
=====================================
  Coverage   94.1%   94.1%           
=====================================
  Files        373     373           
  Lines      31040   31127   +87     
  Branches    4564    4571    +7     
=====================================
+ Hits       29214   29297   +83     
- Misses      1162    1165    +3     
- Partials     664     665    +1

…u/koku into cost-4945_awsec2_compute_api

update unit tests

maskarb

a few comments:

koku/api/query_params.py

koku/api/report/aws/serializers.py

koku/api/report/aws/provider_map.py

koku/api/report/aws/serializers.py

koku/api/report/aws/query_handler.py

koku/api/report/view.py

- move report_specific filters to main filter map - use report_type instead of kwargs in get_paginator - do not use deepcopy - just overwrite query_data - resolution and time_scope_units are always monthly and month respectively - overide start and end date params in base ParamSerializer - overide limit and offset in base FilterSerializer

…sec2_compute_api

koku/api/report/aws/serializers.py

maskarb

Looks good to me! Nicely done.

bacciotti added 2 commits May 20, 2024 14:09

feat: cleaning up old changes. making first changes to create the API…

12e09ce

… pieces.

feat: cleaning up old changes. making first changes to create the API…

9d85c75

… pieces.

github-actions bot added the smokes-required label May 20, 2024

bacciotti added the aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests label May 20, 2024

bacciotti and others added 10 commits May 21, 2024 11:16

Merge branch 'main' into cost-4945_awsec2_compute_api

adf75a3

Merge branch 'main' into cost-4945_awsec2_compute_api

b6f3dd9

feat: insert new filters.

57ae32d

Merge branch 'main' into cost-4945_awsec2_compute_api

0355c52

feat: insert new filters and order by params.

bbce41d

feat: customizing provider map and serializer.

f13f9d2

fix: changing TIME_CHOICES options.

87e56ed

Merge branch 'main' into cost-4945_awsec2_compute_api

6c624c1

feat: first unit tests.

f970d16

feat: wip.

b945ffc

cgoodfred removed the aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests label May 28, 2024

bacciotti and others added 14 commits May 29, 2024 11:35

feat: fixing provider map.

9f6230a

feat: fixing provider map.

e21c004

Merge branch 'main' into cost-4945_awsec2_compute_api

fd34ce8

update ec2 annotations to get all required fields

e863cf9

use AWSEC2ComputeQueryParamSerializer

af45f4f

Merge branch 'cost-4945_awsec2_compute_api' of github.com:project-kok…

1e4b69d

…u/koku into cost-4945_awsec2_compute_api

feat: creating orderby and groupby serializers for ec2

7e5abcb

feat: unit tests for filters

823bdcf

Merge branch 'main' into cost-4945_awsec2_compute_api

ff37a9a

feat: removing group_by - not needed on ec2

f4139ff

feat: fixing orderby serializer and starting units tests

0e3af90

fix: typo

1c2fb62

feat: changing usage_hours to usage_amount

9b5938f

Merge branch 'main' into cost-4945_awsec2_compute_api

1490fe7

bacciotti and others added 6 commits July 9, 2024 14:35

fix: fixing parameters validations.

80b4976

update linting

11700a0

Merge branch 'cost-4945_awsec2_compute_api' of github.com:project-kok…

d8b2440

…u/koku into cost-4945_awsec2_compute_api

squash commits

7aaa6bd

Merge branch 'cost-4945_awsec2_compute_api' of github.com:project-kok…

84e8826

…u/koku into cost-4945_awsec2_compute_api

clean up query params and time period settings

a82249f

djnakabaale added smoke-tests pr_check will build the image and run minimal required smokes and removed aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests labels Jul 11, 2024

djnakabaale added 2 commits July 10, 2024 23:52

do not use filter keyword

cc26986

more code clean up

be974bc

update unit tests

djnakabaale marked this pull request as ready for review July 16, 2024 01:52

djnakabaale requested review from a team as code owners July 16, 2024 01:52

Merge branch 'main' into cost-4945_awsec2_compute_api

d0ad047

maskarb reviewed Jul 16, 2024

View reviewed changes

djnakabaale added 7 commits July 16, 2024 14:33

more unit tests

8547650

update openapi spec

4447066

clean up and add unit tests

7ddbfa4

Merge branch 'main' of github.com:project-koku/koku into cost-4945_aw…

834a95e

…sec2_compute_api

move changes to openapi spec to a separate pr

c3d0ab0

Merge branch 'main' of github.com:project-koku/koku into cost-4945_aw…

924a0d3

…sec2_compute_api

maskarb reviewed Jul 18, 2024

View reviewed changes

koku/api/report/aws/serializers.py Outdated Show resolved Hide resolved

use serializer choice field and not customer validate method

592ef1c

maskarb reviewed Jul 18, 2024

View reviewed changes

koku/api/report/aws/serializers.py Show resolved Hide resolved

maskarb approved these changes Jul 18, 2024

View reviewed changes

Merge branch 'main' into cost-4945_awsec2_compute_api

45decf9

djnakabaale enabled auto-merge (squash) July 19, 2024 10:20

djnakabaale merged commit 45d8e31 into main Jul 19, 2024
11 checks passed

djnakabaale deleted the cost-4945_awsec2_compute_api branch July 19, 2024 11:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[COST-4945] Create a new API to return list of EC2-compute instances #5117

[COST-4945] Create a new API to return list of EC2-compute instances #5117

bacciotti commented May 20, 2024 •

edited by djnakabaale

Loading

codecov bot commented May 20, 2024 •

edited

Loading

maskarb left a comment

maskarb left a comment

[COST-4945] Create a new API to return list of EC2-compute instances #5117

[COST-4945] Create a new API to return list of EC2-compute instances #5117

Conversation

bacciotti commented May 20, 2024 • edited by djnakabaale Loading

Jira Ticket

Description

Testing

Release Notes

codecov bot commented May 20, 2024 • edited Loading

Codecov Report

maskarb left a comment

Choose a reason for hiding this comment

maskarb left a comment

Choose a reason for hiding this comment

bacciotti commented May 20, 2024 •

edited by djnakabaale

Loading

codecov bot commented May 20, 2024 •

edited

Loading