Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloudfront_distribution doesn't recognise S3 origin #1819

Closed
1 task done
gsimon75 opened this issue May 23, 2023 · 0 comments · Fixed by #1821
Closed
1 task done

cloudfront_distribution doesn't recognise S3 origin #1819

gsimon75 opened this issue May 23, 2023 · 0 comments · Fixed by #1821

Comments

@gsimon75
Copy link
Contributor

gsimon75 commented May 23, 2023

Summary

When I refer to an S3 bucket domain in the form {bucket_name}.s3.{region}.amazonaws.com, as per Origin Domain spec, it isn't recognised as an S3 domain here, so a custom_origin_config entry is added automatically here, which results in an error:

"botocore.errorfactory.InvalidOrigin: An error occurred (InvalidOrigin) when calling the CreateDistribution operation: You must specify either a CustomOrigin or an S3Origin. You cannot specify both."

The problem is in the method used for recognising S3 domains: whether it contains .s3.amazonaws.com or not (note the missing region part).

Issue Type

Bug Report

Component Name

cloudfront_distribution

Ansible Version

$ ansible --version
ansible [core 2.14.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/fules/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/fules/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
# /home/fules/.ansible/collections/ansible_collections
Collection    Version
------------- -------
amazon.aws    6.0.1
community.aws 6.0.0

# /usr/lib/python3/dist-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    5.4.0
ansible.netcommon             4.1.0
ansible.posix                 1.5.2
ansible.utils                 2.9.0
ansible.windows               1.13.0
arista.eos                    6.0.1
awx.awx                       21.14.0
azure.azcollection            1.15.0
check_point.mgmt              4.0.0
chocolatey.chocolatey         1.4.0
cisco.aci                     2.6.0
cisco.asa                     4.0.0
cisco.dnac                    6.7.1
cisco.intersight              1.0.27
cisco.ios                     4.5.0
cisco.iosxr                   4.1.0
cisco.ise                     2.5.12
cisco.meraki                  2.15.1
cisco.mso                     2.4.0
cisco.nso                     1.0.3
cisco.nxos                    4.3.0
cisco.ucs                     1.8.0
cloud.common                  2.1.3
cloudscale_ch.cloud           2.2.4
community.aws                 5.4.0
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.12.0
community.digitalocean        1.23.0
community.dns                 2.5.3
community.docker              3.4.3
community.fortios             1.0.0
community.general             6.6.0
community.google              1.0.0
community.grafana             1.5.4
community.hashi_vault         4.2.0
community.hrobot              1.8.0
community.libvirt             1.2.0
community.mongodb             1.5.2
community.mysql               3.6.0
community.network             5.0.0
community.okd                 2.3.0
community.postgresql          2.3.2
community.proxysql            1.5.1
community.rabbitmq            1.2.3
community.routeros            2.8.0
community.sap                 1.0.0
community.sap_libs            1.4.1
community.skydive             1.0.0
community.sops                1.6.1
community.vmware              3.5.0
community.windows             1.12.0
community.zabbix              1.9.3
containers.podman             1.10.1
cyberark.conjur               1.2.0
cyberark.pas                  1.0.17
dellemc.enterprise_sonic      2.0.0
dellemc.openmanage            6.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
dellemc.powerflex             1.6.0
dellemc.unity                 1.6.0
f5networks.f5_modules         1.23.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.2.3
frr.frr                       2.0.2
gluster.gluster               1.0.2
google.cloud                  1.1.3
grafana.grafana               1.1.1
hetzner.hcloud                1.11.0
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.11.0
infinidat.infinibox           1.3.12
infoblox.nios_modules         1.4.1
inspur.ispim                  1.3.0
inspur.sm                     2.3.0
junipernetworks.junos         4.1.0
kubernetes.core               2.4.0
lowlydba.sqlserver            1.3.1
mellanox.onyx                 1.0.0
microsoft.ad                  1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0
netapp.ontap                  22.5.0
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.4.0
netbox.netbox                 3.12.0
ngine_io.cloudstack           2.3.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.3
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.4.1
purestorage.flasharray        1.17.2
purestorage.flashblade        1.11.0
purestorage.fusion            1.4.2
sensu.sensu_go                1.13.2
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.32.2
theforeman.foreman            3.10.0
vmware.vmware_rest            2.3.1
vultr.cloud                   1.7.0
vyos.vyos                     4.0.2
wti.remote                    1.0.4

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.26.137
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.29.137
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.10/dist-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = /etc/ansible/ansible.cfg

OS / Environment

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy

Steps to Reproduce

-
  hosts: localhost
  collections:
    - community.aws
    - amazon.aws
  vars_files:
    - external_vars.yaml
  tasks:
    -
      name: Reading landing page domain from certificate
      acm_certificate_info:
        profile: "{{ aws_profile }}"
        region: "us-east-1"
        certificate_arn: "{{ landing_page_certificate_arn }}"
      register: landing_page_certificates
    -
      set_fact:
        landing_page_domain: "{{ landing_page_certificates.certificates[0].domain_name }}"
    -
      name: Create the S3 bucket
      s3_bucket:
        profile: "{{ aws_profile }}"
        state: present
        name: "{{ landing_page_domain }}"
        object_ownership: "BucketOwnerEnforced"
        encryption: "AES256"
        versioning: false
        public_access:
          block_public_acls: false
          block_public_policy: false
          ignore_public_acls: false
          restrict_public_buckets: false
      register: landing_page_bucket
    -
      name: Create CloudFront Access Identity
      cloudfront_origin_access_identity:
        state: present
        caller_reference: "LandingPageAccessIdentity"
        comment: "no comment"
      register: landing_page_access_identity
    -
      name: Create CloudFront Distribution
      cloudfront_distribution:
        profile: "{{ aws_profile }}"
        state: present
        http_version: "http2"
        caller_reference: "LandingPageDistribution"
        comment: "no comment"
        alias: "{{ landing_page_domain }}"
        viewer_certificate:
          acm_certificate_arn: "{{ landing_page_certificate_arn }}"
          ssl_support_method: "sni-only"
          minimum_protocol_version: "TLSv1.2_2021"
        origins:
          -
            id: "{{ landing_page_domain }}"
            domain_name: "{{ landing_page_bucket.name }}.s3.{{ aws_region }}.amazonaws.com"
            s3_origin_config:
              origin_access_identity: "origin-access-identity/cloudfront/{{ landing_page_access_identity.cloud_front_origin_access_identity.id }}"
            origin_shield:
              enabled: false
        # default_origin_domain_name: "{{ landing_page_bucket.name }}"
        default_root_object: "index.html"
        price_class: "PriceClass_200"
        wait: true
      register: landing_page_distribution

Expected Results

I expected that only the s3_origin_config is generated in the origin, and the custom_origin_config isn't.

Actual Results

botocore.errorfactory.InvalidOrigin: An error occurred (InvalidOrigin) when calling the CreateDistribution operation: You must specify either a CustomOrigin or an S3Origin. You cannot specify both.

Code of Conduct

  • I agree to follow the Ansible Code of Conduct
softwarefactory-project-zuul bot pushed a commit that referenced this issue Jun 22, 2023
Issue 1819 cloudfront distribution origin s3 domain

SUMMARY
Fixes #1819
As per Origin Domain Name spec now the S3 domain names are in the form {name}.s3.{region}.amazonaws.com, so the string fragment .s3.amazonaws.com no longer occurs in them, and therefore they aren't recognised as S3 origin domains.
Consequentially, the origin is treated as a custom one, so a custom_origin_config member is generated into it, which collides with the s3_origin_config and produces an error:

botocore.errorfactory.InvalidOrigin: An error occurred (InvalidOrigin) when calling the CreateDistribution operation: You must specify either a CustomOrigin or an S3Origin. You cannot specify both.

The backward-compatible way is to recognise both {name}.s3.amazonaws.com and {name}.s3.{domain}.amazonaws.com, but for this a regular expression is the most effective solution.
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
cloudfront_distribution
ADDITIONAL INFORMATION
The breakdown of the regex I used: \.s3(?:\.[^.]+)?\.amazonaws\.com$

\.s3 matches ".s3"
\.[^.]+ would match a dot followed by at least one, possibly more non-dot characters
(\.[^]+) would match the same, just grouped, so we could treat it as an atom
(?:\.[^]+) would match the same, just grouped in a non-capturing fashion (we don't want to extract the matched characters)
(?:\.[^]+)? matches the same, occuring 0 or 1 times
\.amazonaws\.com matches ".amazonaws.com"
$ matches the end of the input string

Reviewed-by: Markus Bergholz <[email protected]>
Reviewed-by: Alina Buzachis
patchback bot pushed a commit that referenced this issue Jun 22, 2023
Issue 1819 cloudfront distribution origin s3 domain

SUMMARY
Fixes #1819
As per Origin Domain Name spec now the S3 domain names are in the form {name}.s3.{region}.amazonaws.com, so the string fragment .s3.amazonaws.com no longer occurs in them, and therefore they aren't recognised as S3 origin domains.
Consequentially, the origin is treated as a custom one, so a custom_origin_config member is generated into it, which collides with the s3_origin_config and produces an error:

botocore.errorfactory.InvalidOrigin: An error occurred (InvalidOrigin) when calling the CreateDistribution operation: You must specify either a CustomOrigin or an S3Origin. You cannot specify both.

The backward-compatible way is to recognise both {name}.s3.amazonaws.com and {name}.s3.{domain}.amazonaws.com, but for this a regular expression is the most effective solution.
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
cloudfront_distribution
ADDITIONAL INFORMATION
The breakdown of the regex I used: \.s3(?:\.[^.]+)?\.amazonaws\.com$

\.s3 matches ".s3"
\.[^.]+ would match a dot followed by at least one, possibly more non-dot characters
(\.[^]+) would match the same, just grouped, so we could treat it as an atom
(?:\.[^]+) would match the same, just grouped in a non-capturing fashion (we don't want to extract the matched characters)
(?:\.[^]+)? matches the same, occuring 0 or 1 times
\.amazonaws\.com matches ".amazonaws.com"
$ matches the end of the input string

Reviewed-by: Markus Bergholz <[email protected]>
Reviewed-by: Alina Buzachis
(cherry picked from commit 509ccad)
softwarefactory-project-zuul bot pushed a commit that referenced this issue Jul 2, 2023
…tion origin s3 domain (#1849)

[PR #1821/509ccad9 backport][stable-6] Issue 1819 cloudfront distribution origin s3 domain

This is a backport of PR #1821 as merged into main (509ccad).
SUMMARY
Fixes #1819
As per Origin Domain Name spec now the S3 domain names are in the form {name}.s3.{region}.amazonaws.com, so the string fragment .s3.amazonaws.com no longer occurs in them, and therefore they aren't recognised as S3 origin domains.
Consequentially, the origin is treated as a custom one, so a custom_origin_config member is generated into it, which collides with the s3_origin_config and produces an error:

botocore.errorfactory.InvalidOrigin: An error occurred (InvalidOrigin) when calling the CreateDistribution operation: You must specify either a CustomOrigin or an S3Origin. You cannot specify both.

The backward-compatible way is to recognise both {name}.s3.amazonaws.com and {name}.s3.{domain}.amazonaws.com, but for this a regular expression is the most effective solution.
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
cloudfront_distribution
ADDITIONAL INFORMATION
The breakdown of the regex I used: \.s3(?:\.[^.]+)?\.amazonaws\.com$

\.s3 matches ".s3"
\.[^.]+ would match a dot followed by at least one, possibly more non-dot characters
(\.[^]+) would match the same, just grouped, so we could treat it as an atom
(?:\.[^]+) would match the same, just grouped in a non-capturing fashion (we don't want to extract the matched characters)
(?:\.[^]+)? matches the same, occuring 0 or 1 times
\.amazonaws\.com matches ".amazonaws.com"
$ matches the end of the input string

Reviewed-by: Mark Chappell
abikouo pushed a commit to abikouo/community.aws that referenced this issue Oct 24, 2023
…_certificate and the corresponding _info module (ansible-collections#1819)

Revert pull request ansible-collections#1805 that migrated iam_server_certificate and the corresponding _info module

This reverts commit 978c802, reversing changes made to 06f8f53.
SUMMARY


ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: GomathiselviS
Reviewed-by: Bikouo Aubin
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant