aws_ec2 use_ssm_inventory fails with a large number of instances

[zombrie]

Summary

After upgrading to version 6.0.0 and enabling use_ssm_inventory, dynamic inventory graphing fails with a warning. Our setup has many instances, and so the list of instance ids is well over the 40 character limit for the aws ssm get-inventory api endpoint.

Issue Type

Bug Report

Component Name

aws_ec2

Ansible Version

❯ ansible --version
ansible [core 2.15.3]

Collection Versions

❯ ansible-galaxy collection list

# /opt/homebrew/lib/python3.11/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.0.0  

AWS SDK versions

❯ pip3 show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.26.152
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /opt/homebrew/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.29.152
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /opt/homebrew/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

No response

OS / Environment

No response

Steps to Reproduce

inventory/aws_ec2.yml

---
##
# AWS EC2 Inventory
##

# Dynamic AWS inventory: Uses the aws_ec2 plugin to generate a dynamic inventory.
# For more info and examples see:
#  - https://docs.ansible.com/ansible/latest/collections/amazon/aws/docsite/aws_ec2_guide.html
#  - https://docs.ansible.com/ansible/latest/collections/amazon/aws/aws_ec2_inventory.html#id3
#
# Usage:
# - Run `ansible-inventory --graph` to list the available hosts and groups.

# Enable the aws_ec2 plugin
plugin: aws_ec2

# Enables fetching additional EC2 instance information from the AWS Systems Manager (SSM) inventory service into hostvars.
use_ssm_inventory: true

##
# Hosts
##

# This controls how we display hostnames.
hostnames:

  # Using the Hostname tag is more aligned with the legacy static inventory and easier to reference for humans. However using `instance-id` might be more practical in the future with dynamic groups.
  - tag:Hostname

##
# Groups
##

# Generate dynamic groups to run playbooks against.
keyed_groups:
  # Using tags to generate groups. This gives something similar to our legacy static inventory with familiar names to identify instances with.

  # By Application name
  - key: tags.Application
    prefix: App

  # By Environment
  - key: tags.Environment
    prefix: Env


##
# Compose
##

# Compose creates vars from Jinja2 templates but also allows us to set some connection details for the dynamic inventory.
compose:
  # Use SSM Agent to connect to the instance ID. This means no ssh keys are needed..
  ansible_host: instance_id
  ansible_connection: 'community.aws.aws_ssm'

command used:

❯ ansible-inventory --graph

Expected Results

I expect to get a graph output of instances grouped appropriately. This occurs as expected with the above inventory/aws_ec2.yml with amazon.aws collection version 5.5.0 and earlier.

Actual Results

❯ ansible-inventory --graph
[WARNING]:  * Failed to parse /Users/<>/inventory/aws_ec2.yaml with ansible_collections.amazon.aws.plugins.inventory.aws_ec2 plugin: An error occurred
(ValidationException) when calling the GetInventory operation: 1 validation error detected: Value '[<instance ids list>]' at 'filters.1.member.values' failed to satisfy constraint: Member must have length less than or equal to 40
[WARNING]: Unable to parse /Users/<>/inventory/aws_ec2.yaml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
@all:
  |--@ungrouped:

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibullbot]

Files identified in the description:

• [plugins/inventory/aws_ec2.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/inventory/aws_ec2.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[tomphill16]

The exact same thing is happening to me:

...at 'filters.1.member.values' failed to satisfy constraint: Member must have length less than or equal to 40

[liad5h]

same issue here

[vijayreddiar]

@abikouo The option use_ssm_inventory limits to use only if the number of instances in the syncing account is less than the allowed limit of instances for SSM send_command. Is it possible to sync the inventory in a batch with the option to apply the count of instances so that we can use it to keep it under the allowed limit of instances for SSM send_command?

[vijayreddiar]

I see a fix is merged through #1957
Now, we don't see failures in inventory sync when the count of EC2 instances in an account is more than the allowed limit of instances (default value is 40) for SSM send_command, but no host variables are fetched through SSM now. The scenario is same even if the count of instances in the account is lesser than 40 - which means the parameter use_ssm_inventory is not at all effective now.

[gravesm]

@vijayreddiar Please open a new issue for this.

[vijayreddiar]

@vijayreddiar Please open a new issue for this.

@gravesm I have raised a new issue. Thanks.