EC2 instance tags do not populate ec2_tag_* hostvars

[nethershaw]

SUMMARY

Brevity is not going to be possible. I need to take you through how I got here because that's half the problem.

Please forgive me if I have missed something in the process of filing this report. Everything about the amazon.aws collection appears to still be in an incomplete state of transition; the documentation, examples of working options, and even the plugin's fundamental capabilities are recorded in different places and I have been forced to synthesize what I know from these disjoint sets. There are still gaps.

I am trying to get the aws_ec2 inventory source plugin from this repository to behave exactly as the legacy ec2.py inventory script did with respect to resource tags on EC2 instances, such that any resource tag given as "key:value" in EC2 is recorded as "tag_key_value" in the instance's hostvars. This, of course, does not happen out of the box as it does with the legacy inventory script, and the examples found in this plugin's documentation scarcely begin to approximate feature parity.

The first stop on my adventure to discover how to bend the plugin back toward its legacy behavior led me to a specific commit of a file that no longer exists in the ansible/awx repository. This configuration document was written as someone's attempt at testing whether the inventory plugin provides most of the groups and hostvars that the legacy ec2.py inventory script did. It gets remarkably close. This example should be included in the plugin's documentation -- if someone has been using ec2.py until Ansible 2.10 and needs to migrate to the inventory plugin, this is unquestionably what they want. That should be a separate documentation issue, yes, but I'm only describing all of this once.

You will notice the following in that example:

compose:
  ec2_tag_Name: tags.Name

This does resolve a single tag, the Name tag, from every instance and encodes it as ec2.py would have. Only that tag. Here's the problem: I need it to do that in the general case, for all resource tags on the instance. I can't know what all the keys are going to be a priori so I can't explicitly account for all of them in compose. The developers on my team have 157 locations throughout our repository where we depend on a variable of the form "ec2_tag_key" to directly reference information about the target host's tags and retrofitting all of them to something else would create a proper mess, so I'm back to seeking parity.

In other words, I need this.

I can't get my team off of 2.9 and onto 2.10 until I close that gap, or any playbook where we have to configure a newly launched instance based on its assigned tags will quickly fail. Getting from A to B for me is not optional.

I am going to try to use something like this:

compose:
  ec2_tags: tags

This will create a dictionary at ec2_tags in the instance's hostvars with all of the instance's resource tag key-value pairs as descendants just so that they're available somewhere, and maybe I can put some Jinja2 magic in group_vars/all/vars.yml to flatten that dictionary, but I haven't had much luck before with inline templates in vars files, especially when they have to compose variable names, and especially especially when they have to use a control structure like a loop.

That this was released to stable without being a drop-in replacement for ec2.py is ... the most polite word I can use is "frustrating."

If anyone has a better idea, please let me know. Thanks.

ISSUE TYPE

• Bug Report

COMPONENT NAME

amazon.aws.aws_ec2 (EC2 Inventory Source)

ANSIBLE VERSION

ansible 2.10.2
  config file = /home/mvaughn/Documents/git/ansible/ansible.cfg
  configured module search path = ['/home/mvaughn/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/lib/python-exec/python3.7/ansible
  python version = 3.7.9 (default, Oct 27 2020, 10:17:31) [GCC 9.3.0]

CONFIGURATION

ANSIBLE_PIPELINING(/home/mvaughn/Documents/git/ansible/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/home/mvaughn/Documents/git/ansible/ansible.cfg) = -C -o ControlMaster=auto -o ControlPersist=1800s
ANSIBLE_SSH_CONTROL_PATH(/home/mvaughn/Documents/git/ansible/ansible.cfg) = %(directory)s/%%h-%%p-%%r
ANSIBLE_SSH_CONTROL_PATH_DIR(/home/mvaughn/Documents/git/ansible/ansible.cfg) = ~/.ansible/cp
CACHE_PLUGIN(/home/mvaughn/Documents/git/ansible/ansible.cfg) = redis
CACHE_PLUGIN_CONNECTION(/home/mvaughn/Documents/git/ansible/ansible.cfg) = localhost:6379:0
COMMAND_WARNINGS(/home/mvaughn/Documents/git/ansible/ansible.cfg) = True
DEFAULT_CALLBACK_WHITELIST(/home/mvaughn/Documents/git/ansible/ansible.cfg) = ['datadog_callback']
DEFAULT_FORKS(/home/mvaughn/Documents/git/ansible/ansible.cfg) = 16
DEFAULT_GATHER_TIMEOUT(/home/mvaughn/Documents/git/ansible/ansible.cfg) = 30
DEFAULT_HOST_LIST(env: ANSIBLE_INVENTORY) = ['/home/mvaughn/Documents/git/ansible/inventory/dev.aws_ec2.yml']
DEFAULT_LOG_PATH(/home/mvaughn/Documents/git/ansible/ansible.cfg) = /var/log/ansible.log
DEFAULT_MANAGED_STR(/home/mvaughn/Documents/git/ansible/ansible.cfg) = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
DEFAULT_REMOTE_USER(/home/mvaughn/Documents/git/ansible/ansible.cfg) = ansible
DEFAULT_VAULT_IDENTITY_LIST(/home/mvaughn/Documents/git/ansible/ansible.cfg) = ['ansible/vault/token@contrib/vault/vault-aws-sm-client.py']
HOST_KEY_CHECKING(/home/mvaughn/Documents/git/ansible/ansible.cfg) = False
INVENTORY_ENABLED(/home/mvaughn/Documents/git/ansible/ansible.cfg) = ['amazon.aws.aws_ec2', 'host_list', 'script', 'auto', 'yaml', 'ini', 'toml']

OS / ENVIRONMENT

Somewhere on a Linux machine, talking to the EC2 API.

STEPS TO REPRODUCE

1. Configure aws_ec2 as an inventory source.
2. List inventory.

ansible-inventory --list

EXPECTED RESULTS

EC2 instance resource tags of the form "key: value" must appear as vars of the form "ec2_tag_key: value" in each instance's hostvars. All of them.

E.g., if an instance with the hostname "myhost" has the EC2 resource tags "foo: bar" and "baz: qux" assigned to it, the instance's hostvars must reflect them:

{
	"_meta": {
		"hostvars": {
			"myhost": {
				"another_var": "from_elsewhere",
				...elided for brevity...
				"ec2_tag_foo": "bar",
				"ec2_tag_baz": "qux",
				...elided for brevity...
				"still_more_vars": "not_from_tags"
			},
			"myotherhost": {
				"you_get_the_idea": true
			}
		}
	}
}

ACTUAL RESULTS

Same thing, without the ec2_tag_* keys.

{
	"_meta": {
		"hostvars": {
			"myhost": {
				"another_var": "from_elsewhere",
				...elided for brevity...
				"still_more_vars": "not_from_tags"
			},
			"myotherhost": {
				"you_get_the_idea": true
			}
		}
	}
}

[next-jesusmanuelnavarro]

I feel your pain, I found the same problem when trying to gain access to tags I previously set.

You can use the ec2_metadata_facts module in a task before you need to access one of the EC2 host's facts. I.e.:

  - name: Grabs AWS instances data
    ec2_metadata_facts:

  - name: Shows the host's name on the delegated domain
    debug:
      msg: "Service hostname is: '{{ hostvars[inventory_hostname].tags.Name }}.{{ delegated_dns_domain }}'"

You possibly should output all facts after running ec2_metadata_facts to learn which/where they appear.

[jerm]

Until this is implemented, I've been working around it by having this as a pre-task in my playbook.

   - name: Set up tag facts
     set_fact: {"ec2_tag_{{item.key}}": "{{ item.value }}"}
     with_dict: "{{tags}}"
     tags: always

Problem is you have to include it for EVERY play, so no, skeptical reader, it's not a solution, it's a band-aid 😜

[goneri]

Hi all! It would be great if you can take the time to test #331 .