tests: Randomise VPC/subnet CIDRs and fix elb_target timing issues#2457
Merged
tremble merged 16 commits intoJun 19, 2026
Conversation
Add setup_ec2_instance_env and setup_ec2_vpc roles from amazon.aws and use
setup_ec2_instance_env for VPC setup/cleanup with randomised CIDR allocation.
Add Lambda state polling and load balancer ENI wait to prevent timing failures.
Fixes three root causes identified in PR #3000 CI analysis:
1. ENI Cleanup Timing: Load balancers create ENIs that take 1-5 minutes
to fully delete after the load balancer is removed. The test was
attempting subnet deletion 2-6 seconds after load balancer deletion,
causing DependencyViolation errors. Now explicitly waits for all
load balancer ENIs to be deleted before proceeding with VPC cleanup.
The setup_ec2_instance_env role then handles any remaining ENIs.
2. Lambda State Polling: Lambda functions need time to transition to
Active state after creation (typically 5-10 seconds). The test was
attempting ELB registration only 3 seconds after creation, causing
"not in Active state" errors. Now polls lambda_info until state is
Active before proceeding with registration.
3. Hardcoded CIDR Allocation: The test used 20.0.0.0/16 which is not
in the private address ranges and can cause conflicts. Now uses
randomised 10.x.0.0/16 CIDRs based on resource_prefix following
amazon.aws patterns.
The setup_ec2_instance_env role provides comprehensive VPC setup/cleanup:
- Randomised CIDR allocation (10.{{ 256 | random(seed=vpc_seed) }}.0.0/16)
- VPC, subnets (in 2 AZs), security groups creation
- Instance discovery and termination
- ENI discovery and deletion with retries
- Security group rule cleanup before deletion
- Subnet deletion with retries
- VPC deletion with retries
The setup_ec2_vpc role provides VPC cleanup only (for tests that do
manual VPC setup but want standardised cleanup).
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Update Glue versions from deprecated 2.0/0.9 to currently supported 4.0/5.0. AWS Glue retired older versions (< 3.0), causing CreateJob API calls to fail with "The Glue version selected for the job is no longer available. Select a newer Glue version." Changes: - Replace glue_version "2.0" with "4.0" (stable LTS version) - Replace glue_version "0.9" with "5.0" (latest version for update tests) Fixes glue_job test failure in PR #3000 CI run. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace hardcoded 20.0.0.0/16 CIDRs with randomised 10.x.0.0/16 allocation
across elb_target, elb_target_info tests to prevent conflicts.
Updates:
- elb_target/alb_target.yml: Use setup_ec2_instance_env for VPC setup/cleanup
- elb_target/etg_protocol_version.yml: Update to randomised CIDR
- elb_target_info: Use setup_ec2_instance_env for VPC setup/cleanup
All tests now use the pattern:
cidr_block: "10.{{ 256 | random(seed=resource_prefix) }}.0.0/16"
This follows amazon.aws patterns and ensures each test run gets a unique
CIDR range, preventing VPC conflicts when tests run in parallel on the
same AWS account.
Additionally adds proper ENI cleanup wait and uses setup_ec2_instance_env
role for comprehensive VPC cleanup in alb_target and elb_target_info tests.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add randomised VPC CIDR allocation to test defaults following amazon.aws
patterns to prevent conflicts when tests run in parallel.
Updates defaults/main.yml for:
- autoscaling_launch_config: 10.x.32.0/23 with two /24 subnets
- autoscaling_scheduled_action: 10.x.77.0/24
- dms_replication_subnet_group: 10.x.32.0/23 with two /28 subnets
- efs: 10.x.32.0/23 with two /24 subnets
- eks_cluster: 10.x.0.0-2.0/24 (three zones)
- eks_nodegroup: 10.x.0.0-1.0/24 (two zones)
- elb_network_lb: 10.x.228.0/22 with three /24 subnets
- mq: 10.x.0.0/16 with one /24 subnet
All use pattern: 10.{{ 256 | random(seed=resource_prefix) }}.y.z/prefix
Also updates env_setup.yml and env_cleanup.yml for autoscaling_launch_config
to reference the new variables instead of hardcoded CIDRs.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Update task files to use CIDR variables from defaults instead of hardcoded
values for autoscaling_scheduled_action, dms_replication_subnet_group,
ecs_cluster, efs, and elb_network_lb tests.
Also adds missing subnet_a_cidr and subnet_b_cidr variables to ecs_cluster
defaults/main.yml.
All tasks now reference {{ vpc_cidr }}, {{ subnet_a_cidr }}, etc. instead
of hardcoded 10.x.y.z/prefix values.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Complete the systematic replacement of hardcoded CIDRs with randomised
allocation for all remaining tests:
- elb_network_lb: Add subnet_b_cidr and subnet_d_cidr for all four subnets
- glue_connection: Create defaults/main.yml with vpc_cidr and subnet_cidr
- opensearch: Create defaults/main.yml with vpc_cidr and two subnet CIDRs
- route53_wait: Create defaults/main.yml, change from 192.0.2.0/24
(TEST-NET-1 reserved range) to randomised 10.x allocation
- wafv2: Create defaults/main.yml with vpc_cidr and four subnet CIDRs
All integration tests now use randomised CIDR allocation following the
amazon.aws pattern: 10.{{ 256 | random(seed=resource_prefix) }}.y.z/prefix
This ensures unique network ranges for each test run, preventing conflicts
when tests run in parallel on the same AWS account.
Summary of changes:
- 20 test targets updated with randomised CIDRs
- All hardcoded private IP ranges eliminated
- Consistent pattern across all tests
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fix remaining hardcoded CIDRs in integration tests: - autoscaling_lifecycle_hook: Add vpc_cidr and subnet_cidr to role defaults, update env_setup.yml and env_cleanup.yml to use variables - eks_fargate_profile: Add vpc_cidr_base to defaults, update eks_subnets to use randomised allocation matching eks_cluster and eks_nodegroup pattern - elasticache: Update security group rule to use existing vpc_cidr_prefix instead of hardcoded 10.31.0.0/16 - elb_network_lb: Update security group rule to use vpc_cidr variable instead of hardcoded 10.228.228.0/22 All integration tests now use consistent randomised CIDR allocation. Zero hardcoded CIDRs remain in integration tests (unit tests intentionally use mock data and are unchanged). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace hardcoded 10.228.228.0/22 CIDRs with randomised allocation using
vpc_cidr_base variable. All subnet CIDRs now use the same random base.
Pattern: 10.{{ 256 | random(seed=resource_prefix) }}.228.y/z
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace hardcoded 192.168.178.32 with randomised test_target_ip variable
for IP-type target group testing.
Uses pattern: 192.168.{{ 256 | random(seed=resource_prefix) }}.32
This ensures the test IP address is unique per test run, avoiding potential
conflicts with actual infrastructure using that specific IP address.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Spread CIDR third octets from 252 down in decrements of 4
(252, 248, 244, ..., 160) to reduce collision probability
while maintaining randomised second octet.
Mirrors amazon.aws pattern which spreads from 0 up by 4s.
Previous distribution clustered around 0-3, 32-33, 77, 228-231.
Keeps {{ 256 | random(seed=resource_prefix) }} for second octet
as belt-and-braces protection against conflicts.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add vpc_cidr to elb_target defaults for etg_protocol_version test - Fix elb_network_lb subnet allocation to use all four distinct CIDRs (was incorrectly reusing subnet_b_cidr for both private-a and public-b) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Update ansible_test_splitter and ansible_read_targets actions to use the same commit hash as amazon.aws (6659b608), which includes the fix for correctly ignoring infrastructure roles in test target detection. This prevents the CI from attempting to run setup_ec2_instance_env and setup_ec2_vpc as test targets when they are infrastructure roles. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add setup_ec2_instance_env and setup_ec2_vpc as test dependencies so ansible-test can find the roles when running the integration tests. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Document test CIDR randomisation, infrastructure role additions, and various test fixes in changelog fragment. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Contributor
|
Build succeeded. ❌ ansible-galaxy-importer RETRY_LIMIT Host unreachable in 6m 52s (non-voting) |
tremble
marked this pull request as ready for review
June 16, 2026 14:58
Contributor
|
Build succeeded. ✔️ ansible-galaxy-importer SUCCESS in 4m 19s (non-voting) |
Replace manual subnet CIDR calculations with ansible.utils.ipsubnet
filter for consistency with amazon.aws pattern. Changes:
- Convert hardcoded subnet CIDRs to {{ vpc_cidr | ansible.utils.ipsubnet(size, index) }}
- Apply subnet_X_startswith pattern using {{ (subnet_X_cidr.split('.')[:3] + ['']) | join('.') }}
- Update networkfirewall to use /23 VPCs at 156.0 and 158.0 with ipsubnet
- Remove cidr_prefix pattern in favour of standard vpc_cidr definitions
- Start all subnet allocations at index 0 for consistency
This eliminates manual third octet incrementing and makes subnet
allocation more maintainable across 23 integration test targets.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
Replace regex_replace approach with ansible.utils.ipsubnet filter to properly calculate IPv6 /64 subnet CIDRs from VPC /56 CIDR blocks. The original regex_replace generated invalid IPv6 notation like '2600:1f18:3828:4e00:0::/64' (double :: not allowed). The ipsubnet filter correctly generates '2600:1f18:3828:4e00::/64', '2600:1f18:3828:4e01::/64', etc. Split VPC IPv6 fact definition into separate task to avoid circular dependency when deriving subnet CIDRs. Mirrors fix from amazon.aws commit 71b549a. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
alinabuzachis
approved these changes
Jun 19, 2026
github-actions
Bot
added
the
backport-11
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Contributor
Author
|
The importer test had broken because the configuration to ensure we test using the amazon.aws pre-release (main branch) configuration was accidentally dropped. Restored by ansible/zuul-config#647 |
Contributor
|
Build succeeded (gate pipeline). ✔️ ansible-galaxy-importer SUCCESS in 3m 59s (non-voting) |
Contributor
|
Pull request merge failed: Resource not accessible by integration, You may need to manually rebase your PR and retry. |
Backport to stable-11: 💚 backport PR created✅ Backport PR branch: Backported as #2461 🤖 @patchback |
centosinfra-prod-github-app Bot
pushed a commit
that referenced
this pull request
Jun 19, 2026
…2457) (#2461) This is a backport of PR #2457 as merged into main (e1e0b6d). SUMMARY Fix elb_target test timing issues causing CI failures and systematically randomise VPC/subnet CIDR allocations across integration tests to prevent collisions when running in parallel. Related to #3000 (CI failures analysis) ISSUE TYPE Bugfix Pull Request Test Pull Request COMPONENT NAME elb_target integration tests Integration test infrastructure (VPC/subnet CIDR allocation) Update Integration test GitHub Action to use the latest upstream version ADDITIONAL INFORMATION test fixes ENI Cleanup Timing: Load balancers create ENIs that take 1-5 minutes to fully delete after the load balancer is removed. The test was attempting subnet deletion 2-6 seconds after load balancer deletion, causing DependencyViolation errors. Now explicitly waits for all load balancer ENIs to be deleted before proceeding with VPC cleanup. Lambda State Polling: Lambda functions need to reach Active state before they can be registered with ELB. Added polling to wait for Active state before registration. Infrastructure Setup: Migrated to setup_ec2_instance_env role (copied from amazon.aws) for comprehensive VPC setup/cleanup with ENI handling. Glue Versions: Updated glue_job tests from deprecated versions (0.9, 2.0) to supported versions (4.0, 5.0). CIDR randomisation Tests were using hardcoded or clustered CIDRs (10.X.{0,1,2,3}.0), inreasing collision probability when running in parallel on shared AWS accounts. Changes: Randomise VPC and subnet CIDRs across 24 integration tests Pattern: 10.{{ 256 | random(seed=resource_prefix) }}.Y.0/Z Distribute third octets from 252 down in decrements of 4 (252, 248, 244, ..., 156) Mirrors amazon.aws pattern (which uses 0 up by 4s) from opposite end of range Second octet randomisation ({{ 256 | random(seed=...) }}) provides belt-and-braces protection Migrate all CIDR definitions from task files into defaults files Task files now reference CIDRs via variables only ({{ vpc_cidr }}, {{ subnet_a_cidr }}) Change VPC size from /16 to /22 (more appropriate for test environments) All VPC CIDRs validated to properly encompass their subnet CIDRs Tests affected: autoscaling_, dms_replication_subnet_group, ecs_cluster, efs, eks_, elasticache*, elb_, glue_connection, mq, msk_cluster, opensearch, redshift_subnet_group, route53_wait, setup_ec2_instance_env, wafv2* Assisted-by: Claude Sonnet 4.5 Reviewed-by: Mark Chappell
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUMMARY
Fix elb_target test timing issues causing CI failures and systematically randomise VPC/subnet CIDR allocations across integration tests to prevent collisions when running in parallel.
Related to #3000 (CI failures analysis)
ISSUE TYPE
COMPONENT NAME
ADDITIONAL INFORMATION
test fixes
ENI Cleanup Timing: Load balancers create ENIs that take 1-5 minutes to fully delete after the load balancer is removed. The test was attempting subnet deletion 2-6 seconds after load balancer deletion, causing DependencyViolation errors. Now explicitly waits for all load balancer ENIs to be deleted before proceeding with VPC cleanup.
Lambda State Polling: Lambda functions need to reach Active state before they can be registered with ELB. Added polling to wait for Active state before registration.
Infrastructure Setup: Migrated to setup_ec2_instance_env role (copied from amazon.aws) for comprehensive VPC setup/cleanup with ENI handling.
Glue Versions: Updated glue_job tests from deprecated versions (0.9, 2.0) to supported versions (4.0, 5.0).
CIDR randomisation
Tests were using hardcoded or clustered CIDRs (10.X.{0,1,2,3}.0), inreasing collision probability when running in parallel on shared AWS accounts.
Changes:
10.{{ 256 | random(seed=resource_prefix) }}.Y.0/Z{{ 256 | random(seed=...) }}) provides belt-and-braces protection{{ vpc_cidr }},{{ subnet_a_cidr }})Tests affected: autoscaling_, dms_replication_subnet_group, ecs_cluster, efs, eks_, elasticache*, elb_, glue_connection, mq, msk_cluster, opensearch, redshift_subnet_group, route53_wait, setup_ec2_instance_env, wafv2*
Assisted-by: Claude Sonnet 4.5