[DHPA] Upudate dockerPortMap() in task.go with dynamic host port range support - part 1 #3584

chienhanlin · 2023-02-21T01:10:55Z

Summary

DHPA - Dynamic Host Port Assignment

The target branch of this PR is feature/dynamicHostPortAssignment.

This is a follow-up PR for [DHPA] Add GetHostPort() and update unit tests to utilize function GetHostPort() in dockerPortMap() to construct a docker port map for an application container with the user-specified container port but without an user-defined host port running in bridge network mode.

Comparing to the current dockerPortMap(), main changes in this PR include:

Use GetHostPort() to get a host port from the given dynamicHostPortRange when (a) it's a bridge network mode task and (b) there is the defined container port but no defined host in the port mapping configured by customers. With this change, ECS Agent will be the single source of truth for host port assignment. We will not fall back to Docker dynamic host port assignment if no host port can be found by ECS Agent within the given dynamicHostPortRange.

{
  "networkMode": "bridge",
  "containerDefinitions": [
    {
      "portMappings": [
        {
          "protocol": "tcp",
          "containerPort": 100,  
        }
      ],
      "image": "...",
      "name": "..."
    }
  ]
}

Note that dynamicHostPortRange can be configured by customers using ECS Agent environment variable ECS_DYNAMIC_HOST_PORT_RANGE; if the customized value is not provided, ECS Agent will use the default value returned from GetDynamicHostPortRange().

Update test cases in TestDockerHostConfigPortBinding to validate changes made in this PR
Make function PortIsInRange() and VerifyPortsWithinRange() public
Add dynamicHostPortRange to String() for Agent config.go

Implementation details

agent/api/task/task.go

Update dockerPortMap() to include ECS Agent dynamic host port assignment for a singular port case
Update comments for the function

agent/api/task/task_test.go

Update test cases in TestDockerHostConfigPortBinding to test changes

agent/utils/ephemeral_ports.go

Move VerifyPortsWithinRange() and PortIsInRange() from the test file and make them public for reusing them in task_test.go

agent/utils/ephemeral_ports_test.go

Add unit tests for VerifyPortsWithinRange() and PortIsInRange()

agent/config/config.go

Add dynamicHostPortRange to function String()

level=debug time=xxx msg="Loaded config: Cluster: default,  ..., DynamicHostPortRange: 50000-50010, PauseContainerImageName: amazon/amazon-ecs-pause, PauseContainerTag: 0.1.0" module=agent.go

Testing

New tests cover the changes: yes

Unit test

TestDockerHostConfigPortBinding

--- PASS: TestDockerHostConfigPortBinding (0.00s)
    --- PASS: TestDockerHostConfigPortBinding/user-specified_container_ports_and_host_ports (0.00s)
    --- PASS: TestDockerHostConfigPortBinding/user-specified_container_ports_with_a_ideal_dynamicHostPortRange (0.00s)
    --- PASS: TestDockerHostConfigPortBinding/user-specified_container_ports_with_a_bad_dynamicHostPortRange (0.00s)
    --- PASS: TestDockerHostConfigPortBinding/user-specified_container_port_and_container_port_range_with_a_ideal_dynamicHostPortRange (0.00s)
    --- PASS: TestDockerHostConfigPortBinding/user-specified_container_port_and_container_port_range_with_a_bad_user-specified_dynamicHostPortRange (0.00s)

TestPortIsInRange

--- PASS: TestPortIsInRange (0.00s)
    --- PASS: TestPortIsInRange/in_the_range (0.00s)
    --- PASS: TestPortIsInRange/not_in_the_range (0.00s)

TestVerifyPortsWithinRange

--- PASS: TestVerifyPortsWithinRange (0.00s)
    --- PASS: TestVerifyPortsWithinRange/in_the_expected_range (0.00s)
    --- PASS: TestVerifyPortsWithinRange/not_in_the_expected_range (0.00s)

Manual test

Setup

An ECS cluster with only one registered EC2 instance, which is launched with ECS optimized AMI: amzn2-ami-ecs-hvm-2.0.20230127-x86_64-ebs
Install AWS CLI version aws-cli/2.10.0 Python/3.9.11 Linux/4.14.301-224.520.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off on the host
Run ecs describe-tasks to get the actual networkBindings of the task container
Check the default ephemeral port range for Docker version 1.6.0 and later listed on the instance under /proc/sys/net/ipv4/ip_local_port_range

[ec2-user@ip-xxx]$ cat /proc/sys/net/ipv4/ip_local_port_range
32768   60999

Test cases

Case	ECS_DYNAMIC_HOST_PORT_RANGE	Port mappings in task def	Result	Note
1	no user-specified value	containerPortRange 8080-8084	hostPortRange 332768-32772	bound to host ports in default range
2	invalid value	containerPortRange 8080-8084	hostPortRange 32768-32772	bound to host ports in default range
3	50000-50010	containerPortRange 8080-8084	hostPortRange 50000-50004	bound to host ports in user-specified range
4	50000-50001	containerPortRange 8080-8084	HostConfigError: error retrieving docker port map 5 contiguous host ports are unavailable	the user-specified dynamic host port range is not enough to cover required # of host ports
5	no user-specified value	containerPort 8080	hostPort 32768	bound to a host port in default range
6	invalid value	containerPort 8080	hostPort 32768	bound to a host port in default range
7	50000-50010	containerPort 8080	hostPort 50005	bound to a host port in user-specified range
8	50000-50001	containerPort 8080	HostConfigError: error retrieving docker port map: a host port is unavailable	host port 50000 and 50001 were both unavailable
9	no user-specified value	containerPort 8080 and hostPort 9000	hostPort 9000	bound to the user-specified host port
10	50000-50010	containerPort 8080 and hostPort 9000	hostPort 9000	bound to the user-specified host port

Description for the changelog

[Enhancement] Support user-specified dynamic host port range for a singular port - part 1

Related PRs

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…rt part 1

chienhanlin · 2023-02-21T06:50:34Z

agent/api/task/task.go

+				// If there is no user-specified host port, ECS Agent will find an available host port
+				// within the given dynamic host port range. And if no host port is available within the range,
+				// an error will be returned.
+				logger.Debug("No user-specified host port, ECS Agent will find an available host port within the given dynamic host port range", logger.Fields{


An example ecs-agent.log for line 2415 - 2427

level=info time=xxx msg="Creating container" task=“xxx” container="containerPort" level=debug time=xxx msg="No user-specified host port, ECS Agent will find an available host port within the given dynamic host port range" container="containerPort" dynamicHostPortRange="50000-50001" level=debug time=xxx msg="Port 50000 is unavailable or an error occurred while listening on the local tcp network" module=ephemeral_ports.go level=debug time=xxx msg="Port 50001 is unavailable or an error occurred while listening on the local tcp network" module=ephemeral_ports.go level=error time=xxx msg="Unable to find a host port for container within the given dynamic host port range" task=“xxx” container="containerPort" dynamicHostPortRange="50000-50001" error="a host port is unavailable" level=error time=xxx msg="Error transitioning container" runtimeID="" nextState="CREATED" error="error retrieving docker port map: a host port is unavailable" task=“xxx” container="containerPort" level=info time=xxx msg="Handling container change event" status="CREATED" task=“xxx” container="containerPort" runtimeID="" level=warn time=xxx msg="Error creating container; marking its desired status as STOPPED" task=“xxx” container="containerPort" error="error retrieving docker port map: a host port is unavailable" … level=debug time=xxx msg="Submitted state change to ECS" eventType="task" eventData="TaskChange: [xxx -> STOPPED, Known Sent: STOPPED, PullStartedAt: xxx, PullStoppedAt: xxx, ExecutionStoppedAt: xxx, container change: xxx containerPort -> STOPPED, Reason HostConfigError: error retrieving docker port map: a host port is unavailable, Known Sent: STOPPED] sent: true"

ubhattacharjya · 2023-02-21T19:19:30Z

agent/utils/ephemeral_ports.go

+
+// VerifyPortsWithinRange returns true if the actualPortRange is within the expectedPortRange;
+// otherwise, returns false.
+func VerifyPortsWithinRange(actualPortRange, expectedPortRange string) bool {


QQ. Is this method used anywhere other than in task_test.go ? If not, could you move it to the test file?

Also do we verify that a port is within range in our taskdef validation or elsewhere? Where is this same validation done outside of our tests?

QQ. Is this method used anywhere other than in task_test.go ? If not, could you move it to the test file?

Also do we verify that a port is within range in our taskdef validation or elsewhere? Where is this same validation done outside of our tests?

This method is used in both agent/utils/ephemeral_ports_test.go and agent/api/task/task_test.go in this PR. As it's using in different packages, I have to make it as a public method.

And as fierlion@ suggested, we should validate a host port or a host port range assigned by ECS Agent after calling getHostPort() or getHostPortRange(). Therefore, I would like to keep VerifyPortsWithinRange() in agent/utils/ephemeral_ports.go, and have a follow up PR to implement host port validations outside of our tests.

Changes are made in https://github.com/aws/amazon-ecs-agent/pull/3589/files#diff-d475c5ecd1be9e633e4fde242318c1369112ce30a3a122288779c5cc277a7b93.

agent/api/task/task.go

fierlion · 2023-02-22T19:33:47Z

agent/utils/ephemeral_ports.go

+
+// VerifyPortsWithinRange returns true if the actualPortRange is within the expectedPortRange;
+// otherwise, returns false.
+func VerifyPortsWithinRange(actualPortRange, expectedPortRange string) bool {


Also do we verify that a port is within range in our taskdef validation or elsewhere? Where is this same validation done outside of our tests?

agent/api/task/task.go

1. Add GetHostPort() and update unit tests #3570 2. Upudate dockerPortMap() in task.go with dynamic host port range support part 1 #3584 3. Upudate dockerPortMap() in task.go with dynamic host port range support part 2 #3589 4. Validate the host port/host port range found by ECS Agent before returning it #3589

1. Add GetHostPort() and update unit tests aws#3570 2. Upudate dockerPortMap() in task.go with dynamic host port range support part 1 aws#3584 3. Upudate dockerPortMap() in task.go with dynamic host port range support part 2 aws#3589 4. Validate the host port/host port range found by ECS Agent before returning it aws#3589 5. Refactor buildPortMapWithSCIngressConfig() in task.go aws#3600

1. Add GetHostPort() and update unit tests #3570 2. Upudate dockerPortMap() in task.go with dynamic host port range support part 1 #3584 3. Upudate dockerPortMap() in task.go with dynamic host port range support part 2 #3589 4. Validate the host port/host port range found by ECS Agent before returning it #3589 5. Refactor buildPortMapWithSCIngressConfig() in task.go #3600

Upudate dockerPortMap() in task.go with dynamic host port range suppo…

339e375

…rt part 1

chienhanlin added the bot/test label Feb 21, 2023

amazon-ecs-bot removed the bot/test label Feb 21, 2023

chienhanlin commented Feb 21, 2023

View reviewed changes

chienhanlin marked this pull request as ready for review February 21, 2023 06:51

chienhanlin requested a review from a team as a code owner February 21, 2023 06:51

This was referenced Feb 21, 2023

Upudate dockerPortMap() in task.go with dynamic host port range support part 2 chienhanlin/amazon-ecs-agent#39

Closed

[DHPA] Upudate dockerPortMap() in task.go with dynamic host port range support - part 2 #3585

Closed

chienhanlin changed the title ~~Upudate dockerPortMap() in task.go with dynamic host port range support - part 1~~ [DHPA] Upudate dockerPortMap() in task.go with dynamic host port range support - part 1 Feb 21, 2023

ubhattacharjya reviewed Feb 21, 2023

View reviewed changes

fierlion reviewed Feb 22, 2023

View reviewed changes

agent/api/task/task.go Show resolved Hide resolved

mye956 reviewed Feb 23, 2023

View reviewed changes

agent/api/task/task.go Show resolved Hide resolved

mye956 approved these changes Feb 23, 2023

View reviewed changes

chienhanlin mentioned this pull request Feb 24, 2023

[DHPA] Validate the host port/host port range found by ECS Agent before returning it chienhanlin/amazon-ecs-agent#40

Closed

fierlion approved these changes Feb 24, 2023

View reviewed changes

YashdalfTheGray approved these changes Feb 24, 2023

View reviewed changes

chienhanlin merged commit 59fa3b7 into aws:feature/dynamicHostPortAssignment Feb 24, 2023

chienhanlin mentioned this pull request Feb 24, 2023

[DHPA] Upudate dockerPortMap() in task.go with dynamic host port range support - part 2+ #3589

Merged

chienhanlin mentioned this pull request Mar 3, 2023

[DHPA] Refactor buildPortMapWithSCIngressConfig() in task.go #3600

Merged

chienhanlin mentioned this pull request Mar 4, 2023

ECS Agent dynamic host port assignment #3601

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DHPA] Upudate dockerPortMap() in task.go with dynamic host port range support - part 1 #3584

[DHPA] Upudate dockerPortMap() in task.go with dynamic host port range support - part 1 #3584

chienhanlin commented Feb 21, 2023 •

edited

Loading

chienhanlin Feb 21, 2023

ubhattacharjya Feb 21, 2023

fierlion Feb 22, 2023

chienhanlin Feb 22, 2023

chienhanlin Feb 24, 2023 •

edited

Loading

fierlion Feb 22, 2023

[DHPA] Upudate dockerPortMap() in task.go with dynamic host port range support - part 1 #3584

[DHPA] Upudate dockerPortMap() in task.go with dynamic host port range support - part 1 #3584

Conversation

chienhanlin commented Feb 21, 2023 • edited Loading

Summary

Implementation details

Testing

Unit test

TestDockerHostConfigPortBinding

TestPortIsInRange

TestVerifyPortsWithinRange

Manual test

Setup

Test cases

Description for the changelog

Related PRs

Licensing

chienhanlin Feb 21, 2023

Choose a reason for hiding this comment

ubhattacharjya Feb 21, 2023

Choose a reason for hiding this comment

fierlion Feb 22, 2023

Choose a reason for hiding this comment

chienhanlin Feb 22, 2023

Choose a reason for hiding this comment

chienhanlin Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

fierlion Feb 22, 2023

Choose a reason for hiding this comment

chienhanlin commented Feb 21, 2023 •

edited

Loading

chienhanlin Feb 24, 2023 •

edited

Loading