Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Testing - TestDockerStateToContainerState flaky #1899

Closed
suneyz opened this issue Feb 27, 2019 · 1 comment
Closed

Testing - TestDockerStateToContainerState flaky #1899

suneyz opened this issue Feb 27, 2019 · 1 comment
Assignees
@fenxiong
Labels
kind/testing

Comments

@suneyz
Copy link
Contributor

suneyz commented Feb 27, 2019

Summary

TestDockerStateToContainerState failed intermittently.

Description

=== RUN   TestDockerStateToContainerState
2019-02-27T02:29:04Z [INFO] Image excluded from cleanup: amazon/amazon-ecs-agent:latest
2019-02-27T02:29:04Z [INFO] Image excluded from cleanup: amazon/amazon-ecs-pause:0.1.0
2019-02-27T02:29:04Z [WARN] Unable to fetch user data: blackholed
2019-02-27T02:29:04Z [CRITICAL] Unable to communicate with EC2 Metadata service to infer region: blackholed
2019-02-27T02:29:04Z [CRITICAL] Configuration key not set, key: AWSRegion
2019-02-27T02:29:04Z [INFO] Task engine [test_task]: pulling image 127.0.0.1:51670/amazon/amazon-ecs-netkitten:latest for container netcat concurrently
2019-02-27T02:29:04Z [INFO] Task engine [test_task]: recording timestamp for starting image pulltime: 2019-02-27 02:29:04.891014276 +0000 UTC m=+204.493824478
2019-02-27T02:29:04Z [INFO] Adding image name- 127.0.0.1:51670/amazon/amazon-ecs-netkitten:latest to Image state- sha256:3fa2bb1695db7989ffe173baaa9bc6efebf5cc80a353f6891619c50bd40b6ea0
2019-02-27T02:29:04Z [INFO] Updating container reference netcat in Image State - sha256:3fa2bb1695db7989ffe173baaa9bc6efebf5cc80a353f6891619c50bd40b6ea0
2019-02-27T02:29:04Z [INFO] Task engine [test_task]: finished pulling image 127.0.0.1:51670/amazon/amazon-ecs-netkitten:latest for container netcat in 9.630511ms
2019-02-27T02:29:04Z [INFO] Task engine [test_task]: creating container: netcat
2019-02-27T02:29:04Z [INFO] Task engine [test_task]: created container name mapping for task:  netcat -> ecs-family-1-netcat-cc8480a9a1c1f6acf501
2019-02-27T02:29:05Z [INFO] Task engine [test_task]: created docker container for task: netcat -> faea220e1f818a573da361eb57b37a662934c68edc2bb40e4f30a6a3e8f8700c
2019-02-27T02:29:05Z [INFO] Task engine [test_task]: created docker container for task: netcat -> faea220e1f818a573da361eb57b37a662934c68edc2bb40e4f30a6a3e8f8700c, took 231.992877ms
2019-02-27T02:29:05Z [INFO] Task engine [test_task]: starting container: netcat
2019-02-27T02:29:06Z [INFO] Task engine [test_task]: started docker container for task: netcat -> faea220e1f818a573da361eb57b37a662934c68edc2bb40e4f30a6a3e8f8700c, took 1.142583523s
2019-02-27T02:29:06Z [CRITICAL] Task engine: could not find managed task [test_task] corresponding to a docker event: Status: RUNNING, DockerID: faea220e1f818a573da361eb57b37a662934c68edc2bb40e4f30a6a3e8f8700c, health: UNKNOWN, ExitCode: 0, Labels: map[com.amazonaws.ecs.cluster: com.amazonaws.ecs.container-name:netcat com.amazonaws.ecs.task-arn:test_task com.amazonaws.ecs.task-definition-family:family com.amazonaws.ecs.task-definition-version:1], CreatedAt: 2019-02-27 02:29:04.902801246 +0000 UTC, StartedAt: 2019-02-27 02:29:05.44513855 +0000 UTC, FinishedAt: 2019-02-27 02:29:05.475656586 +0000 UTC
2019-02-27T02:29:06Z [INFO] Task engine [test_task]: stopping container [netcat]
2019-02-27T02:29:06Z [CRITICAL] Task engine: could not find managed task [test_task] corresponding to a docker event: Status: STOPPED, DockerID: faea220e1f818a573da361eb57b37a662934c68edc2bb40e4f30a6a3e8f8700c, health: UNKNOWN, ExitCode: 0, Labels: map[com.amazonaws.ecs.task-definition-version:1 com.amazonaws.ecs.cluster: com.amazonaws.ecs.container-name:netcat com.amazonaws.ecs.task-arn:test_task com.amazonaws.ecs.task-definition-family:family], CreatedAt: 2019-02-27 02:29:04.902801246 +0000 UTC, StartedAt: 2019-02-27 02:29:05.44513855 +0000 UTC, FinishedAt: 2019-02-27 02:29:05.475656586 +0000 UTC
2019-02-27T02:29:06Z [INFO] Task engine [test_task]: removing container: netcat
2019-02-27T02:29:06Z [INFO] Task engine [test_task2]: creating container: netcat
2019-02-27T02:29:06Z [INFO] Task engine [test_task2]: created container name mapping for task:  netcat -> ecs-family-1-netcat-f4c0b295a4e8e099ec01
2019-02-27T02:29:07Z [INFO] Task engine [test_task2]: created docker container for task: netcat -> 483857d327de088210460cab7599d5a532d5c726f27ff9a540a98ace4c7a29b9
2019-02-27T02:29:07Z [INFO] Task engine [test_task2]: created docker container for task: netcat -> 483857d327de088210460cab7599d5a532d5c726f27ff9a540a98ace4c7a29b9, took 268.027248ms
2019-02-27T02:29:07Z [INFO] Task engine [test_task2]: starting container: netcat
2019-02-27T02:29:07Z [INFO] Task engine [test_task2]: started docker container for task: netcat -> 483857d327de088210460cab7599d5a532d5c726f27ff9a540a98ace4c7a29b9, took 511.604322ms
2019-02-27T02:29:07Z [INFO] Task engine [test_task2]: removing container: netcat
--- FAIL: TestDockerStateToContainerState (3.14s)
	<autogenerated>:1: 
			Error Trace:	engine_integ_test.go:135
			Error:      	Not equal: 
			            	expected: 3
			            	actual  : 5
			Test:       	TestDockerStateToContainerState

Expected Behavior

Observed Behavior

Environment Details

Supporting Log Snippets
@suneyz suneyz changed the title Testing - TestDockerStateToContainerState Testing - TestDockerStateToContainerState flaky Feb 27, 2019
@suneyz suneyz mentioned this issue Feb 27, 2019
Merged
@ubhattacharjya ubhattacharjya mentioned this issue Feb 28, 2019
Merged
@fenxiong fenxiong self-assigned this Mar 6, 2019
fenxiong added a commit to fenxiong/amazon-ecs-agent that referenced this issue Mar 6, 2019
@fenxiong
Fix TestDockerStateToContainerState integ test.
d09e73d 
During the workflow of TestDockerStateToContainerState, it starts a container and checks whether it's running by inspecting it. However, the container it starts exit immediately, so it's possible that when we check whether the container is running it's already stopped. This leads to flakey test as per aws#1899. Fixing by letting the container keep running rather than exit itself.
fenxiong added a commit to fenxiong/amazon-ecs-agent that referenced this issue Mar 6, 2019
@fenxiong
Fix TestDockerStateToContainerState integ test.
1b6a8d3 
During the workflow of TestDockerStateToContainerState, it starts a container and checks whether it's running by inspecting it. However, the container it starts exit immediately, so it's possible that when we check whether the container is running it's already stopped. This leads to flakey test as per aws#1899. Fixing by letting the container keep running rather than exit itself.
@fenxiong fenxiong mentioned this issue Mar 6, 2019
Merged
fenxiong added a commit to fenxiong/amazon-ecs-agent that referenced this issue Mar 6, 2019
@fenxiong
Fix TestDockerStateToContainerState integ test.
76062cb 
During the workflow of TestDockerStateToContainerState, it starts a container and checks whether it's running by inspecting it. However, the container it starts exit immediately, so it's possible that when we check whether the container is running it's already stopped. This leads to flakey test as per aws#1899. Fixing by letting the container keep running rather than exit itself.
fenxiong added a commit that referenced this issue Mar 7, 2019
@fenxiong
Fix TestDockerStateToContainerState integ test.
08d3eaf 
During the workflow of TestDockerStateToContainerState, it starts a container and checks whether it's running by inspecting it. However, the container it starts exit immediately, so it's possible that when we check whether the container is running it's already stopped. This leads to flakey test as per #1899. Fixing by letting the container keep running rather than exit itself.
@fenxiong
Copy link
Contributor

fenxiong commented Mar 7, 2019

#1926 fixes this. closing

@fenxiong fenxiong closed this as completed Mar 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fenxiong fenxiong

Labels
kind/testing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@suneyz @fenxiong