Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/ecs exec #2798

Merged
merged 51 commits into from
Jan 21, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
880c298
add TopContainer to dockerapi
fierlion Aug 26, 2020
eb938f6
Rebase against dev branch
fierlion Sep 23, 2020
26ef604
Add ps arguments to docker top to pass in process ID
sharanyad Sep 9, 2020
a3b2a30
Add docker exec create
ubhattacharjya Sep 9, 2020
e6c7460
Add ContainerExecStart to dockerapi
ubhattacharjya Sep 14, 2020
d87d6e7
add ContainerExecInspect to dockerapi
ubhattacharjya Sep 14, 2020
a2009d4
monitor ExecAgent Processes for Exec-enabled tasks
sharanyad Sep 17, 2020
1a6658f
Add integration tests for exec command agent
angelcar Sep 14, 2020
75de50c
Remove unused config.Config argument in initializeExecCommandAgentRes…
angelcar Sep 16, 2020
050cde6
Add exec subdirectory to exec command agent log path
angelcar Sep 16, 2020
a5791c1
Remove ssm-session-logger from exec command agent requirements
angelcar Sep 17, 2020
f78a3ff
Start exec command agent upon container startup
angelcar Sep 17, 2020
fff3e0e
Add integration test for docker exec APIs
ubhattacharjya Sep 18, 2020
6074f8b
Add Container state change events for ManagedAgent.
fierlion Sep 29, 2020
d4bc3e8
Refactor PostUnmarshalTask to abstract away ExecCommandAgent complexity
angelcar Sep 28, 2020
0a56512
Add integ test for ExecCommandAgent start, and monitor
angelcar Oct 6, 2020
98927d3
add jitter between monitoring exec agents for containers
sharanyad Oct 14, 2020
1a92b58
add managedAgent event and update unit test (#2671)
fierlion Oct 21, 2020
9d39e38
cleanup ExecAgent host logs for exec tasks
sharanyad Oct 16, 2020
022090e
Change ExecCommandAgentEnabled to ExecCommandAgentEnabledUnsafe
sharanyad Oct 16, 2020
7449d97
fix unit test failure
sharanyad Oct 22, 2020
02294da
add cleanup logs to integ test
sharanyad Oct 21, 2020
fde9847
validate exec agent start container state change event (#2691)
fierlion Oct 26, 2020
3da0878
add helper methods for creating ExecAgent Config file
sharanyad Oct 23, 2020
0fc85ac
generate execAgent config for default session limit
sharanyad Oct 23, 2020
dc78868
change agent execute command dir
sharanyad Oct 23, 2020
40ccb8c
change host execAgent config dir mount
sharanyad Oct 26, 2020
0db867d
bind mount ExecAgent Config file to each container
sharanyad Oct 27, 2020
c27f22f
Refactor execute-command-agent start process to be done at container …
angelcar Oct 23, 2020
1710ed8
Fix exec command agent configuration template to make it valid JSON
angelcar Oct 29, 2020
5476f76
Add util functions to determine latest ExecuteCommandAgent version
angelcar Oct 29, 2020
64faaf5
add managed agent stopped and running event check (#2702)
fierlion Nov 2, 2020
41566e7
add check for execute-command capability and populate attribute if ap…
cyastella Nov 2, 2020
efe2f25
Use latest version available in deps bin dir
angelcar Oct 30, 2020
ed12294
Fix exec command agent config
angelcar Nov 10, 2020
e29c1b5
add helper function mockPathExists()
cyastella Nov 3, 2020
b752dc3
add select model updates from autogenerated model (#2723)
fierlion Nov 12, 2020
3a21031
generate ExecAgent seelog config for exec enabled containers
sharanyad Nov 3, 2020
1ca05f4
check the validity of exec agent config
sharanyad Nov 17, 2020
dc53b09
Add ManagedAgent changes to SubmitTaskStateChange
angelcar Nov 17, 2020
0384f40
update cert mount source (#2748)
fierlion Nov 30, 2020
ccbbf07
Cleanup config files paths if they are dirs
sharanyad Dec 2, 2020
446304d
change to task level managed agent state change (#2758)
fierlion Dec 14, 2020
4a34811
Create container without enabling exec feature if exec bind mounting …
ubhattacharjya Dec 8, 2020
a9f9cc2
Emit managed agent stateChange event for Exec agent init failure
ubhattacharjya Dec 16, 2020
b68c37e
check at least 1 ssm version is valid instad of checking all versions…
cyastella Dec 22, 2020
24eb5e4
sync managed agent sent state (#2773)
fierlion Dec 23, 2020
2023291
Emit ExecCommandAgent stop event when container stops
angelcar Dec 24, 2020
68ab437
Emit the right reason upon exec command start error
angelcar Jan 6, 2021
60eb959
s/Execute Command Agent/ExecuteCommandAgent/g
sharanyad Jan 7, 2021
4852969
Run ssm-agent with root user
ubhattacharjya Jan 13, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ benchmark-test:

.PHONY: build-image-for-ecr upload-images replicate-images

build-image-for-ecr: netkitten volumes-test image-cleanup-test-images fluentd
build-image-for-ecr: netkitten volumes-test image-cleanup-test-images fluentd exec-command-agent-test

upload-images: build-image-for-ecr
@./scripts/upload-images $(STANDARD_REGION) $(STANDARD_REPOSITORY)
Expand Down Expand Up @@ -225,7 +225,11 @@ volumes-test:
$(MAKE) -C misc/volumes-test $(MFLAGS)

# Run our 'test' registry needed for integ tests
test-registry: netkitten volumes-test pause-container image-cleanup-test-images fluentd
test-registry: netkitten volumes-test pause-container image-cleanup-test-images fluentd exec-command-agent-test

exec-command-agent-test:
$(MAKE) -C misc/exec-command-agent-test $(MFLAGS)

@./scripts/setup-test-registry

.PHONY: fluentd gremlin image-cleanup-test-images
Expand Down Expand Up @@ -319,6 +323,7 @@ clean:
-$(MAKE) -C $(ECS_CNI_REPOSITORY_SRC_DIR) clean
-$(MAKE) -C misc/netkitten $(MFLAGS) clean
-$(MAKE) -C misc/volumes-test $(MFLAGS) clean
-$(MAKE) -C misc/exec-command-agent-test $(MFLAGS) clean
-$(MAKE) -C misc/gremlin $(MFLAGS) clean
-$(MAKE) -C misc/image-cleanup-test-images $(MFLAGS) clean
-$(MAKE) -C misc/container-health $(MFLAGS) clean
Expand Down
16 changes: 16 additions & 0 deletions agent/acs/model/api/api-2.json
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,7 @@
"name":{"shape":"String"},
"overrides":{"shape":"String"},
"portMappings":{"shape":"PortMappingList"},
"managedAgents":{"shape":"ManagedAgentList"},
"mountPoints":{"shape":"MountPointList"},
"volumesFrom":{"shape":"VolumeFromList"},
"dockerConfig":{"shape":"DockerConfig"},
Expand Down Expand Up @@ -496,6 +497,21 @@
},
"exception":true
},
"ManagedAgent":{
"type":"structure",
"members":{
"name":{"shape":"ManagedAgentName"},
"properties":{"shape":"StringMap"}
}
},
"ManagedAgentList":{
"type":"list",
"member":{"shape":"ManagedAgent"}
},
"ManagedAgentName":{
"type":"string",
"enum":["ExecuteCommandAgent"]
},
"FirelensConfiguration":{
"type":"structure",
"members":{
Expand Down
20 changes: 20 additions & 0 deletions agent/acs/model/ecsacs/api.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 6 additions & 6 deletions agent/acs/update_handler/updater_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ func TestPerformUpdateWithUpdatesDisabled(t *testing.T) {
Reason: ptr("Updates are disabled").(*string),
}})

taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil)
taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil, nil)
msg := &ecsacs.PerformUpdateMessage{
ClusterArn: ptr("cluster").(*string),
ContainerInstanceArn: ptr("containerInstance").(*string),
Expand Down Expand Up @@ -181,7 +181,7 @@ func TestFullUpdateFlow(t *testing.T) {

require.Equal(t, "update-tar-data", writtenFile.String(), "incorrect data written")

taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil)
taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil, nil)
msg := &ecsacs.PerformUpdateMessage{
ClusterArn: ptr("cluster").(*string),
ContainerInstanceArn: ptr("containerInstance").(*string),
Expand Down Expand Up @@ -249,7 +249,7 @@ func TestUndownloadedUpdate(t *testing.T) {
MessageId: ptr("mid").(*string),
}})

taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil)
taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil, nil)
msg := &ecsacs.PerformUpdateMessage{
ClusterArn: ptr("cluster").(*string),
ContainerInstanceArn: ptr("containerInstance").(*string),
Expand Down Expand Up @@ -307,7 +307,7 @@ func TestDuplicateUpdateMessagesWithSuccess(t *testing.T) {

require.Equal(t, "update-tar-data", writtenFile.String(), "incorrect data written")

taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil)
taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil, nil)
msg := &ecsacs.PerformUpdateMessage{
ClusterArn: ptr("cluster").(*string),
ContainerInstanceArn: ptr("containerInstance").(*string),
Expand Down Expand Up @@ -376,7 +376,7 @@ func TestDuplicateUpdateMessagesWithFailure(t *testing.T) {

require.Equal(t, "update-tar-data", writtenFile.String(), "incorrect data written")

taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil)
taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil, nil)
msg := &ecsacs.PerformUpdateMessage{
ClusterArn: ptr("cluster").(*string),
ContainerInstanceArn: ptr("containerInstance").(*string),
Expand Down Expand Up @@ -447,7 +447,7 @@ func TestNewerUpdateMessages(t *testing.T) {

require.Equal(t, "newer-update-tar-data", writtenFile.String(), "incorrect data written")

taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil)
taskEngine := engine.NewTaskEngine(cfg, nil, nil, nil, nil, nil, nil, nil, nil)
msg := &ecsacs.PerformUpdateMessage{
ClusterArn: ptr("cluster").(*string),
ContainerInstanceArn: ptr("containerInstance").(*string),
Expand Down
120 changes: 120 additions & 0 deletions agent/api/container/container.go
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,32 @@ type HealthStatus struct {
Output string `json:"output,omitempty"`
}

type ManagedAgentState struct {
// ID of this managed agent state
ID string `json:"id,omitempty"`
// TODO: [ecs-exec] Change variable name from Status to KnownStatus in future PR to avoid noise
// Status is the managed agent health status
Status apicontainerstatus.ManagedAgentStatus `json:"status,omitempty"`
// SentStatus is the managed agent sent status
SentStatus apicontainerstatus.ManagedAgentStatus `json:"sentStatus,omitempty"`
// Reason is a placeholder for failure messaging
Reason string `json:"reason,omitempty"`
// LastStartedAt is the timestamp when the status last went from PENDING->RUNNING
LastStartedAt time.Time `json:"lastStartedAt,omitempty"`
// Metadata holds metadata about the managed agent
Metadata map[string]interface{} `json:"metadata,omitempty"`
// InitFailed indicates if exec agent initialization failed
InitFailed bool `json:"initFailed,omitempty"`
}

type ManagedAgent struct {
ManagedAgentState
// Name is the name of this managed agent. This name is streamed down from ACS.
Name string `json:"name,omitempty"`
// Properties of this managed agent. Properties are streamed down from ACS.
Properties map[string]string `json:"properties,omitempty"`
}

// Container is the internal representation of a container in the ECS agent
type Container struct {
// Name is the name of the container specified in the task definition
Expand All @@ -121,6 +147,8 @@ type Container struct {
TaskARNUnsafe string `json:"taskARN"`
// DependsOnUnsafe is the field which specifies the ordering for container startup and shutdown.
DependsOnUnsafe []DependsOn `json:"dependsOn,omitempty"`
// ManagedAgentsUnsafe presently contains only the executeCommandAgent
ManagedAgentsUnsafe []ManagedAgent `json:"managedAgents,omitempty"`
// V3EndpointID is a container identifier used to construct v3 metadata endpoint; it's unique among
// all the containers managed by the agent
V3EndpointID string
Expand Down Expand Up @@ -672,6 +700,13 @@ func (c *Container) GetKnownPortBindings() []PortBinding {
return c.KnownPortBindingsUnsafe
}

// GetManagedAgents returns the managed agents configured for this container
func (c *Container) GetManagedAgents() []ManagedAgent {
c.lock.RLock()
defer c.lock.RUnlock()
return c.ManagedAgentsUnsafe
}

// SetVolumes sets the volumes mounted in a container
func (c *Container) SetVolumes(volumes []types.MountPoint) {
c.lock.Lock()
Expand Down Expand Up @@ -1188,3 +1223,88 @@ func (c *Container) HasNotAndWillNotStart() bool {
c.DesiredStatusUnsafe.Terminal() &&
c.AppliedStatus == apicontainerstatus.ContainerStatusNone
}

// GetManagedAgentByName retrieves the managed agent with the name specified and a boolean indicating whether an agent
// was found or not.
// note: a zero value for ManagedAgent if the name is not known to this container.
func (c *Container) GetManagedAgentByName(agentName string) (ManagedAgent, bool) {
c.lock.RLock()
defer c.lock.RUnlock()
for _, ma := range c.ManagedAgentsUnsafe {
if ma.Name == agentName {
return ma, true
}
}
return ManagedAgent{}, false
}

// UpdateManagedAgentByName updates the state of the managed agent with the name specified. If the agent is not found,
// this method returns false.
func (c *Container) UpdateManagedAgentByName(agentName string, state ManagedAgentState) bool {
c.lock.Lock()
defer c.lock.Unlock()
for i, ma := range c.ManagedAgentsUnsafe {
if ma.Name == agentName {
// It's necessary to clone the whole ManagedAgent struct
c.ManagedAgentsUnsafe[i] = ManagedAgent{
Name: ma.Name,
Properties: ma.Properties,
ManagedAgentState: state,
}
return true
}
}
return false
}

// UpdateManagedAgentStatus updates the status of the managed agent with the name specified. If the agent is not found,
// this method returns false.
func (c *Container) UpdateManagedAgentStatus(agentName string, status apicontainerstatus.ManagedAgentStatus) bool {
c.lock.Lock()
defer c.lock.Unlock()
for i, ma := range c.ManagedAgentsUnsafe {
if ma.Name == agentName {
c.ManagedAgentsUnsafe[i].Status = status
return true
}
}
return false
}

// UpdateManagedAgentSentStatus updates the sent status of the managed agent with the name specified. If the agent is not found,
// this method returns false.
func (c *Container) UpdateManagedAgentSentStatus(agentName string, status apicontainerstatus.ManagedAgentStatus) bool {
c.lock.Lock()
defer c.lock.Unlock()
for i, ma := range c.ManagedAgentsUnsafe {
if ma.Name == agentName {
c.ManagedAgentsUnsafe[i].SentStatus = status
return true
}
}
return false
}

func (c *Container) GetManagedAgentStatus(agentName string) apicontainerstatus.ManagedAgentStatus {
c.lock.RLock()
defer c.lock.RUnlock()
for i, ma := range c.ManagedAgentsUnsafe {
if ma.Name == agentName {
return c.ManagedAgentsUnsafe[i].Status
}
}
// we shouldn't get here because we'll always have a valid ManagedAgentName
return apicontainerstatus.ManagedAgentStatusNone
}

func (c *Container) GetManagedAgentSentStatus(agentName string) apicontainerstatus.ManagedAgentStatus {
c.lock.RLock()
defer c.lock.RUnlock()
for i, ma := range c.ManagedAgentsUnsafe {
if ma.Name == agentName {
return c.ManagedAgentsUnsafe[i].SentStatus
}
}
// we shouldn't get here because we'll always have a valid ManagedAgentName
return apicontainerstatus.ManagedAgentStatusNone
}
Loading