Skip to content
This repository has been archived by the owner on Jan 8, 2024. It is now read-only.

Waypoint Server on ECS installing already existing instance #1763

Closed
marcotuna opened this issue Jul 2, 2021 · 3 comments
Closed

Waypoint Server on ECS installing already existing instance #1763

marcotuna opened this issue Jul 2, 2021 · 3 comments
Assignees
Labels
bug Something isn't working core/server plugin/aws
Milestone

Comments

@marcotuna
Copy link

Describe the bug
When attempting to install on an already existing Waypoint Server instance running on ECS it throws a panic error while attempting to "Registering Task definition"

Steps to Reproduce

  • Have an already running instance of Waypoint Server on ECS running with version 0.4.0.
  • Have the Waypoint CLI with version 0.4.1 and attempt to install

Log excerpt:

-----------------------------------------------------
2021/07/02 09:37:03 DEBUG: Request ecs/RegisterTaskDefinition Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: ecs.eu-west-1.amazonaws.com
User-Agent: aws-sdk-go/1.36.31 (go1.16.5; linux; amd64)
Content-Length: 1005
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20210702/eu-west-1/ecs/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=REDACTED
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20210702T083703Z
X-Amz-Target: AmazonEC2ContainerServiceV20141113.RegisterTaskDefinition
Accept-Encoding: gzip


-----------------------------------------------------
2021/07/02 09:37:03 DEBUG: Response ecs/RegisterTaskDefinition Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 
Connection: close
Content-Length: 90
Content-Type: application/x-amz-json-1.1
Date: Fri, 02 Jul 2021 08:37:03 GMT
X-Amzn-Requestid: 8cf02279-1377-4374-b61a-7852bf81ccfa


-----------------------------------------------------
2021-07-02T09:37:05.641+0100 [TRACE] waypoint: stopping signal listeners and cancelling the context
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x2577c33]

goroutine 1 [running]:
github.com/hashicorp/waypoint/internal/serverinstall.(*ECSInstaller).Launch(0xc000690380, 0x3209520, 0xc00040c940, 0x323ba18, 0xc0009c6380, 0x3224b10, 0xc0009c01d8, 0xc0009ea000, 0xc000c25de0, 0xc000ace3f0, ...)
        /home/circleci/project/waypoint/internal/serverinstall/ecs.go:286 +0x1133
github.com/hashicorp/waypoint/internal/serverinstall.(*ECSInstaller).Install.func2(0x3224b10, 0xc0009c01d8, 0xd, 0x0)
        /home/circleci/project/waypoint/internal/serverinstall/ecs.go:134 +0xf7
github.com/hashicorp/waypoint/internal/serverinstall.(*Lifecycle).Execute(0xc000e2d7b8, 0x323ba18, 0xc0009c6380, 0x3224b10, 0xc0009c01d8, 0x0, 0x0)
        /home/circleci/project/waypoint/internal/serverinstall/ecs.go:1135 +0x8b
github.com/hashicorp/waypoint/internal/serverinstall.(*ECSInstaller).Install(0xc000690380, 0x3209520, 0xc00040c940, 0xc000408e60, 0xc00036ef28, 0x1, 0x0)
        /home/circleci/project/waypoint/internal/serverinstall/ecs.go:141 +0x2dd
github.com/hashicorp/waypoint/internal/cli.(*InstallCommand).Run(0xc000a7d080, 0xc000133420, 0xa, 0xe, 0x0)
        /home/circleci/project/waypoint/internal/cli/install.go:85 +0x510
github.com/mitchellh/cli.(*CLI).Run(0xc0009c2000, 0xc000abfee8, 0x323ba18, 0xc0009127e0)
        /home/circleci/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:262 +0x41a
github.com/hashicorp/waypoint/internal/cli.Main(0xc000133400, 0xc, 0x10, 0x0)
        /home/circleci/project/waypoint/internal/cli/main.go:120 +0x57a
main.main()
        /home/circleci/project/waypoint/cmd/waypoint/main.go:14 +0xa5

Expected behavior
Running waypoint install should redeploy the Waypoint Server with the new settings specified in the cli.

Waypoint Platform Versions
Additional version and platform information to help triage the issue if
applicable:

  • Waypoint CLI Version: 0.4.1
  • Waypoint Server Platform and Version: ecs 0.4.0
  • Waypoint Plugin: aws/ecs

Additional context

waypoint install -plain -vvv -accept-tos -context-set-default -platform ecs -ecs-region=eu-west-1 -ecs-server-image=hashicorp/waypoint:latest -ecs-execution-role-name=waypoint-server-execution-role -ecs-mem=512 -ecs-cpu=512
@marcotuna marcotuna added the new label Jul 2, 2021
@catsby catsby added bug Something isn't working core/server labels Jul 6, 2021
@krantzinator krantzinator removed the new label Jul 7, 2021
@krantzinator krantzinator added this to the 0.4.x milestone Jul 7, 2021
@catsby
Copy link
Contributor

catsby commented Jul 15, 2021

Hello! The underlying error here is caused due to invalid cpu/memory combination of values. For 512 CPU, if you want to specify the memory as well you must use one of the following memory values: 1024 (1GB), 2048 (2GB), 3072 (3GB), 4096 (4GB). See this page for some more information on the combinations:

I'm currently working on a fix to have Waypoint validate and throw an error when the memory/cpu values are invalid according to that document. Stay tuned!

@catsby
Copy link
Contributor

catsby commented Jul 15, 2021

Short story:
Hello again! I've opened #1872 which should "fix" this issue, by returning an error that users can use to diagnose the cause. Screen shot attached in #1872.

Long story:
Function registerTaskDefinition was failing for an unrecognized error, and it just looped until it ran out of attempts then moved on. The ecs.RegisterTaskDefinitionOutput was then returned as nil, and then we got a nil reference panic on L286

In #1872 we verify the mem/cpu values are a good pair and if not, error. We also verify the taskOut is not nil.

@catsby
Copy link
Contributor

catsby commented Jul 20, 2021

Hello again (again)! I've merged #1872 , it will go out in the next Waypoint release. Thanks for reporting this!

@catsby catsby closed this as completed Jul 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working core/server plugin/aws
Projects
None yet
Development

No branches or pull requests

3 participants