Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

balancer: fix message parsing error -- invalid type: string "true" #1339

Closed
3 tasks
dyc3 opened this issue Feb 16, 2024 · 8 comments · Fixed by #1452
Closed
3 tasks

balancer: fix message parsing error -- invalid type: string "true" #1339

dyc3 opened this issue Feb 16, 2024 · 8 comments · Fixed by #1452
Assignees
Labels
bug Something isn't working regression An intentional behavior that worked before does not work now

Comments

@dyc3
Copy link
Owner

dyc3 commented Feb 16, 2024

Current Behavior

Started seeing this log message from the balancer

2024-02-16T14:03:12.261509Z ERROR ott_balancer::balancer: failed to dispatch monolith message e7768db2-da5a-4726-9c07-c98392daf866: invalid type: string "true", expected a boolean at line 1 column 116

Expected Behavior

All messages from monoliths should be valid json, and the balancer should be able to parse them.

Steps To Reproduce

  1. docker-compose -f docker/with-balancer.docker-compose.yml up -d --scale opentogethertube=10
  2. Watch logs with docker-compose -f docker/with-balancer.docker-compose.yml logs -f --tail 1
  3. See log message

Environment

  • This happens on the official site, opentogethertube.com
  • This happens using a self-hosted version.
  • I'm using the docker image.

Anything else?

I think this has something to do with the M2BGossip mesage

@dyc3 dyc3 added bug Something isn't working regression An intentional behavior that worked before does not work now labels Feb 16, 2024
@dyc3 dyc3 added this to the Load Balancing milestone Feb 16, 2024
@Victor-M-Giraldo
Copy link
Contributor

I would like to work on this.

@dyc3
Copy link
Owner Author

dyc3 commented Feb 23, 2024

I was able to reproduce this again using the average-load load test. Got this log message:

2024-02-23T03:54:39.984832Z ERROR ott_balancer::balancer: failed to dispatch monolith message cfb65b5a-7c1d-479e-9688-eef961984e7e: payload.rooms[0].room.isTemporary: invalid type: string "true", expected a boolean at line 1 column 116

I'm more inclined to think that there is something wrong with the monolith than the balancer.

Full steps:

  1. docker-compose -f docker/with-balancer.docker-compose.yml up -d --scale opentogethertube=10
  2. observe logs from docker-compose -f docker/with-balancer.docker-compose.yml logs -f --tail 1 balancer
  3. run k6 run tests/load/average-load.js

@Victor-M-Giraldo
Copy link
Contributor

Victor-M-Giraldo commented Feb 23, 2024

I have docker working both on my desktop and laptop. Can confirm I'm getting the same error.

I haven't tried it with K6 yet, but I will tomorrow.

To me, this isn't really indicative of where the error might be in the code base. How can I troubleshoot more to try and track it down?

@dyc3
Copy link
Owner Author

dyc3 commented Feb 23, 2024

I haven't tried it with K6 yet, but I will tomorrow.

ah well if you're getting it without k6, don't bother.

We need to track down where isTemporary is getting set as a string. I would start adding logs or placing breakpoints (to use with the debugger) in places where rooms get created. Maybe the Room class constructor would be a good place to start for this.

@Victor-M-Giraldo
Copy link
Contributor

Victor-M-Giraldo commented Feb 23, 2024

Never mind. Can't recreate it without k6 now. Do I have to run the entire load test or is there a way to slow down the logs in the terminal to better see whats going on? They fly by really fast and you can't get any meaningful information from them. I'll also start trying to mess around with the debugger.

@dyc3
Copy link
Owner Author

dyc3 commented Feb 23, 2024

you can reduce the number of vus

@dyc3
Copy link
Owner Author

dyc3 commented Feb 24, 2024

I've found an easier repro:

  1. Start 1 monolith 1 balancer
  2. Get an auth token from GET /api/auth/grant
  3. Create a room via HTTP request: POST /api/room/create
    With header:
Authorization: Bearer $TOKEN

with body

{
	"name": "foobar",
	"isTemporary": "true"
}

This should get the log message to show up again.

dyc3 added a commit that referenced this issue Feb 24, 2024
By default, k6/http will encode post requests as `x-www-form-urlencoded`. This can result in boolean values getting interpreted by the monolith as strings.

Related: #1339
@Victor-M-Giraldo
Copy link
Contributor

Think you have some free time at any point today or tomorrow to go over some of this? I've been trying to use the debugger but haven't been making much progress.

dyc3 added a commit that referenced this issue Feb 24, 2024
By default, k6/http will encode post requests as `x-www-form-urlencoded`. This can result in boolean values getting interpreted by the monolith as strings.

Related: #1339
@dyc3 dyc3 moved this from Todo to In Progress in OTT Horizontal Scaling Feb 25, 2024
@dyc3 dyc3 closed this as completed in #1452 Mar 6, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in OTT Horizontal Scaling Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working regression An intentional behavior that worked before does not work now
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants