Controller lags in relay mode when multiple devices connected to switch via bluetooth

[JuanPotato]

See mart1nro/joycontrol#31

I am looking into how to forward HCI_EVT packets that we receive to the controller. Obviously not all of them because some are responsible for negotiating the encryption keys.

[JuanPotato]

Ok, so it seems that we regularly receive mode change events from the pro controller, and then once we connect a controller to the switch we get a few new events, but this time from the switch. This makes sense.

Legend:

• light red: received event from pro controller
• dark read: sent command to pro controller
• light grey: received event from switch
• dark grey: sent command to switch

The first cluster of switch events are from me removing one joycon, and second is from removing the other joycon.

I believe the most important ones are the max slots change events we receive from the switch.

In order from top to bottom, the max slots changed values were 5, 5, 1, 5

This is when I smacked both joycons into the switch at the same time, probably wouldve been better to just do them one at a time but hey. Again we see the max slots changed events.

In order from top to bottom, the max slots changed values were 1, 5, 1, 5

I'll be honest I don't know what these max slot values mean.

HCI Events I believe we should forward from pro controller to switch

• Maybe mode change

HCI Events I believe we should forward from switch to pro controller

• Max slots changed
• Maybe role change
• Maybe mode change

[JuanPotato]

Also I want to do some graphing of the time between each received packet and each sent packet within this log. I want to figure out if the controller is sending the packets badly or if the switch isn't letting us write all the packets we want as fast. My assumption is the later, which would explain the pauses and then sudden sending of movement packets which make the lag happen. I assume slots would in some way refer to slots that the switch is going to hold for packets? Maybe when more than one controller is connected, the switch tells the other controllers to slow down their polling rate to let the switch be able to handle everything??

[JuanPotato]

I did a little research and collected some links

Very interesting

• http://www.bluez.org/the-management-interface/
  Talks about replacement to hci socks

• https://docs.ubuntu.com/core/en/stacks/bluetooth/bluez/docs/reference/available-commands
  bluez ubuntu docs

• https://github.com/KaiseiYokoyama/joycon-rs
  Joycon api in rust

• https://www.amd.e-technik.uni-rostock.de/ma/gol/lectures/wirlec/bluetooth_info/hci.html
  wild site about hci

Possibly useful

• https://github.com/balle/bluetooth-snippets/blob/master/hcisniff.c
  Some random hci examples
  Should be useful as a starting point

• https://www.spinics.net/lists/linux-bluetooth/msg37345.html
  Explains hci user socket mode

• https://git.congatec.com/arm/imx6_kernel_4.1/commit/23500189d7e03a071f0746f43f2cce875a62c91c
  commit introducing the user socket mode, same as above thing?

• https://github.com/Adam-Langley/pybleno/blob/master/pybleno/hci_socket/BluetoothHCI/BluetoothHCI.py
  someone using hci in python

• https://stackoverflow.com/questions/53857864/details-on-hci-filter-socket-options
  talking about socket filter options

• https://stackoverflow.com/questions/51895739/linux-receive-data-from-bluetooth-le
  I dont know what hci_devid is but I might need it later

• https://gitlab.com/kzapalowicz/hciraw
  Rust bindings for hciraw, might use

• https://lisha.ufsc.br/teaching/shi/ine5346-2003-1/work/bluetooth/hci_commands.html
  List of HCI commands and events

• https://www.programcreek.com/python/example/50991/socket.BTPROTO_HCI
  Weird site giving me tons of examples but not really but sorta?

• https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux/+/refs/heads/master/include/net/bluetooth

• https://kernel.googlesource.com/pub/scm/linux/kernel/git/bluetooth/bluetooth/+/refs/heads/master
  bluetooth kernel source

• https://stackoverflow.com/questions/41944822/bluetooth-over-uart-using-hciattach
  contains useful links

• https://people.csail.mit.edu/albert/bluez-intro/x682.html
  general linux bluetooth bluez help

Possibly not useful

• http://software-dl.ti.com/lprf/simplelink_cc26x2_latest/docs/ble5stack/vendor_specific_guide/BLE_Vendor_Specific_HCI_Guide/hci_interface.html
  TI docs about how the hci interface works, might be a little helpful

• https://stackoverflow.com/questions/28261029/wrong-payload-when-using-bluez-stack-as-peripheral
  Advertizing and cool

• https://stackoverflow.com/questions/53047742/prevent-bluetooth-driver-actions
  not too helpful, keeping anyway for example code

• https://stackoverflow.com/questions/57152408/what-is-the-internal-mechanics-of-socket-function
  Neat link talking about how socket() works

• https://stackoverflow.com/questions/50538888/linux-bluetooth-api-function-locations
  Didn't look too much into this, doesn't seem helpful but they mentioned hci sockets so I want it

• https://reverse-engineering-ble-devices.readthedocs.io/en/latest/index.html
  This looks cool, might not be all to helpful for me though

• https://stackoverflow.com/questions/16151360/use-bluez-stack-as-a-peripheral-advertiser
  Mildly useful, shows hcitool usage

• https://multi-io.github.io/mydocs-pub/bluetooth/bluetooth.txt.html
  bluetooth.txt what

• https://www.real-world-systems.com/docs/hcitool.1.html
  hcitool

Prexisting similar projects as mine

• https://github.com/matlo/l2cap_proxy
• https://github.com/quangthanh010290/BL_keyboard_RPI/
• https://github.com/Alkaid-Benetnash/EmuBTHID
• https://github.com/007durgesh219/BTGamepad
• https://github.com/gkovacs/Ubuntu_Bluetooth_Keyboard_Emulator

Useful programs

• hcidump
• hcitool
• btmon

[JuanPotato]

Ok things I've learned about bluetooth.

Radio -> Baseband -> LMP -> HCI -> L2CAP -> HID
bluetooth stack.

I am opening up sockets in the l2cap layer to forward data from controller to switch and from switch to controller. This works for the most part.

When a second controller is connected via bluetooth to the switch, it sends out a request to change the max slots on the LMP layer to tell the bluetooth controllers to slow down so that everyone has enough room. That data also ripples an event into the HCI layer. Note the request is sent in LMP but you can read an event in HCI and LMP. That event is never sent over to the controller, because I'm only reading L2CAP, so the controller never slows down and then we lag.

I need access to either HCI or LMP for reading, but I definitely need access to LMP for writing. Bluez makes reading HCI stuff easy, but no clue how hard it will be to rw LMP stuff.

[JuanPotato]

A lot of that info was found from this site https://www.amd.e-technik.uni-rostock.de/ma/gol/lectures/wirlec/bluetooth_info/radio.html

[typ127]

Hi JuanPotato,

I used tcpdump to sniff the bluetooth conversation. Then I used Matplotlib to visualize results. I just want to share my results with you.

This is just one controller connected, sending status every 15 ms. the curve only shows distance between timestamps of incoming and outgoing messages. You always count 3 messages send from the controller (near 0.015) and one incoming message (near 0) - handshake?

This is the same thing after a second controller becomes connected. Peaks are the slowdown lags. Right before the peaks I think you miss some incoming signals.

Thats the amount of sent and received messages over time with a second controller connected

Blue: send messages from controller (second controller connected). Red incoming messages from console. Nut sure if this is plotted correctly. I just plot the time delta between each incoming message and plot an x. The blue curve makes sense as it slows down every period

Again the amount of messages over time

Here you see a message sending frequency of 1Hz. So the controller sends his state once a second. Even with this slow frequency there are gaps included. At the end of the curve you see two messages sent at the same time with the gap of two seconds after.

I am sure it is not solvable only by slowing down the send frequency. Maybe the incoming message is not a handshake but a request for status massage (mentioned by @dekuNukem)? So the controller only need to send the status when he is asked for it?

[JuanPotato]

Hi, thank you so much for contributing to this! I have a few questions, the first being:

How did you find this repo? (EDIT: Just realized you were the person who opened mart1nro/joycontrol#31, Got it!)

Also, pairing is a little bad right now due to the fact that we dont flush out all the old packets before connecting to the switch. When the pro controller first connects to the pc, it sends small packets, presumably with bare minimum information for using it as a hid controller. It sends this every 15ms. These pile up very quickly between the time the pro controller connects, and the switch connects, so pairing is not very good while it catches up and floods the switch with old packets. I need to fix that.

So if you have data from within the first ten seconds or so of connecting, its probably misleading.

With regards to the first graph, I haven't graphed anything yet, but just looking at my wireshark capture, I don't have a message from the switch after three packets from the controller.

Second graph, was the second controller already connected? When was it connected?

Fourth graph and Fifth graph, I am generally not sure how to interpret these graphs.

Sixth graph, how did you change the message sending frequency to 1hz?

Thanks for joining in on my wild journey to try and figure out whats going on. I need to look more into what exactly the max slots changed event is changing.

[typ127]

Hi, Okay first my setup, maybe things become more clear.

I use the python joycontrol on raspy 3 as first controller. This is in fact the controller i use for plotting. I use tcpdump to record bluetooth conversation between joycontrol and switch. Tcpdump writes in and out packages with timestamp and content. Sadly the content is not readable as it is binary data (i think). Not sure what wireshark displays here, maybe he can do much better on this.
When I create a plot, I make sure the switch is turned on without any controller connected. after start recording I connect the joycontrol to the switch using the switch mac address. This takes nearly 5 seconds. Then after a while I connect a second controller (original joycon) to the switch. after a view more seconds i stop the recording.

I did some new simpler plottings in the past hour, lets take a look at them:

This is 25 seconds of packet dumps. Blue is outgoing (from joycontrol to switch), red is incoming packages. The vertical axis shows the size of the package. I thougt this is a good indicator to differentiate between different types of packages (all packages in the same y position are of the same size - same message type).

You see 4 Areas. A: Starting phase - joycontrol connects to switch. B: begin broadcasting controller status at 66Hz (every 15ms). This is set up in joycontrol.protocol and can be changed here to different speeds. C: continue broadcasting. Second controller is turned on and connects to switch. D: Second controller is connected to switch - lags occur

Interesting: B and C differ in one periodic send message at size 60. It stops sending when second controller tries to connect to switch. I am not really sure if this is the reason for the 60 size message to stop. I will check it out in the next days

Now we look at each part zoomed

Staring Phase. Note: The one red dot at size 100 is actually much more on top of it (at about 500). I corrected the number to 100 to have a better looking plot.

Broadcasting with additional size 60 package each second. Blue dots are outgoing packages (each 15 ms). Every three sent packages, there is one incoming at size 8 (about 20Hz)

Same thing, continue broadcasting, but without the size 60 package. Second controller is beginning to connect. No lags until now!

Second controller is connected to the switch. Lags are clearly visible in the blue line every 1.25 seconds (nearly). You can also see gaps in the red lines right before the lags in the blue line.

Sure thing: The joycontrol sends to much packages to the switch. If you increase the broadcast frequency to lets say 400Hz, the gaps are much bigger! When you decrease it to 20Hz, it becomes much mor stabil, but 20Hz is a really bad performance when playing games - and this is what we really want to achieve - good performance! (Small Note: 20Hz is exactly the frequency of the incoming packages during broadcast at size 8)

I would really like to help you to find a solution to this. I am right in the middle of a 2 person arcade switch machine project. Maybe you have some ideas to share?

[typ127]

This comes right up in my mind, maybe it's important. I just use the dump from tcpdump for the plots. Maybe some of the incoming packages are not from the switch. Maybe they are just bluetooth protocol garbage and should be ignored? Not sure on this..

[JuanPotato]

Ok, I've done some more research (you can read my stuff in #2 for what I found) and once I implement what I talked about there I will see if that fixes the issue. Those graphs were very interesting.