Weighted loadbalancing (client steering), WG peer config push to clients

README.md

@@ -7,6 +7,7 @@
   - [Overview](#overview)
     - [Frontend broker](#frontend-broker)
       - [POST /api/v1/wg/key/exchange](#post-apiv1wgkeyexchange)
+      - [POST /api/v2/wg/key/exchange](#post-apiv2wgkeyexchange)
     - [Backend worker](#backend-worker)
   - [Installation](#installation)
   - [Configuration](#configuration)
@@ -41,6 +42,7 @@ The frontend broker exposes the following API endpoints for use:
 
 ```
 /api/v1/wg/key/exchange
+/api/v2/wg/key/exchange
 ```
 
 The listen address and port for the Flask server can be configured in `wgkex.yaml` under the `broker_listen` key:
@@ -66,6 +68,35 @@ JSON POST'd to this endpoint should be in this format:
 
 The broker will validate the domain and public key, and if valid, will push the key onto the MQTT bus.
 
+
+#### POST /api/v2/wg/key/exchange
+
+JSON POST'd to this endpoint should be in this format:
+
+```json
+{
+  "domain": "CONFIGURED_DOMAIN",
+  "public_key": "PUBLIC_KEY"
+}
+```
+
+The broker will validate the domain and public key, and if valid, will push the key onto the MQTT bus.
+Additionally it chooses a worker (aka gateway, endpoint) that the client should connect to.
+The response is JSON data containing the connection details for the chosen gateway:
+
+```json
+{
+  "Endpoint": {
+    "Address": "GATEWAY_ADDRESS",
+    "Port": "GATEWAY_WIREGUARD_PORT",
+    "AllowedIPs": [
+      "GATEWAY_WIREGUARD_INTERFACE_ADDRESS"
+    ],
+    "PublicKey": "GATEWAY_PUBLIC_KEY"
+  }
+}
+```
+
 ### Backend worker
 
 The backend (worker) waits for new keys to appear on the MQTT message bus. Once a new key appears, the worker performs
@@ -129,13 +160,25 @@ Worker:
 python3 -c 'from wgkex.worker.app import main; main()'
 ```
 
-## Client usage
+
+## Development
+
+### Unit tests
+
+The test can be run using `bazel test ... --test_output=all` or `python3 -m unittest discover -p '*_test.py'`.
+
+### Client
 
 The client can be used via CLI:
 
 ```
-$ wget -q  -O- --post-data='{"domain": "ffmuc_welt","public_key": "o52Ge+Rpj4CUSitVag9mS7pSXUesNM0ESnvj/wwehkg="}'   --header='Content-Type:application/json'   'http://127.0.0.1:5000/api/v1/wg/key/exchange'
+$ wget -q  -O- --post-data='{"domain": "ffmuc_welt","public_key": "o52Ge+Rpj4CUSitVag9mS7pSXUesNM0ESnvj/wwehkg="}'   --header='Content-Type:application/json'   'http://127.0.0.1:5000/api/v2/wg/key/exchange'
 {
+  "Endpoint": {
+    "Address": "gw04.ext.ffmuc.net:40011",
+    "LinkAddress": "fe80::27c:16ff:fec0:6c74",
+    "PublicKey": "TszFS3oFRdhsJP3K0VOlklGMGYZy+oFCtlaghXJqW2g="
+  },
   "Message": "OK"
 }
 ```
@@ -146,7 +189,7 @@ Or via python:
 import requests
 key_data = {"domain": "ffmuc_welt","public_key": "o52Ge+Rpj4CUSitVag9mS7pSXUesNM0ESnvj/wwehkg="}
 broker_url = "http://127.0.0.1:5000"
-push_key = requests.get(f'{broker_url}/api/v1/wg/key/exchange', json=key_data)
+push_key = requests.get(f'{broker_url}/api/v2/wg/key/exchange', json=key_data)
 print(f'Key push was: {push_key.json().get("Message")}')
 ```
 
@@ -173,6 +216,13 @@ sudo ip link set wg-welt up
 sudo ip link set vx-welt up
 ```
 
+### MQTT topics
+
+- Publishing keys broker->worker: `wireguard/{domain}/{worker}`
+- Publishing metrics worker->broker: `wireguard-metrics/{domain}/{worker}/connected_peers`
+- Publishing worker status: `wireguard-worker/{worker}/status`
+- Publishing worker data: `wireguard-worker/{worker}/{domain}/data`
+
 ## Contact
 
 [Freifunk Munich Mattermost](https://chat.ffmuc.net)

requirements.txt

@@ -9,4 +9,4 @@ waitress~=2.1.2
 ipaddress~=1.0.23
 mock~=5.1.0
 coverage
-paho-mqtt~=1.6.1
+paho-mqtt~=1.6.1

wgkex.yaml.example

@@ -1,3 +1,4 @@
+# [broker] The domains that should be accepted by clients and for which matching WireGuard interfaces exist
 domains:
   - ffmuc_muc_cty
   - ffmuc_muc_nord
@@ -6,20 +7,37 @@ domains:
   - ffmuc_muc_west
   - ffmuc_welt
   - ffwert_city
+# [broker, worker] The prefix is trimmed from the domain name and replaced with 'wg-' and 'vx-'
+# to calculate the WireGuard and VXLAN interface names
+domain_prefixes:
+  - ffmuc_
+  - ffdon_
+  - ffwert_
+# [broker] The dict of workers mapping their hostname to their respective weight for weighted peer distribution
+workers:
+  gw04.in.ffmuc.net:
+    weight: 30
+  gw05.in.ffmuc.net:
+    weight: 30
+  gw06.in.ffmuc.net:
+    weight: 20
+  gw07.in.ffmuc.net:
+    weight: 20
+# [worker] The external hostname of this worker
+externalName: gw04.ext.ffmuc.net
+# [broker, worker] MQTT connection informations
 mqtt:
   broker_url: broker.hivemq.com
   broker_port: 1883
   username: user
   password: SECRET
   keepalive: 5
   tls: False
+# [broker]
 broker_listen:
   host: 0.0.0.0
   port: 5000
-domain_prefixes:
-  - ffmuc_
-  - ffdon_
-  - ffwert_
+# [broker, worker]
 logging_config:
   formatters:
     standard:

wgkex/broker/BUILD

@@ -1,6 +1,26 @@
 load("@rules_python//python:defs.bzl", "py_binary", "py_test")
 load("@pip//:requirements.bzl", "requirement")
 
+py_library(
+    name = "metrics",
+    srcs = ["metrics.py"],
+    visibility = ["//visibility:public"],
+    deps = [
+       "//wgkex/common:mqtt",
+       "//wgkex/common:logger",
+       "//wgkex/config:config",
+    ],
+)
+
+py_test(
+    name="metrics_test",
+    srcs=["metrics_test.py"],
+    deps = [
+       "//wgkex/broker:metrics",
+       requirement("mock"),
+    ],
+)
+
 py_binary(
     name="app",
     srcs=["app.py"],
@@ -11,5 +31,7 @@ py_binary(
         requirement("flask-mqtt"),
         requirement("waitress"),
         "//wgkex/config:config",
+        "//wgkex/common:mqtt",
+        ":metrics"
     ],
 )