Skip to content

Commit cb8bb75

Browse files
reguladregulad
committed
Commit working progress thus far
Ubuntu documentation docker attempt perfect 4-architecture building dist_archive building Test GitHub actions Test GitHub actions Test GitHub actions Test GitHub actions Test GitHub actions Test GitHub actions Test GitHub actions properly implemented caching
1 parent 86e68f0 commit cb8bb75

File tree

15 files changed

+455
-29
lines changed

15 files changed

+455
-29
lines changed

‎.github/workflows/test.yml

+97
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
on:
2+
pull_request:
3+
types: [opened, synchronize]
4+
push:
5+
branches: ["**"]
6+
7+
jobs:
8+
test-compile:
9+
runs-on: ubuntu-latest
10+
name: "Test P4A Compiling"
11+
12+
steps:
13+
- name: Clone LXMF
14+
run: git clone https://github.com/markqvist/LXMF $GITHUB_WORKSPACE/LXMF
15+
16+
- name: Clone LXST
17+
run: git clone https://github.com/markqvist/LXST $GITHUB_WORKSPACE/LXST
18+
19+
- name: Clone rnsh
20+
run: git clone https://github.com/acehoss/rnsh $GITHUB_WORKSPACE/rnsh
21+
22+
- name: Clone Reticulum
23+
run: git clone https://github.com/markqvist/Reticulum $GITHUB_WORKSPACE/Reticulum
24+
25+
- name: Clone rnode-flasher
26+
run: git clone https://github.com/liamcottle/rnode-flasher $GITHUB_WORKSPACE/rnode-flasher
27+
28+
- name: Clone NomadNet
29+
run: git clone https://github.com/markqvist/nomadnet $GITHUB_WORKSPACE/NomadNet
30+
31+
- name: Clone reticulum_website
32+
run: git clone https://github.com/markqvist/reticulum_website $GITHUB_WORKSPACE/reticulum_website
33+
34+
- name: Checkout Sideband
35+
uses: actions/checkout@v3
36+
with:
37+
path: Sideband/
38+
39+
# Set up caching for general cache directory
40+
- name: Cache pip build cache
41+
uses: actions/cache@v3
42+
with:
43+
path: ~/.cache
44+
key: ${{ runner.os }}-build-cache-${{ hashFiles('**/buildozer.spec') }}
45+
restore-keys: |
46+
${{ runner.os }}-build-cache-
47+
48+
# Set up caching for buildozer directories
49+
- name: Cache buildozer
50+
uses: actions/cache@v3
51+
with:
52+
path: |
53+
~/.buildozer
54+
${{ github.workspace }}/**/.buildozer
55+
${{ github.workspace }}/**/.buildozer-container
56+
key: ${{ runner.os }}-buildozer-${{ hashFiles('**/buildozer.spec') }}
57+
restore-keys: |
58+
${{ runner.os }}-buildozer-
59+
60+
# Set up caching for Gradle
61+
- name: Cache Gradle files
62+
uses: actions/cache@v3
63+
with:
64+
path: |
65+
~/.gradle/caches
66+
~/.gradle/wrapper
67+
~/.gradle/daemon
68+
~/.gradle/native
69+
key: ${{ runner.os }}-gradle-${{ hashFiles('**/buildozer.spec') }}
70+
restore-keys: |
71+
${{ runner.os }}-gradle-
72+
73+
- name: Create Android SDK directory
74+
run: mkdir -p ~/.android
75+
76+
- name: Generate debug keystore
77+
run: |
78+
keytool -genkeypair \
79+
-keystore ~/.android/debug.keystore \
80+
-storepass android \
81+
-keypass android \
82+
-alias androiddebugkey \
83+
-keyalg RSA \
84+
-keysize 2048 \
85+
-validity 10000 \
86+
-dname "CN=Android Debug,O=Android,C=US"
87+
88+
- name: Attempt to build Sideband for Android (Debug)
89+
run: |
90+
cd $GITHUB_WORKSPACE/Sideband
91+
./dmake devapk
92+
93+
- name: Upload built APK
94+
uses: actions/upload-artifact@v4
95+
with:
96+
name: android-debug
97+
path: ${{ github.workspace }}/Sideband/sbapp/bin/*.apk

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,4 @@ dist
3333
docs/build
3434
sideband*.egg-info
3535
sbapp*.egg-info
36+
Sideband.iml

‎DEVELOPING.md

+145
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# Establishing a Development environment
2+
3+
Looking to contribute some code to Sideband? Awesome! Follow the guide below to get the repository building on your machine.
4+
5+
## Creating folders
6+
7+
Sideband relies on a certain folder structure to achieve a psuedo-monorepo structure with the other Reticulum projects.
8+
9+
To make sure that the `getrns` target runs successfully, make sure your directory tree looks like this:
10+
11+
```
12+
repositories/
13+
├─ LXMF/
14+
├─ LXST/
15+
├─ rnsh/
16+
├─ rnode-flasher/
17+
├─ Reticulum/
18+
├─ NomadNet/
19+
├─ reticulum_website/
20+
└─ Sideband/
21+
```
22+
23+
Below are the git repositories for some of the above folders:
24+
25+
- `LXMF`: https://github.com/markqvist/LXMF
26+
- `LXST`: https://github.com/markqvist/LXST
27+
- `rnsh`: https://github.com/acehoss/rnsh
28+
- `Reticulum`: https://github.com/markqvist/Reticulum
29+
- `rnode-flasher`: https://github.com/liamcottle/rnode-flasher
30+
- `NomadNet`: https://github.com/markqvist/nomadnet
31+
- `reticulum_website`: https://github.com/markqvist/reticulum_website
32+
33+
> Please note: in order for the docker script and `createshare` make target to work correctly, your directory **must** be laid out like this.
34+
35+
## Required dependencies for development (Docker)
36+
37+
If you have a Fedora-based Linux system (see [Addendum: Fedora](#addendum-fedora)) or simply would not like to install all of P4A's dependencies manually, you may choose to use the containerized build.
38+
39+
This method requires that you have Docker installed on your system: https://docs.docker.com/engine/install/
40+
41+
Additionally, [rootless Docker](https://docs.docker.com/engine/install/) should be used to minimize any possible attack surface on your system. Never run a script you haven't vetted with sudo! The `./dmake.sh` script uses `set -ex` and is designed to be used with rootless docker.
42+
43+
After configuring docker, you can replace any use of the `make` command with `dmake` (i.e. `./dmake devapk`) to run make commands in the container, building it on demand if needed.
44+
45+
Example:
46+
47+
```
48+
./dmake devapk
49+
```
50+
51+
(or if running in sbapp, it is smart enough to run itself in Sideband regardless of where it is called from)
52+
53+
```
54+
../dmake devapk
55+
```
56+
57+
## Required dependencies for development (Native)
58+
59+
Until this repository has a `flake.nix` added, you will need to manually download the following dependencies using your Operating System's package manager.
60+
61+
- `make`
62+
- `adb` (available as a part of the `android-tools` package on Fedora, `adb` on Debian/Ubuntu, `android-platform-tools` on Brew Casks)
63+
- `python3`/`python3-dev(el)` (must be available as `python`, on Ubuntu try `apt install python-is-python3`)
64+
- `patchelf`
65+
- `patch`
66+
- `perl`
67+
- `portaudio19-dev`
68+
- `libopus-dev`
69+
- `libogg-dev`
70+
- `buildozer` (see https://buildozer.readthedocs.io/en/latest/installation.html)
71+
- buildozer's PyPI hosted version is very far behind, therefore you should install from source at https://github.com/kivy/buildozer.git; this is easy with pipx `pipx install git+https://github.com/kivy/buildozer.git`.
72+
- buildozer needs `wheel` to run, but it is not currently marked as a dependency. If you are using pipx, you will need to inject it with `pipx inject buildozer wheel`.
73+
- all of buildozer's Android dependencies
74+
- Ubuntu 22.04 LTS packages `git zip unzip openjdk-17-jdk python3-pip autoconf libtool pkg-config zlib1g-dev libncurses5-dev libncursesw5-dev libtinfo5 cmake libffi-dev libssl-dev`
75+
- Ubuntu 24.04 LTS packages `git zip unzip openjdk-17-jdk python3-pip autoconf libtool pkg-config zlib1g-dev libncurses-dev libtinfo6 cmake libffi-dev libssl-dev`
76+
- Fedora 41 `git zip unzip java-17-openjdk java-17-openjdk-devel python3-pip autoconf libtool pkgconf-pkg-config ghc-zlib-devel ncurses-devel ncurses-compat-libs cmake libffi-devel openssl-devel`
77+
78+
In the root directory of the repository, use `pip install .` to install the package from `setup.py`. The use of a `venv` is strongly recommended.
79+
80+
Make sure you manually install `Cython<3.0` into your Python install or `venv`, as buildozer will need it for Android.
81+
82+
### Addendum: Fedora
83+
84+
As many users of Kivy have noted before, some of Python4Android's recipes do not compile correctly on Fedora/RHEL. For this project, one package of interest is [`freetype-py` and its native dependency](https://github.com/kivy/python-for-android/blob/develop/pythonforandroid/recipes/freetype/__init__.py), which is a direct dependency of pillow, the ubiquitous Python image editing library.
85+
86+
This is due to the fact that Fedora and several other distros include default versions of toolchains, which prompts python4android to abstain from downloading its own. [This issue has been encountered by many other users.](https://groups.google.com/g/kivy-users/c/z46lSJXgbjY/m/M1UoWwtWAgAJ)
87+
88+
If you can't use Docker, use of Ubuntu 24.04 LTS is therefore recommended for developing this project. Ubuntu 22.04 LTS is not supported, as its `cmake` version (even with backports) is below the minimum 3.24.
89+
90+
Sideband does run fine on Fedora, however.
91+
92+
## Compiling and testing Sideband on an Android device
93+
94+
With a correctly configured environment, run the following command to create a development APK.
95+
96+
```
97+
make devapk
98+
```
99+
100+
You can then install it to a connected device with
101+
102+
```
103+
make devinstall
104+
```
105+
106+
If you would like your release to be signed for release, you must configure the following four environment variables:
107+
108+
- `P4A_RELEASE_KEYALIAS`
109+
- `P4A_RELEASE_KEYSTORE_PASSWD`
110+
- `P4A_RELEASE_KEYSTORE`
111+
- `P4A_RELEASE_KEYALIAS_PASSWD`
112+
113+
With `./dmake`, omitting any of these values will cause it to default to the `debug.keystore` generated with each install of the Android SDK.
114+
115+
After it is configured correctly, you may build it with
116+
117+
```
118+
make apk
119+
```
120+
121+
and install it to a connected device with
122+
123+
```
124+
make install
125+
```
126+
127+
The output will be placed in `./sbapp/bin/sideband-*.apk`.
128+
129+
If you have multiple devices connected at once (for example, while developing the BLE interface between devices), you may use `devinstall-multi` or `install-multi` in place of `devinstall` and `install` respectively.
130+
131+
If using an Android that provides the ability to toggle DCL via storage (like GrapheneOS), make sure to enable it for Sideband, as it must load its Cython executables.
132+
133+
## Compiling and testing Sideband for other platforms
134+
135+
For Windows, use the following command: (make sure you have `PyInstaller` in your venv, and you are on Windows, as PyInstaller removed cross-compilation)
136+
137+
```
138+
make build_win_exe
139+
```
140+
141+
Wheels for other platforms can be built with
142+
143+
```
144+
make build_wheel
145+
```

‎Dockerfile

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# exercise extreme caution when bumping this docker image to the newest ubuntu LTS version
2+
FROM ubuntu:24.04
3+
4+
# for rationale behind each of these dependencies, consult the native section of DEVELOPING.md
5+
RUN DEBIAN_FRONTEND=noninteractive \
6+
apt update \
7+
&& DEBIAN_FRONTEND=noninteractive apt install -y curl git libffi-dev python-is-python3 python3-dev python3-wheel python3-setuptools python3-virtualenv libssl-dev autoconf openjdk-17-jdk cmake libtool libssl-dev libncurses5-dev libsqlite3-dev libreadline-dev libtk8.6 libgdm-dev libpcap-dev unzip zip wget apksigner build-essential libopus-dev libogg-dev portaudio19-dev patchelf pipx \
8+
&& apt install --reinstall python3 \
9+
&& rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*
10+
11+
# need to run as root for a rootless docker runtime
12+
# the repository folders owned by 1000 on the host are mounted to 0 on the container; this is intentional and unchangable
13+
14+
# add pipx
15+
ENV PATH=$PATH:/root/.local/bin
16+
17+
# install & inject wheel
18+
RUN pipx install git+https://github.com/kivy/buildozer.git@abc2d7e66c8abe096a95ed58befe6617f7efdad0
19+
RUN pipx inject buildozer wheel
20+
21+
# needed for some transitive deps, like rnsh
22+
RUN pipx install poetry==2.1.1

‎Makefile

+1-1
Original file line numberDiff line numberDiff line change
@@ -40,4 +40,4 @@ upload:
4040
@echo Ready to publish release, hit enter to continue
4141
@read VOID
4242
@echo Uploading to PyPi...
43-
twine upload dist/sbapp-*
43+
twine upload dist/sbapp-*

‎dmake

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#!/bin/bash
2+
3+
set -ex
4+
5+
# any of the tasks that require adb should be run on the host, not the guest docker container
6+
# ad-hoc docker usb passthrough sounds miserable...
7+
if [ $1 == "install" ] || [ $1 == "devinstall" ] || [ $1 == "install-multi" ] || [ $1 == "devinstall-multi" ]; then
8+
echo "Command requiring adb detected, running on host..."
9+
# shellcheck disable=SC2068 # goal is to re-split
10+
make -C sbapp $@
11+
exit 0
12+
fi
13+
14+
if [[ -z $P4A_RELEASE_KEYSTORE ]]; then
15+
echo "P4A_RELEASE_KEYSTORE is not set correctly! Using default debug keystore..."
16+
P4A_RELEASE_KEYSTORE=$(realpath ~/.android/debug.keystore)
17+
P4A_RELEASE_KEYALIAS=androiddebugkey
18+
P4A_RELEASE_KEYSTORE_PASSWD=android
19+
P4A_RELEASE_KEYALIAS_PASSWD=android
20+
fi
21+
22+
# will hit caches automatically, unless the dockerfile was changed
23+
# buildozer writes to its own venv, so it can't be --read-only even though it really should be
24+
# /root/.buildozer has some build artifacts that are reused, and since they are dependent on the libraries available at compile-time, they must be isolated from ~/.buildozer as these may have different fingerprints
25+
# ~/.cache, on the other hand, may be reused because pip is smart enough to download/rebuild native dependencies if they wouldn't be compatible
26+
docker build --network=host -t sideband-dmake .
27+
docker run \
28+
--rm \
29+
--tty \
30+
--network=host \
31+
--tmpfs /tmp:rw,exec \
32+
-e P4A_RELEASE_KEYALIAS=$P4A_RELEASE_KEYALIAS \
33+
-e P4A_RELEASE_KEYSTORE_PASSWD=$P4A_RELEASE_KEYSTORE_PASSWD \
34+
-e P4A_RELEASE_KEYSTORE="/keystore.jks" \
35+
-e P4A_RELEASE_KEYALIAS_PASSWD=$P4A_RELEASE_KEYALIAS_PASSWD \
36+
-v "$(realpath $(dirname "$0")/..):/repositories:rw" \
37+
-v "$(realpath ./.buildozer-container):/root/.buildozer:rw" \
38+
-v "$(realpath ./sbapp/.buildozer-container):/repositories/Sideband/sbapp/.buildozer:rw" \
39+
-v "$(realpath ~/.cache):/root/.cache:rw" \
40+
-v "$P4A_RELEASE_KEYSTORE:/keystore.jks:ro" \
41+
-v "$(realpath ~/.android):/root/.android:rw" \
42+
-v "$(realpath ~/.gradle):/root/.gradle:rw" \
43+
sideband-dmake \
44+
/bin/bash \
45+
-c \
46+
"python3 -m venv --system-site-packages /tmp/venv \
47+
&& source /tmp/venv/bin/activate \
48+
&& cd /repositories/Sideband \
49+
&& pip install -e . \"Cython<3.0\" \
50+
&& make -C sbapp $@"

‎recipes/codec2/__init__.py

+16-2
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,19 @@
11
from os.path import join
2+
from tempfile import TemporaryDirectory
23
from pythonforandroid.recipe import Recipe
34
from pythonforandroid.toolchain import current_directory, shprint
5+
import os
46
import sh
57

68
# For debugging, clean with
79
# buildozer android p4a -- clean_recipe_build codec2 --local-recipes ~/Information/Source/Sideband/recipes
810

911
class Codec2Recipe(Recipe):
12+
# recipe for building codec2 from https://github.com/markqvist/codec2
13+
1014
url = "https://github.com/markqvist/codec2/archive/00e01c9d72d3b1607e165c71c4c9c942d277dfac.tar.gz"
15+
sha512sum = "2f8db660592e19b7f853c146793ccbde90f1d505663084f055172c8e5088a9fc2ddb588cc014ed8dec46a678ec73aaf654bbe77ff29f21caa7c45fb121f2281f"
16+
1117
built_libraries = {'libcodec2.so': 'build_android/src'}
1218

1319
def include_flags(self, arch):
@@ -41,10 +47,18 @@ def build_arch(self, arch):
4147
# cd = sh.cd("build_android")
4248
os.chdir("build_android")
4349
cmake = sh.Command('cmake')
50+
gcc = sh.Command("gcc")
4451

4552
shprint(cmake, *flags, _env=env)
46-
shprint(sh.make, _env=env)
47-
sh.cp("../src/codec2.h", "./codec2/")
53+
54+
# before running the make, we need to compile `generate_codebook` from the codec2 repository for the architecture we are on
55+
# allowing it to be compiled with the rest of the
56+
with TemporaryDirectory() as tmp:
57+
shprint(gcc, "../src/generate_codebook.c", "-o" f"{tmp}{os.sep}generate_codebook", "-lm")
58+
59+
env_tmp = env | {"PATH": f"{env['PATH']}{os.pathsep}{tmp}"}
60+
shprint(sh.make, _env=env_tmp)
61+
sh.cp("../src/codec2.h", "./codec2/")
4862

4963

5064
recipe = Codec2Recipe()

‎recipes/codec2/generate_codebook

-16.1 KB
Binary file not shown.

‎recipes/opusfile/__init__.py

-2
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,6 @@ def build_arch(self, arch):
3434
# env['LDFLAGS'] += openssl_recipe.link_dirs_flags(arch)
3535
# env['LIBS'] = openssl_recipe.link_libs_flags()
3636

37-
from rich.pretty import pprint
38-
pprint(env)
3937
time.sleep(5)
4038

4139
configure = sh.Command('./configure')

0 commit comments

Comments
 (0)