Skip to content

docs: update test instructions to prefer bin/run-tests.sh; #10734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thebentern merged 3 commits into from
Jun 25, 2026
Merged

docs: update test instructions to prefer bin/run-tests.sh; #10734

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
90f01c2
docs: update test instructions to prefer bin/run-tests.sh; fix suite …
NomDeTom Jun 17, 2026
0094ad0
address copilot review
NomDeTom Jun 25, 2026
5a81fb5
fix lint
NomDeTom Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 72 additions & 7 deletions .github/copilot-instructions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
# Meshtastic Firmware - Copilot Instructions

> **TL;DR**
>
> | | |
> | -------------- | -------------------------------------------------------------------------------------------- |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) |
> | Hardware tests | `./mcp-server/run-tests.sh` |
> | Format | `trunk fmt` |
> | Mirror docs | `AGENTS.md` (short pointer for agents that don't read this file) · `CLAUDE.md` (Claude Code) |
>
> **Need this? It's here.**
>
> | | |
> | ------------------------------------------- | ---------------------------------------------------------- |
> | General helpers (clamp, UTF-8, string fmt…) | `src/meshUtils.h` |
> | Logging macros (LOG_DEBUG / INFO / WARN…) | `src/DebugConfiguration.h` |
> | New module skeleton | inherit `ProtobufModule<T>` in `src/mesh/ProtobufModule.h` |
> | Observer / event wiring | `src/Observer.h` |

This document provides context and guidelines for AI assistants working with the Meshtastic firmware codebase.

## Project Overview
Expand Down Expand Up @@ -37,7 +55,7 @@ MQTT provides a bridge between Meshtastic mesh networks and the internet, enabli

Messages are published/subscribed using a hierarchical topic format:

```
```text
{root}/{channel_id}/{gateway_id}
```

Expand Down Expand Up @@ -247,7 +265,7 @@ Unit tests for the conversion layer live in `test/test_type_conversions/test_mai

## Project Structure

```
```text
firmware/
├── src/ # Main source code
│ ├── main.cpp # Application entry point
Expand Down Expand Up @@ -482,7 +500,7 @@ Key defines in variant.h:

## Build System

## Agent Tooling Baseline
### Agent Tooling Baseline

Mirror counterpart: `AGENTS.md` under **Agent Tooling Baseline**.

Expand Down Expand Up @@ -635,35 +653,82 @@ Most workflows can be triggered manually via `workflow_dispatch` for testing.

### Native unit tests (C++)

Unit tests in `test/` directory with 17 test suites:
Unit tests in `test/` directory. The canonical suite count is in `test/native-suite-count` and is cross-checked on every full run. Current suites:

- `test_admin_radio/` - LoRa region/config validation and AdminModule dispatch
- `test_atak/` - ATAK integration
- `test_crypto/` - Cryptography
- `test_default/` - Default configuration
- `test_hop_scaling/` - Hop scaling histogram and required-hop logic
- `test_http_content_handler/` - HTTP handling
- `test_mac_from_string/` - MAC address parsing
- `test_mesh_module/` - Module framework
- `test_meshpacket_serializer/` - Packet serialization
- `test_mqtt/` - MQTT integration
- `test_nexthop_routing/` - Next-hop routing logic
- `test_nodedb_blocked/` - NodeDB blocked-node handling
- `test_packet_history/` - Packet history tracking
- `test_packet_signing/` - Packet signing
- `test_position_module/` - Position module behaviour
- `test_position_precision/` - Position precision helpers
- `test_radio/` - Radio interface
- `test_rtc/` - RTC / time handling
- `test_serial/` - Serial communication
- `test_traffic_management/` - Traffic management
- `test_traffic_management/` - Traffic management (dedup, rate-limit, hop-trim, role exceptions)
- `test_transmit_history/` - Retransmission tracking
- `test_type_conversions/` - NodeDB v25 type conversion (bitfield round-trips, NodeInfoLite)
- `test_utf8/` - UTF-8 utilities
- `test_warm_store/` - Warm-tier node store

**Preferred run command — `bin/run-tests.sh`** (uses the `coverage` env with ASan/LSan sanitizers; emits a machine-readable verdict on the final line; update `test/native-suite-count` when adding or removing suites):

```bash
./bin/run-tests.sh # all suites
./bin/run-tests.sh -f test_traffic_management # single suite (yields FILTERED, not GREEN)
```

Exit codes and verdicts (exact counts will vary; examples below are illustrative):

| Exit | Verdict | Meaning |
| ---- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | `GREEN` | All canonical suites ran, all passed, no ignored test cases |
| 1 | `RED` | At least one failure, build error, or sanitizer fault |
| 2 | `AMBER` | All that ran passed, but something was lost: a suite silently went missing on a full run, individual test cases were skipped (`TEST_IGNORE`), or `test/native-suite-count` disagrees with the `test/` directory count |
| 3 | `FILTERED` | A `-f` run completed cleanly; suites outside the filter were intentionally not run |

Examples — exact counts will vary by suite count and env:

```text
# GREEN: all suites ran and passed
RESULT: GREEN N/N suites passed [canonical: N/N]

# RED: real test failure
RESULT: RED 1 failed

# RED: sanitizer exit-time abort (all tests passed but process aborted at exit)
RESULT: RED exit-time abort (tests passed; likely sanitizer — see hint above)

# AMBER: native-suite-count disagrees with test/ directory count (too low)
RESULT: AMBER test/ has 24 suite directories but native-suite-count says 5 — update test/native-suite-count after registering new suites

# AMBER: native-suite-count disagrees with test/ directory count (too high)
RESULT: AMBER test/ has 24 suite directories but native-suite-count says 99 — update test/native-suite-count after removing suites

# FILTERED: single suite run completed cleanly
RESULT: FILTERED 1/24 suites ran (not run: test_admin_radio test_atak …) — filtered: test_serial [canonical: 1/24]
```

> **Copilot interface note:** When running tests via the Copilot chat interface, edits made through the chat may not be reflected in the on-disk files that the test binary reads. If tests pass in chat but fail locally (or vice versa), verify the files on disk match what you expect before trusting the result. Always confirm with a local terminal run.

Run command (preferred — avoids pipe-buffering and the Ubuntu externally-managed-environment error):
Raw `pio test` (no sanitizers, no verdict logic) — use only when you need to override the env:

```bash
~/.platformio/penv/bin/python -m platformio test -e native -f test_your_suite > /tmp/test_out.txt 2>&1
grep -E 'error:|PASS|FAIL|succeeded|failed' /tmp/test_out.txt
tail -15 /tmp/test_out.txt
```

Do **not** use `pio test … | tail -N` — it discards build errors and shows stale cached results. Do **not** use `pio test … | grep` — line-buffering makes the terminal appear hung until the process exits. Redirect to a file first, then grep.
Do **not** pipe `pio test` — line-buffering makes the terminal appear hung and hides build errors.

Simulation testing: `bin/test-simulator.sh`

Expand Down
44 changes: 31 additions & 13 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
# Agent instructions

> **TL;DR**
>
> | | |
> | -------------- | ------------------------------------------------------------------------- |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) |
> | Hardware tests | `./mcp-server/run-tests.sh` |
> | Format | `trunk fmt` |
> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `CLAUDE.md` (Claude Code) |
>
> **Need this? It's here.**
>
> | | |
> | ------------------------------------------- | ---------------------------------------------------------- |
> | General helpers (clamp, UTF-8, string fmt…) | `src/meshUtils.h` |
> | Logging macros (LOG_DEBUG / INFO / WARN…) | `src/DebugConfiguration.h` |
> | New module skeleton | inherit `ProtobufModule<T>` in `src/mesh/ProtobufModule.h` |
> | Observer / event wiring | `src/Observer.h` |

This repository is the [Meshtastic](https://meshtastic.org) firmware — a C++17 embedded codebase targeting ESP32 / nRF52 / RP2040 / STM32WL / Linux-Portduino LoRa mesh radios — plus a Python MCP server in `mcp-server/` that AI agents use to flash, configure, and test connected devices.

## Primary instruction file
Expand All @@ -10,18 +28,18 @@ This file (`AGENTS.md`) is a short pointer + quick reference for agents that don

## Quick command reference

| Action | Command |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Build a firmware variant | `pio run -e <env>` (e.g. `pio run -e rak4631`, `pio run -e heltec-v3`) |
| Build native macOS host binary | `pio run -e native-macos` (Homebrew prereqs + CH341 LoRa setup in `variants/native/portduino/platformio.ini`) |
| Clean + rebuild | `pio run -e <env> -t clean && pio run -e <env>` |
| Flash a device | `pio run -e <env> -t upload --upload-port <port>` (or use the `pio_flash` MCP tool) |
| Run firmware unit tests (native) | `~/.platformio/penv/bin/python -m platformio test -e native > /tmp/test_out.txt 2>&1` then `grep -E 'error:\|PASS\|FAIL\|succeeded\|failed' /tmp/test_out.txt` (redirect first — piping causes line-buffering) |
| Run MCP hardware tests | `./mcp-server/run-tests.sh` |
| Live TUI test runner | `mcp-server/.venv/bin/meshtastic-mcp-test-tui` |
| Format before commit | `trunk fmt` |
| Regenerate protobuf bindings | `bin/regen-protos.sh` |
| Generate CI matrix | `./bin/generate_ci_matrix.py all [--level pr]` |
| Action | Command |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Build a firmware variant | `pio run -e <env>` (e.g. `pio run -e rak4631`, `pio run -e heltec-v3`) |
| Build native macOS host binary | `pio run -e native-macos` (Homebrew prereqs + CH341 LoRa setup in `variants/native/portduino/platformio.ini`) |
| Clean + rebuild | `pio run -e <env> -t clean && pio run -e <env>` |
| Flash a device | `pio run -e <env> -t upload --upload-port <port>` (or use the `pio_flash` MCP tool) |
| Run firmware unit tests (native) | `./bin/run-tests.sh` (preferred — ASan/LSan + RED/AMBER/GREEN verdict); or raw: `~/.platformio/penv/bin/python -m platformio test -e native > /tmp/test_out.txt 2>&1` |
| Run MCP hardware tests | `./mcp-server/run-tests.sh` |
| Live TUI test runner | `mcp-server/.venv/bin/meshtastic-mcp-test-tui` |
| Format before commit | `trunk fmt` |
| Regenerate protobuf bindings | `bin/regen-protos.sh` |
| Generate CI matrix | `./bin/generate_ci_matrix.py all [--level pr]` |

## MCP server (device + test automation)

Expand Down Expand Up @@ -108,7 +126,7 @@ Sequence these; don't parallelize on the same port.
| `src/modules/` | Feature modules; `Telemetry/Sensor/` has 50+ I2C sensor drivers |
| `variants/` | 200+ hardware variant definitions (`variant.h` + `platformio.ini` per board) |
| `protobufs/` | `.proto` definitions; regenerate with `bin/regen-protos.sh` |
| `test/` | Firmware unit tests (17 suites; `pio test -e native`) |
| `test/` | Firmware unit tests (19 suites; `./bin/run-tests.sh` preferred, falls back to `pio test -e native`) |
| `mcp-server/` | Python MCP server + pytest hardware integration tests |
| `mcp-server/tests/` | Tiered pytest suite: `unit/`, `mesh/`, `telemetry/`, `monitor/`, `recovery/`, `ui/`, `fleet/`, `admin/`, `provisioning/` |
| `.claude/commands/` | Claude Code slash command bodies |
Expand Down
23 changes: 23 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Claude Code instructions

> **TL;DR**
>
> | | |
> | -------------- | ------------------------------------------------------------------ |
> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) |
> | Hardware tests | `./mcp-server/run-tests.sh` |
> | Format | `trunk fmt` |
> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `AGENTS.md` |
>
> **Need this? It's here.**
>
> | | |
> | ------------------------------------------- | ---------------------------------------------------------- |
> | General helpers (clamp, UTF-8, string fmt…) | `src/meshUtils.h` |
> | Logging macros (LOG_DEBUG / INFO / WARN…) | `src/DebugConfiguration.h` |
> | New module skeleton | inherit `ProtobufModule<T>` in `src/mesh/ProtobufModule.h` |
> | Observer / event wiring | `src/Observer.h` |

**Read `.github/copilot-instructions.md` first.** That file is the canonical agent-facing document for this repo. It covers project layout, coding conventions, the build system, CI/CD, the native C++ test suite, and the MCP Server & Hardware Test Harness. Read it top-to-bottom before starting any non-trivial change.

This file (`CLAUDE.md`) is a short pointer for Claude Code sessions. Slash commands live in `.claude/commands/`.
Loading
Loading