Pr1 nodedb warmstore #10705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

NomDeTom merged 16 commits into meshtastic:develop from NomDeTom:pr1-nodedb-warmstore

Jun 18, 2026

.github/copilot-instructions.md

-Original file line number
+Diff line change
@@ Expand Up @@
     ### Eviction
-    Every code path that drops a node from the header table must also evict the satellites. The single chokepoint is `eraseNodeSatellites(NodeNum)`; it's already called from `getOrCreateMeshNode`'s oldest-boring eviction, `removeNodeByNum`, both branches of `resetNodes`, `cleanupMeshDB`, `addFromContact`'s ignored-branch, and `AdminModule`'s `set_ignored_node`. Add new eviction sites here, not by calling `.erase()` directly.
+    Every code path that drops a node from the header table must also evict the satellites. The single chokepoint is `eraseNodeSatellites(NodeNum)`; it's already called from `getOrCreateMeshNode`'s oldest-boring eviction, `demoteOldestHotNodesToWarm` (the over-cap warm-tier migration), `removeNodeByNum`, both branches of `resetNodes`, `cleanupMeshDB`, `addFromContact`'s ignored-branch, and `AdminModule`'s `set_ignored_node`. Add new eviction sites here, not by calling `.erase()` directly. (Note: `enforceSatelliteCaps`/`evictSatelliteOverCap` call `.erase()` directly on purpose — that's a satellite-only cap trim where the node _stays_ in the header, a different operation from this chokepoint.)
+    ### Warm tier (long-tail identity)
+    On every arch except STM32WL and bare nRF52832 (`WARM_NODE_COUNT > 0`), a node evicted from the header table is not forgotten outright: `WarmNodeStore` (`src/mesh/WarmNodeStore.{h,cpp}`) keeps a 40 B `{num, last_heard, public_key}` record per evicted node — primarily so PKI DMs to/from a long-tail node keep decrypting without re-running a NodeInfo exchange (the rest of `NodeInfoLite` rebuilds from traffic in seconds).
+    - **Write:** `getOrCreateMeshNode`'s eviction and `demoteOldestHotNodesToWarm` (the over-cap boot migration) call `warmStore.absorb(num, last_heard, key)` _before_ the node leaves the header.
+    - **Read-back:** `getOrCreateMeshNode` calls `warmStore.take()` to rehydrate `last_heard` + key when a warm node is re-admitted; `copyPublicKey()` falls back to the warm tier so the PKI send path finds keys for evicted peers.
+    - **Persistence:** nRF52840 uses a 12 KB raw-flash record-ring at `0xEA000` (below LittleFS; append + replay + compact-on-rotate, link-guarded by `nrf52840_s140_v7.ld` and `extra_scripts/nrf52_warm_region.py`). Everywhere else: a `/prefs/warm.dat` snapshot flushed by `saveIfDirty()` on the node-DB save cadence.
+    - **Tunables** (`mesh-pb-constants.h`): `WARM_NODE_COUNT` (per-arch; `0` disables the tier) and `MAX_NUM_NODES` (hot cap — 120 on nRF52840/generic ESP32 to fit the 28 KB LittleFS; ESP32-S3 keeps its flash-scaled 100/200/250, portduino 250). Verbose migration/self-care tracing routes through `LOG_MIGRATION`, gated by `MESHTASTIC_NODEDB_MIGRATION_VERBOSE`.
+    ### Satellite caps
+    Only the freshest `MAX_SATELLITE_NODES` nodes keep satellite payloads; the rest of the header table carries just the `NodeInfoLite`. The cap is **per-platform**: 40 on RAM-constrained parts (nRF52840, generic ESP32) since the four maps live in internal SRAM (not PSRAM, ~408 B/node across the four), and 250 on flash-rich hosts (ESP32-S3, portduino) so every hot node can carry rich data as before the cap existed. `enforceSatelliteCaps()` trims each map to the cap on load (returns whether it trimmed); `evictSatelliteOverCap()` trims before each insert. Eviction is by the owning node's hot `last_heard` (stalest first, demoted/absent nodes rank as `last_heard==0`); self is never trimmed.
+    ### On-boot self-care
+    `NodeDB::nodeDBSelfCare()` runs once identity is established (the constructor after key (re)gen, and `reloadFromDisk()` — _not_ inside `loadFromDisk`, where `getNodeNum()` is still 0). It confirms self is present (warns if a non-empty DB is missing us — a foreign/over-cap file), pins self to index 0, demotes/trims only **non-self** overflow into the warm tier, then rewrites `nodes.proto` **once** and only if it healed something — and never while encrypted storage is locked (it would persist placeholder defaults). `loadFromDisk` deliberately leaves the loaded store untrimmed for this pass.
     ### Sync flow: thin NodeInfo + post-COMPLETE_ID replay (no opt-in)
@@ Expand Down @@

extra_scripts/nrf52_warm_region.py

-Original file line number
+Diff line change
@@ -0,0 +1,70 @@
+    #!/usr/bin/env python3
+    # trunk-ignore-all(ruff/F821)
+    # trunk-ignore-all(flake8/F821): For SConstruct imports
+    #
+    # Post-link guard for the warm-node-store raw-flash region on nRF52840.
+    #
+    # The 3 app-region pages below LittleFS (0xEA000-0xED000, reclaimed by whole-image
+    # LTO) are reserved for the WarmNodeStore record-ring (see WarmNodeStore.h). Our
+    # linker scripts (nrf52840_s140_v6.ld and nrf52840_s140_v7.ld) cap the image at
+    # 0xEA000, but boards on the framework-default script (FLASH ending at 0xED000) could
+    # silently place code in those pages — the first warm-store save would then brick the
+    # device. This turns that into a build failure.
+    #
+    # Image flash end = __etext + sizeof(.data) (loaded at LMA __etext); symbols from
+    # the framework's nrf52_common.ld.
+    import os
+    Import("env")
+    WARM_REGION_BASE = 0xEA000  # keep in sync with WARM_FLASH_REGION_BASE in WarmNodeStore.h (3 x 4 KB record-ring)
+    _tc = env.PioPlatform().get_package_dir("toolchain-gccarmnoneeabi") or ""
+    _NM = os.path.join(_tc, "bin", "arm-none-eabi-nm")
+    if not os.path.isfile(_NM):
+        _NM = "arm-none-eabi-nm"  # fall back to PATH
+    def _assert_warm_region_clear(source, target, env):
+        import subprocess
+        import sys
+        try:
+            elf = env.subst("$BUILD_DIR/${PROGNAME}.elf")
+            out = subprocess.check_output([_NM, elf], universal_newlines=True)
+        except Exception as exc:  # tooling hiccup: warn loudly, don't wedge the build
+            print("nrf52_warm_region: WARNING - guard skipped (nm failed: %s)" % exc)
+            return
+        syms = {}
+        for line in out.split("\n"):
+            f = line.split()
+            if len(f) >= 3 and f[-1] in ("__etext", "__data_start__", "__data_end__"):
+                syms[f[-1]] = int(f[0], 16)
+        if len(syms) != 3:
+            print("nrf52_warm_region: WARNING - guard skipped (linker symbols not found)")
+            return
+        flash_end = syms["__etext"] + (syms["__data_end__"] - syms["__data_start__"])
+        if flash_end > WARM_REGION_BASE:
+            sys.stderr.write(
+                "\n*** nrf52 warm-region guard: image ends at 0x%X, past the reserved "
+                "warm-store region at 0x%X ***\n"
+                "The 12 KB region at 0xEA000 holds the WarmNodeStore record-ring; a warm-store\n"
+                "save would overwrite this firmware's tail. Shrink the image, or shrink/move\n"
+                "the region (WARM_FLASH_REGION_BASE in src/mesh/WarmNodeStore.h, the FLASH\n"
+                "LENGTH in src/platform/nrf52/nrf52840_s140_v6.ld and _v7.ld, and this guard).\n\n"
+                % (flash_end, WARM_REGION_BASE)
+            )
+            from SCons.Script import Exit
+            Exit(1)
+        print(
+            "nrf52_warm_region: guard OK -- image ends at 0x%X, %d KB clear of the warm region"
+            % (flash_end, (WARM_REGION_BASE - flash_end) // 1024)
+        )
+    # Attach to the phony "buildprog" alias (not the .elf node) so the guard runs
+    # on incremental relinks too -- same reasoning as nrf52_lto.py's guard.
+    env.AddPostAction("buildprog", _assert_warm_region_clear)

src/graphics/draw/MenuHandler.cpp

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1572,6 +1572,7 @@ void menuHandler::manageNodeMenu()
  
                    nodeDB->set_favorite(false, menuHandler::pickedNodeNum);

                } else {

                    LOG_INFO("Adding node %08X to favorites", menuHandler::pickedNodeNum);

                    // set_favorite() already logs PROTECTED_CAP_WARN_FMT on a cap refusal; don't double-log here.

                    nodeDB->set_favorite(true, menuHandler::pickedNodeNum);

                }

                screen->setFrames(graphics::Screen::FOCUS_PRESERVE);

    @@ -1615,15 +1616,23 @@ void menuHandler::manageNodeMenu()
  
                    return;

                }

                bool changed = false;

                if (nodeInfoLiteIsIgnored(n)) {

                    nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_IGNORED_MASK, false);

                    LOG_INFO("Unignoring node %08X", menuHandler::pickedNodeNum);

                } else {

                    nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_IGNORED_MASK, true);

                    changed = true;

                } else if (nodeDB->setProtectedFlag(n, NODEINFO_BITFIELD_IS_IGNORED_MASK, true)) {

                    LOG_INFO("Ignoring node %08X", menuHandler::pickedNodeNum);

                    changed = true;

                } else {

                    LOG_WARN(NodeDB::PROTECTED_CAP_WARN_FMT, "ignore", menuHandler::pickedNodeNum, MAX_NUM_NODES - 2);

                }

                // Only persist/notify when the ignore bit actually moved; a cap

                // refusal changed nothing and shouldn't trigger a prefs save.

                if (changed) {

                    nodeDB->notifyObservers(true);

                    nodeDB->saveToDisk();

                }

                nodeDB->notifyObservers(true);

                nodeDB->saveToDisk();

                screen->setFrames(graphics::Screen::FOCUS_PRESERVE);

                return;

            }

Navigation Menu

Uh oh!

Uh oh!

Pr1 nodedb warmstore #10705

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!