Add distributed mesh support and fake neighbor handling in BreakMeshByBlock #31636

ChengHauYang · 2025-09-30T15:59:24Z

closes #12033
closes #21161

Reason

Currently, BreakMeshByBlock (bmbb) uses a hacky approach to prepare the mesh.
When nodes are duplicated by bmbb, their former neighbors are no longer neighbors (since neighborhood is based on shared nodes). As a result:

We cannot call prepare_for_use() at the end of bmbb, because it would erase the discontinuous neighbor information.
To work around this, the framework has special hacks for bmbb (e.g., skipping mesh validity checks in debug mode when bmbb is present).

Another limitation is that bmbb cannot run with distributed meshes. This PR addresses that.

Design

Preserving discontinuous neighbors
- Since prepare_for_use() wipes out discontinuous neighbor data, we introduce a new data structure to store this information (referred to as fake neighbors).
- These fake neighbors remain accessible to interface kernels that require them (e.g., CZM).
Support for distributed meshes
Following Roy’s suggestions:
- Unique node IDs: In replicated meshes, new nodes were created with
```
Node::build(*current_node, mesh->n_nodes()).release();
```
  This fails in distributed meshes.
  -> Solution: use communication between processors and assign deterministic IDs based on max_node_id strides.
- Ghost node handling:
  - Phase 0: Each rank computes locally connected blocks for its owned nodes and sends this info to the node owner.
  - Phase 1: Node owners aggregate connected block info and broadcast the global set to all ranks.
- Boundary ID assignment: Updated in line with Roy’s fix (commit link).

Impact

bmbb now supports distributed meshes.
prepare_for_use() can be safely called at the end of bmbb.
Removes the need for special hacks in the MOOSE framework for bmbb.

hugary1995

Some early comments.

hugary1995 · 2025-09-30T21:26:05Z

framework/include/loops/ThreadedElementLoopBase.h

+          const Elem * neighbor_ptr = elem->neighbor_ptr(side);
+          const Elem * neighbor =
+              (neighbor_ptr) ? neighbor_ptr : _mesh.neighbor_fake_ptr(elem, side);


Suggested change

const Elem * neighbor_ptr = elem->neighbor_ptr(side);

const Elem * neighbor =

(neighbor_ptr) ? neighbor_ptr : _mesh.neighbor_fake_ptr(elem, side);

const auto * neighbor_ptr = elem->neighbor_ptr(side) || _mesh.neighbor_fake_ptr(elem, side);

Might have to add an assertion below to make sure it's nullptr.

Thank you for pointing this out.

Using || in this context does not work because in C++ it always evaluates to a bool, not a pointer.

As a result, I go with

const auto * neighbor = elem->neighbor_ptr(side) ? elem->neighbor_ptr(side) : _mesh.neighbor_fake_ptr(elem, side);

hugary1995 · 2025-09-30T21:41:58Z

framework/include/mesh/MooseMesh.h

+  void
+  setElemSideToFakeNeighborElemSideMap(std::unordered_map<std::pair<dof_id_type, unsigned int>,
+                                                          std::pair<dof_id_type, unsigned int>> map)
+  {
+    _elemid_side_to_fake_neighbor_elemid_side = std::move(map);
+  }


We need a more fine-grained API to modify the map.

Thanks for the suggestion! I’ve extended the interface to provide a more fine-grained API:

addFakeNeighbor: add or update a single (elem, side) entry

removeFakeNeighbor: remove a single entry if it exists

hasFakeNeighbor: check whether a mapping exists

getFakeNeighborMap: accessor for read-only access to the whole map

setFakeNeighborMap: still available for bulk replacement

framework/include/mesh/MooseMesh.h

hugary1995 · 2025-09-30T21:43:16Z

framework/include/mesh/MooseMesh.h

+    _elemid_side_to_fake_neighbor_elemid_side.clear();
+  }
+
+  const Elem * neighbor_fake_ptr(const Elem * elem, unsigned int side) const


Suggested change

const Elem * neighbor_fake_ptr(const Elem * elem, unsigned int side) const

const Elem * fake_neighbor_ptr(const Elem * elem, unsigned int side) const

Also, perhaps we should come up with a better name than "fake". Perhaps "additional_neighbor_ptr"? I'll take suggestions from @roystgnr and @lindsayad

I currently just rename it to be fake_neighbor_ptr . Let's wait for the suggestions.

hugary1995 · 2025-09-30T21:43:41Z

framework/include/mesh/MooseMesh.h

+      return nullptr;
+  }
+
+  int neighbor_fake_side(const Elem * elem, unsigned int side) const


similar comment on naming.

Change it to be fake_neighbor_side

hugary1995 · 2025-09-30T21:55:27Z

framework/include/meshgenerators/BreakMeshByBlockGenerator.h

+  std::unordered_map<ElemIDSidePair, ElemIDSidePair> _elemid_side_to_fake_neighbor_elemid_side;
+
+  /// generate the new boundary interface
+  void addInterfaceBoundary(MeshBase & mesh);


I suggest renaming this to addInterface.

hugary1995 · 2025-09-30T21:55:37Z

framework/include/meshgenerators/BreakMeshByBlockGenerator.h

+  std::map<dof_id_type, std::set<subdomain_id_type>> _nodeid_to_connected_blocks;
+
+  /// @brief a set of pairs of block ids between which new boundary sides are created
+  std::set<std::pair<subdomain_id_type, subdomain_id_type>> _new_boundary_sides_list;


I prefer the old name _neighboring_block_list. Why was the renaming, is it now serving different purposes?

This name I grabbed from Roy's implementation: roystgnr@2583950. I will use _neighboring_block_list to replace this.

hugary1995 · 2025-09-30T22:04:01Z

framework/include/meshgenerators/BreakMeshByBlockGeneratorBase.h

+  void findBoundaryName(MeshBase & mesh,
+                        const subdomain_id_type & /*primaryBlockID*/,
+                        const subdomain_id_type & /*secondaryBlockID*/,
+                        std::string & /*boundaryName*/,
+                        const boundary_id_type & /*boundaryID*/,
+                        BoundaryInfo & /*boundary_info*/);


Suggested change

void findBoundaryName(MeshBase & mesh,

const subdomain_id_type & /*primaryBlockID*/,

const subdomain_id_type & /*secondaryBlockID*/,

std::string & /*boundaryName*/,

const boundary_id_type & /*boundaryID*/,

BoundaryInfo & /*boundary_info*/);

BoundaryID getBoundaryIDFromBlockPair(MeshBase & mesh,

const subdomain_id_type & /*primaryBlockID*/,

const subdomain_id_type & /*secondaryBlockID*/);

hugary1995 · 2025-09-30T22:09:00Z

framework/include/utils/MooseMeshUtils.h

 // Used to temporarily store information about which lower-dimensional
 // sides to add and what subdomain id to use for the added sides.
-struct ElemSidePair
+struct ElemIDSidePair


See if you are reinventing BndElement...

I actually wrongly changed here. This is the original implementation inside MOOSE. I will revert this back, and on my side, I will try to use BndElement .

hugary1995 · 2025-09-30T22:09:20Z

framework/src/base/MeshGeneratorSystem.C

-  if (type == "BreakMeshByBlockGenerator")
-    _has_bmbb = true;


moosebuild · 2025-10-02T00:55:03Z

Job Documentation, step Docs: sync website on 9f29e32 wanted to post the following:

View the site here

This comment will be updated on new commits.

libmesh

GiudGiud

early review to keep up to date
looks good

GiudGiud · 2025-11-03T18:59:15Z

framework/include/meshgenerators/BreakMeshByBlockGenerator.h

+  /// @brief a set of pairs of block ids between which new boundary sides are created
  std::set<std::pair<subdomain_id_type, subdomain_id_type>> _neighboring_block_list;
+
+  /// @brief a map from a pair of block ids to a set of element and side pairs


Suggested change

/// @brief a map from a pair of block ids to a set of element and side pairs

/// @brief a map from a pair of block ids to a set of pairs of an element and a side

GiudGiud · 2025-11-03T19:00:06Z

framework/include/meshgenerators/BreakMeshByBlockGenerator.h

+           std::set<std::pair<const Elem *, unsigned int>>>
      _new_boundary_sides_map;
+
+  /// @brief Maps an element pointer and side pair to its corresponding fake neighbor element pointer and side pair.


fake? Why not "disconnected" ?
A disconnected neighbor would make sense to me

This was part of the legacy implementation. I have removed this member map in the latest commit.

GiudGiud · 2025-11-03T19:08:04Z

framework/include/meshgenerators/BreakMeshByBlockGenerator.h

+  std::unordered_map<std::pair<const Elem *, unsigned int>, std::pair<const Elem *, unsigned int>>
+      _elem_side_to_fake_neighbor_elem_side;
+
+  /// @brief Maps a pair of subdomain ids to the corresponding boundary id.


is that an extended map? that includes the disconnected / other-side of disconnection boundaries?

the MooseMesh has such a map already

This was part of the legacy implementation. I have removed this member map in the latest commit.

GiudGiud · 2025-11-03T19:08:54Z

framework/include/relationshipmanagers/FakeNeighborRM.h

@@ -0,0 +1,33 @@
+#pragma once


Suggested change

#pragma once

//* This file is part of the MOOSE framework

//* https://mooseframework.inl.gov

//*

//* All rights reserved, see COPYRIGHT for full restrictions

//* https://github.com/idaholab/moose/blob/master/COPYRIGHT

//*

//* Licensed under LGPL 2.1, please see LICENSE for details

//* https://www.gnu.org/licenses/lgpl-2.1.html

#pragma once

GiudGiud · 2025-11-03T19:24:12Z

modules/solid_mechanics/test/tests/cohesive_zone_model/poly.msh

Yes. I have removed that. Thanks for careful checkings!

GiudGiud · 2025-11-03T19:55:39Z

framework/src/meshgenerators/BreakMeshByBlockGenerator.C

+      fake_neighbor_rm->setFakeNeighborMap(elem_side_to_fake_neighbor_elem_side);
+    }
+    else
+      mooseError("Failed to cast to FakeNeighborRM.");


Suggested change

mooseError("Failed to cast to FakeNeighborRM.");

mooseAssert(false, "Failed to cast to FakeNeighborRM.");

As mentioned above, I moved the ghost-element addition from MOOSE into libMesh.

sounds good. feel free to copy paste that comment where needed and I ll resolve

GiudGiud · 2025-11-03T19:56:42Z

framework/src/meshgenerators/BreakMeshByBlockGenerator.C

-        bool should_create_new_node = true;
-        if (_block_pairs_restricted)
+        // Check if this node is exactly at the boundary between two blocks
+        // If it is junction between more than two blocks, we do not split it


seems like a difference between the two modes for this object? w/ and w/o specifying subdomains

Yes, that’s intentional and matches the original documented behavior in BreakMeshByBlockGenerator.
When a node is shared by more than two subdomains, it will not be duplicated.
For the block_pairs option, only nodes shared by the specified block pairs are newly created.

GiudGiud · 2025-11-03T19:58:15Z

framework/src/meshgenerators/BreakMeshByBlockGenerator.C

+                boundary_info.add_node(new_node, node_boundary_ids);
              }
+              multiplicity_counter--; // node created, update multiplicity counter
+              break; // ones the proper node has been fixed in one element we can break the


Suggested change

break; // ones the proper node has been fixed in one element we can break the

break; // once the proper node has been fixed in one element we can break the

GiudGiud · 2025-11-03T20:01:12Z

framework/src/meshgenerators/BreakMeshByBlockGenerator.C

-          {
-            // assign the newly added node to current_elem
-            Node * new_node = nullptr;
+        if ((block_id != reference_subdomain_id) ||


Suggested change

if ((block_id != reference_subdomain_id) ||

// Work on duplicating the node only from the element that is not on the reference subdomain

if ((block_id != reference_subdomain_id) ||

Added. Thanks!

GiudGiud · 2025-11-03T20:03:41Z

framework/src/meshgenerators/BreakMeshByBlockGenerator.C

+              blockRestrictedElementSubdomainID(connected_elem);

-            if (current_elem != connected_elem && curr_elem_subid < connected_elem_subid)
+          if (current_elem != connected_elem && curr_elem_subid < connected_elem_subid)


add a comment

I d probably move current_elem != connected_elem higher to, with an explicit "continue;"

Thank you, @GiudGiud! I added a higher early out.

…hbor RM 1. Distributed mesh fixes - Node ID assignment: using `mesh->n_nodes()` fails for distributed meshes (IDs are sparse). Switch to `max_node_id` with a stride. - Ghost node handling: - *Phase 0*: Each rank computes locally connected blocks for its owned nodes and sends them to the node owner. - *Phase 1*: Node owners aggregate connected block info and broadcast the global set to all ranks. - Boundary ID assignment aligned with Roy’s fix ([commit link](roystgnr@2583950)). 2. New FakeNeighbor datastructure - Introduce a RelationshipManager for ghosted fake neighbors (CZM discontinuous neighbors, not topological). - Robust initialization: - Override `set_mesh` to ensure `_functor` is created JIT on first use. - Rebuild `_functor` in `internalInitWithMesh` if the neighbor map was populated after an early empty init. - Handles mesh cloning (`SetupMeshAction`) cases where `set_mesh` is called before initialization. 3. Simplifications - Since we now call `prepare_for_use()` at the end of BMBb, remove BreakMeshByBlockGenerator-specific hacks. - Simplify node ID handling. - Drop `MeshGeneratorSystem::_has_bmbb` flag and `hasBreakMeshByBlockGenerator()` API. # Conflicts: # framework/src/base/MeshGeneratorSystem.C

Fixes crashes from map::at when nodes had no valid blocks. Each node now gets an entry (even empty), keeping the map consistent with node_to_elem_map.

… nodes

…ributed mode. The crash was caused by an incorrect decision on node duplication, which stemmed from using incomplete, processor-local data (`node_to_elem_map`). The fix ensures the decision is based on globally synchronized data (`_nodeid_to_connected_blocks`), guaranteeing a consistent and correct outcome across all processors.

… node is in a junction with more than two blocks that share the node. It only happens that the node is in the interface of two blocks and has the block IDs equal to what the user set.

…estriction for bmbb tests (2) regold for the bmbb tests

Refactors the disconnected neighbor mechanism to be robust and safe under parallel (MPI) execution. Previously, the implementation stored raw `Elem*` pointers in a map, which was fundamentally unsafe in a distributed environment. Pointers are only valid on the local rank and become meaningless across MPI processes, leading to segmentation faults in multi-process runs. This commit resolves the issue by transitioning to an ID-based design: - Core map (`_disconnected_neighbors_by_id`) now stores `dof_id_type` pairs instead of raw `ConstBndElement` pointers, ensuring portability and serializability across ranks. - The `disconnectedNeighbor` accessor now returns a `std::optional<ConstBndElement>`, replacing unsafe raw pointers with a safer C++ interface that naturally handles missing neighbors. **Key Learning:** For parallel-safe design, always **store globally unique IDs** and resolve them into rank-local pointers, which avoids pointer validity issues and ensures correctness across distributed executions.

- Replaced raw Elem* usage with ID-based ElemSide pairs for safer parallel handling - Added cache (_cached_disconnected_neighbors) to speed up repeated lookups - Modified neighbor queries across loops (NonlinearThread, ComputeUserObjectsThread, ComputeMaterialsObjectThread, FEProblemBase, etc.) to use disconnectedNeighborPtr

moosebuild · 2025-11-05T15:42:19Z

libmesh

@@ -1 +1 @@
-Subproject commit 910736a3a20aded528c3ba99371ce65980dc8e2c
+Subproject commit d615ffee1aca885edd568b59d1b8dd969eb434c2
