System Stepping implemented as Resource (bevyengine#8453)

# Objective

Add interactive system debugging capabilities to bevy, providing
step/break/continue style capabilities to running system schedules.

* Original implementation: bevyengine#8063
    - `ignore_stepping()` everywhere was too much complexity
* Schedule-config & Resource discussion: bevyengine#8168
    - Decided on selective adding of Schedules & Resource-based control

## Solution
Created `Stepping` Resource. This resource can be used to enable
stepping on a per-schedule basis. Systems within schedules can be
individually configured to:
* AlwaysRun: Ignore any stepping state and run every frame
* NeverRun: Never run while stepping is enabled
    - this allows for disabling of systems while debugging
* Break: If we're running the full frame, stop before this system is run

Stepping provides two modes of execution that reflect traditional
debuggers:
* Step-based: Only execute one system at a time
* Continue/Break: Run all systems, but stop before running a system
marked as Break

### Demo

https://user-images.githubusercontent.com/857742/233630981-99f3bbda-9ca6-4cc4-a00f-171c4946dc47.mov

Breakout has been modified to use Stepping. The game runs normally for a
couple of seconds, then stepping is enabled and the game appears to
pause. A list of Schedules & Systems appears with a cursor at the first
System in the list. The demo then steps forward full frames using the
spacebar until the ball is about to hit a brick. Then we step system by
system as the ball impacts a brick, showing the cursor moving through
the individual systems. Finally the demo switches back to frame stepping
as the ball changes course.


### Limitations
Due to architectural constraints in bevy, there are some cases systems
stepping will not function as a user would expect.

#### Event-driven systems
Stepping does not support systems that are driven by `Event`s as events
are flushed after 1-2 frames. Although game systems are not running
while stepping, ignored systems are still running every frame, so events
will be flushed.

This presents to the user as stepping the event-driven system never
executes the system. It does execute, but the events have already been
flushed.

This can be resolved by changing event handling to use a buffer for
events, and only dropping an event once all readers have read it.

The work-around to allow these systems to properly execute during
stepping is to have them ignore stepping:
`app.add_systems(event_driven_system.ignore_stepping())`. This was done
in the breakout example to ensure sound played even while stepping.

#### Conditional Systems
When a system is stepped, it is given an opportunity to run. If the
conditions of the system say it should not run, it will not.

Similar to Event-driven systems, if a system is conditional, and that
condition is only true for a very small time window, then stepping the
system may not execute the system. This includes depending on any sort
of external clock.

This exhibits to the user as the system not always running when it is
stepped.

A solution to this limitation is to ensure any conditions are consistent
while stepping is enabled. For example, all systems that modify any
state the condition uses should also enable stepping.

#### State-transition Systems
Stepping is configured on the per-`Schedule` level, requiring the user
to have a `ScheduleLabel`.

To support state-transition systems, bevy generates needed schedules
dynamically. Currently it’s very difficult (if not impossible, I haven’t
verified) for the user to get the labels for these schedules.

Without ready access to the dynamically generated schedules, and a
resolution for the `Event` lifetime, **stepping of the state-transition
systems is not supported**

---

## Changelog
- `Schedule::run()` updated to consult `Stepping` Resource to determine
which Systems to run each frame
- Added `Schedule.label` as a `BoxedSystemLabel`, along with supporting
`Schedule::set_label()` and `Schedule::label()` methods
- `Stepping` needed to know which `Schedule` was running, and prior to
this PR, `Schedule` didn't track its own label
- Would have preferred to add `Schedule::with_label()` and remove
`Schedule::new()`, but this PR touches enough already
- Added calls to `Schedule.set_label()` to `App` and `World` as needed
- Added `Stepping` resource
- Added `Stepping::begin_frame()` system to `MainSchedulePlugin`
    - Run before `Main::run_main()`
    - Notifies any `Stepping` Resource a new render frame is starting
    
## Migration Guide
- Add a call to `Schedule::set_label()` for any custom `Schedule`
    - This is only required if the `Schedule` will be stepped

---------

Co-authored-by: Carter Anderson <[email protected]>

Cargo.toml

@@ -71,6 +71,7 @@ default = [
   "tonemapping_luts",
   "default_font",
   "webgl2",
+  "bevy_debug_stepping",
 ]
 
 # Force dynamic linking, which improves iterative compile times
@@ -295,6 +296,9 @@ file_watcher = ["bevy_internal/file_watcher"]
 # Enables watching in memory asset providers for Bevy Asset hot-reloading
 embedded_watcher = ["bevy_internal/embedded_watcher"]
 
+# Enable stepping-based debugging of Bevy systems
+bevy_debug_stepping = ["bevy_internal/bevy_debug_stepping"]
+
 [dependencies]
 bevy_dylib = { path = "crates/bevy_dylib", version = "0.12.0", default-features = false, optional = true }
 bevy_internal = { path = "crates/bevy_internal", version = "0.12.0", default-features = false }
@@ -1566,6 +1570,17 @@ description = "Illustrates creating custom system parameters with `SystemParam`"
 category = "ECS (Entity Component System)"
 wasm = false
 
+[[example]]
+name = "system_stepping"
+path = "examples/ecs/system_stepping.rs"
+doc-scrape-examples = true
+
+[package.metadata.example.system_stepping]
+name = "System Stepping"
+description = "Demonstrate stepping through systems in order of execution"
+category = "ECS (Entity Component System)"
+wasm = false
+
 # Time
 [[example]]
 name = "time"

crates/bevy_app/Cargo.toml

@@ -11,7 +11,8 @@ keywords = ["bevy"]
 [features]
 trace = []
 bevy_ci_testing = ["serde", "ron"]
-default = ["bevy_reflect"]
+bevy_debug_stepping = []
+default = ["bevy_reflect", "bevy_debug_stepping"]
 bevy_reflect = ["dep:bevy_reflect", "bevy_ecs/bevy_reflect"]
 
 [dependencies]

crates/bevy_app/src/main_schedule.rs

@@ -256,6 +256,12 @@ impl Plugin for MainSchedulePlugin {
             .init_resource::<FixedMainScheduleOrder>()
             .add_systems(Main, Main::run_main)
             .add_systems(FixedMain, FixedMain::run_fixed_main);
+
+        #[cfg(feature = "bevy_debug_stepping")]
+        {
+            use bevy_ecs::schedule::{IntoSystemConfigs, Stepping};
+            app.add_systems(Main, Stepping::begin_frame.before(Main::run_main));
+        }
     }
 }
 

crates/bevy_ecs/Cargo.toml

@@ -12,7 +12,8 @@ categories = ["game-engines", "data-structures"]
 [features]
 trace = []
 multi-threaded = ["bevy_tasks/multi-threaded"]
-default = ["bevy_reflect"]
+bevy_debug_stepping = []
+default = ["bevy_reflect", "bevy_debug_stepping"]
 
 [dependencies]
 bevy_ptr = { path = "../bevy_ptr", version = "0.12.0" }

crates/bevy_ecs/src/schedule/executor/mod.rs

@@ -18,7 +18,12 @@ use crate::{
 pub(super) trait SystemExecutor: Send + Sync {
     fn kind(&self) -> ExecutorKind;
     fn init(&mut self, schedule: &SystemSchedule);
-    fn run(&mut self, schedule: &mut SystemSchedule, world: &mut World);
+    fn run(
+        &mut self,
+        schedule: &mut SystemSchedule,
+        skip_systems: Option<FixedBitSet>,
+        world: &mut World,
+    );
     fn set_apply_final_deferred(&mut self, value: bool);
 }
 

crates/bevy_ecs/src/schedule/executor/multi_threaded.rs

@@ -163,7 +163,12 @@ impl SystemExecutor for MultiThreadedExecutor {
         self.num_dependencies_remaining = Vec::with_capacity(sys_count);
     }
 
-    fn run(&mut self, schedule: &mut SystemSchedule, world: &mut World) {
+    fn run(
+        &mut self,
+        schedule: &mut SystemSchedule,
+        _skip_systems: Option<FixedBitSet>,
+        world: &mut World,
+    ) {
         // reset counts
         self.num_systems = schedule.systems.len();
         if self.num_systems == 0 {
@@ -181,6 +186,31 @@ impl SystemExecutor for MultiThreadedExecutor {
             }
         }
 
+        // If stepping is enabled, make sure we skip those systems that should
+        // not be run.
+        #[cfg(feature = "bevy_debug_stepping")]
+        if let Some(mut skipped_systems) = _skip_systems {
+            debug_assert_eq!(skipped_systems.len(), self.completed_systems.len());
+            // mark skipped systems as completed
+            self.completed_systems |= &skipped_systems;
+            self.num_completed_systems = self.completed_systems.count_ones(..);
+
+            // signal the dependencies for each of the skipped systems, as
+            // though they had run
+            for system_index in skipped_systems.ones() {
+                self.signal_dependents(system_index);
+            }
+
+            // Finally, we need to clear all skipped systems from the ready
+            // list.
+            //
+            // We invert the skipped system mask to get the list of systems
+            // that should be run.  Then we bitwise AND it with the ready list,
+            // resulting in a list of ready systems that aren't skipped.
+            skipped_systems.toggle_range(..);
+            self.ready_systems &= skipped_systems;
+        }
+
         let thread_executor = world
             .get_resource::<MainThreadExecutor>()
             .map(|e| e.0.clone());

crates/bevy_ecs/src/schedule/executor/simple.rs

@@ -30,7 +30,20 @@ impl SystemExecutor for SimpleExecutor {
         self.completed_systems = FixedBitSet::with_capacity(sys_count);
     }
 
-    fn run(&mut self, schedule: &mut SystemSchedule, world: &mut World) {
+    fn run(
+        &mut self,
+        schedule: &mut SystemSchedule,
+        _skip_systems: Option<FixedBitSet>,
+        world: &mut World,
+    ) {
+        // If stepping is enabled, make sure we skip those systems that should
+        // not be run.
+        #[cfg(feature = "bevy_debug_stepping")]
+        if let Some(skipped_systems) = _skip_systems {
+            // mark skipped systems as completed
+            self.completed_systems |= &skipped_systems;
+        }
+
         for system_index in 0..schedule.systems.len() {
             #[cfg(feature = "trace")]
             let name = schedule.systems[system_index].name();

crates/bevy_ecs/src/schedule/executor/single_threaded.rs

@@ -38,7 +38,20 @@ impl SystemExecutor for SingleThreadedExecutor {
         self.unapplied_systems = FixedBitSet::with_capacity(sys_count);
     }
 
-    fn run(&mut self, schedule: &mut SystemSchedule, world: &mut World) {
+    fn run(
+        &mut self,
+        schedule: &mut SystemSchedule,
+        _skip_systems: Option<FixedBitSet>,
+        world: &mut World,
+    ) {
+        // If stepping is enabled, make sure we skip those systems that should
+        // not be run.
+        #[cfg(feature = "bevy_debug_stepping")]
+        if let Some(skipped_systems) = _skip_systems {
+            // mark skipped systems as completed
+            self.completed_systems |= &skipped_systems;
+        }
+
         for system_index in 0..schedule.systems.len() {
             #[cfg(feature = "trace")]
             let name = schedule.systems[system_index].name();

crates/bevy_ecs/src/schedule/mod.rs

@@ -8,6 +8,7 @@ mod graph_utils;
 mod schedule;
 mod set;
 mod state;
+mod stepping;
 
 pub use self::condition::*;
 pub use self::config::*;
@@ -1098,4 +1099,60 @@ mod tests {
             assert!(schedule.graph().conflicting_systems().is_empty());
         }
     }
+
+    #[cfg(feature = "bevy_debug_stepping")]
+    mod stepping {
+        use super::*;
+        use bevy_ecs::system::SystemState;
+
+        #[derive(ScheduleLabel, Clone, Debug, PartialEq, Eq, Hash)]
+        pub struct TestSchedule;
+
+        macro_rules! assert_executor_supports_stepping {
+            ($executor:expr) => {
+                // create a test schedule
+                let mut schedule = Schedule::new(TestSchedule);
+                schedule
+                    .set_executor_kind($executor)
+                    .add_systems(|| panic!("Executor ignored Stepping"));
+
+                // Add our schedule to stepping & and enable stepping; this should
+                // prevent any systems in the schedule from running
+                let mut stepping = Stepping::default();
+                stepping.add_schedule(TestSchedule).enable();
+
+                // create a world, and add the stepping resource
+                let mut world = World::default();
+                world.insert_resource(stepping);
+
+                // start a new frame by running ihe begin_frame() system
+                let mut system_state: SystemState<Option<ResMut<Stepping>>> =
+                    SystemState::new(&mut world);
+                let res = system_state.get_mut(&mut world);
+                Stepping::begin_frame(res);
+
+                // now run the schedule; this will panic if the executor doesn't
+                // handle stepping
+                schedule.run(&mut world);
+            };
+        }
+
+        /// verify the [`SimpleExecutor`] supports stepping
+        #[test]
+        fn simple_executor() {
+            assert_executor_supports_stepping!(ExecutorKind::Simple);
+        }
+
+        /// verify the [`SingleThreadedExecutor`] supports stepping
+        #[test]
+        fn single_threaded_executor() {
+            assert_executor_supports_stepping!(ExecutorKind::SingleThreaded);
+        }
+
+        /// verify the [`MultiThreadedExecutor`] supports stepping
+        #[test]
+        fn multi_threaded_executor() {
+            assert_executor_supports_stepping!(ExecutorKind::MultiThreaded);
+        }
+    }
 }

crates/bevy_ecs/src/schedule/schedule.rs

@@ -25,6 +25,8 @@ use crate::{
     world::World,
 };
 
+pub use stepping::Stepping;
+
 /// Resource that stores [`Schedule`]s mapped to [`ScheduleLabel`]s excluding the current running [`Schedule`].
 #[derive(Default, Resource)]
 pub struct Schedules {
@@ -238,6 +240,11 @@ impl Schedule {
         }
     }
 
+    /// Get the `InternedScheduleLabel` for this `Schedule`.
+    pub fn label(&self) -> InternedScheduleLabel {
+        self.label
+    }
+
     /// Add a collection of systems to the schedule.
     pub fn add_systems<M>(&mut self, systems: impl IntoSystemConfigs<M>) -> &mut Self {
         self.graph.process_configs(systems.into_configs(), false);
@@ -324,7 +331,17 @@ impl Schedule {
         world.check_change_ticks();
         self.initialize(world)
             .unwrap_or_else(|e| panic!("Error when initializing schedule {:?}: {e}", self.label));
-        self.executor.run(&mut self.executable, world);
+
+        #[cfg(not(feature = "bevy_debug_stepping"))]
+        let skip_systems = None;
+
+        #[cfg(feature = "bevy_debug_stepping")]
+        let skip_systems = match world.get_resource_mut::<Stepping>() {
+            None => None,
+            Some(mut stepping) => stepping.skipped_systems(self),
+        };
+
+        self.executor.run(&mut self.executable, skip_systems, world);
     }
 
     /// Initializes any newly-added systems and conditions, rebuilds the executable schedule,
@@ -366,6 +383,11 @@ impl Schedule {
         &mut self.graph
     }
 
+    /// Returns the [`SystemSchedule`].
+    pub(crate) fn executable(&self) -> &SystemSchedule {
+        &self.executable
+    }
+
     /// Iterates the change ticks of all systems in the schedule and clamps any older than
     /// [`MAX_CHANGE_AGE`](crate::change_detection::MAX_CHANGE_AGE).
     /// This prevents overflow and thus prevents false positives.
@@ -402,6 +424,36 @@ impl Schedule {
             system.apply_deferred(world);
         }
     }
+
+    /// Returns an iterator over all systems in this schedule.
+    ///
+    /// Note: this method will return [`ScheduleNotInitialized`] if the
+    /// schedule has never been initialized or run.
+    pub fn systems(
+        &self,
+    ) -> Result<impl Iterator<Item = (NodeId, &BoxedSystem)> + Sized, ScheduleNotInitialized> {
+        if !self.executor_initialized {
+            return Err(ScheduleNotInitialized);
+        }
+
+        let iter = self
+            .executable
+            .system_ids
+            .iter()
+            .zip(&self.executable.systems)
+            .map(|(node_id, system)| (*node_id, system));
+
+        Ok(iter)
+    }
+
+    /// Returns the number of systems in this schedule.
+    pub fn systems_len(&self) -> usize {
+        if !self.executor_initialized {
+            self.graph.systems.len()
+        } else {
+            self.executable.systems.len()
+        }
+    }
 }
 
 /// A directed acyclic graph structure.
@@ -1939,6 +1991,12 @@ impl ScheduleBuildSettings {
     }
 }
 
+/// Error to denote that [`Schedule::initialize`] or [`Schedule::run`] has not yet been called for
+/// this schedule.
+#[derive(Error, Debug)]
+#[error("executable schedule has not been built")]
+pub struct ScheduleNotInitialized;
+
 #[cfg(test)]
 mod tests {
     use crate::{