Allow systems to execute with a fixed timestep

[reidbhuntley]

In order to have objects move at the same speed regardless of FPS while still keeping a game's logic deterministic, we need some way to let users mark systems to be run with a fixed timestep. (See the classic Gaffer on Games article here)

Currently a user could achieve this by creating an accumulator resource inside a system and running the system's logic multiple times per call based on that accumulator. However, this stops working correctly when you try it with multiple systems at once. If systems A and B should both run with a fixed timestep, and B is dependent on A, then they should always run one after another (ABABAB); but using this method would first make A run a few times, then B run a few times (AAA...BBB...), so their order would depend on the amount of time in the accumulator, and thus be nondeterministic.

To solve this, I think we should create a second scheduler and push systems to there that a user wants to run with a fixed timestep. On each frame that entire schedule would be executed some number of times based on the amount of time in an accumulator, and it would execute before the main scheduler. This approach of running fixed-timestep logic before other logic is how other engines like Godot and Unity have handled this issue.

Additionally, we'd need some way in the API to configure what the timestep actually is, as well as a way for systems to read that value, ideally through the Time resource. A field containing a float between 0 and 1 indicating how far through the current timestep we are would also be helpful, since this would allow rendered objects to update faster than the fixed timestep through the use of interpolation.

[cart]

Agreed. This is a "must have" feature so we should take the time to get it right.

I think we could approach this from a few angles:

1. Add support for multiple schedules in Apps and give each schedule its own run() function (which could then perform the time step work).
2. Have one global schedule and allow for "sub schedules" via resources and stages:

• Ex: create a new "physics" stage in the global schedule, add a PhysicsSchedule resource, then add this system to the "physics" stage:

fn physics_step(world: &mut World, resources: &mut Resources) {
  let schedule = resources.get::<PhysicsSchedule>();
  schedule.run(world, resources);
}

3. Use an "accumulator" resource like you mentioned and let systems determine when something is executed

(2) and (3) are both limited by the global schedule update, but i do like how (2) builds on top of the existing feature set without adding new things. (1) allows for fully decoupling schedules from each other, which does seem valuable.

[reidbhuntley]

I think (3) is unsuitable for the reasons I gave above. (2) is definitely worth looking into, especially if there are more use cases for the "sub schedule" concept. If not, (1) may be the best way to go due to how simple it could allow the API to be.

[tristanpemble]

I believe Unity DOTS is taking a somewhat similar approach to 2 by allowing system groups to loop: ComponentSystemGroup::UpdateCallback

Relevant forum post describing the design in a little bit more detail

[lwansbrough]

I know amethyst_network uses a SimulationTime resource which covers the use case for network physics, etc.

[tinco]

I just spent a couple days exploring option 1. It's more complicated than I thought it would be. The API definitely is rather simple, app's have multiple schedules, each schedule has a runner. But the internals are more complex. I went for each schedule has its own executor, which means the schedules run fully independent of eachother, but it also means the parallel stages contend for access to the World and Resources, I did this through a RwLock but as I'm finishing this there seem to be a serious drawback.

A RwLock let's all the systems run in parallel, but any time a stage finishes, it will try to get write access on the World and Resources (to run the commands) which will immediately block all systems until the current systems are done running, until the finished stage has done its deeds. I'm not exactly sure how this would go in practice, maybe it all goes fine because most time is spent in the systems, and not as much time is spent in finishing up stages. It feels a bit awkward because in a game engines avoiding randomness in timing is very important, and this introduces some randomness. Maybe this would be solved by a more advanced Resources system where there might be finer granularity locking on the resources.

If we go option 2, it will mean we'll dynamically change the schedule so that the stages are interleaved at their respectively correct rates. This is very attractive if we're thinking of sequential stage execution, but parallel stages seem much more interesting to me, so it's actually possible get some work done in these stages. If the stages run in parallel then we actually get the exact same situation as in option 1.

Just to paint a common game scenario:

We have a animation+render loop running at 60fps. We have a gamelogic loop running at 30fps. While the animation+render loop is interpolating animations and sending stuff to the graphics card, the game logic loop is running a physics step, and then running the game logic systems.

I agree that option 3 does not seem to be able to adequately express this. I must say I'm not fully clear on how you would express this with option 2.

After stepping back and writing this comment, considering option 2 has the same drawbacks, I think option 1 does seem to be most attractive so I'm gonna finish this up and hopefully I'll have something to show that could be a basis for this feature.

[thebluefish]

I have been contemplating a few approaches to this problem. Here is something I have worked out for a separate Physics tick rate via a singleton PhysicsSchedule, system runner, and PhysicsRate component.

use bevy::{prelude::*, ecs::Schedule};
pub struct Physics;

pub struct PhysicsRate(f64);
pub struct PhysicsSchedule(f64, Schedule); // accumulator, schedule

static mut PHYSICS_SCHEDULE: Option<PhysicsSchedule> = None;

pub fn get_schedule<'a>() -> &'a mut PhysicsSchedule {
    unsafe {
        match &mut PHYSICS_SCHEDULE {
            Some(schedule) => schedule,
            None => {
                PHYSICS_SCHEDULE = Some(PhysicsSchedule(0.0, Schedule::default()));
                get_schedule()
            }
        }
    }
}

fn physics_runner(world: &mut World, resources: &mut Resources) {
    unsafe {
        match &mut PHYSICS_SCHEDULE {
            Some(PhysicsSchedule(acc, schedule)) => {
                let delta = resources.get::<Time>().unwrap().delta_seconds_f64;
                let rate = resources.get::<PhysicsRate>().unwrap().0;
                *acc += delta;
                while *acc >= rate {
                    *acc -= rate;
                    schedule.run(world, resources);
                }
            },
            _ => {
                panic!("schedule must exist before running it!")
            }
        };
    }
}

impl Plugin for Physics {
    fn build(&self, app: &mut AppBuilder) {
        unsafe {
            let schedule = get_schedule();
            schedule.1.add_stage("update");
            schedule.1.add_system_to_stage("update", phys_system.system());
        }

        // rate of 30 ticks per second
        app.add_resource(PhysicsRate(1.0 / 30.0))
            .add_stage_after("event_update", "physics_runner")
            .add_system_to_stage("physics_runner", physics_runner.thread_local_system());
    }
}

fn phys_system(rate: Res<PhysicsRate>) {
    println!("physics! delta: {}", rate.0);
}

My concern with this approach is how I would succinctly integrate 3rd party plugins. It seems that I would probably need to duplicate the plugin's build function to run its systems on my custom schedule.

[Ratysz]

@thebluefish, why is PhysicsSchedule a global variable instead of a resource?

[thebluefish]

I am unsure how to run the Schedule if it's a resource, been fighting the borrow checker there. I think ideally I should get child schedules from the parent schedule directly, but I'm torn on the exact approach. I think my sort of ideal approach now would be stage runners - instead of always firing a stage, optionally run a function and give it some way to fire the stage manually; which also covers cases where you might need to run a stage multiple times per frame.

…

On Fri, Sep 25, 2020, 4:08 AM Alexander Sepity ***@***.***> wrote: @thebluefish <https://github.com/thebluefish>, why is PhysicsSchedule a global variable instead of a resource? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#125 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCINHSWYCEB5KJNAR6F4BLSHR22BANCNFSM4P3TYEDA> .

[Ratysz]

been fighting the borrow checker there

Yeah, it gets a bit gnarly, but it works if the resource is Arc<Mutex<PhysicsSchedule>>, and you get it like this in the system

    let physics = resources
        .get::<Arc<Mutex<PhysicsSchedule>>>()  // Option<Ref<Arc<Mutex<PhysicsSchedule>>>>
        .unwrap()  // Ref<Arc<Mutex<PhysicsSchedule>>>
        .deref()  // Arc<Mutex<PhysicsSchedule>>
        .clone();  // Clones the Arc, which lets the compiler infer that `resources` isn't borrowed anymore.
    let mut physics = physics.lock().unwrap();  // MutexGuard<PhysicsSchedule>

---

For what it's worth, the approach I've been using in similar situations is to implement System on the type containing the schedule. That way the schedule is not stored in resources, but indeed in the parent schedule.

[thebluefish]

For what it's worth, the approach I've been using in similar situations is to implement System on the type containing the schedule. That way the schedule is not stored in resources, but indeed in the parent schedule.

Good call, this worked out great for my case.

[smokku]

@thebluefish, I see you've encapsulated your approach in https://github.com/thebluefish/bevy_contrib_schedules
Thanks.

[klaatu01]

@tinco where did you get to with this? Happy to try and pickup where you left off.

[tinco]

hey @klaatu01 I don't recommend it. I abandoned it because it just seems like a bad idea to me now. Especially since @cart recently had this proud post that bevy is now lock free.

What I did adds RwLocks all over the place. Anyway here's the code:

master...tinco:multiple-schedulers

If I'd try now I would really carefully read how the lock free system works and find out how this would fit in there. I bet the new architecture opens up new possibilities.

[tinco]

This is the API I made work on my branch:

use bevy::{app::ScheduleRunnerPlugin, prelude::*};
use std::time::Duration;

// This example shows multiple schedules running at the same time. So you can have
// the default schedule at your target screen framerate, and for example your
// physics game loop at a fixed 30 fps.
fn main() {
    // this app has the default schedule running at 1 fps and a second schedule
    // running at 2 fps.
    App::build()
        .add_plugin(ScheduleRunnerPlugin::run_loop(Duration::from_secs_f64(
            1.0,
        )))
        .add_schedule("faster", ScheduleRunnerPlugin::run_loop(Duration::from_secs_f64(
            2.0,
        )))
        .add_system(hello_world_system.system())
        .add_system_to_schedule("faster", zippy_system.system())
        .run();
}

fn hello_world_system() {
    println!("hello world");
}

fn zippy_system() {
    println!("zip");
}

[klaatu01]

@tinco thanks so much for sharing. I've been looking into this quite a bit, so even just seeing where you got to is very useful!

If I'd try now I would really carefully read how the lock free system works and find out how this would fit in there. I bet the new architecture opens up new possibilities.

I must have missed this, I will need to take a look. Thanks!

[reidbhuntley]

This is now possible thanks to #1021 !