Networking Technology

[zicklag]

Opening this topic to discuss networking technology we might be able to make use of for Punchy multiplayer.

[zicklag]

Originally posted by @erlend-sh here

The most promising multiplayer netcode library in Bevy land these days is https://github.com/lucaspoffo/renet

Web-support is on its tentative roadmap, and a first step towards that is in a draft PR:

lucaspoffo/renet#10
lucaspoffo/renet#14

[zicklag]

I've not got a lot of exposure to what other people think of them, but from my own looking around, here are some other promising projects, that have Bevy examples and support web:

• https://github.com/gschup/ggrs — Peer-to-peer style networking for games that have deterministic gameplay, which we should be able to satisfy with Punchy if we wanted to use this.
  • matchbox as it's socket layer, which we might be able to make use of if we add web support to renet or another project
• https://github.com/naia-lib/naia — Different client-server model networking lib looking pretty similar to renet. Not sure what the differences between them are yet.

@erlend-sh do you have any background on how renet might compare with either of those? They seem like the closest alternatives as far as I can find on GitHub.

[erlend-sh]

ggrs/rollback comes with more unknowns.

naia is promising, but less actively supported than renet at the moment, and apparently not very well aligned with Ultimate’s model.

Here’s another great maniwani excerpt from the Bevy discord:

Also, to the general question of "what kind of networking should X game use?", imo answering these three questions can decide for you.
1. Deterministic or authoritative?
2. How much should clients predict? Nothing? Some things? Everything?
3. Client-server or peer-to-peer?

(1)

• How complex is the simulation? How many entities are there? How large is the world?
• Does fairness require some truly hidden information?
• Are all of the libraries you want to use capable of being deterministic?

With something like an RTS game, you would probably (in 2022) struggle to send enough data under the authoritative model. MTU and packet loss just don't allow for much. On the other hand, there's a limit to how much client hardware can simulate. You can't practically use the deterministic model (e.g. GGPO) with a large open-world game like Genshin Impact since not every player has a literal supercomputer.

Hidden information is not possible with the deterministic model since clients have to simulate everything to stay in sync. All data will be in memory and we can't (in 2022) stop something from reading it without permission. So if you're coding up the next online poker game, you pretty much have to choose an authoritative framework. That said, if you have nothing to hide, determinism has very appealing low bandwidth properties.

But in the end games often have no choice but the authoritative model because some critical dependency (i.e. PhysX, Bullet) doesn't expose its entire state needed to rollback or has some other form of randomness.

(2)

• Is this a realtime or turn-based game?
• Can mispredictions be tolerated?
• Deterministic or authoritative?

If it's a realtime game, players expect immediate input feedback, whereas with turn-based games like chess, it's normal to wait for other players. But even with realtime games, IIRC some genres (like RTS) don't tolerate visual artifacts caused by mispredictions, so they actually won't have client-side prediction (absence of prediction is lockstep). Instead, they'll add a generous amount of "input delay" and hide the wait with some clever animations.

Having clients predict everything (even other players) has an advantage in fighting games and games like Rocket League that have lots of physical contact, the advantage being that all players simultaneously observe the same scene (mostly), so they can react at the same time. The tradeoff is less absolute accuracy because of the constant misprediction.

Authoritative games can have clients only predict "some things", with server-side lag compensation to deal with the dichotomy. Deterministic games don't really have that choice tho, it's predict everything or nothing.

(3)

• Do you want to get intimate with the most frustrating aspects of network infrastructure?

This almost isn't a question. 99/100 times, pick client-server. I think peer-to-peer (the fully-connected mesh kind) is much harder to implement, for next to no benefit. I'd pick host migration any day over more frequent NAT / firewall issues.

Even if your game is going to use deterministic replication (or we lived in an IPv6 world), a literal P2P mesh wouldn't be able to support as many players (on 2022 consumer connections), be as stable, or be as convenient as having clients connect to a central dumb relay.

[zicklag]

Here's a really great resource with thoughts in general on networking in games. We should probably all ( developers who plan on working on networking impl ) read this:

https://github.com/maniwani/rfcs/blob/main/replication_concepts.md

It's apparently part of a Bevy network replication RFC, and has a lot of really easy-to-read breakdowns on techniques.

[erlend-sh]

Yeah, here’s the accompanying RFC discussion.

A starting implementation is being sketched out here: https://github.com/maniwani/parrot

Someone on Discord asked:

Looking at the project's readme and there being no GH issues, do you have a plan for an MVP/POC? I think a well-defined scope (based on the rfc) is gonna be key to figuring out what should be focused on first and help if people wanna contribute, like myself 😊

maniwani’s reply:

yes, I do have a plan to get to an MVP state
I consider the readme already well-scoped tbh

• for the "sending" points, you can probably just substitute with something like renet
• the first two "syncing" bullet points will be accomplished by my alloc sub-crate, both kinds of replication boil down to "memcpy an arena" for me, it doesn't matter if those bytes originally came from the local sim or a server packet
• multiple players on each client just translates to clients and players being separate identifiers
• prediction and interpolation, and some of lag comp will be accomplished by a simple time sync algorithm, the other part of lag comp involves rapier/bevy particulars
• "interest management" does still needs some design work, I'm split on something that has to do with heap-allocating types
• RPCs should be pretty straightforward, it's just going to involve a proc macro for configuration

I spent too much time investigating how I would do the arena allocator / relative pointer thing in Rust, was trying to follow the allocator trait API and pass allocator references around like a Bump, but that was looking uber complicated (I'm pretty sure what I needed to make it work with relative pointers required GATs), so recently I decided I'll just assign a static variable and use that

the biggest unknown to me is actually "bevy integration", first I really need stageless
and another thing is I haven't fully figured out how to not break change detection with the whole rollback thing
simulation component and system change ticks need to be reset like everything else that gets rolled back, so I need us to add a method that lets users manually set change ticks

It’s a promising future, but there’s still some design work to do for the theory to become practice. It won’t land fully until networking is a focus area for Bevy.

[zicklag]

@erlend-sh what is the maximum number of online players in a single game at the same time that you could imagine for Punchy? Just to be clear, I'm talking real players making inputs, not AI.

I'm trying to get some reference as to how much it might cost in a deterministic peer-to-peer model, which seems to be preferable for high-precision games that can limit number of players in a single "room", i.e. no open world. It seems like we'd fit into that category pretty well.

[odecay]

I think 4 players should be the goal for the main Game, though might be different for other GameModes or mods. I remember having some discussion in discord awhile back and I believe 4 players doable in p2p rollback.

[zicklag]

If we can make p2p rollback work, I think we could support a massive amount of players while only using an inconsequential amount of memory/CPU on the our server, which would be really cool.

Biggest questions around p2p remaining then might be how to do deterministic floats, which rapier did somehow so we could look at that. And whether JavaScript mods are deterministic. I think v8 can be started in deterministic mode so I think that might work for JavaScript.

And there's at least one rollback library for JS so I'm guessing that means browsers support enough determinism for it to work, maybe?

[erlend-sh]

4 players is our default, and maximum in custom games would be 8, so we fit within the constraints in that regard.

However, rollback is a lot harder to do right than a classic client-server architecture, especially for integration with Ultimate. The low-latency benefits of rollback are also insignificant for a beatemup, which is not a ‘high precision game’ to the same degree as a classical fighter.

At some point we will for sure be doing a Smashbros/Brawlhalla type game, which by many fighter gamers’ standards requires p2p rollback for satisfactory online play. That will be the more appropriate time to fully grok this tech.

AFAIK rollback is not common in beatemups, but there is some precedent for it, e.g. https://github.com/atom-tm/l2df-engine

[erlend-sh]

p.s. we are gonna be experimenting with the ggpo/rollback approach in this project: https://github.com/gamercade-io/hoppy

The @gamercade-io virtual console is deeply integrated with ggrs, so we’re gonna try make one and possibly several little fighting games with it.

[zicklag]

OK, reading through your comments @erlend-sh I think I'm satisfied that p2p rollback is more work than it's worth right now.

Sounds like we go with Renet, and contribute browser support to it. 👍

I'm comfortable with Rust in the browser, and I've done server-side Rust, too if we need to do some sort of WebRTC server to setup the browser connection, though hopefully we can just copy somebody who's already done that.

So I think I'll be able to add whatever's necessary if he get's the API in place for it like he said he was doing.

[erlend-sh]

The beginnings of our browser requirements in renet has been started here, so if you can complete this that’d really get the ball rolling: lucaspoffo/renet#36

[zicklag]

Cool, I'm trying to finish off some non-punchy work that hopefully won't take more than a week or two, then I'll try to finish of the scripting MVP, then I think working on that could be a good next step.