From 1ba07feebee5a74280ca2a4b3de95fc5f7b82ba3 Mon Sep 17 00:00:00 2001 From: Bryce Wilson Date: Tue, 5 Mar 2024 08:48:37 -0700 Subject: [PATCH] BizHawkClient: Add README (#2689) Co-authored-by: Nicholas Saylor <79181893+nicholassaylor@users.noreply.github.com> --- worlds/_bizhawk/README.md | 279 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 279 insertions(+) create mode 100644 worlds/_bizhawk/README.md diff --git a/worlds/_bizhawk/README.md b/worlds/_bizhawk/README.md new file mode 100644 index 000000000000..ddc70c3dd748 --- /dev/null +++ b/worlds/_bizhawk/README.md @@ -0,0 +1,279 @@ +# BizHawk Client + +`BizHawkClient` is an abstract base class for a client that can access the memory of a ROM running in BizHawk. It does +the legwork of connecting Python to a Lua connector script, letting you focus on the loop of checking locations and +making on-the-fly modifications based on updates from the server. It also provides the same experience to users across +multiple games that use it, and was built in response to a growing number of similar but separate bespoke game clients +which are/were largely exclusive to BizHawk anyway. + +It's similar to `SNIClient`, but where `SNIClient` is designed to work for specifically SNES games across different +emulators/hardware, `BizHawkClient` is designed to work for specifically BizHawk across the different systems BizHawk +supports. + +The idea is that `BizHawkClient` connects to and communicates with a Lua script running in BizHawk. It provides an API +that will call BizHawk functions for you to do things like read and write memory. And on an interval, control will be +handed to a function you write for your game (`game_watcher`) which should interact with the game's memory to check what +locations have been checked, give the player items, detect and send deathlinks, etc... + +Table of Contents: +- [Connector Requests](#connector-requests) + - [Requests that depend on other requests](#requests-that-depend-on-other-requests) +- [Implementing a Client](#implementing-a-client) + - [Example](#example) +- [Tips](#tips) + +## Connector Requests + +Communication with BizHawk is done through `connector_bizhawk_generic.lua`. The client sends requests to the Lua script +via sockets; the Lua script processes the request and sends the corresponding responses. + +The Lua script includes its own documentation, but you probably don't need to worry about the specifics. Instead, you'll +be using the functions in `worlds/_bizhawk/__init__.py`. If you do need more control over the specific requests being +sent or their order, you can still use `send_requests` to directly communicate with the connector script. + +It's not necessary to use the UI or client context if you only want to interact with the connector script. You can +import and use just `worlds/_bizhawk/__init__.py`, which only depends on default modules. + +Here's a list of the included classes and functions. I would highly recommend looking at the actual function signatures +and docstrings to learn more about each function. + +``` +class ConnectionStatus +class BizHawkContext + +class NotConnectedError +class RequestFailedError +class ConnectorError +class SyncError + +async def read(ctx, read_list) -> list[bytes] +async def write(ctx, write_list) -> None: +async def guarded_read(ctx, read_list, guard_list) -> (list[bytes] | None) +async def guarded_write(ctx, write_list, guard_list) -> bool + +async def lock(ctx) -> None +async def unlock(ctx) -> None + +async def get_hash(ctx) -> str +async def get_system(ctx) -> str +async def get_cores(ctx) -> dict[str, str] +async def ping(ctx) -> None + +async def display_message(ctx, message: str) -> None +async def set_message_interval(ctx, value: float) -> None + +async def connect(ctx) -> bool +def disconnect(ctx) -> None + +async def get_script_version(ctx) -> int +async def send_requests(ctx, req_list) -> list[dict[str, Any]] +``` + +`send_requests` is what actually communicates with the connector, and any functions like `guarded_read` will build the +requests and then call `send_requests` for you. You can call `send_requests` yourself for more direct control, but make +sure to read the docs in `connector_bizhawk_generic.lua`. + +A bundle of requests sent by `send_requests` will all be executed on the same frame, and by extension, so will any +helper that calls `send_requests`. For example, if you were to call `read` with 3 items on your `read_list`, all 3 +addresses will be read on the same frame and then sent back. + +It also means that, by default, the only way to run multiple requests on the same frame is for them to be included in +the same `send_requests` call. As soon as the connector finishes responding to a list of requests, it will advance the +frame before checking for the next batch. + +### Requests that depend on other requests + +The fact that you have to wait at least a frame to act on any response may raise concerns. For example, Pokemon +Emerald's save data is at a dynamic location in memory; it moves around when you load a new map. There is a static +variable that holds the address of the save data, so we want to read the static variable to get the save address, and +then use that address in a `write` to send the player an item. But between the `read` that tells us the address of the +save data and the `write` to save data itself, an arbitrary number of frames have been executed, and the player may have +loaded a new map, meaning we've written data to who knows where. + +There are two solutions to this problem. + +1. Use `guarded_write` instead of `write`. We can include a guard against the address changing, and the script will only +perform the write if the data in memory matches what's in the guard. In the below example, `write_result` will be `True` +if the guard validated and the data was written, and `False` if the guard failed to validate. + +```py +# Get the address of the save data +read_result: bytes = (await _bizhawk.read(ctx, [(0x3001111, 4, "System Bus")]))[0] +save_data_address = int.from_bytes(read_result, "little") + +# Write to `save_data_address` if it hasn't changed +write_result: bool = await _bizhawk.guarded_write( + ctx, + [(save_data_address, [0xAA, 0xBB], "System Bus")], + [(0x3001111, read_result, "System Bus")] +) + +if write_result: + # The data at 0x3001111 was still the same value as + # what was returned from the first `_bizhawk.read`, + # so the data was written. + ... +else: + # The data at 0x3001111 has changed since the + # first `_bizhawk.read`, so the data was not written. + ... +``` + +2. Use `lock` and `unlock` (discouraged if not necessary). When you call `lock`, you tell the emulator to stop advancing +frames and just process requests until it receives an unlock request. This means you can lock, read the address, write +the data, and then unlock on a single frame. **However**, this is _slow_. If you can't get in and get out quickly +enough, players will notice a stutter in the emulation. + +```py +# Pause emulation +await _bizhawk.lock(ctx) + +# Get the address of the save data +read_result: bytes = (await _bizhawk.read(ctx, [(0x3001111, 4, "System Bus")]))[0] +save_data_address = int.from_bytes(read_result, "little") + +# Write to `save_data_address` +await _bizhawk.write(ctx, [(save_data_address, [0xAA, 0xBB], "System Bus")]) + +# Resume emulation +await _bizhawk.unlock(ctx) +``` + +You should always use `guarded_read` and `guarded_write` instead of locking the emulator if possible. It may be +unreliable, but that's by design. Most of the time you should have no problem giving up and retrying. Data that is +volatile but only changes occasionally is the perfect use case. + +If data is almost guaranteed to change between frames, locking may be the better solution. You can lower the time spent +locked by using `send_requests` directly to include as many requests alongside the `LOCK` and `UNLOCK` requests as +possible. But in general it's probably worth doing some extra asm hacking and designing to make guards work instead. + +## Implementing a Client + +`BizHawkClient` itself is built on `CommonClient` and inspired heavily by `SNIClient`. Your world's client should +inherit from `BizHawkClient` in `worlds/_bizhawk/client.py`. It must implement `validate_rom` and `game_watcher`, and +must define values for `system` and `game`. + +As with the functions and classes in the previous section, I would highly recommend looking at the types and docstrings +of the code itself. + +`game` should be the same value you use for your world definition. + +`system` can either be a string or a tuple of strings. This is the system (or systems) that your client is intended to +handle games on (SNES, GBA, etc.). It's used to prevent validators from running on unknown systems and crashing. The +actual abbreviation corresponds to whatever BizHawk returns from `emu.getsystemid()`. + +`patch_suffix` is an optional `ClassVar` meant to specify the file extensions you want to register. It can be a string +or tuple of strings. When a player clicks "Open Patch" in a launcher, the suffix(es) will be whitelisted in the file +select dialog and they will be associated with BizHawkClient. This does not affect whether the user's computer will +associate the file extension with Archipelago. + +`validate_rom` is called to figure out whether a given ROM belongs to your client. It will only be called when a ROM is +running on a system you specified in your `system` class variable. In most cases, that will be a single system and you +can be sure that you're not about to try to read from nonexistent domains or out of bounds. If you decide to claim this +ROM as yours, this is where you should do setup for things like `items_handling`. + +`game_watcher` is the "main loop" of your client where you should be checking memory and sending new items to the ROM. +`BizHawkClient` will make sure that your `game_watcher` only runs when your client has validated the ROM, and will do +its best to make sure you're connected to the connector script before calling your watcher. It runs this loop either +immediately once it receives a message from the server, or a specified amount of time after the last iteration of the +loop finished. + +`validate_rom`, `game_watcher`, and other methods will be passed an instance of `BizHawkClientContext`, which is a +subclass of `CommonContext`. It additionally includes `slot_data` (if you are connected and asked for slot data), +`bizhawk_ctx` (the instance of `BizHawkContext` that you should be giving to functions like `guarded_read`), and +`watcher_timeout` (the amount of time in seconds between iterations of the game watcher loop). + +### Example + +A very simple client might look like this. All addresses here are made up; you should instead be using addresses that +make sense for your specific ROM. The `validate_rom` here tries to read the name of the ROM. If it gets the value it +wanted, it sets a couple values on `ctx` and returns `True`. The `game_watcher` reads some data from memory and acts on +it by sending messages to AP. You should be smarter than this example, which will send `LocationChecks` messages even if +there's nothing new since the last loop. + +```py +from typing import TYPE_CHECKING + +from NetUtils import ClientStatus + +import worlds._bizhawk as bizhawk +from worlds._bizhawk.client import BizHawkClient + +if TYPE_CHECKING: + from worlds._bizhawk.context import BizHawkClientContext + + +class MyGameClient(BizHawkClient): + game = "My Game" + system = "GBA" + patch_suffix = ".apextension" + + async def validate_rom(self, ctx: "BizHawkClientContext") -> bool: + try: + # Check ROM name/patch version + rom_name = ((await bizhawk.read(ctx.bizhawk_ctx, [(0x100, 6, "ROM")]))[0]).decode("ascii") + if rom_name != "MYGAME": + return False # Not a MYGAME ROM + except bizhawk.RequestFailedError: + return False # Not able to get a response, say no for now + + # This is a MYGAME ROM + ctx.game = self.game + ctx.items_handling = 0b001 + ctx.want_slot_data = True + + return True + + async def game_watcher(self, ctx: "BizHawkClientContext") -> None: + try: + # Read save data + save_data = await bizhawk.read( + ctx.bizhawk_ctx, + [(0x3000100, 20, "System Bus")] + )[0] + + # Check locations + if save_data[2] & 0x04: + await ctx.send_msgs([{ + "cmd": "LocationChecks", + "locations": [23] + }]) + + # Send game clear + if not ctx.finished_game and (save_data[5] & 0x01): + await ctx.send_msgs([{ + "cmd": "StatusUpdate", + "status": ClientStatus.CLIENT_GOAL + }]) + + except bizhawk.RequestFailedError: + # The connector didn't respond. Exit handler and return to main loop to reconnect + pass +``` + +### Tips + +- Make sure your client gets imported when your world is imported. You probably don't need to actually use anything in +your `client.py` elsewhere, but you still have to import the file for your client to register itself. +- When it comes to performance, there are two directions to optimize: + 1. If you need to execute multiple commands on the same frame, do as little work as possible. Only read and write necessary data, + and if you have to use locks, unlock as soon as it's okay to advance frames. This is probably the obvious one. + 2. Multiple things that don't have to happen on the same frame should be split up if they're likely to be slow. + Remember, the game watcher runs only a few times per second. Extra function calls on the client aren't that big of a + deal; the player will not notice if your `game_watcher` is slow. But the emulator has to be done with any given set of + commands in 1/60th of a second to avoid hiccups (faster still if your players use speedup). Too many reads of too much + data at the same time is more likely to cause a bad user experience. +- Your `game_watcher` will be called regardless of the status of the client's connection to the server. Double-check the +server connection before trying to interact with it. +- By default, the player will be asked to provide their slot name after connecting to the server and validating, and +that input will be used to authenticate with the `Connect` command. You can override `set_auth` in your own client to +set it automatically based on data in the ROM or on your client instance. +- You can override `on_package` in your client to watch raw packages, but don't forget you also have access to a +subclass of `CommonContext` and its API. +- You can import `BizHawkClientContext` for type hints using `typing.TYPE_CHECKING`. Importing it without conditions at +the top of the file will probably cause a circular dependency. +- Your game's system may have multiple usable cores in BizHawk. You can use `get_cores` to try to determine which one is +currently loaded (it's the best we can do). Some cores may differ in the names of memory domains. It's good to check all +the available cores to find differences before your users do. +- The connector script includes a DEBUG variable that you can use to log requests/responses. (Be aware that as the log +grows in size in BizHawk, it begins to stutter while trying to print it.)