Documentation time

README.md

@@ -4,4 +4,50 @@
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier)
 
-This directory is home to the [LIGHT](https//parl.ai/projects/light/) project. At the moment it is very much a migration in progress, but it will be home to all of the LIGHT experiments, reproducible code, and more!
+This directory is home to the [LIGHT](https://parl.ai/projects/light/) project. At the moment it is very much a migration in progress, but it will be home to all of the LIGHT experiments, reproducible code, and more! For now, much of the code is split between the two locations, so missing segments and data can likely be found from the projects game.
+
+## What is LIGHT?
+
+LIGHT is part Text-Adventure game, part Dialogue + Natural Language research platform. We host a game at [light-rpg.ai](https://light-rpg.ai) built upon the research work contained in this repo. Here we're also happy to consider developments from the LIGHT community of researchers, creators, and players. Our long-term goal is to create something that is all of:
+1. A strong multiplayer game environment that is easy and fun to interact with for players.
+2. An AI-Powered storytelling tool for creators to build interactive experiences for others to enjoy.
+3. A platform for exploring advancements in Dialogue and Natural Language research through developing better models to power the game.
+
+## The LIGHT Repo
+
+This repo contains a few directories related to the LIGHT project. More details can be found in each directory:
+- [**`crowdsourcing`**](https://github.com/facebookresearch/LIGHT/tree/main/crowdsourcing): Contains the full setup and deploy code for (nearly) all of the crowdsourced data used in LIGHT. Useful examples of how to collect similar types of data.
+- [**`deploy`**](https://github.com/facebookresearch/LIGHT/tree/main/deploy): Contains code relevant for various deploys of LIGHT for public consumption.
+- [**`light`**](https://github.com/facebookresearch/LIGHT/tree/main/light): Contains the core LIGHT game engine. Includes the `OOGraph` data model, descriptions of all of the node types, actions and manipulations on the graph (`GraphEvent`s), and the abstract API (`Soul`s) for creating agents that can act in LIGHT.
+- [**`modeling`**](): A (currently-nonexistant) directory of code for agents and teachers that can be trained to work in LIGHT.
+- [**`projects`**](https://github.com/facebookresearch/LIGHT/tree/main/projects): Contains code that's still in-development, either for research endeavors that haven't yet been incorporated into the game, or works that derive from work but ultimately don't feed back in. Also contains code relevant preserved to reproduce specific paper results.
+- [**`scripts`**](https://github.com/facebookresearch/LIGHT/tree/main/scripts): Contains execution code for running LIGHT, exploring our dataset, filtering through examples, and more.
+
+## Getting started
+
+First off, to get started you'll need to install LIGHT into your python environment. For this, you can navigate to this repo and run
+```
+pip install -e .
+```
+The rest of the core interactions with LIGHT may require additional environment and other downloads, explained below.
+
+### Interacting with the LIGHT engine (no models)
+To jump right into a LIGHT world on the command line, you can use some of our example code. The following will put you in the simplest world:
+```
+python scripts/examples/play_map.py
+```
+
+You can also try a more complex LIGHT world using the following:
+```
+python scripts/examples/play_map.py --load-map scripts/examples/complex_world.json
+```
+
+Deeper explanations on how the engine itself functions can be found in the [LIGHT graph](https://github.com/facebookresearch/LIGHT/tree/main/light/graph) directory.
+
+### Exploring the environment with models
+
+TODO - Fill out once all of the required minimal models are hosted for access outside of the cluster.
+
+### Launching the full game server locally
+
+TODO - more details are in the [web deploy](https://github.com/facebookresearch/LIGHT/tree/main/deploy/web/) directory.

crowdsourcing/README.md

@@ -1,3 +1,10 @@
-# crowdsourcing
+# Crowdsourcing
 
-All LIGHT crowdsourcing tasks launched through regular crowd providers (like MTurk) belong in here.
+All LIGHT crowdsourcing tasks launched through regular crowd providers (like MTurk) belong in here. Overall this collection should be the full setup and deploy code for (nearly) all of the crowdsourced data used in LIGHT. Some are useful examples of how to collect similar types of data.
+
+**Subdirectories:**
+- **`custom_world_interactions`**: Tasks that together allow us to collect custom `GraphEvent` operations outside of the basic game engine. These allow LIGHT to have "common sense" interactions.
+- **`dialogues`**: Tasks to improve the dialogue capabilities of models in specific situations.
+- **`environment`**: Tasks to expand on or clean up the LIGHT environment.
+- **`filtering`**: Tasks that allow us to filter out data collected in LIGHT for safety or other reasons.
+- **`quests`**: Tasks for collecting quests and motivations for characters, as well as some plans or paths to accomplish those goals.

deploy/README.md

@@ -1,3 +1,8 @@
-# deploy
+# Deploy
 
 Folder containing all of the backend and frontend code for deploying an aspect of LIGHT on one of the surfaces we support.
+
+
+**Subdirectories:**
+- **`web`**: Code for the current [light-rpg.ai](https://light-rpg.ai) deploy. Minus some deploy + safety secrets, of course! You'll need to supply your own for a local deployment.
+- **`messenger`**: Code for the messenger deploy used in our [LIGHT-WILD](https://arxiv.org/abs/2008.08076) data collection. Not in a deploy-ready state, though the underlying ParlAI worlds contain the same logic we used in those experiments.

light/README.md

@@ -1,3 +1,9 @@
-# LIGHT Core
+# light
 
-Folder for all of the generally shared content within the LIGHT project.
+Contains the core LIGHT game engine and data model. Includes the `OOGraph` data model, descriptions of all of the node types, actions and manipulations on the graph (`GraphEvent`s), and the abstract API (`Soul`s) for creating agents that can act in LIGHT.
+
+**Subdirectories:**
+- **`data_model`**: Classes that represent LIGHT data in _cold storage_, absent of a specific LIGHT world. At the moment this is mostly centralized in the `LIGHTDatabase`, which uses SQLite storage, but given new data types and systems this is definitely subject to change.
+- **`graph`**: Classes that represent a LIGHT Graph (the state of a LIGHT world), the elements inside, builders to create a new Graph, events that modify Graphs, and related tooling.
+- **`registry`**: Classes that allow registering LIGHT models to an active LIGHT world, allowing them to interact and fill certain decisions in the LIGHT setup. Currently in development as we transition models out from being initialized at their usage locations.
+- **`world`**: Contains all the higher-level code that is required to run an _active_ LIGHT graph, including processing agent actions into events, executing them on the graph, and handling logging.

light/colors.py

@@ -4,6 +4,11 @@
 # This source code is licensed under the MIT license found in the
 # LICENSE file in the root directory of this source tree.
 
+"""
+Simple shared class of colors we can use for displaying terminal content in
+visually appealing ways.
+"""
+
 
 class Colors:
     CYAN = "\u001b[36m"
@@ -18,4 +23,4 @@ class Colors:
     BOLD_BLUE = "\u001b[34;1m"
     PURPLE = "\u001b[35m"
     BOLD_PURPLE = "\u001b[35;1m"
-    RESET = "\u001b[0m"
+    RESET = "\u001b[0m"

light/graph/README.md

@@ -1,3 +1,41 @@
 # LIGHT Graph
 
 The LIGHT graph is the core state of the environment over which LIGHT takes place. This folder contains the components for managing, building, and operating on Graphs.
+
+## The LIGHT `OOGraph`
+
+Generally, the LIGHT `OOGraph` is constructed of `GraphNode`s and `GraphEdge`s. Nodes contained in or containing other nodes use the `GraphEdge` class. We use a special `NeighborEdge` class to note the connections between rooms, as these may have additional attributes. All of the contents of a graph should be json-serializable, so you can see a JSON representation of a simple graph [here](https://github.com/facebookresearch/LIGHT/tree/main/scripts/examples/simple_world.json).
+
+### Special Conditions
+Beyond the basic structure of nodes and edges, the `OOGraph` keeps track of a few special considerations.
+
+#### The Void
+As all nodes must have a container, we create a special `GraphVoidNode` which is the default continer for all nodes. When creating a node, all nodes are initially created in the void, and can later be moved to their correct containers with `move_to` functions. Room nodes generally will remain in the void.
+
+#### Deletion Lifecycles
+Node deletion is somewhat tricky in LIGHT, as it's possible that an action leads to the deletion of a node that, in the same timestep, needs to be used for a description, log, or otherwise. For this, we can instead mark a node for deletion with `mark_node_for_deletion` and later use `delete_nodes` to clear these. Further, even when a node is deleted, we still maintain a reference to it in `_deleted_nodes` for logging purposes. (This may eventually be removed for memory usage reasons)
+
+#### Agent Death
+Death in the LIGHT world, in a base sense, is the transition from a node being a `GraphAgent` that can be inhabited by a `Soul` to a `GraphNode`. As of now this transition is managed by the _to be deprecated_ `agent_die` function, which makes an object copy of the agent, transfers over the contents to the copy, and then delete the agent node. These special post-death nodes are tracked in the `dead_nodes` array.
+
+#### Node searching
+It is important in many cases (especially parsing) to try and find nodes that match a specific description in the graph. Accessors of this type are bundled in the `desc_to_nodes` function, which can either search the whole graph or can use a `nearby_node` and `nearbytype` to search with a specific strategy.
+
+`nearbytype` is a string of `+` separated options, having the following effects:
+- `'all'`: include the contents of `nearby_node`, its parent, and its parent's neighbors.
+- `'sameloc'`: searches over all nodes sharing the same parent container as `nearby_node`.
+- `'carrying'`: searches over all nodes contained by `nearby_node`.
+- `'path'`: searches over rooms attached to the container of `nearby_node`.
+- `'contains'`: includes the container of `nearby_node`.
+- `'other_agents'`: includes nodes carried by agents in the same **room** as `nearby_node`.
+- `'others'`: extend search to recursively include anything contained in the search list already.
+
+#### Nodes vs IDs
+Some functions in the `OOGraph` still refer to using a node's `node_id` rather than using the reference to the `node` directly. In general, this access pattern is deprecated, but remaining usage tends to very clearly note whether an access is getting a `GraphNode` or is `node_id`. To convert `node_id` to `GraphNode`, you can use `OOGraph.get_node(node_id)`. To get a node's id, you can just use `GraphNode.node_id`.
+
+## Subdirectories
+- **`builders`**: `GraphBuilder`s are utility classes used to create and extend `OOGraph`s with content.
+- **`elements`**: `GraphNode`s are the the contents of a graph, and comprise the information stored within.
+- **`events`**: `GraphEvent`s define the types of operations and manipulations that can be performed on an `OOGraph`.
+- **`tests`**: Some testing to ensure that the graph is acting as expected.
+- **`viz`**: Visualization tooling that allows inspection of an `OOGraph`.

light/graph/builders/README.md

@@ -0,0 +1,12 @@
+# Builders
+
+Core folder containing the LIGHT `GraphBuilder` classes. A `GraphBuilder` is used to directly construct or modify a LIGHT `OOGraph` in a controllable way. The core types are the `GraphBuilder` and the `SingleSuggestionGraphBuilder`. The former exposes an API where you can simply create a graph or add agents to it (to fill for dying agents). The latter provides an abstract API for a graph builder that can later be used to make suggestions for modifications to the graph.
+
+**Contents:**
+- **`base.py`**: Underlying base `GraphBuilder` class definitions.
+- **`base_elements.py`**: Utility classes representing the contents retrieved from the `LIGHTDatabase` as would be useful for a `GraphBuilder`.
+- **`db_utils.py`**: Utility classes to access graph-construction-related content from a `LIGHTDatabase`.
+- **`external_map_json_builder.py`**: Graph builder using a _deprecated_ format (from an early world builder tool) to load a graph directly from that json format.
+- **`one_room_builder.py`**: Graph builder that creates a single room and populates it with the desired number of characters. Often used to create dialogue tasks.
+- **`starspace_all.py`**, **`starspace_assisted.py`**, and **`starspace_neighbor.py`**: Various starspace-based graph builders that use a `starspace` model to make large graphs from knowledge on local connections. Nearly deprecated by Commonsense-based agents, and will likely be deprecated once a `GraphBuilder` is created from them.
+- **`user_world_builder.py`**: Creates a LIGHT Graph based on contents saved in the `LIGHTDatabase` for a given user. Likely to be deprecated when we start saving the new world format.

light/graph/elements/README.md

@@ -0,0 +1,5 @@
+# elements
+
+Contains the main `GraphNode` and `GraphEdge` types that comprise a LIGHT `OOGraph`.
+
+At the moment, `GraphNode`s can have two different edge types. The first is a `GraphEdge`, which represents a relation between two nodes that are in the graph. The second is the more implicit attribute edge, which allows for `GraphNode`s to be identified as containers or having certain properties.