docs: Add Sphinx docs site and move docs into site.

docs/README.md

@@ -0,0 +1,42 @@
+# Docs
+
+This directory contains the files necessary to generate a Sphinx-based documentation website for
+this project:
+
+* `conf` - Configuration files
+* `src` - The actual docs
+
+## Requirements
+
+* [Node.js] >= 16 to be able to [view the output](#viewing-the-output)
+* Python 3.10 or higher
+* [Task] 3.40.0 or higher
+
+## Build commands
+
+* Build the site incrementally:
+
+  ```shell
+  task docs:site
+  ```
+
+  * The output of the build will be in `../build/docs/html`.
+
+* Clean up the build:
+
+  ```shell
+  task docs:clean
+  ```
+
+## Viewing the output
+
+```shell
+task docs:serve
+```
+
+The command above will install [http-server] and serve the built docs site; `http-server` will print
+the address it binds to (usually http://localhost:8080).
+
+[http-server]: https://www.npmjs.com/package/http-server
+[Node.js]: https://nodejs.org/en/download/current
+[Task]: https://taskfile.dev/

docs/conf/conf.py

@@ -0,0 +1,77 @@
+# -- Project information -------------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = "Spider"
+
+# NOTE: We don't include a period after "Inc" since the theme adds one already.
+copyright = "2024-2025 YScope Inc"
+
+# -- General configuration -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = [
+    "myst_parser",
+    "sphinx_copybutton",
+    "sphinx_design",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.viewcode",
+]
+
+# -- MyST extensions -----------------------------------------------------------
+# https://myst-parser.readthedocs.io/en/stable/syntax/optional.html
+myst_enable_extensions = [
+    "attrs_block",
+    "colon_fence",
+]
+
+myst_heading_anchors = 4
+
+# -- Sphinx autodoc options ----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#configuration
+
+autoclass_content = "class"
+autodoc_class_signature = "separated"
+autodoc_typehints = "description"
+
+# -- HTML output options -------------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_favicon = "https://docs.yscope.com/_static/favicon.ico"
+html_title = project
+html_show_copyright = True
+
+html_static_path = ["../src/_static"]
+
+html_theme = "pydata_sphinx_theme"
+
+# -- Theme options -------------------------------------------------------------
+# https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/layout.html
+
+html_theme_options = {
+    "footer_start": ["copyright"],
+    "footer_center": [],
+    "footer_end": ["theme-version"],
+    "navbar_start": ["navbar-logo"],
+    "navbar_end": ["navbar-icon-links", "theme-switcher"],
+    "primary_sidebar_end": [],
+    "secondary_sidebar_items": ["page-toc", "edit-this-page"],
+    "show_prev_next": False,
+    "use_edit_page_button": True,
+}
+
+# -- Theme source buttons ------------------------------------------------------
+# https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/source-buttons.html
+
+html_context = {
+    "github_user": "y-scope",
+    "github_repo": "spider",
+    "github_version": "main",
+    "doc_path": "docs/src",
+}
+
+# -- Theme custom CSS and JS ---------------------------------------------------
+# https://pydata-sphinx-theme.readthedocs.io/en/stable/user_guide/static_assets.html
+
+
+def setup(app):
+    app.add_css_file("custom.css")

docs/requirements.txt

@@ -0,0 +1,5 @@
+myst-parser>=4.0.0
+pydata-sphinx-theme>=0.16.1
+sphinx_design>=0.6.1
+sphinx-copybutton>=0.5.2
+sphinx>=8.1.3

docs/src/_static/custom.css

@@ -0,0 +1,32 @@
+html[data-theme="dark"], html[data-theme="light"] {
+    --pst-color-primary: #3399ff;
+    --pst-color-secondary: #9580ff;
+}
+
+a {
+    text-decoration: none;
+}
+
+a:hover {
+    text-decoration: underline;
+}
+
+/*
+Use the bottom border that's used for indicating the current page as the hover style (for a more
+cohesive look).
+NOTE: This selector matches the one in pydata-sphinx-theme
+pydata/pydata-sphinx-theme@v0.14.4/src/pydata_sphinx_theme/assets/styles/sections/_header.scss#L86
+*/
+.bd-header .navbar-nav li a.nav-link:hover {
+    border-bottom: max(3px,.1875rem,.12em) solid var(--pst-color-secondary);
+    text-decoration: none;
+}
+
+/*
+Remove margin from sidebar-primary-items__end so that we don't have an unnecessary scrollbar. We're
+not using the end items currently.
+*/
+.bd-sidebar-primary .sidebar-primary-items__end {
+    margin-top: 0;
+    margin-bottom: 0;
+}

docs/src/dev-docs/index.md

@@ -0,0 +1,21 @@
+# Developer docs
+
+This section contains docs for developing Spider. Choose one of the sections below or use the left
+sidebar (if it's hidden, click the <i class="fa fa-bars"></i> icon) to navigate to specific docs.
+
+::::{grid} 1 1 2 2
+:gutter: 2
+
+:::{grid-item-card}
+:link: testing
+Testing
+^^^
+How to test Spider.
+:::
+::::
+
+:::{toctree}
+:hidden:
+
+testing
+:::

docs/testing.md → docs/src/dev-docs/testing.md

@@ -57,4 +57,4 @@ You can use the following tasks to run integration tests.
 | `test:integration`            | Runs all integration tests.                                       |
 
 
-[gh-workflow-unit-tests]: ../.github/workflows/unit-tests.yaml
+[gh-workflow-unit-tests]: https://github.com/y-scope/spider/blob/main/.github/workflows/unit-tests.yaml

docs/src/index.md

@@ -0,0 +1,31 @@
+# Spider
+
+Spider is a distributed system for executing user-defined tasks. It is designed to achieve low
+latency, high throughput, and robust fault tolerance.
+
+Spider's docs are separated into two categories:
+
+::::{grid} 1 1 2 2
+:gutter: 2
+
+:::{grid-item-card}
+:link: user-docs/index
+🧑 User docs 
+^^^
+Docs for those interested in using and operating Spider.
+:::
+
+:::{grid-item-card}
+:link: dev-docs/index
+🛠 Developer docs
+^^^
+Docs for those interested in developing Spider.
+:::
+::::
+
+:::{toctree}
+:hidden:
+
+user-docs/index.md
+dev-docs/index.md
+:::

docs/src/user-docs/guides-overview.md

@@ -0,0 +1,14 @@
+# Overview
+
+The tutorials below guide you on how to use and operate Spider.
+
+::::{grid} 1 1 2 2
+:gutter: 2
+
+:::{grid-item-card}
+:link: guides-quick-start
+Quick start
+^^^
+How to get started with running a task on Spider.
+:::
+::::

docs/quick-start.md → docs/src/user-docs/guides-quick-start.md

@@ -1,8 +1,5 @@
 # Quick start
 
-Spider is a distributed system for executing user-defined tasks. It is designed to achieve low
-latency, high throughput, and robust fault tolerance.
-
 The guide below briefly describes how to get started with running a task on Spider. At a high-level,
 you'll need to:
 
@@ -13,11 +10,14 @@ you'll need to:
 * Set up a Spider cluster
 * Run the client
 
-The example source code for this guide is in `examples/quick-start`.
+The example source code for this guide is in [examples/quick-start].
+
+:::{note}
+In the rest of this guide:
 
-> [!NOTE] In the rest of this guide:
-> 1. we specify source file paths relative to `examples/quick-start`.
-> 2. all CMake commands should be run from inside `examples/quick-start`.
+1. we specify source file paths relative to `examples/quick-start`.
+2. all CMake commands should be run from inside `examples/quick-start`.
+:::
 
 # Requirements
 
@@ -39,15 +39,25 @@ In Spider, a task is a C++ function that satisfies the following conditions:
   * All other parameters must have types that conform to the `Serializable` or `Data` interfaces.
 * It returns a value that conforms to the `Serializable` or `Data` interfaces.
 
-> [!NOTE]
-> You don't immediately need to understand the TaskContext, Serializable, or Data types as we'll
-> explain them in other guides.
+:::{note}
+You don't immediately need to understand the TaskContext, Serializable, or Data types as we'll
+explain them in other guides.
+:::
 
-For example, the task in `src/tasks.cpp` computes and returns the sum of two integers.
+For example, the task in `src/tasks.cpp` computes and returns the sum of two integers:
 
-> [!NOTE]
-> The task is split into a header file and an implementation file so that it can be loaded as a
-> library in the worker, as we'll see in later sections.
+:::{literalinclude} ../../../examples/quick-start/src/tasks.cpp
+:caption: src/tasks.cpp: The example task.
+:language: cpp
+:lines: 5-12
+:lineno-start: 5
+:linenos: true
+:::
+
+:::{note}
+The task is split into a header file and an implementation file so that it can be loaded as a
+library in the worker, as we'll see in later sections.
+:::
 
 The integer parameters and return value are `Serializable` values.
 
@@ -76,13 +86,22 @@ To make Spider to run a task, we first need to write a client application. Gener
 4. and then handles the result.
 
 For example, the client in `src/client.cpp` runs the `sum` task from the previous section and
-verifies its result.
+verifies its result:
+
+:::{literalinclude} ../../../examples/quick-start/src/client.cpp
+:caption: src/client.cpp: A snippet of the example client.
+:language: cpp
+:lines: 24-35
+:lineno-start: 24
+:linenos: true
+:::
 
 When we submit a task to Spider, Spider returns a `Job`, which represents a scheduled, running, or
 completed task (or `TaskGraph`) in a Spider cluster.
 
-> [!NOTE]
-> `Job`s and `TaskGraph`s will be explained in another guide.
+:::{note}
+`Job`s and `TaskGraph`s will be explained in another guide.
+:::
 
 # Building the client
 
@@ -122,12 +141,14 @@ docker run \
         --publish 3306:3306 mariadb:latest
 ```
 
-> [!WARNING]
-> When the container above is stopped, the database will be deleted. In production, you should set
-> up a database instance with some form of data persistence.
+:::{warning}
+When the container above is stopped, the database will be deleted. In production, you should set up
+a database instance with some form of data persistence.
+:::
 
-> [!WARNING]
-> The container above is using hardcoded default credentials that shouldn't be used in production.
+:::{warning}
+The container above is using hardcoded default credentials that shouldn't be used in production.
+:::
 
 Alternatively, if you have an existing MySQL/MariaDB instance, you can use that as well. Simply
 create a database and authorize a user to access it.
@@ -183,9 +204,10 @@ NOTE:
 * You can specify multiple task libraries to load. The task libraries must be built with linkage
   to the Spider client library. 
 
-> [!TIP]
-> You can start multiple workers to increase the number of concurrent tasks that can be run on the
-> cluster.
+:::{tip}
+You can start multiple workers to increase the number of concurrent tasks that can be run on the
+cluster.
+:::
 
 # Running the client
 
@@ -207,3 +229,4 @@ support for fault tolerance.
 
 [Docker]: https://docs.docker.com/engine/install/
 [docker-non-root]: https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user
+[examples/quick-start]: https://github.com/y-scope/spider/tree/main/examples/quick-start

docs/src/user-docs/index.md

@@ -0,0 +1,24 @@
+# User docs
+
+This section contains docs for using and operating Spider. Choose one of the sections below or use
+the left sidebar (if it's hidden, click the <i class="fa fa-bars"></i> icon) to navigate to specific
+docs.
+
+::::{grid} 1 1 2 2
+:gutter: 2
+
+:::{grid-item-card}
+:link: guides-overview
+Guides
+^^^
+Guides for using and operating Spider.
+:::
+::::
+
+:::{toctree}
+:hidden:
+:caption: Guides
+
+guides-overview.md
+guides-quick-start.md
+:::