Merge pull request #142 from Point72/pavithraes/docs-restructure

Documentation Restructure

Author: robambalu

docs/wiki/0.-Introduction.md

@@ -1,70 +1,38 @@
-`csp` ("Composable Stream Processing") is a functional-like reactive
-language that makes time-series stream processing simple to do.  The
-main reactive engine is a C++ based engine which has been exposed to
-python ( other languages may optionally be extended in future versions
-). `csp` applications define a connected graph of components using a
-declarative language (which is essentially python).  Once a graph is
-constructed it can be run using the C++ engine. Graphs are composed of
-some number of "input" adapters, a set of connected calculation "nodes"
-and at the end sent off to "output" adapters. Inputs as well as the
-engine can be seamlessly run in simulation mode using historical input
-adapters or in realtime mode using realtime input adapters.
+<picture>
+  <source media="(prefers-color-scheme: light)" srcset="https://raw.githubusercontent.com/Point72/csp/main/docs/img/csp-light.png">
+  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/Point72/csp/main/docs/img/csp-dark.png">
+  <img alt="CSP logo mark - text will be black in light color mode and white in dark color mode." width="50%"/>
+</picture>
 
-# Contents
+CSP (Composable Stream Processing) is a library for high-performance real-time event stream processing in Python.
 
-- [0. Introduction](https://github.com/Point72/csp/wiki/0.-Introduction)
-- [1. Generic Nodes (csp.baselib)](<https://github.com/Point72/csp/wiki/1.-Generic-Nodes-(csp.baselib)>)
-- [2. Math Nodes (csp.math)](<https://github.com/Point72/csp/wiki/2.-Math-Nodes-(csp.math)>)
-- [3. Statistics Nodes (csp.stats)](<https://github.com/Point72/csp/wiki/3.-Statistics-Nodes-(csp.stats)>)
-- [4. Random Time Series Generation](<https://github.com/Point72/csp/wiki/4.-Random-Time-Series-Generation-(csp.random)>)
-- [5. Adapters](https://github.com/Point72/csp/wiki/5.-Adapters)
-- [6. Dynamic Graphs](https://github.com/Point72/csp/wiki/6.-Dynamic-Graphs)
-- [7. csp.Struct](https://github.com/Point72/csp/wiki/7.-csp.Struct)
-- [8. Profiler](https://github.com/Point72/csp/wiki/8.-Profiler)
-- [9. Caching](https://github.com/Point72/csp/wiki/9.-Caching)
+## Key Features
 
-# Installation
+- **Powerful C++ Engine:** Execute the graph using CSP's C++ Graph Processing Engine
+- **Simulation (i.e., offline) mode:** Test workflows on historical data and quickly move to real-time data in deployment
+- **Infrastructure-agnostic:** Connect to any data format or storage database, using built-in (Parquet, Kafka, etc.) or custom adapters
+- **Highly-customizable:** Write your own input and output adapters for any data/storage formats, and real-time adapters for specific workflows
+- **PyData interoperability:** Use your favorite libraries from the Scientific Python Ecosystem for numerical and statistical computations
+- **Functional/declarative style:** Write concise and composable code for stream processing by building graphs in Python
 
-We ship binary wheels to install `csp`  on MacOS and Linux via `pip`:
+<!-- ## Applications -->
 
-```bash
-pip install csp
-```
+## Get Started
 
-Other platforms will need to see the instructions to [build `csp` from
-source](https://github.com/Point72/csp/wiki/98.-Building-From-Source).
+- [Install CSP](Installation) and [write your first CSP program](First-Steps)
+- Learn more about [nodes](CSP-Node), [graphs](CSP-Graph), and [execution modes](Execution-Modes)
+- Learn to extend CSP with [adapters](Adapters)
 
-We plan to create conda packages on conda-forge and ship binaries for Windows in
-the near future.
+<!-- - Check out the [examples](Examples) for various CSP features and use cases -->
 
-# Contributing
+> \[!TIP\]
+> Find relevant docs with GitHub’s search function, use `repo:Point72/csp type:wiki <search terms>` to search the documentation Wiki Pages.
 
-Contributions are welcome on this project. We distribute under the terms of the [Apache 2.0 license](https://github.com/Point72/csp/blob/main/LICENSE).
+## Community
 
-For **bug reports** or **small feature requests**, please open an issue on our [issues page](https://github.com/Point72/csp/issues).
+- [Contribute](Contribute) to CSP and help improve the project
+- Read about future plans in the [project roadmap](Roadmap)
 
-For **questions** or to discuss **larger changes or features**, please use our [discussions page](https://github.com/Point72/csp/discussions).
+## License
 
-For **contributions**, please see our [developer documentation](https://github.com/Point72/csp/wiki/99.-Developer). We have `help wanted` and `good first issue` tags on our issues page, so these are a great place to start.
-
-For **documentation updates**, make PRs that update the pages in `/docs/wiki`. The documentation is pushed to the GitHub wiki automatically through a GitHub workflow. Note that direct updates to this wiki will be overwritten.
-
-# Roadmap
-
-We do not have a formal roadmap, but we're happy to discuss features, improvements, new adapters, etc, in our [discussions area](https://github.com/Point72/csp/discussions). Here are some high level items we hope to accomplish in the next few months:
-
-- Support `clang` compiler and full MacOS support ([#33](https://github.com/Point72/csp/issues/33) / [#132](https://github.com/Point72/csp/pull/132))
-- Support `msvc` compiler and full Windows support ([#109](https://github.com/Point72/csp/issues/109))
-- Establish a better pattern for adapters ([#165](https://github.com/Point72/csp/discussions/165))
-
-## Adapters and Extensions
-
-- Redis Pub/Sub Adapter with [Redis-plus-plus](https://github.com/sewenew/redis-plus-plus) ([#61](https://github.com/Point72/csp/issues/61))
-- C++-based websocket adapter
-  - Client adapter in [#152](https://github.com/Point72/csp/pull/152)
-- C++-based HTTP/SSE adapter
-- Add support for other graph viewers, including interactive / standalone / Jupyter
-
-## Other Open Source Projects
-
-- `csp-gateway`: Application development framework, built with [FastAPI](https://fastapi.tiangolo.com) and [Perspective](https://github.com/finos/perspective). This is a library we have built internally at Point72 on top of `csp` that we hope to open source later in 2024. It allows for easier construction of modular `csp` applications, along with a pluggable REST/WebSocket API and interactive UI.
+CSP is licensed under the Apache 2.0 license. See the [LICENSE](https://github.com/Point72/csp/blob/main/LICENSE) file for details.

docs/wiki/5.-Adapters.md

@@ -0,0 +1 @@
+_This wiki is autogenerated. To made updates, open a PR against the original source file in [`docs/wiki`](https://github.com/Point72/csp/tree/main/docs/wiki)._

docs/wiki/6.-Dynamic-Graphs.md

@@ -0,0 +1,61 @@
+<!--
+This sidebar is displayed on the GitHub Wiki section instead of the default sidebar.
+Notes for editors:
+- Ensure links don't have the file extensions (i.e., `.md`)
+- Do not use colons (':') in page titles, they don't render properly as links in the sidebar
+- Use only the filenames in this page (without the filepath and file extension)
+-->
+
+**[Home](Home)**
+
+**Get Started (Tutorials)**
+
+- [Installation](Installation)
+- [First steps](First-Steps)
+
+<!-- Add more pages when ready -->
+
+**Concepts**
+
+- [CSP Node](CSP-Node)
+- [CSP Graph](CSP-Graph)
+- [Historical Buffers](Historical-Buffers)
+- [Execution Modes](Execution-Modes)
+- [Adapters](Adapters)
+
+**How-to guides**
+
+- [Use Statistical Nodes](Use-Statistical-Nodes)
+- Use Adapters (coming soon)
+- [Add Cycles in Graphs](Add-Cycles-in-Graphs)
+- [Create Dynamic Baskets](Create-Dynamic-Baskets)
+- Write Adapters:
+  - [Write Historical Input Adapters](Write-Historical-Input-Adapters)
+  - [Write Realtime Input Adapters](Write-Realtime-Input-Adapters)
+  - [Write Output Adapters](Write-Output-Adapters)
+- [Profile CSP Code](Profile-CSP-Code)
+
+**References**
+
+- API Reference
+  - [Base Nodes API](Base-Nodes-API)
+  - [Base Adapters API](Base-Adapters-API)
+  - [Math and Logic Nodes API](Math-and-Logic-Nodes-API)
+  - [Statistical Nodes API](Statistical-Nodes-API)
+  - [Functional Methods API](Functional-Methods-API)
+  - [Adapters (Kafka, Parquet, DBReader) API](Input-Output-Adapters-API)
+  - [Random Time Series Generators API](Random-Time-Series-Generators-API)
+  - [`csp.Struct` API](csp.Struct-API)
+  - [`csp.dynamic` API](csp.dynamic-API)
+  - [`csp.profiler` API](csp.profiler-API)
+- [Examples](Examples)
+- [Glossary of Terms](Glossary)
+
+**Developer Guide**
+
+- [Contributing](Contribute)
+- [Development Setup](Local-Development-Setup)
+- [Build CSP from Source](Build-CSP-from-Source)
+- [GitHub Conventions (for maintainers)](GitHub-Conventions)
+- [Release Process (for maintainers)](Release-Process)
+- [Roadmap](Roadmap)

docs/wiki/9.-Caching.md

@@ -0,0 +1,110 @@
+`csp.baselib` defines some generally useful adapters, which are also imported directly into the CSP namespace when importing CSP.
+
+These are all graph-time constructs.
+
+## Table of Contents
+
+- [Table of Contents](#table-of-contents)
+- [`csp.timer`](#csptimer)
+- [`csp.const`](#cspconst)
+- [`csp.curve`](#cspcurve)
+- [`csp.add_graph_output`](#cspadd_graph_output)
+- [`csp.feedback`](#cspfeedback)
+
+## `csp.timer`
+
+```python
+csp.timer(
+    interval: timedelta,
+    value: '~T' = True,
+    allow_deviation: bool = False
+)
+```
+
+This will create a repeating timer edge that will tick on the given `timedelta` with the given value (value defaults to `True`, returning a `ts[bool]`)
+
+Args:
+
+- **`interval`**: how often to tick value
+- **`value`**: the actual value that will tick every interval (defaults to the value `True`)
+- **`allow_deviation`**: When running in realtime the engine will ensure timers execute exactly when they requested on their intervals.
+  If your engine begins to lag, timers will still execute at the expected time "in the past" as the engine catches up
+  (imagine having a `csp.timer` fire every 1/2 second but the engine becomes delayed for 1 second.
+  By default the half seconds will still execute until time catches up to wallclock).
+  When `allow_deviation` is `True`, and the engine is in realtime mode, subsequent timers will always be scheduled from the current wallclock + interval,
+  so they won't end up lagging behind at the expensive of the timer skewing.
+
+## `csp.const`
+
+```python
+csp.const(
+    value: '~T',
+    delay: timedelta = timedelta()
+)
+```
+
+This will create an edge that ticks one time with the value provided.
+By default this will tick at the start of the engine, delta can be provided to delay the tick
+
+## `csp.curve`
+
+```python
+csp.curve(
+    typ: 'T',
+    data: typing.Union[list, tuple]
+)
+```
+
+This allows you to convert a list of non-CSP data into a ticking edge in CSP
+
+Args:
+
+- **`typ`**: is the type of the value of the data of this edge
+- **`data`**: is either a list of tuples of `(datetime, value)`, or a tuple of two equal-length numpy ndarrays, the first with datetimes and the second with values.
+  In either case, that will tick on the returned edge into the engine, and the data must be in time order.
+  Note that for the list of tuples case, you can also provide tuples of (timedelta, value) where timedelta will be the offset from the engine's start time.
+
+## `csp.add_graph_output`
+
+```python
+csp.add_graph_output(
+    key: object,
+    input: ts['T'],
+    tick_count: int = -1,
+    tick_history: timedelta = timedelta()
+)
+```
+
+This allows you to connect an edge as a "graph output".
+All edges added as outputs will be returned to the caller from `csp.run` as a dictionary of `key: [(datetime, value)]`
+(list of datetime, values that ticked on the edge) or if `csp.run` is passed `output_numpy=True`, as a dictionary of
+`key: (array, array)` (tuple of two numpy arrays, one with datetimes and one with values).
+See [Collecting Graph Outputs](https://github.com/Point72/csp/wiki/0.-Introduction#collecting-graph-outputs)
+
+Args:
+
+- **`key`**: key to return the results as from `csp.run`
+- **`input`**: edge to connect
+- **`tick_count`**: number of ticks to keep in the buffer (defaults to -1 - all ticks)
+- **`tick_history`**: amount of ticks to keep by time window (defaults to keeping all history)
+
+## `csp.feedback`
+
+```python
+csp.feedback(typ)
+```
+
+`csp.feedback` is a construct that can be used to create artificial loops in the graph.
+Use feedbacks in order to delay bind an input to a node in order to be able to create a loop
+(think of writing a simulated exchange that takes orders in and needs to feed responses back to the originating node).
+
+`csp.feedback` itself is not an edge, its a construct that allows you to access the delayed edge / bind a delayed input.
+
+Args:
+
+- **`typ`**: type of the edge's data to be bound
+
+Methods:
+
+- **`out()`**: call this method on the feedback object to get the edge which can be wired as an input
+- **`bind(x: ts[object])`**: call this to bind an edge to the feedback